US20110185269A1 - Audio/visual editing tool - Google Patents
Audio/visual editing tool Download PDFInfo
- Publication number
- US20110185269A1 US20110185269A1 US13/079,565 US201113079565A US2011185269A1 US 20110185269 A1 US20110185269 A1 US 20110185269A1 US 201113079565 A US201113079565 A US 201113079565A US 2011185269 A1 US2011185269 A1 US 2011185269A1
- Authority
- US
- United States
- Prior art keywords
- audio
- visual
- timeline
- data sequence
- event
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/91—Television signal processing therefor
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
- G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/79—Processing of colour television signals in connection with recording
- H04N9/80—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
- H04N9/82—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only
- H04N9/8205—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only involving the multiplexing of an additional signal and the colour video signal
Definitions
- Non-expert video editors are likely to fail at producing video productions that fulfill their personal expectations, or the expectations of their highly-conditioned audience.
- Non-expert users of non-linear video editing (e.g., digital video editing) systems typically cannot create attractive looking and sounding videos.
- non-expert users Unlike professional video editors who have knowledge and experience in making effective editing decisions, non-expert users have little to no experience, and would therefore benefit greatly from a coach.
- Such non-expert video editors may need editing assistance to create higher-quality products.
- Conventional media editing tools provide users with the ability to capture, edit, import, and modify both visual and audio media for the development of audio/visual works.
- a user can create an audio/visual work with a media editing tool (e.g., Microsoft® Windows Movie Maker Version 2.1) and the media editing tool can provide the user with several functions, including trimming, splicing, and cropping video, merging image, audio, and video media, adding transitions and effects between constituent parts, and overlaying additional audio, to name a few.
- a media editing tool e.g., Microsoft® Windows Movie Maker Version 2.1
- the media editing tool can provide the user with several functions, including trimming, splicing, and cropping video, merging image, audio, and video media, adding transitions and effects between constituent parts, and overlaying additional audio, to name a few.
- such a conventional system can provide a means for merging several different types of media together into a unified work, different than all of its constituent portions.
- aspects of the invention provide for analyzing data sequences and extracting metadata from the data sequences to provide information related to events of the data sequences. These events provide specific editing guidance to a user.
- aspects of the invention permit the dissemination of information regarding key events and features of the data sequences for use by a user in editing the sequences.
- determining which of such metadata occur at substantially the same time and associating the metadata with a timeline provide information to users related to the coordination of editing events from different data sequences, thereby facilitating editing by a user.
- FIG. 1 is a diagram of an aggregate timeline and its constituent parts of one embodiment of the invention
- FIG. 2 is a flow diagram of a method of one embodiment of the invention.
- FIG. 3 is an extension of the flow diagram of Fig, 2 ;
- FIG. 4 is another flow diagram of a method of another embodiment of the invention.
- FIG. 5 is an exemplary system memory of another embodiment of the invention.
- FIG. 6 is an exemplary data record of another embodiment of the invention.
- FIG. 7 is a block diagram illustrating one example of a suitable computing system environment in which one embodiment of the invention may be implemented.
- the constituent parts of the aggregate timeline 21 include several objects commonly associated with media editing. Those objects may include but are not limited to textual messages TE 1 , TE 2 (e.g., titles, credits, presentation information, etc.), video segments VS 1 , VS 2 , VS 3 , VS 4 (e.g., video clips, animations, etc.), digital images I 1 , I 2 (e.g., digital photographs, documents, screenshots, and computer-generated images, etc.), and transitions, or effects, between other objects T 1 , T 2 , T 3 , T 4 , and T 5 .
- textual messages TE 1 , TE 2 e.g., titles, credits, presentation information, etc.
- video segments VS 1 , VS 2 , VS 3 , VS 4 e.g., video clips, animations, etc.
- digital images I 1 , I 2 e.g., digital photographs, documents, screenshots, and computer-generated images, etc.
- transitions, or effects
- the aggregate timeline 21 further includes blank space BS as an object for holding timeline space open, without any objects.
- media editing software embodying aspects of the invention analyze multiple data sequences according to characteristics to extract metadata from each sequence.
- This metadata might include a transition to a new video scene, a new face entering a video scene, or a music beat in an audio sequence, among others, as described below.
- This permits the dissemination of metadata information regarding events and features of the data sequences for use in editing the sequences.
- Providing such metadata to a user will assist in the editing process, as the use of such metadata can offer convenient and effective editing choices (e.g., for a novice use).
- determining which of such metadata occur at substantially the same time and associating the metadata with a timeline provide information related to the coordination of editing events from different data sequences, which is also helpful when editing.
- the diagram also illustrates a non-aggregated timeline, generally indicated 23 , which depicts each of the objects separately.
- a timeline is particularly useful for understanding the details of how a media production involving multiple media objects and transitions may be constructed.
- the first textual message TE 1 extends from the beginning, or left edge, of the timeline until the end, or right edge, of the first transition T 1 .
- the first video segment VS 1 extends from the beginning, or left edge, of the first transition T 1 until the end, or right edge of the second transition T 2 .
- the viewer of the media would begin by seeing primarily the first textual message.
- each of the video segments VS 1 , VS 2 , VS 3 , VS 4 and digital images I 1 , 12 is depicted as staggered from adjacent video segments and digital images to facilitate viewing of the entire scope of the video segment or image, without overlap.
- Audio track 25 A depicted in the diagram is the aggregation of the audio associated with each of the video segments VS 1 , VS 2 , VS 3 , VS 4 and digital images I 1 , 12 , described above.
- the audio track 25 A depicts the intensity of the aggregated audio over time, such that the intensity of the audio—and indirectly the activities of the video segments VS 1 , VS 2 , VS 3 , VS 4 and the digital images I 1 , I 2 —can be understood as a function of time.
- Audio track 25 B (e.g., a musical overlay) included in the diagram depicts the intensity of music currently present in the timeline 23 .
- the music track 25 B depicts the intensity of the music track or tracks over time, thereby providing an indication of the nature of the music.
- Overlay track 25 C depicted in the diagram depicts items that may be composited, or added, onto the audio/visual work over time. The duration, blending, and intensity of each is specified over time. Examples of these items are text that may be used for titles and captions, other images, animations, or video.
- Other audio characteristics, other than audio intensity can also be depicted on the aural tracks 25 , without departing from the scope of the embodiments of the present invention.
- other aural tracks 25 can be included without departing from the scope of the embodiments of the present invention.
- both video segments VS and still digital images I are included.
- the timeline may contain any number and typed of items.
- the timeline may include at least one of video, video with associated audio, images, audio, music, text, overlay text, images, animations, and other text, without departing from the scope of the present invention.
- the timeline can contain only video segments VS, or only digital images I, without departing from the scope of the embodiments of the present invention.
- digital images can include digital photographs, but can also include other digital items, including documents, scanned images, and computer-generated images, to name a few.
- a system flow diagram, generally indicated 29 depicts one embodied method of the present invention.
- the system flow begins with raw components, generally indicated 31 , as described generally above.
- the raw components include a video segment VS, an audio track 25 A associated with the video segment, and a music track 25 B.
- each of these raw components 31 can be referred to more generally as data sequences, such as a visual data sequence VS, an audio data sequence 25 A, and a second, or overlay, audio data sequence 25 B.
- These raw components 31 are each sent to an analysis module, generally indicated 33 , which analyzes the underlying data sequences of the raw components.
- the video segment VS is sent to a video and image analysis submodule 33 A
- the audio track 25 A is sent to an audio analysis submodule 33 B
- the music track 25 B is sent to a second audio analysis submodule 33 C.
- the analysis module 33 analyzes the underlying data sequences VS, 25 A, 25 B and extracts metadata related to the data sequences. This metadata can be useful to an editor, for example a novice editor, in selecting appropriate locations for particular editing decisions, as discussed below.
- the audio analysis submodule 33 B is a speech analysis submodule.
- the second audio analysis submodule 33 C is a music analysis submodule, and more particularly a music beat and audio peak analysis submodule. It should also be noted that other submodules may be added to perform additional analyses without departing from the scope of the present invention.
- a storage area receives the extracted metadata from the video and image analysis submodule 33 A
- an audio analysis storage area 35 B receives the extracted metadata from the audio analysis submodule 33 B
- a second audio analysis storage area 35 C receives the extracted metadata from the second audio analysis submodule 33 C.
- the storage areas 35 A, 35 B, and 35 C are depicted as separate in FIG. 2 , it should be understood by one skilled in the art that two or more of those storage areas can be combined without departing from the scope of the embodiments of the present invention.
- an analysis engine 37 receives the stored metadata and analyzes the metadata. As discussed below in greater detail, the analysis engine 37 analyzes the content of the data sequences VS, 25 A, 25 B and makes meaningful editing suggestions, in the form of events, based upon the actual content of the data sequence (e.g., scene changes, music beats, audio peaks, spaces between spoken words, etc.). In one alternative embodiment, the analysis engine 37 determines the intersection of metadata events occurring at substantially the same time within different raw components 31 and associates such metadata with a common timeline associated with the raw components. The results of this analysis are stored in storage area 39 .
- the analysis engine 37 is also responsible for displaying the metadata and the results of its analysis, such as in depicted in the user interface, generally indicated 41 , of FIG. 3 .
- the editing suggestions noted above are displayed in the timeline of the user interface 41 as “snap-to-points” (also known as “sticky points”) corresponding to the events.
- the analysis engine 37 is adapted to receive control parameters from the user via the user interface 41 of FIG. 3 . Such control parameters can also be stored in the storage area 39 .
- FIG. 3 is an extension of the flow diagram of FIG. 2 .
- the exemplary user interface 41 is in the form of a timeline, including each of the data sequences VS, 25 A, 25 B depicted on the same timeline. This allows the user to readily determine what portion of each data sequence VS, 25 A, 25 B is utilized at any given time. The details of the user interface 41 will be discussed in greater detail below.
- a method for analyzing data sequences (such as data sequences VS, 25 A, 25 B introduced above) and extracting metadata associated with the data sequences for providing information related to events of the data sequences is generally indicated 45 in FIG. 4 .
- the method comprises analyzing, at 47 , a visual data sequence VS according to at least one characteristic of the visual data sequence and extracting, also at 47 , metadata associated with the visual data sequence according to the at least one characteristic of the visual data sequence.
- the visual data sequence VS is analyzed, at 47 , according to at least one visual characteristic. Moreover, the metadata associated with the visual data sequence VS is extracted according to the visual characteristic of the visual data sequence.
- visual characteristics of the video data sequence can comprise video dynamic peaks, dynamic image changes, color entropy characteristics, chroma values and patterns, luma values and patterns, and image pattern recognition, among others.
- video dynamic peaks can comprise detectable changes in the visual data sequence VS, such as scene changes (e.g., different scenes stored adjacent one another in the visual data sequence) and stark image transitions (e.g., large changes in brightness, large changes in color).
- dynamic image changes can comprise detectable changes in the visual data sequence VS (e.g., quick zooming, quick panning, removal of the lens cap, among others).
- Recognition of chroma values and patterns can detect the type of video, such as sports (e.g., baseball) or other known video pattern characteristics.
- the following patent application describes how to perform detection of the video type: U.S. patent application entitled Video Search and Services, filed Feb. 27, 2006, assigned to Microsoft Corporation of Redmond, Washington, U.S.A.
- the image pattern recognition visual characteristic can comprise recognition of particular images, such as people generally, a particular person, a particular face (i.e., face patterns), animals generally, a particular animal (e.g., a dog), and other items (e.g., a boat, a car, etc.) among others.
- aspects of the invention may allow the user to select one or more images for pattern recognition analysis, such as from a drop-down menu.
- visual data sequence VS is analyzed, at 47 , based upon its content to determine if metadata related to the content can be extracted, also at 47 , such as where detectable changes in the data sequence occur.
- the following patent application describes how to perform face detection: U.S.
- the method 45 further comprises analyzing, at 49 , an audio data sequence 25 A, substantially corresponding to the visual data sequence VS, according to at least one characteristic of the audio data sequence and extracting, also at 49 , metadata associated with the audio data sequence according to at least one characteristic of the audio sequence data.
- the audio data sequence 25 A substantially corresponding to the visual data sequence VS may simply mean that the data sequences are related to one another.
- the typical audio recorded along with a video recording would be such an audio data sequence 25 A substantially corresponding to the visual data sequence VS, or video recording.
- the audio data sequence 25 A is analyzed, at 49 , according to at least one audio characteristic. Moreover, the metadata associated with the audio data sequence 25 A is extracted, at 49 , according to the at least one audio characteristic of the audio data sequence.
- audio characteristics of the audio data sequence 25 A can comprise music beats, audio dynamic peaks, speech characteristics, changes in the person speaking, recorded sounds, word boundary detection, and word and phrase detection (e.g., identify all occurrences of the phrase “Happy Birthday”), among others.
- the audio data sequence 25 A can be analyzed according to any characteristic that can yield metadata regarding events of interest in the audio data sequence.
- the audio data sequence 25 A can be analyzed to create a metadata event at significant music beats, or at all music beats. Identification of such events related to music beats can be useful to an editor, as the introduction of a new scene or new camera angle is often timed to coincide with the beat of background music.
- audio dynamic peaks such as loud noises or other sharp increases and subsequent decreases in audio intensity, or volume, that are part of the audio data sequence can be identified as potential editing events.
- One skilled in the art would readily understand the other audio characteristics identified above. For example, the following patent application describes how to segment and classify an audio data sequence: U.S. patent application entitled Audio Segmentation and Classification, filed Apr. 19, 2000, issued as U.S. Pat. No.
- the method 45 also comprises analyzing, at 51 , a second audio data sequence 25 B according to at least one characteristic of the second audio data sequence and extracting, also at 51 , metadata associated with the second audio data sequence according to at least one characteristic of the second audio data sequence.
- the second audio data sequence 25 B does not necessarily substantially correspond to the visual data sequence VS.
- both audio data sequences 25 A, 25 B will correspond to the video data sequence.
- the extracting, at 51 extracts metadata associated with a second audio data sequence 25 B not in substantial correspondence with the visual data sequence VS.
- the extracting metadata, at 51 associated with the second audio data sequence 25 B can comprise extracting metadata associated with an audio overlay, such as a musical piece (e.g., a song) or vocal narration.
- the second audio data sequence 25 B is analyzed, at 51 , according to at least one audio characteristic of the second audio data sequence.
- the metadata associated with the second audio data sequence 25 B is extracted, at 51 , according to the at least one audio characteristic of the second audio data sequence.
- audio characteristics of the second audio data sequence 25 B can comprise music beats, audio dynamic peaks, speech characteristics, particular recorded sounds, word boundary detection, and word and phrase detection, among others, generally as discussed above with respect to the audio data sequence 25 A.
- an intersection can be determined.
- this metadata intersection event may be more useful that a solitary metadata event because the intersection event brings together features of interest from two or more data sequences, or parts of the final production.
- the method further aggregates, at 55 , metadata and metadata intersections.
- the method 45 further comprises associating, at 57 , metadata and metadata intersections with a timeline associated with the data sequences.
- This association, at 57 places each piece of extracted metadata and determined metadata intersection into a common timeline.
- This association with a common timeline provides for ready review of all the metadata and metadata intersections by the user of one or more of the metadata events, whereby further editing decisions are based upon the proximity, frequency, and density of the metadata associated with the timeline.
- the method further comprises rendering, at 59 , a user interface depicting the timeline with the data sequences VS, 25 A, 25 B, the metadata, and the intersections of metadata.
- the rendered user interface 41 (see FIG. 3 ) is a useful tool for editing, as it incorporates all of the extracted metadata and metadata intersections into a single interface for review by the user. The details of the user interface 41 will be discussed in greater detail below.
- the method may further comprise automatically rendering, at 60 , a resultant audio/visual work. For example, without user direction, the method may utilize the previous analyzing and extracting 47 , 49 , 51 , determining 53 , aggregating 55 , and associating 57 processes to render 60 a resultant audio-visual work automatically. Such an automatic rendering may be particularly useful for a user seeking a completed audio/visual work with relatively little effort and in as short a time period as possible.
- the method may further comprise allowing, at 62 , the user to modify the resultant audio/visual work.
- the method may further comprise receiving, at 61 , user input regarding changes in one or more of the data sequences VS, 25 A, 25 B via the user interface 41 .
- the user can elect to change any number of parameters, including moving, modifying, and deleting one or more of the data sequences VS, 25 A, 25 B.
- the user can elect to add one or more additional data sequences.
- changes of this type will often change the location of metadata events and metadata intersections, requiring a new set of analysis, extractions, and determinations, generally as set forth above.
- the user can decide to crop a particular portion of the visual data sequence VS and the associated portion of the audio data sequence 25 A, thereby causing substantial changes in the location of metadata events.
- the method repeats the analyzing and extracting 47 , 49 , 51 metadata associated with each of the data sequences, the determining 53 , the aggregating 55 , the associating 57 , and the rendering 59 to ensure that the user interface 41 rendered is reflective of the location of metadata and metadata intersections created during the latest changes.
- a user interface 41 for integrating visual and audio data sequences VS, 25 A, 25 B together for creating an audio-visual work is depicted in FIG. 3 .
- the user interface 41 includes a timeline, generally indicated 61 , extending left to right and demarking the time for each of the data sequences on the timeline 61 .
- the scale of the timeline 61 can be adjusted to suit the user, such as to focus in on a particular portion of interest, or to review a greater portion of the timeline 61 for a more general view.
- the timeline 61 extends from about 0 seconds to about 32 seconds. Any scale can be used without departing from the scope of the embodiments of the present invention.
- the user interface 41 further comprises a visual timeline VT associated with the visual data sequence VS comprising one or more visual elements.
- the visual timeline VT depicts the beginning and the end of each of the one or more visual elements.
- a first video segment VS 1 extends from about zero seconds to about 22.5 seconds
- a second video segment VS 2 extends from about 20.8 seconds and at least until the end of the visible visual timeline VT at about 32 seconds.
- a first transition T 1 between the first video segment VS 1 and the second video segment VS 2 is a first transition T 1 between the first video segment VS 1 and the second video segment VS 2 .
- the visual timeline VT of the embodiments of the present invention further include visual event markers, generally indicated 65 , indicating the corresponding time on the timeline 61 of a particular visual event in the visual data sequence VS.
- the visual event markers 65 provide visual indications regarding events within the visual data sequence VS that are helpful to users when editing.
- the visual event markers 65 further comprise at least one of an icon 67 indicative of the nature of the corresponding event and a value indicator 69 indicative of the relative value of the corresponding event.
- the icons 67 corresponding to the visual event markers 65 can depict any class of events, or individual events.
- a new face icon, a face exit icon, and a new scene icon are included. Two of those, a new face icon 67 A and a new scene icon 67 B, are depicted in the example of FIG. 3 .
- a new face icon 67 A appears the editor knows that a new person has entered the scene
- a new scene icon 67 B appears, the editor knows that a new scene has begun. This is important information to the editor, which can now be used as a basis for other editing decisions. With conventional editing systems, the user would need to carefully review the visual data sequence VS manually to determine such editing events.
- the automatic extraction and rendering of these events on a user interface 41 provides this information to an editor directly.
- Visual demonstratives other than icons 67 are also contemplated as within the scope of the embodiments of the present invention (e.g., user-configurable thumbnails, including ones representative of the items detected, such as specific faces, boats, cars, etc.).
- the exemplary user interface 41 of FIG. 3 may also include a tile (video) overlay visual demonstrative (not shown). Such demonstratives would be depicted similar to the visual demonstratives of the first video segment VS 1 .
- the audio timeline AT further includes audio event markers, generally indicated 71 , indicating the corresponding time on the timeline 61 of a particular audio event in the audio data sequence 25 A.
- the audio event markers 71 provide visual indications regarding events within the audio data sequence 25 A that are helpful during editing.
- the audio event markers 71 further comprise at least one of an icon 73 indicative of the nature of the corresponding event and a value indicator 75 indicative of the relative value of the corresponding event.
- the icons 73 and value indicators 75 function similarly to those described above with respect to the visual data sequence VS.
- an end of phrase icon For example, an end of phrase icon, a beginning of phrase icon, an audio peak icon, a silence icon, a music beat icon, and a dynamic audio change icon are common examples.
- an end of phrase icon 73 is depicted in FIG. 3 .
- an end of phrase icon 67 A appears, the editor knows that a person has stopped speaking, which can correspond to a convenient edit point.
- the user interface 41 further comprises an overlay audio timeline OAT associated with an overlay audio data sequence 25 B comprising one or more overlay audio elements.
- the overlay audio timeline OAT corresponds to the same time scale and position in time as the visual timeline VT and the audio timeline AT.
- the overlay audio timeline OAT depicts at least one audio characteristic of the overlay audio data sequence over time, such as the output level of the overlay audio data sequence 25 B over time.
- the overlay audio data sequence 25 B is a musical work, such as a song, which can be used in conjunction with the video data sequence VS and the audio data sequence 25 A.
- an audio segment AS extends from about zero seconds to about 32 seconds. Multiple audio segments AS can be utilized without departing from the scope of the embodiments of the present invention.
- the overlay audio timeline OAT further includes overlay audio event markers, generally indicated 81 , indicating the corresponding time on the timeline of a particular overlay audio event in the overlay audio data sequence 25 B.
- the overlay audio event markers 81 provide visual indications regarding events within the overlay audio data sequence 25 B, as did the event markers described above with respect to their data sequence.
- the overlay audio event markers 81 further comprise at least one of an icon 83 indicative of the nature of the corresponding event and a value indicator 85 indicative of the relative value of the corresponding event. It should be noted here that each occurrence of an event marker 81 , an icon 83 , or a value indicator 85 is not marked with a reference numeral in FIG. 3 . Several reference numerals were intentionally not included to decrease clutter in the view of the overlay audio timeline OAT.
- the icons 83 and value indicators 85 function similarly to those described above with respect to the visual data sequence VS.
- an end of phrase icon, a beginning of phrase icon, an audio peak icon, a silence icon, a music beat icon, and a dynamic audio change icon are common examples.
- Two of those, a music beat icon 83 A and a dynamic audio change icon 83 B, are depicted in FIG. 3 .
- a music beat icon 83 A and/or a dynamic audio change icon 83 B appears, the editor knows that the audio has changed significantly, which can correspond to a convenient edit point.
- the user interface 41 includes value indicators 69 , 75 , 85 corresponding to each of the respective visual event markers 65 , audio event markers 71 , and overlay audio event markers 81 .
- these value indicators 69 , 75 , 85 indicate increased value when two or more of the event markers 65 , 71 , 81 correspond to a substantially similar time on their respective timelines VT, AT, OAT.
- the majority of the value indicators 85 are relatively short and correspond to a single event marker 81 (e.g., a music beat, see icon 83 A).
- Other value indicators 85 A include lines that are relatively long and correspond to two event markers 81 occurring at a substantially similar time. In the example of FIG.
- the relative large value indicator 85 A is utilized to increase the relative value of the event marker 81 .
- Such an increase in value is appropriate because of the intersection of two event markers 83 A, 83 B at substantially the same time.
- the substantially similar time exhibited by two events need not be the exact same time. For example, by default, events no more than about five seconds apart can still be considered to have occurred at a substantially similar time. This number is readily configurable by the user, thereby allowing the user to customize the sensitivity of finding intersections and thereby the number of such intersections.
- events on separate timelines can also be determined as an intersection and given a higher value, as indicated above.
- three or more event markers 65 , 71 , 81 occurring at a substantially similar time can indicate a value indicator of even greater value.
- At least one of the visual event markers 65 , audio event markers 71 , and overlay audio event markers 81 corresponds to a selection element 89 for selection by a user.
- the selection element 89 is adapted for selection of the event by a user and movement of the event to another time location on the respective timeline VT, AT, OAT by the user.
- the selection element 89 is a vertical guideline spanning each of the respective timelines VT, AT, OAT and indicating what event is invoked on each timeline at the same moment in time.
- this selection element 89 has further functions. For example, selection of the selection element 89 can increase the granularity of event markers 65 , 71 , 81 within the visual elements, the audio elements, and the overlay audio elements occurring at the selected time. In other words, increased granularity can be invoked over only a portion of the timeline including those elements associated with the time of the selection element 89 .
- a selection element 91 (e.g., a check box) for selecting the inclusion or exclusion of event markers 65 , 71 , 81 indicating particular types of events is included in the user interface 41 .
- such selection elements 91 for selecting the inclusion or exclusion of event markers 65 , 71 , 81 indicate particular types of events.
- a new face selection element 91 A, a new scene selection element 91 B, an end-of-phrase selection element 91 C, a music beat selection element 91 D, and a dynamic audio change selection element 91 E are depicted.
- selection elements 91 By selecting and deselecting one or more of such selection elements 91 , particular event markers 65 , 71 , 81 are included or excluded, respectively. For example, a user wishing to focus editing efforts only on new faces and music beat could select the new face selection element 91 A and the music beat selection element 91 D, while deselecting the remaining selection elements, such that only event markers 65 , 71 , 81 associated with new faces and music beat are shown.
- Other selection elements directed to event markers based upon other characteristics are also contemplated as within the scope of the embodiments of the present invention.
- the selection elements 91 for selecting the inclusion or exclusion of event markers 65 , 71 , 81 can also each comprise a corresponding selection element (not shown) (e.g., a slider bar) adapted for adjusting the granularity of the inclusion or exclusion of the event markers associated with a particular selection element.
- a granularity selection element associated with the new face selection element 91 A can be adjusted to increase or decrease the frequency of new face events by increasing or decreasing the sensitivity of the analysis engine 37 .
- a selection element 93 for increasing or decreasing the granularity of the placement of all of the event markers 65 , 71 , 81 provides a mechanism for quickly increasing or decreasing the population of all events, depending upon the preference of the user.
- the user interface 41 can be configured to provide functions even more specific than those discussed above.
- a selection element can be included for automatically adjusting the viewing time of at least one of the digital images I to begin and end during the visual data sequence VS between audio beats of the overlay audio data sequence 25 B.
- a selection element can be included for automatically adjusting the timeline position of at least one of the digital images I including the image of a particular item to appear substantially adjacent other video segments VS or digital images also including the image of the same item.
- Such a selection element can be useful in grouping similar items near one another in the timelines VT, AT, OAT.
- the image of a particular item is the image of a person, whereby video segments and digital images of particular people can be automatically placed adjacent one another by selecting a single selection element.
- the user interface 41 further comprises an aggregate timeline (see FIG. 1 ) depicting the visual data sequence VS, the audio data sequence 25 A, and the overlay audio data sequence 25 B in a single, aggregate timeline.
- a system, generally indicated 101 , for analyzing data sequences VS, 25 A, 25 B and extracting metadata associated with the data sequences for providing information related to events of the data sequences is depicted in FIG. 5 .
- the system 101 comprises application programs 103 , including an analysis engine 105 , and application data 107 .
- the analysis engine 105 comprises a metadata extraction module 105 A for extracting metadata associated with the data sequences VS, 25 A, 25 B, generally as set forth above.
- the analysis engine 105 further comprises a metadata aggregation module 105 B for aggregating the extracted metadata from the data sequences VS, 25 A, 25 B.
- the analysis engine 105 further comprises a user interface metadata association module 105 C for providing information related to the aggregated metadata to a user via a user interface.
- Other modules 105 D directed to other aspects of the embodiments of the present invention are also contemplated herein and depicted generally in FIG. 5 .
- the application data 107 comprises at least one visual data sequence VS, at least one audio data sequence 25 A, and at least one second, or overlay, audio data sequence 25 B, each stored as application data and generally as set forth above.
- the application data 107 further comprises metadata associated with the visual data sequence 107 A extracted by the metadata extraction module, metadata associated with the audio data sequence 107 B extracted by the metadata extraction module, and metadata associated with the second, or overlay, audio data sequence 107 C extracted by the metadata extraction module.
- the extracted metadata associated with the visual data sequence 107 A comprises at least one of face patterns, color entropy characteristics, chroma values and patterns, luma values and patterns, among others.
- Each metadata element is associated with a timeline location corresponding to a timeline relating the data sequences to one another.
- the extracted metadata associated with the audio data sequence comprises at least one of pauses between phrases of an audio data sequence comprising speech, music, and recorded sounds, among others.
- the application data further comprises other data 107 D, such as a particular user-selected view, or other data.
- the application data 107 comprises aggregated metadata 107 E aggregated by the metadata aggregation module 105 B. Beyond aggregated metadata, the aggregated metadata 107 E may further comprise other related data, such as the sorting order for the resultant set of metadata. For example when sorting a spreadsheet application, one may select which rows or columns have precedence in sorting hierarchy and how the sorted hierarchy should be ordered. With the present example, the user may configure the sorting hierarchy of the event detection (e.g. determine and visualize a specific face detection over a generic face, and then determine how many faces are present in a scene, etc.).
- an exemplary data record corresponding to an event marker 65 , 71 , 81 on a timeline VT, AT, OAT is depicted.
- a data record 111 would be generated for each event marker 65 , 71 , 81 associated with a particular data sequence VS, 25 A, 25 B.
- the exemplary data record 111 comprises a timeline location 111 A of the event on the timeline VT, AT, OAT. This timeline location can be readily amended as the position of the event on the timeline changes, while maintaining the other information related to the event.
- the data record 111 further includes a type indicator 111 B of the type of event indicated on the timeline. These data are useful in indicating the nature of the event, as discussed above.
- the data record 111 further comprises a sequence indicator 111 C, or individual sequence number, indicating the location of the event relative to other events of the timeline VT, AT, OAT.
- the data record 111 further comprises an icon 111 D indicative of the nature of the corresponding event and an alternate icon 111 E indicative of the nature of the corresponding event.
- the alternate icon can provide the user with a choice of icons for a particular event, or serve as a substitute icon if the first icon is unavailable or otherwise unusable.
- the data record 111 further comprises a data explaining the function of the data record 111 F, such as for use with a tool tip text that appears when hovering a mouse cursor over an object (see FIG. 3 ).
- An alternate data 111 G explaining the function of the data record is also included.
- the data record 111 further comprises a value indicator indicative of the relative value of the corresponding event.
- the value indicator is a line and the data record 111 further comprises a line height value 111 H, a line thickness value 111 I, and a line color value 111 J, generally as set forth above.
- FIG. 7 shows one example of a general purpose computing device in the form of a computer 130 .
- a computer such as the computer 130 is suitable for use in the other figures illustrated and described herein.
- Computer 130 additionally has one or more processors or processing units 132 and a system memory 134 .
- the computer 130 typically has at least some form of computer readable media.
- Computer readable media which include both volatile and nonvolatile media, removable and non-removable media, may be any available medium that can be accessed by computer 130 .
- Computer readable media comprise computer storage media and communication media.
- Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information.
- Communication media typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media.
- a user may enter commands and information into computer 130 through input devices or user interface selection devices such as a keyboard 180 and a pointing device 182 (e.g., a mouse, trackball, pen, or touch pad).
- Other input devices may include a microphone, joystick, game pad, camera, scanner, or the like.
- processing unit 132 through a user input interface 184 that is coupled to system bus 136 , but may be connected by other interface and bus structures, such as a parallel port, game port, or a Universal Serial Bus (USB).
- a monitor 188 or other type of display device is also connected to system bus 136 via an interface, such as a video interface 190 .
- the computer 130 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 194 .
- the remote computer 194 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer 130 .
- the embodiments of the invention are operational with numerous other general purpose or special purpose computing system environments or configurations.
- the computing system environment is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention.
- the computing system environment should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment.
- Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the embodiments of the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Television Signal Processing For Recording (AREA)
- Management Or Editing Of Information On Record Carriers (AREA)
Abstract
Analyzing data sequences, such as video and audio segments, and extracting metadata from the data sequences for providing information related to events of the data sequences. The metadata extracted is used for making editing decisions, such as for an audio-visual production comprised of a collection of video and audio segments. The metadata identifies events of importance in the video and audio segments, which may then be applied to the editing process. A user interface including the extracted metadata in a timeline format facilitates editing the audio-visual production.
Description
- This application is a divisional of U.S. Ser. No. 11/424,509, filed Jun. 15, 2006, the entire contents of which are incorporated herein by reference.
- Many people are accustomed to consuming sophisticated, well edited media, as seen on television, in movies, and in other professionally produced media. Because of these heightened expectations, non-expert video editors are likely to fail at producing video productions that fulfill their personal expectations, or the expectations of their highly-conditioned audience. Non-expert users of non-linear video editing (e.g., digital video editing) systems typically cannot create attractive looking and sounding videos. Unlike professional video editors who have knowledge and experience in making effective editing decisions, non-expert users have little to no experience, and would therefore benefit greatly from a coach. Such non-expert video editors may need editing assistance to create higher-quality products. For example, if a non-expert user had one-on-one professional advice on how to make a high impact, upbeat video, he would likely receive particular editing suggestions (e.g., use jump-cuts in places where music beats and loud audio peaks, like explosions, are located). With this advice, such a non-expert user could cut to dramatic video image changes at particular points corresponding to the audio. Unfortunately, many users are unable to hire an expert video editing coach, and are therefore unable to apply this advice or any other techniques applied by expert editors.
- Conventional media editing tools (e.g., video editing software) provide users with the ability to capture, edit, import, and modify both visual and audio media for the development of audio/visual works. For example, a user can create an audio/visual work with a media editing tool (e.g., Microsoft® Windows Movie Maker Version 2.1) and the media editing tool can provide the user with several functions, including trimming, splicing, and cropping video, merging image, audio, and video media, adding transitions and effects between constituent parts, and overlaying additional audio, to name a few. In the hands of a skilled user, such a conventional system can provide a means for merging several different types of media together into a unified work, different than all of its constituent portions. Unfortunately, such conventional media editing tools do not provide the type of advanced guidance required to substantially improve editing for the novice user. Without assistance from a human expert, most users are relegated to creating basic video productions without the tools and expertise available to a professional editor. A way to provide some of these tools and expertise to novice users based upon their specific media content would be useful. Moreover, a way to automatically edit and produce an audio/visual work or semi-automatically produce one by allowing modification of the results yielded by the automatic process and then producing (rendering) the audio/visual work would also be useful, such as to a novice user.
- Moreover, even for the expert editor who understands how to effectively manually edit media, additional tools identifying critical events in the media being edited can facilitate more efficient editing. In other words, automatically identifying potential edit events to the skilled editor can facilitate more efficient and effective editing because effort can be applied to the creative aspects of the editing process, rather than to the identification of potential editing events. Thus, a way to provide tools to an experienced editor in identifying potential editing events would be useful.
- The following simplified summary provides a basic overview of some aspects of the present technology. This summary is not an extensive overview. It is not intended to identify key or critical elements or to delineate the scope of this technology. This Summary is not intended to be used as an aid in determining the scope of the claimed subject matter. Its purpose is to present some simplified concepts related to the technology before the more detailed description presented below.
- Accordingly, aspects of the invention provide for analyzing data sequences and extracting metadata from the data sequences to provide information related to events of the data sequences. These events provide specific editing guidance to a user. By analyzing multiple data sequences according to characteristics to extract metadata from each sequence, aspects of the invention permit the dissemination of information regarding key events and features of the data sequences for use by a user in editing the sequences. Moreover, determining which of such metadata occur at substantially the same time and associating the metadata with a timeline provide information to users related to the coordination of editing events from different data sequences, thereby facilitating editing by a user.
-
FIG. 1 is a diagram of an aggregate timeline and its constituent parts of one embodiment of the invention; -
FIG. 2 is a flow diagram of a method of one embodiment of the invention; -
FIG. 3 is an extension of the flow diagram of Fig, 2; -
FIG. 4 is another flow diagram of a method of another embodiment of the invention; -
FIG. 5 is an exemplary system memory of another embodiment of the invention; -
FIG. 6 is an exemplary data record of another embodiment of the invention; and -
FIG. 7 is a block diagram illustrating one example of a suitable computing system environment in which one embodiment of the invention may be implemented. - Corresponding reference characters indicate corresponding parts throughout the drawings.
- Referring now to
FIG. 1 , a diagram of an aggregate timeline, generally indicated 21, and its constituent parts of one embodiment of the invention is shown. The constituent parts of theaggregate timeline 21 include several objects commonly associated with media editing. Those objects may include but are not limited to textual messages TE1, TE2 (e.g., titles, credits, presentation information, etc.), video segments VS1, VS2, VS3, VS4 (e.g., video clips, animations, etc.), digital images I1, I2 (e.g., digital photographs, documents, screenshots, and computer-generated images, etc.), and transitions, or effects, between other objects T1, T2, T3, T4, and T5. Theaggregate timeline 21 further includes blank space BS as an object for holding timeline space open, without any objects. Advantageously, media editing software embodying aspects of the invention analyze multiple data sequences according to characteristics to extract metadata from each sequence. This metadata might include a transition to a new video scene, a new face entering a video scene, or a music beat in an audio sequence, among others, as described below. This permits the dissemination of metadata information regarding events and features of the data sequences for use in editing the sequences. Providing such metadata to a user will assist in the editing process, as the use of such metadata can offer convenient and effective editing choices (e.g., for a novice use). Moreover, determining which of such metadata occur at substantially the same time and associating the metadata with a timeline provide information related to the coordination of editing events from different data sequences, which is also helpful when editing. - The diagram also illustrates a non-aggregated timeline, generally indicated 23, which depicts each of the objects separately. Such a timeline is particularly useful for understanding the details of how a media production involving multiple media objects and transitions may be constructed. For example, the first textual message TE1 extends from the beginning, or left edge, of the timeline until the end, or right edge, of the first transition T1. Similarly, the first video segment VS1 extends from the beginning, or left edge, of the first transition T1 until the end, or right edge of the second transition T2. As would be readily understood by one skilled in the art, during the first transition, the viewer of the media would begin by seeing primarily the first textual message. As time passed during the first transition T1, more of the first video segment VS1 would be visible while less of the first textual message TE1 would be visible. The remaining timeline objects function similarly. As would be readily understood by one skilled in the art, any number of different types of objects can be included in any arrangement without departing from the scope of the embodiments of the present invention. Moreover, in the
non-aggregated timeline 23 ofFIG. 1 , each of the video segments VS1, VS2, VS3, VS4 and digital images I1, 12 is depicted as staggered from adjacent video segments and digital images to facilitate viewing of the entire scope of the video segment or image, without overlap. - In addition to the visual aspects of the aggregated
timeline 21 and the non-aggregatedtimeline 23, additional information is included in the form of aural tracks 25, three of which are depicted inFIG. 1 for purposes of illustration.Audio track 25A depicted in the diagram is the aggregation of the audio associated with each of the video segments VS1, VS2, VS3, VS4 and digital images I1, 12, described above. In this exemplary diagram, theaudio track 25A depicts the intensity of the aggregated audio over time, such that the intensity of the audio—and indirectly the activities of the video segments VS1, VS2, VS3, VS4 and the digital images I1, I2—can be understood as a function of time.Audio track 25B (e.g., a musical overlay) included in the diagram depicts the intensity of music currently present in thetimeline 23. As with theaudio track 25A, themusic track 25B depicts the intensity of the music track or tracks over time, thereby providing an indication of the nature of the music.Overlay track 25C depicted in the diagram depicts items that may be composited, or added, onto the audio/visual work over time. The duration, blending, and intensity of each is specified over time. Examples of these items are text that may be used for titles and captions, other images, animations, or video. Other audio characteristics, other than audio intensity, can also be depicted on the aural tracks 25, without departing from the scope of the embodiments of the present invention. In addition, other aural tracks 25 can be included without departing from the scope of the embodiments of the present invention. - In the example shown in
FIG. 1 , both video segments VS and still digital images I are included. It should be understood that the timeline may contain any number and typed of items. For example, the timeline may include at least one of video, video with associated audio, images, audio, music, text, overlay text, images, animations, and other text, without departing from the scope of the present invention. In another example, the timeline can contain only video segments VS, or only digital images I, without departing from the scope of the embodiments of the present invention. Moreover, as used herein the term digital images can include digital photographs, but can also include other digital items, including documents, scanned images, and computer-generated images, to name a few. - Referring now to
FIGS. 2 and 3 , a system flow diagram, generally indicated 29, depicts one embodied method of the present invention. The system flow begins with raw components, generally indicated 31, as described generally above. In this embodiment, the raw components include a video segment VS, anaudio track 25A associated with the video segment, and amusic track 25B. As would be readily understood by one skilled in the art, each of theseraw components 31 can be referred to more generally as data sequences, such as a visual data sequence VS, anaudio data sequence 25A, and a second, or overlay,audio data sequence 25B. - These
raw components 31 are each sent to an analysis module, generally indicated 33, which analyzes the underlying data sequences of the raw components. In particular, the video segment VS is sent to a video andimage analysis submodule 33A, theaudio track 25A is sent to anaudio analysis submodule 33B, and themusic track 25B is sent to a secondaudio analysis submodule 33C. As will be discussed in greater detail below, theanalysis module 33 analyzes the underlying data sequences VS, 25A, 25B and extracts metadata related to the data sequences. This metadata can be useful to an editor, for example a novice editor, in selecting appropriate locations for particular editing decisions, as discussed below. In one embodiment, theaudio analysis submodule 33B is a speech analysis submodule. In another embodiment, the secondaudio analysis submodule 33C is a music analysis submodule, and more particularly a music beat and audio peak analysis submodule. It should also be noted that other submodules may be added to perform additional analyses without departing from the scope of the present invention. - After analysis by each submodule 33A, 33B, 33C, the extracted metadata from each
component 31 is collected in a storage area, generally indicated 35. In particular, a video and imageanalysis storage area 35A receives the extracted metadata from the video andimage analysis submodule 33A, an audioanalysis storage area 35B receives the extracted metadata from theaudio analysis submodule 33B, and a second audioanalysis storage area 35C receives the extracted metadata from the secondaudio analysis submodule 33C. Although thestorage areas FIG. 2 , it should be understood by one skilled in the art that two or more of those storage areas can be combined without departing from the scope of the embodiments of the present invention. - After collecting and storing the extracted metadata in the
storage areas analysis engine 37 receives the stored metadata and analyzes the metadata. As discussed below in greater detail, theanalysis engine 37 analyzes the content of the data sequences VS, 25A, 25B and makes meaningful editing suggestions, in the form of events, based upon the actual content of the data sequence (e.g., scene changes, music beats, audio peaks, spaces between spoken words, etc.). In one alternative embodiment, theanalysis engine 37 determines the intersection of metadata events occurring at substantially the same time within differentraw components 31 and associates such metadata with a common timeline associated with the raw components. The results of this analysis are stored instorage area 39. - The
analysis engine 37 is also responsible for displaying the metadata and the results of its analysis, such as in depicted in the user interface, generally indicated 41, ofFIG. 3 . The editing suggestions noted above are displayed in the timeline of theuser interface 41 as “snap-to-points” (also known as “sticky points”) corresponding to the events. Moreover, theanalysis engine 37 is adapted to receive control parameters from the user via theuser interface 41 ofFIG. 3 . Such control parameters can also be stored in thestorage area 39. -
FIG. 3 is an extension of the flow diagram ofFIG. 2 . In particular, theexemplary user interface 41 is in the form of a timeline, including each of the data sequences VS, 25A, 25B depicted on the same timeline. This allows the user to readily determine what portion of each data sequence VS, 25A, 25B is utilized at any given time. The details of theuser interface 41 will be discussed in greater detail below. - A method for analyzing data sequences (such as data sequences VS, 25A, 25B introduced above) and extracting metadata associated with the data sequences for providing information related to events of the data sequences is generally indicated 45 in
FIG. 4 . The method comprises analyzing, at 47, a visual data sequence VS according to at least one characteristic of the visual data sequence and extracting, also at 47, metadata associated with the visual data sequence according to the at least one characteristic of the visual data sequence. - In one embodiment, the visual data sequence VS is analyzed, at 47, according to at least one visual characteristic. Moreover, the metadata associated with the visual data sequence VS is extracted according to the visual characteristic of the visual data sequence. In one alternative embodiment, visual characteristics of the video data sequence can comprise video dynamic peaks, dynamic image changes, color entropy characteristics, chroma values and patterns, luma values and patterns, and image pattern recognition, among others. In another alternative embodiment, video dynamic peaks can comprise detectable changes in the visual data sequence VS, such as scene changes (e.g., different scenes stored adjacent one another in the visual data sequence) and stark image transitions (e.g., large changes in brightness, large changes in color). Also, for example, dynamic image changes can comprise detectable changes in the visual data sequence VS (e.g., quick zooming, quick panning, removal of the lens cap, among others). Recognition of chroma values and patterns can detect the type of video, such as sports (e.g., baseball) or other known video pattern characteristics. In particular, the following patent application describes how to perform detection of the video type: U.S. patent application entitled Video Search and Services, filed Feb. 27, 2006, assigned to Microsoft Corporation of Redmond, Washington, U.S.A. In addition, the image pattern recognition visual characteristic can comprise recognition of particular images, such as people generally, a particular person, a particular face (i.e., face patterns), animals generally, a particular animal (e.g., a dog), and other items (e.g., a boat, a car, etc.) among others. In one embodiment, aspects of the invention may allow the user to select one or more images for pattern recognition analysis, such as from a drop-down menu. In other words, visual data sequence VS is analyzed, at 47, based upon its content to determine if metadata related to the content can be extracted, also at 47, such as where detectable changes in the data sequence occur. In particular, the following patent application describes how to perform face detection: U.S. patent application entitled Pose-Adaptive Face Detection System and Process, filed May 26, 2000, issued as U.S. Pat. No. 6,671,391, and assigned to Microsoft Corporation of Redmond, Wash., U.S.A. The following patent application describes how to perform face recognition: U.S. patent application entitled Pose-Invariant Face Recognition System and Process, filed Nov. 5, 2004, published as U.S. publication number US 2005-0147292 A1, and assigned to Microsoft Corporation of Redmond, Wash., U.S.A.
- The
method 45 further comprises analyzing, at 49, anaudio data sequence 25A, substantially corresponding to the visual data sequence VS, according to at least one characteristic of the audio data sequence and extracting, also at 49, metadata associated with the audio data sequence according to at least one characteristic of the audio sequence data. Here, theaudio data sequence 25A substantially corresponding to the visual data sequence VS may simply mean that the data sequences are related to one another. In one alternative embodiment, the typical audio recorded along with a video recording would be such anaudio data sequence 25A substantially corresponding to the visual data sequence VS, or video recording. - In one embodiment, the
audio data sequence 25A is analyzed, at 49, according to at least one audio characteristic. Moreover, the metadata associated with theaudio data sequence 25A is extracted, at 49, according to the at least one audio characteristic of the audio data sequence. In one alternative embodiment, audio characteristics of theaudio data sequence 25A can comprise music beats, audio dynamic peaks, speech characteristics, changes in the person speaking, recorded sounds, word boundary detection, and word and phrase detection (e.g., identify all occurrences of the phrase “Happy Birthday”), among others. In other words, theaudio data sequence 25A can be analyzed according to any characteristic that can yield metadata regarding events of interest in the audio data sequence. In another alternative embodiment, theaudio data sequence 25A can be analyzed to create a metadata event at significant music beats, or at all music beats. Identification of such events related to music beats can be useful to an editor, as the introduction of a new scene or new camera angle is often timed to coincide with the beat of background music. In another example, audio dynamic peaks, such as loud noises or other sharp increases and subsequent decreases in audio intensity, or volume, that are part of the audio data sequence can be identified as potential editing events. One skilled in the art would readily understand the other audio characteristics identified above. For example, the following patent application describes how to segment and classify an audio data sequence: U.S. patent application entitled Audio Segmentation and Classification, filed Apr. 19, 2000, issued as U.S. Pat. No. 6,901,362, and assigned to Microsoft Corporation of Redmond, Wash., U.S.A. For example, the following patent application describes how to determine if speaker identity has changed: U.S. patent application entitled Method of Real-time Speaker Change Point Detection, Speaker Tracking and Speaker Model Construction, filed Nov. 29, 2002, published as U.S. patent application number US 2004/0107100 A1, and assigned to Microsoft Corporation of Redmond, Washington, U.S.A. - The
method 45 also comprises analyzing, at 51, a secondaudio data sequence 25B according to at least one characteristic of the second audio data sequence and extracting, also at 51, metadata associated with the second audio data sequence according to at least one characteristic of the second audio data sequence. Unlike theaudio data sequence 25A discussed above, the secondaudio data sequence 25B does not necessarily substantially correspond to the visual data sequence VS. In one exemplary embodiment, where two microphones record two, separate audio tracks corresponding to the same video data sequence VS, bothaudio data sequences audio data sequence 25B not in substantial correspondence with the visual data sequence VS. For example, the extracting metadata, at 51, associated with the secondaudio data sequence 25B can comprise extracting metadata associated with an audio overlay, such as a musical piece (e.g., a song) or vocal narration. - In one embodiment, the second
audio data sequence 25B is analyzed, at 51, according to at least one audio characteristic of the second audio data sequence. Moreover, the metadata associated with the secondaudio data sequence 25B is extracted, at 51, according to the at least one audio characteristic of the second audio data sequence. For example, audio characteristics of the secondaudio data sequence 25B can comprise music beats, audio dynamic peaks, speech characteristics, particular recorded sounds, word boundary detection, and word and phrase detection, among others, generally as discussed above with respect to theaudio data sequence 25A. - The
method 45 further determines, at 53, intersections of metadata from two or more of the extracted metadata occurring at substantially the same time. Determining intersections of metadata means reviewing the metadata extracted from each of the data sequences VS, 25A, 25B and determining if any of the metadata from one data sequence occurs at substantially the same time as metadata from another of the data sequences. Where metadata occur at substantially the same time, an intersection is determined. These intersections represent valuable events to the novice or experienced editor, as significant events have occurred at the same time on two or more data sequences. Noting these intersections of events for the user, as will be discussed in greater detail below, provides tangible, specific guidance to the user regarding effective editing strategies. For example, where a dynamic image change occurs in the visual data sequence VS at substantially the same time as an audio dynamic peak in theaudio data sequence 25A, an intersection can be determined. In many cases, this metadata intersection event may be more useful that a solitary metadata event because the intersection event brings together features of interest from two or more data sequences, or parts of the final production. With the intersections determined, the method further aggregates, at 55, metadata and metadata intersections. - Once the intersections are determined, at 53, and the extracted metadata and metadata intersections are aggregated, at 55, with the timeline, the
method 45 further comprises associating, at 57, metadata and metadata intersections with a timeline associated with the data sequences. This association, at 57, places each piece of extracted metadata and determined metadata intersection into a common timeline. This association with a common timeline provides for ready review of all the metadata and metadata intersections by the user of one or more of the metadata events, whereby further editing decisions are based upon the proximity, frequency, and density of the metadata associated with the timeline. - The method further comprises rendering, at 59, a user interface depicting the timeline with the data sequences VS, 25A, 25B, the metadata, and the intersections of metadata. The rendered user interface 41 (see
FIG. 3 ) is a useful tool for editing, as it incorporates all of the extracted metadata and metadata intersections into a single interface for review by the user. The details of theuser interface 41 will be discussed in greater detail below. In one example, the method may further comprise automatically rendering, at 60, a resultant audio/visual work. For example, without user direction, the method may utilize the previous analyzing and extracting 47, 49, 51, determining 53, aggregating 55, and associating 57 processes to render 60 a resultant audio-visual work automatically. Such an automatic rendering may be particularly useful for a user seeking a completed audio/visual work with relatively little effort and in as short a time period as possible. The method may further comprise allowing, at 62, the user to modify the resultant audio/visual work. - The method may further comprise receiving, at 61, user input regarding changes in one or more of the data sequences VS, 25A, 25B via the
user interface 41. The user can elect to change any number of parameters, including moving, modifying, and deleting one or more of the data sequences VS, 25A, 25B. Moreover, the user can elect to add one or more additional data sequences. As would be understood by one skilled in the art, changes of this type will often change the location of metadata events and metadata intersections, requiring a new set of analysis, extractions, and determinations, generally as set forth above. For example, the user can decide to crop a particular portion of the visual data sequence VS and the associated portion of theaudio data sequence 25A, thereby causing substantial changes in the location of metadata events. It should be readily appreciated here that even relatively minor changes to the data sequences VS, 25A, 25B can create significant changes to the extracted metadata and metadata intersections. Thus, after the user invokes such a change, the method repeats the analyzing and extracting 47, 49, 51 metadata associated with each of the data sequences, the determining 53, the aggregating 55, the associating 57, and therendering 59 to ensure that theuser interface 41 rendered is reflective of the location of metadata and metadata intersections created during the latest changes. - As introduced above, a
user interface 41 for integrating visual and audio data sequences VS, 25A, 25B together for creating an audio-visual work is depicted inFIG. 3 . Theuser interface 41 includes a timeline, generally indicated 61, extending left to right and demarking the time for each of the data sequences on thetimeline 61. As would be readily understood by one skilled in the art, the scale of thetimeline 61 can be adjusted to suit the user, such as to focus in on a particular portion of interest, or to review a greater portion of thetimeline 61 for a more general view. In the example depicted inFIG. 3 , thetimeline 61 extends from about 0 seconds to about 32 seconds. Any scale can be used without departing from the scope of the embodiments of the present invention. - The
user interface 41 further comprises a visual timeline VT associated with the visual data sequence VS comprising one or more visual elements. The visual timeline VT depicts the beginning and the end of each of the one or more visual elements. In the example ofFIG. 3 , a first video segment VS1 extends from about zero seconds to about 22.5 seconds, while a second video segment VS2 extends from about 20.8 seconds and at least until the end of the visible visual timeline VT at about 32 seconds. Between about 20.8 seconds and about 22.5 seconds is a first transition T1 between the first video segment VS1 and the second video segment VS2. - Depiction of video segments VS1, VS2 and transitions T1 is well-known in the art. The visual timeline VT of the embodiments of the present invention, however, further include visual event markers, generally indicated 65, indicating the corresponding time on the
timeline 61 of a particular visual event in the visual data sequence VS. Thevisual event markers 65 provide visual indications regarding events within the visual data sequence VS that are helpful to users when editing. Moreover, thevisual event markers 65 further comprise at least one of an icon 67 indicative of the nature of the corresponding event and avalue indicator 69 indicative of the relative value of the corresponding event. - The icons 67 corresponding to the
visual event markers 65 can depict any class of events, or individual events. In one alternative embodiment, a new face icon, a face exit icon, and a new scene icon are included. Two of those, anew face icon 67A and anew scene icon 67B, are depicted in the example ofFIG. 3 . Thus, wherever anew face icon 67A appears, the editor knows that a new person has entered the scene, and wherever anew scene icon 67B appears, the editor knows that a new scene has begun. This is important information to the editor, which can now be used as a basis for other editing decisions. With conventional editing systems, the user would need to carefully review the visual data sequence VS manually to determine such editing events. The automatic extraction and rendering of these events on auser interface 41 provides this information to an editor directly. Visual demonstratives other than icons 67 are also contemplated as within the scope of the embodiments of the present invention (e.g., user-configurable thumbnails, including ones representative of the items detected, such as specific faces, boats, cars, etc.). Note that theexemplary user interface 41 ofFIG. 3 may also include a tile (video) overlay visual demonstrative (not shown). Such demonstratives would be depicted similar to the visual demonstratives of the first video segment VS1. - The
value indicators 69 depicted in the visual data sequence VS ofFIG. 3 are vertical lines. Each of thevalue indicators 69 depicted in the visual data sequence VS ofFIG. 3 are the same length, thickness, line-type, and color. By sharing the same length, thickness, line-type, and color, thesevalue indicators 69 each correspond to avisual event marker 65 of similar value. The length, thickness, line-type, and color of theindividual value indicators 69 can also be modified to indicate a value indicator of greater or lesser value, as compared with theother value indicators 69. For example, the line associated with aparticular value indicator 69 can be lengthened to indicate its greater relative value. Similarly, ared value indicator 69 can indicate a higher value than a brown value indicator. In another example, the icon 67 can also indicate the value of the visual event marker 65 (e.g., a larger icon indicates greater relative value). Any such changes to indicate a greater or lesser relative value of a particularvisual event marker 65 are contemplated as within the scope of the embodiments of the present invention. - The
user interface 41 further comprises an audio timeline AT associated with anaudio data sequence 25A comprising one or more audio elements. The audio timeline AT corresponds to the same time scale and position in time as the visual timeline VT. The audio timeline AT depicts an audio characteristic of theaudio data sequence 25A, such as the output level of the audio data sequence over time. Like the visual timeline, the audio timeline AT depicts the beginning and the end of each of the one or more audio elements. In the example ofFIG. 3 , a first audio segment AS1 extends from about zero seconds to about 22.5 seconds, while a second audio segment AS2 extends from about 20.8 seconds and at least until the end of the visible audio timeline AT at about 32 seconds. As with the visual timeline VT, between about 20.8 seconds and about 22.5 seconds is the first transition T1 between the first audio segment AS1 and the second audio segment AS2. - Depiction of audio segments AS1, AS2 and transitions T1 is well known in the art. In addition, however, the audio timeline AT further includes audio event markers, generally indicated 71, indicating the corresponding time on the
timeline 61 of a particular audio event in theaudio data sequence 25A. Theaudio event markers 71 provide visual indications regarding events within theaudio data sequence 25A that are helpful during editing. Moreover, theaudio event markers 71 further comprise at least one of anicon 73 indicative of the nature of the corresponding event and avalue indicator 75 indicative of the relative value of the corresponding event. Theicons 73 andvalue indicators 75 function similarly to those described above with respect to the visual data sequence VS. For example, an end of phrase icon, a beginning of phrase icon, an audio peak icon, a silence icon, a music beat icon, and a dynamic audio change icon are common examples. One of those, an end ofphrase icon 73, is depicted inFIG. 3 . Thus, wherever an end ofphrase icon 67A appears, the editor knows that a person has stopped speaking, which can correspond to a convenient edit point. - The
user interface 41 further comprises an overlay audio timeline OAT associated with an overlayaudio data sequence 25B comprising one or more overlay audio elements. The overlay audio timeline OAT corresponds to the same time scale and position in time as the visual timeline VT and the audio timeline AT. The overlay audio timeline OAT depicts at least one audio characteristic of the overlay audio data sequence over time, such as the output level of the overlayaudio data sequence 25B over time. In one alternative embodiment, the overlayaudio data sequence 25B is a musical work, such as a song, which can be used in conjunction with the video data sequence VS and theaudio data sequence 25A. In the example ofFIG. 3 , an audio segment AS extends from about zero seconds to about 32 seconds. Multiple audio segments AS can be utilized without departing from the scope of the embodiments of the present invention. - The overlay audio timeline OAT further includes overlay audio event markers, generally indicated 81, indicating the corresponding time on the timeline of a particular overlay audio event in the overlay
audio data sequence 25B. The overlayaudio event markers 81 provide visual indications regarding events within the overlayaudio data sequence 25B, as did the event markers described above with respect to their data sequence. Moreover, the overlayaudio event markers 81 further comprise at least one of an icon 83 indicative of the nature of the corresponding event and avalue indicator 85 indicative of the relative value of the corresponding event. It should be noted here that each occurrence of anevent marker 81, an icon 83, or avalue indicator 85 is not marked with a reference numeral inFIG. 3 . Several reference numerals were intentionally not included to decrease clutter in the view of the overlay audio timeline OAT. - The icons 83 and
value indicators 85 function similarly to those described above with respect to the visual data sequence VS. For example, an end of phrase icon, a beginning of phrase icon, an audio peak icon, a silence icon, a music beat icon, and a dynamic audio change icon are common examples. Two of those, amusic beat icon 83A and a dynamicaudio change icon 83B, are depicted inFIG. 3 . Thus, wherever amusic beat icon 83A and/or a dynamicaudio change icon 83B appears, the editor knows that the audio has changed significantly, which can correspond to a convenient edit point. - As discussed above, the
user interface 41 includesvalue indicators visual event markers 65,audio event markers 71, and overlayaudio event markers 81. In one alternative embodiment, thesevalue indicators event markers FIG. 3 , the majority of thevalue indicators 85 are relatively short and correspond to a single event marker 81 (e.g., a music beat, seeicon 83A).Other value indicators 85A include lines that are relatively long and correspond to twoevent markers 81 occurring at a substantially similar time. In the example ofFIG. 3 , where amusic beat icon 83A and a dynamicaudio change icon 83B occur at a substantially similar time, the relativelarge value indicator 85A is utilized to increase the relative value of theevent marker 81. Such an increase in value is appropriate because of the intersection of twoevent markers more event markers - In another embodiment, at least one of the
visual event markers 65,audio event markers 71, and overlayaudio event markers 81 corresponds to aselection element 89 for selection by a user. Theselection element 89 is adapted for selection of the event by a user and movement of the event to another time location on the respective timeline VT, AT, OAT by the user. In the example depicted inFIG. 3 , theselection element 89 is a vertical guideline spanning each of the respective timelines VT, AT, OAT and indicating what event is invoked on each timeline at the same moment in time. By moving the event of interest (e.g., a video segment) along its respective timeline with theselection element 89, at least a portion of the data sequence (e.g., the visual data sequence VS) associated with the selection element also moves along the timeline. Moreover, at least one of thevisual event markers 65,audio event markers 71, and overlayaudio event markers 81 can be selected and snapped to the location of another of the event markers located on another of the timelines. In this manner, the various video segments VS and audio segments AS can be moved and snapped to appropriate locations wherebyevent markers - In this manner, the user can select a particular point to determine exactly which of the media from each data sequence VS, 25A, 25B will be shown at the particular time associated with the
selection element 89. In another embodiment, thisselection element 89 has further functions. For example, selection of theselection element 89 can increase the granularity ofevent markers selection element 89. - In still another embodiment, a selection element 91 (e.g., a check box) for selecting the inclusion or exclusion of
event markers user interface 41. In one embodiment, such selection elements 91 for selecting the inclusion or exclusion ofevent markers FIG. 3 , a newface selection element 91A, a newscene selection element 91B, an end-of-phrase selection element 91C, a musicbeat selection element 91D, and a dynamic audiochange selection element 91E are depicted. By selecting and deselecting one or more of such selection elements 91,particular event markers face selection element 91A and the musicbeat selection element 91D, while deselecting the remaining selection elements, such thatonly event markers - The selection elements 91 for selecting the inclusion or exclusion of
event markers face selection element 91A can be adjusted to increase or decrease the frequency of new face events by increasing or decreasing the sensitivity of theanalysis engine 37. Moreover, aselection element 93 for increasing or decreasing the granularity of the placement of all of theevent markers - The
user interface 41 can be configured to provide functions even more specific than those discussed above. For example, a selection element can be included for automatically adjusting the viewing time of at least one of the digital images I to begin and end during the visual data sequence VS between audio beats of the overlayaudio data sequence 25B. In another alternative embodiment, a selection element can be included for automatically adjusting the timeline position of at least one of the digital images I including the image of a particular item to appear substantially adjacent other video segments VS or digital images also including the image of the same item. Such a selection element can be useful in grouping similar items near one another in the timelines VT, AT, OAT. In one further alternative embodiment, the image of a particular item is the image of a person, whereby video segments and digital images of particular people can be automatically placed adjacent one another by selecting a single selection element. - In another embodiment, the
user interface 41 further comprises an aggregate timeline (seeFIG. 1 ) depicting the visual data sequence VS, theaudio data sequence 25A, and the overlayaudio data sequence 25B in a single, aggregate timeline. - A system, generally indicated 101, for analyzing data sequences VS, 25A, 25B and extracting metadata associated with the data sequences for providing information related to events of the data sequences is depicted in
FIG. 5 . Broadly, thesystem 101 comprisesapplication programs 103, including ananalysis engine 105, andapplication data 107. Theanalysis engine 105 comprises ametadata extraction module 105A for extracting metadata associated with the data sequences VS, 25A, 25B, generally as set forth above. Theanalysis engine 105 further comprises ametadata aggregation module 105B for aggregating the extracted metadata from the data sequences VS, 25A, 25B. - The
analysis engine 105 further comprises a user interfacemetadata association module 105C for providing information related to the aggregated metadata to a user via a user interface.Other modules 105D directed to other aspects of the embodiments of the present invention are also contemplated herein and depicted generally inFIG. 5 . - The
application data 107 comprises at least one visual data sequence VS, at least oneaudio data sequence 25A, and at least one second, or overlay,audio data sequence 25B, each stored as application data and generally as set forth above. Theapplication data 107 further comprises metadata associated with thevisual data sequence 107A extracted by the metadata extraction module, metadata associated with theaudio data sequence 107B extracted by the metadata extraction module, and metadata associated with the second, or overlay,audio data sequence 107C extracted by the metadata extraction module. The extracted metadata associated with thevisual data sequence 107A comprises at least one of face patterns, color entropy characteristics, chroma values and patterns, luma values and patterns, among others. Each metadata element is associated with a timeline location corresponding to a timeline relating the data sequences to one another. The extracted metadata associated with the audio data sequence comprises at least one of pauses between phrases of an audio data sequence comprising speech, music, and recorded sounds, among others. In another embodiment, the application data further comprisesother data 107D, such as a particular user-selected view, or other data. Moreover, theapplication data 107 comprises aggregatedmetadata 107E aggregated by themetadata aggregation module 105B. Beyond aggregated metadata, the aggregatedmetadata 107E may further comprise other related data, such as the sorting order for the resultant set of metadata. For example when sorting a spreadsheet application, one may select which rows or columns have precedence in sorting hierarchy and how the sorted hierarchy should be ordered. With the present example, the user may configure the sorting hierarchy of the event detection (e.g. determine and visualize a specific face detection over a generic face, and then determine how many faces are present in a scene, etc.). - Referring to
FIG. 6 , an exemplary data record, generally depicted 111, corresponding to anevent marker data record 111 would be generated for eachevent marker exemplary data record 111 comprises atimeline location 111A of the event on the timeline VT, AT, OAT. This timeline location can be readily amended as the position of the event on the timeline changes, while maintaining the other information related to the event. Thedata record 111 further includes atype indicator 111B of the type of event indicated on the timeline. These data are useful in indicating the nature of the event, as discussed above. Thedata record 111 further comprises asequence indicator 111C, or individual sequence number, indicating the location of the event relative to other events of the timeline VT, AT, OAT. Thedata record 111 further comprises anicon 111D indicative of the nature of the corresponding event and analternate icon 111E indicative of the nature of the corresponding event. The alternate icon can provide the user with a choice of icons for a particular event, or serve as a substitute icon if the first icon is unavailable or otherwise unusable. - The
data record 111 further comprises a data explaining the function of thedata record 111F, such as for use with a tool tip text that appears when hovering a mouse cursor over an object (seeFIG. 3 ). Analternate data 111G explaining the function of the data record is also included. Thedata record 111 further comprises a value indicator indicative of the relative value of the corresponding event. In the example shown, the value indicator is a line and thedata record 111 further comprises aline height value 111H, a line thickness value 111I, and aline color value 111J, generally as set forth above. -
FIG. 7 shows one example of a general purpose computing device in the form of acomputer 130. In one embodiment of the invention, a computer such as thecomputer 130 is suitable for use in the other figures illustrated and described herein.Computer 130 additionally has one or more processors orprocessing units 132 and asystem memory 134. - The
computer 130 typically has at least some form of computer readable media. Computer readable media, which include both volatile and nonvolatile media, removable and non-removable media, may be any available medium that can be accessed bycomputer 130. By way of example and not limitation, computer readable media comprise computer storage media and communication media. Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information. Communication media typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media. - A user may enter commands and information into
computer 130 through input devices or user interface selection devices such as akeyboard 180 and a pointing device 182 (e.g., a mouse, trackball, pen, or touch pad). Other input devices (not shown) may include a microphone, joystick, game pad, camera, scanner, or the like. These and other input devices are connected toprocessing unit 132 through auser input interface 184 that is coupled tosystem bus 136, but may be connected by other interface and bus structures, such as a parallel port, game port, or a Universal Serial Bus (USB). Amonitor 188 or other type of display device is also connected tosystem bus 136 via an interface, such as avideo interface 190. - The
computer 130 may operate in a networked environment using logical connections to one or more remote computers, such as aremote computer 194. Theremote computer 194 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative tocomputer 130. - Although described in connection with an exemplary computing system environment, including
computer 130, the embodiments of the invention are operational with numerous other general purpose or special purpose computing system environments or configurations. The computing system environment is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention. Moreover, the computing system environment should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the embodiments of the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. - Those skilled in the art will note that the order of execution or performance of the methods illustrated and described herein is not essential, unless otherwise specified. That is, it is contemplated by the inventors that elements of the methods may be performed in any order, unless otherwise specified, and that the methods may include more or less elements than those disclosed herein.
- When introducing elements of the present invention or the embodiment(s) thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.
- As various changes could be made in the above products and methods without departing from the scope of the invention, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
Claims (20)
1. A user interface for integrating visual data and audio data together for creating an audio-visual work, said user interface comprising:
a visual timeline associated with a visual data sequence comprising one or more visual elements, said visual timeline depicting the beginning and end of each of said one or more visual elements, said visual timeline further including visual event markers indicating the corresponding time on the timeline of a particular visual event in the visual data sequence; and
an audio timeline associated with an audio data sequence comprising one or more audio elements, said audio timeline corresponding to the same time scale and position in time as the visual timeline, said audio timeline depicting at least one audio characteristic of the audio data sequence over time, said audio timeline further including audio event markers indicating the corresponding time on the timeline of a particular audio event in the audio data sequence;
said visual event markers and audio event markers providing visual indications regarding events within the respective visual data sequence and audio data sequence.
2. A user interface as set forth in claim 1 further comprising an overlay audio timeline associated with an overlay audio data sequence comprising one or more overlay audio elements, said overlay audio timeline corresponding to the same time scale and position in time as the visual timeline and the audio timeline, said overlay audio timeline depicting at least one audio characteristic of the overlay audio data sequence over time, said overlay audio timeline further including overlay audio event markers indicating the corresponding time on the timeline of a particular overlay audio event in the overlay audio data sequence, said overlay audio event markers providing visual indications regarding events within the overlay audio data sequence.
3. A user interface as set forth in claim 2 wherein each of said visual event markers, audio event markers, and overlay audio event markers further comprises at least one of an icon indicative of the nature of the corresponding event and a value indicator indicative of the relative value of the corresponding event.
4. A user interface as set forth in claim 2 wherein a value indicator of at least one of said visual event markers, audio event markers, and overlay audio event markers indicates increased value when two or more of said event markers correspond to a substantially similar time on their respective timelines.
5. A user interface as set forth in claim 2 wherein at least one of said visual event markers, audio event markers, and overlay audio event markers corresponds to a selection element for selection by a user, said selection element adapted for selection of the event by a user and movement of the event to another time location on the respective timeline by the user, thereby also moving at least a portion of the visual data sequence or the audio data sequence, or the overlay audio data sequence associated with the selection element with the respective event marker.
6. A user interface as set forth in claim 5 wherein at least one of said visual event markers and audio event markers can be selected and snapped to the location of another of said visual event markers and audio event markers located on another of said timelines
7. A user interface as set forth in claim 1 wherein at least one of said visual event markers and audio event markers further comprises at least one of an icon indicative of the nature of the corresponding event and a value indicator indicative of the relative value of the corresponding event.
8. A user interface as set forth in claim 1 further comprising at least one of a selection element for selecting the inclusion or exclusion of event markers indicating a particular type of event and a selection element for increasing or decreasing the granularity of the placement of the event markers.
9. A user interface as set forth in claim 1 wherein the visual data sequence comprises at least one of video segments and still digital images said user interface further comprising a selection element for automatically adjusting the timeline position of at least one of said digital images including the image of a particular item to appear substantially adjacent other video segments or digital images also including the image of the same item.
10. A user interface as set forth in claim 1 further comprising an aggregate timeline depicting the visual data sequence comprising at least one of textual messages, video segments, images, effects, and transitions and the audio data sequence comprising at least one of speech, music, and recorded sounds.
11. A user interface as set forth in claim 1 further comprising a selection element for selecting a time along each timeline, such that selection of the same time on each timeline increases the granularity of event markers within at least one of the visual elements and the audio elements occurring at said selected time.
12. A method for generating a user interface for an audio-visual work, said method comprising:
extracting metadata associated with a visual data sequence according to at least one characteristic of said visual data sequence;
extracting metadata associated with an audio data sequence according to at least one characteristic of said audio data sequence;
determining intersections of metadata from two or more of said extracted metadata occurring at substantially the same time;
associating metadata and metadata intersections with a visual timeline associated with the visual data sequence;
associating metadata and metadata intersections with an audio timeline associated with the audio data sequence; and
integrating the visual timeline and the audio timelines onto a user interface.
13. The method of claim 12 , further comprising:
displaying an overlay audio timeline associated with an overlay audio data sequence and comprising one or more overlay audio elements, said overlay audio timeline corresponding to the same time scale and position in time as the visual timeline and the audio timeline;
depicting, via the displayed overlay audio timeline, at least one audio characteristic of the overlay audio data sequence over time; and
displaying overlay audio event markers in the overlay audio timeline, said overlay audio markers indicating the corresponding time on the overlay audio timeline of a particular overlay audio event in the overlay audio data sequence, wherein the overlay audio markers provide visual indications regarding events within the overlay audio data sequence,
wherein the overlay audio data sequence further comprises one or more overlay audio elements.
14. The method of claim 13 , further comprising providing at least one of an icon indicative of the nature of the corresponding event and a value indicator indicative of the relative value of the corresponding event for each of the following: the overlay audio event markers, one or more visual event markers indicating the corresponding time on the visual timeline of a particular visual event in the visual data sequence, and one or more audio event markers indicating the corresponding time on the audio timeline of a particular audio event in the audio data sequence.
15. The method of claim 14 , further comprising:
increasing the value of a value indicator of each of the visual event markers, the audio event markers, and the overlay audio event markers when two or more of said event markers correspond to a substantially similar time on their respective timelines; and
indicating said increase in value via the user interface.
16. The method of claim 14 , further comprising:
providing a selection element for selection by a user, wherein at least one of the visual event markers, audio event markers, and overlay audio event markers corresponds to said selection element;
receiving, via said selection element, a selection of an event by the user and movement of the event to another time location on the respective timeline by the user; and
moving at least a portion of the visual data sequence or the audio data sequence, or the overlay audio data sequence associated with the selection element with the respective event marker.
17. The method of claim 16 , further comprising:
receiving a selection of at least one of the visual event markers and audio event markers; and
snapping to the location of another of the visual event markers and audio event markers located on another of the timelines
18. The method of claim 12 further comprising at least one of a selection element for selecting the inclusion or exclusion of event markers indicating a particular type of event and a selection element for increasing or decreasing the granularity of the placement of the event markers.
19. The method of claim 12 wherein the visual data sequence comprises at least one of textual messages, video segments, images, effects, and transitions and the audio data sequence comprises at least one of speech, music, and recorded sounds,
wherein when the visual data sequence comprises still digital images, the method further comprises providing a selection element for automatically adjusting the timeline position of at least one of the digital images including the image of a particular item to appear substantially adjacent other video segments or digital images also including an image of the same item.
20. A system for generating a user interface for creating an audio-visual work, said system comprising:
a metadata extraction module for extracting metadata associated with a visual data sequence according to at least one characteristic of said visual data sequence and for extracting metadata associated with an audio data sequence according to at least one characteristic of said audio data sequence, wherein an intersection of metadata from two or more of said extracted metadata occurs at substantially the same time and wherein each metadata and the intersection of metadata is associated with a timeline location corresponding to a timeline relating the data sequences; and
a user interface metadata association module for associating metadata and metadata intersections with a visual timeline associated with the visual data sequence and an audio timeline associated with the audio data sequence, said user interface metadata association module integrating the visual timeline and the audio timelines onto a user interface.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/079,565 US20110185269A1 (en) | 2006-06-15 | 2011-04-04 | Audio/visual editing tool |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/424,509 US7945142B2 (en) | 2006-06-15 | 2006-06-15 | Audio/visual editing tool |
US13/079,565 US20110185269A1 (en) | 2006-06-15 | 2011-04-04 | Audio/visual editing tool |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/424,509 Division US7945142B2 (en) | 2006-06-15 | 2006-06-15 | Audio/visual editing tool |
Publications (1)
Publication Number | Publication Date |
---|---|
US20110185269A1 true US20110185269A1 (en) | 2011-07-28 |
Family
ID=38861664
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/424,509 Expired - Fee Related US7945142B2 (en) | 2006-06-15 | 2006-06-15 | Audio/visual editing tool |
US13/079,565 Abandoned US20110185269A1 (en) | 2006-06-15 | 2011-04-04 | Audio/visual editing tool |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/424,509 Expired - Fee Related US7945142B2 (en) | 2006-06-15 | 2006-06-15 | Audio/visual editing tool |
Country Status (1)
Country | Link |
---|---|
US (2) | US7945142B2 (en) |
Cited By (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130239158A1 (en) * | 2012-03-12 | 2013-09-12 | Cisco Technology, Inc. | Method and System to Locally Cache and Inform Users of Partial Video Segments and Data Objects |
WO2016014591A1 (en) * | 2014-07-21 | 2016-01-28 | Avigilon Corporation | Timeline synchronization control method for multiple display views |
US9361521B1 (en) | 2015-06-14 | 2016-06-07 | Google Inc. | Methods and systems for presenting a camera history |
USD796540S1 (en) | 2015-06-14 | 2017-09-05 | Google Inc. | Display screen with graphical user interface for mobile camera history having event-specific activity notifications |
USD797131S1 (en) | 2015-06-14 | 2017-09-12 | Google Inc. | Display screen with user interface for mode selector icons |
USD797772S1 (en) | 2015-06-14 | 2017-09-19 | Google Inc. | Display screen with user interface for a multifunction status and entry point icon and device state icons |
USD803241S1 (en) | 2015-06-14 | 2017-11-21 | Google Inc. | Display screen with animated graphical user interface for an alert screen |
USD809522S1 (en) | 2015-06-14 | 2018-02-06 | Google Inc. | Display screen with animated graphical user interface for an alert screen |
USD812076S1 (en) | 2015-06-14 | 2018-03-06 | Google Llc | Display screen with graphical user interface for monitoring remote video camera |
USD843398S1 (en) | 2016-10-26 | 2019-03-19 | Google Llc | Display screen with graphical user interface for a timeline-video relationship presentation for alert events |
US10263802B2 (en) | 2016-07-12 | 2019-04-16 | Google Llc | Methods and devices for establishing connections with remote cameras |
USD848466S1 (en) | 2015-06-14 | 2019-05-14 | Google Llc | Display screen with animated graphical user interface for smart home automation system having a multifunction status |
US10386999B2 (en) | 2016-10-26 | 2019-08-20 | Google Llc | Timeline-video relationship presentation for alert events |
USD882583S1 (en) | 2016-07-12 | 2020-04-28 | Google Llc | Display screen with graphical user interface |
US10972685B2 (en) | 2017-05-25 | 2021-04-06 | Google Llc | Video camera assembly having an IR reflector |
US11035517B2 (en) | 2017-05-25 | 2021-06-15 | Google Llc | Compact electronic device with thermal management |
US11238290B2 (en) | 2016-10-26 | 2022-02-01 | Google Llc | Timeline-video relationship processing for alert events |
US11240542B2 (en) | 2016-01-14 | 2022-02-01 | Avigilon Corporation | System and method for multiple video playback |
WO2022067007A1 (en) * | 2020-09-25 | 2022-03-31 | Wev Labs, Llc | Methods, devices, and systems for video segmentation and annotation |
WO2022083223A1 (en) * | 2020-10-20 | 2022-04-28 | 深圳市前海手绘科技文化有限公司 | Animation video export optimization method and apparatus |
US11689784B2 (en) | 2017-05-25 | 2023-06-27 | Google Llc | Camera assembly having a single-piece cover element |
US11705161B2 (en) | 2020-09-25 | 2023-07-18 | Wev Labs, Llc | Methods, devices, and systems for video segmentation and annotation |
US11817131B2 (en) * | 2020-12-17 | 2023-11-14 | Gopro, Inc. | Interface for indicating video editing decisions |
Families Citing this family (52)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4543261B2 (en) * | 2005-09-28 | 2010-09-15 | 国立大学法人電気通信大学 | Playback device |
US20080071717A1 (en) * | 2006-09-06 | 2008-03-20 | Motti Nisani | Method and system for scenario investigation |
JP4660861B2 (en) * | 2006-09-06 | 2011-03-30 | 富士フイルム株式会社 | Music image synchronized video scenario generation method, program, and apparatus |
TWI395476B (en) * | 2006-12-20 | 2013-05-01 | Princeton Technology Corp | Method and related device for determining timing sequence of video and audio data for high density multimedia interface |
US7599475B2 (en) * | 2007-03-12 | 2009-10-06 | Nice Systems, Ltd. | Method and apparatus for generic analytics |
US8307287B2 (en) * | 2007-04-13 | 2012-11-06 | Apple Inc. | Heads-up-display for use in a media manipulation operation |
US20080263450A1 (en) * | 2007-04-14 | 2008-10-23 | James Jacob Hodges | System and method to conform separately edited sequences |
US20080256136A1 (en) * | 2007-04-14 | 2008-10-16 | Jerremy Holland | Techniques and tools for managing attributes of media content |
US8751022B2 (en) * | 2007-04-14 | 2014-06-10 | Apple Inc. | Multi-take compositing of digital media assets |
KR20090002292A (en) * | 2007-06-26 | 2009-01-09 | 삼성전자주식회사 | Apparatus and method for synchronizing virtual character with music and sharing it through xml based language |
US8326444B1 (en) * | 2007-08-17 | 2012-12-04 | Adobe Systems Incorporated | Method and apparatus for performing audio ducking |
US7825322B1 (en) | 2007-08-17 | 2010-11-02 | Adobe Systems Incorporated | Method and apparatus for audio mixing |
KR101138396B1 (en) * | 2007-09-11 | 2012-04-26 | 삼성전자주식회사 | Method and apparatus for playing contents in IPTV terminal |
US8428758B2 (en) * | 2009-02-16 | 2013-04-23 | Apple Inc. | Dynamic audio ducking |
US8392004B2 (en) * | 2009-04-30 | 2013-03-05 | Apple Inc. | Automatic audio adjustment |
US20110035700A1 (en) * | 2009-08-05 | 2011-02-10 | Brian Meaney | Multi-Operation User Interface Tool |
CN101727949B (en) * | 2009-10-31 | 2011-12-07 | 华为技术有限公司 | Device, method and system for positioning playing video |
US20120072845A1 (en) * | 2010-09-21 | 2012-03-22 | Avaya Inc. | System and method for classifying live media tags into types |
US8923607B1 (en) | 2010-12-08 | 2014-12-30 | Google Inc. | Learning sports highlights using event detection |
US8862254B2 (en) | 2011-01-13 | 2014-10-14 | Apple Inc. | Background audio processing |
US8842842B2 (en) | 2011-02-01 | 2014-09-23 | Apple Inc. | Detection of audio channel configuration |
US8621355B2 (en) | 2011-02-02 | 2013-12-31 | Apple Inc. | Automatic synchronization of media clips |
US8874525B2 (en) | 2011-04-19 | 2014-10-28 | Autodesk, Inc. | Hierarchical display and navigation of document revision histories |
US8533594B2 (en) * | 2011-04-19 | 2013-09-10 | Autodesk, Inc. | Hierarchical display and navigation of document revision histories |
US8533595B2 (en) * | 2011-04-19 | 2013-09-10 | Autodesk, Inc | Hierarchical display and navigation of document revision histories |
US8533593B2 (en) * | 2011-04-19 | 2013-09-10 | Autodesk, Inc | Hierarchical display and navigation of document revision histories |
US9524651B2 (en) * | 2011-07-25 | 2016-12-20 | Raymond Fix | System and method for electronic communication using a voiceover in combination with user interaction events on a selected background |
US8965774B2 (en) | 2011-08-23 | 2015-02-24 | Apple Inc. | Automatic detection of audio compression parameters |
US9344606B2 (en) * | 2012-01-24 | 2016-05-17 | Radical Switchcam Llc | System and method for compiling and playing a multi-channel video |
JP2014106637A (en) * | 2012-11-26 | 2014-06-09 | Sony Corp | Information processor, method and program |
IL223381B (en) | 2012-12-02 | 2018-01-31 | Berale Of Teldan Group Ltd | Automatic summarising of media content |
KR101978216B1 (en) * | 2013-01-04 | 2019-05-14 | 엘지전자 주식회사 | Mobile terminal and method for controlling thereof |
US8537983B1 (en) * | 2013-03-08 | 2013-09-17 | Noble Systems Corporation | Multi-component viewing tool for contact center agents |
US9411882B2 (en) | 2013-07-22 | 2016-08-09 | Dolby Laboratories Licensing Corporation | Interactive audio content generation, delivery, playback and sharing |
WO2015061964A1 (en) * | 2013-10-29 | 2015-05-07 | Hua Zhong University Of Science Technology | Simulataneous metadata extraction of moving objects |
US10349093B2 (en) * | 2014-03-10 | 2019-07-09 | Cisco Technology, Inc. | System and method for deriving timeline metadata for video content |
US9535654B2 (en) * | 2014-11-13 | 2017-01-03 | Here Global B.V. | Method and apparatus for associating an audio soundtrack with one or more video clips |
JP2016111472A (en) * | 2014-12-04 | 2016-06-20 | 株式会社リコー | Image forming apparatus, voice recording method, and voice recording program |
JP6569687B2 (en) * | 2014-12-15 | 2019-09-04 | ソニー株式会社 | Information processing method, video processing apparatus, and program |
US10984248B2 (en) * | 2014-12-15 | 2021-04-20 | Sony Corporation | Setting of input images based on input music |
US10529383B2 (en) * | 2015-04-09 | 2020-01-07 | Avid Technology, Inc. | Methods and systems for processing synchronous data tracks in a media editing system |
US10432987B2 (en) | 2017-09-15 | 2019-10-01 | Cisco Technology, Inc. | Virtualized and automated real time video production system |
CN107978310B (en) * | 2017-11-30 | 2022-11-25 | 腾讯科技(深圳)有限公司 | Audio processing method and device |
US10575069B2 (en) | 2017-12-20 | 2020-02-25 | International Business Machines Corporation | Method and system for automatically creating narrative visualizations from audiovisual content according to pattern detection supported by cognitive computing |
CN108600825B (en) * | 2018-07-12 | 2019-10-25 | 北京微播视界科技有限公司 | Select method, apparatus, terminal device and the medium of background music shooting video |
US10915566B2 (en) * | 2019-03-01 | 2021-02-09 | Soundtrack Game LLC | System and method for automatic synchronization of video with music, and gaming applications related thereto |
KR102656963B1 (en) | 2019-04-03 | 2024-04-16 | 삼성전자 주식회사 | Electronic device and Method of controlling thereof |
US11475867B2 (en) * | 2019-12-27 | 2022-10-18 | Spotify Ab | Method, system, and computer-readable medium for creating song mashups |
US11295782B2 (en) * | 2020-03-24 | 2022-04-05 | Meta Platforms, Inc. | Timed elements in video clips |
US11875781B2 (en) * | 2020-08-31 | 2024-01-16 | Adobe Inc. | Audio-based media edit point selection |
CN113038034A (en) * | 2021-03-26 | 2021-06-25 | 北京达佳互联信息技术有限公司 | Video editing method and video editing device |
CN113473204B (en) * | 2021-05-31 | 2023-10-13 | 北京达佳互联信息技术有限公司 | Information display method and device, electronic equipment and storage medium |
Citations (98)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5353391A (en) * | 1991-05-06 | 1994-10-04 | Apple Computer, Inc. | Method apparatus for transitioning between sequences of images |
US5467288A (en) * | 1992-04-10 | 1995-11-14 | Avid Technology, Inc. | Digital audio workstations providing digital storage and display of video information |
US5513306A (en) * | 1990-08-09 | 1996-04-30 | Apple Computer, Inc. | Temporal event viewing and editing system |
US5706417A (en) * | 1992-05-27 | 1998-01-06 | Massachusetts Institute Of Technology | Layered representation for image coding |
US5729252A (en) * | 1994-12-27 | 1998-03-17 | Lucent Technologies, Inc. | Multimedia program editing system and method |
US5832495A (en) * | 1996-07-08 | 1998-11-03 | Survivors Of The Shoah Visual History Foundation | Method and apparatus for cataloguing multimedia data |
US5929867A (en) * | 1996-08-07 | 1999-07-27 | Adobe System Incorporated | Floating keyframes |
US5990980A (en) * | 1997-12-23 | 1999-11-23 | Sarnoff Corporation | Detection of transitions in video sequences |
US6014706A (en) * | 1997-01-30 | 2000-01-11 | Microsoft Corporation | Methods and apparatus for implementing control functions in a streamed video display system |
US6028603A (en) * | 1997-10-24 | 2000-02-22 | Pictra, Inc. | Methods and apparatuses for presenting a collection of digital media in a media container |
US6081299A (en) * | 1998-02-20 | 2000-06-27 | International Business Machines Corporation | Methods and systems for encoding real time multimedia data |
US6154601A (en) * | 1996-04-12 | 2000-11-28 | Hitachi Denshi Kabushiki Kaisha | Method for editing image information with aid of computer and editing system |
US6167083A (en) * | 1997-04-04 | 2000-12-26 | Avid Technology, Inc. | Computer system and process for capture editing and playback of motion video compressed using interframe and intraframe techniques |
US6173317B1 (en) * | 1997-03-14 | 2001-01-09 | Microsoft Corporation | Streaming and displaying a video stream with synchronized annotations over a computer network |
US6188831B1 (en) * | 1997-01-29 | 2001-02-13 | Fuji Xerox Co., Ltd. | Data storage/playback device and method |
US6195088B1 (en) * | 1998-12-04 | 2001-02-27 | France Telecom | Method and system for controlling multimedia streams using dynamic prototypes |
US6204840B1 (en) * | 1997-04-08 | 2001-03-20 | Mgi Software Corporation | Non-timeline, non-linear digital multimedia composition method and system |
US6230172B1 (en) * | 1997-01-30 | 2001-05-08 | Microsoft Corporation | Production of a video stream with synchronized annotations over a computer network |
US6243725B1 (en) * | 1997-05-21 | 2001-06-05 | Premier International, Ltd. | List building system |
US6317795B1 (en) * | 1997-07-22 | 2001-11-13 | International Business Machines Corporation | Dynamic modification of multimedia content |
US6351765B1 (en) * | 1998-03-09 | 2002-02-26 | Media 100, Inc. | Nonlinear video editing system |
US20020038456A1 (en) * | 2000-09-22 | 2002-03-28 | Hansen Michael W. | Method and system for the automatic production and distribution of media content using the internet |
US20020054742A1 (en) * | 2000-11-07 | 2002-05-09 | The Furukawa Electric Co., Ltd. | Optical fiber and optical transmission line using the optical fiber |
US6400996B1 (en) * | 1999-02-01 | 2002-06-04 | Steven M. Hoffberg | Adaptive pattern recognition based control system and method |
US6424789B1 (en) * | 1999-08-17 | 2002-07-23 | Koninklijke Philips Electronics N.V. | System and method for performing fast forward and slow motion speed changes in a video stream based on video content |
US20020112226A1 (en) * | 1998-01-21 | 2002-08-15 | Rainer Brodersen | Menu authoring system and methd for automatically performing low-level dvd configuration functions and thereby ease an author's job |
US20020126552A1 (en) * | 2001-03-08 | 2002-09-12 | Jeong Jong Hyeok | Method of reproducing a multi-session disk |
US20020146235A1 (en) * | 2001-02-06 | 2002-10-10 | Pioneer Corporation, | Method and apparatus for playing back and editing information |
US20020147728A1 (en) * | 2001-01-05 | 2002-10-10 | Ron Goodman | Automatic hierarchical categorization of music by metadata |
US6476826B1 (en) * | 2000-08-22 | 2002-11-05 | Vastvideo, Inc. | Integrated system and method for processing video |
US6487360B1 (en) * | 1997-08-29 | 2002-11-26 | Hitachi Denshi Kabushiki Kaisha | Method, apparatus, and computer program product for editing moving image and displaying selected groups of representative images of cuts of the moving image such that representative images having a predetermined feature can be distinguished from representative images not having the predetermined feature |
US20020184180A1 (en) * | 2001-03-27 | 2002-12-05 | Debique Kirt A. | Meta data management for media content objects |
US20020193895A1 (en) * | 2001-06-18 | 2002-12-19 | Ziqiang Qian | Enhanced encoder for synchronizing multimedia files into an audio bit stream |
US20030009469A1 (en) * | 2001-03-09 | 2003-01-09 | Microsoft Corporation | Managing media objects in a database |
US20030032033A1 (en) * | 2001-04-16 | 2003-02-13 | Anglin Hugh W. | Watermark systems and methods |
US20030052910A1 (en) * | 2001-09-18 | 2003-03-20 | Canon Kabushiki Kaisha | Moving image data processing apparatus and method |
US6546188B1 (en) * | 1998-01-16 | 2003-04-08 | Sony Corporation | Editing system and editing method |
US6549922B1 (en) * | 1999-10-01 | 2003-04-15 | Alok Srivastava | System for collecting, transforming and managing media metadata |
US20030090506A1 (en) * | 2001-11-09 | 2003-05-15 | Moore Mike R. | Method and apparatus for controlling the visual presentation of data |
US6577807B1 (en) * | 1996-11-15 | 2003-06-10 | Hitachi Denshi Kabushiki Kaisha | Editing method and apparatus for moving pictures |
US6597859B1 (en) * | 1999-12-16 | 2003-07-22 | Intel Corporation | Method and apparatus for abstracting video data |
US20030142955A1 (en) * | 1997-09-12 | 2003-07-31 | Aki Hashizume | Apparatus for correcting an abnormality of video signal of a video system, its method, and recording medium storing the method |
US20030146915A1 (en) * | 2001-10-12 | 2003-08-07 | Brook John Charles | Interactive animation of sprites in a video production |
US6606388B1 (en) * | 2000-02-17 | 2003-08-12 | Arboretum Systems, Inc. | Method and system for enhancing audio signals |
US6606409B2 (en) * | 1997-07-29 | 2003-08-12 | Eastman Kodak Company | Fade-in and fade-out temporal segments |
US20030192044A1 (en) * | 2002-04-04 | 2003-10-09 | Huntsman Robert Ames | Content filtering system and method |
US20030192049A1 (en) * | 2002-04-09 | 2003-10-09 | Schneider Tina Fay | Binding interactive multichannel digital document system |
US20030225777A1 (en) * | 2002-05-31 | 2003-12-04 | Marsh David J. | Scoring and recommending media content based on user preferences |
US20030227493A1 (en) * | 2002-06-05 | 2003-12-11 | Canon Kabushiki Kaisha | System and method for creating screen saver |
US20030234805A1 (en) * | 2002-06-19 | 2003-12-25 | Kentaro Toyama | Computer user interface for interacting with video cliplets generated from digital video |
US20030237091A1 (en) * | 2002-06-19 | 2003-12-25 | Kentaro Toyama | Computer user interface for viewing video compositions generated from a video composition authoring system using video cliplets |
US20030236832A1 (en) * | 2002-06-19 | 2003-12-25 | Eastman Kodak Company | Method and system for sharing images over a communication network among a plurality of users in accordance with a criteria |
US6678332B1 (en) * | 2000-01-04 | 2004-01-13 | Emc Corporation | Seamless splicing of encoded MPEG video and audio |
US20040049419A1 (en) * | 2002-09-06 | 2004-03-11 | Knight Leonard K. | Advertising on an interactive screensaver |
US20040064500A1 (en) * | 2001-11-20 | 2004-04-01 | Kolar Jennifer Lynn | System and method for unified extraction of media objects |
US20040061791A1 (en) * | 2002-09-27 | 2004-04-01 | Fuji Photo Film Co., Ltd. | Image editing apparatus, image editing program, and image editing method |
US6721781B1 (en) * | 2000-01-25 | 2004-04-13 | International Business Machines Corporation | Method of providing an alternative audio format of a web page in response to a request for audible presentation of the same |
US6721361B1 (en) * | 2001-02-23 | 2004-04-13 | Yesvideo.Com | Video processing system including advanced scene break detection methods for fades, dissolves and flashes |
US20040070678A1 (en) * | 2001-10-09 | 2004-04-15 | Kentaro Toyama | System and method for exchanging images |
US6728729B1 (en) * | 2003-04-25 | 2004-04-27 | Apple Computer, Inc. | Accessing media across networks |
US20040085341A1 (en) * | 2002-11-01 | 2004-05-06 | Xian-Sheng Hua | Systems and methods for automatically editing a video |
US20040095374A1 (en) * | 2002-11-14 | 2004-05-20 | Nebojsa Jojic | System and method for automatically learning flexible sprites in video layers |
US20040128308A1 (en) * | 2002-12-31 | 2004-07-01 | Pere Obrador | Scalably presenting a collection of media objects |
US6760721B1 (en) * | 2000-04-14 | 2004-07-06 | Realnetworks, Inc. | System and method of managing metadata data |
US20040143604A1 (en) * | 2003-01-21 | 2004-07-22 | Steve Glenner | Random access editing of media |
US20040143590A1 (en) * | 2003-01-21 | 2004-07-22 | Wong Curtis G. | Selection bins |
US20040143598A1 (en) * | 2003-01-21 | 2004-07-22 | Drucker Steven M. | Media frame object visualization system |
US20040177319A1 (en) * | 2002-07-16 | 2004-09-09 | Horn Bruce L. | Computer system for automatic organization, indexing and viewing of information from multiple sources |
US6807306B1 (en) * | 1999-05-28 | 2004-10-19 | Xerox Corporation | Time-constrained keyframe selection method |
US6813313B2 (en) * | 2000-07-06 | 2004-11-02 | Mitsubishi Electric Research Laboratories, Inc. | Method and system for high-level structure analysis and event detection in domain specific videos |
US20050005308A1 (en) * | 2002-01-29 | 2005-01-06 | Gotuit Video, Inc. | Methods and apparatus for recording and replaying sports broadcasts |
US20050010953A1 (en) * | 2003-07-11 | 2005-01-13 | John Carney | System and method for creating and presenting composite video-on-demand content |
US20050033758A1 (en) * | 2003-08-08 | 2005-02-10 | Baxter Brent A. | Media indexer |
US20050053356A1 (en) * | 2003-09-08 | 2005-03-10 | Ati Technologies, Inc. | Method of intelligently applying real-time effects to video content that is being recorded |
US20050071774A1 (en) * | 2003-09-29 | 2005-03-31 | Lipsky Scott E. | Method and system for displaying multiple aspect ratios of a viewport |
US6877134B1 (en) * | 1997-08-14 | 2005-04-05 | Virage, Inc. | Integrated data and real-time metadata capture system and method |
US20050097120A1 (en) * | 2003-10-31 | 2005-05-05 | Fuji Xerox Co., Ltd. | Systems and methods for organizing data |
US6898799B1 (en) * | 2000-10-23 | 2005-05-24 | Clearplay, Inc. | Multimedia content navigation and playback |
US20050114357A1 (en) * | 2003-11-20 | 2005-05-26 | Rathinavelu Chengalvarayan | Collaborative media indexing system and method |
US6912693B2 (en) * | 1998-09-14 | 2005-06-28 | Microsoft Corporation | Computer-implemented image acquisition system |
US6928613B1 (en) * | 2001-11-30 | 2005-08-09 | Victor Company Of Japan | Organization, selection, and application of video effects according to zones |
US20050216454A1 (en) * | 2004-03-15 | 2005-09-29 | Yahoo! Inc. | Inverse search systems and methods |
US20050229225A1 (en) * | 2002-05-08 | 2005-10-13 | Wolfgang Klausberger | Appliance-guided edit-operations in advanced digital video recording systems |
US6961549B2 (en) * | 2001-08-02 | 2005-11-01 | Sun Microsystems, Inc. | Method for recording an audio broadcast by user preference |
US20050249080A1 (en) * | 2004-05-07 | 2005-11-10 | Fuji Xerox Co., Ltd. | Method and system for harvesting a media stream |
US20050257151A1 (en) * | 2004-05-13 | 2005-11-17 | Peng Wu | Method and apparatus for identifying selected portions of a video stream |
US20050281535A1 (en) * | 2000-06-16 | 2005-12-22 | Yesvideo, Inc., A California Corporation | Video processing system |
US20060059426A1 (en) * | 2004-09-15 | 2006-03-16 | Sony Corporation | Image processing apparatus, method, and program, and program storage medium |
US7016540B1 (en) * | 1999-11-24 | 2006-03-21 | Nec Corporation | Method and system for segmentation, classification, and summarization of video images |
US7027509B2 (en) * | 2000-03-07 | 2006-04-11 | Lg Electronics Inc. | Hierarchical hybrid shot change detection method for MPEG-compressed video |
US7054888B2 (en) * | 2002-10-16 | 2006-05-30 | Microsoft Corporation | Optimizing media player memory during rendering |
US7099946B2 (en) * | 2000-11-13 | 2006-08-29 | Canon Kabushiki Kaishsa | Transferring a media browsing session from one device to a second device by transferring a session identifier and a session key to the second device |
US20060288288A1 (en) * | 2005-06-17 | 2006-12-21 | Fuji Xerox Co., Ltd. | Methods and interfaces for event timeline and logs of video streams |
US20070009231A1 (en) * | 2003-08-22 | 2007-01-11 | Sony Corporation | Reproducing apparatus, method, method and program |
US20070044010A1 (en) * | 2000-07-24 | 2007-02-22 | Sanghoon Sull | System and method for indexing, searching, identifying, and editing multimedia files |
US20070074115A1 (en) * | 2005-09-23 | 2007-03-29 | Microsoft Corporation | Automatic capturing and editing of a video |
US7203380B2 (en) * | 2001-11-16 | 2007-04-10 | Fuji Xerox Co., Ltd. | Video production and compaction with collage picture frame user interface |
US7222300B2 (en) * | 2002-06-19 | 2007-05-22 | Microsoft Corporation | System and method for automatically authoring video compositions using video cliplets |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5999173A (en) | 1992-04-03 | 1999-12-07 | Adobe Systems Incorporated | Method and apparatus for video editing with video clip representations displayed along a time line |
US6574417B1 (en) | 1999-08-20 | 2003-06-03 | Thomson Licensing S.A. | Digital video processing and interface system for video, audio and ancillary data |
KR100295610B1 (en) | 1999-11-09 | 2001-07-12 | 장영규 | System for editing moving images and method for providing object information employing the same |
US7334191B1 (en) * | 2000-05-09 | 2008-02-19 | International Business Machines Corporation | Segmentation and detection of representative frames in video sequences |
US7398004B1 (en) * | 2001-10-16 | 2008-07-08 | Sonic Solutions | Software methods for authoring multimedia content to be written to optical media |
US7251413B2 (en) * | 2002-04-26 | 2007-07-31 | Digital Networks North America, Inc. | System and method for improved blackfield detection |
US7027513B2 (en) * | 2003-01-15 | 2006-04-11 | Microsoft Corporation | Method and system for extracting key frames from video using a triangle model of motion based on perceived motion energy |
JP2005198165A (en) * | 2004-01-09 | 2005-07-21 | Canon Inc | Device and method for reproducing image, computer program, and computer readable recording medium |
US7975062B2 (en) * | 2004-06-07 | 2011-07-05 | Sling Media, Inc. | Capturing and sharing media content |
US9430587B2 (en) * | 2006-06-05 | 2016-08-30 | Qualcomm Incorporated | Techniques for managing media content |
US8261191B2 (en) * | 2006-08-04 | 2012-09-04 | Apple Inc. | Multi-point representation |
-
2006
- 2006-06-15 US US11/424,509 patent/US7945142B2/en not_active Expired - Fee Related
-
2011
- 2011-04-04 US US13/079,565 patent/US20110185269A1/en not_active Abandoned
Patent Citations (99)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5513306A (en) * | 1990-08-09 | 1996-04-30 | Apple Computer, Inc. | Temporal event viewing and editing system |
US5353391A (en) * | 1991-05-06 | 1994-10-04 | Apple Computer, Inc. | Method apparatus for transitioning between sequences of images |
US5467288A (en) * | 1992-04-10 | 1995-11-14 | Avid Technology, Inc. | Digital audio workstations providing digital storage and display of video information |
US5706417A (en) * | 1992-05-27 | 1998-01-06 | Massachusetts Institute Of Technology | Layered representation for image coding |
US5729252A (en) * | 1994-12-27 | 1998-03-17 | Lucent Technologies, Inc. | Multimedia program editing system and method |
US6154601A (en) * | 1996-04-12 | 2000-11-28 | Hitachi Denshi Kabushiki Kaisha | Method for editing image information with aid of computer and editing system |
US5832495A (en) * | 1996-07-08 | 1998-11-03 | Survivors Of The Shoah Visual History Foundation | Method and apparatus for cataloguing multimedia data |
US5929867A (en) * | 1996-08-07 | 1999-07-27 | Adobe System Incorporated | Floating keyframes |
US6577807B1 (en) * | 1996-11-15 | 2003-06-10 | Hitachi Denshi Kabushiki Kaisha | Editing method and apparatus for moving pictures |
US6188831B1 (en) * | 1997-01-29 | 2001-02-13 | Fuji Xerox Co., Ltd. | Data storage/playback device and method |
US6014706A (en) * | 1997-01-30 | 2000-01-11 | Microsoft Corporation | Methods and apparatus for implementing control functions in a streamed video display system |
US6230172B1 (en) * | 1997-01-30 | 2001-05-08 | Microsoft Corporation | Production of a video stream with synchronized annotations over a computer network |
US6173317B1 (en) * | 1997-03-14 | 2001-01-09 | Microsoft Corporation | Streaming and displaying a video stream with synchronized annotations over a computer network |
US6167083A (en) * | 1997-04-04 | 2000-12-26 | Avid Technology, Inc. | Computer system and process for capture editing and playback of motion video compressed using interframe and intraframe techniques |
US6204840B1 (en) * | 1997-04-08 | 2001-03-20 | Mgi Software Corporation | Non-timeline, non-linear digital multimedia composition method and system |
US6243725B1 (en) * | 1997-05-21 | 2001-06-05 | Premier International, Ltd. | List building system |
US6317795B1 (en) * | 1997-07-22 | 2001-11-13 | International Business Machines Corporation | Dynamic modification of multimedia content |
US6606409B2 (en) * | 1997-07-29 | 2003-08-12 | Eastman Kodak Company | Fade-in and fade-out temporal segments |
US6877134B1 (en) * | 1997-08-14 | 2005-04-05 | Virage, Inc. | Integrated data and real-time metadata capture system and method |
US6487360B1 (en) * | 1997-08-29 | 2002-11-26 | Hitachi Denshi Kabushiki Kaisha | Method, apparatus, and computer program product for editing moving image and displaying selected groups of representative images of cuts of the moving image such that representative images having a predetermined feature can be distinguished from representative images not having the predetermined feature |
US20030142955A1 (en) * | 1997-09-12 | 2003-07-31 | Aki Hashizume | Apparatus for correcting an abnormality of video signal of a video system, its method, and recording medium storing the method |
US6028603A (en) * | 1997-10-24 | 2000-02-22 | Pictra, Inc. | Methods and apparatuses for presenting a collection of digital media in a media container |
US5990980A (en) * | 1997-12-23 | 1999-11-23 | Sarnoff Corporation | Detection of transitions in video sequences |
US6546188B1 (en) * | 1998-01-16 | 2003-04-08 | Sony Corporation | Editing system and editing method |
US20020112226A1 (en) * | 1998-01-21 | 2002-08-15 | Rainer Brodersen | Menu authoring system and methd for automatically performing low-level dvd configuration functions and thereby ease an author's job |
US6081299A (en) * | 1998-02-20 | 2000-06-27 | International Business Machines Corporation | Methods and systems for encoding real time multimedia data |
US6351765B1 (en) * | 1998-03-09 | 2002-02-26 | Media 100, Inc. | Nonlinear video editing system |
US6912693B2 (en) * | 1998-09-14 | 2005-06-28 | Microsoft Corporation | Computer-implemented image acquisition system |
US6195088B1 (en) * | 1998-12-04 | 2001-02-27 | France Telecom | Method and system for controlling multimedia streams using dynamic prototypes |
US6400996B1 (en) * | 1999-02-01 | 2002-06-04 | Steven M. Hoffberg | Adaptive pattern recognition based control system and method |
US6807306B1 (en) * | 1999-05-28 | 2004-10-19 | Xerox Corporation | Time-constrained keyframe selection method |
US6424789B1 (en) * | 1999-08-17 | 2002-07-23 | Koninklijke Philips Electronics N.V. | System and method for performing fast forward and slow motion speed changes in a video stream based on video content |
US6549922B1 (en) * | 1999-10-01 | 2003-04-15 | Alok Srivastava | System for collecting, transforming and managing media metadata |
US7016540B1 (en) * | 1999-11-24 | 2006-03-21 | Nec Corporation | Method and system for segmentation, classification, and summarization of video images |
US6597859B1 (en) * | 1999-12-16 | 2003-07-22 | Intel Corporation | Method and apparatus for abstracting video data |
US6678332B1 (en) * | 2000-01-04 | 2004-01-13 | Emc Corporation | Seamless splicing of encoded MPEG video and audio |
US6721781B1 (en) * | 2000-01-25 | 2004-04-13 | International Business Machines Corporation | Method of providing an alternative audio format of a web page in response to a request for audible presentation of the same |
US6606388B1 (en) * | 2000-02-17 | 2003-08-12 | Arboretum Systems, Inc. | Method and system for enhancing audio signals |
US7027509B2 (en) * | 2000-03-07 | 2006-04-11 | Lg Electronics Inc. | Hierarchical hybrid shot change detection method for MPEG-compressed video |
US6760721B1 (en) * | 2000-04-14 | 2004-07-06 | Realnetworks, Inc. | System and method of managing metadata data |
US20050281535A1 (en) * | 2000-06-16 | 2005-12-22 | Yesvideo, Inc., A California Corporation | Video processing system |
US6813313B2 (en) * | 2000-07-06 | 2004-11-02 | Mitsubishi Electric Research Laboratories, Inc. | Method and system for high-level structure analysis and event detection in domain specific videos |
US20070044010A1 (en) * | 2000-07-24 | 2007-02-22 | Sanghoon Sull | System and method for indexing, searching, identifying, and editing multimedia files |
US6476826B1 (en) * | 2000-08-22 | 2002-11-05 | Vastvideo, Inc. | Integrated system and method for processing video |
US20020038456A1 (en) * | 2000-09-22 | 2002-03-28 | Hansen Michael W. | Method and system for the automatic production and distribution of media content using the internet |
US6898799B1 (en) * | 2000-10-23 | 2005-05-24 | Clearplay, Inc. | Multimedia content navigation and playback |
US20020054742A1 (en) * | 2000-11-07 | 2002-05-09 | The Furukawa Electric Co., Ltd. | Optical fiber and optical transmission line using the optical fiber |
US7099946B2 (en) * | 2000-11-13 | 2006-08-29 | Canon Kabushiki Kaishsa | Transferring a media browsing session from one device to a second device by transferring a session identifier and a session key to the second device |
US20020147728A1 (en) * | 2001-01-05 | 2002-10-10 | Ron Goodman | Automatic hierarchical categorization of music by metadata |
US20020146235A1 (en) * | 2001-02-06 | 2002-10-10 | Pioneer Corporation, | Method and apparatus for playing back and editing information |
US6721361B1 (en) * | 2001-02-23 | 2004-04-13 | Yesvideo.Com | Video processing system including advanced scene break detection methods for fades, dissolves and flashes |
US20020126552A1 (en) * | 2001-03-08 | 2002-09-12 | Jeong Jong Hyeok | Method of reproducing a multi-session disk |
US20030009469A1 (en) * | 2001-03-09 | 2003-01-09 | Microsoft Corporation | Managing media objects in a database |
US20020184180A1 (en) * | 2001-03-27 | 2002-12-05 | Debique Kirt A. | Meta data management for media content objects |
US20030032033A1 (en) * | 2001-04-16 | 2003-02-13 | Anglin Hugh W. | Watermark systems and methods |
US20020193895A1 (en) * | 2001-06-18 | 2002-12-19 | Ziqiang Qian | Enhanced encoder for synchronizing multimedia files into an audio bit stream |
US6961549B2 (en) * | 2001-08-02 | 2005-11-01 | Sun Microsystems, Inc. | Method for recording an audio broadcast by user preference |
US20030052910A1 (en) * | 2001-09-18 | 2003-03-20 | Canon Kabushiki Kaisha | Moving image data processing apparatus and method |
US20040070678A1 (en) * | 2001-10-09 | 2004-04-15 | Kentaro Toyama | System and method for exchanging images |
US20030146915A1 (en) * | 2001-10-12 | 2003-08-07 | Brook John Charles | Interactive animation of sprites in a video production |
US20030090506A1 (en) * | 2001-11-09 | 2003-05-15 | Moore Mike R. | Method and apparatus for controlling the visual presentation of data |
US7203380B2 (en) * | 2001-11-16 | 2007-04-10 | Fuji Xerox Co., Ltd. | Video production and compaction with collage picture frame user interface |
US20040064500A1 (en) * | 2001-11-20 | 2004-04-01 | Kolar Jennifer Lynn | System and method for unified extraction of media objects |
US6928613B1 (en) * | 2001-11-30 | 2005-08-09 | Victor Company Of Japan | Organization, selection, and application of video effects according to zones |
US20050005308A1 (en) * | 2002-01-29 | 2005-01-06 | Gotuit Video, Inc. | Methods and apparatus for recording and replaying sports broadcasts |
US20030192044A1 (en) * | 2002-04-04 | 2003-10-09 | Huntsman Robert Ames | Content filtering system and method |
US20030192049A1 (en) * | 2002-04-09 | 2003-10-09 | Schneider Tina Fay | Binding interactive multichannel digital document system |
US20050229225A1 (en) * | 2002-05-08 | 2005-10-13 | Wolfgang Klausberger | Appliance-guided edit-operations in advanced digital video recording systems |
US20030225777A1 (en) * | 2002-05-31 | 2003-12-04 | Marsh David J. | Scoring and recommending media content based on user preferences |
US20030227493A1 (en) * | 2002-06-05 | 2003-12-11 | Canon Kabushiki Kaisha | System and method for creating screen saver |
US7222300B2 (en) * | 2002-06-19 | 2007-05-22 | Microsoft Corporation | System and method for automatically authoring video compositions using video cliplets |
US20030234805A1 (en) * | 2002-06-19 | 2003-12-25 | Kentaro Toyama | Computer user interface for interacting with video cliplets generated from digital video |
US20030237091A1 (en) * | 2002-06-19 | 2003-12-25 | Kentaro Toyama | Computer user interface for viewing video compositions generated from a video composition authoring system using video cliplets |
US20030236832A1 (en) * | 2002-06-19 | 2003-12-25 | Eastman Kodak Company | Method and system for sharing images over a communication network among a plurality of users in accordance with a criteria |
US20040177319A1 (en) * | 2002-07-16 | 2004-09-09 | Horn Bruce L. | Computer system for automatic organization, indexing and viewing of information from multiple sources |
US20040049419A1 (en) * | 2002-09-06 | 2004-03-11 | Knight Leonard K. | Advertising on an interactive screensaver |
US20040061791A1 (en) * | 2002-09-27 | 2004-04-01 | Fuji Photo Film Co., Ltd. | Image editing apparatus, image editing program, and image editing method |
US7054888B2 (en) * | 2002-10-16 | 2006-05-30 | Microsoft Corporation | Optimizing media player memory during rendering |
US20040085341A1 (en) * | 2002-11-01 | 2004-05-06 | Xian-Sheng Hua | Systems and methods for automatically editing a video |
US20040095374A1 (en) * | 2002-11-14 | 2004-05-20 | Nebojsa Jojic | System and method for automatically learning flexible sprites in video layers |
US7131059B2 (en) * | 2002-12-31 | 2006-10-31 | Hewlett-Packard Development Company, L.P. | Scalably presenting a collection of media objects |
US20040128308A1 (en) * | 2002-12-31 | 2004-07-01 | Pere Obrador | Scalably presenting a collection of media objects |
US20040143590A1 (en) * | 2003-01-21 | 2004-07-22 | Wong Curtis G. | Selection bins |
US20040143598A1 (en) * | 2003-01-21 | 2004-07-22 | Drucker Steven M. | Media frame object visualization system |
US20040143604A1 (en) * | 2003-01-21 | 2004-07-22 | Steve Glenner | Random access editing of media |
US6728729B1 (en) * | 2003-04-25 | 2004-04-27 | Apple Computer, Inc. | Accessing media across networks |
US20050010953A1 (en) * | 2003-07-11 | 2005-01-13 | John Carney | System and method for creating and presenting composite video-on-demand content |
US20050033758A1 (en) * | 2003-08-08 | 2005-02-10 | Baxter Brent A. | Media indexer |
US20070009231A1 (en) * | 2003-08-22 | 2007-01-11 | Sony Corporation | Reproducing apparatus, method, method and program |
US20050053356A1 (en) * | 2003-09-08 | 2005-03-10 | Ati Technologies, Inc. | Method of intelligently applying real-time effects to video content that is being recorded |
US20050071774A1 (en) * | 2003-09-29 | 2005-03-31 | Lipsky Scott E. | Method and system for displaying multiple aspect ratios of a viewport |
US20050097120A1 (en) * | 2003-10-31 | 2005-05-05 | Fuji Xerox Co., Ltd. | Systems and methods for organizing data |
US20050114357A1 (en) * | 2003-11-20 | 2005-05-26 | Rathinavelu Chengalvarayan | Collaborative media indexing system and method |
US20050216454A1 (en) * | 2004-03-15 | 2005-09-29 | Yahoo! Inc. | Inverse search systems and methods |
US20050249080A1 (en) * | 2004-05-07 | 2005-11-10 | Fuji Xerox Co., Ltd. | Method and system for harvesting a media stream |
US20050257151A1 (en) * | 2004-05-13 | 2005-11-17 | Peng Wu | Method and apparatus for identifying selected portions of a video stream |
US20060059426A1 (en) * | 2004-09-15 | 2006-03-16 | Sony Corporation | Image processing apparatus, method, and program, and program storage medium |
US20060288288A1 (en) * | 2005-06-17 | 2006-12-21 | Fuji Xerox Co., Ltd. | Methods and interfaces for event timeline and logs of video streams |
US20070074115A1 (en) * | 2005-09-23 | 2007-03-29 | Microsoft Corporation | Automatic capturing and editing of a video |
Cited By (49)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130239158A1 (en) * | 2012-03-12 | 2013-09-12 | Cisco Technology, Inc. | Method and System to Locally Cache and Inform Users of Partial Video Segments and Data Objects |
US9661357B2 (en) * | 2012-03-12 | 2017-05-23 | Cisco Technology, Inc. | Method and system to locally cache and inform users of partial video segments and data objects |
US9659598B2 (en) | 2014-07-21 | 2017-05-23 | Avigilon Corporation | Timeline synchronization control method for multiple display views |
WO2016014591A1 (en) * | 2014-07-21 | 2016-01-28 | Avigilon Corporation | Timeline synchronization control method for multiple display views |
US10741220B2 (en) | 2014-07-21 | 2020-08-11 | Avigilon Corporation | Timeline synchronization control method for multiple display views |
US10269393B2 (en) | 2014-07-21 | 2019-04-23 | Avigilon Corporation | Timeline synchronization control method for multiple display views |
US10558323B1 (en) | 2015-06-14 | 2020-02-11 | Google Llc | Systems and methods for smart home automation using a multifunction status and entry point icon |
USD848466S1 (en) | 2015-06-14 | 2019-05-14 | Google Llc | Display screen with animated graphical user interface for smart home automation system having a multifunction status |
USD797131S1 (en) | 2015-06-14 | 2017-09-12 | Google Inc. | Display screen with user interface for mode selector icons |
USD797772S1 (en) | 2015-06-14 | 2017-09-19 | Google Inc. | Display screen with user interface for a multifunction status and entry point icon and device state icons |
USD803242S1 (en) | 2015-06-14 | 2017-11-21 | Google Inc. | Display screen with animated graphical user interface for an alarm silence icon |
USD803241S1 (en) | 2015-06-14 | 2017-11-21 | Google Inc. | Display screen with animated graphical user interface for an alert screen |
USD809522S1 (en) | 2015-06-14 | 2018-02-06 | Google Inc. | Display screen with animated graphical user interface for an alert screen |
USD810116S1 (en) | 2015-06-14 | 2018-02-13 | Google Inc. | Display screen with graphical user interface for mobile camera history having collapsible video events |
USD812076S1 (en) | 2015-06-14 | 2018-03-06 | Google Llc | Display screen with graphical user interface for monitoring remote video camera |
US10133443B2 (en) | 2015-06-14 | 2018-11-20 | Google Llc | Systems and methods for smart home automation using a multifunction status and entry point icon |
US11599259B2 (en) | 2015-06-14 | 2023-03-07 | Google Llc | Methods and systems for presenting alert event indicators |
USD796540S1 (en) | 2015-06-14 | 2017-09-05 | Google Inc. | Display screen with graphical user interface for mobile camera history having event-specific activity notifications |
US9380274B1 (en) * | 2015-06-14 | 2016-06-28 | Google Inc. | Methods and systems for presenting alert event indicators |
US10921971B2 (en) | 2015-06-14 | 2021-02-16 | Google Llc | Methods and systems for presenting multiple live video feeds in a user interface |
US10296194B2 (en) | 2015-06-14 | 2019-05-21 | Google Llc | Methods and systems for presenting alert event indicators |
US10871890B2 (en) | 2015-06-14 | 2020-12-22 | Google Llc | Methods and systems for presenting a camera history |
US10444967B2 (en) | 2015-06-14 | 2019-10-15 | Google Llc | Methods and systems for presenting multiple live video feeds in a user interface |
US10552020B2 (en) | 2015-06-14 | 2020-02-04 | Google Llc | Methods and systems for presenting a camera history |
US9361011B1 (en) | 2015-06-14 | 2016-06-07 | Google Inc. | Methods and systems for presenting multiple live video feeds in a user interface |
USD879137S1 (en) | 2015-06-14 | 2020-03-24 | Google Llc | Display screen or portion thereof with animated graphical user interface for an alert screen |
US11048397B2 (en) | 2015-06-14 | 2021-06-29 | Google Llc | Methods and systems for presenting alert event indicators |
USD889505S1 (en) | 2015-06-14 | 2020-07-07 | Google Llc | Display screen with graphical user interface for monitoring remote video camera |
USD892815S1 (en) | 2015-06-14 | 2020-08-11 | Google Llc | Display screen with graphical user interface for mobile camera history having collapsible video events |
US9361521B1 (en) | 2015-06-14 | 2016-06-07 | Google Inc. | Methods and systems for presenting a camera history |
US11240542B2 (en) | 2016-01-14 | 2022-02-01 | Avigilon Corporation | System and method for multiple video playback |
US10263802B2 (en) | 2016-07-12 | 2019-04-16 | Google Llc | Methods and devices for establishing connections with remote cameras |
USD882583S1 (en) | 2016-07-12 | 2020-04-28 | Google Llc | Display screen with graphical user interface |
US10386999B2 (en) | 2016-10-26 | 2019-08-20 | Google Llc | Timeline-video relationship presentation for alert events |
USD920354S1 (en) | 2016-10-26 | 2021-05-25 | Google Llc | Display screen with graphical user interface for a timeline-video relationship presentation for alert events |
US11036361B2 (en) | 2016-10-26 | 2021-06-15 | Google Llc | Timeline-video relationship presentation for alert events |
USD997972S1 (en) | 2016-10-26 | 2023-09-05 | Google Llc | Display screen with graphical user interface for a timeline-video relationship presentation for alert events |
USD843398S1 (en) | 2016-10-26 | 2019-03-19 | Google Llc | Display screen with graphical user interface for a timeline-video relationship presentation for alert events |
US11238290B2 (en) | 2016-10-26 | 2022-02-01 | Google Llc | Timeline-video relationship processing for alert events |
US11680677B2 (en) | 2017-05-25 | 2023-06-20 | Google Llc | Compact electronic device with thermal management |
US11353158B2 (en) | 2017-05-25 | 2022-06-07 | Google Llc | Compact electronic device with thermal management |
US11156325B2 (en) | 2017-05-25 | 2021-10-26 | Google Llc | Stand assembly for an electronic device providing multiple degrees of freedom and built-in cables |
US10972685B2 (en) | 2017-05-25 | 2021-04-06 | Google Llc | Video camera assembly having an IR reflector |
US11689784B2 (en) | 2017-05-25 | 2023-06-27 | Google Llc | Camera assembly having a single-piece cover element |
US11035517B2 (en) | 2017-05-25 | 2021-06-15 | Google Llc | Compact electronic device with thermal management |
WO2022067007A1 (en) * | 2020-09-25 | 2022-03-31 | Wev Labs, Llc | Methods, devices, and systems for video segmentation and annotation |
US11705161B2 (en) | 2020-09-25 | 2023-07-18 | Wev Labs, Llc | Methods, devices, and systems for video segmentation and annotation |
WO2022083223A1 (en) * | 2020-10-20 | 2022-04-28 | 深圳市前海手绘科技文化有限公司 | Animation video export optimization method and apparatus |
US11817131B2 (en) * | 2020-12-17 | 2023-11-14 | Gopro, Inc. | Interface for indicating video editing decisions |
Also Published As
Publication number | Publication date |
---|---|
US20070292106A1 (en) | 2007-12-20 |
US7945142B2 (en) | 2011-05-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7945142B2 (en) | Audio/visual editing tool | |
US20210233570A1 (en) | Methods and systems for annotation and truncation of media assets | |
US10031649B2 (en) | Automated content detection, analysis, visual synthesis and repurposing | |
US7362946B1 (en) | Automated visual image editing system | |
JP2994177B2 (en) | System and method for locating boundaries between video segments | |
US8594440B2 (en) | Automatic creation of a scalable relevance ordered representation of an image collection | |
JP5533861B2 (en) | Display control apparatus, display control method, and program | |
US7432940B2 (en) | Interactive animation of sprites in a video production | |
US8548249B2 (en) | Information processing apparatus, information processing method, and program | |
CN111683209A (en) | Mixed-cut video generation method and device, electronic equipment and computer-readable storage medium | |
US20030234805A1 (en) | Computer user interface for interacting with video cliplets generated from digital video | |
US20230140369A1 (en) | Customizable framework to extract moments of interest | |
JP2007336106A (en) | Video image editing assistant apparatus | |
US20100185628A1 (en) | Method and apparatus for automatically generating summaries of a multimedia file | |
Amir et al. | Automatic generation of conference video proceedings | |
Valdés et al. | On-line video abstract generation of multimedia news | |
Augst et al. | No longer a shot in the dark: Engineering a robust environment for film study | |
AU745436B2 (en) | Automated visual image editing system | |
US20240127857A1 (en) | Face-aware speaker diarization for transcripts and text-based video editing | |
Coimbra et al. | The shape of the game | |
US20240127820A1 (en) | Music-aware speaker diarization for transcripts and text-based video editing | |
Rehatschek et al. | A tool supporting annotation and analysis of videos | |
JP6979738B1 (en) | Servers and animation recommendation systems, animation recommendation methods, programs | |
EP4216206A1 (en) | System and method for ai/xi based automatic song finding method for videos | |
WO2019030551A1 (en) | Method for applying metadata to immersive media files |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MICROSOFT CORPORATION, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:FINKELSTEIN, CHARLES A.;MERCER, IAN C.;SIGNING DATES FROM 20060519 TO 20060606;REEL/FRAME:026147/0863 |
|
STCB | Information on status: application discontinuation |
Free format text: EXPRESSLY ABANDONED -- DURING EXAMINATION |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034766/0509 Effective date: 20141014 |