WO2015169165A1

WO2015169165A1 - Method and system for creating replate videos

Info

Publication number: WO2015169165A1
Application number: PCT/CN2015/077766
Authority: WO
Inventors: Pedro Vieira Sander; Ge Chen; Diego FERNANDES NEHAB
Original assignee: The Hong Kong University Of Science And Technology
Priority date: 2014-05-06
Filing date: 2015-04-29
Publication date: 2015-11-12

Abstract

The present teaching relates to creating videos of an event. In one example, an original video is provided to a user. The original video describes an event having a plurality of objects present therein. An input is received from the user identifying at least one of the plurality of objects from the original video and a plurality of instances of the identified at least one object from different parts of the original video. A replate video is created based on the original video and the input from the user. The plurality of instances of the at least one object are played simultaneously in the replate video.

Description

METHOD AND SYSTEM FOR CREATING REPLATE VIDEOS

BACKGROUND

1. Technical Field

The present teaching generally relates to multimedia technology. More specifically, the present teaching is directed to methods, systems, and programming for creating videos.

2. Discussion of Technical Background

Fans of any sport frequently watch and exchange replays of their favorite plays. Although each sporting event is now captured from multiple angles by multiple high-definition cameras (perhaps even in 3D) , surprisingly little has changed in the way replays are experienced, ever since the advent of recorded video made the replays popular and widespread.

Oftentimes, the viewer wants to focus on particular events: a fantastic dribble, the moment at which the player strikes the ball, or the ball hits the net. Unfortunately, replays typically show an entire continuous play. To see the event of interest multiple times, the viewer has to repeatedly rewind and fast-forward the video to the important parts, or cut a piece of the video and play it in a loop. Neither alternative is desirable: the first is tedious, and the second destroys the context in which the play took place.

To make matters worse, replays are often available from a single point of view under camera pans and zooms that reflect the intentions of the camera operator. The replay viewer may be motivated by entirely different reasons. Furthermore, replays typically focus too narrowly into a particular action. This often makes it difficult for viewers to see the big picture.

Therefore, there is a need to provide an improved solution for creating videos of an event to solve the above-mentioned problems.

SUMMARY

In one example, a method, implemented on a computing device having at least one processor, storage, and a communication platform capable of connecting to a network for creating a replate video is disclosed. An original video is provided to a user. The original video describes an event having a plurality of objects present therein. An input is received from the user identifying at least one of the plurality of objects from the original video and a plurality of instances of the identified at least one object from different parts of the original video. A replate video is created based on the original video and the input from the user. The plurality of instances of the at least one object are played simultaneously in the replate video.

In a different example, a system for creating a replate video is disclosed. The system includes an original video module, a user configuration module, and a replate video module. The original video module is configured to provide an original video to a user. The original video describes an event having a plurality of objects present therein. The user configuration module is configured to receive an input from the user identifying at least one of the plurality of objects from the original video and a plurality of instances of the identified at least one object from different parts of the original video. The replate video module is configured to create a replate video based on the original video and the input from the user. The plurality of instances of the at least one object are played simultaneously in the replate video.

Other concepts relate to software for implementing the present teaching on creating a replate video. A software product, in accord with this concept, includes at least one non-transitory machine-readable medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters in association with the executable program code, and/or information related to a user, a request, content, or information related to a social group, etc.

In one example, a non-transitory machine readable medium having information recorded thereon for creating a replate video is disclosed. The recorded information, when read by the machine, causes the machine to perform a series of processes. An original video is provided to a user. The original video describes an event having a plurality of objects present therein. An input is received from the user identifying at least one of the plurality of objects from the original video and a plurality of instances of the identified at least one object from different parts of the original video. A replate video is created based on the original video and the input from the user. The plurality of instances of the at least one object are played simultaneously in the replate video.

Additional features will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings or may be learned by production or operation of the examples. The features of the present teachings may be realized and attained by practice or use of various aspects of the methodologies, instrumentalities and combinations set forth in the detailed examples discussed below.

BRIEF DESCRIPTION OF THE DRAWINGS

The methods, systems, and/or programming described herein are further described in terms of exemplary embodiments. These exemplary embodiments are described in detail with reference to the drawings. These embodiments are non-limiting exemplary embodiments, in which like reference numerals represent similar structures throughout the several views of the drawings, and wherein:

FIG. 1 is an exemplary system diagram of a replate video creation system, according to an embodiment of the present teaching；

FIG. 2 is a flowchart of an exemplary process for a replate video creation system, according to an embodiment of the present teaching；

FIG. 3 is an exemplary screen shot of a replate video of racing in which multiple racing car instances are present, according to an embodiment of the present teaching；

FIG. 4 is a detailed exemplary system diagram of an original video module, according to an embodiment of the present teaching；

FIG. 5 is an exemplary illustration of frame registration of an original video, according to an embodiment of the present teaching；

FIG. 6 is a flowchart of an exemplary process for an original video module, according to an embodiment of the present teaching；

FIG. 7 is a detailed exemplary system diagram of a user configuration module, according to an embodiment of the present teaching；

FIG. 8 is an illustration of various exemplary modes and special effects that can be set via a user interface, according to an embodiment of the present teaching；

FIG. 9 is an exemplary screen shot of a replate video of soccer in which markers and path lines special effects are applied, according to an embodiment of the present teaching；

FIG. 10 is an exemplary screen shot of a replate video of soccer in which slow motion special effect is applied, according to an embodiment of the present teaching；

FIGs. 11-13 illustrate exemplary screen shots of various replate videos with different duration settings, according to various embodiments of the present teaching；

FIG. 14 is an exemplary screenshot of a user interface, according to an embodiment of the present teaching；

FIG. 15 is a detailed exemplary system diagram of a replate video module, according to an embodiment of the present teaching；

FIG. 16 is a flowchart of an exemplary process for a replate video module, according to an embodiment of the present teaching；

FIGs. 17-18 illustrate exemplary networked environments in which a replate video creation system can be deployed, according to various embodiments of the present teaching；

FIG. 19 depicts the architecture of a mobile device which can be used to implement a specialized system incorporating the present teaching； and

FIG. 20 depicts the architecture of a computer which can be used to implement a specialized system incorporating the present teaching.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that the present teachings may be practiced without such details. In other instances, well known methods, procedures, systems, components, and/or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

Throughout the specification and claims, terms may have nuanced meanings suggested or implied in context beyond an explicitly stated meaning. Likewise, the phrase “in one embodiment/example” as used herein does not necessarily refer to the same embodiment and the phrase “in another embodiment/example” as used herein does not necessarily refer to a different embodiment. It is intended, for example, that claimed subject matter include combinations of example embodiments in whole or in part.

In general, terminology may be understood at least in part from usage in context. For example, terms, such as “and” , “or” , or “and/or, ” as used herein may include a variety of meanings that may depend at least in part upon the context in which such terms are used. Typically, “or” if used to associate a list, such as A, B or C, is intended to mean A, B, and C, here used in the inclusive sense, as well as A, B or C, here used in the exclusive sense. In addition, the term “one or more” as used herein, depending at least in part upon context, may be used to describe any feature, structure, or characteristic in a singular sense or may be used to describe combinations of features, structures or characteristics in a plural sense. Similarly, terms, such as “a, ” “an, ” or “the, ” again, may be understood to convey a singular usage or to convey a plural usage, depending at least in part upon context. In addition, the term “based on” may be understood as not necessarily intended to convey an exclusive set of factors and may, instead, allow for existence of additional factors not necessarily expressly described, again, depending at least in part on context.

The present teaching describes method, system, and programming aspects of creating replate videos of an event. The system and method of the present teaching allow users to include multiple instances of animated objects (e.g., players, balls, referees, the crowd) over a static background, offset by regular intervals in time. The frequency with which each action occurs is therefore multiplied. Together, these modifications enable the viewers of the replate videos to quickly visualize and inspect any part of the play, at any time, in a global view. Replate videos take a fraction of the size of the original input video, and produce an effect reminiscent of video game reenactments of famous sporting events. The footage, however, is real.

One of the great side-effects of replate videos is a significant size compression. To see this, imagine a player takes 90 frames to move from the start to the end of the play. Now assume three copies (instances) of this player in the replate video, each separated by 30 frames. Finally, assume the replate video is played in a loop. Clearly, only 30 frames need to be included in the replate video. Yet, a viewer can track the continuous motion of each instance of the player across the entire duration of the play. No useful information is lost, but the replate video has been compressed to a third of the original size. Replate videos are therefore light-weight and can be viewed over the web without requiring lengthy download times.

In some embodiments of the present teaching, the system and method disclosed herein can align all frames in the original video with regard to each other. Using this alignment, a single background plate can be generated as it would be seen from a wide- angle static camera. Objects and actions are then reinserted into this arena, recovering the big picture that is typically lost due to camera panning and zoom.

In some embodiments of the present teaching, the system and method disclosed herein can show some of the objects in slow motion for a part (or the entire extent of) their roles, while others remain at the regular speed. The system and methodallow the user to synchronize the objects in slow motion to the standard speed of the play at a given point in the timeline.

In some embodiments of the present teaching, the system and method disclosed herein can have a viewer listen to the sound from different parts of a replate video by simply moving the mouse cursor over the relevant region of the replate video. Alternatively, with the aid of a webcam, gaze tracking can be used.

In some embodiments of the present teaching, the system and method disclosed herein can allow users to choose from a variety of special effects that help them construct a narrative of their favorite plays. Objects can optionally be shown as blurred trails, have superimposed tracking markers, dotted lines indicating their paths, and varying fade-in/fade-out and opacity properties based on the importance of their role at each moment in the play.

FIG. 1 is an exemplary system diagram of a replate video creation system, according to an embodiment of the present teaching. In this embodiment, the replate video creation system 100 obtains one or more original input videos of an event from a video source 104 and receives inputs from a user 102 with respect to the original input videos. Based on the user input, the replate video creation system 100 creates a replate video, which can be provided to a publisher 106 for downloading or streaming by viewers.

The video source 104 in this embodiment may be any data source that stores compressed or non-compressed video files in any suitable format, such as but not limited to, AVI, QuickTime, Flash Video, RealMedia, MPEG, etc. Each original video stored in the video source 104 describes an event having multiple objects present therein. The event may be, for example, a sporting event such as soccer, basketball, car racing, skating, bowling, etc., a social event, a performance event, or any other kind of event. The objects present in the event may be people (e.g., players and referees in a sporting event) , things (e.g., balls or cars in a sporting event) , animals, plants, buildings, etc. The publisher 106 in this embodiment may be any entity that hosts spaces in its assets (e.g., websites, applications, television channels) for presenting the replate videos created by the replate video creation system 100. For example, the publisher 106 may provide replate videos of ongoing FIFA World Cup games on its website and/or mobile apps for viewers to download or stream. The publisher 106 may be a television station and can broadcast replate videos in its sports programs.

The users 102 in this embodiment may be authors of the replate videos. The users 102 may provide configuration inputs with respect to the original videos via a user interface of the replate video creation system 100 so that the replate video creation system 100 can create the replate video accordingly. The users 102 may enjoy the replate videos created by him/herself as viewers or instruct the replate video creation system 100 to publish the replate videos to the public via the publisher 106.

The replate video creation system 100 in this embodiment includes an original video module 108, a user configuration module 110, and a replate video module 112. The original video module 108 may be operatively connected to the video source 104 for retrieving original input videos. As will be described later in detail, the original video module 108 may perform preprocessing on the original videos, such as stitching the input video frames into a consistent static background plate (i.e., a global background on which all objects and actions thereof occur) . In other words, the original video module 108 may use all the frames to generate a frame-aligned video, where all the frames are transformed based on the resulting alignment and superimposed over a blank background. The original video module 108 may further provide the original videos (either in their native forms or being registered into the background plate) to the users 102. The users 102 may interact with the original videos, for example, by playing/pausing the videos or fast-forwarding/rewinding the videos.

The user configuration module 110 in this embodiment provides a user interface with the original videos to the users 102 to facilitate the users 102 to provide configuration input for creating the replate video. The users 102 may identify one or more objects of interest from all the objects present in the original video. The users 102 may also identify a number of instances (copies) for any object from different parts of the original video by setting the duration of the replate video. For example, for an original video of a soccer event having 90 frames, the users may set the duration of the replate video to be 1/3 of the original video (30 frames) so that an object in the soccer event (e.g., the ball or a player) can have a maximum number of three instances that will be played simultaneously in the replate video. Each instance of the object is offset by 30 frames, i.e., the first instance corresponding to the object appearing in frames 1-30, the second instance corresponding to the object appearing in frames 31-60, and the third instance corresponding to the object appearing in frames 61-90. As will be described later in detail, additional settings may be made by the users 102 via the user interface of the user configuration module 110, such as modes of each identified object, special effects of each identified object, playing speeds of each identified object, and localized sounds in different sound regions in the replate video.

The replate video module 112 in this embodiment creates the replate video based on the original video and the input of user configuration. The multiple instances of the same object of interest are played simultaneously in the replate video. Thus, viewers of the replate video can quickly and repeatedly visualize and inspect any part of the event related to the object of interest. That is, viewers can track the continuous motion of each instance of the object of interest across the entire duration of the play. In some embodiments, if multiple objects of interest are set at different playing speeds, for example one of them at the regular speed of the original video while the other one in slow motion, the replate video module 112 may synchronize the multiple objects at a given time point when the multiple objects interact with each other. That is, the replate video module 112 can synchronize the actions of different objects happening at different speeds so they meet at a relevant moment. For example, if the ball in a soccer replate video is at the regular speed while the player striking the ball is in slow motion in the replate video, then at least at the time when the foot of the player hits the ball, the ball and the player are synchronized. Moreover, any special effects that have been specified by the users 102 to be added to any object of interests are blended with the corresponding object by the replate video module 112 when the object is rendered and inserted into the static background plate’s reference frames.

FIG. 2 is a flowchart of an exemplary process for a replate video creation system, according to an embodiment of the present teaching. Starting at 202, an original video is provided to a user. The video describes an event having a plurality of objects present therein. It is understood that more than one original video describing the same event, for example multiple replays of the same soccer event taken by different cameras at different angles, may be provided to the user. The events can be of any kinds, such as sporting events, social events, performance events, etc. The objects may include people, things, animals, plants, buildings, etc. In some embodiments, the original video is registered into the static background formed by all the aligned frames of the original video. At 204, an input from the user is received. The user input identifies one or more objects of interest from the original video. The user input may further identify instances of an object of interest from different parts of the original video. The same object appears in different frames in the original video separated by a regular interval. Each instance of the object corresponds to a respective fraction of frames in which the object appears. At 206, a replate video is created based on the original video and the user input. The multiple instances of the object are played simultaneously in the replate video. The different instances of the same object usually occur at different locations in the background plate. For example, the same soccer player runs across the soccer field to strike the ball in a soccer replay video so that the different instances of the player occur at different locations of the soccer field. Thus, when the different instances of the same object are played simultaneously in the replate video, they usually would not interfere with each other.

FIG. 3 is an exemplary screen shot of a replate video of racing in which multiple racing car instances are present, according to an embodiment of the present teaching. The replate video shows overtaking during a Formula One race. All frames in the original video are aligned with regard to each other to form the background plate 300, which has a significantly wider angle as taken by a wide-angle static camera. For each of the two Formula One cars, four instances 302, 304, 306, 308 are created and simultaneously displayed at different locations of the speedway. Thus, viewers of the replate video can easily inspect the overtaking at different stages by focusing on the respective instance 302, 304, 306, 308. The linear presentation of actions of each object in the original video is reorganized by the replate video creation system 100 such that different actions of the same object occurred at different times in the original video can be present simultaneously in the replate video.

FIG. 4 is a detailed exemplary system diagram of an original video module, according to an embodiment of the present teaching. The original video module 108 in this example includes an original video obtaining unit 402, a video frame registration unit 404, a background plate generation unit 406, and an original video playing unit 408. The original video obtaining unit 402 in this embodiment is operatively coupled to the video source 104 to obtain original input videos of an event. In one example, the user 102 may specify a particular event of interest, and the original video obtaining unit 402 then fetches the related one or more videos from the video source 104. For example, the user 102 may be interested in the replay of the first goal in FIFA World Cup 2014. The original video obtaining unit 402 may fetch the corresponding video from a website specializing in collecting FIFA World Cup 2014 replays. In some embodiments, the user 102 may provide the video file or link to the video to the original video obtaining unit 402 for downloading and/or storing.

The video frame registration unit 404 in conjunction with the background plate generation unit 406 performs the preprocessing on the original input video obtained by the original video obtaining unit 402. The video frame registration unit 404 in this embodiment stitches the input video frames into a consistent background plate by any known video frame registration techniques. For example, automatic panorama stitching pipeline may be used to align video frames. For feature extraction and matching, SURF features may be used due to their high efficiency and quality. In one example, due to the sequential nature as well as high temporal coherence and overlap of the video frames, only the two nearest neighbors may be considered for each feature during matching. Additionally, instead of selecting the image pairs with the most feature matches among all video candidates, only the n (e.g., n ＝ 5) frames immediately before or after the current frame may be considered in one example. Note that this simply provides a speed optimization, and the matches may be verified using the probabilistic model. FIG. 5 is an exemplary illustration of frame registration of an original video, according to an embodiment of the present teaching. FIG. 5 shows three aligned frames overlaid over an empty background plate. In this embodiment, all the frames of the original input video are used to generate a frame-aligned video, where all the frames are transformed based on the resulting alignment and superimposed over a blank background.

The background plate generation unit 406 in this embodiment generates the background plate based on the frame-aligned video. Some of the moving objects in the frame-aligned video processed by the video frame registration unit 404 may get blended on the background plate. The background plate generation unit 406 in this embodiment masks out all the moving objects. To do so, the background plate generation unit 406 may perform foreground detection by applying the ViBe algorithm on the frame-aligned video. In one example, for better results, the background plate generation unit 406 may bootstrap ViBe’s cache using the average color value of each pixel location throughout the frame-aligned video. Pixels that ViBe labels as foreground are then mapped to their original image frames and masked out (i.e., removed from consideration for the background plate) . Additionally, the users 102 can optionally select static regions of the video to be masked out (e.g., the region where a digital scoreboard is overlaid by a television broadcaster) . Once these masks are determined, the background plate generation unit 406 stitches the unmasked regions of the remaining images by feathering, resulting in a cleaner background. Additionally, the background plate generation unit 406 may compute dense optical flow for all pixels on the frame-aligned video. The optical flow information may be used for object tracking.

The original video playing unit 408 in this embodiment may play the original video for the user 102. The original video may be in its native form without being preprocessed by the video frame registration unit 404 and the background plate generation unit 406 or may be registered into the static background. In some embodiments, the original video may be presented to the user 102 in both its native form and preprocessed form.

FIG. 6 is a flowchart of an exemplary process for an original video module, according to an embodiment of the present teaching. Starting at 602, an event is identified. The event may be specified by a user or automatically determined by the original video module 108. At 604, one or more original videos describing the identified event are obtained. It is understood that in some embodiments, 602 may be omitted and an original video may be obtained by its link or provided by the user directly. At 606, frames of each original video are registered to generate a frame-aligned video. For example, all the video frames may be aligned with each other to generate a panoramic video. At 608, a background plate is generated based on the frame-aligned video. For example, moving objects in the frame-aligned video are masked out in the background plate. At 610, the original video registered into the background plate is provided to the user.

FIG. 7 is a detailed exemplary system diagram of a user configuration module, according to an embodiment of the present teaching. The user configuration module 110 in this embodiment includes a user interface 702, a timeline unit 704, an object identifying unit 706, a mode selection unit 708, a special effect unit 710, a playing speed setting unit 712, an object instance identifying unit 714, a duration setting unit 716, a sound region setting unit 718, and a localized sound designating unit 720. The user interface 702 facilities user interaction with each of the other units in the user configuration module 110.

The object identifying unit 706 in this embodiment allows the user 102 to identify any one of the objects present in the original video, which is to be displayed in the replate video. In one example, the user 102 may manually select any object in the original video by any suitable input device, such as mouse, touch pad, touch screen, keyboard, etc. In another example, the object identifying unit 706 may automatically recognize some or all of the objects present in the original video to facilitate the user 102’s selection of any one of them. In still another example, the user 102 may first designate an area in the original video and the object identifying unit 706 then may automatically recognize any object in the designated area as the selected object.

The mode selection unit 708 in this embodiment allows the user 102 to select one of the predefined modes for each of the identified objects of interest. Referring now to FIG. 8, the modes 802 include, for example, a still mode, a motion mode, a trail mode, and an inpaint/black mode. Each mode defines how the corresponding object is displayed in the replate video. The still mode allows the user 102 to place a static image region from one of the video frames into the replate video. For example, the user 102 may want the referees in a sporting event to stay still during the event since their movements often irrelevant and distracting. When a still region is selected from the original video, a timeline entry of the object may be added by the timeline unit 704 with a particular vertical bar indicating the frame from the input video where this still was taken from. When generating the replate video, feathering may be used to blend the still into the background.

The motion mode is the mode for moving objects, such as players and the ball in a sporting event. The user 102 may select the object and the desired range of frames during which the object should appear. For example, the scoring player in a soccer relay video appears from the time he starts running until after the goal is scored. The chosen range of frames from the original input video may be identified by the timeline unit 704 as a particular bar in the object’s timeline. Note that the range may later be changed by dragging the edges of the bar. Feathering may be used for compositing the moving objects into the background in the replate video. The user interface 702 may provide an object tracking tool to help the user 102 quickly select an interest region of any shape around the object and track it through the entire time range. Tracking of the interest region may be accomplished by using the dense optical flow computed during preprocessing and Kalman filtering to further improve the estimate. Additionally, the user interface 702 allows for simple and fast manual editing to modify the tracked path for instances where automatic tracking based on optical flow is not appropriate (e.g., occlusion, fast moving balls, and blurred regions due to abrupt camera motions) .

The trail mode operates just like the motion mode, but instead of showing the moving object, it blends faded-out versions of the object along its path in the replate video. The result is blurred trail or streak effect. The user 102 can optionally apply Laplacian smoothing to the path of the object to achieve a more pleasant rendering. Oftentimes the user 102 wants to remove an undesirable region from the replate video. Two different options are thus provided in the inpaint/black mode: Inpainting the selected regions of the replate video, or filling the region in black. It is understood that other techniques such as texture synthesis could be used as well to remove undesired objects from the replate video.

The special effect unit 710 in this embodiment allows the user 102 to apply special effects for each of the identified objects of interest. Referring now to FIG. 8, the special effects 804 include, for example, slow motion, markers, path lines, fade-in/fade-out, opacity, z-order, and z-subject. For markers special effects, any kind of markers can be automatically added above an object as it moves along in the replate video. If there are multiple instances of the object, each instance may be given the same type of marker with different colors. The colors are consistently assigned across different objects based on their temporal position in the timeline. In a soccer replate video, the color of the marker shown above an instance of a passing player may match the color of the marker shown above the instance of a running player that will receive that pass. This helps the viewer more easily track which instance of the player will be receiving the pass. The special effect unit 710 may allow the marker to be displayed over only of the instances. This helps prevent clutter while still allowing easier tracking in some of the instances.

The special effect unit 710 allows the path of any object to be overlaid with a dotted line (path line special effect) . This is a less obtrusive alternative to rendering objects using the trail mode. It is useful to show important aspects of the play, such as the path of the ball in a soccer replate video. As with markers, Laplacian smoothing is applied to path lines. FIG. 9 is an exemplary screen shot of a replate video of soccer in which markers and path lines special effects are applied, according to an embodiment of the present teaching. This example shows the placement of one marker over at most one instance of the passing and scoring players. Note that the markers are synchronized on the same instance of the ball. A dotted line that tracks the ball location is added as the path line special effect.

As to the fade-in/fade-out special effect, the special effect unit 710 allows objects to be faded in and out at different speeds at the discretion of the user 102. Different fading speed settings (e.g., 5, 10, and 15 frames) may be set by the user 102 via the user interface 702. Objects can be drawn with different opacity settings by the special effect unit 710. For example, lead players are normally opaque, while supporting objects can be deemphasized with a lower opacity. This helps reduce excessive clutter in the replate video. For Z-order special effect, occlusion relations of different objects can be specified. For example, a Z-value between 0 and 80 may be specified by the user 102 via the user interface 702. During composition, a greater-than Z-test may be used to select the order. Preset depth values associated to the following tags may be provided in different modes: black or inpaint (70) , motion (50) , trail (20) , and still (10) . The user 102 can then fine-tune the relative ordering by changing the Z-value of each timeline entry of the objects. When multiple instances of the same object are added to the timeline, it is possible for them to overlap one another. In that case, the user 102 has the option to specify whether the rendering of instances from a later part of a play should appear in front or behind earlier instances (i.e., z-subject special effect) .

The slow motion special effect may be achieved by the special effect unit 710 in conjunction with the playing speed setting unit 712. The playing speed setting unit 712 allows objects to appear at different playing speeds in the replate video. It is helpful in highlighting a particular aspect of the play (e.g., slow motion to show how a player strikes a soccer ball) , as well as realism (e.g., regular speed to emphasize the power of the kick) . Actions happening at different speeds are synchronized so they meet at a relevant moment (e.g., the time when the foot hits the ball) . The user interface 702 allows the user 102 to specify a speed factor for each object of interest, as well as the synchronization frame between them. For example, a first object may be specified by the user 102 at a first playing speed and a second object may be specified by the user 102 at a second playing speed. The first and second objects in the replate video are synchronized with each other at a time when the first and second objects interact with each other. In one example, the first playing speed is the same as a playing speed of the original video (regular playing speed) , and the second playing speed is slower than the playing speed of the original video (in slow motion) . FIG. 10 is an exemplary screen shot of a replate video of soccer in which slow motion special effect is applied, according to an embodiment of the present teaching. In this example, selective slow motion (factor of 3x) special effect is applied to the scoring player and the two nearest defenders (1002) . They are synchronized with the rest of the scene based on the position of the ball.

The same object may also be shown at different playing speeds during the play. To achieve this, different timeline entries may be created for the same object, each representing adjacent time segments in the original input video, and synchronized at the transition frame between them. The slow motion may be previewed by simple frame duplication. When creating the replate video, optical flow may be used to create smoother interpolated animations.

In this embodiment, the object instance identifying unit 714 in conjunction with the duration setting unit 716 allows the user 102 to include multiple instances of the same subject, separated by regular intervals in time. In order to enable viewers to see any part of the event at any time, replate videos often last a short time period, for example about 1–2 seconds (which may be automatically played in a continuous loop) . The shorter the duration, the more instances of each object may be included. This can cause too much overlap and clutter. Thus, the user 102 has a balance between the two extremes. This sweet spot is specific to each play. FIGs. 11-13 illustrate exemplary screen shots of various replate videos with different duration settings, according to various embodiments of the present teaching. In each case, the timeline for the scoring player appears below the replate video. Note that the instances of players in FIG. 11 are very sparse with too much time between actions. In contrast, the instances of players in FIG. 13 are too cluttered. The number of instances in FIG. 12 provides a more reasonable tradeoff for this play.

The number of instances of an object that are visible in the replate video at any given time also depends on the length of the object’s role in the play (the number of frames in which the object appears in the original video) . Furthermore, the user 102 can also change the number of instances of an object temporally along the replate video, by fading it in and out at the beginning and end of its role. In FIGs. 11-13, the player on the top-right who passes the ball has fewer instances than the player who makes the run and scores the goal.

In this embodiment, the sound region setting unit 718 in conjunction with the localized sound designating unit 720 allows the user 102 to include snippets of the audio that have the duration of the replate video and selectively play them based on viewer demand. That is, when one of the sound regions in the replate video is activated (e.g., by moving the mouse cursor to the sound region, finger touching the sound region, gazing at the sound region, etc. ) , the corresponding sound is played. For example, in a bowlingreplate video, a viewer may want to hear the sound of the ball striking the pins. The replate video may play that looping sound continuously whenever the mouse cursor is placed over the region of the replate video where the strike occurs. The viewer can hear the sound of the ball striking the floor after the throw by moving the mouse cursor over the bowling player. The sound region setting unit 718 allows the user 102 to configure these sound regions, and the localized sound designating unit 720 allows the user 102 to designate specific sound snippets to each of them. A default sound may also be played if the mouse cursor is outside all of those regions.

In some embodiments, off-the-shelf gaze tracking software with a webcam may be used to determine the region of the screen the viewer is currently looking at. This information may be used to drive the selection of the localized sound to be played. This provides an intuitive way to visualize a play, since the viewer only repeatedly hears the localized sound from the part of the play they are currently looking at.

FIG. 14 is an exemplary screenshot of a user interface, according to an embodiment of the present teaching. The user interface 1400 includes four

main panels

1402, 1404, 1406, 1408. The input panel 1402 shows the original video registered into the background plate. The result panel 1404 continuously loops a preview of the replate video being created. The timeline panel 1406 shows the timeline for each selected object in the replate video. It is used to specify when and how objects appear and disappear in the replate video. Each timeline is parameterized by time over the original input video. In this example, when a still region is selected from the input panel 1402 in the still mode, a timeline entry is added with a vertical yellow bar (e.g., 1410 in the top timeline entry) indicating the frame from the input video where this still was taken from. In the motion mode, the object appears in the timeline as a green bar (e.g., 1412 in the second timeline entry) representing the chosen range of frames from the original input video. Note that the range may later be changed by dragging the edges of the bar. The length of the green bar also represents the object’s role in the play. Moreover, in the timelines shown in FIG. 14, the space between vertical blue lines (e.g., between 1414 and 1416) represents the selected replate duration. These lines are updated dynamically when the user 102 modifies the duration.

The control panel 1208 is used to perform seeking over the original input video, add objects to the timeline, and specify settings of each object instance that is added to the timeline. As mentioned above with respect to FIG. 7, duration, modes, speed, fade, mark, path, and other configuration parameters may be specified by the user 102 via the control panel 1408 of the user interface 1400.

FIG. 15 is a detailed exemplary system diagram of a replate video module, according to an embodiment of the present teaching. Based on the original video and the user input, the replate video module 112 in this embodiment creates the replate video to be provided to the publisher 106 for downloading or streaming by viewers or provided to the user 102 for previewing. The replate video module 112 in this embodiment includes a plurality of static object rendering units 1502, a plurality of dynamic object rendering units 1504, a dynamic object synchronization unit 1506, and a replate video rendering unit 1508.

Each of the static object rendering units 1-n 1502 is configured to render a respective static object of interest. As mentioned above, the user 102 may via the user interface 702 to select objects of interest and designate some of them as in the still mode. Objects in the still mode (i.e., static objects) are kept still in the replate video. The static object rendering unit 1502 renders the object based on the specific frame from the original input video where this still was taken from. For example, feathering may be used to blend the still into the background by the static object rendering units 1-n 1502.

Each of the dynamic object rendering units 1-n 1504 is configured to render a respective dynamic object of interest and includes a plurality of object instance rendering units 1-m 1510, an object instance synchronization unit 1512, and a special effect applying unit 1514. As mentioned above, the user 102 may via the user interface 702 to select objects of interest and designate some of them in the motion mode or trail mode. Each of the object instance rendering units 1-m 1510 is configured to render a respective instance of the dynamic object. For example, feathering may be used to blend the dynamic object instances into the background by the dynamic object instance rendering units 1-m 1510. The object instance synchronization unit 1512 synchronizes all object instances, so each of them can be seen over the entire length of the replate video, merging seamlessly when the replate video is looped. As mentioned above, different timeline entries may be created for different instances of an object, each representing adjacent time segments in the original input video. The timeline entries are synchronized at the transition frame between them by the object instance synchronization unit 1512. The special effect applying unit 1514 applies the special effect (s) identified by the user 102 for each object instance.

The dynamic object synchronization unit 1506 is responsible for synchronizing different objects at different playing speeds to the regular speed of the original video at a given point in the timeline. The replate video rendering unit 1508 in this embodiment receives the rendered static objects and synchronized dynamic objects (and the multiple instances thereof if any) and inserts them into the background plate to form the replate video. The replate video rendering unit 1508 may automatically play the replate video in a continuous loop. By placing all objects of interest in short looping animations over a static background, replate videos allow viewers to quickly analyze and inspect any moment of the play.

FIG. 16 is a flowchart of an exemplary process for a replate video module, according to an embodiment of the present teaching. Starting at 1602, objects of interest to be presented in the replate video are identified from the original video. At 1604, whether each object is in the still mode or motion mode is determined. If an object is in the motion mode (i.e., the object is a dynamic object) , at 1606, it is further determined whether there are one or more instances of the dynamic object. If there are multiple instances of the object, then at 1608, each object is rendered with the designated special effect (s) if any. At 1610, the multiple object instances are synchronized. If at 1606, it is determined that there is only one instance of the object, then at 1612, the dynamic object is rendered with designated special effect (s) if any. At 1614, it is determined whether there are more dynamic objects to be rendered. Once all the dynamic objects are rendered, at 1616, dynamic objects with different playing speeds (e.g., in slow motion) are synchronized. Back to 1604, if it is determined that the object is in the still mode (i.e., the object is a static object) , then at 1618, the static object is rendered. At 1620, it is determined whether there are more static objects to be rendered. Once all the static objects are rendered, at 1622, the background plate is obtained. At 1624, the replate video is created by inserting all the rendered static and dynamic objects into the background plate.

Although the processing blocks of FIG. 16 are illustrated in a particular order, those having ordinary skill in the art will appreciate that the processing can be performed in any suitable orders. For example, 1622 may be performed prior to 1602. That is, the creation of the background plate can be performed earlier than identifying the objects. It is also understood that any one of the multiple dynamic objects can have one instance or multiple instances as needed.

FIGs. 17-18 illustrate exemplary networked environments in which a replate video creation system can be deployed, according to various embodiments of the present teaching. In FIGs. 17-18, the networked environments include the replate video creation system 100, the users 102, the publisher 106, a network 1702, and content providers 1704 including a content provider 1 1704-1, a content provider 2 1704-2, …, a content provider n 1704-n.

The network 1702 may be a single network or a combination of different networks. For example, a network may be a local area network (LAN) , a wide area network (WAN) , a public network, a private network, a proprietary network, a Public Switched Telephone Network (PSTN) , the Internet, a wireless network, a cellular network, a virtual network, or any combination thereof. A network 1702 may also include various network access points, e.g., wired or wireless access points such as base stations or Internet exchange points 1702-1, …, 1702-2, through which a data source may connect to the network 1702 in order to transmit information via the network 1702, and a network node may connect to the network 1702 in order to receive information.

The users 102 may be of different types such as end users connected to the network 1702 via desktop connections (102-1) , users connecting to the network 1702 via wireless connections such as through a laptop (102-2) , a handheld device (102-4) , or a built-in device in a mobile vehicle such as a motor vehicle (102-3) . The users 102 may be connected to the network 1702 and able to communicate with the replate video creation system 100.

The publisher 106 may be any entity that hosts one or more spaces in its assets (e.g., web sites, applications, television channels, etc. ) for presenting content items, e.g., videos, to viewers. For example, replate videos created by the replate video creation system 100 may be provided by the publisher 106 for downloading or streaming by viewers on their computers or mobile devices. The publisher 106 may also be a search engine, a blogger, a television station, a newspaper issuer, a web page host, a content portal, an online service provider, or a game server. For example, replate videos created by the replate video creation system 100 may be provided to a television station for analysis during intermissions of sporting events.

The content providers 1704 include multiple content providers 1704-1, 1704-2, …, 1704-n, such as video sources from which original input videos can be obtained by the replate video creation system 100. A content provider 1704 may correspond to a content source, including any web site hosted by an entity, whether an individual, a business, or an organization such as USPTO. gov, cnn. com and Yahoo. com, or a content feed source. In addition to providing original input videos to the replate video creation system 100, the content providers 1704 may also provide content items to the publisher 106 so that they can be presented to the viewers with the replate videos.

In this embodiment, as described above in detail, the replate video creation system 100 obtains original input video (s) describing an event, such as a sporting event, and having multiple objects present in the event. The replate video creation system 100 then provides a user interface to the users 102 with the original input video (s) registered into a background plate and receives user inputs with respect to the original input videos. The user inputs may identify various types of information, such as object (s) of interest in the original input video, instances of each object from different parts of the original input videos to be played simultaneously in the replate video, modes of each object, special effects of each object, playing speed of each object, etc. Based on the registered original video (s) and user inputs, the replate video creation system 100 creates the replate video in which the multiple instances of each object appear simultaneously while being kept synchronized, so that viewers can quickly and repeatedly inspect any actions of the object. The created replate video may be provided by the replate video creation system 100 to the publisher 106 for streaming or downloading. For example, a replate video may be played on the website of the publisher 106 in a continuous loop.

FIG. 17 shows a networked environment 1700 in which the replate video creation system 100 serves as an independent service provider in relation to the publisher 106 and the content providers 1702. That is, in this networked environment 1700, the replate video creation system 100 may independently provide service of creating replate videos to the publisher 106, any of the content providers 1704, or any other entities.

FIG. 18 presents a slightly different networked environment 1800 in which the replate video creation system 100 serves as a backend sub-system of the publisher 106 for dedicatedly providing replate video creation service to the particular publisher 106. In other words, the publisher 106 deploys a dedicated backend sub-system including the replate video creation system 100 for improving its content serving quality, thereby attracting more traffic to its assets. Although not know in figures, one or more content providers 1704 may also deploy a dedicated backend sub-system including the replate video creation system 100 or subscribe the services provided by the replate video creation system 100 in order to provide replate videos to its customers in addition to original videos.

FIG. 19 depicts the architecture of a mobile device which can be used to realize a specialized system implementing the present teaching. In this example, the user device on which the user interface with original video is presented and interacted with is a mobile device 1900, including, but is not limited to, a smart phone, a tablet, a music player, a handled gaming console, a global positioning system (GPS) receiver, and a wearable computing device (e.g., eyeglasses, wrist watch, etc. ) , or in any other forms. The mobile device 1900 in this example includes one or more central processing units (CPUs) 1902, one or more graphic processing units (GPUs) 1904, a display 1906, a memory 1908, a communication platform 1910, such as a wireless communication module, storage 1912, and one or more input/output (I/O) devices 1914. Any other suitable component, including but not limited to a system bus or a controller (not shown) , may also be included in the mobile device 1900. As shown in FIG. 19, a mobile operating system 1916, e.g., iOS, Android, Windows Phone, etc., and one or more applications 1918 may be loaded into the memory 1908 from the storage 1912 in order to be executed by the CPU 1902. The applications 1918 may include a browser or any other suitable mobile apps for rendering and interacting with the user interface and original video on the mobile device 1900. User interactions with the user interface and original video may be achieved via the I/O devices 1914 and provided to the replate video creation system 100 via the communication platform 1910.

To implement various modules, units, and their functionalities described in the present disclosure, computer hardware platforms may be used as the hardware platform (s) for one or more of the elements described herein (e.g., the replate video creation system 100 described with respect to FIGs. 1-18) . The hardware elements, operating systems and programming languages of such computers are conventional in nature, and it is presumed that those skilled in the art are adequately familiar therewith to adapt those technologies to video creation as described herein. A computer with user interface elements may be used to implement a personal computer (PC) or other type of work station or terminal device, although a computer may also act as a server if appropriately programmed. It is believed that those skilled in the art are familiar with the structure, programming and general operation of such computer equipment and as a result the drawings should be self-explanatory.

FIG. 20 depicts the architecture of a computing device which can be used to realize a specialized system implementing the present teaching. Such a specialized system incorporating the present teaching has a functional block diagram illustration of a hardware platform which includes user interface elements. The computer may be a general purpose computer or a special purpose computer. Both can be used to implement a specialized system for the present teaching. This computer 2000 may be used to implement any component of video creation techniques, as described herein. For example, the replate video creation system 100 may be implemented on a computer such as computer 2000, via its hardware, software program, firmware, or a combination thereof. Although only one such computer is shown, for convenience, the computer functions relating to video creation as described herein may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load.

The computer 2000, for example, includes COM ports 2002 connected to and from a network connected thereto to facilitate data communications. The computer 2000 also includes a central processing unit (CPU) 2004, in the form of one or more processors, for executing program instructions. The exemplary computer platform includes an internal communication bus 2006, program storage and data storage of different forms, e.g., disk 2008, read only memory (ROM) 2010, or random access memory (RAM) 2012, for various data files to be processed and/or communicated by the computer, as well as possibly program instructions to be executed by the CPU 2004. The computer 2000 also includes an I/O component 2014, supporting input/output flows between the computer and other components therein such as user interface elements 2016. The computer 2000 may also receive programming and data via network communications.

Hence, aspects of the methods of video creation and/or other processes, as outlined above, may be embodied in programming. Program aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of executable code and/or associated data that is carried on or embodied in a type of machine readable medium. Tangible non-transitory “storage” type media include any or all of the memory or other storage for the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide storage at any time for the software programming.

All or portions of the software may at times be communicated through a network such as the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer of a search engine operator into the hardware platform (s) of a computing environment or other system implementing a computing environment or similar functionalities in connection with video creation. Thus, another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software. As used herein, unless restricted to tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.

Hence, a machine-readable medium may take many forms, including but not limited to, a tangible storage medium, a carrier wave medium or physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer (s) or the like, which may be used to implement the system or any of its components as shown in the drawings. Volatile storage media include dynamic memory, such as a main memory of such a computer platform. Tangible transmission media include coaxial cables； copper wire and fiber optics, including the wires that form a bus within a computer system. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.

Those skilled in the art will recognize that the present teachings are amenable to a variety of modifications and/or enhancements. For example, although the implementation of various components described above may be embodied in a hardware device, it may also be implemented as a software only solution-e.g., an installation on an existing server. In addition, the video creation as disclosed herein may be implemented as a firmware, firmware/software combination, firmware/hardware combination, or a hardware/firmware/software combination.

While the foregoing has described what are considered to constitute the present teachings and/or other examples, it is understood that various modifications may be made thereto and that the subject matter disclosed herein may be implemented in various forms and examples, and that the teachings may be applied in numerous applications, only some of which have been described herein. It is intended by the following claims to claim any and all applications, modifications and variations that fall within the true scope of the present teachings.

Claims

A method, implemented on a computing device having at least one processor, storage, and a communication platform capable of connecting to a network for creating a replate video, comprising:

providing an original video to a user, wherein the original video describes an event having a plurality of objects present therein；

receiving an input from the user identifying at least one of the plurality of objects from the original video and a plurality of instances of the identified at least one object from different parts of the original video； and

creating a replate video based on the original video and the input from the user, wherein the plurality of instances of the at least one object are played simultaneously in the replate video.
The method of claim 1, wherein

the at least one object includes a first object and a second object；

the input from the user identifies a first playing speed of the first object in the replate video and a second playing speed of the second object in the replate video； and

the first and second objects in the replate video are synchronized with each other at a time when the first and second objects interact with each other.
The method of claim 2, wherein the first playing speed is the same as a playing speed of the original video, and the second playing speed is slower than the playing speed of the original video.
The method of claim 1, wherein the original video is a replay of a sporting event.
The method of claim 1, further comprising:

providing a user interface with the original video to the user to facilitate the user to identify at least one of:

at least some of the plurality of objects to be played in the replate video；

a mode associated with each of the at least some of the plurality of objects；

a playing speed of each of the at least some of the plurality of objects；

a special effect of each of the at least some of the plurality of objects； and

a duration of the replate video.
The method of claim 5, wherein the number of the plurality of instances of the at least one object is determined based on the duration of the replate video.
The method of claim 5, wherein the mode includes at least one of:

a still mode in which the corresponding object is kept still in the replate video；

a motion mode in which the corresponding object is played in the replate video；

a trail mode in which a trail of the corresponding object is played in the replate video； and

an inpaint/black mode in which the corresponding object is removed from the replate video.
The method of claim 5, wherein the special effect includes at least one of:

markers；

path lines；

fade-in and/or fade-out；

opacity；

z-order； and

z-subject.
The method of claim 1, wherein

the input from the user identifies a plurality of regions in the replate video, each of which is associated with a sound； and

when one of the plurality of regions in the replate video is activated, the corresponding sound is played.
The method of claim 1, further comprising:

automatically playing the replate video in a continuous loop.
A system for creating a replate video, comprising:

an original video module configured to provide an original video to a user, wherein the original video describes an event having a plurality of objects present therein；

a user configuration module configured to receive an input from the user identifying at least one of the plurality of objects from the original video and a plurality of instances of the identified at least one object from different parts of the original video； and

a replate video module configured to create a replate video based on the original video and the input from the user, wherein the plurality of instances of the at least one object are played simultaneously in the replate video.
The system of claim 11, wherein

the at least one object includes a first object and a second object；

the input from the user identifies a first playing speed of the first object in the replate video and a second playing speed of the second object in the replate video； and

the first and second objects in the replate video are synchronized with each other at a time when the first and second objects interact with each other.
The system of claim 12, wherein the first playing speed is the same as a playing speed of the original video, and the second playing speed is slower than the playing speed of the original video.
The system of claim 11, wherein the original video is a replay of a sporting event.
The system of claim 11, wherein the user configuration module is further configured to provide a user interface with the original video to the user to facilitate the user to identify at least one of:

at least some of the plurality of objects to be played in the replate video；

a mode associated with each of the at least some of the plurality of objects；

a playing speed of each of the at least some of the plurality of objects；

a special effect of each of the at least some of the plurality of objects； and

a duration of the replate video.
The system of claim 15, wherein the number of the plurality of instances of the at least one object is determined based on the duration of the replate video.
The system of claim 15, wherein the mode includes at least one of:

a still mode in which the corresponding object is kept still in the replate video；

a motion mode in which the corresponding object is played in the replate video；

a trail mode in which a trail of the corresponding object is played in the replate video； and

an inpaint/black mode in which the corresponding object is removed from the replate video.
The system of claim 15, wherein the special effect includes at least one of:

markers；

path lines；

fade-in and/or fade-out；

opacity；

z-order； and

z-subject.
The system of claim 11, wherein

the input from the user identifies a plurality of regions in the replate video, each of which is associated with a sound； and

when one of the plurality of regions in the replate video is activated, the corresponding sound is played.
The system of claim 11, wherein the replate video module is further configured to automatically play the replate video in a continuous loop.
A non-transitory machine readable medium having information recorded thereon for creating a replate video, wherein the information, when read by a machine, causes the machine to perform the steps of:

providing an original video to a user, wherein the original video describes an event having a plurality of objects present therein；

receiving an input from the user identifying at least one of the plurality of objects from the original video and a plurality of instances of the identified at least one object from different parts of the original video； and

creating a replate video based on the original video and the input from the user, wherein the plurality of instances of the at least one object are played simultaneously in the replate video.
The medium of claim 21, wherein

the at least one object includes a first object and a second object；

the input from the user identifies a first playing speed of the first object in the replate video and a second playing speed of the second object in the replate video； and

the first and second objects in the replate video are synchronized with each other at a time when the first and second objects interact with each other.
The medium of claim 22, wherein the first playing speed is the same as a playing speed of the original video, and the second playing speed is slower than the playing speed of the original video.
The medium of claim 21, wherein the original video is a replay of a sporting event.
The medium of claim 21, further comprising:

providing a user interface with the original video to the user to facilitate the user to identify at least one of:

at least some of the plurality of objects to be played in the replate video；

a mode associated with each of the at least some of the plurality of objects；

a playing speed of each of the at least some of the plurality of objects；

a special effect of each of the at least some of the plurality of objects； and

a duration of the replate video.
The medium of claim 25, wherein the number of the plurality of instances of the at least one object is determined based on the duration of the replate video.
The medium of claim 25, wherein the mode includes at least one of:

a still mode in which the corresponding object is kept still in the replate video；

a motion mode in which the corresponding object is played in the replate video；

a trail mode in which a trail of the corresponding object is played in the replate video； and

an inpaint/black mode in which the corresponding object is removed from the replate video.
The medium of claim 25, wherein the special effect includes at least one of:

markers；

path lines；

fade-in and/or fade-out；

opacity；

z-order； and

z-subject.
The medium of claim 21, wherein

the input from the user identifies a plurality of regions in the replate video, each of which is associated with a sound； and

when one of the plurality of regions in the replate video is activated, the corresponding sound is played.
The medium of claim 21, further comprising:

automatically playing the replate video in a continuous loop.