JP2002518723A - Creating animation from video - Google Patents

Abstract

(57) Abstract: An apparatus and method for creating and storing animation and linking animation to video. The sequence of video images (10) is examined to identify a first variant of the scene depicted in the sequence of video images. The first image and the second image are obtained from a sequence of video images, wherein the first image represents the scene before the first deformation and the second image represents the scene after the first deformation. For the storage of animation (12), a set of keyframes created from the video is stored in an animation object (30). One or more values (33, 35) indicating a first sequence of keyframes selected from the set of keyframes, along with information for interpolating between the keyframes of the first sequence, are included in the animation object (30). ) Is stored. One or more values indicating a second sequence of keyframes selected from the set of keyframes are also stored in the animation object along with information for interpolating between keyframes of the second sequence. The number of key frames in the second sequence is less than the number of key frames in the first sequence. As for the connection between the video and the animation, a data structure including elements corresponding to respective frames of the first video is generated. Information indicating the animation created from the second video is stored in one or more elements of the data structure.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001] The present invention relates to the field of image animation. More particularly, it relates to automatically creating animations from videos.

BACKGROUND OF THE INVENTION The Internet has become an increasingly popular medium for delivering complete motion video to end users. However, due to limited bandwidth, most users cannot download and watch high quality video on demand. For example, 640 x 480 at 30 frames per second
In order to deliver a video having a resolution compressed to pixels, the image data is converted to about 8 Mbs (
Megabits per second), the required bandwidth is about 300 times the modem speed of 28.8 Kbs (kilobits per second) available to most Internet users today. Industrial standard compression techniques (eg, MPEG-Mobi
Even with the use of the ng Picture Expert Group, video effects on the Internet are often like today's lower quality slide shows than experienced on television.

[0003] Animations that use keyframes and interpolation to create video effects potentially require much less bandwidth to transmit the video. As personal computer performance has improved, television quality video effects can be synthesized in real time from relatively few key frames that can be received using a low bandwidth modem. Animation sequences that require the transmission of key frames every few seconds can be delivered with enormous bandwidth savings compared to video, and still provide very good image quality.

In addition to requiring less bandwidth, animation is more scalable than video in both playback quality and frame rate. As video effects are synthesized on the fly during playback, frame rate and image quality can be dynamically determined based on several factors, such as playback processor speed, network bandwidth, and user preferences. Can be adjusted.

[0005] Adding features for user interaction and other types of editing is also much easier with animation than with video. For example, to adjust the panning path of the camera or the speed of the subject's movement, it is only necessary to change the motion parameters associated with a few keyframes of the animation. Editing a video clip to achieve the same effect requires hundreds of frame changes. Similarly, installing a hot spot that tracks a moving subject over time can be achieved much more easily with animation than with video.

[0006] Animation has drawbacks. Since skilled human animators have traditionally been required to create high quality animations, the animation process is often expensive and expensive. Furthermore, since human animators often sketch keyframes by hand, animations tend to look like cartoons, often with the true-to-life images needed to portray real world scenes. Missing. In some cases, animations are created with primitive two-dimensional and three-dimensional objects, such as building blocks. This type of animation also tends to have an artificial rather than a natural appearance, and is usually limited to displaying graphic information.

SUMMARY OF THE INVENTION A method and apparatus for creating an animation is disclosed. The sequence of video images is examined to identify a first variation of the scene depicted in the sequence of video images. The first image and the second image are obtained from a sequence of video images. First
The image represents the scene before the first transformation, and the second image represents the scene after the first transformation. First
Information is generated that indicates the deformation and that can be used to interpolate between the first and second images to create a video effect similar to displaying a sequence of video images.

[0008] A method and apparatus for storing animation is also disclosed. A set of keyframes created from the video are stored in the animation object. 1 indicating the first sequence of keyframes selected from the set of keyframes
The one or more values are stored in the animation object along with information for interpolating between the first sequence of keyframes. The one or more values indicating a second sequence of keyframes selected from the set of keyframes are used to interpolate between the first sequence of keyframes storing the set of keyframes created from the video in the animation object. Is stored in the animation object together with the information of The number of key frames in the second sequence is less than the number of key frames in the first sequence.

A method and apparatus for linking video and animation are also disclosed. A data structure is generated that includes an element corresponding to each frame of the first video, and information indicating an image of the animation created from the second video is stored in one or more elements of the data structure.

[0010] Other features and advantages of the invention will be apparent from the accompanying drawings, and from the detailed description that follows.

The present invention is illustrated by way of example, and there is no limitation on the figures of the accompanying drawings that show similar elements as well as references.

DETAILED DESCRIPTION According to embodiments described herein, a video is analyzed to automatically create animations that include keyframes and information for interpolating between keyframes. The keyframes and interpolation information can be used to synthesize the image immediately (on the fly) during animation playback. When displayed, the composited image creates a video effect that approximates the original video. Because video effects, such as image motion and color changes, can be displayed with much less information in animation than normal video, animation has very low bandwidth when transmitted over communications networks such as the Internet. Just spend the breadth. For example, using the methods and apparatus described herein to create an animation that includes information for interpolating a video containing hundreds of image frames between a very small number of keyframes and keyframes. It is possible to use. When receiving an animation on a playback system, such as a desktop computer having animation playback capabilities, the playback system will provide keyframes and interpolation information provided with the animation to synthesize and display the image as the animation is received. Can be used. Animations are more compact than video and automatically create animations based on video because animations can be received and displayed simultaneously, even by playback systems that do not have the bandwidth to receive and display video at the same time. This is the intended advantage of the embodiments disclosed in the present invention.
Further, it is a further intended advantage of the embodiments disclosed herein that the animation and the video be crosslinked so that the user can switch between watching the animation and watching the video during playback. Providing a server system that provides animations with selectable temporal and spatial resolutions, selects animations with temporal and spatial resolutions appropriate for the characteristics of the playback system, and distributes the animations to the playback system It is a further intended advantage of the embodiments disclosed herein.

[0013] These and other intended advantages are described below.

Terminology As used herein, the term “video” refers to an image sequence acquired by a camera at a predetermined rate or generated by an image generator for playback at a predetermined rate. Each picture in the sequence of pictures is contained in a frame of video, and the real-world subject represented by the picture is called a scene. The video data is stored such that for each frame there is data representing an image of the frame. This data is in a compressed form or an uncompressed bit map. While any acquisition rate can be used in theory, the normal acquisition rate is fast enough (eg, 10 frames per second or more) to acquire human perceptible motion in the scene.

Film, NTSC Video (National Television St)
video can be provided from, including, but not limited to, standard code, or any other relay or recorded video format. Video can be displayed on a cathode ray tube display (CRT), liquid crystal display,
It is possible to display on several different types of displays, including plasma displays.

As used herein, the term “animation” refers to a data structure that includes information for interpolating between keyframes and keyframes. A keyframe is an image that describes, or can be used to describe, incremental deformations of a scene. In one embodiment, a new keyframe is added to each incremental transf
and the criteria that determine what constitutes an incremental deformation can be adjusted according to system requirements and user preferences. As the criteria become more sensitive (ie, as the deformation of the scene decreases), more keyframes will be present in the animation.

According to one embodiment, there are two types of key frames, background frames and object frames, in the animation. Background frames are key frames that result from background motion or color changes. Background motion is usually caused by changes in the arrangement of cameras used to record the scene. Typical camera placement changes include, but are not limited to, camera translation, rotation, panning, tilting, or zooming. Color changes are often due to changes in scene lighting (sometimes due to changes in camera placement, such as changes in aperture), but can also be caused by color changes in a wide area of the scene. There is also.

An object frame is a key frame that results from a movement or color change of an object in the scene, and not from a change in the arrangement of cameras used to record the scene. Objects in the scene that change motion or change color independently of camera motion are referred to herein as dynamic objects. It will be appreciated that whether a given object is a dynamic object or a background part of the scene will depend in part on how large the object is relative to the rest of the scene. When the object is large enough (eg, physically or optically close to the camera), the dynamic object effectively becomes the scene background.

According to the embodiments disclosed herein, the sequence of background frames and information for interpolating between background frames is stored in a data structure called a background track. Similarly, information for interpolating between sequences of object frames and object frames is stored in a data structure called an object track. Animations created with the methods and apparatus disclosed herein include at least one background track and zero or more object tracks. Background tracks and object tracks are stored in data structures called animation objects. Animations can be specified in an animation object for storage in memory or in an animation data stream for sequential transmission between communication network or device subsystems.

Creation and Distribution of Animation FIG. 1 shows creation of animation 14 and animation 1 to playback system 18.
4 is shown. The animation is created by an animation production system 12 using a video source 10. Animation 1 after or during creation
4 is converted to an animation data stream 15 and distributed to a playback system 18 via a communication network 20. Alternatively, animation 14 is distributed to playback system 18 on a distributable storage medium 21 readable by subsystems of playback system 18 for displaying the animation. Distributable storage media are, for example, magnetic tapes, magnetic disks, compact disks, read-only storage (CD ROM), digital video diskettes (D
VD) and the like, but are not limited thereto. The playback system 18 obtains a device (eg, a DVD or cassette player) or an animation 14 (eg, via a communication network or a distributable medium) designed specifically for playing the animation, and displays the animation 14 to display the animation 14. It can be a general purpose computer system programmed to execute playback software. For example, a web browsing application
The program may be executed on any number of different types of computers (eg, Apple Macintosh computers, IBM compatible personal computers, workstations, etc.) that implement the animation playback system. The program code for playing animation 14 is:
When the web browsing application determines that the animation data stream 15 is to be received, the web browsing application program itself or an extension of the web browsing application provided in the operating memory of the computer. Can be included.

As indicated by the dotted arrow 19 and the dotted transmission path 17, the server system 1
6 can be used to control animation distribution to a playback system on the network 20. For example, the server system 16 may prioritize animation download requests from playback systems belonging to a class of subscribers, or provide access to a menu of available animations based on service configuration or other criteria. Can be used to limit. As a more specific example, the World Wide Web site used to provide instructional animations (eg, tiles, decorate doors, mount fans on ceilings, etc.) to improve projects at home (ie, Server computer). Site providers may wish to have at least one animation at their disposal that allows interested visitors to know about the usefulness of the service. Other animations
It is downloadable on a pay-per-view basis. The site provider can also sell subscription rights to the site so that subscribers have free access to download all animations in return for periodic payments. The server system 16 can be used to distinguish download requests from these different classes of requesters and respond accordingly.

Another use of server system 16 is to provide animation 14 to playback system 18 in one of a variety of different animation formats. The particular format used can be determined based on the transmission network bandwidth and the capabilities of the playback system. For example, a given playback system 18 may use a particular format or language in which the animation 14 can be understood by the playback system 18 (eg, Java, Dynamic Hypertext Markup Language (D-HTML), Virtual
It may require that it be described in Reality Modeling Langedge (VRML), Macromedia Flash format, etc. Also, the background and object frames of the animation 14 are stored in the normal playback system 18.
In order to avoid exceeding the bandwidth of the transmission network, which is limited by the current download rate (eg, modem speed), it must be sent at a specific spatial and temporal resolution. In some embodiments, the animation 14 is stored in a language and bandwidth independent format to accommodate many possible changes in animation language and network bandwidth. The server system 16 can then be used to dynamically create the animation data stream, depending on the format and bandwidth requirements of the playback system 18. The operation of the server system will be described in more detail below.

Still referring to FIG. 1, the playback system 18 includes a storage medium (eg, DVD, CDROM) that is accessible from a communication network 20 or locally.
An animation data stream can be obtained by reading an animation object stored on a storage device (e.g., cassette tape, etc.).
In one embodiment, playback system 18 is a time-based controller that includes play, pause, fast forward, rewind, and step functions. In another embodiment, the playback system 18 can switch between an animation playback mode and a video playback mode to display an animation or video. The playback system 18 provides a bi-directional, non-time based playback mode so that the user can click on hot spots in the animation, pan and zoom within the frames of the animation, and download still frames of the animation. Can also be included. Additional embodiments of the playback system are described below.

Animation Production System FIG. 2 is a block diagram of an animation production system 12 according to one embodiment. The animation production system 12 includes a background track generator 25, an object track generator 27, and an animation object generator 29. The video source 10 is first received at a background track generator 25 that analyzes a sequence of frames at the video source 10 to generate a background track 33. The background track 33 contains a sequence of background frames and information that can be used to interpolate between the background frames. In one embodiment, the background track generator 25 outputs the background track 33 to the symmetric object track generator 27 and outputs it to the animation object generator 29 after the background track is completed. In another embodiment, the background track generator 25 outputs the background track 33 to the symmetric object track generator 27 and to the animation object generator 29 each time a new background frame in the background track 33 is completed. .

As can be seen from FIG. 2, the object track generator 27 receives the background track 33 from the background track generator 25 and the video source 10. object·
The track generator 27 generates zero or more object tracks 35 based on the background track 33 and the video source 10,
5 to the animation object generator 29. Each object track 35 contains a sequence of object frames and transmission information that can be used to interpolate between object frames.

The animation object generator 29 receives the background track 33 from the background track generator 25 and zero or more object tracks 35 from the object track generator 27, and receives the animation object 30.
Write track to As described below, the animation object 30 can be formatted to include multiple temporal and spatial resolutions of the background track and the object track.

FIG. 3 is a block diagram of the background track generator 25 according to one embodiment. The background track generator 25 includes a scene change estimator 41, a background frame constructor 43, a background motion estimator 45, and a background mixture estimator 47.

The scene change estimator 41 compares successive frames of the video source 10 with each other to determine when the scene deformation of a video frame exceeds a threshold.
When applied to the entire video source 10, the effect of the scene change estimator 41 is to divide the sequence of frames at the video source 10 into one or more sub-sequences of video frames (ie, video segments). Yes, each sub-sequence indicates a scene deformation below a predetermined threshold. Each video segment can be processed by a background motion estimator 45, and a background frame constructor 43 and a background blend estimator 47 generate scene change estimators to generate background frames and interpolation information for the video segments. Process each video segment identified by device 41. Thus, the predetermined threshold applied by the scene change estimator 41 defines the incremental deformation of the scene that results in the construction of a new background frame. In one embodiment, the background frame roughly corresponds to the start and end of each video segment, and the background frame corresponding to the end of one video segment corresponds to the start of the next video segment. Thus, each video segment is described by a background frame, and, except for the first video segment (where the start and end background frames are constructed), certain background frames are constructed for each video segment of video source 10. Have been.

FIG. 4A shows a video segment 54 identified by the scene change estimator 41 of FIG. According to one embodiment, the scene change estimator 41 includes a video
It operates by determining a deformation vector for each pair of adjacent video frames in segment 54. In this specification, a first frame is considered to be adjacent to a second frame if the first frame immediately follows or immediately follows the second frame in the time sequence of the frames.

The deformation vector for each pair of adjacent video frames is represented by a delta (
That is, it is shown in FIG. 4A by the symbol “Δ”. According to one embodiment, the deformation vector includes a plurality of scalar elements in the video segment 54, each indicating a degree of change of the scene from one video frame to the next. For example, the scalar elements of the deformation vector are the following changes in the scene: translation, scaling, rotation,
Includes pan, tilt, skew, color change, and degree of elapsed time.

According to one embodiment, the scene change estimator 41 calculates a spatial delta between video frames 54 in order to increase the shading of the images in the video segments 54 before calculating the deformation delta between adjacent frames. Add a low pass filter. After filtering with a low pass filter, the individual images of video segment 54 contain less information than before filtering because fewer calculations are required to determine the deformation delta. In one embodiment, the modified delta calculated for each adjacent frame pair of video segment 54 is added to the modified delta calculated for the previous adjacent frame pair to accumulate the sum of the modified deltas. In effect, the sum of the deformation deltas represents the deformation between the first video frame 54A in video segment 54 and the most recently compared video frame in video segment 54. In some embodiments, the sum of the deformation deltas is compared against a predetermined deformation threshold to determine whether the most recently compared video frame has exceeded the deformation threshold. It should be understood that the deformation threshold is a vector quantity that includes multiple scalar thresholds, including thresholds for scene color change, translation, scaling, rotation, panning, tilt, skew, and elapsed time. In an alternative embodiment, video source 1
Dynamically adjust the deformation threshold to achieve the desired video segment to frame ratio at zero. In another alternative embodiment, the desired average video segment size (ie, the desired number of video frames per video segment)
, Dynamically adjust the deformation threshold. In yet another alternative embodiment,
The deformation threshold is dynamically adjusted to achieve the desired average elapsed time per video segment. In general, any technique for dynamically adjusting the deformation threshold may be used without departing from the spirit and scope of the present invention.

In one embodiment, if the most recently compared video frame 54C exceeds a threshold, the scene is considered to have changed and the video segment preceding the most recently compared video frame 54C 54B is indicated to be the end frame of the video frame 54. Thus, if a predetermined deformation threshold is used, it is ensured that each video segment of video source 10 has an entire deformation that is less than the deformation threshold. If a variable deformation threshold is used, each video
Substantial changes occur throughout the deformation delta of the segment, and it is necessary to add a repetitive scene change estimator to reduce the change in the deformation delta.

According to the embodiment shown in FIG. 3, the background track generator 25 uses the background motion estimator 45, each time a new video segment is determined (ie, each time a new scene change is detected). The background frame constructor 43 and the background mixing estimator 47 are called. In an alternative embodiment, the scene change estimator 4
1 is used to completely resolve the video into sub-sequences before any sub-sequences are processed by the background frame constructor 43, the background motion estimator 45, and the background blend estimator 47.

As shown in FIG. 4A and described above, the video source within a given video segment
Frames are subsequently selected and compared until the deformation delta accumulation exceeds the deformation threshold. In some embodiments, when the last frame of the video is reached, the last frame is automatically considered to be the end of the video segment. Also, after each new video segment has been processed by the background frame constructor 43, the sum of the deformation deltas is erased. In an embodiment where the entire video is analyzed by the scene change estimator 41 before every video segment is processed, the deformation delta associated with each video segment is determined by the background motion estimator 45.
And recorded by the background frame constructor 43 for later use.

FIG. 4B is a flowchart 57 illustrating the operation of the background motion estimator 45, the background frame constructor 43, and the background mixture estimator 47 shown in FIG. Beginning at block 59, the background motion estimator extracts the video segment 54 indicated by the scene change estimator (ie, a sub-sequence of video frame 54 constrained by BF _i and BF _{i + 1 in} FIG. 4A). Inspect to identify the main motion of the scene shown in those frames. This primary movement is considered a background movement.

There are several techniques that can be used to identify background motion in a video segment. One technique, called feature tracking, involves identifying features of a video frame (eg, using edge detection techniques) and tracking the movement of features from one video frame to the next. Features exhibiting motion that is statistically abnormal with respect to other features are considered dynamic objects and are temporarily ignored. Motion shared by a large number of features (or large features) usually results from changes in the arrangement of cameras used to record video and is considered background motion.

Another technique for identifying background motion in a video segment correlates the frames of the video segment with each other based on a common region, and then determines the frame-to-frame displacement of those regions. The frame-to-frame shift can then be used to determine the background motion of the video segment.

Still another technique to consider the background motion of a video segment is a coarse-textured survey method that uses a spatially gradual decomposition of the frames in the video segment, and an estimated primary fiber. Robust estimation techniques, such as, but not limited to, M estimation, which remove elements of the video frame that do not follow the motion.
Gradual decomposition measures changes in the characteristics of a video frame histogram over time to identify scene changes, to highlight features of video segments that can be used for motion identification Pixel format conversion, which replaces color displays (including grayscale) to achieve higher filtration, optical flow measurement and analysis, faster processing speed, greater reliability, or both.

Still referring to flowchart 57 of FIG. 4B, the background frame constructor receives background motion information from the background motion estimator at block 61 and uses the background motion information to register frames of the video segment relative to each other. . Registration is
Refers to correlating video frames in a manner that accounts for changes caused by background motion. By registering a video frame based on background motion information, regions of the frame exhibiting motion different from the background motion (ie, dynamic objects) appear in a very small number of fixed locations in the registered video frame. Will be. That is, the area moves from frame to frame as compared to the stationary background. This area is a dynamic object. At block 63, the background frame constructor removes dynamic objects from the video segment to create a processed sequence of video frames. Background frame at block 65
The constructor generates a background frame based on the processed sequence of video frames and the background motion information. Depending on the nature of the deformation, the construction of the background frame comprises combining two or more processed video frames into one background image;
Or selecting one of the processed video frames as a background frame. In some embodiments, the composite background frame is a panoramic image or a high-resolution still image. Panoramic images are created by stitching two or more processed video frames together and can be used to display a background scene obtained by panning, tilting, or translating the camera. High resolution still images are used when the subject of the processed sequence of video frames is a relatively static background scene (ie, when the position of the camera used to record the video source has not changed significantly). Is appropriate. One technique for creating high resolution still images is to analyze the processed sequence of video frames to identify sub-pixel motion between frames. Sub-pixel motion is caused by slight camera motion and can be used to create a composite image with a higher resolution than any individual frame obtained by the camera. As described below, high resolution still images are particularly useful because they can be displayed to provide details not present in video source 10. Also, when multiple high-resolution still images of the same subject are constructed, the high-resolution still images can be combined to form a still image having regions of varying resolution. In the present specification, such an image is referred to as a multi-resolution still image. As described below, the user can stop playing the animation to zoom in and out at different regions of such a still image. Similarly, the user can stop playing the animation to pan about the panoramic image. A combination of pan and zoom is also possible. Additionally, animations can be cross-linked with the video source so that during playback of the video source, the user can be prompted to stop the video playback to see a high resolution still image, panoramic still image, or zoomable still image It is possible to Crosslinks are described in more detail below.

FIG. 5 shows a background image set 70 generated by the background frame constructor 43 shown in FIG. The background frame BF _i is a processed video frame,
A background image 71 that is not a composite image. Typically, this type of background image results from scaling (ie, zooming in or out) or sudden cuts between successive video frames. The background frame BF _{i + 1} refers to a high resolution still image 73 synthesized from a plurality of processed video frames of essentially the same scene.
As noted above, this type of image is particularly useful for providing details that are not perceptible at the video source. The background frames BF _{i + 2} , BF _{i + 3} , and BF _{i + 4} each relate to a different area of the panoramic background image 75. As already indicated, the panoramic image frame 75 comprises one or more portions 7 of the processed video frame.
6 by stitching it to another processed video frame.
In this example, the camera may have been translated down and to the right, or panned to the right and tilted down to gain incrementally more of the scene. Other shapes of the composite background image are due to different types of camera movement.

Returning to the last block of flowchart 57 of FIG. 4B, the background blend estimator (eg, element 47 of FIG. 3) performs background blending based on the background motion information and the newly constructed background frame in block 67. Generate information. Regarding the operation of the mixing estimation device,
This will be described in more detail below.

FIG. 6 is a block diagram of the object track generator 27 according to one embodiment. Object track generator 27 receives as input a background track 33 and a video source 10 generated by a background track generator (eg, element 25 of FIG. 2). The object track generator 27 identifies dynamic objects in the scene based on the difference between the background track 33 and the video source 10 and
An object frame (OF) containing a dynamic object is recorded along with the object motion (OM) and object mixture (OB) information of the track 35.

In one embodiment, the object track generator 27 includes an object frame constructor 81, an object motion estimator 83, and an object mixture estimator 85. The object frame constructor 81 compares the video frames of the video source 10 to the background frames of the background track 33 to construct an object frame (OF). As described below, each object frame constructed by the object frame constructor 81 contains a dynamic object. In some embodiments, at least per dynamic object detected in a given video segment (ie, per dynamic object detected in the sequence of video frames identified by scene change estimator 41 of FIG. 3), One object frame is generated. The object motion estimating device 83 is provided with object motion information (OM)
The motion of the dynamic object in the video segment is tracked to generate Object mixture information (OB) is generated based on the object motion information.

FIGS. 7A and 7B illustrate in detail the operation of the object track generator 27 of FIG. FIG. 7A shows a video segment 54 identified by the scene change estimator 41 of FIG. Video segment 54 is constrained by background frames BF _i and BF _{i + 1} and includes dynamic object 56. FIG. 7B is a flowchart 100 of the operation of the object track generator 27.

Beginning at block 101 of flowchart 100, the object frame constructor (eg, element 81 of FIG. 6) converts background frame BF _i to video frame VF of video segment 54 to generate difference frame 91. Compare with _j . As shown in FIG. 7A, a small difference between BF _i and VF _j creates a somewhat random difference (noise) in the difference frame 91. However, a region where the difference 92 between BF _i and VF _j is relatively concentrated occurs when the dynamic object has been removed from the background frame BF _i by the background frame constructor (eg, element 43 of FIG. 3). At block 103 of the flowchart 100, a spatial low pass filter is applied to the difference frame 91 to create a filtered difference frame 93. In the filtered difference frame 93,
The random differences (ie, high frequency elements) have disappeared, and the concentrated area of difference 92 indicates that the shading has increased. As a result, the outline of the concentrated area of the difference 92 can be more easily identified. Thus, at block 105 of the flowchart 100, the object frame constructor determines the difference 9 in the filtered difference frame 93.
A feature survey (eg, using edge detection techniques) is performed to identify the two focus areas. At block 107, the object frame constructor as an object frame 56, the difference 92 in filtered difference frame 93 to select the area of the video frame VF _j corresponding to the region to focus. In some embodiments,
The object frame constructor selects the object frame 56 as a rectangular area (eg, having the same x, y offset) corresponding to the rectangular area of the filtered difference frame 93 surrounding the concentrated area of the difference 92. Alternative object
It is possible to use the shape of the frame. It should be understood that if there are no concentrated regions of difference in the filtered difference frame 93, no object frame will be selected by the object frame constructor. Conversely, if the filtered difference frame 93 has a plurality of difference concentration regions, a plurality of object frames can be selected. Each concentrated area of the difference of the filtered difference frame 93 is
It is considered to correspond to a dynamic object in a subsequence of frame 54.

The object frame 56 is generated by the object frame constructor.
After the dynamic object has been identified and configured in, the motion of the dynamic object is determined by tracking changes in the position of the object frame 56 passing through the sequence of frames of the video segment 54. Accordingly, at block 109 of the flowchart 100, the object motion estimator (eg, element 83 of FIG. 6) determines the dynamics identified and configured by the object frame constructor from the video frame at video segment 54 to the next frame. Track the movement of an object. According to one embodiment, tracking of object motion is performed by performing a feature survey within each successive video frame of video segment 54 to determine a new position of the dynamic object of interest. . Using the frames that make up the motion of the dynamic object, the motion estimator generates motion information that can be used to interpolate between successive object frames to approximate the motion of the dynamic object. At block 111 of flowchart 100,
An object mixture estimator (eg, element 85 of FIG. 6) generates object mixture information based on the object motion information and the object frame.
In one embodiment, the operation of the object mixture estimator is the same as the operation of the background mixture estimator. However, alternative techniques for generating information that mixes successive object frames can be used without departing from the spirit and scope of the present invention.

As described above, in one embodiment of the object track generator 27 of FIG. 3, at least one object frame is identified by each dynamic object identified by the object frame constructor 81 in the video segment. Is generated for If the object motion estimator 83 determines that the motion of the dynamic object in the video segment is so complex that it cannot be properly indicated by interpolating between the constraining object frames. For example, the object motion estimator 83 indicates the need to construct one or more additional object frames for the video segment. The object frame constructor will then use the techniques described above to generate additional object frames at the junctions in the video segment indicated by the object motion estimator. As described above in connection with background frame construction, an object frame can include image data derived from a composite image area. If one or more additional object frames are constructed to indicate the dynamic object undergoing complex movements, the dynamic object may be overlaid with other features of the scene during playback of the animation. It is possible to organize additional frames with animated objects.

Dynamic objects may occasionally overlap each other in a scene. According to one embodiment of the object track generator 27, when the dynamic objects represented by the separate object tracks are occluded overlapping each other, the occluded dynamics will occur if the occluded object reappears. Object of object
The track ends and a new object track is created. Therefore,
If the dynamic objects repeatedly shield each other, many discrete objects
Tracks can be created. In an alternative embodiment of the object track generator, if the screen location of two dynamic objects is concentrated, the information can be associated with an object track that indicates which one to display on top of the other.

As in the case of a background image, an image of a dynamic object (ie, an object image) can be composited from multiple video frames. Composite object images include, but are not limited to, panoramic object images, high resolution still object images, and multi-resolution still object images. In general, any composite of images that can be used to create a composite background image can also be used to create a composite object image.

FIG. 8 is a diagram of an animation object 30 according to one embodiment.
The animation object 30 includes a background track 33 and a plurality of object tracks 35A, 35B, 35C. As described above, the number of object tracks depends on the number of dynamic objects identified in the scene shown in the video source; There is no.

In one embodiment, the animation object 30 includes a background track and
And a linked list 121 of object tracks. Height
The scene track itself includes a background track element BT and a background frame BF.₁From BF_Nof
Implemented by a linked list of sequences. Each object track
Is also the object track element OT₁, OT_Two, OT_R, And obgi
Each of the sequences of the target frame (OF1₁From OF1_M, OF2₁From
OF2_K, OFR₁OFR to_J) By a linked list. Ah
In one embodiment, the background track element BT and the object track element OT ₁ , OT_2,OT_RAlso an animation / object linked list 121
And a pointer that implements That is, the background track element BT is the first object.
OT track element₁The first object trajectory.
Lock element OT₁Is the next object track element OT_TwoContains a pointer to
Object track OT_RContinue until you reach. An implementation
In the state, the animation / object linked list 121 and the individual
The end of the linked list of scenes and object tracks is
Each is indicated by a null pointer. Other to indicate the end of the linked list
The technique can be used in alternative embodiments. For example, animation
The object 30 is a list 1 to which the animation object is linked.
At 21, the first pointer pointing to the background track 33, and the last object
Includes a data structure that includes a tail pointer pointing to track 35C. Similarly, the background
Track element BT and each object track element OT₁, OT_Two, OT_RIs
, Respectively, to indicate the end of each linked list.
In yet another embodiment, a flag in an element of a linked list is displayed at the end of the list.
Can be used to indicate completion.

With continued reference to FIG. 8, data structure 123 is used to implement a background frame according to an embodiment. A member of the background frame data structure 123 includes a next pointer (NEXT P) pointing to the next background frame in the background track 33.
TR) and a preceding pointer (PR) indicating the preceding background frame on the background track 33.
EV PTR), an image pointer (IMAGE PTR) indicating the position of the image data with respect to the background frame, and an interpolation pointer (INT) indicating the interpolation data structure.
ERP PTR) and a timestamp (relative to the background frame)
TIMESTAMP). As described below, the background frame data structure 123 may further include one or more members for crosslinks having frames of the video source.

Recalling that the image to be displayed for a given background frame can be obtained from a non-composite or composite background image, the background frame data structure 1
The image pointer itself at 23 has a data structure that occupies the position of the background image in memory, a shift in the background image from which to obtain the image data for the background frame (eg, rows and columns), and to generate the background frame. May be a pointer to the video segment to use. As described below, pointers to video segments are used to link animation and video sources. In some embodiments, the pointer to the video segment is at least a pointer to the first video frame of the video segment. Other techniques for linking background frames to video segments can be used without departing from the spirit and scope of the present invention.

In one embodiment, the background interpolation data structure 125 includes data for interpolating between a given background frame and its adjacent background frame. The information for interpolating between a background frame and a background frame adjacent thereto (ie, the next background frame) includes forward background motion information (BM FORWARD) and forward background mixed information (BB FORWARD). Similarly, information for interpolating between a background frame and a preceding background frame adjacent to the background frame is backward background motion information (BM R).
EVERSE) and rear background mixed information (BB REVERSE). The background motion information itself in a given direction (ie, forward or backward) can be a data structure that includes several members. In the exemplary embodiment shown in FIG. 8, the forward background motion information (BM FORWARD) is X and Y reaching the next background frame.
Members indicating the translation of the background scene in the directions (i.e., horizontal and vertical in the image plane); scale factors, rotation factors, panning in the X and Y directions (i.e., changes in camera zoom in / out and aspect ratio). Factor, slope factor,
And skew factors. It should be appreciated that alternative embodiments use more or fewer parameters. Back background motion information (BM RE
VERSE) can be indicated by a similar set of motion parameters.

In one embodiment, each individual object frame is implemented by an object frame data structure 127 similar to the background frame data structure 123 described above. For example, the object frame data structure 127 contains a pointer (N) to the next object frame in the object track.
EXT PTP), a pointer to the preceding object frame in the object track (PREV PTR), and an image pointer (IMAGE PTP).
TR), an interpolation pointer (INTERPTR), and a time stamp (TIM).
ESTAMP), each performing a function similar to the function of the same member in the background track data structure 123. Of course, object frames
The image pointer of the data structure 127 indicates object image data instead of the background image data, and the interpolation pointer indicates object interpolation data instead of the background interpolation data. As shown in FIG. 8, an exemplary object interpolation data structure includes forward and backward object motions (OM FORWARD and OM R respectively).
EVERSE) and forward and backward object mixture information (OB respectively)
FORWARD and OB REVERSE).

FIG. 9A illustrates an exemplary embodiment of a background frame mixing data structure 135A, 137A that can be used to implement background mixing. It should be understood that the object mixture data can be similarly organized. In one embodiment, each mixed data structure 135A, 137A includes a mixed operator in the form of a polynomial (A, B, C, D) coefficient and a frame to which the mixed operator is applied over two consecutive background frames. And a pointer to the next mixed data structure that allows successive background frames to be displayed by multiple mixing operators.

[0057] In Figure 9A, the forward background mixed data 135A for the background frame BF _i, and the rear background mixed data 137A for the background frame BF _{i + 1,} for mixing the background frame BF _i and BF _{i + 1} 139 showing a method of applying the mixed data to the data
Shown together. The mixing operation shown in the graph is known as a cross-dissolve operation because during the mixing interval (ie, the time between background frames), the background frame BF _i is effectively dissolved into the background frame BF _{i + 1.} ing. To generate the interpolated frame at time t _INT, background frame BF _i is deformed in a forward direction based on the frontward background motion information for the frame BF _i, background frame BF i _{+ 1} is a frame BF i _{+ 1} Is deformed in the backward direction based on the backward background motion information for. The respective weights (ie, multipliers) are calculated for frames BF _i and BF _{i + 1} using the mixing information of those frames. Weight frame BF _i is based on the forward mixing information for background frame BF _i, the weight of the background frame BF i _{+ 1} is based on the rear mixing information for a frame BF i _{+ 1.} Then the weight of the frame BF _i and a frame BF i _{+ 1,} respectively added to the modified version of the background frame BF _i and BF i _{+ 1,} the variations and weighted image obtained as a result to produce an interpolated frame (E.g., using pixel x pixel addition).

As described above, in one embodiment, the mixing operator is implemented by storing the coefficients of the polynomial and the portion of the mixing interval to which the polynomial is applied. For example, the forward mixed data 135A for the frame BF _i is obtained by calculating the coefficient A of the mixed data 135A,
The mixing operator denoted by B, C, D is the total mixing interval (in this case, tBF _i and t
1 interval fraction (indicating that it has been applied over BF _{i + 1} intervals)
INTV = 1). In general, fractional fractions less than one are used when the entire mixture function contains discontinuities that cannot be adequately represented by a polynomial of limited order. However, the mixing operation shown in the graph 139 shows a continuous primary mixing operation. Therefore, when the coefficients A, B, C, and D specified in the mixed data structure 135A are applied to the polynomial weight (T) = AT ³ + BT ² + CT + D, the weight BF _i (T) = 1−T. According to one embodiment, the value of T is
The mixing operators A = 0, B = 0, C = -1, D = 1 are normalized to the range of 0 to 1 over the relevant mixing interval fraction so that the multipliers result in multipliers that decrease linearly with time over the mixing interval. Be transformed into The multiplier for BF _i starts at 1 and decreases linearly to 0 at the end of the mixing interval. Referring to the mixing operator for frame BF _{i + 1, the} coefficients A = 0, B = 0, C = 1, specified in the mixing data structure 137A
Applying D = 0 results in a weight BF _{i + 1} (T) = T. Therefore, frame B
The F _{i + 1} multiplier starts at 0 and increases linearly to 1 during the mixing interval.

FIG. 9B shows the discontinuous mixing function 141. In this case, the background frame BF_iWhen
BF_{i + 1}The mixing intervals of the three interval fractions 146, 147, and 148
Is divided into During the first fraction 146 of the mixing interval, the background frame BF_iTo
The applied weight is kept constant at 1 and the background frame BF_{i + 1}Weight added to
The volume is kept constant at zero. During the second fraction 147 of the mixing interval,
A loss dissolve occurs and during the third fraction 148 of the mixing interval, frame B
F_iAnd BF_{i + 1}Is again kept constant, but the first fraction 1 of the mixing interval
It takes the opposite value to the value of 46. In one embodiment, the discontinuous mixing function 141 is
Indicated by a linked list of combined data structures 135B, 135C, 135D
And each mixed data structure in the list has its own INTV parameter
2 shows the fraction of the mixing interval applied in. Therefore, the background frame
BF_iThe first forward mixed data structure 135B for the
= 0.25, and the multiplier chunk is the first 25% of the mixing interval (ie, the interval
146) Frame BF_iApplied to a transformed version of
Show mixed operator weight BF_i(T) = 1. Background frame BF_iSecond blend against
The combined data structure 135C contains the interval fraction INTV = 0.5, and the mixed interval
Of the frame BF during the middle 50% of_iWeight added to
Operator weight BF indicating that decreases linearly from 1 to 0_i(T) = 1-T
including. For ease of explanation, the value of T is between 0 and 1 during each interval fraction.
Note that it is assumed that they are normalized in the range Other indications are
Of course, it is possible and considered within the scope of the invention. Background frame BF _i Of the third mixed data structure 135D is that the interval fraction INTV = 0.25,
BF during the last 25% of the mixing interval (ie, interval 148)_iIs supplement
Weighted BF, indicating no contribution to the interposed background frame_i(T) = 0
Contains the resulting mixed operators.

Still referring to FIG. 9B, the linked list of mixed data structures 137B, 137C, 137D for background frame BF _{i + 1} shows the inverse mixing function of the list shown for background frame BF _i . That is, the first 25% of the mixing interval
, A weight of 0 is added to the transformed version of frame BF _{i + 1} (indicating that there is no contribution from frame BF _{i + 1} to the interpolated background frame during that time) and the mixing interval The weight added to the deformed version of frame BF _{i + 1} increases linearly from 0 to 1 during the middle 50% of the last 25 minutes of the mixing interval.
During%, the multiplier chunk (ie, weight = 1) is added to the modified version of frame BF _{i + 1} to create an interpolated background frame.

One reason for applying a discontinuous mixture function of the type shown in FIG.
The purpose is to reduce the distortion associated with the frame. Fraction of mixing interval
By keeping the contribution of a given key frame constant during the _i And BF_{i + 1}Can reduce the distortion caused by the difference between the front and rear deformation
it can. In some embodiments, the input of the operator will make the contribution of a given keyframe constant.
Animation production system to select the fraction of the mixing interval to keep
System (eg, element 12 of FIG. 1). In an alternative embodiment, some images or
Or automatic interval fractions where the contribution of other images must be kept constant
The degree of sharpness of the image (for example, the image gradient) to determine
And unmixed images. Also, linear cross dissolve operation
The work has been described above, but other types of cross-dissolve behavior can be changed to different polynomials.
Therefore, it is also possible to regulate. Also, a polynomial to indicate the type of mixed operation
Instead of using the coefficient, other indices can be used. For example, a line
The value that indicates whether to apply shape, quadratic, transcendental, logarithmic, or other mixed operations
, Can be stored in a mixed data structure. Cross dizo for background mixing
Although described first from the viewpoint of the lube operation, other mixing effects can also be applied to a certain background frame.
Can be used to transition to background frames, fading and various
Including, but not limited to, screen wipes.

[0062] Figure 10 illustrates exemplary animation object 30 of the background track 33 and the object track 35A, 35B is the method used to synthesize the frame IF _t interpolated in the animation playback.

At a given time t, an interpolated frame IF _t is generated based on each pair of adjacent frames in the background track 33 and the object tracks 35A, 35B. Adjacent pairs of background frames BF _i , BF _{i + 1} are respectively transformed and weighted using the background motion information and the background mixing information associated with those background frames. Background frame BF _i is deformed by the forward background motion information associated with the frame BF _i (BM), then weighted by the forward background mixed information associated with the frame BF _i (BB). The effect is to transform the pixels of the background frame BF _i to their respective positions based on forward motion information (eg, translation, rotation, scaling, pan, tilt, or skew), and then pixel values based on the blend operator Is to reduce the intensity level of each pixel value.
The pixels of the background frame BF _{i + 1} are similarly deformed and weighted by the backward motion information and the backward mixed information (BM, BB) for the frame FB _{i + 1} . The resulting deformed image is then interpolated background frame 1 representing the background scene at time t
Combined to create 51A. Object frames OF1 _i and OF1 _{i + 1} are similarly transformed using forward and backward object motion information (OM), respectively, weighted using forward and backward object mixed information (OB), respectively, and then combined. The resulting interpolated object frame is then combined with the interpolated background frame 15 to create an interpolated frame 151B containing the interpolated background and the interpolated dynamic object.
Overlaid on 1A. Object frames OF2 _i and OF2 _{i + 1} are also transformed, weighted, combined using the object motion information and object blending information associated with those object frames (OM, OB), and then overlaid on the interpolated background. Is done. The result is the completed interpolated frame 151C. Consecutive interpolated frames are similarly created using different values of the temporal deformation blend operator and progressive deformation on background and object frames based on motion information. The net effect of animation playback is to create a video effect that approximates the original video used to create the animation object 30. The soundtrack from the original video can also be played along with the animation.

FIGS. 11 and 12 illustrate techniques for providing multiple resolutions of animation in an animation object. FIG. 11 illustrates a technique for providing multiple temporal resolutions of animation keyframes, and FIG. 12 illustrates a technique for providing multiple spatial resolutions of animation keyframes. In one embodiment, the animation object is constructed to provide both types, ie, spatial and temporal multiple playback resolution. This provides the user of the playback system with the option to increase or decrease the resolution of the animation sequence in spatial and / or temporal dimensions. If the playback system has sufficient download bandwidth and processing power, it is possible to select the maximum resolution in time and space to present the highest resolution animation playback. If the playback system does not have sufficient download bandwidth, or the processing power to handle the maximum resolution in space and time, the playback system will be able to provide the space for the animation being played based on the criteria selected by the user. Target or temporal resolution can be reduced automatically. For example, if a user indicates that they want to see the image with the highest spatial resolution (ie, a larger and higher resolution image), it means fewer key frames and more interpolated frames. Again, selecting a keyframe track with fewer keyframes per unit time (ie, a background track or object track) while selecting to display the largest or near maximum spatial resolution keyframes. It is possible. Thus, if a user desires a higher temporal resolution (ie, more keyframes per unit time), the maximum or near maximum temporal resolution keyframes, even if the spatial resolution must be reduced. It is possible to select a track, but each key frame will be displayed with reduced spatial resolution.

Another use for reducing the temporal resolution of an animation is when scanning quickly both forward and backward within an animation.
During animation playback, the user can signal a time multiplier (eg, 2x, 5x, 10x, etc.) requesting to view the animation at a faster rate. In one embodiment, the need for fast scanning is satisfied by using a temporal multiplier, with the playback system's bandwidth ability to select an appropriate temporal resolution of the animation. At very fast playback rates, the spatial resolution of the animation can also be reduced. Similarly, a temporal multiplier can be used to slow down the playback of the animation to a slower rate than the natural rate to achieve a slow motion effect.

FIG. 11 depicts a multi-temporal level background track 161. object·
Tracks can be similarly configured. First Level Background Track 35A
In, the maximum number of background frames (each named "BF") is provided with background motion information and background mixing information for interpolating between successive pairs of background frames. The number of background frames per unit of time is determined by a small amount of the video frame rate from one video frame rate (in which case, the motion and blending information shows no information, simply cuts to the next frame). May vary in the range of the fraction. The second level background track 35B has fewer background frames than the first level background track 35A, and the third level background track 35C has
Continue to the Nth level background track 35D, such as having less background frames than the second level background track. Although the number of background frames in the second level background track 35B is shown in FIG. 11 as being half of the number of level 1 background tracks 35A, other ratios can be used. In the second level background track 35B,
Mixing and motion information (B) for interpolating between successive pairs of background frames
M ₂ , BB ₂ ) is different from the mixed information and motion information (BM ₁ , BB ₁ ) for the first level background track 35A because the transformation from the background frame to the background frame on the different level track is different. Similarly, the third level background track 35C has fewer background frames than the second level background track 35B, and therefore has different motion and blending information (BM ₃ , BB ₃ ) for each frame. As the level of the background track increases, the temporal resolution gradually decreases until the lowest resolution background track is reached at level N.

In one embodiment, the background track levels above the first level 35 A do not actually include a separate sequence of background frames. Instead, a pointer to a background frame in the first level background track 35A is provided. For example, the first background frame 62B of the second level background track 35B can be indicated by a pointer to the first background frame 62A of the first level background track 62A, and the second background frame 63B of the second level background track 35B. Is the first
It can be indicated by a pointer to the third background frame 63A of the level background track 35A. Level 1 background track 35A
Can be combined in the data structure with motion and mixing information indicating a transformation to the next background frame and motion and mixing information indicating a transformation to the preceding background frame. further,
A linked list of such data structures can be used to indicate a sequence of background frames. Other data structures and techniques for indicating the sequence of background frames can be used without departing from the spirit and scope of the present invention.

In an alternative embodiment, each background track level 35A, 35B, 35C, 35D
Is formed by some criteria that selects a background frame from a set (or pool) of background frames. In this embodiment, the reference value used to form a given level of background track effectively regulates the sequence of key frames having a temporal resolution determined by the number of reference values. The reference value used to select the background frame can be a pointer to the background frame, an index that indicates the position of the background frame in the table, or any other value that can be used to identify the background frame. is there.

In some embodiments, the blending and motion information for higher level background tracks in the multi-level background track 161 may be obtained by combining multiple sets of motion and blending information from lower level background tracks. It is possible. For example, in background track level 2, background frame 62B
Background motion information and background mixed information (BM ₂ , BB ₂ ) used for transition between
Creating by combining background motion information and background mixing information used for transition between background frames 62A and 64, and background motion information and background mixing information used for transition between background frames 64 and 63A. Is possible. In an alternative embodiment, the background motion information and background mixing information for the higher level background track may be created based on the background frame of the track without using the mixing information and motion information from the lower level background track. It is possible.

FIG. 12 shows a multi-spatial resolution background frame. Each background frame in the background track of the multi-temporal-resolution background track can include various resolution background frame from BF _1, BF ₂ to BF _N. Background frame BF ₁ is the background frame with the highest spatial resolution and, in one embodiment, includes the same number of pixels as the original video frame. Background frame BF ₂ has a lower spatial resolution than BF ₁ , which means that BF ₂ has fewer pixels (ie, smaller image) than BF ₁ or has a larger block size. . Block size refers to the size of a pixel, usually an elemental unit of visual information used to describe an image. Smaller block sizes result in images with higher spatial resolution because more delicate element units are used to describe the features of the image. block·
Larger sizes result in lower spatial resolution images, but require less overall information because only one pixel value is applied to the group of pixels in the pixel block.

FIG. 13 shows how to use the server system 16 for controlling the contents of the animation data stream distributed to the playback systems 18A, 18B, 18C. According to one embodiment, server system 16 includes various animation objects 14A, 14B,
Receive a request to download 14C. Before receiving the animation object, the server system 16 first queries the playback systems 18A, 18B, 18C to determine the capabilities of the system. For example, animation
In response to a request from playback system 18A to download object 30C,
The server system 16 features one of the playback system features that the server system 16 can use to generate the appropriate animation data stream.
It requests the playback system 18A to provide the set. As shown in FIG. 13, the set of playback system features associated with a given playback system 18A, 18B, 18C includes the playback system download bandwidth or its network access medium, playback system processing power (eg, The number of processors, the speed of the processor, etc.), the graphics display capabilities of the playback system, the software application used by the playback system (e.g., the type of web browser), the operating system running the software application, and the preferences of the user. A set may be included, but is not limited thereto.
User preferences include preferences of favoring spatial resolution and sacrificing temporal resolution, and vice versa. Also, user preferences can be dynamically adjusted by the user of the playback system during the downloading and display of the animation.

In one embodiment, the animation objects 14A, 14B, 14C are stored in multiple temporal and multiple spatial resolution formats, and
System 16 selects a background track and an object track from an animation object (e.g., animation object 30C) having a temporal and spatial resolution that is optimal for the features provided by the target playback system. Thus, as shown in the graph 172, the server system 16 provides different temporal / spatial resolution versions 17 of the same animation object 30C for downloading to the playback systems 18A, 18B, 18C.
4A, 174B, 174C can be selected based on their characteristics. Further, the server system may be provided with a given playback system 18A, 18B, 18C.
Can be dynamically adjusted based on changes in the characteristics of the playback system.

FIG. 13 illustrates the use of a server system to control the content of an animation data stream via a communication network, but dynamically between multiple temporal and spatial resolution animation tracks. Similar techniques can be applied within the playback system to make the selection. For example, the selection logic in the playback system can provide an animation data stream to display the logic in the playback system having a temporal / spatial resolution appropriate for the features of the playback system. For example, a DVD player may adjust the temporal or spatial resolution of animation playback to one or more other videos or animations (
It can be designed to reduce based on whether or not it is being displayed (e.g. in other areas of the display).

As described above, associating key frames of an animation with video frames of a video source so that a user can watch the animation and switch video sources during playback is an embodiment of the present invention. Is the intended advantage. This association of keyframes and video frames is called "crosslinking" and if one display of animation or video offers advantages over another,
Particularly useful. For example, in one embodiment of the animation playback system described below, a user receives a notification during the playback of a video when a sequence of video frames is linked to a still image that forms part of the animation. As explained below, a still image has a higher or more deformable resolution,
It is possible to have a wider field of view (eg, a panoramic image), a higher dynamic range, or a different aspect ratio than the video frame. Still images can also include stereo parallax information or other depth information that allows for a stereo three-dimensional (3D) view. When notified that a still image is available,
The user can provide an input to switch from a video display to an animated display during use to achieve the benefits associated with the animation (eg, higher resolution images). Alternatively, the user can stop the video display to steer within the panoramic image of the animation, or zoom in or out on the still image of the animation. In other embodiments, a user can play animations and videos on a picture in picture mode, or switch from displaying animations to cross-link video.

In one embodiment, cross-linking involves generating a still image from the video and then creating a cross-link between the still image and the frames of the video. In an alternative embodiment, the still image can be generated using a video other than the video to which the still image is cross-linked. Techniques for creating still images from video are described below. It should be understood that other similar techniques could be used to create a still image without departing from the spirit and scope of the present invention.

Still images having a higher spatial resolution than the frames of the video source can be achieved by integrating multiple video frames over time.
Images of video frames that are close together in time are typically panned by a camera,
It presents a small amount of position shift (eg, sub-pixel movement) as a result of zooming or other movement. The shift allows multiple video frames to be spatially registered to create higher resolution images. High resolution still images can then be created by interpolating between adjacent pixels of the spatially registered video frame.

[0077] Alternatively, the still image can be derived from a second video source that exhibits a higher resolution than the video to which the still image is linked. For example, motion pictures are typically recorded on film having a resolution many times higher than the NTSC video format commonly used for video tape.

A still image can also have a wider dynamic range than the video frames to which the still image is cross-linked. The dynamic range relates to the range of recognizable intensity levels for each pixel color component in the image. Since the exposure settings of the camera can be changed from frame to frame to accommodate changing lighting conditions (eg, automatic iris), the sequence of video frames will increase the dynamic range for each video frame. A slight change in color can be presented that can be integrated into the still image having. Still images can also be created from video sources with a large dynamic range (eg, film) and then cross-linked to videos with a smaller dynamic range.

A still image can also have a different aspect ratio than the video frame to which the still image is cross-linked. The aspect ratio indicates the ratio of the width to the height of an image. For example, a still image can be created from a video source with a relatively wide aspect ratio, such as film, and then cross-linked to a different video source with a narrower aspect ratio, such as NTSC video. The typical aspect ratio of the film is 2.2 × 1.
In contrast, NTSC video has a 4 × 3 aspect ratio.

The video frames resulting from panning the camera can be registered and composited to create a panorama. Video frames resulting from zooming the camera can be registered and composited to create large still images with different resolutions in different regions (ie, multi-resolution images). The multi-resolution area when camera zoom occurs will include higher resolution than other areas of the multi-resolution image. Panoramic and multi-resolution still images are a class of images referred to herein as steerable images. Generally, steerable images are:
Any image that can be panned or zoomed to provide a different view, or any image that can be used, including a three-dimensional image. While the panoramic image and the multi-resolution still image can be displayed by a composite image, the panoramic image or the multi-resolution still image can also be displayed by a spatially registered discrete still image.

A stereo image pair can be obtained from a sequence of video frames showing the tracking motion of the camera in the horizontal direction. Two video frames recorded from different viewpoints (eg, viewpoints separated by interpupillary distance) can be selected as a stereo image pair from the video sequence. Stereo images can be displayed using a number of different stereo visual devices, such as stereo 3D displays, stereo glasses, and the like.

Further, the stereo images may be used to identify corresponding pixels or image features in a given stereo image pair, for example, using image modification or feature matching techniques.
Can be analyzed. The corresponding pixel or image features can then be used to establish the pixel depth and thus create a 3D range image. 3
Building D-models, creating new views or scenes from images,
Range images can be used in several applications, including interpolating between images to create new views.

FIG. 14A illustrates the use of the crosslink generator 203 to establish a crosslink between the video source 10 and the animation 14 created by the animation production system 12 from the video source 10. The video source may be compressed by a video encoder 201 (eg, a vector quantizer) before being received at crosslink generator 203. According to one embodiment, the crosslink generator 203
Generates a crosslink data structure that includes respective pointers to keyframes in the animation to which the frames of the video source correspond.

FIG. 14B illustrates the use of a crosslink generator 203 to establish a crosslink between a video source 10 and an animation 205 created from another video source 204. Another video source 204 may have been used to create video source 10, or the two video sources 10, 204 may not be related. If the two video sources 10, 204 are not related, operator assistance is required to identify which image of the animation 205 will be cross-linked with the frames of the video source 10. there is a possibility. If two video sources 10, 204 are associated (eg, one is film and the other is NTSC format video), the temporal correlation or scene correlation automatically converts the image of the animation 205 and the frames of the video source 10. It can be used by a cross-link generator that cross-links dynamically.

FIG. 15 illustrates a crosslink data structure 212 according to one embodiment. Each data element of the crosslink data structure 212 is a video frame element (VFE).
) And corresponds to each frame of the video source. Therefore, the element VF
E ₁ , VFE ₂ , VFE ₃ , VFE _i , and VFE _{i + 1} are the frames V
_{F 1, VF 2, VF 3} , VF i, and corresponds to the VF _{i + 1} (not shown). As shown, the crosslink data structure 212 shows that each video frame element contains a pointer to the next video frame, and a background frame 215 for the animation.
, 216, implemented as a linked list that contains pointers to them. In an alternative embodiment, the cross-link data structure 212 may be implemented as an array of video frame elements rather than as a linked list. In yet other embodiments, the cross-link data structure 212 can be implemented as a tree data structure instead of a linked list. The tree data structure is useful for establishing associations between non-adjacent video segments and conducting searches to find specific video frames. In general, a cross-link data structure may be represented by any type of data structure without departing from the spirit and scope of the present invention.

In one embodiment, the background frames of the animation are each a pointer to the next background frame data structure (NEXT PTR), a pointer to the previous background frame data structure (PREV PTR), and an image pointer (IM
AGE PTR), a pointer to the interpolation information (INTERPTR),
Background frame data structure 21 including a timestamp and a pointer (VF PTR) to one or more elements in crosslink data structure 212
5, 216. NEXT PTR, PREV PTR, IMA
The GE PTR and the INTERP PTR are described above in connection with FIG.

The pointer to the VF PTR of a particular background frame data structure 215, 216 and the background frame data structure of the corresponding element of the crosslink data structure 212 forms a crosslink 217. That is, the background frame data structure and the video frame element each include an association with each other. The association can be a uniform resource location input device, a memory address, an array index, or any other value that associates a background frame data structure and video frame elements.

Referring to the background frame data structure 215, the VF PTR is shown in FIG. 15 to point to only one video frame element (VFE ₁ ) in the crosslink data structure 212, but the VF PTR is For each video frame element, it is possible to again include a separate pointer to it. For example, VF
PTR, for each video frame elements _{VFE 1, VFE 2, VFE 3} , can be a data structure including separate pointers. Alternatively, VF PTR
May be a data structure that includes a pointer to a value indicating the total number of video frame elements to which the video frame element (eg, VFE ₁ ) and the background frame data structure 215 are linked. The construction of other data that crosslinks the background frame data structure and the sequence of video frame elements
It can be used in alternative embodiments.

In one embodiment, the image pointer (IMAGE PTR) of each background frame data structure 215, 216 indicates that the background image acquiring image data for the background frame is, for example, a non-composite still image (ie, a moving image if any). (A video frame with the target object removed), a high resolution still image, a panorama or other composite image. The image pointer includes a member indicating a position of the background image in the memory and a shift in the background image where the image data is located with respect to the background frame.

A text descriptor (TEXT DESCR) can also be included as part of the background frame data structure 215, 216. In some embodiments,
A text descriptor is a pointer to a text description (eg, a string) that describes the part of the animation spanned by the background frame.
The text description can be displayed as an overlay on the animation or elsewhere on the display (eg, a control bar). While cross-linking, appropriate default values are assigned to each text description based on the type of exercise identified. Referring to FIG. 16, for example, the default text descriptions for each of the three depicted animation segments 221, 223, 225 are "Camera Still", "Camera Pan",
"Camera zoom". These default values can be edited by the user during cross-linking or during later playback of a video or animation. In an alternative embodiment, text in the background frame data structures 215, 216
The descriptor (TEXT DESCR) is not a pointer, but is an index that can be used to select a text description from a table of text descriptions.

When a video frame is being displayed using the above crosslink configuration, the corresponding video frame element of the crosslink data structure 212 identifies the crosslink background frame data structure 215, 216 of the animation. It is possible to refer to. The image pointer in the background frame data structures 215, 216 can then be consulted to determine if the background frame has been derived from a composite or non-composite image. In the case of a composite image, the user may be notified (eg, by a visual or audio support message) that the composite image is available during the playback of the video. The user then chooses to play the animation or look and maneuver in the background image.
For example, in the case of a panorama, the user can view the panorama using a panoramic viewing tool (ie, a software program that can be executed on a general purpose computer to display the portion of the composite image selected by the user). is there.
Similarly, for high resolution still images, the user may want to view the image as a still frame to recognize details that were unavailable or difficult to recognize with the video source. For a zoomable still image, the user may want to zoom in and out on a still frame. Other activities possible with the animation can also be performed, such as selecting a designated hot spot in the animation, isolating dynamic objects in the animation, directing object or background motion.

FIG. 16 shows a sequence of video frames 230 in a video source and a background image 231 from an animation created using the animation production techniques described above.
It is a figure of the cross link relationship between. As can be seen, the sequence of video frames is transmitted via the cross link 217 to the background images 221 and 223, respectively.
, 225, 227 associated with four video segments 222, 22
4, 226, 228. Video segment 222 shows a still scene (ie, is still within a certain motion threshold) and is cross-linked to a corresponding still background image 221. Video segment 224 shows a scene caused by camera panning and is cross-linked to a corresponding panorama 223 created by processing or stitching two or more frames from video segment 224. Video segment 226 shows the scene created by the camera zoom and is cross-linked to a high resolution, zoomable still image 225. Video segment 228 shows a scene created by moving the camera around one or more 3D objects, and is cross-linked to the 3D object image. As described above, the high resolution still image and the 3D object image are converted to video segments (eg, video segments 222, 224).
, 226, 228) by processing and combining the frames.

FIG. 17 shows a display 241 generated by the playback system. According to some embodiments, the playback system may display video or animation on the display 241. As shown in FIG. 17, the playback system is creating a video on the display 241. At the bottom of the display 241, controls including a rewind button, a play button, a pause button, and a stop button
A bar 242 is presented. According to one embodiment, as each video frame is created, the crosslink between the corresponding video frame element and the animated background frame is such that the background frame is from a high resolution still image, panoramic image or zoomable image. You will determine if it has been withdrawn. For example,
If the background frame is derived from the panoramic image, the icon indicating PAN is shown in FIG. 17 as being displayed, highlighted, or otherwise active. An audible tone can also be generated to indicate that a panoramic image is available. In response to the indication that a panoramic image is available, the user can either stop displaying the video, or display the panoramic image,
The PAN icon can be clicked or otherwise selected (e.g., using a mouse or other hand-held control). When a panoramic image is being displayed, the program code for manipulating the panoramic image, if not already provided, is automatically installed in the operating system's operating memory, allowing the user to pan and tilt the panoramic perspective view. And it is possible to zoom. Static or zoom icon S, as in the case of the PAN icon
When TILL, ZOOM is active, the user can click on the appropriate STILL or ZOOM icon to view the high resolution still image or zoomable image.

A video can be linked to one or more three-dimensional objects, or scenes associated with the video. When a link to a three-dimensional object is invoked during video playback, a particular view of the three-dimensional object is displayed in a manner similar to that described above. To generate different perspectives of the object, program code is executed to allow a user to change the direction and position of the virtual camera in a three-dimensional coordinate system.

In one embodiment, control bar 242 also includes an icon ANIM / VIDEO that can be used to switch between displaying a video and displaying an animation cross-linked to the video. The user is ANIM / VIDE
When the O button is clicked, the video frame element corresponding to the currently displayed video frame is examined to identify cross-link frames in the animation. The time stamp of the crosslink frame in the animation is used to determine the relative start time in the background and object tracks of the animation, and the playback system creates the animation accordingly. During the playback of the animation, the user re-enters ANIM / VI
Clicking on the DEO icon causes the current background track data structure to be examined to identify cross-link frames in the video. Then, the reproduction of the video is restarted at the cross link frame.

FIG. 18 depicts an alternative display 261 created by playing an animation in a playback system. In one embodiment, control bar 262 in display 261 includes icons for rewind, play, pause, stop, and play animation (ie, icons REWIND, PLAY, PAU).
SE, STOP). The control bar 262 also includes a resolution selector in the form of a slide bar 264 that allows a user of the playback system to indicate relative preferences for temporal and spatial resolution in playing the animation. By selecting slide 265 with a cursor control and moving slide 265 left and right within slide bar 264, the user can adjust his preferences for spatial and temporal resolution. For example, the leftmost position of slide 265 within slide bar 264 indicates a preference for maximum spatial resolution,
As slide 265 moves to the far right position within slide bar 264, a preference for maximum temporal resolution is indicated.

The ANIM / VIDEO icon is on the control bar 262 that allows the user to switch between cross-linked video and animation display. FIG.
According to the embodiment shown in FIG. 8, when selecting to display the animation, the cross-linked video is displayed simultaneously in the sub-window 268 in a picture-in-picture format. When the user clicks on the ANIM / VIDEO icon, the video is displayed in the first view area of display 261 and the animation is presented in sub-window 268. The picture-in-picture capability may be enabled or disabled from a menu (not shown) shown on display 261.

[0098] Crosslinks between animation and video can be used to provide some useful effects. For example, by cross-linking a steerable image of a market to a frame of video that includes a storefront, prompting the user to watch the video to switch to a panoramic image for shopping for goods or services depicted in the market. It is possible. Business processing of goods and services can be performed electronically via a communication network. Cross-linking the steerable image with the video is particularly effective where the steerable image is a panorama or other composite image of a location in the video scene. For example, when the video includes a steerable environment (eg, an airplane, spacecraft, submarine, cruise ship, building, etc.). For example, imagine a character on a cruise ship walking in front of a souvenir shop. The person watching it can voluntarily and intuitively stop the video and browse the souvenir shop.

[0099] Another useful application of crosslinks allows a user to compose a video. The user can cross-link the animation sequence to the video so that the animation sequence is automatically invoked when a cross-linked frame of the video is reached. When the end of the animation sequence is reached, the display of the video can resume with another cross-linked video frame. The user can selectively add outtakes to the video scene or replace portions of the video with animation sequences.

The foregoing description of the invention has been described with reference to specific exemplary embodiments. However, it will be apparent that various modifications and changes can be made to certain exemplary embodiments without departing from the broader spirit and scope of the invention, as set forth in the appended claims. Become. Accordingly, the specification and figures are to be regarded in an illustrative, rather than a restrictive, sense.

[Brief description of the drawings]

FIG. 1 is a diagram showing creation and distribution of animation.

FIG. 2 is a block diagram of an animation production system according to one embodiment.

FIG. 3 is a block diagram of a background track generator according to one embodiment.

FIG. 4A illustrates a video segment identified by a scene change estimator in a background track generator.

FIG. 4B is a flowchart illustrating the operation of the background motion estimator, background frame constructor, and background mixing estimator depicted in FIG. 3;

FIG. 5 is a diagram illustrating a background image set generated by the background frame constructor depicted in FIG. 3;

FIG. 6 is a block diagram of an object track generator according to one embodiment.

FIG. 7A depicts a video segment identified by the scene change estimator 41 of FIG.

FIG. 7B is a flowchart 100 of the operation of the object track generator according to one embodiment.

FIG. 8 is a diagram of an animation object according to one embodiment.

FIG. 9A illustrates an exemplary embodiment of a background frame mixing data structure that can be used to implement background mixing.

FIG. 9B illustrates a discontinuous mixing function.

FIG. 10 illustrates how background tracks and object tracks of exemplary animation objects can be used to synthesize interpolated frames during animation playback.

FIG. 11 illustrates a technique for providing multiple temporal resolutions of animation keyframes.

FIG. 12 illustrates a technique for providing multiple spatial resolutions of animation keyframes.

FIG. 13 illustrates the use of a server system to control the content of an animation data stream delivered to a playback system.

FIG. 14A illustrates the use of a crosslink generator to establish a crosslink between a video source created from a video source and an animation.

FIG. 14B illustrates the use of a crosslink generator to establish a crosslink between a first video and an animation created from a second video.

FIG. 15 illustrates a crosslink data structure according to one embodiment.

FIG. 16 is a flow diagram of a crosslink relationship between a sequence of video frames of a video source and a background image from an animation.

FIG. 17 depicts a display generated by the playback system.

FIG. 18 is a diagram depicting an alternative display generated by playing an animation in the playback system.

──────────────────────────────────────────────────続 き Continued on the front page (31) Priority claim number 09 / 096,487 (32) Priority date June 11, 1998 (June 11, 1998) (33) Priority claim country United States (US) ( 81) Designated countries EP (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, SE), OA (BF, BJ, CF, CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG), AP (GH, GM, KE, LS, MW, SD, SL, SZ, UG, ZW) , EA (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), AE, AL, AM, AT, AU, AZ, BA, BB, BG, BR, BY, CA, CH, CN , CU, CZ, DE, DK, EE, ES, FI, GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK , LR, LS, LT, LU, LV, MD, MG, MK, MN, MW, MX, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, TR, TT, UA, UG, UZ, VN, YU, ZA, ZW (72) Inventor Brandt, Jonathan United States, 95065, California, Santa Cruz, Rider Ridge Ridge Road, 377 F Term (Reference) BA06 BA08 BA09 BA10 BA11 BA13 CA05 CA06 CA08 DA04 DA07 EA03 EA06 EA09 EA13 EA18 EA19 EA24 EA27 EA28 FA02 FA05 FA06 GA08 [Continuation of summary] Less than the number of frames. For the connection between video and animation, a data structure is generated that includes elements corresponding to each frame of the first video. Information indicating the animation created from the second video is stored in one or more data structure elements.

Claims (84)

[Claims] 1. Examining a sequence of video images to identify a first variation of a scene depicted in the sequence of video images; and a first image representing a scene prior to the first variation from the sequence of video images. And obtaining a second image representing the scene after the first deformation, and between the first and second images to create a video effect indicating the first deformation and approximating the display of the sequence of video images. Generating information that can be used to interpolate in a computer-implemented method for creating an animation. 2. Examining the sequence of video images to identify a first variant of the scene determines when a difference between a selected one of the video images and a next one of the video images exceeds a threshold. The method of claim 1, wherein the selected one of the video images and the next one of the video images indicate a start image and an end image of a segment of the video image, respectively. 3. The method according to claim 2, wherein the start picture of the segment of the video picture indicates the end picture of the preceding section of the video picture. Determining when a difference between a selected one of the video images and the next one of the video images exceeds a threshold, selecting a video image following the starting image from the sequence of video images. Comparing the video image following the starting image with the adjacent preceding video image from the sequence of video images to generate an incremental difference value, adding the incremental difference value to the sum of the incremental difference values, selecting 3. The method of claim 2, comprising: repeating the comparing, comparing, and adding acts until the sum of the incremental difference values exceeds a threshold. 5. When the next one of the video images is added to the sum of the incremental differences,
5. The method of claim 4, wherein the video image is used to generate an incremental difference that causes the sum of the incremental differences to exceed a threshold. 6. The method of claim 5, wherein the end image of the set of video images is adjacent to the next one of the video images. 7. The method of claim 2, wherein the difference between the selected one of the video images and the next one of the video images includes a difference caused by a change in a camera arrangement used to record the sequence of video images. The described method. 8. The method of claim 2, wherein the difference between the selected one of the video images and the next one of the video images includes a color difference. 9. The method of claim 2, wherein the difference between the selected one of the video images and the next one of the video images includes a difference in elapsed time between the selected video image and the next one of the video images. The method described in. 10. The method of claim 1, wherein obtaining a first image and a second image from a sequence of video images includes selecting a start image and an end image of a set of video images as the first image and the second image, respectively. Item 3. The method according to Item 2. 11. A method for obtaining a second image from a sequence of video images, identifying one or more dynamic objects in a final image, and one or more dynamic objects to create a second image. 3. The method of claim 2, comprising removing the strategic object. 12. The method of claim 1, wherein identifying one or more dynamic objects in the end image comprises receiving one or more of the set of video images undergoing a second deformation not indicated by the first deformation in the set of video images. 12. The method of claim 11, comprising identifying a characteristic of: 13. The method according to claim 12, wherein the second variation comprises a change in the arrangement of one or more dynamic objects that is not due to a change in the arrangement of the cameras used to record the sequence of video images. Method. 14. Generating information that is indicative of a first deformation and that can be used to interpolate between the first and second images, comprising: The method of claim 1, further comprising: generating a value indicating the time elapsed between displaying the first image and displaying the second image. 15. The method of claim 15, wherein generating a value indicative of a degree of change comprises generating a value indicative of a degree of change caused by a change in an arrangement of cameras used to record the sequence of video images. Item 15. The method according to Item 14. 16. The method of claim 14, wherein generating a value indicative of a degree of change comprises generating a value indicative of a degree of color change. 17. Identifying a first variation of a scene depicted in the sequence of video images that indicates a change in camera arrangement used to record the sequence of video images; and identifying a first variation of the scene depicted in the sequence of video images. Identifying an object from the first and second images of the sequence of video images to identify a second deformation of the scene that indicates a change in the position of the object in the scene, and to generate first and second background images. Removing each region including the first and second background images between the first and second background images to create an interpolated background image that is displayable to approximate the first deformation of the scene and to approximate the first deformation of the scene. Generating background information that can be used to interpolate. A computer-implemented method for creating an animation. 18. Generating first and second object images including respective regions removed from the first and second images of a sequence of video images, wherein the first object image is a dynamic object before a second deformation. And the second object image represents a dynamic object after the second transformation. Interpolation between the first and second object images to create the interpolated object image, indicating the second transformation. 18. The method of claim 17, further comprising: generating object information that can be used to: display the interpolated object image to approximate a change in position of an object in the scene. 19. The method according to claim 19, further comprising the steps of:
And storing the second background image and the background information; and storing the first and second background images in the object track of the animation object.
19. The method of claim 18, further comprising storing the object image and the object information.
The method described in. 20. The method of claim 19, further comprising transmitting the animation object to a computer network in response to a request from the animation playback device. 21. A background track generator that examines a sequence of video images and generates a background track therefrom, wherein the background track is between a sequence of background frames and a background frame to synthesize additional images. A background track generator that includes deformation information that can be used to interpolate; and an object track generator that examines a sequence of video images and generates an object track therefrom, wherein the object track is , An object track generator comprising a sequence of object frames and deformation information that can be used to interpolate between the object frames to synthesize additional object images. . 22. The animation production system according to claim 21, further comprising an animation object generator for storing a background track and an object track in the animation object for later recall. 23. The animation production system of claim 22, further comprising: receiving a request to download the animation object from one or more client devices and responding to the one or more client devices. An animation distribution system comprising a communication device for transmitting an animation. 24. The animation production system according to claim 22, wherein reproduction time information is stored in the animation object to indicate a relative reproduction time with respect to the object track and the background track. 25. The system of claim 2, wherein at least one of the background track generator and the object track generator is implemented by a programmed processor.
2. The animation production system according to 1. 26. A scene change estimator for resolving a sequence of video images into one or more video segments, a background track generator, and in one or more video segments, based on respective deformations. 3. A background motion estimator for generating deformation information, and a background frame constructor for generating a sequence of background frames based on each deformation in one or more video segments.
2. The animation production system according to 1. 27. The animation production system according to claim 26, wherein the background track generator further comprises a mixing estimator for generating mixing information for combining background frames in a sequence of background frames. 28. The animation production system according to claim 27, wherein the mixed information indicates a cross dissolve operation. 29. The background frame constructor generates at least one background frame of a sequence of background frames by combining one or more images from one or more video segments. Animation production system. 30. The animation production system according to claim 29, wherein the background frame constructor combines one or more images into a panoramic image by stitching the one or more images. 31. The animation production system according to claim 29, wherein the background frame constructor combines one or more images into a high resolution image. 32. A computer-readable medium having instructions stored thereon, the instructions being, when executed by a processor, for indicating to a processor a first variation of a scene depicted in a sequence of the video image. Examining the sequence; obtaining a first image and a second image from the sequence of video images, wherein the first image represents the scene before the first deformation and the second image represents the scene after the first deformation. Generating information that can be used to interpolate between the first image and the second image to create a video effect that approximates the display of the sequence of video images; A medium that does things. 33. The computer readable medium of claim 32, wherein the computer readable medium includes one or more mass storage disks. 34. The computer readable medium according to claim 33, wherein the computer readable medium is a computer data signal encoded on a carrier wave. 35. The instructions for causing the processor to examine a sequence of video images to identify a first variant of the scene, the instructions comprising, when executed, causing the processor to determine a difference between a selected one of the video images and a next one of the video images. 34. The computer-readable method of claim 33, further comprising instructions for determining when is greater than a threshold, wherein the selected one of the video images and the next one of the video images indicate a start image and an end image of the set of video images, respectively. Medium. 36. An instruction, which when executed, causes the processor to determine when a difference between a selected one of the video images and a next one of the video images exceeds a threshold, the processor comprising: Selecting the following video image; comparing the video image following the start image with an adjacent preceding video image from the sequence of video images to generate an incremental difference value; 36. The computer-readable medium of claim 35, comprising instructions for adding to the sum and repeating the act of selecting, comparing, and adding until the sum of the incremental difference values exceeds a threshold. 37. An animation comprising: inspecting the sequence of video images to identify a first variation of the scene depicted in the sequence of video images; and displaying the scene representing the scene prior to the first variation from the sequence of video images. Acquiring one image and a second image representing the scene after the first deformation; and information indicating the first deformation and the first image and the second image to create a video effect that approximates the display of the sequence of video images. A computer-readable medium having data stored for displaying a sequence of images from its animation that has been created by generating information that can be used to interpolate between the images. 38. Generating a data structure including elements corresponding to respective frames of the first video, and storing one or more elements of data structure information indicating an image of an animation created from the second video. How to link videos and animations, including: 39. The method of claim 38, wherein generating a data structure including the element comprises generating a data structure including a respective element for each frame of the first video. 40. The method of claim 38, wherein storing information indicative of an image of the animation comprises storing a reference value indicative of a key frame of the animation. 41. The method of claim 40, wherein storing a reference to a key frame of the animation comprises storing a reference value indicating a background frame of the animation object. 42. Storing a reference value indicating a background frame includes storing an address of a background frame data structure, wherein the background frame data structure includes information indicating a background image and the background image includes a composite image. 42. The method of claim 41, comprising information indicating whether 43. The method of claim 42, wherein the information indicating whether the background image is a composite image includes information indicating whether the background image is a panoramic image. 44. The method of claim 38, wherein the data structure is an array of elements. 45. The data structure linked to a list of elements.
The method described in. 46. The first video and the second video are the same video.
9. The method according to 8. 47. The method of claim 3, wherein the first video is generated using the second video.
9. The method according to 8. 48. The method of claim 38, wherein the animation comprises a high resolution still image. 49. The animation comprising a multi-resolution still image having first and second regions, wherein the first region has a higher pixel resolution than the second region.
The method described in. 50. The method of claim 38, wherein the animation comprises a still image having a wider field of view than the frames of the first video. 51. The method of claim 38, wherein the animation comprises a still image having a wider dynamic range than the frames of the first video. 52. The method of claim 38, wherein the animation comprises a still image having an aspect ratio different from an aspect ratio of a frame of the first video. 53. The method of claim 38, wherein the animation comprises a pair of still images forming a stereo image pair. 54. The animation as claimed in claim 38, wherein the animation comprises an image containing depth information.
The method described in. 55. The method of claim 38, wherein the animation comprises an object having three-dimensional geometric properties. 56. The method of claim 38, wherein the text description is associated with at least one image of the animation. 57. The method of claim 38, wherein the animation comprises an animation object having a plurality of elements corresponding to the images of the animation, wherein the method further comprises: converting one or more frames of the first video. A method comprising storing in one or more of a plurality of elements in the indicated animation object information. 58. The method of claim 38, wherein the animation comprises an animation object having a plurality of elements corresponding to the images of the animation, wherein the method further comprises the plurality of animation object information indicating a sequence of frames. Storing in one or more of the elements of the. 59. Displaying a frame of a video on a display of a playback system; and when the animation key frame is automatically generated using the frame of the video, the animation key corresponding to the frame of the video. Playing a data element associated with the frame of the video to identify the frame; and prompting a user of the playback system to begin displaying an image associated with the animation keyframe. How to display video on the system. 60. A method for determining whether an image associated with an animation keyframe is a composite image, and for viewing the composite image if the image associated with the animation keyframe is a composite image. 60. The method of claim 59, further comprising: signaling a user. 61. The method of claim 60, wherein determining whether an image associated with the animation keyframe is a composite image includes determining whether the image associated with the animation keyframe is a panoramic image. the method of. 62. Receiving a request from a user to view a panoramic image, and executing the program code in response to a request from the user to view the panoramic image in response to a steering input from the user. 63. The method of claim 61, further comprising: 63. The method of claim 62, wherein the maneuver input from the user includes a command to pan a perspective view of the scene depicted in the panoramic image in the horizontal direction. 64. The method of claim 62, wherein the maneuver input from the user includes a command to tilt a perspective view of the scene depicted in the panoramic image. 65. Determining whether the image associated with the animation keyframe is a composite image includes determining whether the image associated with the animation keyframe is a high resolution still image. 60. The method according to item 59. 66. Receiving a request from a user to view a high-resolution still image and responding to a zoom input from the user to scale the viewing of the high-resolution still image to the request from the user. 66. The method of claim 65, further comprising executing the program code in response. 67. Prompting the user of the playback system to begin displaying an image associated with the animation keyframe signals the user that it is possible to view the image associated with the animation keyframe. 60. The method of claim 59, comprising displaying an indicator on a display of the playback system to do so. 68. Prompting the user of the playback system to begin displaying an image associated with the animation keyframe signals the user that it is possible to view the image associated with the animation keyframe. 60. The method of claim 59, comprising activating an indicator on a display of the playback system to do so. 69. Activating an indicator on a playback system includes activating an indicator on a handheld control of the playback system.
9. The method according to 8. 70. A method for displaying a video on a playback system, the method comprising: displaying a frame of the video on a display of the playback system; and identifying an animation keyframe corresponding to the frame of the video. Inspects the data elements associated with the frames of the video and displays that the animation keyframes are automatically generated using the frames of the video, and simultaneously displays the frames of the video and animates them in a window on the display. Displaying an image associated with the keyframe; 71. A processor, a display coupled to the processor, a media reader coupled to the processor, and a memory coupled to the processor, wherein the memory is configured to execute to the processor, to a video frame. Video data with associated elements
Signaling a media reader to provide video data including a sequence of frames from a machine-readable medium, causing a sequence of video frames to be displayed on a display, and animation keyframes corresponding to one or more video frames Inspect the data structure elements associated with the video frames to identify the animation frame, and an animation keyframe is automatically generated using one or more of the video frames to provide the playback system user with an animation. A playback system comprising program code for prompting to start displaying an image associated with the key frame. 72. Displaying a frame of a video on a display of a playback system, and switching from displaying the video to displaying an image of a 3D object steerable image associated with the frame of the video. A method comprising: receiving input from a requesting user; and displaying a steerable image. 73. The method of claim 72, further comprising panning a perspective view of the steerable image in response to input from a user. 74. The method of claim 72, further comprising processing a sale of the item depicted in the steerable image in response to input from a user. 75. The method of claim 72, further comprising processing an arrangement to perform a service indicated by one or more characteristics of the steerable image in response to input from a user. 76. The method of claim 72, further comprising zooming a perspective view of the steerable image in response to input from a user. 77. The method of claim 72, wherein the steerable image is a panoramic image of a market that includes merchandise that can be purchased electronically. 78. The method according to claim 72, wherein the steerable image includes one or more three-dimensional objects. 79. A user requesting to display a frame of a video on a display of a playback system and to switch from displaying the video to displaying a three-dimensional object associated with the frame of the video. Receiving the input of the user. 80. The method of claim 79, further comprising changing a viewpoint from a viewpoint at which the three-dimensional object is displayed in response to a user input. 81. A computer readable medium having data stored for displaying a sequence of images from an animation, the method comprising: generating a data structure including an element corresponding to each frame of a first video; A computer-readable medium wherein the animation is linked to the video by storing one or more elements of data structure information indicating an image of the animation created from the video. 82. A method for storing a set of keyframes created from a video in an animation object and indicating a first sequence of keyframes selected from the set of keyframes.
Storing in the animation object one or more values and information for interpolating between the first sequence of keyframes, the number of keyframes of the first sequence of keyframes selected from the set of keyframes; One or more values indicating a lesser number of second sequences;
An animation object that stores information to interpolate between keyframes in a sequence. 83. Storing a set of keyframes includes storing first and second subsets of keyframes in an animation object, wherein the second subset of keyframes is stored in a first subset of keyframes. 83. The method of claim 82, wherein the included image comprises a reduced resolution version. 84. The method of claim 82, wherein each of the one or more values indicating the first sequence of selected keyframes is a reference value that identifies a respective keyframe in the set of keyframes.