WO2001045036A1

WO2001045036A1 - Slow motion system

Info

Publication number: WO2001045036A1
Application number: PCT/US2000/033674
Authority: WO
Inventors: Steven D. Edelson; Klaus Diepold
Original assignee: Dynapel Systems, Inc.
Priority date: 1999-12-14
Filing date: 2000-12-13
Publication date: 2001-06-21

Abstract

In a slow motion system, interpolated frames are provided between original motion picture frames by comparing the original motion picture frames to generate dense forward (224) and backward (226) vectors to represent the motion of image elements from frame to frame. The dense forward and backward motion vector fields are used to generate motion adjusted images which are merged to form the interpolated frames. The interpolated frames are then combined by interleaving with the original motion picture frames to provide a slow motion depiction of the motion depicted in the original motion picture frames.

Description

SLOW MOTION SYSTEM

This invention relates to an improved system for showing motion pictures in slow motion.

Background of the Invention In a conventional slow motion picture system, picture frames of a scene are obtained at a high rate several times the conventional frame rate of 24 to 30 frames a second and then the frames are displayed at a much lower frame rate than the frame rate at which the frames are obtained. This displayed frame rate should be rapid enough to provide quality simulation of motion in which the motion is perceived without the motion being jerky wherein the moving objects in the scene are perceived to move in abrupt transitions. The displayed frame rate accordingly should be at least 24 frames per second and, typically, will be in the range of 24 to 30 frames a second. The above-described slow motion system requires a motion picture camera which is capable of obtaining the frames of the scene at the desired high frame rate and cannot be used to generate slow motion from a conventional motion picture which is already in existence. An alternative slow motion system has been proposed wherein the motion picture has been encoded into compression coded picture data in accordance with the Motion Picture Experts Group (MPEG) Standard, which is a compression that is typically used to transmit motion pictures to remote locations. The compression coded motion picture data, coded according to the MPEG Standard is called MPEG Code or MPEG Data. The MPEG Code includes motion vector data which represents vectors intended to represent motion corresponding to blocks of pixels from scene to scene in the motion picture. In accordance with the slow motion technique, these motion vectors are used to generate inteφolated frames between the original frames and the resulting sequence of frames are displayed at a frame rate to reproduce the original motion picture in slow motion. The resulting slow motion depiction suffers in quality because the motion vectors in the MPEG Data frequently include vectors representing nonexistent motion between similar appearing parts of a scene. In addition, the system has a disadvantage in that it requires the original motion picture, from which the slow motion display is to be produced, to be encoded in the MPEG Code.

Summary of the Invention The system of the present invention provides high quality slow motion depiction from any conventional motion picture in which the motion picture frames are represented by digital data. In accordance with the invention, the successive frames of the digital data are compared to generate forward and backward dense vector fields representing the change in position of image elements from frame to frame. The term "image element" as used herein refers to a pixel sized element of the depicted scene and objects in that scene whereby an image element representing a part of an object which is moving is considered to move with the object. A "dense vector field" as used herein means a set of vectors representing image element motion wherein the vectors have substantially the same density as the pixels or image elements of the motion picture frames. The forward and backward dense vector fields are scaled to be a fraction of the original magnitude and then are used to generate motion adjusted images which are merged together to produce inteφolated frames between the original set of picture frames. In the motion adjusted image generated from the forward motion vectors, the image elements are moved from the preceding motion picture frame to positions in the motion adjusted image in accordance with the scaled forward motion vectors. In the motion adjusted image generated from the scaled backward motion vectors, the image elements in the succeeding scene are moved to positions in the motion adjusted image in accordance with the scaled backward motion vectors of the dense backward motion vector field. After the motion adjusted images have been merged into the inteφolated images, the inteφolated images are then interleaved with the original frames to provide a succession of frames which are then reproduced at a rate to produce a slow motion representation of the original motion picture. Brief Description of the Drawings

Fig. 1 schematically illustrates the method of the slow motion system of the present invention.

Fig. 2 is a block diagram of the system of the present invention. Fig. 3 is a block diagram of an inteφolator employed in the system of the present invention.

Description of the Preferred Embodiments

In accordance with the method of the invention illustrated in Fig. 1 , frames are inteφolated between successive frames of a scene representing a motion picture containing one or more moving objects. In Fig. 1 , two successive frames, frame A and frame B, of a motion picture are shown. The frames depict a stationary object in the form of a station house 200 and a moving object in the form of an engine 212. As shown in frames A and B, the engine 212 has moved from a central position in frame A toward the lower right-hand corner of the scene as shown in frame B. The frames A and B are compared to generate a dense forward vector field wherein a vector value is generated for each pixel in the image shown in frame A to indicate where the corresponding image element has been moved in the image in frame B. In Figure 1 , vectors 214 are part o the dense forward vector field. In the example represented in Figure 1 , all of the image elements stay in the same position except the image elements representing the engine 212. For the image elements representing the engine 212, vectors 214 in the dense forward vector field are computed from the change in the position of the image elements depicting the engine 212 from frame A to frame B. Since the remaining image elements do not change position from frame A to frame B, the vectors for these stationary image elements will have a zero magnitude in the dense forward vector field. In a similar manner, a dense backward vector field is generated representing the difference in the image element position from frame B to frame A. Thus, the dense backward vector field will contain backward vectors 234 corresponding to the image elements depicting the engine computed from the difference in the positions of the image elements depicting engine 212 in frame A from their position in frame B. As in the case of the dense forward vector field, vectors in the dense backward vector field which correspond to image elements which do not change in position from frame B to frame A, have zero magnitude. The magnitudes of the vectors 214 and 234 are scaled to correspond to the location in time of the frame being inteφolated between the frames A and B. For puφoses of simplifying the explanation of the invention, the method illustrated in Fig. 1 generates only a single inteφolated frame between each pair of original motion picture frames. As will be explained below, the method of the invention will readily produce a multiple number of inteφolated frames between each pair of original motion picture frames. Since the inteφolated frame in Fig. 1 will be equally spaced in time from frame A and frame B, the magnitudes of the forward vectors and the backward vectors are reduced by one-half, resulting 14 scale forward vectors 224 and 14 scale backward vectors 226. A first motion adjusted image is then generated from frame A and the 14 scale dense forward vector field. The first motion adjusted image will be derived from frame A by moving the image elements of frame A in accordance with the 14 scale forward vectors. All of the image elements of frame A which do not depict the engine 212 and which have a magnitude of zero stay in the same position in the first motion adjusted image. Thus, the image elements depicting the engine 212 are moved in the first motion adjusted image to positions as indicated by the 14 scale forward vectors 224, and, in the first motion adjusted image, the engine will appear halfway between its position in frame A and frame B. A second motion adjusted image is generated in a similar manner from frame B and the 14 scale backward vector field. In a practical application, the dense motion vector field may represent changes in object size, three-dimensional motion and other complex motion. When the vectors of the dense motion vector field are scaled, they will sometimes call for placing more than one image element in the motion adjusted image in the same position. In the simplified example of Fig. 1, the image element depicting the lower lefthand corner of the engine 212 will be moved by its scaled forward vector to the position of a stationary image element so that in the motion adjusted image, two image elements will be assigned to the same position. This phenomena may be handled by a simple algorithm, such as by averaging the image elements, or by a simple rule or by means of additional image information. The end result may not be an ideal representation of the corresponding image element in the motion adjusted image, but the result does not critically affect the overall motion picture quality.

After the first and second motion adjusted images are generated, they are merged to form an inteφolated image. A simple algorithm for merging the two motion adjusted images would be to average the corresponding image elements in the two motion adjusted images. Preferably, the algorithm for blending the image elements in the merged image determines whether the image element from the first frame or the second frame is a better representation of a given pixel in the inteφolated frame. The preferred merging algorithm detects for each image element in the inteφolated frame whether it is subject to occlusion or innovation phenomena. An occlusion phenomena is detected for an image element in the inteφolated frame when the image element is visible in the first frame and is not visible in the second frame, thus indicating that the image element in the second frame has been occluded by a moving object. In this case, the image element from the first motion adjusted image is selected for the corresponding pixel in the inteφolated image. An innovation phenomena is detected when an image element that was not visible in the first frame becomes visible in the second frame. In this case, the image element from the second motion adjusted image is selected for the corresponding pixel in an inteφolated frame. Image elements which are not detected as being subject to occlusion or innovation phenomena from the image elements as represented in the first and second motion adjusted images are averaged in the inteφolated frame.

The process as represented by Fig. 1 is repeated for each frame inteφolated from each successive overlapping pair of adjacent frames in the original motion picture. The term "successive overlapping pairs of adjacent frames" refers to the sequence wherein, if the successive frames are labeled A, B, C, D, then the frames B and C are a successive overlapping pair of adjacent frames to the pair of adjacent frames A and B and the frames C and D are a successive overlapping pair of adjacent frames to the pair of adjacent frames B and C. In the system of the invention as shown in Fig. 2, digital data representing frames of the motion picture are provided on a channel 406 by a motion picture source 407. The motion picture frames represented by the digital data are designed to be shown at a frame rate of at least 24 frames per second and, preferably, in the range of 24-30 frames per second, and when displayed at the designed rate, the motion picture frames will reproduce the depicted motion at the same speed that the motion actually occurred. The video frames are applied to a motion extractor 412 of a motion enhancement module 410, which also includes a motion adjusted inteφolator 416. The motion extractor 412 computes forward and backward dense motion vector fields for the image elements for each successive overlapping pair of adjacent frames as described above in connection with Fig. 1. The details of motion extraction are disclosed in German application No. 95912250.8 and in published PCT application No. WO99/07156, which are hereby incoφorated by reference. The forward and backward dense vector fields are applied to the motion adjusted inteφolator 416, which also receives the video data on channel 406. The inteφolator 416 generates inteφolated frames from the received forward and backward dense vector fields and from the corresponding motion picture frames in the manner described above in connection with Fig. 1. The inteφolator 416 applies the inteφolated frames to the interleavor 420 which interleaves the inteφolated frames between the corresponding pairs of frames in video data on channel 406 to produce a sequence of frames on channel 430 combining the original motion picture frames with the inteφolated frames.

The video data on channel 430 is then applied to a slow motion video display device 432, which stores the received motion picture frames in a buffer storage device and displays the motion picture frames at a frame rate to produce a slow motion display. In the specific embodiment of the invention, the frames received by the slow motion display device are displayed at the display frame rate of the original motion picture on channel 406. Thus, if there were just one frame inteφolated between each pair of original frames, the slow motion would be at half speed. Frequently, it will be desirable to slow the motion down by more than a factor of two. For this puφose, a multiple number of inteφolated images are created between each pair of original motion picture frames. To produce these multiple inteφolated frames, the vectors of the forward and reverse dense vector fields must be appropriately scaled in accordance with where in time, the inteφolated frames are located between successive original motion picture frames. For example, to slow the motion down by a factor of 4, three inteφolated images are generated between each pair of adjacent frames. To generate these inteφolated images, the forward and backward vectors are scaled at 1/4, 1/2, and 3/4 to produce three motion adjusted images from the scaled forward vectors and three motion adjusted images from the scaled backward vectors. The first inteφolated image in the interval between adjacent original motion picture frames would then be produced from a merger of the motion adjusted image produced from the 1/4 scale forward vectors and the motion adjusted image produced from the 3/4 scale backward vectors. The second inteφolated image located in time midway between the original frames would be generated by merging the motion adjusted images produced from the one-half forward and backward scale vectors respectively. The third inteφolated image in the interval between the two adjacent original motion picture frames is generated by merging the motion adjusted image from the 3/4 scale forward vectors with the motion adjusted image produced from the 1/4 scale backward image. In the examples described above, an integer number of inteφolated frames are added for each original motion picture frame. However, the number of inteφolated frames does not have to be restricted in this manner. If the number of inteφolated frames added per original frame is not an integer, some of the original motion frames will have to be discarded in the interleavor. The slow motion system c ould be provided with the capability of varying the degree of slow motion over a continuous range. In such a system, to provide for some portions of the slow motion range, while maintaining a constant display frame rate, all but one of the original motion picture frames might have to be discarded. In the slow motion system described above, the sequence of motion picture frames comprising the original motion picture frames and the inteφolated frames are stored in a buffer storage device in the slow motion display device 432. Alternatively, the original motion picture frames could be presented by the motion picture source 407 on channel 406 to the motion adjusted enhancement model and interleavor 420 at the desired slow motion rate whereupon the interleavor 420 would produce a sequence of motion picture frames combining the original frames with the inteφolated frames at the desired slow motion rate on channel 430. The output frames from the interleavor could then be displayed by a conventional video display device to produce the slow motion display. The above description is of preferred embodiments of the invention and modification may be made thereto without departing from the spirit and scope of the invention which is defined in the appended claims.

Claims

1. A method of generating a slow motion display from the motion picture comprising a first sequence of motion picture frames, said method comprising comparing each motion picture frame with the succeeding motion picture frame in said first sequence to generate dense forward vector fields representing the difference in position of image elements from each motion picture frame to the succeeding motion picture frame and to generate dense backward vector fields representing the difference in position of image elements from each motion picture frame to the preceding motion picture frame, generating a first set of motion adjusted images, corresponding to inteφolated motion picture frames, in accordance with said dense forward vector fields, wherein the image elements of each motion picture frame are moved to positions in said first set of motion adjusted images in accordance with said dense forward vector fields scaled in accordance with the position in time of the corresponding inteφolated motion picture frames, generating a second set of motion adjusted images, corresponding to said inteφolated frames, in accordance with said dense backward vector fields wherein the image elements of each motion picture frame are moved to positions in said second set of motion adjusted images in accordance with said dense backward vector fields scaled in accordance with the position in time of the corresponding inteφolated images, merging said first set of motion adjusted images with said second set of motion adjusted images to produce said inteφolated motion picture frames, and displaying said inteφolated motion picture frames in a second sequence of motion picture frames to produce a slow motion display of the motion depicted by said first sequence motion picture frames.

2. A method as recited in claim 1 , further comprising interleaving said inteφolated motion picture frames with said first sequence of motion picture frames to produce said second sequence of output motion picture frames.

3. A slow motion display system comprising a motion picture source providing a first sequence of motion picture frames for depicting motion, a motion extractor connected to receive said motion picture frames of said first sequence and compare each motion frame of said first sequence with the succeeding motion picture frame in said first sequence to generate dense forward field representing the change in position of image elements from each motion picture frame of said first sequence to the succeeding motion picture frames of said first sequence and to generate a dense backward vector field representing the difference in position of image elements from each motion picture frame of said first sequence to the preceding motion picture frame of said first sequence, an image element moving circuit connected to receive said dense forward vector fields and generating a first set of motion adjusted images by moving the image elements of said motion picture frames of said first sequence to positions in accordance with said dense forward vector fields and generating a second set of motion adjusted images by moving the image elements of said motion picture frames of said first sequence to positions in accordance with said dense backward vector field, a merging module connected to receive said first and second sets of motion adjusted images and merging corresponding ones of said first and second motion adjusted images to produce a set of inteφolated frames, an interleavor for combining said inteφolated frames with said motion picture frames of said first sequence to produce a second sequence of motion picture frames, and a motion picture display device for displaying said second sequence of motion picture frames at a rate to provide a slow motion display of the motion depicted by said motion picture frames of said first sequence.