WO2002023901A2

WO2002023901A2 - Special effects in video in response to depicted motion

Info

Publication number: WO2002023901A2
Application number: PCT/US2001/028359
Authority: WO
Inventors: Steven D. Edelson
Original assignee: Dynapel Systems, Inc.
Priority date: 2000-09-11
Filing date: 2001-09-12
Publication date: 2002-03-21
Also published as: WO2002023901A9; WO2002023901A3

Abstract

In a video processing system (15), dense motion vector fields are generated representing the motion of image elements from frame to frame. One or more pixel characteristics of the pixels in the video frames are adjusted in accordance with the corresponding vectors of the dense motion vector fields. The pixel characteristics that are adjusted include color, brightness, saturation, and contrast. The pixel characteristics are adjusted in accordance with the magnitude and/or direction of the dense motion field vectors and in accordance with the acceleration of the corresponding image elements.

Description

SPECIAL EFFECTS IN VIDEO IN RESPONSE TO DEPICTED MOTION

This invention relates to video processing and more particularly to a technique of processing video to provide a change in the depiction of objects and background in a video motion picture based on the motion depicted in the video. Background of the Invention

In a co-pending application entitled, "System for the Estimation of Optical Flow" filed June 14, 2000 by Siegfried Wonneberger, Max Griessl, and Markus Wittkop, there is disclosed a method and apparatus for generating a dense motion vector field for a motion picture wherein the motion of pixel sized image elements from frame to frame is detected and represented by vectors. This copending application is hereby incorporated by reference. The word "image element" is used herein in distinction from the word "pixel" in that a pixel represents a color and brightness in a particular location in the video frame whereas an image element means a pixel sized piece of an object or background that is depicted in the video frame. The image element will change position from frame to frame when the object of which the image element is a part moves from frame to frame, or when the background of which the image element is a part is moved from frame to frame, as would occur when the video camera is panned, h a dense motion vector field from a first frame to a successive second frame, there will be vectors representing the change in position of the image elements from the first frame to the second frame wherein each motion vector has an origin at the corresponding image element in the first frame and has its termination point at the position of this image element in the second frame. The vectors of the dense motion vector fields are indications of the motion of the image elements from frame to frame. The present invention is directed to a system making novel use of the dense motion vector fields.

Summary of the Invention In accordance with the invention a dense vector field is generated for each pair of adjacent pixel frames in a motion picture video. In response to the vectors of the dense motion vector fields, the pixels corresponding to the vectors are changed in accordance with the motion vectors so that depicted objects in rapid motion appear differently than objects not in motion or moving slowly. In accordance with the invention, one or more pixel characteristics of the pixels corresponding to the motion vectors are changed in accordance with the magnitude and/or direction of the motion vectors. For example, the color of the pixels may be changed in accordance with the magnitudes of the motion vectors so that rapidly moving objects are tinted towards a particular color. Alternatively, the brightness, the saturation or the contrast may each be modified in accordance with motion vectors. In addition, combinations of color, brightness, saturation, and contrast may be changed in accordance with the magnitude of the vectors. Alternatively, the color of the pixels corresponding to the motion vectors may be changed in accordance with the direction of the motion vectors so that the objects moving in a particular direction are tinted towards a particular color. For example, all of the pixels corresponding to vectors pointed upwardly could have their color changed to tint the objects moving upwardly toward yellow for example. Alternatively, the brightness, saturation or contrast could be modified in accordance with the direction of the motion vectors. For example, all of the pixels corresponding to motion vectors pointing in the leftward direction could have increased saturation whereas all of the pixels corresponding to motion vectors pointed in the rightward direction could have reduced saturation, making these latter pixels grayer in the depicted scene. Different combinations of color, brightness, saturation and contrast may be changed in accordance with motion vector direction.

In addition to the above described alternatives, the change in one or more pixel characteristics may be controlled in accordance with a combination of motion and direction. Another alternative is to have all of the pixels corresponding to image elements moving with a velocity over a predetermined magnitude or moving with a velocity in a predetermined direction over a predetermined magnitude to be tinted toward a selected color, or to be changed in brightness, saturation, or contrast by a particular amount. Another alternative is make all of pixels corresponding to vectors with greater than a predetermined magnitude or moving in a particular direction have a selected color, brightness, saturation, or contrast.

In accordance with still another alternative the pixel characteristics are changed in accordance with acceleration or deceleration or in accordance with velocity duration. These criteria for controlling the change in the pixel characteristics can also be used in combination with velocity magnitude and direction.

Brief Description of the Drawings Fig. 1 is a block diagram of the system of the present invention. Fig. 2 is a flow chart illustrating the process performed by the video processor of Figure 1 to adjust pixel characteristics in accordance with motion vector magnitude.

Figure 3 is a flow chart of a process performed by the video processor of Figure 1 to adjust pixel characteristics in accordance with vector direction. Figure 4 is a flow chart illustrating a process performed by the video processor of Figure 1 to adjust pixel characteristics in accordance with image element acceleration.

Description of a Preferred Embodiment of the Invention As shown in Figure 1, the system of the invention comprises a video source 11 which provides video frames in sequence wherein the pixels of each frame is represented in digital form. The video frames from source 11 are stored in storage device 13 by a video processor 15. The video processor is programmed to process the video data in the storage device 13 and is also capable of transmitting the video stored in the storage device 13 to the video display 17 to display the stored video. This flow chart of Figure 2 illustrates a process carried out by the video processor 15 on a video stored in the storage device 13 in accordance one set of embodiments of the invention. As shown in Figure 2, the processor 15 generates dense motion vector fields representing the motion or change in position of image elements from frame to frame in the video. The result of this process will be sets of vectors, one set for each pair of adjacent frames in the video and each set comprising a dense motion vector field. In each vector set, each vector represents the change in position of a pixel sized image element from frame to frame. The image elements represent a small pixel sized pieces of a depicted object or background. If the object is moving as represented in adjacent video frames, then the image elements of such objects will move with the object. Similarly if the video camera is panned so that the background in effect is moving, the image element of the moving background will change position from frame to frame. Thus, each vector of a dense motion field will represent the motion of a corresponding image element. The image elements normally won't be precisely aligned with pixels. A set of four pixels in each video frame will normally be overlapped by and correspond to each image element. Four pixels will thus correspond to each motion vector origin and each vector termination point.

Following the generation of dense motion vector field sets for pairs of adjacent video frames, the video processor in the embodiments represented in Figure 2 compares the vector magnitudes of the vectors in the dense motion vector field to a standard to compute an adjustment factor. For example, the adjustment factor could be an expected average of the magnitudes of the vectors of the dense vector field. The video processor then adjusts one or more characteristics of the pixels corresponding to the vectors in accordance with the adjustment factor. The adjusted one or more pixel characteristics may be color, brightness, saturation, contrast, or combinations thereof, hi each adjustment the one or more characteristics of the corresponding pixels are adjusted in proportion to how much the corresponding image element overlaps the corresponding pixels. By adjusting the overlapped pixels in this manner, the characteristics of the equivalent of a pixel at the termination point and/or the origin of a motion vector is adjusted in accordance with vector magnitude.

When the pixel characteristic being adjusted is color, the color controlling value or values of the video color system is adjusted in accordance with the adjustment factor. For Example, if the video color system in HSL (hue, saturation, lightness) then to change color, the H value is adjusted in accordance with the motion vector. If the video color system in RBG, then one or two of the three color separations will be increased in accordance with the adjustment factor and the other one or two of the color separation elements will be decreased in accordance with the adjustment factor. The result of this adjustment in response to each vector in the dense motion vector field will be to tint the color of rapidly moving objects toward a selected color wherein the more rapidly an object is moving the more intense will be the color tinting. By controlling the relative degree with which the color values are adjusted in accordance with the adjustment factor the moving objects can be tinted toward any selected color. Alternatively, the tinting can be of stationary and slower moving objects. One way of achieving this latter effect is to invert the adjustment factor. This action would cause stationary objects to be tinted to a maximum color value while the slower moving objects will be tinted toward the selected color. The system of the invention is applicable to other video color systems including YUN, and YIQ. Instead of using the adjustment factor to adjust color the video processor may be programmed to adjust other pixel characteristics as shown in Figure 2 including brightness, saturation, or contrast, or any combination of these characteristics with or without color.

In another alternative arrangement, instead of tinting the pixels proportionally to an adjustment factor, all the pixel equivalents corresponding to vector magnitudes over a predetermined value would be changed to a given color or tinted by a fixed amount toward a selected color.

Instead of adjusting the selected pixel characteristic or characteristics in accordance with vector magnitude the pixel characteristics could be adjusted in accordance with vector direction. For example, all of the objects moving upwardly could be tinted toward yellow. Another example would be to make leftward moving objects more saturated and rightward moving objects less saturated which would mean that the rightward moving objects would become grayer.

Figure 3 is a flow chart of the operation of the video processor 15 to adjust a pixel characteristic in accordance with vector direction. As shown in Figure 3, the processor generates the dense motion vector field for each pair of adjacent video frames in the video stored in the storage device 13. The processor then compares the direction of the vectors of the dense motion vector fields with a selected direction to compute an adjustment factor, hi this case the adjustment factor would be proportional to the component of the motion vector aligned with the selected direction. The video processor would then adjust one or more pixel characteristics in accordance with the adjustment factor wherein the pixel characteristics may be color, brightness, saturation, contrast or combinations thereof. Alternatively, the processor 15 could adjust the pixel equivalent to one pixel characteristic value, such as one color, for all the vectors over a certain magnitude and aligned to within a selected number of degrees with the selected direction.

Instead of adjusting the pixel characteristic or characteristics in accordance with velocity, the characteristics or characteristic can be adjusted in accordance with acceleration. Figure 4 illustrates the process carried out by the video processor 15 to adjust pixel characteristics in accordance with acceleration. As shown in Figure 4, the processor 15 first generates a dense motion vector field for each pair of adjacent frames to represent the motion of image elements. The video processor then computes the acceleration of the image elements from frame to frame. To make this computation the corresponding vectors in successive dense motion vector fields have to be identified. Successive dense motion vector fields are the fields between successive adjacent overlapping pairs of video frames wherein each pair includes as its first frame the second frame of the preceding pair of frames. A vector in a successive dense motion vector field will correspond to a vector in the preceding dense motion vector field if the vector in the second field originates approximately at the termination point of a vector in the preceding field. To compute acceleration in a sealer form, the video processor subtracts the magnitude of each vector in each preceding field from the magnitude of the corresponding vector in the succeeding field to obtain a sealer measure of the acceleration, which is the change in speed from frame and to frame. Negative values will represent sealer deceleration. The sealer acceleration for each image element computed by the video processor is then compared with a standard to compute in an adjustment factor. Alternatively, a vector subtraction could be carried out to obtain an acceleration vector for each of the image elements. If vector acceleration is computed, the magnitude or direction of the acceleration vectors would be compared with standards to produce an adjustment factor which would then be used to adjust the pixel characteristics.

In the above described specific embodiments, the pixel characteristics are controlled in response to different motion criteria as represented by the dense motion vector fields. The different motion criteria as described above are velocity magnitude and direction and acceleration. In addition, to these criteria the control of pixel characteristics could be made responsive to velocity or acceleration duration, h addition the criteria controlling the change in pixel characteristics could be the turning of the image elements in the represented motion. The turning of the image element would be detectable by the acceleration vector being in a different direction than the corresponding velocity vector. In addition, to the above described pixel characteristics which could be controlled, the system could also control the focus of the depicted objects in accordance with the motion criteria. All of the above described criteria and pixel characteristics can be used in different and diverse combinations with each other. Other modifications include making the changed pixel characteristic persist over a plurality of frames or make the changed pixel characteristic occur prior to or be delayed following the velocity effect causing the change. To provide these anticipary or delayed effects, a pixel characteristic of pixels corresponding to vectors of a dense motion vector field is changed in frames preceding or following the frames between which the dense motion field represents image element motion, hi the system described above, the motion of the image elements is represented by vectors of dense motion vector fields and the pixel characteristics are controlled in accordance with these vectors. The same results can be achieved by using other forms of motion representation such as parametric representation including affine and perspective transformation These and other modifications may be made to the described specific embodiments of the invention without departing from the sprit and scope of the invention which is defined in the appended claims.

Claims

CLAIMSWhat is claimed is:

1. A method of controlling video in accordance with depicted motion comprising generating indications of motion of depicted image elements in a scene from frame to frame in said video and adjusting at least one pixel characteristic of the pixels in the frames of said video in accordance with said indications of motion of the corresponding image elements.

2. A method as recited in claim 1 wherein said at least one pixel characteristic is color.

3. A method as recited in claim 1 wherein said at least one pixel characteristic is brightness.

4. A method as recited in claim 1, wherein said at least one said pixel characteristic is saturation.

5. A method as recited in claim 1, wherein said at least one said pixel characteristic is contrast.

6. A method as recited in claim 1, wherein said at least one pixel characteristic is adjusted proportionally in accordance with the magnitudes of motion represented by said indications.

7. A method as recited in claim 1, wherein said at least one pixel characteristic is adjusted in accordance with the directions of motion represented by said indications.

8. A method as recited in claim 1, wherein the step of controlling at least one pixel characteristic includes detecting acceleration of image elements from said indications and adjusting said at least one pixel characteristic in the frames of said video in accordance with the acceleration of the corresponding image elements.

9. A method as recited in claim 1 wherein said indications comprise vectors of dense motion vector fields.

10. A video processing system for adjusting video in accordance with depicted motion comprising a storage device to store frames of a video comprising a motion picture, and a video processor programmed to process the frames in said storage device wherein said video processor generates indications of motion representing the motion of image elements from frame to frame of said video and to adjust at least one pixel characteristic of the pixels in the frames of said video in accordance with the indications of motion of the corresponding image elements.

11. A system as recited in claim 10 wherein said at least one pixel characteristic is color.

12. A system as recited in claim 10 wherein said at least one pixel characteristic is brightness.

13. A system as recited in claim 10 wherein said at least pixel characteristic is saturation.

14. A system as recited in claim 10 wherein said at least one pixel characteristic is contrast.

15. A system as recited in claim 10 wherein said video processor is programmed to adjust said at least one pixel characteristic in accordance with the magnitudes of motion represented by said indications.

16. A system as recited in claim 9 wherein said video processor is programmed to adjust said at least one pixel characteristic in accordance with the direction of the motion represented by said indications.

17. A system as recited in claim 10 wherein said video processor is programmed to detect from said indications, the acceleration of said image elements, and to adjust said at least one pixel characteristic of the pixels in said video frames in accordance with the acceleration of the corresponding image elements.

18. A system as recited in claim 10 wherein said indicators comprise vectors of dense motion vector fields.