GB2373946A

GB2373946A - Method of synthesizing motion blur in a video sequence

Info

Publication number: GB2373946A
Application number: GB0107887A
Authority: GB
Inventors: Andrew David Raine Cotton
Original assignee: Snell and Wilcox Ltd
Current assignee: Snell Advanced Media Ltd
Priority date: 2001-03-29
Filing date: 2001-03-29
Publication date: 2002-10-02
Also published as: GB0107887D0

Abstract

A method of synthesizing motion blur in a video sequence in which information from a corresponding video sequence which is at a higher frame-rate and a lower spatial resolution is used ti calculate a blur signal. The blur signal may be a linear combination of successive images in the corresponding video sequence and the combination may comprise a comparison of weighted averages of successive images. The blur signal may be up-converted to a higher spatial resolution and combined with the lower frame-rate video sequence. An additional high spatial frequency component of the blur signal may be calculated from the low frame-rate sequence using motion compensation.

Description

DIGITAL SIGNAL PROCESSING

This invention is directed to digital video processing techniques, and in particular to methods of exploiting such techniques to generate special effects.

Digital and computer assisted film production is becoming more and more widespread within the movie and television industries. This is expected to bring important benefits, but care must taken to ensure that all useful features of conventional production are maintained and, preferably, provided in an enhanced and more flexible form.

It is important, for example, that any digital image capture technique is capable of avoiding the motion judder at relatively lower frame rates. that conventional film production avoids by maintaining the film shutter open long enough to introduce motion blur. It would be very useful if a tool were available to introduce the motion blur normally associated with a long camera shutter time, artificially in the post-production process, once the material has been shot.

It is also desirable that the technique employed to avoid motion judder will cope with slow motion effects and film speed changes used in conventional film production for artistic effect.

When a conventional film camera is used to shoot a scene that is to be replayed in slow motion, the film speed at the time of capture is increased so that playback of the film material at the standard 24 frames per second (fps) will result in the desired slow motion effect. Thus, if a scene is to be played back at 1/3 speed, the film at the time of shooting is passed through the camera at 3 x 24 = 72 fps.

It is common practice to run the film camera with a 1800 shutter. That is to say that the rotating shutter is open for 1800 of its 3600 rotation, in effect the shutter is open for 50% of the frame period. Thus if the film were running at 24 fps, the shutter speed would typically be 1/48 of a second. A longer exposure time would lead to undesirable blur of moving objects, whilst a shorter exposure time would lead to excessive motion judder. If the film passes through the camera shutter at a higher than normal speed, the 50: 50 shutter timing remains, resulting in the appropriate degree of motion blur for replay at 24 fps.

It is an object of the invention to provide a system capable of improved motion effects in electronically captured image material.

Accordingly, the invention consists in one aspect in a method of synthesising motion blur in a video sequence, characterised in that information from a corresponding video sequence which is at a higher frame-rate and a lower spatial resolution than said video sequence is used to calculate a blur signal.

Advantageously, the blur signal is a linear combination of successive images in said corresponding video sequence.

Suitably, the linear combination comprises the comparison of a weighted average of successive images with one of those images.

The invention will now be described by way of example with reference to the accompanying drawings, in which: Figure 1 is a diagram illustrating a prior art method of image capture; Figure 2 is a diagram illustrating an embodiment of the invention; Figure 3 is a diagram illustrating a method of image capture according to an embodiment of the invention; Figure 4 is a diagram illustrating another embodiment of the invention; and Figure 5 is a diagram illustrating a further embodiment of the invention.

Referring initially to Figure 1, PCT/GB00/02411 describes an electronic camera which provides two digital outputs. Each output represents the same scene in front of the camera, but is in a different output format. The primary output is at high spatial resolution but low temporal resolution, typically 24 fps.

However, an additional output is also provided at a lower spatial resolution but a higher temporal resolution or frame rate.

As shown in Figure 2, the scheme described herein similarly utilises output from a digital camera based on one or more light sensors capable of producing a high resolution output at a high frame rate. The camera provides two outputs: the primary output is a full spatial resolution image but at a relatively low temporal resolution ; the secondary output is at a low spatial resolution but at a high frame rate.

One important feature of the proposed camera is that it operates without a mechanical or electronic shutter. Each image therefore captures the light that falls on the sensor in the period between sampling instants. Thus image S2 captures the light failing on the sensor in period t2 to t3, S3 represents the period t3 to t4, and so on. As a result, both outputs from the camera have a very short shutter time compared with a standard mechanical film camera, and so the images have little motion blur. In this example the effective shutter time is 1/ (24x8) = 1/192 of a second. Whilst this is good for motion estimation on the lower resolution image and for exploiting those motion vectors to re-create high resolution images at intermediate points in time, it is not so good when the high resolution images are viewed directly as they will suffer from apparent motion "judder". In order to reduce the judder it is necessary to put motion blur back in to the high resolution images.

A key feature of motion blur is that it is a smearing of the picture, and contains very few high spatial frequencies. As a result it is possible to calculate a separate motion blur"component"from the low spatial resolution camera output.

The blur component can then be up-converted to the higher resolution image size, and the two signals added together, effectively increasing the shutter time of the high resolution image.

Referring to Figure 2, in order to achieve an effective shutter time of 1/48 of a second for image hO, it is necessary to simulate the effect of capturing all of the light that falls on the sensor in the period tO to t4. Image hO can be considered to already have captured light from tO to t1, thus the additional blur signal to be up-converted and added in to hO must represent the additional light signal from t1 to t4. This can easily be calculated by adding images together on a

pixel by pixel basis, and then subtracting 10, the low resolution version of hO, thus :

Additional Blur Component = (10+11+12+13)/4-10

In one embodiment, both output signals are derived from the same light sensor. As shown in Figure 3, images hO and h1 are copies of captured images SO and S8. Lower resolution images 10 to 18 are derived by spatial downconversion of images SO to S8. In this example the high resolution images might be output at the standard film camera frame rate of 24 fps and, as drawn, the low resolution images 10 to 18 at eight times that rate. It should be noted, however, that any combination of frame rates might be chosen.

In an aspect of the invention, the high frame rate output may be used to accurately measure the movement within any scene to generate a set of motion vectors. Figure 4 shows how these motion vectors can then be applied to the high resolution output to artificially generate high resolution images at any temporal position between the existing lower rate frames. Such a technique can be used to artificially generate slow motion sequences from high resolution images shot at"normal"film speed.

In an alternative, the same accurate motion vector field can be used to perform"standards conversion"for playback at other frame rates.

In order to create a half speed slow motion sequence from the above data, as shown in Figure 4, motion estimation and motion compensation techniques can be used to synthesise a high resolution image, hO. 5, at time t4. In this instance, in order to prevent motion judder an effective shutter speed of 1/96 second is required i. e. tO to t2 and t4 to t6 etc. An appropriate compensation

signal for hO is simply :

Blur compensationho = (10 +11)/2-10

And similarly for the synthetic frame, hO. 5, at t4 :

Blur compensationhO. 5' = (14+15)/2-14

The degree of motion blur, or the effective shutter time, is simply adjusted by changing the number of low resolution images that are added together, and dividing by the appropriate factor.

However, it should be noted that in the case of hO. 5 the motion compensated low frequency component of hO. 5 may not match 14 which could give rise to a blur mismatch. In this case, alternative methods might include : 1. Adding blur to the existing image hO derived from 10 and 11 prior to motion compensation, and then motion compensating the combined image to obtain hO. 5.

2. Deriving the low frequency component of hO. 5, complete with blur, directly from 14 and 15 and using motion-compensation of hO to derive only the high frequency component of hO. 5.

Of course it is also possible to apply motion compensation to the low frequency images to derive low frequency and blur components for synthesised high resolution images that are not time coincident with a pre-existing low frequency image. Such a technique may be required for frame-rate conversion, where the input frame rate and the output frame rate are not a simple multiple of one another.

In a further embodiment, the effects of"non-integral"shutter speeds are simulated by multiplying the contributions from the lower resolution images by a scalar between 0 and 1, rather than simply switching them on and off.

In the example above a shutter speed of 3.5/192 (or 1/55) of a second can

be achieved thus :

Additional Blur Component = (10+11 +12+0. 5xt3)/3. 5-10

In a further embodiment, the technique described above that calculates the motion blur on a low resolution version of the image is combined with motion vector based schemes to improved performance. Figure 5 shows how the low frequency motion blur can be derived from the robust low resolution images, whilst the high frequency component can be derived from motion vector based methods. Should the motion estimation fail, only the high frequency component is affected and the low frequency component will remain intact. It should be noted that in order to ensure a flat frequency response in the blurred image hO 48, the high-pass filter will typically have a response equal to one minus the response of the combined down-filtering, sub-sampling and up-conversion process.

It will be appreciated by those skilled in the art that the invention has been described by way of example only, and a wide variety of alternative approaches may be adopted.

Claims

CLAIMS 1. A method of synthesising motion blur in a video sequence, characterised in that information from a corresponding video sequence which is at a higher frame-rate and a lower spatial resolution than said video sequence is used to calculate a blur signal.
2. A method according to Claim 1, in which the blur signal is a linear combination of successive images in said corresponding video sequence.
3. A method according to Claim 2, in which the linear combination comprises the comparison of a weighted average of successive images with one of those images.
4. A method according to Claim 3, in which the weights are calculated to correspond to a specified desired camera integration period.
5. A method according to Claims 1-4, in which the blur signal is upconverted to a higher spatial resolution and combined with the lower frame-rate video sequence at the higher spatial resolution.
6. A method according to Claim 5, in which motion-compensated frame-rate conversion is additionally applied to either or both of the low frame-rate sequence and the corresponding high frame-rate sequence.
7. A method according to any of the preceding claims, in which an additional high spatial frequency component of the blur signal is calculated from the low frame-rate sequence using motion compensation.