GB2264020A - Fast/slow motion video signal processing - Google Patents

Abstract

A method of processing an input digital video signal enables display of the video image in fast or slow motion at a predetermined field or frame display rate. Output fields or frames are produced from the input fields or frames, at least some of the output fields or frames being produced by motion compensated temporal interpolation. The ratio of the number of output fields or frames to input fields or frames corresponding to a given portion of the image is set in dependence upon the desired increase or decrease in the motion rate of the image when displayed at the predetermined display rate. Interlaced fields are first convened to progressive scan, Fig 4b, after which the appropriate number of intermediate fields are interpolated, Fig 4c, and finally the interpolated fields are read out at the same rate as the input signal. Preferably all output fields are derived by interpolation (Fig 5(c)). <IMAGE>

Description

DIGITAL VIDEO SIGNAL PROCESSING The present invention relates to processing of digital video signals.

It is a common practice in the film industry to acquire film pictures at a variety of different rates (frames/s) even though the film material may be displayed at a fixed display rate, eg 24Hz (24 frames/s). For example, sports such as football may be filmed at a rate slightly higher than normal, for example 28 to 30Hz, to provide for slow motion display of the film image when the film is shown at 24Hz. Miniature models may be filmed at 7 to 13 times the normal rate so that details are clearly visible at 24Hz. Films may be shot at a lower rate than usual, for example 12 to 15Hz, in order to produce comic effects when displayed at 24Hz such as simulation of old movies.

While it is a simple matter to vary the acquisition and display rates of film cameras, the techniques used in film cannot be used in video systems since the image acquisition and display rates are both fixed at standards of either 60 fields/s or 50 fields/s. Display of video images in fast or slow motion is usually achieved by operating a VTR in "stunt mode". In this mode, the VTR runs at a non-standard speed, and the temporally nearest field to that required will be displayed. This process is illustrated in Figure 1 of the accompanying drawings for the case where a slow motion display is to be produced, the motion rate in the displayed image being one quarter that of the recorded image.

The top line of Figure 1 shows schematically a series of odd and even fields, "0" and "1" respectively, of a recorded video signal acquired at 60 fields/s. With the VTR running at one quarter normal replay speed, the raw output of the VTR, as shown by the middle line of Figure 1, consists of a series of 60 fields/s fields formed by repeating each field of the recorded signal four times. To produce the final output of the VTR, the field polarity of certain fields of the raw output must be reversed, in each case by linear vertical interpolation between pixels of the corresponding raw output field, to produce the required alternating field polarity. The final output of the VTR is shown by the bottom line of Figure 1, the interpolated fields being indicated by broken lines.

In the above system, temporally coincident portions of the source image are repeated over a four-field period, so that when the final output of the VTR is displayed, the result is an image moving in jerky slow motion. In addition, vertical resolution is impaired in the displayed image since four in every eight fields of the final output are produced by linear interpolation from raw output fields of the opposite polarity.

Where the motion rate in the displayed image is related to that in the recorded image by a non-integral factor, additional motion judder artifacts will be exhibited. For example, if the motion rate in the displayed image is to be two thirds that in the recorded image, then in each successive group of three fields of the final VTR output two adjacent fields will correspond to one field of the original video signal and the next output field to the following field of the original signal. The displayed image will therefore change irregularly with time. Further, in a series of six output fields, one group of three consecutive fields will be derived directly from the original fields, and another group of three consecutive fields will be polarity reversed and thus suffer from impaired vertical resolution.This resolution switching cycle at 10Hz or 50/6Hz may be noticeable in the displayed image.

The present invention is based in part on the realisation that, in order to provide smooth motion portrayal when displaying a video image in fast or slow motion, each field of the displayed image should ideally be temporally offset from the previous one, ie immediately successive output fields should correspond to temporally successive portions of the source image.

A technique for format conversion of digital video signals is known in which output fields or frames are generated at new temporal positions with respect to the input fields or frames. According to this technique, output fields/frames are created by "motion compensated temporal interpolation" such that the image data in these output fields/frames is temporally offset with respect to the image data in the input fields/Prames. Format conversion using motion compensated temporal interpolation is described in detail in UK patent applications nos GB2231228A and 9024836.0, the contents of which are incorporated herein by reference. GB2231228A relates particularly to video to film conversion in which an input 60 fields/s 2::1 interlace scan format video signal is processed to produce a 211 frames/s progressive scan format video signal which is used to drive an electron beam recorder which writes the video image to film. UK patent application no 902836.0 discloses use of motion compensated interpolation to perform format conversions other than the specific example described in GB2231228A. In all these processes, however, the object is to allow an image recorded in one format to be identically reproduced in a different format with no alteration of the rate of motion in the image when displayed in the different format. Thus, the ratio of the number of output frames/fields to input frames/fields corresponding to a given portion of the image is set in dependence upon the format of the desired output signal as compared with that of the input signal.

Output fields/frames are created at the appropriate temporal positions to allow reproduction of the image with the same motion rate in the new format.

According to one aspect of the present invention there is provided a method of processing an input digital video signal to enable display of the video image in fast or slow motion at a predetermined field or frame display rate, wherein output fields or frames are produced from the input fields or frames, at least some of the output fields or frames being produced by motion compensated temporal interpolation, the ratio of the number of output fields or frames to input fields or frames corresponding to a given portion of the image being set in dependence upon the desired increase or decrease in the motion rate of the image when displayed at the predetermined display rate. Thus, output fields/frames are created at new temporal positions with respect to the input fields/frames such that, for a given portion of the image, the temporal offsets of the image data in the output fields/frames corresponding to that portion with respect to the input fields/frames corresponding to that portion depend upon the desired change in the motion rate of the image when displayed.

It will be appreciated that the invention may be carried out in conjunction with a format conversion process such that the output signal has a different format to the input signal. In this case, the ratio of the number of output fields/frames to input fields/frames corresponding to a given portion of the image will depend on the desired format change as well as the desired change in the motion rate of the image when displayed in the new format. More usually, however, the input and output signals will be of the same format in which case the said predetermined display rate will be equal to the field/frame rate of the input signal. In this case, the ratio of the number of output fields/frames to input fields/frames corresponding to a given portion of the image may be set solely in dependence upon the desired change in the motion rate of the image.In particular, in order to multiply the motion rate by a factor of n, one output field/frame may be produced for every n input fields/frames. For example, where the input and output signals are both 50Hz 2:1 interlace format or 60Hz 2:1 interlace format, to produce a slow motion display in which the motion rate is one quarter that of the recorded image, four output fields are produced for each input field. To increase the motion rate in the displayed image by a factor of 2 as compared with the recorded image, one output field is produced for every two input fields.

According to another aspect of the invention there is provided a method of processing an input digital video signal representing a series of input fields or frames to produce an output digital video signal representing a series of output fields or frames with a different image motion rate to that of the input signal, wherein at least some of the output fields or frames are produced by motion compensated temporal interpolation to simulate image acquisition at temporal sites offset from the temporal acquisition sites of the input fields or frames in dependence upon the desired increase or decrease in the image motion rate, and wherein the temporal sites of the output fields or frames are equally spaced.As previously mentioned, if the ratio of the image motion rate of the output signal to that of the input signal is n, one output field or frame may be produced for every n input fields or frames.

Where the input signal has a field format, for the purpose of the subsequent processing, and for improved vertical resolution in the output fields/frames, it is preferred that a series of progressive scan format frames are produced from the input fields, and each of the interpolated output fields/frames are temporally interpolated between a corresponding pair of the progressive scan frames. The process of progressive scan conversion is described in detail in UK patent applications nos GB2231228A and 9024836.0 referred to above. Briefly, however, the process involves producing progressive scan frames, at the same rate of the input fields, each either from one or from three of the input fields.

The appropriate temporal positions for some of the output frames (or fields) may correspond precisely to the positions of input frames (or fields of the same polarity), so that each of these input frames/fields may be used directly as an output frame/field, other output frames/fields being produced by motion compensated temporal interpolation. However, when an output frame or field is formed by motion compensated interpolation, any noise associated with the input frames or fields from which it is formed is substantially reduced.

Thus, noise level fluctuation may result if some output frames/fields are formed by motion compensated interpolation and others directly from input frames/fields. According to the temporal positions of the output frames/fields with respect to the input frames/fields may be offset appropriately in dependence upon the ratio of output frames/fields to input frames/fields corresponding to a given portion of the image to ensure that there is no coincidence of input and output frame/field position. Thus, substantially all output frames/fields may be produced by motion compensated temporal interpolation.

Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings in which: Figure 1 illustrates schematically the known process of a VTR operating in stunt mode to provide a slow motion display; Figure 2 is a block diagram of processing apparatus embodying the invention; Figure 3 is a more detailed block diagram of part of the apparatus of Figure 2; Figure 4 illustrates schematically a processing operation carried out by the apparatus of Figures 2 and 3; Figure 5 illustrates the processing operation of Figure 4 modified to reduce noise fluctuation; Figure 6 illustrates a further processing operation carried out by the apparatus of Figures 2 and 3; and Figure 7 illustrates the processing operation of Figure 6 modified to reduce noise fluctuation.

The apparatus of Figure 2 comprises a first high definition digital VTR 1, a first silicon frame recorder 2, a motion rate converter 3, a second silicon frame recorder 4, and a second high definition VTR 5 connected as shown. A monitor 6 is associated with the VTR 5. The elements 1 to 5 are under the control of a system controller 7.

In operation, a video signal recorded on tape is reproduced by the VTR 1 and applied via the frame recorder 2 to the input of the motion rate converter 3. The motion rate converter 3 processes the input video signal using motion compensated temporal interpolation to generate an output signal which is supplied to the frame recorder 4 and then recorded by the VTR 5. The motion rate conversion process is such that when the output signal is reproduced by the VTR 5, the video image is displayed in fast or slow motion as compared with the originally recorded image.

The operation of the motion rate converter 3 in this example is such that it produces an output to the frame recorder 4 at one tenth normal video rate, so that each output field is repeated ten times and only one in ten output fields is temporarily stored by the frame recorder 4. When the frame recorder 4 is full, the stored frames are output and recorded by the VTR 5 operating in burst mode. The frame recorder 2 is required to allow the input signal to be supplied to the motion rate converter 3 at a variety of rates depending on the particular motion rate conversion to be carried out. The first VTR 1 is operated in burst mode and its output fields are temporarily stored in the first frame recorder 2. The stored frames are output from the frame recorder 2 to the motion rate converter 3 under control of the system controller 7 at the rate required by the motion rate converter 3. The results of the processing operation may be periodically evaluated by the system controller 7 and on the monitor 6.

Figure 3 is a block diagram of the motion rate converter 3 shown in Figure 2. The motion rate converter 3 comprises an input 10 which receives the signal from the frame recorder 2. The input 10 is connected to a progressive scan converter 11 for producing progressive scan frames from input fields in the usual case where the fields of an input frame are temporally offset. Otherwise, the progressive scan converter 11 may be bypassed or operated in previous field replacement mode. In either case, progressive scan frames are supplied in bursts to a time-base corrector 12 where they are temporarily stored so that four particular progressive scan frames are available at the same time.

These frames are supplied by the time-base corrector 12 to a motion processor generally indicated at 13.

The motion processor 13 comprises a direct block matcher 14 which compares blocks of pixels in a pair of the progressive scan frames supplied by the time-base corrector 12 and produces correlation surfaces. These correlation surfaces are analysed by a motion vector estimator 15 which derives and supplies motion vectors to a motion vector reducer 16 in which additional motion vectors are assigned to each block of pixels. The motion vectors are then supplied to a motion vector selector 17 which selects the best motion vector for each output pixel using the vectors supplied by the motion vector estimator for the two progressive scan frames of the original pair, and the immediately preceding anci immediately succeeding progressive scan frames supplied by the time-base corrector 12.Any irregularity in the selection of the motion vectors by the motion vector selector 17 is removed by a motion vector post-processor 18 from which the processed motion vectors are supplied to an interpolator 19.

The interpolator 19 generates the pixels of each output field by interpolation between the two progressive scan frames of the original pair from the time-base corrector 12 which are supplied to the interpolator by a system delay compensator 20 after a time which takes into account the delay introduced by the operation of the motion processor 13. For each output pixel, the interpolator 19 uses the motion vector supplied by the motion processor 13, and the correct temporal position along the motion vector for output pixels in that field as determined by the system controller 15, to locate the appropriate pixel patches in the pair of progressive scan frames from which the output pixel is to be formed. The value of each output pixel is determined by combining the appropriately filtered and weighted values of the pixel patches located.

The correct temporal offset of the output fields with respect to the progressive scan frame pairs depends on the ratio of output fields to input fields corresponding to a given portion of the image, and hence upon the desired motion rate conversion.

The operation of the progressive scan converter 11, motion processor 13 and interpolator 19 is described in detail in UK patent applications no GB2231228A and 9024836.0. Briefly, however, the operation of the motion processor 13 in the present case is as follows.

The contents of blocks of pixels in the first progressive scan frame of a pair supplied by the time-base corrector 12 are compared with the content of the following frame of the pair and correlation surfaces are produced which represent the difference in the contents so compared between the frames. These correlation surfaces are analysed by the motion vector estimator 15 which derives motion vectors for the respective blocks representing the motion of the content of each block between the two frames. The derived motion vectors are supplied to the motion vector reducer 16 which assigns additional motion vectors to each block, selected from the zero, that is stationary, motion vector and those derived for other blocks, until, for example, five unique motion vectors are associated with each block.The motion vectors associated with the blocks are then passed to the motion vector selector 17 for selection of motion vectors to be associated with respective output pixels to be produced. The pixel positions for the output field to be produced lie temporally somewhere between the original pair of input frames. The motion vector selector 17 performs a two-stage selection process. The first stage involves testing the motion vectors supplied for both of the original pair of frames against the immediately preceding and immediately succeeding progressive scan frames. The second stage involves testing using only the motion vectors for the two frames of the original pair to select the appropriate output pixel motion vectors. These are then supplied to the interpolator 19 as previously described.

The operation of the apparatus of Figure 3 will now be described for a particular motion rate conversion process with reference to Figure 4. Figure 4 illustrates schematically a processing operation performed on a 60 fields/s 2:1 interlace format video signal to enable display at 60Hz of the video image in slow motion, the motion rate being one quarter that of the originally recorded image. Line a of Figure 4 shows a series of odd and even fields ("0" and "1" respectively) of the 60Hz input signal from the VTR 1. Line b shows a series of 60 frames/s progressive scan frames formed by the progressive scan converter from the input fields, such that the image data in each progressive scan frame is temporally coincident with that of a corresponding input field.The progressive scan frames are then supplied as previously described to the time-base corrector 12 where they are temporarily stored to allow multiple output fields to be generated from each pair of frames. Since the motion rate is to be decreased by a factor of four, and the output signal to be displayed at 60Hz, the ratio of the number of output fields to input fields corresponding to a given portion of the image is to be four. Thus, four output fields are to be produced from each pair of progressive scan frames.

Line c of Figure 4 shows the output fields generated by the interpolator 29, the temporal position of each output field with respect to the corresponding progressive scan frame pair indicating the degree to which the image data in that field is temporally offset with respect to the input fields. Thus, considering the four output fields produced from the first pair of progressive scan frames, the first output field is temporally aligned with the first progressive scan frame, and hence the first input field. The temporal offset utilised by the motion vector selector 17 and interpolator 19 for the pixels of this field is therefore zero. That is to say, the first output field is formed directly from the first progressive scan frame and hence corresponds directly to the first input field.The second output field of the four is temporally interpolated one quarter of the way between the respective pair of progressive scan frames. The image data in this field is thus temporally offset by 1/240s from the first progressive scan frame. Similarly, the third and fourth output fields are temporally interpolated one half and three quarters of the way respectively between the respective pair of progressive scan frames so that the image data in these fields is temporally offset from that of the first frame by 1/120s and 1/80s respectively from the first frame.

This process is repeated for each subsequent progressive scan frame pair as shown in the figure. The temporally interpolated output fields are thus simulations of the fields which would have been acquired at temporal sites correspondingly offset from the temporal acquisition sites of the input fields. It will be seen that a given portion of the image is represented by four times as many output fields as input fields, so when the output signal is displayed at the input field rate of 60Hz as indicated in line d of the figure, the motion rate in the image will be decreased by a factor of four.

It can be seen from Figure 4 that one in four output fields is formed from a single progressive scan frame without motion compensated temporal interpolation. As previously described, this may give rise to noise level modulation in the output signal. This noise modulation may be eliminated if all output fields are formed by motion compensated temporal interpolation. This can be achieved as shown in Figure 5 by shifting the temporal positions of all output fields by one eighth of the input field period so that the four output fields produced from each progressive scan frame pair are interpolated 1/8, 3/8, 5/8, and 7/8 of the way respectively between the two progressive scan frames.

In this way, temporal coincidence of output field and input field image data is avoided as shown in line c of Figure 5.

Figures 6 and 7 are diagrams similar to Figures 4 and 5 for processing operations in which the motion rate in the displayed image is to be increased by a factor of 3/2. As before, progressive scan frames are produced at the same rate as the input fields, and output fields are produced by interpolation between pairs of progressive scan frames. Here, however, the ratio of output fields to input fields corresponding to a given portion of the image is 2/3. Figure 6 illustrates the case where the first output field is temporally aligned with the first progressive scan frame. The second output field is interpolated half way between the second and third progressive scan frames, the third output field being formed directly from the fourth progressive scan frame, and so on.In Figure 7, the output field to input field ratio is the same as in Figure 6, but the output fields are temporally offset with respect to those of Figure 6 by one quarter of the input field period to avoid noise level fluctuation in the output signal. In both cases, however, when the output signal, as shown in line c of each figure, is displayed at the input field rate of 60Hz as shown in line d, the motion rate in the displayed image will be 3/2 times that of the originally recorded image.

It will be appreciated that the processes described above enable display of video images with altered motion rates with smooth motion portrayal and improved vertical resolution as compared with operation of a VTR in stunt mode.

As previously mentioned, format conversion of the input signal may be carried out during the motion compensated interpolation process as well as motion rate conversion. In such cases, the ratio of output frames/fields to input frames/fields corresponding to a given portion of the image will depend upon the required format conversion as well as the desired alteration in the motion rate of the image.

Claims (10)

1. A method of processing an input digital video signal to enable display of the video image in fast or slow motion at a predetermined field or frame display rate, wherein output fields or frames are produced from the input fields or frames, at least some of the output fields or frames being produced by motion compensated temporal interpolation, the ratio of the number of output fields or frames to input fields or frames corresponding to a given portion of the image being set in dependence upon the desired increase or decrease in the motion rate of the image when displayed at the predetermined display rate.

2. A method as claimed in claim 1 wherein the said predetermined field or frame display rate equals the field or frame rate of the input signal, and, to multiply the motion rate by a factor of n, 1 output field or frame is produced for every n input fields or frames.

3. A method as claimed in claim 1 or claim 2, including displaying the processed signal at the said predetermined display rate.

4. A method of processing an input digital video signal representing a series of input fields or frames to produce an output digital video signal representing a series of output fields or frames with a different image motion rate to that of the input signal, wherein at least some of the output fields or frames are produced by motion compensated temporal interpolation to simulate image acquisition at temporal sites offset from the temporal acquisition sites of the input fields or frames in dependence upon the desired increase or decrease in the image motion rate, and wherein the temporal sites of the output fields or frames are equally spaced.

5. A method as claimed in claim 4, wherein the ratio of the image motion rate of the output signal to that of the input signal is n, one output field or frame being produced for every n input fields or frames.

6. A method as claimed in any preceding claim, wherein the input signal has a field format, and including the step of producing a series of progressive scan format frames from the input fields, wherein those of the output fields or frames which are temporally interpolated are so interpolated between a corresponding pair of the progressive scan frames.

7. A method as claimed in claim 6, wherein substantially all output fields or frames are produced by motion compensated temporal interpolation between a pair of the progressive scan frames.

8. A method of processing a digital video signal substantially as hereinbefore described with reference to Figures 2 to 7 of the accompanying drawings.

9. Apparatus adapted to perform the method of any one of the preceding claims.

10. Apparatus substantially as hereinbefore described with reference to Figures 2 to 7 of the accompanying drawings.