GB2313514A - Motion compensated video signal processing - Google Patents

Abstract

Motion compensated video signal processing apparatus comprises means for generating a plurality of motion vectors to represent image motion between a pair of input images of an input video signal; and means for testing a group of the motion vectors, to select a motion vector for use in interpolation of an output pixel of an output image, the means for testing comprising: ```means 672 for detecting the degree of correlation between test blocks of the input images pointed to by a motion vector under test; ```means 1020 for generating an analysis value indicative of the image detail (spatial frequency content) of the test blocks; and ```means for selecting a motion vector from the group of motion vectors, in dependence on the degree of correlation between the test blocks pointed to by the motion vectors and the analysis value for the test blocks pointed to by the motion vectors.

Description

MOTION COMPENSATED VIDEO SIGNAL PROCESSING This invention relates to motion compensated video signal processing.

Motion compensated video signal processing is used in applications such as television standards conversion, film standards conversion and conversion between video and film standards.

In a motion compensated television standards converter, such as the converter described in the British Published Patent Application number GB-A-2 231 749, pairs of successive input images are processed to generate sets of motion vectors representing image motion between the pair of input images. The processing is carried out on discrete blocks of the images, so that each motion vector represents the inter-image motion of the contents of a respective block.

Each set of motion vectors is then supplied to a motion vector reducer which derives a subset of the set of motion vectors for each block. The subset, which includes the zero motion vector, is then passed to a motion vector selector which assigns one of the subset of motion vectors to each picture element (pixel) in each block of the image. The selected motion vector for each pixel is supplied to a motion compensated interpolator; the interpolator operates to interpolate successive output images, taking into account the motion between the input images.

Vector selection is used to select a motion vector, for use in interpolating an output pixel from two input images, from the subset of the motion vectors (e.g. four motion vectors) supplied by the vector reducer. The vector selection process involves detecting the degree of correlation between test blocks of the two input images pointed to by a motion vector under test. The motion vector having the greatest degree of correlation between the test blocks is selected for use in interpolation of the output pixel.

A problem can occur when vector selection is performed on input images which have a large noise content. In this case, spurious apparent correlation between areas of noise in the input images can lead to an incorrect motion vector being selected. A motion compensated video signal processing apparatus operating under these conditions is likely to fail dramatically when a certain level of image noise is reached, and then to have a worse performance than a non-motion-compensated apparatus.

This invention provides a motion compensated video signal processing apparatus comprising: means for generating a plurality of motion vectors to represent image motion between a pair of input images of an input video signal; and means for testing a group of the motion vectors, including a zero motion vector indicative of zero image motion, to select a motion vector from the group for use in interpolation of an output pixel of an output image, the means for testing comprising: means for detecting a correlation value indicative of the degree of correlation between test blocks of the input images pointed to by a motion vector under test; means for weighting the correlation values, the weighting being such that the degree of correlation indicated by the correlation value for the zero motion vector is increased with respect to the degrees of correlation indicated by the correlation values for other motion vectors in the group; and means for selecting a motion vector from the group of motion vectors, the selected motion vector having a weighted correlation value indicative of the highest degree of correlation.

In a motion compensated video signal processing apparatus according to the invention, the result of the vector selection correlation test obtained for the zero motion vector is weighted with respect to the other motion vectors in the group. This means that, other things being equal, the zero motion vector is more likely to be selected during motion vector selection unless another one of the subset of motion vectors provides a significantly better correlation between the test blocks.

Since the vector selection process is biased towards selection of the zero motion vector, the motion compensated video signal processing apparatus is more likely to revert to a performance which is no worse than a non-motion-compensated apparatus when noisy input images prevent reliable motion vector selection.

Preferably the detecting means comprises: means for detecting the absolute luminance difference between pairs of pixels at corresponding positions in the test blocks; and means for summing the absolute luminance differences to generate the correlation value. In this case, it is preferred that the means for weighting comprises means for multiplying the correlation value for the zero motion vector by a predetermined constant value, the constant value being less than unity. In other words, the weighting operates to reduce the correlation value for the zero motion vector (to indicate a greater degree of correlation) with respect to those for the other motion vectors.

Preferably the means for weighting comprises means for multiplying the correlation values for one or more motion vectors in the group other the zero motion vector by respective multiplicative weighting values, the multiplicative weighting values being greater than unity.

As an alternative, additive (or subtractive) weighting could be employed. To this end, it is preferred that the means for weighting comprises means for subtracting a predetermined constant value from the correlation value for the zero motion vector.

Also, it is preferred that the means for weighting comprises means for adding respective additive weighting values to the correlation values for one or more motion vectors in the group other the zero motion vector.

In further preferred embodiments, the means for weighting is operable to weight the correlation values such that the degree of correlation indicated by the correlation value for each motion vector in the group is decreased by an amount proportional to the magnitude of that motion vector. This means that in order to be selected, a large motion vector would have to result in a better correlation between the test blocks than a smaller motion vector, all other factors being equal.

Viewed from a second aspect this invention provides a method of motion compensated video signal processing, the method comprising the steps of: generating a plurality of motion vectors to represent image motion between a pair of input images of an input video signal; and testing a group of the motion vectors, including a zero motion vector indicative of zero image motion, to select a motion vector from the group for use in interpolation of an output pixel of an output image, the step of testing comprising: detecting a correlation value indicative of the degree of correlation between test blocks of the input images pointed to by a motion vector under test; weighting the correlation values, the weighting being such that the degree of correlation indicated by the correlation value for the zero motion vector is increased with respect to the degrees of correlation indicated by the correlation values for other motion vectors in the group; and selecting a motion vector from the group of motion vectors, the selected motion vector having a weighted correlation value indicative of the highest degree of correlation.

When the vector selection process is applied to video material having finely detailed portions (which are smaller than one of the test blocks used in the correlation test) moving on a stationary background, an incorrect vector can be selected for interpolation of the moving portions. This is because the correlation between areas surrounding the moving portions in the two input images may be completely different, with this difference dominating the correlation test results. An example of this potential problem is illustrated in Figures la, ib, 2a and 2b.

Figures la, ib, 2a and 2b illustrate the vector selection process for an image of a moving car. Vector estimation will have identified at least the zero motion vector and a horizontal motion vector as representing the motion of the stationary image background and the moving car respectively. For areas of the body of the car, the correlation test performed during vector selection should select the "car" motion vector with little difficulty. However, for items such as the car radio antenna, the test blocks 10, 20 (Figures la and lb) pointed to by the "car" motion vector are overwhelmingly dominated by different parts of the image background. The correlation between these test blocks is poor. In contrast, the correlation between test blocks 30, 40 (Figures 2a and 2b) pointed to by the zero motion vector is high, since these test blocks represent identical parts of the image background, with only a slight difference in that one of the test blocks contains a portion of the car radio antenna.

Accordingly, since the correlation test result for the zero motion vector indicates a greater correlation than that for the "car" motion vector, the zero motion vector is incorrectly selected for use in interpolation of a portion of the car radio antenna.

Although the test blocks could be reduced in size to overcome this problem, this measure would increase the susceptibility of the vector selection process to noise in the input images.

Viewed from a third aspect this invention provides a motion compensated video signal processing apparatus comprising: means for generating a plurality of motion vectors to represent image motion between a pair of input images of an input video signal; means for testing a group of the motion vectors to select a motion vector from the group for use in interpolation of an output pixel of an output image; and a motion compensated interpolator for interpolating the output pixel from a pair of input images of the input video signal using the selected motion vector; the means for testing comprising: image enhancement means for enhancing the image content of at least test blocks of the input images pointed to by a motion vector under test; means for detecting the degree of correlation between the image-enhanced test blocks; and means for selecting a motion vector having the highest degree of correlation between the respective image-enhanced test blocks.

In accordance with this aspect of the present invention, image enhancement is employed to enhance at least the test blocks of the pair of input images. This can increase the visibility; or significance within the test blocks, of fine detail such as the car radio antenna described above. Because the fine detail is enhanced, it will have more influence on the results of the correlation test performed during vector selection.

This means that the vector selection process is more likely to select the correct motion vector for e.g. small, finely detailed objects moving against a stationary background.

The image enhancement may be performed before or after the test blocks are derived from the input images.

In one preferred embodiment the image enhancement means comprises a twodimensional high-pass spatial filter. However, although the use of high-pass filters can enhance fine detail within the test blocks, the filters can also increase the prominence of image noise, which may degrade the overall system performance.

Accordingly, in another preferred embodiment a vertical Sobel image filter and a horizontal Sobel image filter are employed (these filters perform high-pass filtering in one direction and low-pass filtering in another direction), with means for detecting an output by the vertical Sobel image filter or the horizontal Sobel image filter indicative of greater than a predetermined degree of detail surrounding one or more pixels in the test blocks; and means, responsive to the detecting means, for replacing the one or more pixels of the test blocks with pixels of a predetermined luminance.

It is preferred that the pixels of a predetermined luminance are white pixels.

In another embodiment the image enhancement means comprises: a vertical Sobel image filter; a horizontal Sobel image filter; second detecting means for detecting an output by the vertical Sobel image filter or the horizontal Sobel image filter indicative of less than a predetermined degree of detail surrounding one or more pixels in the test blocks; and means, responsive to the second detecting means, for replacing the one or more pixels of the test blocks with pixels of a second predetermined luminance.

In this case, it is preferred that the pixels of a second predetermined luminance are black pixels.

In a further preferred embodiment, the image enhancement means comprises: means for detecting an output by the vertical Sobel image filter or the horizontal Sobel image filter indicative of greater than a predetermined degree of detail surrounding one or more pixels in the test blocks; and means, responsive to the detecting means, for adding a predetermined luminance component to the luminance component of the one or more pixels of the test blocks.

Preferably this aspect of the invention is combined with the weighting of the correlation value of the zero motion vector. To this end, it is preferred that the group of motion vectors comprises a zero motion vector indicative of zero image motion, and that the apparatus comprises: means for detecting a correlation value indicative of the degree of correlation between the image-enhanced test blocks of the input images pointed to by a motion vector under test; and means for weighting the correlation values, the weighting being such that the degree of correlation indicated by the correlation value for the zero motion vector is increased with respect to the degrees of correlation indicated by the correlation values for other motion vectors in the group; and in which the means for selecting is operable to select a motion vector having a weighted correlation value indicative of the highest degree of correlation.

Preferably the detecting means comprises: means for detecting the absolute luminance difference between pairs of pixels at corresponding positions in the imageenhanced test blocks; and means for summing the absolute luminance differences, thereby generating a sum of absolute differences (SAD) value.

Viewed from a fourth aspect this invention provides a method of motion compensated video signal processing, the method comprising the steps of: generating a plurality of motion vectors to represent image motion between a pair of input images of an input video signal; testing a group of the motion vectors to select a motion vector from the group for use in interpolation of an output pixel of an output image; and interpolating the output pixel from a pair of input images of the input video signal using the selected motion vector; the step of testing comprising: image enhancing the image content of at least test blocks of the input images pointed to by a motion vector under test; detecting the degree of correlation between the imageenhanced test blocks; and selecting a motion vector having the highest degree of correlation between the respective image-enhanced test blocks.

Figure 3 illustrates a further problem which can occur when a detailed object is moving on a very plain background. In this case, the correlation test performed in vector selection can give a very good match between two areas of the plain background. In fact, the match obtained between the two plain areas can often appear to be a better match than that obtained between the two corresponding portions of the detailed object. Since the motion vector having the highest degree of correlation between the test blocks is selected for use in interpolation, the very good match between the plain areas can result in an incorrect vector being selected in preference to the correct vector representing the motion of the detailed object. The visible result of this error is a gap (in fact an area of background) appearing in the detailed object.

Viewed from a fifth aspect this invention provides a motion compensated video signal processing apparatus comprising: means for generating a plurality of motion vectors to represent image motion between a pair of input images of an input video signal; and means for testing a group of the motion vectors, to select a motion vector for use in interpolation of an output pixel of an output image, the means for testing comprising: means for detecting the degree of correlation between test blocks of the input images pointed to by a motion vector under test; means for generating an analysis value indicative of the image detail of the test blocks; and means for selecting a motion vector from the group of motion vectors, in dependence on the degree of correlation between the test blocks pointed to by the motion vectors and the analysis value for the test blocks pointed to by the motion vectors.

In this aspect of the invention an analysis value is generated to reflect the degree of detail in the test blocks and is then used to influence the vector selection process. In this way, greater weight can be attached to correlation detected between detailed test blocks than that detected between plain (non-detailed) test blocks. This can alleviate the problem of erroneous vector selection described above.

Many different ways of detecting the analysis value are possible. For example, the output of one or more Sobel filters may be employed. In one preferred embodiment the means for generating an analysis value comprises means for detecting the difference between the highest luminance component of pixels in a test block and the lowest luminance component of pixels in that test block.

In another preferred embodiment the means for generating an analysis value comprises means for summing the luminance differences between pairs of adjacent pixels within each test block. Preferably the means for summing comprises means for summing, for each pixel in a test block, the higher of the luminance difference between that pixel and a vertically adjacent pixel and the luminance difference between that pixel and a horizontally adjacent pixel.

The detail test result (the analysis value) may be used in various ways to influence the vector selection process. In one preferred embodiment the apparatus comprises means for preventing selection of a motion vector, for which the test blocks pointed to by that motion vector have an analysis value indicative of less than a predetermined degree of image detail.

Alternatively, a weighting could be applied to a test result indicative of correlation between the test blocks. In this case it is preferred that the apparatus comprises means for detecting a correlation value indicative of the degree of correlation between the test blocks; and means for weighting the correlation values in dependence on the analysis value, the weighting being such that the degree of correlation indicated by the correlation value for a motion vector is increased with increasing detected image detail; and in which the means for selecting is operable to select a motion vector having a weighted correlation value indicative of the highest degree of correlation.

In an advantageously simple embodiment the means for detecting the degree of correlation comprises: means for detecting the absolute luminance difference between pairs of pixels at corresponding positions in the test blocks; and means for summing the absolute luminance differences, thereby generating a sum of absolute differences (SAD) value. In this case, it is preferred that the means for weighting comprises means for multiplying the sum of absolute difference value by a weighting coefficient, the weighting coefficient decreasing with increasing detected image detail.

Image enhancement may also be employed with this aspect of the invention, by employing image enhancement means for enhancing the image content of at least the test blocks of the input images pointed to by a motion vector under test.

Also, it is preferred that the feature of zero vector weighting is employed. To this end, it is preferred that the group of motion vectors comprises a zero motion vector indicative of zero image motion, the apparatus comprises: means for detecting a correlation value indicative of the degree of correlation between the test blocks of the input images pointed to by a motion vector under test; and means for weighting the correlation values, the weighting being such that the degree of correlation indicated by the correlation value for the zero motion vector is increased with respect to the degrees of correlation indicated by the correlation values for other motion vectors in the group; and that the means for selecting is operable to select a motion vector having a weighted correlation value indicative of the highest degree of correlation.

In a preferred embodiment the pair of input images comprises two successive fields of an interlaced input video signal.

Preferably the output image comprises a field of an interlaced output video signal.

Viewed from a sixth aspect this invention provides a method of motion compensated video signal processing, the method comprising the steps of: generating a plurality of motion vectors to represent image motion between a pair of input images of an input video signal; and testing a group of the motion vectors, to select a motion vector for use in interpolation of an output pixel of an output image, the step of testing comprising: detecting the degree of correlation between test blocks of the input images pointed to by a motion vector under test; generating an analysis value indicative of the spatial frequency content of the test blocks; and selecting a motion vector from the group of motion vectors, in dependence on the degree of correlation between the test blocks pointed to by the motion vectors and the analysis value for the test blocks pointed to by the motion vectors.

Apparatus according to the invention is particularly advantageously employed in a television standards conversion apparatus.

An embodiment of the invention will now be described, by way of example only, with reference to the accompanying drawings, throughout which like parts are referred to by like references, and in which: Figures la and 1b illustrate the vector selection process for an image of a moving car; Figures 2a and 2b illustrate the vector selection process for an image of a moving car; Figure 3 illustrates the vector selection process for an image containing large plain areas; Figure 4 is a schematic block diagram of a motion compensated standards converter; Figure 5 schematically illustrates vertical subsampling of interlaced video fields; Figure 6 is a schematic diagram showing the comparison of test blocks during vector selection; Figure 7 is a schematic block diagram of a motion vector selector; Figure 8 is a schematic block diagram of a part of the motion vector selector of Figure 7, according to one embodiment of the invention; Figure 9 illustrates the coefficients of a two-dimensional high-pass spatial filter; Figure 10 illustrates the coefficients of another two-dimensional high-pass spatial filter; Figure 11 illustrates the coefficients of a vertical Sobel spatial filter; Figure 12 illustrates the coefficients of a horizontal Sobel spatial filter; Figure 13 illustrates an image enhancement circuit; Figures 14a and 14b illustrate the vector selection process for an image of a moving car, using image-enhanced test blocks; Figure 15 is a schematic block diagram of a part of the motion vector selector of Figure 7, according to a second embodiment of the invention; and Figure 16 is a schematic block diagram of a part of the motion vector selector of Figure 7, according to a third embodiment of the invention.

Figure 4 is a schematic block diagram of a motion compensated television standards conversion apparatus. The apparatus receives an input interlaced digital video signal 250 (e.g. an 1125/60 2:1 high definition video signal (HDVS)) and generates an output interlaced digital video signal 260 (e.g a 1250/50 2:1 signal).

The input video signal 250 is first supplied to an input buffer/packer 310. In the case of a conventional definition input signal, the input buffer/packer 310 formats the image data into a high definition (16:9 aspect ratio) format, padding with black pixels where necessary. For a HDVS input the input buffer/packer 310 merely provides buffering of the data.

The data are passed from the input buffer/packer 310 to a matrix circuit 320 in which (if necessary) the input video signal's colorimetry is converted to the colorimetry of the desired output signal, such as the standard "CCIR recommendation 601" (Y,Cr,Cb) colorimetry.

From the matrix circuit 320 the input video signal is passed to a time base changer and delay 330, and via a sub-sampler 370 to a subsampled time base changer and delay 380. The time base changer and delay 330 determines the temporal position of each field of the output video signal, and selects the two fields of the input video signal which are temporally closest to that output field for use in interpolating that output field. For each field of the output video signal, the two input fields selected by the time base changer are appropriately delayed before being supplied to an interpolator 340 in which that output field is interpolated. A control signal t, indicating the temporal position of each output field with respect to the two selected input fields, is supplied from the time base changer and delay 330 to the interpolator 340.

The subsampled time base changer and delay 380 operates in a similar manner, but using spatially subsampled video supplied by the subsampler 370. Pairs of fields, corresponding to the pairs selected by the time base changer 330, are selected by the subsampled time base changer and delay 380 from the subsampled video, to be used in the generation of motion vectors.

The time base changers 330 and 380 can operate according to synchronisation signals associated with the input video signal, the output video signal, or both. In the case in which only one synchronisation signal is supplied, the timing of fields of the other of the two video signals is generated deterministically within the time base changers 330, 380.

The pairs of fields of the subsampled input video signal selected by the subsampled time base changer and delay 380 are supplied to a motion processor 385 comprising a direct block matcher 390, a correlation surface processor 400, a motion vector estimator 410, a motion vector reducer 420, a motion vector selector 430 and a motion vector post-processor 440. The pairs of input fields are supplied first to the direct block matcher 390 which calculates correlation surfaces representing the spatial correlation between search blocks in the temporally earlier of the two selected input fields and (larger) search areas in the temporally later of the two input fields.

From the correlation surfaces output by the block matcher 390, the correlation surface processor 400 generates a larger number of interpolated correlation surfaces, which are then passed to the motion vector estimator 410. The motion vector estimator 410 detects points of greatest correlation in the interpolated correlation surfaces. (The original correlation surfaces actually represent the difference between blocks of the two input fields; this means that the points of maximum correlation are in fact minima on the correlation surfaces, and are referred to as "minima"). In order to detect a minimum, additional points on the correlation surfaces are interpolated, providing a degree of compensation for the loss of resolution caused by the use of subsampled video to generate the surfaces. From the detected minimum on each correlation surface, the motion vector estimator 410 generates a motion vector which is supplied to the motion vector reducer 420.

The motion vector estimator 410 also performs a confidence test on each generated motion vector to establish whether that motion vector is significant above the average data level, and associates a confidence flag with each motion vector indicative of the result of the confidence test. The confidence test, known as the "threshold" test, is described (along with certain other features of the apparatus of Figure 4) in GB-A-2 231 749.

A test is also performed by the motion vector estimator 410 to detect whether each vector is aliased. In this test, the correlation surface (apart from an exclusion zone around the detected minimum) is examined to detect the next lowest minimum.

If this second minimum does not lie at the edge of the exclusion zone, the motion vector derived from the original minimum is flagged as being potentially aliased.

The motion vector reducer 420 operates to reduce the choice of possible motion vectors for each pixel of the output field, before the motion vectors are supplied to the motion vector selector 430. The output field is notionally divided into blocks of pixels, each block having a corresponding position in the output field to that of a search block in the earlier of the selected input fields. The motion vector reducer compiles a group of four motion vectors to be associated with each block of the output field, with each pixel in that block eventually being interpolated using a selected one of that group of four motion vectors.

Vectors which have been flagged as "aliased" are re-qualified during vector reduction if they are identical to non-flagged vectors in nearby blocks.

As part of its function, the motion vector reducer 420 counts the frequencies of occurrence of "good" motion vectors (i.e. motion vectors which pass the confidence test and the alias test, or which were re-qualified as non-aliased), with no account taken of the position of the blocks of the input fields used to obtain those motion vectors. The good motion vectors are then ranked in order of decreasing frequency. The most common of the good motion vectors which are significantly different to one another are then classed as "global" motion vectors. Three motion vectors which pass the confidence test are then selected for each block of output pixels and are supplied, with the zero motion vector, to the motion vector selector 430 for further processing. These three selected motion vectors are selected in a predetermined order of preference from: (i) the motion vector generated from the corresponding search block (the "local" motion vector"); (ii) those generated from surrounding search blocks ("neighbouring" motion vectors); and (iii) the global motion vectors.

The motion vector selector 430 also receives as inputs the two input fields which were selected by the subsampled time base changer and delay 380 and which were used to calculate the motion vectors. These fields are suitably delayed so that they are supplied to the motion vector selector 430 at the same time as the vectors derived from them. The motion vector selector 430 supplies an output comprising one motion vector per pixel of the output field. This motion vector is selected from the four motion vectors for that block supplied by the motion vector reducer 420.

The vector selection process involves detecting the degree of correlation between test blocks of the two input fields pointed to by a motion vector under test.

The motion vector having the greatest degree of correlation between the test blocks is selected for use in interpolation of the output pixel. A "motion flag" is also generated by the vector selector. This flag is set to "static" (no motion) if the degree of correlation between blocks pointed to by the zero motion vector is greater than a preset threshold.

The vector post-processor reformats the motion vectors selected by the motion vector selector 430 to reflect any vertical or horizontal scaling of the picture, and supplies the reformatted vectors to the interpolator 340. Using the motion vectors, the interpolator 340 interpolates an output field from the corresponding two (nonsubsampled) interlaced input fields selected by the time base changer and delay 330, taking into account any image motion indicated by the motion vectors currently supplied to the interpolator 340.

If the motion flag indicates that the current output pixel lies in a moving or temporally changing part of the image, pixels from the two selected fields supplied to the interpolator are combined in relative proportions depending on the temporal position of the output field with respect to the two input fields (as indicated by the control signal t), so that a larger proportion of the nearer input field is used. If the motion flag is set to "static" then the temporal weighting is fixed at 50% of each input field. The output of the interpolator 340 is passed to an output buffer 350 for output as a high definition output signal, and to a down-converter 360 which generates a conventional definition output signal 365, using the motion flag.

The down-converter 360 allows a representation of the output of the apparatus (which may be, for example, a high definition video signal) to be monitored, transmitted and/or recorded using conventional definition apparatus. This has benefits because conventional definition recording equipment is significantly cheaper and very much more widespread than high definition equipment. For example, a simultaneous output of conventional and high definition video may be required for respective transmission by terrestrial and satellite channels.

The subsampler 370 performs horizontal and vertical spatial subsampling of the input video fields received from the matrix 320, before those input fields are supplied to the time base changer 380. Horizontal subsampling is a straightforward operation in that the input fields are first prefiltered by a half-bandwidth low pass filter (in the present case of 2:1 horizontal decimation) and alternate video samples along each video line are then discarded, thereby reducing by one half the number of samples along each video line.

Vertical subsampling of the input fields is complicated by the fact that, in this embodiment, the input video signal 250 is interlaced. This means that successive lines of video samples in each interlaced field are effectively two video lines apart, and that the lines in each field are vertically displaced from those in the preceding or following field by one video line of the complete frame.

One approach to vertical subsampling would be to perform progressive scan conversion (to generate successive progressively scanned video frames each having 1125 lines) and then to subsample the progressively scanned frames by a factor of 2 to perform the vertical subsampling. However, efficient progressive scan conversion would require a degree of motion compensated processing, which processing could adversely affect the operation of the motion processor 385. Furthermore, real-time progressive scan conversion of a high definition video signal would require particularly powerful and complex processing apparatus.

A simpler approach to vertical spatial subsampling is shown in Figure 5, in which the input fields are first low pass filtered in the vertical direction (to reduce potential aliasing) and a filtering operation is then performed which effectively displaces each pixel vertically by half a video line downwards (for even fields) or upwards (for odd fields). The resulting displaced fields are broadly equivalent to progressively scanned frames which have been subsampled vertically by a factor of two.

In summary, therefore, the result of the subsampling operations described above is that the motion processor 385 operates on pairs of input fields which are spatially subsampled by a factor of two in the horizontal and the vertical directions.

This reduces the processing required for motion vector estimation by a factor of four.

Figure 6 is a schematic diagram showing the comparison of test blocks during vector selection.

Figure 6 shows two of the motion vectors 500, 510 associated with a block containing an output pixel 520 in an output field 530. For clarity of the Figure, the other two motion vectors associated with the output pixel 520 by the motion vector reducer 420 have not been shown.

In order to test one of the four motion vectors (e.g. the motion vector 510) associated with the output pixel 520, that motion vector is extrapolated to point to respective test blocks of pixels 540, 550 in the input fields 560, 570 from which the output field 530 is to be interpolated. The degree of correlation between the test blocks of pixels is then determined, by calculating the absolute luminance difference between pairs of pixels at corresponding positions in the two test blocks. These absolute luminance difference values are then added to produce a sum of absolute luminance differences ('SAD') associated with the motion vector under test. A high SAD value indicates a low degree of correlation between the blocks in the input frames surrounding the pixels pointed to by that motion vector, and a low SAD value indicates a high degree of correlation between those blocks. This test is performed for each of the four motion vectors supplied to the motion vector selector 430 by the motion vector reducer 420; the motion vector having the lowest SAD value from the test is selected by the motion vector selector 430 for use in interpolation of the output pixel 520.

Figure 7 is a schematic block diagram of the motion vector selector 430.

Four motion vectors for each block of output pixels are supplied to the motion vector selector 430 by the motion vector reducer 420. These four motion vectors, namely the zero motion vector and three other vectors referred to as vl, v2 and v3, are supplied to four respective processing units 600, 610, 620 and 630. Each of the processing units 600, 610, 620 and 630 comprises an address offset calculator 640, two random access memories (RAMs) 650, 660, each storing a relevant portion of a respective one of the pair of input fields selected by the time base changer and delay 380, and a block matcher and comparator 670.

In each of the processing units, the address offset calculator 640 receives the motion vector for that processing unit along with the temporal offset control signal (t) generated by the time base changer and delay 380. From these two values, the address offset calculator generates a plurality of memory addresses for accessing test blocks of pixels, stored in the RAMs 650, 660, which are pointed to by that motion vector. In response to the addresses supplied by the address offset calculator, each of the RAMs 650, 660 supplies an array of pixel values representing a test block to the block matcher and comparator 670.

The block matcher and comparator 670 compares pixel values (in particular the luminance component of the pixel values) at corresponding positions in the two test blocks The absolute luminance difference between each pair of pixels is calculated and a sum of all of the absolute luminance differences (SAD) is generated to indicate the overall correlation between the two test blocks. A low SAD value for a particular motion vector indicates a high degree of correlation between the test blocks pointed to by that motion vector.

The processing unit 630 receives the motion vector v3 and calculates the SAD value for test blocks pointed to by that motion vector. The processing unit 630 then supplies the SAD value and an identifier of the vector v3 to the processing unit 620.

In the processing unit 620 the SAD value for the vector v2 is generated by the respective block matcher and comparator 670. The SAD value for v2 is then compared with the SAD value for v3 received from the processing unit 630. The lower of these two SAD values represents the motion vector (v2 or v3) having the higher degree of correlation between test blocks pointed to by that motion vector.

Accordingly, the processing unit 620 outputs the lower of the SAD values 675 and a vector identifier 680 identifying the vector from which the lowest SAD value was derived.

The processing unit 610 receives the vector v1 and calculates a SAD value from that vector. The SAD value for vl is then compared with the lowest SAD value 675 of the vectors v2 and v3, as supplied by the processing unit 620. The processing unit 610 then outputs the lowest SAD value of the vectors vl, v2 and v3, along with an identifier of the vector from which that SAD value was generated.

The processing unit 600 generates a SAD value for the zero motion vector and compares this with the current lowest SAD value for the vectors vl, v2 and v3 received from the processing unit 610. From this comparison, a selected vector identifier 690, indicating that one of the four motion vectors for which the lowest SAD value was generated, is output by the block matcher and comparator 670 in the processing unit 600.

The SAD value for the zero motion vector (generated by the block matcher and comparator 670 in the processing unit 600) is supplied to a comparator 700 in which it is compared with a preset threshold value 710. The SAD value for the zero motion vector is supplied to the comparator 600 regardless of which of the four motion vectors (zero, vl, v2, v3) was selected for use in interpolation of the output pixel 300.

The comparator 700 generates a motion flag 720 which is "set" (indicating image motion) if the SAD value for the zero motion vector is greater than the threshold 710. If the SAD value for the zero motion vector is less than the threshold 710 then the motion flag is not set, thereby indicating that the current output pixel lies in a substantially stationary portion of the picture.

Figure 8 is a schematic block diagram of a part of the motion vector selector of Figure 7, according to one embodiment of the invention.

In Figure 8, the two test blocks pointed to by the zero motion vector (output by the RAMs 650, 660 in the processing unit 600) are supplied to a block matcher 672 forming part of the block matcher and comparator 670. The block matcher 672 compares pixels at corresponding positions in each of the two test blocks to generate a SAD value representing correlation between the test blocks pointed to by the zero motion vector. This SAD value is output directly to the comparator 700 and is also supplied to a multiplier 750.

The multiplier 750 multiplies the SAD value generated for the zero motion vector by a coefficient 1-6. This applies a proportional weighting to reduce the SAD value for the zero motion vector. For example, if 6 is equal to 0.2 then the SAD value for the zero motion vector is reduced 20%.

The weighted SAD value for the zero motion vector output by the multiplier 750 is supplied to a comparator 674 which also forms part of the block matcher and comparator 670. The comparator 674 receives the current lowest SAD value and an identifier indicating which of the motion vectors vl, v2 and v3 corresponds to that current lowest SAD value, from the processing unit 610. The comparator 674 compares the current lowest SAD value with the weighted SAD value output by the multiplier 750, and generates the output 690 identifying either the zero motion vector (if the weighted SAD value for the zero motion vector is lower than or equal to the current lowest SAD value received from the processing unit 610) or the vector identifier received from the processing unit 610 (if the weighted SAD value for the zero motion vector is greater than the current lowest SAD value received from the processing unit 610).

Using the apparatus of Figure 8, a weighting is applied to the SAD for the zero motion vector only, so that the correlation between test blocks pointed to by one of the other three (non-zero) motion vectors must be proportionally better than the correlation for the zero motion vector, if one of those other three motion vectors is to be selected.

As an alternative, instead of (or in addition to) multiplying the SAD value for the zero motion vector by 1-6, where 6 is less than 1, the SAD values for the other motion vectors (vl, v2 and v3) could be multiplied by a coefficient which is greater than 1. In another embodiment, an additive weighting could be used, in which a weighting value is either subtracted from the SAD value for the zero motion vector or added to the SAD values for the remaining three motion vectors, or both.

Alternatively, the additive weighting value could be added to the SAD value for the best of the three motion vectors vl, v2, v3 (the current lowest SAD) when that SAD value is passed from the processing unit 610 to the processing unit 600.

In all of the above cases in which weighting is applied to the SAD values for one or more of the three vectors vl, v2 and v3, the weighting could be made to increase in dependence on the magnitude of the respective motion vector (e.g. in proportion to the vector magnitude). This would mean that in order to be selected, a large motion vector would have to result in a better correlation between the test blocks (i.e. a lower SAD value) than a smaller motion vector, all other factors being equal In another embodiment of the present invention, image enhancement is applied to the test blocks used in motion vector selection in order to enhance small areas of fine detail within those test blocks. This process increases the visibility of the small detailed objects so that they have greater influence in the block match process performed by the block matcher and comparator 670.

One technique of image enhancement suitable for use with the test blocks used in motion vector selection is to apply high frequency emphasis to those test blocks using a conventional two-dimensional filter arrangement. For example, a 3 x 3 spatial filter may be used employing two-dimensional high-pass spatial filtering coefficients such as the coefficients illustrated in Figures 9 and 10. However, although the use of high-pass filters such as those shown in Figures 9 and 10 will enhance fine detail within the test blocks, the filters increase the prominence of image noise, which may degrade the overall system performance.

An alternative image enhancement technique modifies the image data in response to the output of a so-called Sobel filtering process. With these filters, the image data is effectively filtered by a low-pass filter in one direction and a high-pass filter in the other direction. With this technique, the resulting enhanced images are more immune to noise than similar images filtered using the two-dimensional highpass filters of Figures 9 and 10. The Sobel filtering processing is also sensitive to natural image features such as horizontal or vertical lines.

Figure 11 illustrates a set of coefficients for a vertical Sobel spatial filter, and Figure 12 illustrates a set of coefficients for a horizontal Sobel spatial filter. The Sobel filtered versions of the test blocks are used to control the selective substitution of pixel data, as described below.

Figure 13 is a schematic block diagram of an image enhancement circuit. The image enhancement circuit of Figure 13 could be inserted into the apparatus of Figure 7 to process the test blocks output by the RAMs 650, 660 before those test blocks are passed to the block matcher and comparator 670. Alternatively, the image enhancement circuit of Figure 13 could be used to enhance the entire images to be stored in the RAMs 650 and 660. In either case, image enhancement is applied to the test blocks used in vector selection, but not to the images from which the output pixels are ultimately generated.

In the apparatus of Figure 13, input video data 800 (e.g. from the RAMs 650, 660) is supplied in parallel to a vertical Sobel filter 810 and a horizontal Sobel filter 820. The vertical Sobel filter 810 generates, for each pixel of the input video data 800, an output dv indicative of the vertical image detail at that position. Similarly, the horizontal Sobel filter 820 generates an output dh indicative of horizontal image detail at that position.

The outputs from the vertical and horizontal Sobel filters 810, 820 are supplied in parallel to respective pairs of comparators. In particular, the output dv from the vertical Sobel filter 810 is supplied to comparators 830, 840, and the output dh from the horizontal Sobel filter 820 is supplied to comparators 850, 860. The comparators 830 and 850 compare their respective input dv and dh with a threshold (t) 870, generating an output indicating whether dv (or dh) is greater than the threshold value 870. Similarly, the comparators 840 and 860 compare the respective detail value dv or dh with a negated threshold value equal in magnitude to the threshold 870. The negated threshold values are generated by multipliers 880, 890 which each multiply the threshold value 870 by -1. The comparators 840 and 860 generate respective outputs indicating whether the detail value dv or dh is less than the negated threshold (-t).

The outputs of the comparators 830 and 850 are supplied to an OR-gate 900, the output of which controls a switch 910 such that if either the horizontal detail or the vertical detail is greater than the threshold 870, a white pixel is substituted at that position in the input video. Similarly, the outputs of the comparators 840 and 860 are supplied to a OR-gate 920 which controls a switch 930 such that if either of the detail values dv or dh is less than the negated threshold value a black pixel is substituted into the input video at that position.

The output of the switch 930 is supplied as output video (e.g. to the block matcher and comparator 670).

In an alternative embodiment, the substitution of black and white video by the switches 910, 930 could also be subject to the average intensity in a small block surrounding the current pixel, such that in low intensity areas fine detail is substituted by white pixels and in high intensity areas the fine detail is substituted by black pixels.

In another alternative embodiment, instead of substituting black or white pixels in areas of fine detail, a constant signal could be added to the value of the pixels in the areas of fine detail.

Since the performance of the various alternative embodiments described above is dependent on the video material in use, a dynamic system may be employed in which it is possible to switch between these various alternatives.

Figures 14a and 14b illustrate the vector selection process for an image of a moving car (as shown in Figures la, lb, 2a and 2b) using image enhanced test blocks 950, 960. The image enhancement performed by the apparatus of Figure 13 is illustrated schematically in Figures 14a and 14b by the use of bold print to represent the car radio antenna. The result of the image enhancement is that the antenna becomes a much more significant part of the test blocks 950, 960 used in motion vector selection. This means that the correlation test in vector selection is more likely to select the correct vector (the 'car' motion vector) for use in interpolating pixels representing the car radio antenna.

In further embodiments of the invention, the test blocks used in vector selection are analyzed to detect the amount of detail contained in the blocks. The correlation test results are then adjusted or weighted so that a high degree of correlation obtained using test blocks containing little detail (e.g. blocks from plain background areas) is treated as being less significant than a lower degree of correlation obtained from more detailed test blocks.

The test for detail in the test blocks can be performed in a number of ways.

For example, the dynamic range (maximum pixel value - minimum pixel value) in the test blocks could be quantified. As a refinement, the blocks could be smoothed (by a conventional two-dimensional low-pass filtering arrangement) before testing so that one spurious pixel on an otherwise plain background is not taken to indicate detail in the test blocks. In another alternative, a Sobel filtering process could be used.

Another method of assessing the detail in the blocks is to calculate the differences between adjacent pixel values and sum these differences across the block:

Detail = , S MAX ( Pixel~diff, Line~di) lines 1 ptxeis - I where: Pixel~diff = absolute difference between horizontally adjacent pixels; and Line~diff = absolute difference between vertically adjacent pixels.

If there is detail in the block then the differences in luminance values between adjacent pixels causes the above summation to be non-zero by an amount dependent on the high frequency content.

Figure 15 is a schematic block diagram of a part of the motion vector selector of Figure 7, in which the SAD values generated in vector selection are weighted according to the detail detected in the test blocks This means that a correlation test result is not rejected simply because the blocks lack detail. Bearing in mind that a higher degree of correlation between the test blocks is indicated by a lower SAD value generated from the test blocks, the weighting could be performed as follows: difference value = original~SAD x weighting~value where, for the case where the detail test result is greater than a predetermined threshold value, the weighting~value = threshold/(detail test result + 1); and, for the case where the detail test result is less than or equal to the predetermined threshold value, the weighting~value = 1.

In Figure 15, each of the two test blocks pointed to by a motion vector under test is supplied in parallel to a detail detector 1000 and to the block matcher 672 in each of the processing units 600, 610, 620 and 630. The SAD value generated by the block matcher 672 is supplied to a multiplier 1010, in which the SAD value is multiplied by a weighting value generated by the detail detector 1000 as described above.

In another embodiment, illustrated schematically in Figure 16, the test result indicative of detail in the test blocks may be compared with a threshold value; if there is less detail than that represented by the threshold value then the correlation test performed using those blocks is ignored.

In the apparatus of Figure 16, test blocks output by the RAMs 650, 660 are again supplied in parallel to the block matcher 672 and a detail detector 1020. The detail detector 1020 outputs a detail value indicative of the degree of detail in the test blocks. This detail value is passed to a comparator 1030 in which it is compared with a detail threshold 1040.

The block matcher 672 generates a SAD value from the test blocks which is supplied to a comparator 1050 along with the current lowest SAD value 675 from a previous processing unit (as described with reference to Figure 7). The comparator 1050 generates an output indicating whether the SAD value generated by the block matcher 672 is lower than the current lowest SAD value 675.

The outputs of the comparators 1030 and 1050 are supplied to an AND-gate 1060. The output of the AND-gate controls a multiplexer 1070 which generates an output identifying either the motion vector under test 1080 or the current vector identifier 680 received from a previous processing unit. The AND-gate 1060 and the multiplexer 1070 operate so that the output of the multiplexer 1070 identifies the current motion vector under test only if both of the following conditions are satisfied, namely the SAD value for the current vector under test is lower than the current lowest SAD value 675 received from a previous processing unit; and the amount of detail detected by the detail detector 1020 in the test blocks pointed to by the current motion vector under test is greater than or equal to the detail threshold 1040.

Although various discrete embodiments of the invention have been described, it should be noted that the features of the various embodiments (e.g. the weighting of the zero vector SAD, the use of image enhancement and the use of a detail test) may be combined together in various permutations.

Claims (20)

1. Motion compensated video signal processing apparatus comprising: means for generating a plurality of motion vectors to represent image motion between a pair of input images of an input video signal; and means for testing a group of the motion vectors, to select a motion vector for use in interpolation of an output pixel of an output image, the means for testing comprising: means for detecting the degree of correlation between test blocks of the input images pointed to by a motion vector under test; means for generating an analysis value indicative of the image detail of the test blocks; and means for selecting a motion vector from the group of motion vectors, in dependence on the degree of correlation between the test blocks pointed to by the motion vectors and the analysis value for the test blocks pointed to by the motion vectors.

2. Apparatus according to claim 1, in which the means for generating an analysis value comprises means for detecting the difference between the highest luminance component of pixels in a test block and the lowest luminance component of pixels in that test block.

3. Apparatus according to claim 1, in which the means for generating an analysis value comprises means for summing the luminance differences between pairs of adjacent pixels within each test block.

4. Apparatus according to claim 2, in which the means for summing comprises means for summing, for each pixel in a test block, the higher of the luminance difference between that pixel and a vertically adjacent pixel and the luminance difference between that pixel and a horizontally adjacent pixel.

5. Apparatus according to any one of claims 1 to 4, comprising means for preventing selection of a motion vector, for which the test blocks pointed to by that motion vector have an analysis value indicative of less than a predetermined degree of image detail.

6. Apparatus according to any one of claims 1 to 4, comprising: means for detecting a correlation value indicative of the degree of correlation between the test blocks; and means for weighting the correlation values in dependence on the analysis value, the weighting being such that the degree of correlation indicated by the correlation value for a motion vector is increased with increasing detected image detail; and in which the means for selecting is operable to select a motion vector having a weighted correlation value indicative of the highest degree of correlation.

7. Apparatus according to claim 6, in which the means for detecting the degree of correlation comprises: means for detecting the absolute luminance difference between pairs of pixels at corresponding positions in the test blocks; and means for summing the absolute luminance differences, thereby generating a sum of absolute differences (SAD) value.

8. Apparatus according to claim 7, in which the means for weighting comprises means for multiplying the sum of absolute difference value by a weighting coefficient, the weighting coefficient decreasing with increasing detected image detail.

9. Apparatus according to any one of claims 1 to 8, comprising: image enhancement means for enhancing the image content of at least the test blocks of the input images pointed to by a motion vector under test.

10. Apparatus according to any one of claims 1 to 9, in which the group of motion vectors comprises a zero motion vector indicative of zero image motion, the apparatus comprising: means for detecting a correlation value indicative of the degree of correlation between the test blocks of the input images pointed to by a motion vector under test; and means for weighting the correlation values, the weighting being such that the degree of correlation indicated by the correlation value for the zero motion vector is increased with respect to the degrees of correlation indicated by the correlation values for other motion vectors in the group; and in which the means for selecting is operable to select a motion vector having a weighted correlation value indicative of the highest degree of correlation.

11. Apparatus according to any one of claims 1 to 10, in which the pair of input images comprises two successive fields of an interlaced input video signal.

12. Apparatus according to any one of claims 1 to 11, in which the output image comprises a field of an interlaced output video signal.

13. A method of motion compensated video signal processing, the method comprising the steps of: generating a plurality of motion vectors to represent image motion between a pair of input images of an input video signal; and testing a group of the motion vectors, to select a motion vector for use in interpolation of an output pixel of an output image, the step of testing comprising: detecting the degree of correlation between test blocks of the input images pointed to by a motion vector under test; generating an analysis value indicative of the spatial frequency content of the test blocks; and selecting a motion vector from the group of motion vectors, in dependence on the degree of correlation between the test blocks pointed to by the motion vectors and the analysis value for the test blocks pointed to by the motion vectors.

14. Television standards conversion apparatus comprising apparatus according to any one of claims 1 to 12.

15. Motion compensated video signal processing apparatus substantially as hereinbefore described with reference to the accompanying Figures 4, 5, 6, 7 and 15.

16. A method of motion compensated video signal processing, the method being substantially as hereinbefore described with reference to the accompanying Figures 4, 5, 6, 7 and 15.

17. Television standards conversion apparatus substantially as hereinbefore described with reference to the accompanying Figures 4, 5, 6, 7 and 15.

18. Motion compensated video signal processing apparatus substantially as hereinbefore described with reference to the accompanying Figures 4, 5, 6, 7 and 16.

19. A method of motion compensated video signal processing, the method being substantially as hereinbefore described with reference to the accompanying Figures 4, 5, 6, 7 and 16.

20. Television standards conversion apparatus substantially as hereinbefore described with reference to the accompanying Figures 4, 5, 6, 7 and 16.