GB2277003A

GB2277003A - Determining the motion of regular patterns in video images

Info

Publication number: GB2277003A
Application number: GB9307448A
Authority: GB
Inventors: Morgan William Amos David; Martin Rex Dorricott; Shima Ravji Varsani; James Edward Burns; Carl William Walters; Stephen Mark Keating
Original assignee: Sony United Kingdom Ltd
Current assignee: Sony Europe Ltd
Priority date: 1993-04-08
Filing date: 1993-04-08
Publication date: 1994-10-12
Anticipated expiration: 2013-04-08
Also published as: GB9307448D0; JPH0750815A; GB2277003B

Abstract

The movement of a regular pattern in a video image is determined by forming a correlation surface between search blocks in a pair of images; corresponding luminance values may be subtracted, so that points of maximum correlation are represented by minima. The position of a first maximum 430 is determined, and an exclusion region 460 set up about it in which further maxima are not detected. The position of a second maximum 440 is detected and an alias flag set if it is adjacent to the first maximum to indicate to further parts of the circuitry that the motion vector corresponding to the second maximum is potentially aliassed. Further circuitry may compare motion vectors having the alias flag set with other motion vectors and reset the alias flag as a result. <IMAGE>

Description

MOTION COMPENSATED IMAGE PROCESSING This invention relates to motion compensated image processing.

Motion compensated image processing is used in applications such as television standards conversion, film standards conversion and conversion between video and film standards.

In a motion compensated television standards converter, such as the converter described in the British Published Patent Application number GB-A-2 231 749, pairs of successive input images are processed to generate sets of motion vectors representing image motion between the pair of input images. The processing is carried out on discrete blocks of the images, so that each motion vector represents the interimage motion of the contents of a respective block.

In the motion vector estimation process a correlation surface representing spatial correlation between blocks of the two input images is examined to detect points of maximum correlation. (The correlation surface actually represents the difference between the two input images; this means that the points of maximum correlation are in fact minima on the correlation surface, and will be referred to herein as "minima"). When a minimum has been detected, a motion vector is generated from the spatial position of the minimum in the correlation surface. Tests are performed to determine whether that minimum represents a significant peak in correlation with respect to the remainder of the correlation surface. If the minimum passes this test it is treated as being "valid", and a validity flag associated with the motion vector is set.

Each set of motion vectors is then supplied to a motion vector reducer which derives a subset of the set of (valid) motion vectors for each block, which is then passed to a motion vector selector which assigns one of the subset of motion vectors to each picture element (pixel) in each block of the image. The selected motion vector for each pixel is supplied to a motion compensated interpolator which interpolates output images from the input images, taking into account the motion between the input images.

A problem can occur with images having areas of regularly spaced patterning, such as the image illustrated in Figure 1 of the accompanying drawings. The image is subject to horizontal motion (resulting from a horizontal camera pan). A search block 10 used to generate a correlation surface encompasses more than one "cycle" of the regularly spaced patterning.

When motion vector estimation is performed on the image shown in Figure 1, multiple minima can be detected which can be spaced across the correlation surface. The multiple minima are illustrated in Figure 2 of the accompanying drawings, which is a schematic representation of a cross-section through a correlation surface generated from the search block 10.

One of the detected minima 20 represents the correct motion of the scene; the other minimum 30 results from alias interactions between the image patterning and the size of the blocks used to generate the correlation surfaces. The vector estimation process has no way of identifying which one of the minima represents the correct image motion, and in fact an aliased (incorrect) vector is often generated.

Using this aliased vector in interpolation of an output image can lead to subjectively disturbing image artifacts, such as the artifacts 40 illustrated in Figure 3 of the accompanying drawings.

This invention provides a motion compensated image processing apparatus in which motion vectors are generated to represent image motion between a pair of images of a video signal, the apparatus comprising: means for generating a correlation surface representing correlation between respective blocks of the pair of images; means for detecting a first point of maximum correlation from the correlation surface; means for generating a motion vector in dependence on the first correlation value; means for detecting a second point of maximum correlation from the correlation surface apart from an excluded region surrounding the first point of maximum correlation; and means for detecting whether the second point of maximum correlation is adjacent to the excluded region of the correlation surface, thereby detecting whether the motion vector is potentially aliased.

Apparatus according to the invention allows the detection of multiple correlation maxima (i.e. potentially aliased motion vectors) by examination of the correlation surface. Various measures (such as not using potentially aliased motion vectors for interpolation of an output image) can then be employed to reduce the occurrence of disturbing image artifacts such as those shown in Figure 3.

In order that the detection of a potentially aliased motion vector can be indicated to other parts of the apparatus, it is preferred that the apparatus comprises means for setting an alias flag, associated with the motion vector, to indicate that the motion vector is potentially aliased if the second point of maximum correlation is not adjacent to the excluded region of the correlation surface.

A motion vector which has been detected to be potentially aliased could simply be discarded, or not used in the interpolation of an output image. In such a case, the zero motion vector (for example) could be substituted for the potentially aliased motion vector.

However, it is preferred that further investigation is made to determine whether the motion vector is in fact useable. To this end, a preferred embodiment of the apparatus comprises means for comparing a motion vector under test, for which the associated alias flag is set, with a predetermined group of other motion vectors generated from the pair of images; and means, responsive to a detection that the magnitude of the motion vector under test is within a predetermined threshold of the magnitude of at least one of the predetermined group of motion vectors, for resetting the alias flag associated with the motion vector under test. In this way, if the potentially aliased motion vector is substantially similar to surrounding motion vectors, it can be requalified as non-aliased (or aliased but re-qualified), and then used as part of the interpolation process.

In a preferred embodiment the correlation surface represents the correlation between a search block in one of the pair of images and a search area, comprising a rectangular array of blocks, in the other of the pair of images.

It is preferred that the re-qualification of potentially aliased motion vectors is performed by comparing a motion vector under test with other motion vectors derived from nearby portions of the input images. To this end, it is preferred that the predetermined group of motion vectors comprises those motion vectors corresponding to a predetermined number of search blocks adjacent to the search block corresponding to the motion vector under test, for which the respective alias flag is not set.

Preferably the predetermined number of adjacent search blocks comprises a 5 x 5 array of adjacent search blocks.

Although the correlation surface may be represented in various ways, it is preferred that the correlation surface comprises an array of correlation values representing the difference between the content of the search block and the search area. In this way the point of maximum correlation can be detected by simply detecting the correlation value in the array representing the minimum difference between the respective blocks.

Although the correlation values could relate to chrominance differences, or combined luminance and chrominance differences, it is preferred that the correlation values represent the difference between the luminance content of the search block and the search area.

Although the excluded region could comprise one of a number of patterns of correlation values surrounding the first point of maximum correlation, such as an annular region, it is preferred that the excluded region comprises a rectangular array of correlation values surrounding the first point of maximum correlation. Preferably the excluded region comprises an array of 3 x 3 correlation values.

In a preferred embodiment the apparatus comprises a motion compensated interpolator operable to interpolate an output image using motion vectors generated from the pair of input images and which were detected not to be potentially aliased.

In order to distinguish genuine correlation maxima from noiserelated artifacts, it is preferred that the apparatus comprises means for performing a predetermined confidence test on each motion vector to determine whether the first point of maximum correlation represents a significant correlation maximum in the correlation surface. Again, in order that a potentially aliased motion vector is not re-qualified on the basis of other motion vectors generated from noise phenomena, it is preferred that the predetermined group of motion vectors comprises motion vectors which have passed the confidence test.

Apparatus according to the invention is advantageously employed in a motion compensated television standards converter.

Viewed from a second aspect this invention provides a method of motion compensated image processing in which motion vectors are generated to represent image motion between a pair of images of a video signal, the method comprising the steps of: generating a correlation surface representing correlation between respective blocks of the pair of images; detecting a first point of maximum correlation from the correlation surface; generating a motion vector in dependence on the first correlation value; detecting a second point of maximum correlation from the correlation surface apart from an excluded region surrounding the first point of maximum correlation; and detecting whether the second point of maximum correlation is adjacent to the excluded region of the correlation surface, thereby detecting whether the motion vector is potentially aliased.

The invention will now be described by way of example with reference to the accompanying drawings, throughout which like parts are referred to by like references, and in which: Figure 1 is a schematic diagram of a television image; Figure 2 is a schematic representation of a cross-section through a correlation surface; Figure 3 illustrates image artifacts resulting from the use of an aliased motion vector; Figure 4 is a schematic block diagram of a motion compensated television standards conversion apparatus; Figure 5 is a schematic diagram of a correlation surface; Figures 6 and 7 are schematic representations of cross sections through two respective correlation surfaces; and Figure 8 is a schematic block diagram of a part of a motion vector reducer.

Figure 4 is a schematic block diagram of a motion compensated television standards conversion apparatus. The apparatus receives an input interlaced digital video signal 50 (e.g. an 1125/60 2:1 high definition video signal (HDVS)) and generates an output interlaced digital video signal 60 (e.g a 1250/50 2:1 HDVS).

The input video signal 50 is first supplied to an input buffer/packer 110. In the case of a conventional definition input signal, the input buffer/packer 110 formats the image data into a high definition (16:9 aspect ratio) format, padding with black pixels where necessary. For a HDVS input the input buffer/packer 110 merely provides buffering of the data.

The data are passed from the input buffer/packer 110 to a matrix circuit 120 in which (if necessary) the input video signal's format is converted to the standard "CCIR recommendation 601" (Y,Cr,Cb) format.

From the matrix circuit 120 the input video signal is passed to a time base changer and delay 130, and via a sub-sampler 170 to a subsampled time base changer and delay 180. The time base changer and delay 130 determines the temporal position of each field of the output video signal, and selects the two fields of the input video signal which are temporally closest to that output field for use in interpolating that output field. For each field of the output video signal, the two input fields selected by the time base changer are appropriately delayed before being supplied to an interpolator 140 in which that output field is interpolated. A control signal t, indicating the temporal position of each output field with respect to the two selected input fields, is supplied from the time base changer and delay 130 to the interpolator 140.

The subsampled time base changer and delay 180 operates in a similar manner, but using spatially subsampled video supplied by the subsampler 170. Pairs of input fields are selected by the subsampled time base changer and delay 180 from the subsampled video, to be used in the generation of motion vectors.

The time base changers 130 and 180 can operate according to synchronisation signals associated with the input video signal, the output video signal, or both. In the case in which only one synchronisation signal is supplied, the timing of fields of the other of the two video signals is generated deterministically within the time base changers 130, 180.

The pairs of fields of the subsampled input video signal selected by the subsampled time base changer and delay 180 are supplied to a motion processor 185 comprising a direct block matcher 190, a data stripper 200, a motion vector estimator 210, a motion vector reducer 220, a motion vector selector 230 and a motion vector post-processor 240. The pairs of input fields are supplied first to the direct block matcher 190 which calculates correlation surfaces representing the spatial correlation between search blocks in the temporally earlier of the two selected input fields and (larger) search areas in the temporally later of the two input fields. Data representing these correlation surfaces are reformatted by the stripper 200 and are passed to the motion vector estimator 210. The motion vector estimator 210 detects points of greatest correlation in the correlation surfaces.

(The correlation surfaces actually represent the difference between blocks of the two input fields; this means that the points of maximum correlation are in fact minima on the correlation surface, and are referred to as "minima"). In order to detect a minimum, additional points on the correlation surfaces are interpolated, providing a degree of compensation for the loss of resolution caused by the use of subsampled video to generate the surfaces. From the detected minimum on each correlation surface, the motion vector estimator 210 generates a motion vector which is supplied to the motion vector reducer 220.

The motion vector estimator 210 also performs a confidence test on each generated motion vector to establish whether that motion vector is significant above the general noise level, and associates a confidence flag with each motion vector indicative of the result of the confidence test. The confidence test, known as the "threshold" test, is described (along with other features of the apparatus of Figure 4) in GB-A-2 231 749.

A test is also performed by the motion vector estimator 210 to detect whether each vector is aliased. In this test, (which is described below in more detail) the correlation surface (apart from an exclusion zone around the detected minimum) is examined to detect the next lowest minimum. If this second minimum does not lie at the edge of the exclusion zone, the motion vector derived from the original minimum is flagged as being potentially aliased.

The motion vector reducer 220 operates to reduce the choice of possible motion vectors for each pixel of the output field, before the motion vectors are supplied to the motion vector selector 230. The output field is notionally divided into blocks of pixels, each block having a corresponding position in the output field to that of a search block in the earlier of the selected input fields. The motion vector reducer compiles a group of four motion vectors to be associated with each block of the output field, with each pixel in that block eventually being interpolated using a selected one of that group of four motion vectors.

Vectors which have been flagged as "aliased" are re-qualified during vector reduction if they are substantially identical to nonflagged vectors in adjacent blocks.

As part of its function, the motion vector reducer 220 counts the frequencies of occurrence of "good" motion vectors (i.e. motion vectors which pass the confidence test and the alias test, or which were requalified as non-aliased), with no account taken of the position of the blocks of the input fields used to obtain those motion vectors. The good motion vectors are then ranked in order of decreasing frequency.

The most common of the good motion vectors which are significantly different to one another are then classed as "global" motion vectors.

Three motion vectors which pass the confidence test are then selected for each block of output pixels and are supplied, with the zero motion vector, to the motion vector selector 230 for further processing.

These three selected motion vectors are selected in a predetermined order of preference from: (i) the motion vector generated from the corresponding search block; (ii) those generated from surrounding search blocks ("local" motion vectors); and (iii) the global motion vectors.

The motion vector selector 230 also receives as inputs the two input fields which were selected by the subsampled time base changer and delay 180 and which were used to calculate the motion vectors.

These fields are suitably delayed so that they are supplied to the motion vector selector 230 at the same time as the vectors derived from those fields. The motion vector selector 230 supplies an output comprising one motion vector per pixel of the output field. This motion vector is selected from the four motion vectors for that block supplied by the motion vector reducer 220.

The vector selection process involves detecting the degree of correlation between test blocks of the two input fields pointed to by a motion vector under test. The motion vector having the greatest degree of correlation between the test blocks is selected for use in interpolation of the output pixel. A "motion flag" is also generated by the vector selector. This flag is set to "static" (no motion) if the degree of correlation between blocks pointed to by the zero motion vector is greater than a preset threshold.

The vector post-processor reformats the motion vectors selected by the motion vector selector 230 to reflect any vertical scaling of the picture, and supplies the reformatted vectors to the interpolator 140. Using the motion vectors, the interpolator 140 interpolates an output field from the corresponding two (non-subsampled) interlaced input fields selected by the time base changer and delay 130, taking into account any image motion indicated by the motion vectors currently supplied to the interpolator 140.

If the motion flag indicates that the current output pixel lies in a moving part of the image, pixels from the two selected fields supplied to the interpolator are combined in relative proportions depending on the temporal position of the output field with respect to the two input fields (as indicated by the control signal t), so that a larger proportion of the nearer input field is used. If the motion flag indicates is set to "static" then temporal weighting is not used.

The output of the interpolator 140 is passed to an output buffer 150 for output as a HDVS output signal, and to a down-converter 160 which generates a conventional definition output signal 165.

The subs ampler 170 performs horizontal and vertical spatial subsampling of the input fields received from the matrix 120.

Horizontal subsampling is a straightforward operation in that the input fields are first pre-filtered by a half-bandwidth low pass filter (in the present case of 2:1 horizontal decimation) and alternate video samples along each video line are then discarded, thereby reducing by one half the number of samples along each video line.

Vertical subsampling of the input fields is complicated by the fact that the input video signal is interlaced. This means that successive lines of video samples in each interlaced field are effectively separated by two video lines of the complete frame, and that the lines in each field are vertically displaced from those in the preceding or following field by one video line of the complete frame.

The method of vertical subsampling actually used involves a first stage of low pass filtering in the vertical direction (to avoid aliasing), followed by a filtering operation which effectively displaces each pixel vertically by half a video line downwards (for even fields) or upwards (for odd fields). The resulting displaced fields are broadly equivalent to progressively scanned frames which have been subsampled vertically by a factor of two.

Figure 5 is a schematic diagram of a correlation surface 300.

The correlation surface represents the difference between a search block of the earlier of the two input fields from which the surface is generated and a larger search area in the later of the two input fields. A peak in correlation is therefore represented by a minimum point 310 on the correlation surface 300. The position of the minimum (point of maximum correlation) 310 on the correlation surface 300 determines the magnitude and direction of the motion vector derived from that correlation surface.

In order to detect whether the motion vector derived from the correlation surface 300 is potentially aliased, an exclusion region 320 is defined around the minimum 310. The correlation surface (apart from the exclusion region 320 around the minimum 310) is then re-examined to identify the next lowest minimum. If the next minimum does not lie at the edge of the exclusion region then the generated motion vector is flagged as being potentially aliased.

Two examples of the process used to detect potentially aliased motion vectors will now be described with reference to Figures 6 and 7, which are schematic representations of cross sections through two respective correlation surfaces 400, 420. The correlation surface 400 in Figure 6 has a single minimum and therefore does not suffer from alias problems as described above. When an exclusion region 410 is defined around the minimum 405, the next lowest minimum 415 is found to lie at the edge of the exclusion region. This indicates that the surface is not aliased and an alias flag associated with the motion vector generated from the correlation surface 400 is set to indicate a "not aliased" condition.

In contrast, the correlation surface 420 shown in Figure 7 has a number of distinct minima 430, 440, 450. This correlation surface represents the correlation between two images having regularly spaced patterning. During motion vector estimation from the surface 420, the lowest minimum 430 (the point of maximum correlation) is detected and an exclusion region 460 defined around the minimum 430. The next lowest minimum is then found to be the minimum 440 which does not lie at the edge of the exclusion region. The motion vector generated from the minimum 430 is therefore flagged as being potentially aliased; an alias flag associated with the vector is set, indicating an "aliased" condition.

Figure 8 is a schematic block diagram of a part of the motion vector reducer 220, which operates to re-qualify motion vectors which have been flagged as being potentially aliased.

Each of the motion vectors generated during motion vector estimation is stored at a position in a memory array 500 depending on the position (in the input image) of a search block from which the corresponding correlation surface was derived. The confidence flags associated with the motion vectors are stored in a corresponding confidence flag array 510 and the aliased flags (generated as described above) are stored in an input alias flag array 520.

The re-qualification of potentially aliased motion vectors comprises a sequential scan through the array memories 500, 510, 520 under the control of an address generator 530. For each of the motion vectors, the alias flag, the vector value (magnitude and direction) and the confidence flag for that motion vector 525 are supplied to a comparator 540. The address generator 530 also controls the reading of the alias flags, confidence flags and vector values for the 24 motion vectors surrounding the vector under test in a 5 x 5 array pattern 535.

These 24 sets of flags and values are also supplied to the comparator 540.

The comparator 540 tests the vector under test against the surrounding 24 vectors according to a set of rules given below, and generates a modified alias flag for the vector under test. This modified flag is written to a corresponding position 550 in an output alias flag array 560. under the control of the address generator 530.

The rules obeyed by the comparator 540 are as follows: 1. If the confidence flag for the vector under test is not set (indicating that the vector failed the confidence test) then the corresponding alias flag is written without modification into the output alias flag array 560. In this case, the state of the alias flag is immaterial, since a vector which failed the confidence test is not used in motion compensated interpolation.

2. If the confidence flag is set and the alias flag is not set for the vector under test (indicating that the vector under test passed the confidence test and is not aliased), then the alias flag for the vector under test is written without modification into a corresponding position in the output alias flag array 560.

3. If the alias flag and the confidence flag for the vector under test are set (indicating that the vector passed the confidence test but is potentially aliased), then: a) if the value of the vector under test lies within a preset threshold 570 of the value of one or more of the surrounding 24 vectors (the one or more vectors being non aliased and having passed the confidence test) then the alias flag for the vector under test is re-qualified to indicate "not-aliased". The re-qualified alias flag is written into the output alias flag array 560; b) if the value of the vector under test does not lie within the preset threshold 570 of the value of one or more of the surrounding 24 vectors, then the alias flag for the vector under test is not re-qualified, and is written without modification into the output alias flag array 560.

The use of separate input and output alias flag arrays prevents the propagation of errors through the array during aliased vector requalification.

Only those motion vectors which pass the confidence test and for which the alias flag is not set in the output alias flag array 560 (i.e. the vectors were detected as being non-aliased during vector estimation or have been re-qualified during vector reduction) are output by the motion vector reducer 220, for possible use by the interpolator 140.

Claims

1. Motion compensated image processing apparatus in which motion vectors are generated to represent image motion between a pair of images of a video signal, the apparatus comprising: means for generating a correlation surface representing correlation between respective blocks of the pair of images; means for detecting a first point of maximum correlation from the correlation surface; means for generating a motion vector in dependence on the first correlation value; means for detecting a second point of maximum correlation from the correlation surface apart from an excluded region surrounding the first point of maximum correlation; and means for detecting whether the second point of maximum correlation is adjacent to the excluded region of the correlation surface, thereby detecting whether the motion vector is potentially aliased.

2. Apparatus according to claim 1, comprising means for setting an alias flag, associated with the motion vector, to indicate that the motion vector is potentially aliased if the second point of maximum correlation is not adjacent to the excluded region of the correlation surface.

3. Apparatus according to claim 2, comprising: means for comparing a motion vector under test, for which the associated alias flag is set, with a predetermined group of other motion vectors generated from the pair of images; and means, responsive to a detection that the magnitude of the motion vector under test is within a predetermined threshold of the magnitude of at least one of the predetermined group of motion vectors, for resetting the alias flag associated with the motion vector under test.

4. Apparatus according to any one of the preceding claims, in which the correlation surface represents the correlation between a search block in one of the pair of images and a search area, comprising a rectangular array of blocks, in the other of the pair of images.

5. Apparatus according to claim 3 and claim 4, in which the predetermined group of motion vectors comprises those motion vectors corresponding to a predetermined number of search blocks adjacent to the search block corresponding to the motion vector under test, for which the respective alias flag is not set.

6. Apparatus according to claim 5, in which the predetermined number of adjacent search blocks comprises a 5 x 5 array of adjacent search blocks.

7. Apparatus according to any one of claims 4 to 6, in which the correlation surface comprises an array of correlation values representing the difference between the content of the search block and the search area.

8. Apparatus according to claim 7, in which the correlation values represent the difference between the luminance content of the search block and the search area.

9. Apparatus according to claim 7 or claim 8, in which the excluded region comprises a rectangular array of correlation values surrounding the first point of maximum correlation.

10. Apparatus according to claim 9, in which the excluded region comprises an array of 3 x 3 correlation values.

11. Apparatus according to any one of the preceding claims, comprising a motion compensated interpolator operable to interpolate an output image using motion vectors generated from the pair of input images and which were detected not to be potentially aliased.

12. Apparatus according to any one of the preceding claims, comprising means for performing a predetermined confidence test on each motion vector to determine whether the first point of maximum correlation represents a significant correlation maximum in the correlation surface.

13. Apparatus according to claim 5 and claim 12, in which the predetermined group of motion vectors comprises motion vectors which have passed the confidence test.

14. A motion compensated television standards converter comprising apparatus according to any one of claims 1 to 13.

15. A method of motion compensated image processing in which motion vectors are generated to represent image motion between a pair of images of a video signal, the method comprising the steps of: generating a correlation surface representing correlation between respective blocks of the pair of images; detecting a first point of maximum correlation from the correlation surface; generating a motion vector in dependence on the first correlation value; detecting a second point of maximum correlation from the correlation surface apart from an excluded region surrounding the first point of maximum correlation; and detecting whether the second point of maximum correlation is adjacent to the excluded region of the correlation surface, thereby detecting whether the motion vector is potentially aliased.

16. Motion compensated image processing apparatus substantially as hereinbefore described with reference to the accompanying drawings.

17. A method of motion compensated image processing, the method being substantially as hereinbefore described with reference to the accompanying drawings.

18. A motion compensated television standards converter substantially as hereinbefore described with reference to the accompanying drawings.