GB2268351A - Motion analysis of moving images - Google Patents

Abstract

A motion analysis system is described that determines and utilises the second differential d2(CS)1dx2 of a correlation surface in identifying correlation maxima 16 that are sufficiently sharp (i.e. second differential exceeds a threshold Ths) to be valid. In a surface with more than one correlation maximum 14, 16 the system scans a complete correlation surface and determines the best (CSb) and next best (CSnb) correlation maxima in that surface. The correlation surface is rejected unless the difference between these two maxima exceeds a predetermined threshold THA and the best maximum is also the point of maximum correlation included the correlation surface. <IMAGE>

Description

MOTION ANALYSIS OF MOVING IMAGES This invention relates to motion analysis of moving images. More particularly, but not exclusively, this invention relates to motion analysis of moving images as part of motion compensated moving image standards conversion, e.g. high definition television format to 50Hz 2:1 PAL television format.

A motion compensated video standards converter is described in British Published Patent Application GB-A-2 231 749 (Sony Corporation).

The system of GB-A-2 231 749 performs motion compensated interpolation to derive a sequence of output images from a sequence of input images having a different format. As illustrated in Figure 1 of GB-A-2 231 749 (a copy of which is reproduced herein as Figure 1 of the accompanying drawings), the standards converter comprises a sequence of processing units.

A video signal is supplied to an input terminal 1. The input terminal 1 is connected to a progressive scan converter 2 in which the input video fields are converted into video frames which are supplied to a direct block matcher 3 wherein correlation surfaces are created.

These correlation surfaces are analyzed by a motion vector estimator 4, which derives and supplies. motion vectors to a motion vector reducer 5, wherein the number of motion vectors for each pixel is reduced, before they are supplied to a motion vector selector 6 which also receives an output from the progressive scan converter 2. Any irregularity in the selection of the motion vectors by the motion vector selector 6 is removed by a motion vector post processor 7 from which the processed motion vectors are supplied to and control an interpolator 8, which also receives an input from the progressive scan converter 2. The output from the interpolator 8, which is a standards-converted and motion-compensated video signal, is supplied to an output terminal 9.

The quality of output of such standards converters is affected by the accuracy with which motion vectors are identified. Measures which improve the accuracy with which motion vectors are identified will produce an improvement in the quality of the output.

The invention provides apparatus for motion analysis of moving images, said apparatus comprising: means for calculating a correlation surface representing image correlation between a search block of image data within a current image and portions of a temporally adjacent image displaced by differing displacement vectors from said search block; means for identifying points in said correlation surface corresponding to correlation maxima; means for calculating a value of a second differential of said correlation surface in at least one direction at least said points of correlation maxima; means for comparing said at least one value with a predetermined second differential threshold value in a threshold test; and means for rejecting any points of correlation maxima that fail said threshold test.

The invention recognises that sharper correlation maxima are more likely to relate to valid motion vectors than less sharp correlation maxima. This is exploited by including a threshold test at candidate points of correlation maxima to determine if the second differential of the correlation surface at these points exceeds a predetermined threshold value. The second differential gives a good measure of correlation maximum sharpness with an increase in magnitude of the second differential corresponding to an increase in sharpness. Points of correlation maxima that fail this test will be rejected as potential source for a motion vector.

A second differential measures the sharpness of a correlation maxima along a particular line. Whilst using the second differential along a single line will produce a course test for sharpness, it may produce a misleading result in the case of a correlation maxima that is sharp in one direction but not sharp in another direction. A preferred compromise between ease-of implementation and the effectiveness of the test is one in which said means for calculating calculates a value of a second differential of said correlation surface in two directions.

It will be appreciated that the effectiveness of the test provided by determination of the second differential in the two directions will be increased in embodiments in which said two directions are orthogonal.

The co-ordinate systems relative to which the second differentials are calculated can vary. Embodiments of the invention utilising polar co-ordinates and calculating second differentials along the r and e axis would be possible. However, in preferred embodiments of the invention said two directions correspond to cartesian coordinate directions in said correlation surface.

One way in which points of correlation maxima may be identified is to test the values surrounding a given point to see if they all indicate a lower correlation. If this is the case, then the point will be a correlation maximum. However, in preferred embodiments of the invention said means for identifying calculates a value of a first differential of said correlation surface in two directions and identifies as correlation maxima points at which said values both pass from indicating a correlation increase to indication a correlation decrease.

The use of the first differential to identify points of correlation maxima is complementary to the use of the second differential to test those points since determination of the second differential involves previously determining the first differential.

A feature of preferred embodiments of the invention that allows unsuitable correlation surfaces to be detected is provided by means for detecting a correlation magnitude maximum point in said correlation surface at which correlation is at a maximum; and means for rejecting said correlation surface for use in motion analysis if said correlation magnitude maximum point does not correspond to a point of a correlation maximum that passed said threshold test. If the correlation magnitude maximum point in a surface does not correspond to a sufficiently sharp correlation maximum, then it is unlikely that reliable motion vectors can be determined from that surface. This feature allows such correlation surfaces to be detected and rejects them as potential sources of motion vectors.

Another problem that can arise in the analysis of correlation surfaces results from the occurrence of multiple, typically closely spaced, points of correlation maxima resulting from alias effects. A problem that can also arise is having a correlation maxima that is sharp in one direction but not sharp in another (i.e. a trough minimum in a non-inverted surface).In order to help in identifying such problem situations, preferred embodiments of the invention comprise means for determining, in a correlation surface with at least two correlation maxima, a difference value of correlation magnitude between a point of a correlation maximum having greatest correlation magnitude and a point of a correlation maximum having next greatest correlation magnitude; and means for rejecting said correlation surface for use in motion analysis if said difference value is less than a predetermined difference threshold value.

This feature can be thought of as testing for the "uniqueness" of points of correlation maxima. If a correlation surface does not include a point of correlation maximum that is sufficiently distinct from any others (e.g. due to aliasing or a trough) then that correlation surface will be rejected as a source of motion vectors. In a surface where the two greatest correlation maxima have the same correlation magnitude, one will be assigned as the greatest and one will be assigned the next greatest. In this case the difference value will be zero and the surface will fail the test.

It will be appreciated that this invention is particularly applicable to the field of motion compensated image standards conversion. Such standards conversion typically takes place prior to an image signal being broadcast and in the broadcast field the requirements for image quality are particularly strict.

The invention also provides a method of motion analysis of moving images, said method comprising the steps of: calculating a correlation surface representing image correlation between a search block of image data within a current image and portions of a temporally adjacent image displaced by differing displacement vectors from said search block; identifying points in said correlation surface corresponding to correlation maxima; calculating a value of a second differential of said correlation surface in at least one direction at least said points of correlation maxima; comparing said at least one value with a predetermined threshold value in a threshold test; and rejecting any points of correlation maxima that fail said threshold test.

An embodiment of the invention will. now be described, by way of example only, with reference to the accompanying drawings in which: Figure 1 is a block diagram of a motion compensated moving image standards converter of the type disclosed in GB-A-2231749; Figure 2 shows an inverted correlation surface; Figure 3 shows a section through two aliased correlation maxima; Figure 4 shows a trough correlation maximum; Figure 5 illustrates a line through a correlation surface with corresponding first and second derivatives; Figures 6, 7 and 8 illustrate a circuit for identifying -and testing correlation maxima; and Figures 9 and 10 are flow diagrams illustrating the tests performed by the circuits of Figures 6, 7 and 8.

Figure 2 illustrates an inverted correlation surface. The X and Y axis represent the position within a search area in one frame where tests for a matching pattern to that from a search block within a temporally adjacent frame were made. This correlation test is performed by the direct block matcher 3 of Figure 1 and involves the summing of the difference in amplitude values for the pixels within the search block compared to the corresponding pixels at that position in the search area. A high resulting value represents low correlation and a low resulting value represents high correlation. The correlation surface shown in Figure 2 has had its Z axis inverted. Accordingly, a maximum in Figure 2 corresponds to a correlation maximum. Without this inversion, a minimum in the correlation surface corresponds to a correlation maximum.

The correlation surface of Figure 2 shows two closely spaced correlation maxima 2, 4. Such closely spaced correlation maxima are typically produced due to alias effects. The correlation surface of Figure 2 also includes a ridge 6 (in a non-inverted correlation surface this would be a trough). Such a ridge may be the result of correlation within an image frame rather than between image frames and can have the effect of disrupting the motion vector identification process.

Figure 3 illustrates a section through a non-inverted correlation surface. This surface includes two distinct correlation maxima 8, 10.

Both these correlation maxima 8, 10 have the same correlation magnitude Mag. In this circumstance, it may not be possible to determine which of the correlation maxima 8, 10 represents the true motion vector.

Figure 4 illustrates a correlation surface including a trough correlation maximum 12. When such a correlation surface is searched for correlation maxima, a succession of points of correlation maxima running along the base of the trough 12 will be identified. The system may not be able to determine which of these points in fact represents the true motion vector.

Figure 5 illustrates a section through a correlation surface along a line x together with the first and second differentials along that line. The section through the correlation surface shown in graph a) includes two minimum points 14, 16 corresponding to points of correlation maxima. The correlation maximum at point 16 is sharper than that at point 14. The magnitude of the correlation value at point 16 is sufficiently different to that at point 14 such that an aliasing threshold value THA is exceeded. Accordingly, the correlation surface would pass the test that its greatest magnitude correlation maximum was sufficiently unique so as not to be effected by aliasing or a trough.

Graph b) shows the first differential along the line x of the correlation surface shown in a). At the points corresponding to the correlation maxima 14, 16, the value of the first differential is zero and is passing from negative to positive. In contrast, at point 22 corresponding to a correlation minimum 24, the value of the first derivative is zero but this time passing from positive to negative. It will be understood that if the correlation surface were inverted, then the sign change at a zero crossing indicating a correlation maximum would also be inverted. It will be seen that the relative sharpness of the correlation maximum 16 has resulted in a rapid change in the first differential about point 20. This rapid change may be classed as a discontinuity.

Graph c) shows the second differential of the correlation surface in the x direction. It will be seen that the sharp correlation maximum 16 that produced a rapid change in the first differential about point 20 has produced a large value of the second differential at point 26.

In contrast, the less sharp correlation maximum at point 14 with its more gradual change in the first differential at point 18 has produced a lower peak 28 in the second differential. This difference is used to identify correlation maximum 16 as sharper since the second differential exceeds the second differential threshold value Ths The circuits illustrated in Figures 6, 7 and 8 form part of the motion vector estimator 4 illustrated in Figure 1. As shown in Figure 6, correlation surface data from the direct block matcher 3 of Figure 1 is received at an input node 30 and stored in a correlation surface store 32. The correlation surface data values CS are then sequentially read out of the correlation surface store 32 under control of a read address generator 34.If the read address generator 34 produces addresses raster scanning through the correlation surface along lines in the X direction, then this is the direction in which subsequent first and second derivatives are taken. If the read address generator 34 is modified to scan in the Y direction, then the same circuitry will take first and second derivatives in the Y direction.

The correlation surface values CS read from the correlation surface store 32 are fed to a subtracter 36 both directly at a subtracter input 38 and indirectly at a subtracter input 40 via a one value delay unit 42. The action of the subtracter 36 is to calculate a difference between adjacent correlation surface values CS. Such a difference is representative of the first derivative of the correlation surface along the scanning direction x.

The output from the subtracter 36 that represents the first derivative is fed to a subtracter 44 both directly at a subtracter input 46 and indirectly at a subtracter input 48 via a one value delay unit 50. The action of the subtracter 44 is to calculate a difference between adjacent first derivative values. This difference is representative of the second derivative of the correlation surface values CS in thex direction.

The output from the correlation surface store 32 is also fed to a minimum value comparator 52 and a lowest value store 54. If the lowest value comparator 52 determines that the current correlation surface value CS is lower than that currently stored in the lowest value store 54, then the lowest value store is write enabled with a signal W to replace its previous contents with that of the current correlation surface value CS. After the entire correlation surface has been scanned out of the correlation surface store 32, the lowest value store 54 will store the lowest correlation surface value within the correlation surface being analysed.

The output from the subtracter 36 representing the first derivative value is fed directly to a positive value detector 56 and indirectly to a negative value detector 58 via a one value delay unit 60. The outputs from the positive value detector 56 and the negative value detector 58 are fed to a two input AND gate 62. In the case of a zero crossing going from a negative value to a positive value the outputs from the positive value detector 56 and the negative value detector 58 will be simultaneously asserted true and the output from the AND gate 62 Foin will indicate a correlation maximum in the x direction.

The output from the subtracter 44 that is representative of the second derivative is fed to a threshold value detector 64. If the second derivative exceeds a predetermined threshold value Ths. then a sharpness flag Fxs is asserted true.

The circuit of Figure 6 will be substantially duplicated to provide a circuit that simultaneously scans in the y direction and calculates and tests first and second derivative values in the y direction in the same way as those of Figure 6. The lowest value comparator 52 and the lowest value store 54 need not be duplicated.

Figure 7 illustrates a circuit for processing the outputs of the circuit of Figure 6 as well as the corresponding (not illustrated) circuit performing simultaneous analysis in the y direction. The flags Fxrnin and F indicating the presence of a correlation maximum together with the flags Fxs and F indicating sharpness are fed to a four input AND gate 66. The output from the AND gate 66 Fv indicating the detection of a sufficiently sharp corre]ation maximum is fed to one input of a two input AND gate 68 and one input of a three input AND gate 70.

The correlation surface value CS currently being read from the correlation surface store 32 is supplied to the input of a best correlation maximum store 72 and via a multiplexer 74 to the input of a next best correlation maximum store 76. The output from the AND gate 70 controls the multiplexer 74 such that if the output is false the input of the next best correlation magnitude store 76 is connected to the best correlation magnitude store 72 and if the output is true is connected directly to the correlation surface value CS.

A first comparator 78 determines if the correlation surface value CS is less than the value currently stored in the best correlation magnitude store 72 (i.e. with a non-inverted surface a lower value indicates better correlation). A second comparator 80 detects whether the correlation surface value CS is less than the value currently stored in the next best correlation magnitude store 76. If the first comparator 78 gives a true result and the flag Fy is true, then the AND gate 68 write enables the best correlation magnitude store 72 and the next best correlation magnitude 76 via an OR gate 82.If the correlation surface value CS is less than the value in the next best correlation magnitude store 76 but greater than the value in the best correlation magnitude store 72 and the flag Fv is true, then the AND gate 70 write enables the next best correlation magnitude 76 via the OR gate 82 to directly receive the correlation surface value CS via the multiplexer 74.

The two stores 72, 76 are initialised with default high values at the beginning of each scan.

The combined action of the elements of the circuit of Figure 7 is to detect and store the best and next best correlation maxima encountered during a complete scan. The best magnitude correlation store 72 can also include a section for storing the address value produced by the address generator 34 that is associated with the correlation value it is storing since this address value is indicative of the motion vectdr associated with that correlation maximum.

The circuit of Figure 8 is operative at the end of the scan of a correlation surface to compare the best correlation maximum found with the lowest value stored in the lowest value store 54 using a comparator 84. A subtracter 86 determines the difference between the correlation magnitude of the best correlation maximum encountered and the next best correlation maximum encountered and feeds this to a comparator 88 where a test is made as to whether this difference is above an alias threshold value THA. If both of the outputs of the comparators 84, 88 are true, then the AND gate 90 will reset a flag indicating that the correlation surface should not be used as a source of motion vectors.

Figure 9 illustrates the operation of the circuit of Figure 7 in flow diagram form. Step 92 tests if all of the flags F,in' Fxs' Fymin and F are true. If this condition is not met, then the process ends.

If the condition is met, then the process passes to step 94 where the correlation surface value CS is compared with the current best correlation magnitude value CSb. If "CS < CSb?" is true, then the process passes to step 96 where the next best correlation magnitude value is set to the current best correlation magnitude value and then the best correlation magnitude value is set to the correlation surface value. The process then ends.

If the test at step 94 is false, then the process passes to step 98 where a test is made as to whether the correlation surface value CS is less than the next best correlation magnitude value CSnb. If this condition is met, then the process passes to step 100 where the next best correlation magnitude value is set to the correlation surface value. If the test at step 98 is not met, then the process ends.

Figure 10 illustrate the operation of the circuit of Figure 8 in flow diagram form. Step 102 tests if the best correlation magnitude value is equal to the lowest value found in the correlation surface.

Unless this is the case the process ends. Step 104 tests if the difference between the next best correlation magnitude value and the best correlation magnitude value exceeds a predetermined threshold ThA.

Unless this condition is also met the process ends. If both the tests at steps 102 and 104 are met, then the process passes to step 106 where a flag that otherwise prevents any motion vectors being calculated from the correlation maxima of a correlation surface is reset.

The overall operation of the circuits can be considered in the following way. The circuit of Figure 6 and the four input AND gate 66 of Figure 7 serve to identify sufficiently sharp correlation maxima.

The remainder of the circuit of Figure 7 is responsible for storing the best and next best correlation maxima encountered during the scanning of a complete correlation surface. The circuit of Figure 8 performs two further validity tests upon those sufficiently sharp correlation maxima that have been detected, i.e. that the lowest point in the correlation surface corresponds to the best correlation maximum detected and that the difference between the best correlation maximum detected and the next best correlation maximum detected exceeds a threshold value.

Claims (12)

1. Apparatus for motion analysis of moving images, said apparatus comprising: means for calculating a correlation surface representing image correlation between a search block of image data within a current image and portions of a temporally adjacent image displaced by differing displacement vectors from said search block; means for identifying points in said correlation surface corresponding to correlation maxima; means for calculating a value of a second differential of said correlation surface in at least one direction at least said points of correlation maxima; means for comparing said at least one value with a predetermined second differential threshold value in a threshold test; and means for rejecting any points of correlation maxima that fail said threshold test.

2. Apparatus as claimed in claim 1, wherein said means for calculating calculates a value of a second differential of said correlation surface in two directions.

3. Apparatus as claimed in claim 2, wherein said two directions are orthogonal.

4. Apparatus as claimed in claim 3, wherein said two directions correspond to cartesian coordinate directions in said correlation surface.

5. Apparatus as claimed in any one of the preceding claims, wherein said means for identifying calculates a value of a first differential of said correlation surface in two directions and identifies as correlation maxima points at which said values both pass from indicating a correlation increase to indication a correlation decrease.

6. Apparatus as claimed in any one of the preceding claims, comprising: means for detecting a correlation magnitude maximum point in said correlation surface at which correlation is at a maximum; and means for rejecting said correlation surface for use in motion analysis if said correlation magnitude maximum point does not correspond to a point of a correlation maximum that passed said threshold test.

7. Apparatus as claimed in any one of the preceding claims, comprising: means for determining, in a correlation surface with at least two correlation maxima, a difference value of correlation magnitude between a point of a correlation maximum having greatest correlation magnitude and a point of a correlation maximum having next greatest correlation magnitude; and means for rejecting said correlation surface for use in motion analysis if said difference value is less than a predetermined difference threshold value.

8. Apparatus for motion compensated image standards conversion including apparatus for motion analysis of moving images as claimed in any one of the preceding claims.

9. A method of motion analysis of moving images. said method comprising the steps of: calculating a correlation surface representing image correlation between a search block of image data within a current image and portions of a temporally adjacent image displaced by differing displacement vectors from said search block; identifying points in said correlation surface corresponding to correlation maxima; calculating a value of a second differential of said correlation surface in at least one direction at at least said points of correlation maxima; comparing said at least one value with a predetermined second differential threshold value in a threshold test; and rejecting any points of correlation maxima that fail said threshold test.

10. Apparatus for motion analysis of moving images substantially as hereinbefore described with reference to the accompanying drawings.

11. Apparatus for motion compensated image standards conversion substantially as hereinbefore described with reference to the accompanying drawings.

12. A method of motion analysis of moving images substantially as hereinbefore described with reference to the accompanying drawings.