GB2265516A - Motion analysis of moving images - Google Patents

Abstract

A composite correlation surface having a high resolution first correlation surface and a lower resolution second correlation surface is calculated. The composite correlation surface thus calculated is then searched for correlation maximum M1, M2. The correlation maximum M1 from within a group of equally valid correlation maxima within the composite correlation surface that has the lowest magnitude displacement vector V1 is used to determine the motion vector for that correlation surface. Once motion vectors have been identified for all of the blocks making up the image, a small group of motion vectors is picked for each block in a vector reduction step prior to further processing. The search for motion vectors for inclusion in this group uses a predetermined selection test and takes place in a primary set of blocks abutting the target block in question and then in a secondary set of blocks surrounding the target block but spaced from the primary set blocks, <IMAGE>

Description

MOTION ANALYSIS OF MOVING IMAGES This invention relates to motion analysis of moving images. More particularly, but not exclusively, this invention relates to motion analysis of moving images as part of motion compensated moving image standards conversion, e.g. high definition television format to 50Hz 2:1 PAL television format.

A motion compensated video standards converter is described in British Published Patent Application GB-A-2 231 749 (Sony Corporation).

The system of GB-A-2 231 749 performs motion compensated interpolation to derive a sequence of output images from a sequence of input images having a different format. As illustrated in Figure 1 of GB-A-2 231 749 (a copy of which is reproduced herein as Figure 1 of the accompanying drawings), the standards converter comprises a sequence of processing units.

A video signal is supplied to an input terminal 1. The input terminal 1 is connected to a progressive scan converter 2 in which the input video fields are converted into video frames which are supplied to a direct block matcher 3 wherein correlation surfaces are created.

These correlation surfaces are analyzed by a motion vector estimator 4, which derives and supplies motion vectors to a motion vector reducer 5, wherein the number of motion vectors for each pixel is reduced, before they are supplied to a motion vector selector 6 which also receives an output from the progressive scan converter 2. Any irregularity in the selection of the motion vectors by the motion vector selector 6 is removed by a motion vector post processor 7 from which the processed motion vectors are supplied to and control an interpolator 8, which also receives an input from the progressive scan converter 2. The output from the interpolator 8, which is a standards-converted and motion-compensated video signal, is supplied to an output terminal 9.

Such standards converters require a large amount of data processing to be performed for each output image being interpolated.

Measures which can speed up this processing and thereby ease hardware requirements are highly advantageous. Similarly, whilst images produced by motion-compensated standards converters are generally of good quality, measures which improve this quality further are highly advantageous.

In known standards converters the correlation surface is calculated over an array of equally spaced points each corresponding to a different candidate displacement vector This calculation of the correlation surface requires a large amount of data processing to be performed and this can have a detrimental effect on the overall speed of the system.

Viewed from a first aspect the invention provides apparatus for motion analysis of moving images, said apparatus comprising: means for calculating a first correlation surface representing image correlation between a first search block of image data within a current image and first portions of a temporally adjacent image displaced by differing displacement vectors from said first search block, and means for calculating a second correlation surface representing image correlation between a second search block of image data within said current image and second portions of said temporally adjacent image displaced by differing displacement vectors from said second search block, wherein said first portions extend over a first search area that is smaller than and concentric with a second search area over which said second portions extend, and said first correlation surface is calculated to a higher spatial resolution than said second correlation surface.

The invention recognises that the subjective effect of a given error in an estimated motion vector will be greater for a small magnitude motion vector than for a large magnitude motion vector; if an object is rapidly moving across an image, then it is difficult for the human eye to accurately track it and a perturbation in its motion will be less visually apparent. This is exploited by calculating the portion of the correlation surface corresponding to low magnitude displacement vectors to a higher spatial resolution than that corresponding to high magnitude displacement vectors. In this way, the amount of processing required to produce the correlation surface can be reduced without a subjective degradation in the quality of the interpolated output images.The first correlation surface corresponds to the low magnitude displacement vectors and the second correlation surface corresponds to the high magnitude displacement vectors.

It will be appreciated that the calculation of the first correlation surface and the second correlation surface may be performed on differing shapes and forms of search blocks However, in preferred embodiments of the invention said first search block and said second search block are coextensive within said current image.

This feature makes it easier to combine the results produced in calculating the first correlation surface and the second correlation surface.

A preferred way in which the spatial resolution of the first correlation surface and the second correlation surface can be made to differ is that said first search block and said first portion are of a higher spatial resolution than said second search block. This can be achieved without undue complexity if said second search block and said second portion are sub-sampled from said first search block and said first portion respectively. Preferably these sub sampled versions are also filtered.

It will be appreciated that the first correlation surface and the second correlation surface could be separately analyzed to identify correlation maxima therein and the identified correlation maxima then reconciled. However, in preferred embodiments of the invention said first correlation surface and said second correlation surface are combined to form a composite correlation surface in which said first correlation surface replaces any corresponding portions within said second correlation surface. It will also be appreciated that the correlation surfaces can be calculated in a number of different way and that in some cases a minimum in the calculated correlation surface in fact corresponds to a point of maximum correlation.

This approach of producing a combined correlation surface avoids disadvantageous complexity in the later processing steps.

In preferred embodiments of the invention said first correlation surface calculating means and said second correlation surface calculating means operate in parallel.

The distinct nature of the first correlation surface and the second correlation surface provides a separation that can be exploited to parallelize the processing and thereby increase processing speed.

It will be appreciated that the correlation surface could be calculated in several different ways, the two main ways being phase correlation and direct block matching. In preferred embodiments of the invention direct block matching is used in which said first correlation surface calculating means and said second correlation surface calculating means calculate said first correlation surface and said second correlation surface respectively to have a height at each point proportional to the magnitude of the difference in image data values at that point between said current image and said preceding image, whereby higher points in said first correlation surface or said second correlation surface correspond to points of lower correlation and a minimum in said first correlation surface or said second correlation surface corresponds to a correlation maximum.

Motion analysis apparatus using first and second correlation surfaces as discussed above is particularly useful within motion compensated image standards converters Viewed from a second aspect the invention provides a method of motion analysis of moving images, said method comprising the steps of:: calculating a first correlation surface representing image correlation between a first search block of image data within a current image and first portions of a temporally adjacent image displaced by differing dispclacement vectors from said first search block, and calculating a second correlation surface representing image correlation between a second search block of image data within said current image and second portions of said temporally adjacent image displaced by differing displacement vectors from said second search block, wherein said first portions extend over a first search area that is smaller than and concentric with a second search area over which said second portions extend, and said first correlation surface is calculated to a higher spatial resolution than said second correlation surface.

In known motion vector estimators, the motion vector has been generated from a correlation surface by searching for the largest amplitude correlation maximum in the correlation surface. Thus, when the correlation surface contains more than one correlation maximum, the largest magnitude maximum is used to determine the motion vector.

Viewed from a third aspect this invention provides apparatus for motion analysis of moving images, said apparatus comprising: means for calculating a correlation surface representing image correlation between a search block of image data within a current image and portions of a temporally adjacent image displaced by differing displacement vectors from said search block, and means for estimating a motion vector to represent motion of that part of said current image within said search block between said temporally adjacent image and said current image, wherein, when a plurality of correlation maxima of equal validity are present in said correlation surface, said estimating means estimates said motion vector as the lowest magnitude displacement vector terminating at one of said plurality of correlation maxima.

When faced with a number of apparently equally valid correlation maxima, the invention uses the correlation maximum with the lowest magnitude displacement vector to estimate the motion vector. The invention can be considered as recognising and exploiting that low speed motion resulting in correlation maxima at low displacement vectors (and therefore low magnitude motion vectors) is statistically more common within images than high speed motion. Accordingly, motion vector estimation performed in a way commensurate with this has a higher likelihood of producing good results.

It will be appreciated that the entire correlation surface may be searched (e.g. using a raster scan search pattern) and then the correlation maximum with the lowest magnitude displacement vector selected from amongst all the correlation maxima that have been detected (the correlation maxima can be subject to tests, such as threshold tests, to distinguish genuine maxima from variations in the background noise in the correlation surface). However, in preferred embodiments of the invention said estimating means searches for correlation maximum by testing points within said correlation surface of monotonically increasing displacement vector magnitude, whereby the first correlation maximum encountered from among the group of apparently equally valid correlation maxima is used to estimate said motion vector.

This feature recognises that once you have found the correlation maximum that has the lowest magnitude displacement vector from among this group then there is no point in searching the rest of the correlation surface for similar correlation maxima.

In preferred embodiments of the invention said correlation surface calculating means stores data representing said correlation surface in an addressable memory In this way, the functions of calculating the correlation surfaces and then analyzing the correlation surfaces can be conveniently separated and pipeline processed.

Particularly high speed motion vector estimation may by performed in preferred embodiments of the invention in which: said estimating means includes a logic gate array configured to detect correlation maximum in said correlation surface from data received from said addressable memory.

A simple and effective way of implementing the preferred search pattern through the correlation surface is that said estimating means includes an address generator for generating addresses within said addressable memory from which data is passed to said logic gate array in an order such that said logic gate array tests for correlation maximum at points within said correlation surface of monotonically increasing displacement vector magnitude.

It will be appreciated that the search block could take differing shapes and forms However, a simplified implementation can be achieved when said search block is a rectangular array of image data elements from within said current image Similarly, in preferred embodiments of the invention said portions of said preceding image form a rectangular search area within said temporally adjacent image.

As previously mentioned, correlation surfaces can be calculated in many different ways. However, in preferred embodiments of the invention said correlation surface calculating means calculates said correlation surface to have a height at each point proportional to the magnitude of the difference in image data values at that point between said current image and said preceding image, whereby higher points in said first correlation surface or said second correlation surface correspond to points of lower correlation and a minimum in said first correlation surface or said second correlation surface corresponds to a correlation maximum.

The correlation surface calculation technique may be adapted to more effectively exploit the invention by providing that said correlation surface calculating means weights said correlation surface to have higher values for lower displacement vector magnitudes.

This weighting towards lower displacement vector magnitudes increases the likelihood that the resulting motion vectors will be reliable low magnitude motion vectors.

It will be appreciated that the invention is particularly suited for use in motion compensated image standards conversion where output image quality and high processing speed are of considerable importance.

Viewed from a fourth aspect the invention provides a method of motion analysis of moving images, said method comprising the steps of: calculating a correlation surface representing image data correlation between a search block of image data within a current image and portions of a temporally adjacent image displaced by differing displacement vectors from said search block, and estimating a motion vector to represent motion of that part of said current image within said search block between said temporally adjacent image and said current image, wherein, when a plurality of correlation maxima of equal validity are present in said correlation surface, said estimation takes as said motion vector the lowest magnitude displacement vector terminating at one of said plurality of correlation maxima.

In known standards converters, a motion vector reducer is provided downstream of the motion vector estimator. The function of the motion vector reducer is to provide a small set of motion vectors for each part of the image for use in interpolating that part of the image. The small set of motion vectors is chosen from the motion vectors calculated for the full image. In known systems with rectangular search blocks, the small set of motion vectors for a given target block is chosen from the motion vector for the target block itself, the motion vectors for the eight blocks abutting the target block and any global motion vectors.

It is important that the motion vectors produced by the motion vector reducer should represent a broad cross section of different types of motion vector, i.e. a mixture of local motion vectors and global motion vectors. Providing a broad cross section of types of motion vector for use in interpolating each part of the image increases the likelihood of being able to use a motion vector closely matched to the true motion within the image. This is desirable to produce good quality interpolated output images.

The above requirements are complicated by the need to ensure that erroneous/spurious motion vectors are not used in interpolation. To achieve this, the motion vectors are subject to validity checks based upon the expected characteristics of valid motion vectors. If a particular motion vector fails the validity check, then it is marked as invalid and will not be included within the small set of motion vectors produced by the motion vector reducer.

Viewed from a fifth aspect the invention provides apparatus for motion analysis of moving images, said apparatus comprising: means for calculating an array of motion vectors for a corresponding array of blocks within a current image, said motion vectors representing motion of that part of said current image within each of said blocks between a temporally adjacent image and said current image, means for compiling a plurality of motion vectors to be associated with each block, as a current target block, for use in further analysis, said means for compiling comprising:: means for detecting if said motion vectors within said array of motion vectors are valid motion vectors that meet a predetermined validity test, means for associating with said current target block, if valid, that motion vector calculated for said target block, and local vector supplementing means for associating with said current target block, if valid, differing from any previously associated motion vectors and satisfying a predetermined selection test, one or more supplementary motion vectors taken from one or more sets of blocks that surround said current target block.

The invention recognises that the selection of which supplementary motion vectors should be used can have a significant impact upon the accuracy of the interpolation achievable. To this end, the invention provides an addition layer of sophistication by introducing a predetermined selection test to determine those motion vectors from the one or sets of blocks that surround the target block that are best suited. The previous technique was to go through the vectors from the eight abutting blocks in a fixed, and somewhat arbitrary, order and select the first valid and not previously selected ones that were reached.

One preferred predetermined selection test is implemented in systems in which said means for compiling comprises: means for determining if any of said motion vectors from said one or more sets of blocks occurs a plurality of times within all of said motion vectors in said one or more sets of blocks, and wherein said local vector supplementing means associates any motion vectors with plural occurrences prior to those with a single occurrence.

Another advantageous selection test is implemented in systems in which said means for compiling comprises: means for determining if any of said motion vectors from said one or more sets of blocks have a direction towards said current target block, and wherein said local vector supplementing means associate any motion vectors having a direction towards said current target block prior to those not having a direction towards said current target block.

Another feature of preferred embodiments of the invention is that said local vector supplementing means takes supplementary vectors from a primary set of blocks that are closest to said current target block and then, if sufficient supplementary motion have not been associated with said current target block, from a secondary set of blocks that surround said current target block and are spaced from said primary set of blocks by intervening blocks.

This feature recognises that if a given motion vector is invalid, then there is an increased probability that the abutting motion vectors will also be invalid. As a consequence of this, if a given motion vector is invalid, then sometimes all eight abutting motion vectors will also be invalid. In this circumstance, the known system will include only global and zero motion vectors within the small set of motion vectors for that block. The absence of a local motion vector within the small set is likely to degrade the quality of an interpolated output image at that point.

Having recognised this problem, this feature provides a primary set of blocks that are closest to the current target block and a secondary set of blocks surrounding the current target block but spaced from the primary set of blocks. Embodiments of the invention including this preferred feature search first for local vectors in the primary set of blocks and then, if necessary, search the secondary set of blocks. The spacing between the primary set of blocks and the secondary set of blocks increases the likelihood that the motion vectors of the blocks from the secondary set of blocks will be valid even if those of the primary set of blocks are invalid.

If it is the case that only the current target block is invalid, then local vector information can be found from spatially close blocks within the primary set of blocks which are most likely to be similar to the value that should have been obtained for the current target block.

However, if the problem is one which means that both the current target block and some or all of the primary set of blocks have invalid motion vectors, then at least some local vector information will be extracted from the secondary set of blocks and the system will not have to solely rely on global vectors.

As previously mentioned, if local vector information is not available, then it is preferably replaced with global vector information. To this end, in preferred embodiments of the invention said means for compiling comprises: global vector supplementing means, operative if said local vector supplementing means has not been able to associate sufficient supplementary motion vectors with said current target block, for associating with said current target block one or more global vectors derived from motion of said current image as a whole.

The blocks could take many different shapes and forms, but in advantageously less complex embodiments of the invention, said blocks are rectangular arrays of image data elements from within said current image With such rectangular blocks, in preferred embodiments, said current target block abuts eight touching blocks with said primary set of blocks comprising every alternate block from said eight touching blocks.

Only every alternate block is included within the primary set since it is likely that if these alternating blocks are all invalid then the intervening blocks will also be invalid. It is an inefficient use of processing time to check all of the touching blocks for validity.

With the above arrangement it has been found advantageous that said secondary set of blocks comprise four blocks spaced from said current target block by those blocks of said eight touching blocks not included in said primary set of blocks. This arrangement provides a good balance between the secondary set of blocks having a low probability of sharing the same invalidity as the primary sets of blocks and yet not being too distant from the current target block.

The predetermined validity test could take many forms, but is preferably a simple to implement correlation surface amplitude threshold criterion.

The invention provides a motion vector reducer that is particularly suitable for use in motion compensated image standards conversion.

Viewed from a sixth aspect the invention provides a method of motion analysis of moving images, said method comprising the steps of: calculating an array of motion vectors for a corresponding array of blocks within a current image, said motion vectors representing motion of that part of said current image within each of said blocks between a temporally adjacent image and said current image, compiling a plurality of motion vectors to be associated with each block, as a current target block, for use in further analysis, said step of compiling comprising the steps of:: detecting if said motion vectors within said array of motion vectors are valid motion vectors that meet a predetermined validity test, associating with said current target block, if valid, that motion vector calculated for said target block, and associating with said current target block, if valid, differing from any previously associated motion vectors and satisfying a predetermined selection test, one or more supplementary motion vectors taken from one or more sets of blocks that surround said current target block.

An embodiment of the invention will now be described, by way of example only, with reference to the accompanying drawings in which: Figure 1 is a block diagram of a motion compensated moving image standards converter of the type disclosed in GB-A-2 231 749; Figure 2 shows an inverted correlation surface; Figure 3 illustrates the use of different spatial resolutions for the central and peripheral portions of a correlation surface; Figure 4 schematically illustrates the manner in which the correlation surface of Figure 3 is calculated; Figure 5 illustrates a circuit for calculating the correlation surface of Figures 3 and 4; Figures 6A and 6B illustrate a potentially troublesome situation for motion vector identification; Figures 7A and 7B illustrate inverted correlation surfaces calculated from the images in Figures 6A and 6B;; Figures 8A, 8B and 8C illustrate scanning patterns for searching for correlation maxima in a correlation surface; Figures 9A and 9B illustrate the variation in displacement vector magnitude during the scanning patterns of Figures 8B and 8C; Figure 10 illustrates a circuit for controlling the scanning of a correlation surface; Figure 11 illustrates one strategy for motion vector reduction; Figure 12 illustrates another strategy for motion vector reduction; Figures 13A and 13B illustrate one reduction process; and Figures 14A and 14B illustrate another reduction process.

Figure 2 illustrates an inverted correlation surface. The X and Y axis represent the position within a search area in one frame where each test for a matching pattern to that from a search block within a temporally adjacent frame was made. This correlation test is made by summing the difference in amplitude values for the pixels within the search block compared to the corresponding pixels at that position in the search area. A high resulting value represents low correlation and a low resulting value represents high correlation The correlation surface shown in Figure 2 has had its Z axis inverted. Accordingly, a maximum 2 in the correlation surface of Figure 2 corresponds to a correlation maximum.

Figure 3 illustrates the spatial resolution of the correlation surface. The composite correlation surface 4 in Figure 3 can be considered to be composed of a first high resolution correlation surface 6 and a second low resolution correlation surface 8. The first correlation surface 6 is smaller than and concentric with the second correlation surface 8. In this illustration, the first correlation surface 6 has a spatial resolution four times that of the second correlation surface 8. Intersections in the grid of lines drawn in Figure 3 represent points at which a correlation test is made within the corresponding search area.

Figure 4 illustrates the relationship between the way in which the image being tested is searched for areas of high correlation and the composite correlation surface 4 of Figure 3. The search is carried out by performing the previously described correlation calculation for a search block 12 at a sequence of differing positions within a search area 10 of a temporally adjacent image.

The inner area 14 within the boundary of crosses is that part of the search area 10 that corresponds to the first correlation surface 6 of a higher spatial resolution. The outer area 16 outside of the boundary of crosses is that part of the search area 10 that corresponds to the second correlation surface 8 of a lower spatial resolution.

Within the outer area 16 the search block 12 is moved to positions centred on a sequence of points A, B, C etc. These points are spaced apart by a distance 6. At each point a correlation value representing the degree of image match is calculated as discussed above.

Within the inner area 14 the search block 18 is again moved to a sequence of positions entered on points X, Y, Z, etc. These positions are separated by a distance 6/2. At each point a correlation value representing the degree of image match is calculated as discussed above.

The search block 12 is a sub sampled version (which may also be filtered) of the search block 18. Similarly, the outer area 16 is a sub sampled version (which may also be filtered) of the current image whereas the inner area 14 is a full resolution representation of the corresponding part of the current image. In this way, movement between adjacent pixel positions within the outer area 16 will correspond to a greater spatial displacement within the current image than movement between adjacent pixels within the inner area. This is reflected by the differing spacing of the points A, B, C and points X, Y, Z in Figure 4.

To illustrate the reduction in the number of calculations that need be performed when using this invention consider the following examples: Example 1 The search area measures 80*48 pixels and the search block measures 16*16 pixels. The range of movement of the search block within the search area is +32 pixels in the 80 pixel direction and t16 pixels in the 48 pixel direction, Accordingly, the resulting correlation surface will have 64*32 points at which it will be necessary to calculate a correlation value by performing 16*16 difference calculations. This results in a total of 524,288 difference calculations being performed for each correlation surface. At each point within the correlation surface the 16*16 difference calculations for that point will also need to be summed to produce the correlation value.This results in a total of 2048 sum calculations being performed for each correlation surface.

Example 2 An inner area and an outer area having different resolution characteristics are used.

Within an inner area measuring 48*32 pixels, a search block measuring 16*16 pixels is used. Within this inner area, this corresponds to a range of motion of #16 pixels in the 48 pixel direction and 8 pixels in the 32 pixel direction. Accordingly, the corresponding first high spatial resolution correlation surface has a number of points equal to 32*16. As above, for each of the points within the first correlation surface, 16*16 difference calculations must be performed. This results in a total of 131,672 difference calculations being needed for the first correlation surface. For each point within the first correlation surface a sum of the 16*16 differences is needed. This corresponds to 512 sum calculations for the first correlation surface.

In order to calculate the second correlation surface a 2:1 sub sampled version of the full 80*48 search area of Example 1 is used. In this example, the second correlation surface in fact includes in its centre portions regions for which more accurate determinations have been made in the first correlation surface. The sub sampled search area for the second correlation surface has a size of 40*24 pixels.

The corresponding sub sampled search block has a size of 8*8 pixels.

The range of pixel movements within the sub sampled search area are t16 in the 40 pixel direction and #8 in the 24 pixel direction.

Accordingly the second correlation surface has a size of 32*16. In the case of the second correlation surface, for each point within the correlation surface, 8*8 difference calculations are needed. Thus, the total number of difference calculations needed to generate the second correlation surface is 32,768. At each point within the correlation surface, a sum calculation must also be performed corresponding to a total of 512 sum calculations for the second correlation surface.

Comparing Example 1 and Example 2, it can be seen that the total number of difference calculations has been reduced from 524,288 to 163,840 and the total number of sum calculations has been reduced from 2048 to 1024. This advantageous reduction of the number of calculations that need be performed has been achieved at the expense of reduced spatial resolution for larger amplitude displacement vectors falling within the second correlation surface. However, as previously discussed, the subjective impact of the lower spatial resolution to which large amplitude motion vectors can be calculated does not significantly degrade the perceived image quality.

Figure 5 illustrates a circuit for calculating the correlation surface discussed in relation to Figures 3 and 4. A current frame is stored within a frame store 20. A previous frame is stored within frame store 22. A search block selector 24 picks out an n*m search block from within the current image. The function of the circuit is to search within the previous frame for a corresponding n*m block that matches the search block. An array of n*m pixel values from the search block are fed down a bus 26 to a correlation evaluator 28. Arrays of n*m pixel values from differing positions within the search area of the previous frame are sequentially fed via a bus 30 to the correlation evaluator 28.

An address generator 32 produces an address that indicates the position within the search area from which the n*m pixel values will be selected for feeding via the bus 30 to the correlation evaluator 28.

This address value is also fed to the search block selector 24, the correlation evaluator 28 and the correlation surface memory 34. The sequence of address values generated by the address generator 32 is conveniently produced by having a counter 36 addressing a PROM 38 for mapping an incrementing count value into a predetermined sequence of addresses. The address value fed to the correlation surface memory 34 is used to control the position within the correlation surface memory 34 at which each output value from the correlation evaluator 28 is stored.

The address value fed to the correlation evaluator 28 is used to control the application of a weighting function to the correlation values generated by the correlation evaluator 28. This weighting function serves to increase the indicated correlations for smaller amplitude displacement vectors. In addition, the address value fed to the correlation evaluator 28 serves to switch the correlation evaluator 28 between a high resolution calculation mode and a low resolution calculation mode with different numbers of differences calculations being employed in dependence upon whether the address value indicates a position within the first correlation surface or the second correlation surface.

The address value fed to the search block selector 24 also serves to control the search block selector 24 to either feed a full resolution or sub sampled search block onto the bus 26 in dependence upon whether the address value indicates a position within the first correlation surface or the second correlation surface.

In the circuit of Figure 5 the calculation of the first correlation surface and the second correlation surface are not performed in parallel. It will be appreciated that parallel operation can be readily achieved by providing two circuits such as that illustrated in Figure 5 with one permanently switched into low resolution calculation and one permanently switched into high resolution calculation. The respective first and second correlation surfaces stored within the respective correlation surface memories could then be superimposed to generate a composite correlation surface.

In practice, all that need be done is an appropriate switching of the point at which correlation surface data is read from the respective correlation surface memories in dependence upon the position within the composite correlation surface.

Figures 6A and 6B illustrate two temporally adjacent images. The image of Figure 6A precedes the image of Figure 6B. Both images include a human figure 40 and a cross 42. The human figure 40 is moving leftwards and the cross 42 is moving leftwards and downwards.

In the illustrated example, it is desired to determine a motion vector representing the movement of the cross 42 between the image of Figure 6A and the image of Figure 6B. This motion vector could then be used to interpolate an image temporally intermediate of the images of Figures 6A and 6B.

A search block 44 surrounding the cross 42 is tested at various positions within the image of Figure 6B. By chance, the central portion of the human figure 40 is a good correlation match to the search block 44. The motion vector for the true movement is V1 and the motion vector corresponding to the false correlation match from the central portion of the human figure 40 is V2.

Figure 7A represents the inverted correlation surface generated for the search block 44 within the image of Figure 6B. The correlation surface shows a correlation maximum M1 corresponding to the true motion vector and a correlation maximum M2 corresponding to the false motion vector. Given the finite resolution of the correlation determining process and normal variations between pixel values representing the same object between frames (e.g. an object may move into shade), it is quite possible that the false correlation maximum M2 will have the same magnitude as the true correlation maximum M1. In this case some way is needed to help choose the correct result. The point C within the correlation surface represents the centre point of the correlation surface corresponding to a zero displacement of the object between adjacent frames.

Figure 7B schematically illustrates a plan view of the correlation surface of Figure 7A with the correlation maximum M1 and M2 illustrated by elevation contours. As can be seen from Figure 7B, the magnitude of the displacement vector V2 for the false maximum M2 is greater than the displacement vector V1 for the true correlation maximum M1. As a general rule, in a correlation surface with more than one correlation maximum of equal height (or depth) and therefore validity, the true correlation maximum is most likely to be that with the smallest displacement vector magnitude. Whilst this rule does not hold in every case, its use provides an overall improvement in motion vector identification. Thus the displacement vector V, should be used as the motion vector.

Figures 8A, 8B and 8C illustrate various patterns in which the correlation surface may be searched to identify correlation maxima.

Figure 8A illustrates a simple raster scan search pattern. As each correlation maximum is encountered during this search pattern, its corresponding displacement vector magnitude is compared with that of any previously identified correlation maxima. If the displacement vector magnitude of the newly identified correlation maximum is less than that of the previously stored correlation maximum then the newly encountered correlation maximum replaces that previously stored as the current best candidate correlation maximum. At the end of the raster scan search, the correlation maximum and corresponding displacement vector stored will be the one with the smallest displacement vector magnitude that was encountered during the search pattern.

Figure 8B illustrates another possible search pattern. In this search pattern, points laying along an outwardly expanding spiral from the centre of the correlation surface are sequentially tested to see if they represent a correlation maximum. In this way, points with a monotonically increasing displacement vector magnitude are sequentially tested. This search pattern ensures that the first correlation maximum encountered within a group of equally valid maxima will be that having the smallest displacement vector magnitude.

Whilst the search pattern of Figure 8B represents an elegant ideal, in practice for finite resolution correlation surfaces, a pure spiral search pattern cannot readily be achieved. The search pattern of Figure 8C illustrates a solution. The number within each square illustrates the order in which the point in the correlation surface at the centre of that square is tested to see if it represents a correlation maximum. It will be seen that the order illustrated in Figure 8C again shows a monotonically increasing variation in displacement vector magnitude.

Figures 9A and 9B represent the variation in displacement vector magnitude for the search patterns of Figures 8B and 8C respectively.

It will be seen that whilst the idealised search pattern of Figure 8B results in a smoothly increasing displacement vector magnitude as shown in Figure 9A, the finite resolution of the system and the accompanying search pattern of Figure 8C result in a staircase like increase in displacement vector magnitude as shown in Figure 9B. The displacement vector magnitude illustrated in Figure 9B is nonetheless still monotonically increasing since it only ever increases or stays the same and never decreases.

Figure 10 illustrates a circuit for carrying out the search patterns of Figures 8A, 8B and 8C. The correlation surface calculated by the circuit of Figure 5 is stored within the correlation surface memory 46. As each point in the correlation surface is tested, an array of pixel values 48 centred upon the pixel under test (indicated by a cross) is fed via a multibit bus to a logic gate array 50. The logic gate array provides a high speed hardwired comparison of the correlation value for the point under test compared to its surrounding points. If the correlation value at the point under test is higher than its surrounding points and meets other test criteria, such as a threshold criteria, then it is identified as a maximum and the address of the pixel under test is stored in the maximum location memory 52 upon issue of a MAX signal from the logic gate array 50.

ThW sequence in which the points within the correlation surface are tested is controlled by a counter 54 whose incrementing count value is mapped via a PROM address map 56 into a predetermined sequence of address positions within the correlation surface. By appropriately programming the PROM address map 56, any one of the desired search patterns illustrated in Figures 8A and 8C may be performed.

It is known to provide systems such as motion compensated standards converters of the type described in British Published Patent Application GB-A-2 231 749 with a processing stage termed 'vector reduction'. In this stage a small selection of appropriate motion vectors is determined for each area within the image. Subsequently, a pixel by pixel selection of one vector from amongst this small set of motion vectors is made in a 'vector selection' stage and this selected motion vector is then used to interpolate that pixel of the image to which it relates.

It is important that an appropriate set of motion vectors should be provided from which this selection can be made. It is advantageous to have local motion vector information derived from that particular location itself or a closely adjacent location.

It will be appreciated that motion vector analysis is a complex multistage process which can break down at many different points.

Accordingly, it is not uncommon that, for a particular block of an image, a motion vector has not been calculated. In this circumstance, it is been previously proposed to adopt the search pattern illustrated in Figure 11 to look for a substitute motion vector from the abutting eight blocks. The blocks are tested in the order indicated to see if they contain a valid motion vector and if so this is selected for the target block. Even in the case where the target block has a valid motion vector, it is advantageous to select one or more other local motion vectors from the surrounding blocks for potential use in subsequent processing.

With the search pattern illustrated in Figure 11, the fixed, and somewhat arbitrary, order in which local vector information is chosen can result in degraded performance. In addition, there is a disadvantageously high probability that if the target block in question has an invalid motion vector, then all the surrounding eight blocks will also suffer from the same defect. In this case, no local motion vector information would be included within the set of motion vectors between which a selection must be made. The absence of any local motion vector can degrade the quality of the interpolation made.

The search pattern illustrated in Figure 12 provides a primary set of supplementary blocks (1, 2, 3, 4) and a secondary set of supplementary blocks (5, 6, 7, 8). The primary set of supplementary blocks directly abut the central target block. If there is no large scale error then local vector information can be extracted from the target and the primary supplementary blocks. Since this local vector information is extracted from that part of the image closest to the part on which interpolation is to be performed, it is likely to be the most accurate.

However, if a large area error has occurred, then the provision of the secondary supplementary blocks spaced from the primary supplementary blocks means that any error in the target and primary supplementary blocks is unlikely to also affect the secondary supplementary blocks. In this case, although a large scale error has occurred, at least some local vector information can be extracted and there is no need for sole reliance on global and zero motion vectors.

One way of providing the selection test in the motion vector reduction process is provided by the strategy illustrated in Figures 13A and 13B. Consider the example that is desired to select three local motion vectors for use in subsequent processing.

At steps 58 and 60 the motion vector for the target block in question is selected if it is valid.

At step 62 all of the motion vectors from the primary and secondary sets are compared and any which occur more than once are marked.

At step 64 vectors that are valid and occur more than once are selected from the primary set.

At step 66 a test is made as to whether any more motion vectors need be selected. If motion vectors are still required, then step 68 selects those motion vectors from the primary set that are valid and have not already been selected.

Step 70 again checks whether more motion vectors are required.

If more motion vectors are still required, then step 72 selects any motion vectors from the secondary set that occur more than once and are valid.

Step 74 again checks whether any more motion vectors are required. If more motion vectors are required, then step 76 selects those motion vectors from the secondary set that are valid and have not already been selected.

Steps 78 and 80 fill any remaining requirement for motion vectors with global motion vectors.

Overall, this selection strategy selects vectors in the order: 1. Valid target block vector.

2. Valid primary set vectors occurring more than once.

3. Valid primary set vectors.

4. Valid secondary set vectors occurring more than once.

5. Valid secondary set vectors.

6. Global vectors Another advantageous strategy is illustrated in Figures 14A and 14B. At steps 82 and 84, the vector for the target block is selected if it is valid.

At step 86, all the vectors from the primary and secondary sets are tested to see if they point towards the target block. Any vectors that meet this requirements are marked.

At steps 88 and 92, the vectors in the primary and then, if necessary (step 90), secondary sets are examined and any that are valid and are marked as converging are selected.

At step 94, a test is made as to whether any more vectors are required If more vectors are required, then steps 96 and 100 select vectors from the primary and then, if necessary (step 98), secondary sets that are valid.

At step 102, another test as to whether more vectors are required is made. If more vectors are required, then global vectors are selected at step 104.

Overall, this selection strategy selects vectors in the order: 1. Valid target block vector.

2. Valid primary set vectors converging towards the target block.

30 Valid secondary set vectors converging towards the target block.

4. Valid primary set vectors.

5. Valid secondary set vectors.

6. Global vectors It will be appreciated that the strategies illustrated in Figures 13A, 13B, 14A and 14B can be implemented on a general purpose computer that is appropriately programmed. If speed is critical requirement, then appropriate logic can be hardwired.

It will also be appreciated that a combination of the tests described above could be used.

In a preferred high speed embodiment, all the possible valid supplementary motion vectors are calculated in order of desired use and stored onto a FIFO memory. By subsequently reading the stored vectors one at a time from the FIFO memory, they will have the correct priority sequence. Thus, if a set of three vectors is required for a block, then the first three values will be read off the FIFO memory and these will correspond to the highest priority suitable vector that were previously identified. This approach removes the need for repeated test steps. as shown in Figures 13A, 13B, 14A and 14B, to determine if sufficient vectors have yet been identified.

Claims (34)

1. Apparatus for motion analysis of moving images, said apparatus comprising: means for calculating a first correlation surface representing image correlation between a first search block of image data within a current image and first portions of a temporally adjacent image displaced by differing displacement vectors from said first search block, and means for calculating a second correlation surface representing image correlation between a second search block of image data within said current image and second portions of said temporally adjacent image displaced by differing displacement vectors from said second search block, wherein said first portions extend over a first search area that is smaller than and concentric with a second search area over which said second portions extend, and said first correlation surface is calculated to a higher spatial resolution than said second correlation surface.

2. Apparatus according to claim 1, wherein said first search block and said second search block are coextensive within said current image.

3. Apparatus according to any one of claims 1 and 2, wherein said first search block and said first portion are of a higher spatial resolution than said second search block.

4. Apparatus according to claims 2 and 3, wherein said second search block and said second portion are sub-sampled from said first search block and said first portion respectively.

5. Apparatus according to any one of the preceding claims, wherein said first correlation surface and said second correlation surface are combined to form a composite correlation surface in which said first correlation surface replaces any corresponding portions within said second correlation surface.

6 Apparatus according to any one of the preceding claims, wherein said first correlation surface calculating means and said second correlation surface calculating means operate in parallel

7. Apparatus according to any one of the preceding claims, wherein said first correlation surface calculating means and said second correlation surface calculating means calculate said first correlation surface and said second correlation surface respectively to have a height at each point proportional to the magnitude of the difference in image data values at that point between said current image and said preceding image, whereby higher points in said first correlation surface or said second correlation surface correspond to points of lower correlation and a minimum in said first correlation surface or said second correlation surface corresponds to a correlation maximum.

8. Apparatus for motion compensated image standards conversion including apparatus for motion analysis of moving images according to any one of the preceding claims.

9. A method of motion analysis of movingcimages, said method comprising the steps of: calculating a first correlation surface representing image correlation between a first search block of image data within a current image and first portions of a temporally adjacent image displaced by differing displacement vectors from said first search block, and calculating a second correlation surface representing image correlation between a second search block of image data within said current image and second portions of said temporally adjacent image displaced by differing displacement vectors from said second search block. wherein said first portions extend over a first search area that is smaller than and concentric with a second search area over which said second portions extend, and said first correlation surface is calculated to a higher spatial resolution than said second correlation surface.

10. Apparatus for motion analysis of moving images, said apparatus comprising: means for calculating a correlation surface representing image correlation between a search block of image data within a current image and portions of a temporally adjacent image displaced by differing displacement vectors from said search block, and means for estimating a motion vector to represent motion of that part of said current image within said search block between said temporally adjacent image and said current image, wherein, when a plurality of correlation maxima of equal validity are present in said correlation surface, said estimating means estimates said motion vector as the lowest magnitude displacement vector terminating at one of said plurality of correlation maxima.

11. Apparatus according to claim 10, wherein said estimating means searches for correlation maximum by testing points within said correlation surface of monotonically increasing displacement vector magnitude, whereby the first correlation maximum encountered is used to estimate said motion vector.

12. Apparatus according to anyone of claims 10 or 11, wherein said correlation surface calculating means stores data representing said correlation surface in an addressable memory.

13. Apparatus according to claim 12, wherein said estimating means includes a logic gate array configured to detect correlation maximum in said correlation surface from data received from said addressable memory.

14. Apparatus according to claims 11, 12 and 13, wherein said estimating means includes an address generator for generating addresses within said addressable memory from which data is passed to said logic gate array in an order such that said logic gate array tests for correlation maximum at points within said correlation surface of monotonically increasing displacement vector magnitude.

15. Apparatus according to any one of claims 10 to 14, wherein said search block is a rectangular array of image data elements from within said current image.

16. Apparatus according to any one of claims 10 to 15, wherein said portions of said preceding image form a rectangular search area within said temporally adjacent image.

17. Apparatus according to any one of claims 10 to 16, wherein said correlation surface calculating means calculates said correlation surface to have a height at each point proportional to the magnitude of the difference in image data values at that point between said current image and said preceding image, whereby higher points in said first correlation surface or said second correlation surface correspond to points of lower correlation and a minimum in said first correlation surface or said second correlation surface corresponds to a correlation maximum.

18. Apparatus according to any one of claims 10 to 17, wherein said correlation surface calculating means weights said correlation surface to have higher values for lower displacement vector magnitudes.

19. Apparatus for motion compensated image standards conversion including apparatus for motion analysis of moving images according to any one of claims 10 to 18.

20. A method of motion analysis of moving images, said method comprising the steps of: calculating a correlation surface representing image data correlation between a search block of image data within a current image and portions of a temporally adjacent image displaced by differing displacement vectors from said search block, and estimating a motion vector to represent motion of that part of said current image within said search block between said temporally adjacent image and said current image, wherein, when a plurality of correlation maxima of equal validity are present in said correlation surface, said estimation takes as said motion vector the lowest magnitude displacement vector terminating at one of said plurality of correlation maxima.

21. Apparatus for motion analysis of moving images, said apparatus comprising: means for calculating an array of motion vectors for a corresponding array of blocks within a current image, said motion vectors representing motion of that part of said current image within each of said blocks between a temporally adjacent image and said current image, means for compiling a plurality of motion vectors to be associated with each block, as a current target block, for use in further analysis, said means for compiling comprising:: means for detecting if said motion vectors within said array of motion vectors are valid motion vectors that meet a predetermined validity test, means for associating with said current target block, if valid, that motion vector calculated for said target block, and local vector supplementing means for associating with said current target block, if valid, differing from any previously associated motion vectors and satisfying a predetermined selection test, one or more supplementary motion vectors taken from one or more sets of blocks that surround said current target block.

22. Apparatus according to claim 21, wherein said means for compiling comprises: means for determining if any of said motion vectors from said one or more sets of blocks occurs a plurality of times within all of said motion vectors in said one or more sets of blocks, and wherein said local vector supplementing means associates any motion vectors with plural occurrences prior to those with a single occurrence.

23. Apparatus according to claim 21, wherein said means for compiling comprises: means for determining if any of said motion vectors from said one or more sets of blocks have a direction towards said current target block, and wherein said local vector supplementing means associate any motion vectors having a direction towards said current target block prior to those not having a direction towards said current target block.

24. Apparatus according to any one of claims 21, 22 or 23, wherein said local vector supplementing means takes supplementary vectors from a primary set of blocks that are closest to said current target block and then, if sufficient supplementary motion have not been associated with said current target block, from a secondary set of blocks that surround said current target block and are spaced from said primary set of blocks by intervening blocks.

25. Apparatus according to any one of claims 21 to 24, wherein said means for compiling comprises: global vector supplementing means, operative if said secondary local vector supplementing means has not been able to associate sufficient supplementary motion vectors with said current target block, for associating with said current target block one or more global vectors derived from motion of said current image as a whole.

26. Apparatus according to any one of claims 21 to 25, wherein said blocks are rectangular arrays of image data elements from within said current image.

27. Apparatus according to claim 26, wherein said current target block abuts eight touching blocks with said primary set of blocks comprising every alternate block from said eight touching blocks.

28. Apparatus according to claim 27, wherein said secondary set of blocks comprise four blocks spaced from said current target block by those blocks of said eight touching blocks not included in said primary set of blocks.

29. Apparatus according to any one of claims 21 to 28, wherein said predetermined test criterion is an amplitude threshold criterion.

30. Apparatus for motion compensated image standards conversion including apparatus for motion analysis of moving images according to any one of claims 21 to 29.

31. A method of motion analysis of moving images, said method comprising thie steps of: calculating an array of motion vectors for a corresponding array of blocks within a current image, said motion vectors representing motion of that part of said current image within each of said blocks between a temporally adjacent image and said current image, compiling a plurality of motion vectors to be associated with each block, as a current target block, for use in further analysis, said step of compiling comprising the steps of:: detecting if said motion vectors within said array of motion vectors are valid motion vectors that meet a predetermined validity test, associating with said current target block, if valid, that motion vector calculated for said target block, and associating with said current target block, if valid, differing from any previously associated motion vectors and satisfying a predetermined selection test, one or more supplementary motion vectors taken from one or more sets of blocks that surround said current target block.

32. Apparatus for motion analysis of moving images substantially as hereinbefore described with reference to Figures 2 to 10 and 12 to 14B of the accompanying drawings.

33. Apparatus for motion compensated image standards conversion substantially as hereinbefore described with reference to Figures 2 to 10 and 12 to 14B of the accompanying drawings.

34. A method for motion analysis of moving images substantially as hereinbefore described with reference to Figures 2 to 10 and 12 to 14B of the accompanying drawings.