GB2293713A

GB2293713A - Digital image processing apparatus

Info

Publication number: GB2293713A
Application number: GB9419909A
Authority: GB
Inventors: Barry Isaacs; Mike Pooley
Original assignee: British Aerospace PLC
Current assignee: BAE Systems PLC
Priority date: 1994-09-30
Filing date: 1994-09-30
Publication date: 1996-04-03
Anticipated expiration: 2014-09-30
Also published as: GB9419909D0; DE69517487T2; NO953830D0; EP0704711B1; NO307317B1; NO953830L; EP0704711A2; DK0704711T3; JPH08126000A; GB2293713B; CA2159285A1; EP0704711A3; DE69517487D1

Description

1 DiQital Imacle Processing Apparatus 2293713 This invention relates to

digital image processing apparatus for correlating a two-dimensional reference image over a series of positions in an input image.

Such apparatus is typically used in a correlation tracker which functions by comparing each frame of video data with a small rectangular "reference image" of the object being tracked. The correlator finds the position in the current image where the reference image best matches the current video image. This position data is used to cause the tracker to "follow" the object and may also be used to provide navigational data to allow the object to be intercepted. Correlation trackers such as this are known but there is a substantial delay between scanning or read out of the digital image data for a TV field and completion of the correlation operation. This can slow operation of the tracker and makes it difficult to adapt the various parameters of the correlation in accordance with the results of the correlation. Thus a need exists for a correlation tracker which is capable of providing correlation data for a TV field in near real time, so that the correlation operation can be carried out between TV fields. A need also exists for a correlation tracker in which the reference image and various parameters associated therewith, such as the weighting of the pixels making up the reference image, the size and shape of the reference image and the degree of 2 confidence or the speed of adaptation to new reference values, may be varied in accordance with correlation data and associated parameters determined by the tracker.

We have designed a correlator which is capable of correlating a twodimensional reference image against every possible position in a raster scanned image, producing a correlation surface for the entire image field, with virtually zero delay from receipt of the last pixel of the image field. The reference image can be completely changed during the flyback interval, and each pixel of the reference image can be weighted, to lower its contribution to the overall correlation sum, or turned off completely. This allows the reference image to assume shapes other than rectangular, e.g. a doughnut-shaped ring reference image can be selected.

The correlator architecture, by virtue of its very high speed of operation, and by incorporation of the reference image pixel weighting, allows an ASIC implementation, in association with a suitable high performance digital signal processor, to be used for full field correlation (with complete reference image update during f lyback), and for limited field advanced correlation techniques.

Accordingly, this invention provides digital image processing apparatus for correlating a reference image with an input image at a plurality of correlation positions and for determining a position of best registration thereof, said apparatus comprising:- image data input means for receiving digital data o 1 1 1 3 representing the pixel values of said input image; a series of processing cells connected one to another and to said image input means for receiving data therefrom substantially in parallel, each processing cell including:

- means for storing a respective reference value representing the value of an associated pixel in the reference image; means for performing a correlation operation on the reference value stored in the processing cell and the input image data supplied to the processing cell to determine partial correlation data; partial correlation data output means for outputting partial correlation data, wherein each processing cell downstream of the first cell in the series includes partial correlation data input means for receiving the partial correlation data from the previous cell, and the last processing cell in the series has correlation data output means for outputting a stream of digital correlation data indicating the degree of correlation at each of the correla- tion positions.

Preferably the apparatus includes means responsive to said stream of digital correlation data for determining the position of best registration of the reference image with the input image.

The correlation operation in each processing cell preferably includes determining the absolute difference between the image data value input to that cell and the reference value for that cell. The correlation operation 1 4 preferably further includes combining or summing said absolute difference with the partial correlation data received from the upstream processing cell, to form partial correlation data for output to the downstream processing cell-. Each processing cell preferably includes means for storing a respective correlation weighting value associated with said cell, and means for weighting or multiplying said absolute difference value with said correlation weighting value. The respective correlation weighting values of the processing cells are preferably modified in accordance with at least one of:- (i) the current values of the pixels in the best registration array defined by that part of the image corresponding to the reference image at the position of best registration; (ii) the pixels in the reference image, and (iii) the current pixels in arrays of similar size and adjacent to the best registration array.

Each processing cell preferably includes latch means or other delay means for applying a predetermined delay to the partial correlation data supplied to the downstream processing cell.

Preferably the values of the input image pixels in the best registration array are used to modify the values of the corresponding pixels of the reference image.

Said modification preferably includes modifying the values of the pixels of the reference image in accordance with the values of the corresponding pixels in the best registration array, and in accordance with a respective reference weighting value for each pixel. Each reference weighting value may be determined using the absolute difference between the pixel values of the reference image 5 and-the pixel values of the best registration array.

In another aspect of this invention, there is provided digital image processing apparatus for successively correlating a digitised reference image with a digitised input image at a plurality of search positions to determine a position of best registration of said reference image with a correspondingly sized portion of the input image, including store means for storing a plurality of correlation weight values representing an array corresponding to the reference image, said correlation weight values being used during said correlation to adjust the weighting of the correlation term corresponding to the associated value of the reference image, and means for adjusting the stored correlation values in accordance with the previous correlation.

In a further aspect, this invention provides digital image processing apparatus for successively correlating a digitised reference image with a digitised input image at a plurality of search positions to determine a position of best registration of said reference image with a corresponding sized portion of the input image, including reference store means for storing data representing said digitised reference image, means for modifying the stored data representing said digitised reference image in accordance 6 with the previous correlation, wherein said means for modifying includes means for storing filter values associated one with each location of said reference store means respectively, and said means f or modifying adjusts the value of each of the stored reference data in accordance with the corresponding values of the best registration area on the input image, to an extent dependent on the current f ilter value associated therewith.

In another aspect this invention provides a method of correlating data substantially as implemented in the above apparatus.

Whilst the invention has been described above, it extends to any inventive combination of the features set out above or in the following description.

The invention may be performed in various ways and, by way of example only, various embodiments and modifications thereof will now be described in detail, reference being made to the accompanying drawings, in which:Figure 1 is a schematic diagram representing a 3 x 3 convolver architecture; Figure 2 is a schematic diagram representing an inverted 3 x 3 correlator architecture in accordance with this invention; Figure 3 is a schematic diagram of a correlator in accordance with the invention made up of m rows of n cells; Figure 4 is a block diagram illustrating a processing cell for the correlator of Figure 3; Figure 5 is a block diagram of an application specific 7 integrated circuit (ASIC) providing 2 lines of 32 processing cells.

Figure 6 shows the general area used to calculate the values of the ACwt array; - Figure 7 shows the two areas used in the correlation calculation and the region of overlap between the two; Figures 8(a) and (b) show two examples where the correlation minimum rectangle is near two boundaries of the full field of view; and

Figure 9 is a schematic diagram of an embodiment of correlator tracker incorporating an eight ASIC correlator.

A digitised raster scanned monochrome image (video) consists of a stream of data words (corresponding to the pixel intensities of the source image or scene), a pixel clock, and line and field synchronisation pulses. The image or scene is scanned line by line, left to right and top to bottom. A correlator operates on this data stream in a similar manner to a two-dimensional convolver, using a variation on already well established techniques. The correlator differs from these by virtue of its size, which is very much larger than previously available convolvers (32 16 array size, compared with 9 9 typical maximum array size), and its basic cell structure, which is designed to implement a weighted difference correlation algorithm to be described in further detail below.

The description of the mode of operation of the correlator assumes that the image data consists of 512 pixels per line. The pixel clock rate, and the number of

8 lines in the image are not important for this description. The pixel data is eight bits, i.e. 256 grey levels. The correlator reference image is assumed to be a maximum of 32 16 pixels (512 in total), but may be larger due to the expandable architecture. A conventional 3 3 convolver architecture 10 is shown in Figure 1. Image data enters the convolver at the top left, 12, and is separated into three line channels, 14, 16, 18, by the 512 stage delays. Each channel consists of three 10 delay stages, 20, where each delay is equal to one cycle of the pixel clock. The image data has thus been manipulated so that all 9 pixels which are required for the convolution operation are available simultaneously at the outputs of the nine delay states. Nine multipliers, 22, and a single 15 adder, 24, complete the convolver. Note that the adder, 24, is required to add nine numbers, each greater than eight bits (when multiplied). The 32 16 correlation function could be implemented in a similar manner, replacing the 3 channels by 16 channels 20 each comprising 32 delay states. 512 differencers would replace the 9 multipliers, and the adder would be required to add 512 eight bit numbers. This latter requirement would result in a lengthy delay through the system, and extensive hardware for generating the sum. Referring to Figure 2, we have found that the basic convolver architecture can be reversed, so that the image data is fed simultaneously to all of the differencers, 30, and the 9 delays, 32, similarly organised into 3 channels,

9 34, 36, 38, each 2 delays long, with line delays 39, between channels, are used to shift partial sums of the final addition between differencers. It will readily be seen that this architecture may be extended to provide a 32 16 corr,--lation operation with 512 absolute differences, and with 16 channels each with 32 delay stages with appropriate line delays between each channel.

The principal advantage obtained from this inverted architecture is that the addition process becomes distributed and need not contribute to the overall delay through the system. Advantage is taken of this in the correlator design to minimise the delay from arrival of the last pixel of the image to outputting of the last value of correlation to 2 pixel clock cycles.

Referring to Figure 3 this shows a correlator in accordance with the invention for correlating an m x n reference image with an input image, illustrating the interconnection between the processing cells 40. In brief, the processing cells are connected up in m lines of n cells, the nth cell in each line being connected via a line store 41 to the f irst cell in the next line. The pixel data is fed to each cell 40 in parallel and there are lines for clocking in the respective reference data values and correlation weight values into each cell. The output of the last processing cell 40 provides the aggregate value of all the partial correlation sums calculated by the processing cells and provides a first correlation value for being stored in the correlation map, the correlation values collectively representing a correlation surface. The correlation values provide a measure of the correlation between the reference image and the input image for respective overlaps of the reference and input image, with a low value indicating good correlation. A processing cell is shown in Figure 4. This comprises an 8 bit absolute differencer, 42, a multiplier, 44 and a partial sum adder, 46. The partial sum output to the next cell is delayed by a register 48 for one pixel clock cycle. 10 Also within the cell are registers 50, 52 for holding the reference image pixel data and a multiplication factor (AC weight). In the present implementation, for reasons of minimising complexity, the multiplication factor values are limited to the values 1, 112, 114, 118, 1116, 1132, as well 15 as 0, corresponding to the cell being switched off (i.e. not contributing to the overall correlation sum). The reference data register, 50 and weight factor register, 52 have their inputs connected to the outputs of the equivalent registers in the preceding cell. This is done so that the entire 512 20 bytes of the reference and 512 weight factor values can be loaded into the cells in series through a single parallel port. Similarly the reference data and weight data can be read out of the system through a similar port, and thus the registers can act as read/write stores for an external process requiring access to the data. The clock used for loading the registers is completely separate from the video data clock so that if required the reference and weight data can be changed at any time without stopping the video data 11 f low.

The main requirement for the differencing correlator is to search for the best match between the reference and the image data. This corresponds to the lowest correlation sum.

Consequently a minimum detector at the output of the correlator will indicate when the last best match was located, and if the line number and pixel clock cycles into that line of the image are recorded when the minimum occurs, the best fit part of the image can be subsequently examined.

Figure 5 illustrates an ASIC providing 64 differencing cells, in 32 2 array, with two line delays on a single ASIC, but the full 32 16 array can be achieved simply by connecting ASICs in series, in exactly the same manner as the cells are internally connected within each ASIC. This ability to extend the array size is not limited to 32 16.

The array of processing cells described above provides correlation data in near real time, so that the correlation values making up the correlation surface over the search area are available within a few pixel clock cycles of the last relevant pixel from that area of the image currently being correlated with the reference image. This means that the correlation minimum, i.e. the precise location of the best registration between the reference image and the image input can be determined in the flyback time before the data from the next TV field arrives. Within this time the reference image and the correlation weights may be updated in accordance with the data from the immediately previous TV field, as to be described.

12 The correlation tracker will be a sub-system of a larger tracking system. It will search for matches between a Reference image and an area of the incoming real time video data, the Search area, and provide real time tracking information, ie target co-ordinates and tracking quality values. It will be possible to load the Reference array from an area of the incoming video data - target designation operation, or to load it from pre-recorded images - target acquisition.

The correlation function is a pixel by pixel comparison - differencing of the Reference image and a same size area of the Search area. The correlation calculation involves additional information to the Reference and Search arrays. During the correlation calculation each pixel comparison is factored by a 'weight' obtained from a data area called the ACwt array. This array is the same size as the Reference array and has a direct Pixel to pixel correspondence with it. The contents of the ACwt array are preset at the start of the track such that each value is a maximum. The individ- ual values within the ACwt array vary field by field as the track continues. The variation of each individual entry depends on the image surrounding the located match for the Reference array, field by field. An example of a technique for doing this is given below.

Each correlation calculation is defined as:- L A C W, 'a j] A&kd " L _,4 c ret4 3-Z)jz 1.

where @ j 9 ACwt' 1 pres 1 Iref 1 13 is the pixel counter to process the array. is the line counter to process the array. is the scaling value for the pixel [i,j). is the present array, grey scale values from the Search Array corresponding to the (i,j] position of the overlapping Reference Array. is the grey scale value from the Ref erence Array at position (i,j).

The calculation is performed for each possible overlap of the Reference Array and Search Array producing a Correlation Sum Array or correlation surface. The best correlation match corresponds to the location of the minimum value.

On completion of a correlation run the results are available to the controlling processor and the Reference Array is updated to take account of the variation between the original Reference and the located target as the image changes. The Reference Array update is performed by combin ing a fraction of the original array with the reciprocal fraction of the located target image. The fraction, weight', values are obtained from the ARwt array. This array is the same size as the Reference Array and has a direct correspondence with it. The contents of this array are preset at the start of the track such that each value is a minimum. The values within the ARwt array vary field by

14 field as the track continues. The variation of each individual entry depends on the dif f erence between the values in the Reference Array and the corresponding values in the Reference Array sized area defined by the detected correla- tion minimum (see Figure 6). An example of a technique for calculating the ARwt values is given below.

A typical sequence of correlation tracking is as follows:- STEP/ ACTION 0/ The correlator is in idle waiting f or start co-ordinates from the main system. Each of the values in the ACwt array is set to maximum and each of the values in the ARwt array is set to minimum.

11 The main system defines the co-ordinates of the target area, which also defines the first search area.

2/ The correlator constantly stores the incoming video data in a field store. After receiving target coordinates the correlator loads the Reference array from the designated area of the field store.

3/ On the next field the correlator performs the correlation calculations over the Search area producing a Correlation Sum Array.

4/ The following data is used by the correlator tracker system.

a) Location and value of the correlation minimum, to define the coordinates of the Search area in the next field and to define the field area to be used to modify the Reference array.

The correlation values of the four locations, North, East, South and West of the detected minimum.

The centroid of the Reference Array.

The centroid of the Reference Array sized area within the current field defined by the correlation minimum position.

The average value of the Reference Array grey value.

The average value of the Reference Array sized area within the current field defined by the correlation minimum position.

b) c) d) e) f) The data in b) to f) inclusive are used to produce track quality values, ie how well the located object matches the required object. If the quality value is low then the tracker may be allowed to coast, and the ACwt, ARwt, and Reference arrays are not updated - go to step 6.

g) Location and value of the second best correlation minimum, to be used for possible decoy evasion.

16 5/ If the track quality is high the location of the detected correlation minimum is used to update the ACwt and ARwt arrays.

6/. Using the new ARwt array and the newly located correla tion minimum update the Reference Array.

7/ The controlling processor defines the next field position of the Search area.

8/ Go to step 3.

Figure 7 shows an example of the two areas used in the correlation calculation and the region of overlap between the two. The Search Area defines the sub region of the FFOV over which the Reference Area is passed. The Search Area may extend beyond the boundaries of the FFOV but the Reference Area may not. Image data def ined by the Search Area but overlapping the FFOV shall be not be processed. The result is a Correlation Area occupying the same size and image position as the Search Area. 512 valid correlations are performed, 32 pixels 16 lines, in this example.

The Search Area in this example is 16 lines of 32 pixels. During each field correlation this area is nominally centred around the expected position of the target, there is no true centre to a 32x16 area. The position of this search area is defined by the controlling processor at the start of

17 each field's processing.

The Reference Area is 16 lines of 32 pixels. It contains a record of the image to be searched f or within the Search Area. The contents of the Reference Area may be modified field by field as the target image changes, see below. At the start of the correlation sequence, designation or acquisition, the Reference Array is loaded by the correlator from the incoming image data at a location defined by the controlling processor or from a pre-recorded image list.

Referring again to Figure 6, this shows the general area used to calculate the values of the ACwt array. The Correlation Minimum (CM) location is an area the size of the Reference Array located at the detected correlation minimum for the current field, ie after the correlation process is complete. The image area used to calculate the change in ACwt value is equivalent to 8 similar areas surroundng the Correlation Minimum location.

In the general case the position of the detected correlation minimum will not be near the FFOV boundary and the indicated data area is used. If the correlation minimum should approach the FFOV boundary such that some of the surrounding areas would extend beyond the boundary then the image area used is modified. The area is always equivalent to 8 additional array areas but they are located dif f erently with respect to the correlation minimum.

Figure 8 (a) and (b) shows two examples where the 18 located Correlation Minimum approaches the FFOV boundaries causing the required surrounding areas to overlap the boundaries, left and top in a) and right and bottom in b). It is also possible that only a single boundary is crossed in which case CM occupies one of the central locations of the group of nine array areas.

As mentioned above, the image data area required to update the ACwt and ARwt arrays is the image data in the position of the newly found correlation minimum and the 8 similar areas located around it as allowed by the position of the correlation minimum within the FFOV, see Figures 7 and 8.

An example of an update process is as follows.

1/ A temporary array, 16 lines of 32 pixels, is produced. Each entry is the average of the sum of the absolute differences between the reference array and the 8 areas surrounding the correlation minimum minus the absolute difference between the reference array and the correlation minimum array. The temp entries are signed and may contain binary fractions.

temp[i,j] {Sum(k=1:8) abs(Ak[i,j]-ref[i,j])}18 abs(cm[i,j]-ref[i,j]) where i = temp array pixel value.

19 J = temp array line value. A = one of the data areas surrounding the correlation minimum. cm the correlation minimum array. ref the reference array.

21 3/ The temp array maximum, average, and minimum values are noted.

The temp array is further modified as follows. For each entry in the array If temp[i,j] < average value then temp[i,j] = (temp[i,j]-average) / (average-minimum) else If temp[i,j] ≥ average value then temp[i,j] = (temp[i,j]-average) 1 (maximum-average) 4/ The ACwt array is then updated as follows, the ACwt array truncates the effects of the fractions in the temp array.

ACwt[i,j] = temp[i,j] + ACwt[i,j] The individual entries of the ACwt array allow long term object features to accentuated and short term features, ie crossing or short duration decoys, to be ignored. Pixel locations not associated with long duration features tend to have low values and contribute small amounts to the correlation sum.

- The way in which the ARwt values are calculated will now be described.

The ARwt array is updated using data from the reference array and the scene data at the position of the detected correlation minimum. The update process is as follows.

1/ A temporary array, 16 lines of 32 pixels, is produced.

Each entry is the absolute difference between the reference array and the correlation minimum array.

temp[i,j] = abs(cm[i,j]-ref[i,j]) where is 2/ cm is the correlation minimum array.

The temp array maximum, average, and minimum values are noted. The average can contain a binary fraction.

The temp array is further modified as follows. For each entry in the array If temp[i,j] < average value then temp[i,j] = (average-temp[i,j]) / [2(average-minimum)] 21 else If temp[i,j] ≥ average value then temp[i,j] = (average-tempi,j]) / (maximum-average) 4/ -The ARwt array is then updated as follows.

ARwt[i,j] = temp[i,j] + ARwt[i,j] The individual entries of the ARwt array allow long term object features to persist in the reference record. Short term features, ie crossing or short duration decoys, are ignored.

On completion of the ARwt update the Reference Array can be updated as follows:- ref [i,jl = (ARwt[i,jlcm[i,j]) + (1-ARwt[i,j]ref [i,j]) The hardware for the correlator may be implemented in various ways, but we have designed a correlator made up of 8 similar ASICS, each ASIC comprising 2 lines of 32 process- ing cells, the ASICS being connected in series to Provide acorrelator capable of storing a reference array of 16 lines of 32 pixels each. The input of pixel data, the transfer of partial correlation data between cells and lines, and the clocking of Reference Data and correlation weighting data into the respective processing cells, is carried out in a similar manner to that indicated in relation to Figure 5.

An example of a correlator tracker 56 incorporating a 22 multiple (8) ASIC correlator 58 is shown in Figure 9.

The various processing steps for updating the reference data and correlation weighting data and for detailing the co-ordinates of the tracked object within the input image are implemented by a digital signal processor 60. The correlator also includes a field data store 62 whichstores each video field frame for use in the various updating steps described above, a correlation map store 64 for storing the correlation values making up the correlation map, together 10 with a field control 66 and buffers 68.

23

Claims

1. Digital image Processing apparatus for correlating a reference image with an input image at a plurality of correlation positions and for determining a position of best registration thereof, said apparatus comprising:- image data input means for receiving digital data representing the pixel values of said input image; a series of processing cells connected one to another and to said image input means for receiving data there substantially in parallel, each processing cell including:means for storing a respective reference value representing the value of an associated pixel in the reference image; means for performing a correlation operation on the reference value stored in the processing cell and the input image data supplied to the processing cell to determine partial correlation data; partial correlation data output means for outputting partial correlation data, wherein each processing cell downstream of the first cell in the series includes partial correlation data input means for receiving the partial correlation data from the previous cell, and the last processing cell in the series has correlation data output means for outputting a stream of digital correlation data indicating the degree of correlation at each of the correlation positions.

2. Digital image processing apparatus according to 24 Claim 1, including means responsive to said stream of digital correlation data f or determining the position of best registration of the reference image with the input image.

-

3. Digital image processing apparatus according to Claim 1 or 2, wherein the correlation operation in each processing cell includes determining the absolute difference between the image data value input to that cell and the reference value for that cell.

4. Digital image processing apparatus according to Claim 3 wherein said correlation operation further includes combining or summing said absolute difference with the partial correlation data received from the upstream processing cell, to form partial correlation data for output to the downstream processing cell.

5. Digital image processing apparatus according to Claim 3 or Claim 4, wherein each processing cell includes means for storing a respective correlation weighting value associated with said cell, and means for weighting or multi- plying said absolute value with said correlation weighting value.

6. Digital image processing apparatus according to Claim 5, including means for modifying each of the respective correlation weighting values of the cells in accordance with at least one of:- (i) the current values of the pixels in the best registration array defined by that part of the image corresponding to the reference image at the position of best registration, (ii) the pixels in the reference image, and (iii) the current pixels in arrays of similar size and adjacent to the best registration array.

-

7. Digital image processing apparatus according to any preceding Claim, where in each processing cell includes latch means or other delay means for applying a prede termined delay to the partial correlation data supplied to the downstream processing cell.

8. Digital image processing apparatus according to any preceding Claim, wherein the values of the input image pixels in the best registration array are used to modify the values of the corresponding pixels of the reference image.

9. Digital image processing apparatus according to Claim 8, wherein said modification includes modifying the values of the pixels of the reference image in accordance with the values of the corresponding pixels in the best registration array, and in accordance with a respective reference weighting value for each pixel.

10. Digital image processing apparatus according to Claim 8, wherein each reference weighting value is determined using the absolute difference between the pixel values of the reference image and the pixel values of the best registration array.

11. Digital image processing apparatus for successively correlating a digitised reference image with a digitised input image at a plurality of search positions to determine a position of best registration of said reference image with 26 a correspondingly sized portion of the input image, including store means for storing a plurality of correlation weight values representing an array corresponding to the reference image, said correlation weight values being used during said correlation to adjust the weighting of the correlation term corresponding to the associated value of the reference image, and means for adjusting the stored correlation values in accordance with the previous correlation.

12. Digital image processing apparatus for successive- ly correlating a digitised reference image with a digitised input image at a plurality of search positions to determine position of best registration of said reference image with correspondingly sized portion of the input image, includ- ing reference store means for storing data representing said digitised reference image, means for modifying the stored data representing said digitised reference image in accordance with the previous correlation, wherein said means for modifying includes means for storing filter values associ- ated one with each location of said reference store means respectively, and said means for modifying adjusts the value of each of the stored reference data in accordance with the corresponding values of the best registration area on the input image, to an extent dependent on the current filter value associated therewith.

13. Digital image processing apparatus substantially as hereinbefore described with reference to and as illustrated in any of Figures 2 to 9 of the accompanying drawings.