DE69126424T2 - Compare patterns with confirmed limits - Google Patents

Description

The invention relates to a method for matching a first image and a second image of the same scene, both the first and second images having an array of pixels, comprising the steps of: forming a reference map of a portion of the first image, the reference map having an array of pixels each corresponding to a corresponding pixel of the portion of the first image, and the values of the pixels of the reference map indicating the position of one or more edges shown in this portion of the first image, the reference map being compared sequentially with the portions of the second image and a mark being calculated for each portion of the second image, the mark indicating the probability that the pixels in the portion of the second image corresponding to the pixels in the reference map have values indicating one or more edges in the second image; and selecting the portion of the second image producing the higher mark to correspond to the portion of the first image.

In certain military operations, it is important to fire a guided missile from an aircraft towards a specific target on the ground. A significant military advantage is gained if the guided missile guides itself to the target. This allows the aircraft and its human pilot to stay outside the danger zone surrounding the target. To make this possible, the guided missile must have access to the target data in order to be able to identify the specific target on approach.

US-A-3 737 120 describes a radar map comparison tracking system for an aircraft-launched guided missile, the aircraft being equipped with a variable resolution synthetic aperture mapping radar capable of obtaining high resolution maps of the features of a target and its immediate surrounding area from a distance of 50 nautical miles, and the missile being equipped with a conventional non-processing X-band (X-band) radar which, with a 2-foot X-band antenna, is only 300 feet from the target when it generates the resolution obtained from the aircraft's synthetic aperture radar at 50 nautical miles. The aircraft's radar

produces a series of maps of the target area at different resolutions, each resolution corresponding to the resolution of the X-band missile radar at a given distance from the target. These maps are obtained at a distance of 50 nautical miles from the target and are stored in separate memory-type CRTs in the missile. When the missile is fired, it is guided to the target by an inertial tracking system. After launch, under the control of a timer in the missile, instantaneous radar maps from its X-band radar are displayed on the corresponding CRTs at predetermined times corresponding to predetermined distances. Each of these CRTs produces an input image to a corresponding image comparator which, at an appropriate time, receives the image of the stored map from the corresponding memory CRT. Thus, as the missile approaches the target, the resolution of the maps to be compared increases, with the inertial tracking system being corrected by the error signals emitted by the image comparators. However, the image comparators are not described in detail.

In general, synthetic aperture radars (SARs) are known. Suffice it to say that a missile-mounted SAR produces an image of objects lying within a field of view on the ground below. The image can be displayed in a variety of ways. For computer-processed images, the image is most often an array of digital words, with each word describing the intensity or brightness of radar reflections from a small area within the field of view. Alternatively, an image of the ground area can be captured with a photographic camera if it is to be displayed to a human.

In the weapon system under consideration, both the aircraft and the guided missile are equipped with synthetic aperture radar (SAR).

In the weapon system under consideration, it is desirable for a person in the aircraft to be able to identify a specific target in the image. The image section adjacent to the target becomes the reference image. An image processing device in the aircraft then transmits the reference image to the guided missile. After firing, the guided missile flies towards the target, generating its own SAR images. An image processing system in the guided missile compares the reference image with the image of the guided missile. The guided missile tracking system can then use the results of the image comparison to guide the guided missile towards the specific target.

From US-A-4 581 762 a method of the type mentioned above is known, in which the first image is an image of a known object while the second image is an image of an unknown but related object. The method described is used in particular for the visual inspection of manufactured objects such as speed measuring devices on motor vehicles. A camera of the visual inspection system records the brightness values around a point that has been selected by a user as being in an area where the edges of a known, mass-produced object are present. An area of 9x9 pixels around this point is stored. Using a technique known as the Sobel method (page 337, Digital Image Processing by Gonzales and Wintz, Addison Wesley Company, 1977), the gradient of a center point of each 3x3 matrix of pixel values within the 9x9 pixel area is processed to generate a 7x7 gradient field, which is subsequently used as a model for detecting the same edge configuration in a 40x32 pixel search area of the unknown object. During the test phase, the 40x32 pixel search area is used to generate a 38x30 gradient map using the Sobel method. The mathematical correlation values are processed for each 7x7 pixel area of the 38x30 gradient map. Thus, 720 correlation values are calculated. A correlation value exceeding 0.85 is considered to indicate a probable correspondence between the area of the unknown object producing this value and the area surrounding the selected point of the known object.

However, in order for the weapon system in question to work, various problems must be overcome. One significant problem is the fact that the angle of view changes as the missile approaches the target. As a result, the SAR images produced by the missile during flight differ from the SAR image produced by the aircraft the further the missile moves away from the launch point. Therefore, the image processing must be relatively insensitive to changes in the angle of view. Another problem to be overcome is that the missile cannot detect the SAR Images must be processed quickly so that the missile tracking system can react more quickly to necessary course changes. In addition, the image processing in the missile should require relatively simple circuit technology, since the entire missile is a non-reusable device.

According to the invention, the method described at the outset is characterized in that the reference map is formed by the following steps: forming a first map of the section of the first image, the first map having an array of pixels each corresponding to a corresponding pixel of the section of the first image and having a value that indicates whether the pixel represents an edge in a local window surrounding the corresponding pixel in the first image; forming a second map of the section of the first image, the second map consisting of an array of pixels each corresponding to a corresponding pixel of the section of the first image and having a value that indicates whether the corresponding pixel in the first image has a value that exceeds a threshold value; and combining the first map with the second map, thereby forming the reference map.

Preferred embodiments of this invention produce: a signal processing method for registering two images; a signal processing method for registering two images when the viewing angles of the images differ; and a signal processing method for registering two images using relatively fast and simple processing.

Therefore, in a preferred embodiment of this invention, the first of the two images is processed to produce a reference model. The reference model is compared with each region of the second image to find the region of the second image that matches the reference model. The reference model is produced by filtering the first image to reduce noise and improve contrast between different regions in the image. Two maps are then produced from the filtered image. The first map has an indication of the edge locations of the object in the image and is formed by finding the locations in the image where the intensity or brightness suddenly changes. changes. The second map is formed by adjusting the brightness of each pixel in the filtered image to a very small number of image brightnesses and then displaying an edge at each pixel adjacent to a pixel of different brightness. The two maps are then compared to form a new map containing only the edges generated in both maps. Additional processing adds edges to the new map corresponding to edges occurring in only one of the maps, provided those edges meet additional criteria suggesting they are actual edges. The resulting map is then encoded as a model or template, which allows for quick comparison with the second image.

In particular, the first map is generated by working in sub-fields of the pixels in the first image. Each sub-field is divided into two equally sized groups of adjacent pixels. The sum of the brightnesses of the pixels in the group is calculated and the ratio between the two sums is determined. Each sub-field is decomposed in four different ways, choosing the decomposition that produces the largest ratio. If this ratio exceeds a predetermined threshold, the pixel in the first map corresponding to the center of the sub-field is set to a value that indicates an edge towards the dividing line between the two groups of pixels.

The second map is first created by calculating a threshold. This threshold is determined by calculating a ratio corresponding to each pixel in the first image that reflects the ratio of the brightness of that pixel compared to the pixel near the image with the largest brightness difference. The average brightness of the pixels in the image with the high brightnesses and the high ratios is then calculated as the average of the pixels with low brightnesses and the high ratios.

The average or mean of the two averages forms the threshold, with an edge shown in the second map where a pixel in the first image with a brightness on one side of the threshold is adjacent to a pixel with a brightness on the other side of the threshold.

The maps are combined into a new map that has true edges by first including in the true edge map the pixels corresponding to edges in both the first and second maps. The pixels corresponding to edges in the second map but not corresponding to the first map are selected for further processing. The selected pixels are tested in much the same way as used to identify edges, with their angles used to create the first map, except for different relaxed parameters to identify a pixel more likely as an edge. If the angles of the edge pixels identified with relaxed parameters are similar to the angles of the neighboring pixels already identified as edges in the new map, these pixels are added to the new map in addition.

The reference model consists of a sub-array of pixels and a correlation threshold. The sub-array has two sets of pixels describing pixels near the identified edges, the first set corresponding to pixels on the side of the edge with higher brightnesses and the second set corresponding to pixels on the side of the edge with lower brightnesses. The correlation threshold is calculated to represent the ratio of the average brightness of the pixels in the set with higher brightnesses to the average brightness of the pixels in the set with lower brightnesses.

The sub-field with the model is matched to a portion of the second image by comparing the model with the sub-field formed on the second image. The matching is first performed by two thresholds. The first threshold is determined by averaging the pixels in the sub-field of the second image, with the pixels in the model on the side of the edges with the higher brightness values weighted by the inverse correlation threshold, and the second threshold is determined by averaging the pixels in the sub-field of the second image that correspond to the pixels in the model on the side of the edges with the lower brightness values weighted by the correlation threshold. A correlation variable is then calculated that represents the number of pixels in the sub-field of the second image that correspond to the pixels in the set with the higher brightness of the model above the second threshold, and the number of pixels in the subfield of the second image corresponding to the pixels in the set with the lower brightness of the model below the first threshold. A match of the model with a subfield of the second image is obtained by selecting the subfield that has the largest value of the correlation variable.

Short description of the drawings

The invention is described in more detail below using embodiments with reference to the drawings.

Show it:

Figure 1 is a sketch of the system in which the invention is used;

Figure 2 is a block diagram of the processing that generates the target coordinates as used in tracking a guided missile;

Figure 3A is a flowchart of processing performed by the global edge detector in Figure 2;

Figure 3B is a flowchart of processing performed by the model preparation in the global edge detector of Figure 3A;

Figures 4A, 4B, 4C and 4D show masks used in the processing performed by the local edge detector of Figure 2;

Figure 5A is a first portion of a flow diagram of the processing performed by the edge extractor of Figure 2 to form an alternative ratio image;

Figure 5B is a continuation of the flow chart of Figure 5A;

Figures 5C, 5D, 5E and 5F show masks used in the processing of Figure 5A;

Figure 6A is the first portion of a flow chart of the processing performed by the edge extractor of Figure 2 to form an edge-extended boundary formation;

Figure 6B shows the continuation of the image diagram according to Figure 6A;

Figure 7A is a flowchart of the processing performed by the reference model generator of Figure 2 to form a reference model;

Figure 7B is a flow chart of the processing performed by the reference model generator of Figure 2 to form a correlation threshold;

Figure 5A is the first portion of a flow chart of the processing performed by the missile processor of Figure 2;

Figure 8B shows the continuation of the flow chart according to Figure 8A;

Figure 9A is a flow chart of processing performed according to an alternative embodiment of the reference model generator of Figure 2;

Figure 9B is a sketch of the pixels viewed during processing according to Figure 9A;

Figure 9C shows the first portion of a flow diagram of the model preparation performed as part of the processing of Figure 9A;

Figure 9D is a continuation of the flow chart of Figure 9C; and Figure 10 is a flow chart of processing performed in the missile processor of Figure 2 according to an alternative embodiment.

Description of the preferred embodiment

Figure 1 shows an aircraft 14 armed with a guided missile 16. Both the aircraft 14 and the guided missile 16 are equipped with synthetic aperture radar (SAR) 18A and 18B. A targeting device 19 receives the images received from the SAR 18A. In this embodiment, the targeting device 19 consists of a digital computer with a screen for displaying the images from the SAR 18A. The targeting device 19 also has some input devices (not shown) which enable the pilot (not shown) of the aircraft 14 to designate a target in the image (in this case a carrier T over a river). The targeting device 19 selects an image region as a reference image which is fed to the aircraft processing 20. The region in the reference image has a fixed relationship to the target. In this case, the aircraft processing 20 consists of a program described in detail below which is executed on a conventional digital computer to generate a model (not shown) of the SAR output signals. As shown, the SAR output signals consist of signals representing the river bank (not numerated) to the carrier T and other objects in the field of view, such as a building and trees (without reference numerals). The model and location of the target relative to the reference image are fed to the missile processing 40 via a digital link 15 and stored in the missile processing 40. The digital link 15 in this case consists of any known bus for transferring digital information from one computer to another.

When the missile 16 is fired at the carrier T, the digital link 15 is interrupted. As the missile 16 flies toward the carrier T, additional images of the target area are generated by the SAR 18B. The missile processing 40 compares the model formed by the aircraft processing 20, which is stored in the missile processing 40, with each of the images generated by the missile processing 40 based on the signals from the SAR 18B. The missile processing 20 matches the model with a portion of the image generated by the missile processing 40. The missile tracking system (not shown) is then actuated to control the missile 16. The missile is guided in the direction of the ground point which has the same relationship to the portion of the image matching the model as the target used to form the model had to the reference image. The results calculated by the missile processing are thus used to track the missile 16 to a specific target.

terminology

Before describing the processing performed on the SAR images, introductory concepts and terminology are explained. The SAR image is divided into a two-dimensional array of pixels, each represented by a digital word. In this case, the array is assumed to be 36x36, but it will be clear to those skilled in the art that the methods described below are also applicable to other sizes and shapes of arrays. The value of each digital word corresponds to the intensity or brightness of the image at that pixel. The methods for displaying the images displayed in this way, as well as the methods for transferring the images from one processor to another, are known.

Hereinafter, the images formed by the SAR, or any processed version of that image, are referred to simply as an image. However, when processing the image, individual pixels are sometimes considered. In these cases, each image should be thought of as a two-dimensional array of pixels. Each pixel has an x and y position in the array. An array name followed by two numbers in parentheses (e.g. SRBE (i, j)) means that the pixel in the designated array has the xy position in the array.

In some cases, a value or group of values is assigned to a pixel in the image. The array of these values is also called a "map".

In some examples, an image is processed in several different ways and comparisons are made between the resulting images or maps. In some cases, the processed images may have a different number of pixels from the unprocessed image or from each other. When comparing such images, it is important that the correct pixels in each image are related to each other. The center pixels of the images correspond to each other and have the same xy field coordinates. The pixels near the center pixels also correspond. Corresponding pixels may be identified from the center of each image outward until corresponding pixels have been identified for all pixels in the smaller image. In the larger image, there are no corresponding pixels for the remaining pixels. The comparison results for these pixels are undefined and thus are not used in further processing. In many cases, groups of pixels in an image are selected for simultaneous viewing. Such a selection process is called a "local window." For example, if a 3x3 sub-array of pixels is to be viewed, that group is placed in a 3x3 local window. The local window is centered over the pixel in the center of the window. Often it is necessary to process every such group of pixels that can be generated by an image. In these cases, the local window "slides" over the image, as the local window is centered over one pixel, then moves, is centered over another pixel, moves again, and so on. It is It should be noted that if the local window is centered over a pixel along the side of the array, a subarray of pixels cannot be formed around the pixel because no pixels exist on one side of that pixel. When a local window slides over an image, it will generally not be centered over the pixels along the sides of the array because certain pixels below the window would have undefined values. However, some algorithms have been created to handle these possible cases. In these algorithms, the local window can be centered over the pixels along the sides of the array. In some cases, a large number of numbers representing ratios are compared. Comparing these numbers "in terms of a ratio" involves more than simply selecting the largest number. The numbers are compared as if they were all positive. In addition, the multiplicative inverses of numbers less than one are taken into account (i.e., all ratios greater than one are converted to positive numbers for comparison).

Pre-processing filtering

Referring to Figure 2, additional details of the aircraft processing 20 and the missile processing 40 can be seen. First, the operation of the aircraft processing 20 will be considered.

A non-coherent integrator 22A, a brightness level spreader 24 and a smoothing filter 26 enhance the features of the image prior to generating a model. Those skilled in the art will recognize a variety of other techniques that can be used for feature enhancement. The non-coherent integrator 22A generates an average of a number, e.g. four, of images taken by the SAR using different radar frequencies. The average is processed on a pixel-by-pixel basis (i.e., the corresponding pixels in the different images are averaged, producing the corresponding pixel in the output image).

The brightness level spreader 24 changes the brightness of each pixel in the image, thereby concentrating the distribution of brightness levels in the image around two brightness levels. In this case, each pixel is represented by a digital word consisting of a discrete number, e.g. 256, of values representing a brightness level. The input image has some arbitrary brightness distributions which are modified by the brightness level spreader 24. First, the global median value of the brightness of all pixels in the 36x36 image is determined. Next, the output image in the brightness level spreader 24 is generated by operating on each possible group of new pixels selected by sliding a 3x3 local window over the input image. The value of the pixel in the output image is calculated from the new pixels below the local window corresponding to the pixel in the center of the 3x3 local window. For each position of the local window, that is, each group of new pixels, the local median value of the pixels in the window is determined. If the local median value is less than the global median value, the output pixel is set to the local minimum, that is, it is assigned a value equal to the lowest brightness of all pixels in the window. On the other hand, if the local median is not less than the global median, the output pixel is set to a value equal to the local maximum. Since the local window cannot be centered over every pixel on the outer edge of the 36x36 patch, the image resulting from sliding the local window over the entire input image produces a 34x34 output patch.

The smoothing filter 26 uses the 34x34 output field of the brightness level spreader 24 as its input field. First, the global median of the input field is calculated by finding the median of all pixel values in the input field. A 3x3 window slides over the input image, with the pixel in the output image being calculated to correspond to the pixel in the center of the window at each window position. If the center cell of the local window is greater than or equal to the global median, the pixel in the output image is set equal to the median of all pixels in the local window that are greater than or equal to the global median. On the other hand, if the center cell in the local window is less than the global median, the pixel in the output image is set equal to the median of all pixels in the local window that are less than or equal to the global median. The resulting filtered image is described by a 32x32 array of pixels and is subsequently referred to as an EPS image.

In alternative embodiments of the brightness level spreader 24 and the smoothing filter 26, the filtered image may have the same number of pixels as the reference image. For example, the pixels located near the edge of the reference image, which are not processed because no local window can be centered on them, may simply be copied unchanged into the filtered image.

The filtered image is then processed by a local edge detector 28 and a global edge detector 30. Each produces an edge map. The edge maps, like the filtered images, are represented as arrays of digital words. The value of each word is used by the local edge detector 28 or the global edge detector 30 to indicate whether or not the corresponding pixel in the filtered image represents an edge between two objects in the image. The local edge detector 28 and the global edge detector 30 use different methods to identify edges in the input image. Therefore, the resulting edge maps may differ from each other. The edge extractor 32 combines both edge maps into a confirmed edge map using the information from the filtered image. The confirmed edge map includes edges that are likely to represent actual edges. A reference model generator 34 uses the confirmed edge map to generate the model and transmit it to missile processing where it is compared with the images formed by the SAR 18B using the edge model correlator 42 to locate the target. The operation of each of these elements is described in detail below.

Global edge detector

Figure 3A shows a flow chart of the processing performed in the global edge detector 30 to generate a global edge map, hereinafter referred to as an SRBE map. The rectangular elements (typified by element 50) are referred to as "process blocks" and represent instructions or a group of instructions of a computer program. The diamond-shaped elements (typified by element 54) are referred to as "decision blocks" and represent instructions or groups of instructions of a computer program that cause execution of computer program instructions represented in the process blocks. The Flowchart does not denote a syntax of a particular computer programming language. Rather, the flowchart represents the functional information needed by a person skilled in the art to create a computer program that performs the processing required by the global edge detector 30. It should be noted that a number of routine program elements, such as initialization of loops and variables and the use of temporary variables, are not shown.

Referring to Figure 2, the global edge detector 30 processes the filtered image produced by the smoothing filter 26. When starting the processing according to Figure 3A, the image is represented by a 32x32 array of digital words, i.e., pixels. The global edge detector 30 converts the pixels in the image into one or two values (hereinafter referred to as binarization), depending on whether the pixel brightness is above or below a threshold value. An edge is indicated at any point in the image where a pixel of one value is in close proximity to a pixel of the other value. As can be seen from combining Figures 3A and 3B, an important aspect of the global edge detector 30 is the selection of an appropriate threshold value for binarizing the image.

Referring to Figure 3A, process block 50 changes any pixel having a value of zero to a value of 0.1. This avoids undefined values that could result from division by zero when the pixel values are used to calculate the ratios.

Process blocks 52 and 56 and decision block 54 implement a loop in which ratios are calculated according to the pixels in the image. A 3x3 local window slides over the input image while process block 52 calculates a ratio for the center pixels. The ratio is equal to the ratio of the brightness of the center pixel to either the third highest brightness or the third lowest brightness of the other pixels in the local window, whichever is greater in terms of ratio. Decision block 54 checks whether every possible 3x3 group of pixels has been used to calculate a ratio in the image. If every possible group of pixels has not yet been used, process block 56 selects the next group, repeating process block 52 and decision block 54. If decision block 54 determines that all groups of pixels have been used, a ratio has been assigned to each pixel except those in the edges of the image.

Process block 58 sorts the pixels in the image according to their brightness. It will be clear to those skilled in the art that any known sorting method can be used in process block 58, such as a "bubble sort" or a "shell sort" method. The output signal of process block 58 does not depend on the position of the pixels in the image, but the original image is retained for further processing. However, the ratios calculated by process block 52 are still related to the sorted pixels calculated in process block 58.

Process block 60 selects the pixels having brightness values in the thirty-fifth to ninetieth percentiles, which calculate an upper average, i.e., the average of the pixels likely to be above the optimal threshold, in the manner described below. A process block 62 re-orders the selected pixels into two groups: first, those pixels that, when used to create a ratio in process block 52, exhibited higher brightness than the pixels selected to form the ratio; and second, those pixels that, when used to form a ratio in process block 52, exhibited lower brightness than the other pixels selected to form the ratio. Within each group, the pixels are further ordered such that the pixels with larger ratios precede those with smaller ratios in terms of their ratio. A process block 63 selects the top 2% of the pixels ordered by the process block 62 for further processing.

A process block 64 deletes from the considered pixels those with ratios between 0.5 and 2.0. The decision block 66 checks whether the deletion in the process block 64 has left a sufficient number of pixels for further processing. If less than 1%, ie half of the pixels selected in the process block 63, remain, the processing by the global edge detector 30 (Figure 2) is terminated. This termination corresponds to the occurrence of an exception condition. Empirical evidence has shown that when the exception condition occurs, there is a low probability that good edges or boundaries will be found in the input image. Therefore, once the exception condition occurs, the target address 19 may select a different reference image, which image would be processed starting with the non-coherent integrator 22A (Figure 2).

Assuming that processing has not been completed after decision block 66, processing of the pixels selected by process block 63 takes place in process block 70. Figure 3B shows the details of process block 70, the processing according to Figure 3B has the advantageous effect of removing pixels corresponding to singularities rather than actual edges which are considered when setting the threshold value.

A process block 100 calculates the mean and standard deviation of the ratios of the selected pixel values using known techniques. For each pixel that is less than 1, the inverted value is used for the calculation.

A decision block 102 determines whether the ratio of the standard deviation to the mean is less than 0.1. If the ratio is less than 0.1, processing in process block 70 is completed. If not, processing continues in decision block 104 by determining whether the ratio of the standard deviation to the mean is less than or equal to a threshold T2, which in this case is 0.5. If so, a second threshold THR is calculated in process block 106 according to the following equation:

THR=M+KD(1-D/M)

where M is the mean,

D is the standard deviation, and

K is a constant, which in this case has the value 2.

Otherwise, THR is calculated in a process block 108 according to the following equation:

THR=M+KD/2

A decision block 110 then checks whether the input ratios exceed the threshold calculated in process blocks 106 or 108. If the ratios exceed the threshold, a process block 116 eliminates those ratios from further processing and inserts additional ratios in their place. Recall that process block 63 (Figure 3A) selects only the top 2% from a list of ratios sorted by their value. A process block 116 replaces the eliminated ratios with the next in that list if the ratios in the list exceed a minimum value of 2.0.

A decision block 118 decides whether to repeat the processing shown in Figure 3B. If the processing has performed less than a maximum number of passes, in this case three, and if more than half the number of ratios selected in process block 63 (Figure 3A) remain, the processing in Figure 3B is repeated in process block 100 using the prepared list of ratios. Otherwise, processing is terminated. Processing may also terminate after decision block 110 if none of the ratios exceed the threshold THR. A decision block 112 checks whether at least 1% of the pixels remain in the list, i.e., half of the pixels originally selected in process block 63 (Figure 3A). If a sufficient number of pixels remain, processing is terminated in process block 70. Otherwise, a process block 114 adds previously removed ratios back to the list in the reverse order of their elimination until the list has a number of pixels equal to half the number of pixels selected in process block 63 (Figure 3A). The need for process block 114 arises from the fact that process block 116 adds pixels to the list that does not have a ratio of at least 2.0. Thus, it may sometimes be the case that less than 1% of the pixels remain when decision block 118 is reached.

Returning again to Figure 3A, it can be seen that the rendered list of ratios is used by process block 72. The brightness levels corresponding to the pixels in the rendered list are averaged to produce an average. The process of calculating an average is repeated twice, once using the upper pixel group selected in process block 60 and once using a lower pixel group selected in process block 76. A decision block 74 determines whether processing has been repeated for the lower pixel group. If not, processing block 76 selects the pixels with brightness values in the tenth through sixty-fifth percentiles. The selection is simplified by the fact that process block 58 has already sorted the pixels by their brightness. The ratios corresponding to the selected pixels are then inverted in process block 78 and processed in the same manner as the upper pixel group. Note that process block 62 arranges the pixels for the lower pixel group into two groups as described above, except that those pixels used to form a ratio in process block 52 have a lower brightness than the other pixels selected to form the ratio that precede the pixels that have a higher value than the other pixels selected to form the ratio. However, after processing the lower pixel group, decision block 74 recognizes that the lower group has been processed and processing continues in process block 80.

In process block 80, the average of the two averages calculated in process block 72, i.e. one for the upper pixel group and one for the lower pixel group, is calculated. This "average of the averages" is used in process block 82 as the threshold for splitting the image. The pixels in the global edge map that correspond to the values in the input image above the threshold are set to "4", while those below the threshold are set to "1". Hereinafter, the global edge map at the output of the global edge detector 30 is also referred to as a split or binarized image or a particular SRBE. A particular pixel in the split image is called SRBE(i,j).

In an alternative processing scheme, the process for determining a threshold may be repeated two more times to produce three thresholds prior to splitting the image in process block 82. The second threshold is essentially determined by repeating the process of Figures 3A and 3B on the group of pixels above the "mean of mean" threshold calculated in process block 80. The third threshold may essentially be determined by repeating the process of Figures 3A and 3B on the group of pixels below the "mean of mean" threshold calculated in process block 80. Splitting the image in process block 82 then consists of assigning a number between 1 and 4 to a respective pixel in the input image. Here, a value of 1 would be assigned if the pixel brightness is below all three thresholds; a value of 2 would be assigned if it is below two of the three thresholds; a value of 3 if it is only below one of the three thresholds; and a value of 4 would be assigned if it is above all three thresholds. The edges would be identified if neighboring pixels have different values. Alternatively, edges could be identified only if the values assigned to neighboring pixels differ by two or more.

Local edge detector

Returning briefly to Figure 2, it can be seen that the local edge detector 28 operates on the same input image as the global edge detector 30. The output of the local edge detector 28 is used by the edge extractor 32 in the same way as the output of the global edge detector 30. The processing represented by the local edge detector 28 can therefore be performed at the same time as the processing of the global edge detector 30. When both processes are performed using a general purpose digital computer, the processes are more likely to be performed sequentially.

In the local edge detector 28, a 7x7 local window slides over the 32x32 pixel input array. An output edge map of 26x26 pixels is formed by assigning a value to the pixel in the edge map corresponding to the pixel in the center of the 7x7 local window. The pixel value in the edge map indicates whether the pixel of an edge corresponds to the input image and the orientation of that edge. The processing required to identify an edge and its orientation can be better understood by referring to Figures 4A, 4B, 4C and 4D. Each of these figures shows a 7x7 pixel window, with some pixels labeled "+" and others labeled "-". The windows thus determined are called "masks". As the window slides over the input field, two sums are calculated at each window layer: one with all the pixels corresponding to a "+" in the mask and one with all the pixels corresponding to a "-" in the mask. The ratio of these two sums is calculated. The masks in Figures 4A, 4B, 4C and 4D are applied to each window layer in turn, yielding four ratios. The largest ratio in terms of ratios is selected. If the selected ratio is larger in terms of its ratio than a threshold, in this case 2, the pixel in the local edge map is set to +1, indicating an edge. The angular orientation of the edge is determined from the mask corresponding to the largest ratio and whether or not an inverse ratio has been selected. The masks in Figures 4A, 4B, 4C and 4D are associated with angles 0º, -45º, -90º and -135º. If the selected ratio is greater than the threshold in terms of an absolute value, the angle of the mask corresponds to the angle corresponding to the pixel in the output edge map. If the selected ratio is less than the threshold in terms of its absolute value, the negative angle of the mask corresponds to the angle associated with the pixel in the output edge map. Finally, the local edge detector 28 eliminates from the local edge map all pixels that indicate an edge but are not adjacent to a pixel that indicates an edge with an angle within 45º of the angle of the first pixel. The output of the local edge detector 28 is then also referred to as AED, with a particular pixel in the image being referred to as AED (i, j).

Edge Extractor

Returning to Figure 2, it can be seen that the edge map generated by the local edge detector 28, the split image generated by the global edge detector 30, and the filtered image generated by the smoothing filter 26 are fed to the edge extractor 32. The edge extractor 32 combines the information from all three sources, producing a confirmed edge map with edges that are most likely to be detected in the images processed by the missile processor 40. The details of the process performed by the edge extractor 32 are shown in Figures 5A through 5F and Figure 6.

The edge extractor 32 (Figure 2) first forms an edge boundary cross-correlated map from the AED map generated by the local edge detector 28 and the SRBE image generated by the global edge detector 30. Each pixel in the edge boundary cross-correlated map, referred to as EBCC (i, j), is calculated according to the following equation: Equation (1)

IS (i, j) corresponds to a logical 1 (ie a Boolean true) if a pixel SRBE (i, j) in the output image of the global edge detector 30 differs from any neighboring pixel in that image. The expression in square brackets in equation 1 corresponds to a logical 1 as soon as one of the pixels in the AED map corresponding to SRBE (i, j) or any of its neighboring pixels is a logical 1 (ie a edge). The EBCC image thus has a 1 in each pixel where both the local edge detector 28 and the global edge detector 30 have identified an edge, and a 0 (ie a Boolean false) in all other pixels.

The edge extractor 32 next forms a difference map, denoted by "D", in which each individual pixel in the map is denoted by "D" (i, j). Each pixel in the difference map is calculated according to the following equation: Equation (2)

where IS (i, j) has the same meaning as in equation (1). Each pixel in the D-map with a value of 1 means that the global edge detector 30 detected an edge at that pixel, but the local edge detector 28 did not.

The edge extractor 32 also forms an alternative ratio map RD. The alternative ratio map indicates where the local edge detector 28 would have detected an edge if less stringent edge detection techniques had been used. Figures 5A and 5B show the process performed to generate the RD map. The process is performed once for each pixel set to 1 in the D map. Process block 130 selects a 3x3 local window from the EPS map. The local window consists of the pixels centered on the pixel corresponding to the pixel set to 1 in the D map. The local window is then processed using the mask pairs shown in Figures 5C, 5D, 5E and 5F. A process block 132 applies the first mask of each pair to the selected local window. Each number in the mask is multiplied by the value corresponding to one pixel of the EPS image in the selected local window. The sum of the products is the value corresponding to the mask. A process block 134 applies the second mask in the pair in the same way. A process block 136 then calculates the difference and the ratio of the values calculated in the process blocks 132 and 134. If each of the four pairs of the values shown in Figures 5C, 5D, 5E and 5F shown masks have been applied to the local window, process block 140 selects the next pair and calculates values for that pair.

In process block 142, the absolute values of the differences calculated in process block 136 are sorted. A decision block 144 checks whether the ratio corresponding to the largest difference found in process block 142 exceeds a threshold THR, in this case 2.0. In the event that more than one mask pair in process block 136 produces differences equal to the largest difference, the ratios corresponding to all pairs with that difference are checked by decision block 144 to determine whether any of their ratios exceed the threshold. If the threshold is not exceeded, the pixel in the RD map that corresponds to a pixel in the D map set to 1 is set to 0 in process block 160. If the threshold THR is exceeded, decision block 146 directs the process to a decision block 147 if more than one mask pair has produced differences equal to the largest difference, or to process block 152 if a single mask pair has produced the difference equal to the largest difference. A decision block 147 checks all ratios corresponding to the mask pairs that produce differences equal to the largest. If only one of the ratios corresponding to the largest difference exceeds the threshold THR, the process proceeds to process block 152. Otherwise, the process continues at decision block 148. Decision block 148 checks whether either of the two largest values was produced by an orthogonal mask pair. The mask pair 170A and 170B (Figure 5C) is orthogonal to the mask pair 174A and 174B (Figure 5E). The mask pair 172A and 172B (Figure 5D) is orthogonal to the mask pair 176A and 176B (Figure 5F). If the differences have values that are produced by an orthogonal set, the pixel in the RD image is set to 0 in process block 160. Otherwise, process block 150 selects one of the largest values by selecting the pair with the largest ratio as calculated in process block 136.

The method then continues in process block 152. If only one largest difference was identified in decision block 146, or only one largest difference with a ratio exceeding the threshold THR was identified in decision block 147, the method is continued in The process continues in the same manner at process block 152. In process block 152, an edge angle A₁ is selected based on the mask pair with the largest difference calculated in process block 136, or based on the pair selected in process block 150 if that process block was performed. For that pair, the mask producing the largest value in process block 132 or in process block 134 is identified and the angle associated with that mask is selected. The angles 45º, 90º, 135º, 180º, 225º, 270º, 315º and 360º are assigned to the masks 170A (Figure 5C), 172A (Figure 5D), 174A (Figure 5E), 176A (Figure 5F), 170B (Figure 5C), 172B (Figure 5D), 174B (Figure 5E) and 176B (Figure 5F), respectively.

A process block 154 calculates a second angle A2 that is equal to the arctan of the ratio of the difference calculated in process block 136 for the mask in Figure 5E and the difference calculated for the mask in Figure 5F. The signs of the differences are taken into account so that the arctan A2 can take on any value between 0º and 360º. The calculated difference for the mask pair of Figure 5D represents the strength of an edge in a 90º direction, while the calculated difference for the mask pair of Figure 5F represents the strength of an edge in a 360º (or 0º) direction. Thus, the arctan of the differences represents the angle of an edge.

Decision block 156 determines whether the absolute value of the difference between the angles A1 and A2 calculated in processing blocks 152 and 154 is greater than 45°. Recall that A1 and A2 represent the angles of an edge calculated in two different ways for a selected pixel. If the difference is greater than 45°, the pixel is set to 0 in the RD map in processing block 160 (i.e., the selected pixel is not shown as a possible edge in the RD map). Otherwise, the pixel is set to 1 in processing block 158. The complete RD map is obtained by repeating the processing described above (as shown in Figures 5A and 5B) for each pixel that is equal to 1 in the D-image.

Figures 6A show further details of how the output of the edge extractor 32 (Figure 2) is formed from the EBCC, D, SRBE and RD maps. The output of the edge extractor 32 (Figure 2) consists of an array of pixels referred to as EEB (i, j). Each Each pixel has either a value of 1 indicating an edge or a value of 0 indicating no edge. The method of Figures 6A and 6B sets each pixel in the EEB map to 1 corresponding to a 1 in the EBCC image. The other pixels in the EEB map that correspond to the pixels in the D map that represent probable edges are also set to 1.

In process block 180, the EBCC map is copied into a variable array called "TEMP". The "TEMP" array is processed one pixel at a time, with the pixel currently to be processed being selected by process block 182. In decision block 184, a check is made to see if the pixel in the D-map corresponding to the selected pixel is 0. If the pixel is 0, process block 190 ignores the selected pixel and the next pixel is considered.

On the other hand, if the pixel is not equal to 0, it may represent an edge. In decision block 186, the selected pixel and its four nearest pixels in the TEMP array are examined. The four nearest neighbors are the pixels below and to the sides of the selected pixel in the two-dimensional array. If all of these pixels are, the selected pixel is no longer considered an edge in process block 190 and the next pixel is selected. However, if any of the four nearest neighbor pixels is different from 0 (i.e., is an edge), process block 188 retains the selected pixel as a possible edge for further processing.

If it is determined in decision block 192 that more pixels remain to be processed, process block 182 selects the next pixel to be processed in a similar manner. When all pixels in the image have been processed, decision block 194 determines whether to repeat the selection process. The process is repeated if two conditions are met (1) the entire array has been processed less than fifty times; and (2) the group of pixels retained in the current iteration is different from the group of pixels retained in the last iteration. If it is determined in decision block 194 that another iteration will be performed, Process block 196 sets the pixels in the TEMP field corresponding to the retained pixels to 1 and all other pixels to 0.

Thus, the TEMP array begins with all pixels where edges have been detected by both the local edge detector 28 and the global edge detector 30 as possible edges identified. In addition, pixels that potentially represent edges indicated by a 1 in the D-map are added to the TEMP array at each iteration if they are adjacent to a pixel already identified as a possible edge. Once a sufficient number of iterations, as determined in decision block 184, have been performed, the remaining pixels, i.e., the len in the TEMP array, represent the pixels that are potentially edges.

All retained pixels in the TEMP field are considered to determine which pixels actually represent edges and should appear in the EEB map at the output of the edge extractor 32 (Figure 2).

A process block 198 selects a pixel at a time of processing. If decision block 200 determines that the pixel in the EBCC map is equal to 1, process block 202 sets the corresponding pixel in the EEB map to 1 and the next pixel is processed. If the selected pixel in the EBCC map is 0, a decision block 204 checks whether the corresponding pixel in the TEMP field is equal to 1, i.e., not considered as a pixel representing an edge.

If the pixel in the TEMP field is 0, this pixel is ignored and the next pixel is considered.

On the other hand, if the pixel in the TEMP field is 1, the corresponding pixel in the replacement ratio map RD is considered. If the pixel in the RD map is determined to be 0 by decision block 206, the next pixel is considered. If the pixel in the RD map is not, a determination is made in process block 208 as to whether the corresponding pixel in the output field EEB should be set to 1 using the following equation: Equation (3)

where "pixel" refers to a pixel corresponding to the selected pixel and "neighbor" refers to a pixel near the pixel corresponding to the selected pixel. Note that each pixel has eight neighbors - one on each of the four sides and one on each of the four diagonals. Thus, the right-hand side of equation (3) must be evaluated up to eight times, once for each neighbor pixel, with EEB(pixel) taking on a logical 1 if any of the eight neighbors on the right-hand side is a logical 1. If decision block 210 determines that more pixels remain, process block 198 selects the next pixel and processing repeats.

After completing the processing of Figure 6B, the EEB image has a 1 at each pixel where both the local edge detector 28 (Figure 2) and the global edge detector 30 (Figure 2) have reported an edge. In addition, the results of the processing of Figure 5A are used in decision block 206 and decision block 208 to identify and set to 1 additional pixels that are likely to represent extents of the confirmed edges. The EEB image thus represents the edges in the reference image that are most likely to represent image features that are consistent with subsequent images that may be taken from different angles of the target area. These edges in the EEB image represent confirmed edges.

Reference model generator

The reference model generator 34 (Figure 2) creates a model from the description of the edges in the EEB map. The model is a two-dimensional array of the same size as the EEB map. The reference model generator 34 assigns either a +1, -1, or 0 to each element in the model, based on the edge information contained in the EEB map and the pitch information contained in the SRBE map. The reference model generator 34 (Figure 2) assigns a +1 to those pixels within three (or other relatively small number) pixels of a confirmed edge that likely correspond to a pixel with high brightness if the model field is matched against a SAR image of the target area, while it assigns a -1 to pixels within the three pixels of a confirmed edge that likely correspond to pixels of low brightness when the model field is matched against the SAR image. Pixels that are not within the three pixels of a confirmed edge are assigned a value of 0. The reference model generator 34 (Figure 2) also calculates a correlation threshold k that is used by the edge model correlator 42 (Figure 2) to match the model to a SAR image taken by the SAR 18B (Figure 1) during the approach of the guided missile 16 (Figure 1) to its target.

Figures 7A and 7B show details of the processing performed by the reference model generator 34 (Figure 2) to calculate the model and the correlation threshold. All pixels in the EEB image are processed sequentially as pixels selected by process block 240. If this pixel does not correspond to an edge, as determined in decision block 242, the next pixel is processed. If the pixel represents an edge, i.e., the pixel is non-zero in the EEB map, a plurality of pixels are set accordingly in the model.

A decision block 246 checks whether the selected pixel is on the light side or the dark side of the edge. To determine which side of the edge the pixel is on, the level of the corresponding pixel in the SRBE map is compared to the value of the neighboring pixel in the SRBE map, which has a different value. If the value of the corresponding pixel is greater than the neighboring pixel, the pixel is on the light side of the edge. If the pixel is on the light side of the edge, a variable BRTT is set to +1 in process block 248. Otherwise, BRTT is set to -1 in process block 250.

A process block 252 selects the pixels in a 5x5 local window in the SRBE map that is centered on the pixel selected in process block 240. When processing the pixels under the local window, the four outermost edge pixels are excluded. If the local window is centered over the pixels near the sides of the SRBE image, portions of the local window may "spread" beyond the edges of the image. In this case, only the pixels in the image below the window are considered.

A process block 254 identifies any pixel in the local window that has the same value as the pixel in the SRBE map that corresponds to the pixel selected by process block 240 (i.e., the pixels on the same side of the edge). The corresponding pixels in the model are set to a respective value assigned to the BRTT by process block 248 or 250. The remaining pixels in the EEB map are then processed until decision block 256 determines that all have been processed, thereby completing the model.

Figure 7B shows the processing required to calculate the correlation threshold K. Each pixel in the model is processed in turn as selected by process block 258. If the pixel is determined to be zero by decision block 260, the next pixel is processed. If the pixel is determined to be greater than zero by decision block 262, the value of the corresponding pixel in the EPS image is added to a variable THRP by process block 264 and a counter variable is incremented in process block 266. On the other hand, if the pixel is less than zero, the value of the corresponding pixel in the EPS image is added to a variable THRM in process block 268 and a counter variable NTHRM is incremented in process block 270. The process is repeated for all pixels in the image as determined by decision block 272. A process block 274 calculates a value of a variable CR according to the equation

CR-(THRP/NTHRP)/(THRM/THRM) equation (4)

Note that CR should exceed the value 1. A CR value of less than 1 means that the reference image selected by the target address 19 (Figure 1) does not contain any edges that produce a good model. If CR is less than 1, the target address 19 (Figure 1) can select a different reference image. Finally, in process block 276, the correlation threshold K is calculated according to the following equation:

K = CR (In CR)/(CR-1) Equation (5)

,where In represents the natural logarithm function.

In a further embodiment of the invention, the entire processing according to Figure 78 can be omitted by setting the value K to 1.4.

Guided missile processing

The model and correlation threshold are then transmitted to the missile processing 40 (Figure 1). The missile processing 40 (Figure 1) compares the model and the images taken by the SAR 18B (Figure 1). Note that the processing required to generate the model is performed once in the aircraft processing 20 (Figure 1). As shown in Figure 2, identifying the target from the image taken by the SAR 18B (Figure 1) requires less processing.

It should be noted that the model was generated from a reference image consisting of a piece of an image taken by the SAR 18A (Figure 1). This piece of the image is selected to have sufficient edges and to yield a relatively high value for the CR calculated according to equation (4). For example, the reference image may be a 32x32 pixel array selected from a larger 64x64 pixel array. The target area image generated by the SAR 18B (Figure 1) will then similarly have a 64x64 pixel array. According to a method described in detail below, the missile processor 40 (Figure 2) would match the model to a 32x32 pixel portion of the target area image taken by the SAR 18B (Figure 1). The location of the alignment sections in the target area image provides information about the direction of the target in relation to the flight path of the missile 16 (Figure 1). For example, if the desired target location in the alignment section is directly to the right of the center of the target area image, the missile processing 40 (Figure 1) would determine that the target is to the right of the flight path of the missile 16 and send a signal to the missile tracking system (not shown), thereby guiding the missile 16 to the right. The missile processing 40 (Figure 1) can provide additional information to the missile Tracking system, after which the amount of trajectory tracking is adjusted.

The details of the processing to be performed by the edge model correlator 42 (Figure 2) to match the model with the target area image made by the SAR 18B (Figure 1) are shown in Figures 8A and 8B. Before discussing Figures 8A and 8B in detail, it should be noted that the target area image produced by the SAR 18B (Figure 1) may be filtered to remove noise and interference, such as by the non-coherent integrator 22B (Figure 2), before being processed by the edge model correlator 42 (Figure 2). The non-coherent integrator 22B (Figure 2) operates in a similar manner to the non-coherent integrator 22A (Figure 2) described above.

Returning now to Figures 8A and 8B, processing begins in the edge model correlator 42 (Figure 2) at process block 300, which selects a local window in the target area image generated by the SAR 18B (Figure 1). The local window is the same size as the model generated by the reference model generator 34 (Figure 2). The local window is slid over the image generated by the SAR 18B, with process block 300 selecting a local window position that has not been previously viewed.

A process block 302 selects a pixel in the model. Each pixel in the model is considered, with a process block 302 selecting a pixel that has not been previously considered. If the selected pixel is equal to zero, as determined by decision block 304, the next pixel in the model is considered. If the pixel is not zero, decision block 306 determines whether the pixel is greater or smaller.

If the pixel is greater than 0, the value of the corresponding pixel in the captured image is added to a variable SUMP in process block 308. In addition, a counter NPOS is incremented. Otherwise, if the pixel is less than 0, the value of the corresponding pixel in the captured image is added to a variable SUMN in process block 310 and a counter NNEG is incremented by one. Each pixel is processed sequentially until all pixels have been processed, which is done by decision block 312. Once all pixels within a local window have been processed, a process block 314 calculates a positive threshold THRP according to the following equation:

THRP = (1/K) (SUMP/NPOS),

where K is the correlation threshold generated by the reference model generator 34 (Figure 2) and SUMP and NPOS represent the variables set in the process block 308. A process block 316 calculates a negative threshold THRM according to the following equation:

THRM = K (SUMN/NNEG),

where SUMN and NNEG represent the variables set in process block 310.

A process block 318 creates a binarized array of the same size as the model. All pixels in the binarized array correspond to zero pixels in the model. For each pixel in the model that has a value greater than zero, the corresponding pixel in the local window selected by process block 300 is compared to the threshold THRM. For each pixel in the model that has a value less than zero, the corresponding pixel in the local window is compared to the threshold THRP. For each threshold value, the corresponding pixel in the binarized array is equal to +1 if the pixel in the local window is above the threshold. Otherwise, the pixel in the binarized array is equal to -1.

A process block 320 compares the pixels in the binarized image with the model, thereby determining the values for the following variables:

R�1 = number of pixels in the model with values less than 0;

R₂ = number of pixels in the model with values greater than 0;

S�1 = number of pixels in the binarized field with values less than 0;

S₂ = number of pixels in the binarized field with values

H₁₁ = number of pixels in the model with values less than 0, where the corresponding pixel in the binarized field is less than 0;

H₁₂ = number of pixels in the model with values less than 0,

where the value of the corresponding pixel in the binized field is greater than 0;

H₂₁ = number of pixels in the model with values greater than 0,

where the value of the corresponding pixel in the binized field is less than 0;

H₂₂ = number of pixels in the model with values greater than 0, where the value of the corresponding pixel in the binized field is greater than 0.

The numerical values R₁, R₂, S₁, S₂, H₁₁, H₁₂, H₁₂, H₁₂, H₂₁, and H₂₂ are used in process block 322 to calculate the correlation map CM according to the following equation: Equation (6)

,where

k = 2CC/(R₁ + R₂), and

A and CC are constants chosen empirically to vary the enhancement of the correlation map. Typically, CC is between 0 and 41 while A is between 1 and 7. Note that process block 320 falls within a loop of the flow chart so that a value of the CM is calculated for each local window selected from the target region image. A process block 324 checks each value of the CM and stores the largest one. Process block 324 also stores the location in the target region image at which the local window selected by process block 300 found the largest value of the CM. A decision block 326 causes the entire process to be repeated for each possible local window in the acquired image.

When processing is complete according to Figure 8, the largest value of the CM has been identified by process block 324. The largest value of the CM implies that the model fits best with the local window in the

Calculating the value of the CM used to best match the target area image. The location of the matching local window in the target area image formed by the SAR 18B (Figure 1) indicates the location of the target in the target area image (i.e., the center of the matching local window in the image generated by the SAR 18B has the same relationship to the target as the center of the reference image selected by the target address 19 (Figure 1) has to the target in the image generated by the SAR 18A). This location can be fed to the missile control system for adjusting the missile trajectory.

Alternative embodiment of the model generator and the edge model correlator

An alternative method may be used to generate a model and match that model to the target region image. The alternative method replaces the methods of Figures 7A, 7B, 8A, and 8B. The alternative methods require more processing steps in the reference model generator 34 (Figure 2), making them less desirable than the preferred embodiment in some applications. However, this alternative embodiment described below may generate a model that more closely matches the target region image, often resulting in fewer processing steps in the edge model correlator 42 (Figure 2).

In the preferred embodiment, the model was a field representing confirmed edges detected in the reference image. A simple threshold K was calculated for the model and used by the edge model correlator 42 to match the model to the target area image. In the alternative embodiment, the model consists of a pair of pixel lists for each edge identified in the reference image. A separate threshold is calculated for each edge. The separate thresholds for each edge increase the likelihood that the edge model correlator 42 will match the model to the portion of the target area image that actually contains the target. The alternative representation of the model provides an additional advantage in that the computer implementation of the missile processing 40 (Figure 2) requires less digital memory to store of the model than if the model were created using the preferred embodiment.

Figure 9A shows the processing performed by reference model generator 34 (Figure 2) according to the alternative embodiment. A process block 350 selects a pixel from the EEB map. Since each pixel in this map is processed, the process block 350 selects a pixel that has not been previously processed. A decision block 352 determines whether the selected pixel is equal to 0, i.e., it does not represent an edge. If the selected pixel does not represent an edge, the next pixel is selected from the EEB map if there are more pixels to process. A decision block 354 determines whether there are more pixels.

The selected pixel representing an edge is the first entry in a list of pixel positions along an edge. Subsequent processing adds the positions of the other pixels along the edge, checking neighboring pixels to see whether they fall under the edge.

A decision block 358 determines whether each of the four closest neighbor pixels to the selected pixel have been checked for possible addition to the list. If one or more of the closest neighbor pixels remain to be tested, the process block 360 selects the remaining neighbor pixels. On the other hand, if all of the neighbor pixels have been tested, processing proceeds to decision block 354 where another pixel is selected. Note that if two pixels are adjacent and the second is tested to see if it is on the same edge as the first, it is not necessary to test the first to see if it is on the same edge as the second. A decision block 358 takes advantage of this commutative property, thereby avoiding redundant processing of the neighboring pixels.

A decision block 362 skips the selected neighboring pixels if the pixel in the SRBE map that corresponds to the pixel selected in process block 350 and the pixel selected by process block 360 adjacent pixel has the same value (ie, it lies on the same side of the edge). The adjacent pixel is also skipped if its value is 0 (ie, it does not lie on an edge) or if the pixel pair has already been processed in conjunction with a pixel list along a different edge.

If it is determined at decision block 362 that the neighboring pixel was not skipped, its location becomes part of the pixel location list along an edge. A process step 364 uses the two selected pixels as a starting point to identify the pixels along that edge. Figure 9B shows a portion 410 of a two-dimensional array and shows how the process block 364 selects a pixel to add to the pixel list of an edge. Pixel 4001 represents the pixel selected by process block 350 (Figure 9A). Pixel 4002 represents a neighboring pixel selected by process block 360 (Figure 9A). Nodes 4021 and 4022 represent the points at which four pixels are connected together. Since the pixels 4001 and 4002 were selected on different sides of an edge, the edge segment 404 represents the edge termination.

The edge is extended by finding additional pairs of pixels on opposite sides of the edge and, in addition, on at least one of the pixels already found on the edge. The edge can be extended from either of its two sides. In Figure 9B, the edge can be extended from node 4021 or from node 4022. In this case, it has been arbitrarily assumed that the edge is extended from node 4022.

In extending the edge from node 402₂, it may be extended to possible nodes 406₁, 405₂, or 406₃. The direction in which the edge is extended depends on which side the edge pixels 408₁ and 408₂ fall on. As described above in connection with the description of the global edge detector 30 (Figure 2), the SRBE map has divided the reference image into a number of regions and, for the purposes of this model generation, defined the edges to fall between these regions. If pixel 408₁ is in the same region as pixel 400₁ as described by the SRBE map, and if the If pixel 408₂ is in the same region as pixel 400₂, then the edge must have candidate node 406₂, so pixels 408₁ and 408₂ are added to the list of edge pixels. Another possibility is that pixels 408₁ and 408₂ are in the same region as pixel 400₁, in which case the edge is extended to include candidate node 406₃, so pixel 408₂ is added to the list of edge pixels. The final possibility is that pixels 408₁ and 408₂ are in the same region as pixel 400₂, in which case the edge is extended to include candidate node 406₁. , which is why pixel 408₁ is added to the list of edge pixels.

The edge is further extended by following the pattern shown in Figure 9B. Node 4022 was the last node added to the edge, while node 4021 was the second to last node added to the edge. Once one of the possible nodes 4061, 4062, or 4063 is added to the edge, node 4022 is the second to last node and whatever possible node was added to the list now represents the last node. Using the same pattern shown in Figure 9B, a new set of possible nodes is determined near the last node on the edge. When following the pattern, the two pixels that are near the last node added to the edge and near the second to last node added to the edge are used where pixels 400₁ and 400₂ were used. Similarly, where pixels 408₁ and 408₂ were used, the two pixels that are directly adjacent to the last node added to the edge and not adjacent to the second to last node added to the edge are used.

The pattern shown in Figure 9B is traced until the entire edge has been captured. The end of an edge is detected when two pixels adjacent to the last node and not adjacent to the second to last node (for example pixels 4081 and 4082) do not lie on an edge as represented by the EEB map (ie, the corresponding values in the EEB map are 0). The end of an edge can also be detected when the pixels considered for tracing the pattern do not exist, since a boundary of the field is reached. On the other hand, the edge end is detected when any of the pixels considered for tracing the pattern of Figure 9B have already been added to a list of edge pixels, as is the case when a closed edge is being traced in a circle. The edge end is also detected when two pixels adjacent to the last node and not the second to last node (e.g., pixels 4081 and 4082) do not have the same value as one of the pixels selected by process block 350 or 360 (i.e., pixels 4001 or 4002). This condition can only occur when the alternative method of forming the SRBE image is used such that the pixels in that image take on values of 1, 2, 3, or 4.

Once process block 364 has completely traced the edge in one direction, processing proceeds to process block 366 where the edge is traced in the other direction. As with tracing through process block 364, the same pattern is followed, except that in process block 366, node 402₁ is initially considered the last node and node 402₂ is initially considered the second to last node. The edge is then extended from node 402₁. In the special case that the edge was traced in a closed loop in process block 364, process block 366 performs no further processing.

Once the edge has been traced to both ends by process blocks 364 and 366, the list of pixels on the edge is broken into two lists by process block 368. Any pixel that is on the high brightness side of the edge, as indicated by a large value in the SRBE map, is placed in the high brightness list. The remaining pixels are placed in the low brightness list. The lists also have the pixels that are close to the edges, for example within three pixels on either edge side.

A process block 370 calculates a correlation threshold K for the edge represented by the first pair of lists generated by the process block 368. The process block 370 uses the method described above and shown in Figure 7B. Since the edge pixels have already been identified and are in high and low brightness pixel lists of the Edge sides have been deshuffled, for simplicity the processing of Figure 7B which performs this function is omitted here. Decision block 260 in Figure 7B checks whether a selected pixel is on an edge. Since all of the pixels in the list are on an edge, this decision block can be omitted. Decision block 262 checks whether a selected pixel is on a high or low brightness edge side. Since the pixels have already been sorted into a high and low brightness list, decision block 262 need only delegate program execution to process block 260 when pixels are selected from the high brightness list in process block 258, and delegate it to process block 268 when pixels are selected from the low brightness list.

Returning to Figure 9A, decision block 372 checks whether the value of K exceeds 1 once a correlation threshold K has been calculated in process block 276 (Figure 7B). If the value of K is too small, the pair of lists is discarded by process block 274 such that these lists are not further considered for inclusion in the model. On the other hand, if K is large enough, the lists are stored for further processing by process block 276. Process block 276 also stores the correlation threshold K calculated for these lists. Regardless of whether the lists are retained, processing returns to decision block 358.

Decision block 358 determines, as described above, that all neighbors of the pixel selected by process block 350 have been checked for location along the same edge as the selected pixel. Once all neighbors of the selected pixel have been checked, another pixel is selected until decision block 354 determines that no more pixels remain to test. However, decision block 362 skips any pair (of a pixel and a neighbor) if the pixel is already in a pixel list near an edge and the neighbor is present on the list on the other side of the edge. On the other hand, even if the pixel is already on a list, the selected pixel may be a pixel on a different edge than the one previously tracked. In In this case, the edge is followed further by process blocks 364 and 366, possibly resulting in another list pair. This list pair may also be stored by process block 376 depending on the comparison results made by decision block 372. Thus, if all pixels in the EEB map have been selected and processed, multiple list pairs of pixels along different edges may be retained by process block 376. If it is determined in decision block 354 that all pixels in the image have been processed, the retained list pairs form the basis for the model.

The retained list pairs may also be further processed by process block 356. The details of this processing are described below in connection with Figures 9C and 9D. The output of process block 356 corresponds to the output of reference model generator 34 (Figure 2). Alternatively, process block 356 may be skipped, with the output of reference model generator 34 (Figure 2) consisting of all of the list pairs retained by process block 376.

Referring to Figures 9C and 9D, the processing of the retained list pairs of pixels on the edges is shown. Each list pair represents an estimate of the position of an edge. For a variety of reasons, the estimate of the edge position is not the best possible estimate of that edge position. The processing shown in Figures 9C and 9D serves to modify the list pairs describing the edge, thereby finding a better estimate of the edge position.

A process block 500 selects one of the list pairs stored by process block 376 (Figure 9A). Each list pair must be processed, with process block 500 selecting a pair that has not been previously processed.

A process block 502 selects the pixels from the filtered image produced by the smoothing filter 26 (Figure 2). The process block 502 selects these pixels according to the pixels in the high brightness list of the selected list pairs.

The process block 504 sorts the selected pixels according to their respective brightnesses. A decision block 505 checks whether a new threshold can be calculated to identify the edges by dividing the selected pixels into those above and below the threshold (the edge is likely to be between the areas of the pixels above and below the threshold). To determine if a suitable threshold exists, decision block 505 identifies the pixel produced by process block 504 at one quarter of its way from the ground. Decision block 504 then determines if any other pixel selected by process block 502 has values that are different from the value of the pixel that is one quarter from the ground. If such a pixel with a different value exists, a new threshold is calculated in process block 506. Otherwise, processing resumes at decision block 512.

Process block 506 calculates the new threshold. The threshold represents the average of the value of the pixel that is one-quarter from the bottom of the sequence generated by process block 504 and the value of the next neighboring pixel in that sequence that has a different value. In this case, "closeness" is measured proportionally to the end of the sequence generated by process block 504. For example, a pixel one-eighth from the bottom of the sequence is farther from the pixel one-quarter from the bottom than a pixel one-half from the bottom, since one-eighth is half of one-quarter from the bottom, while half is only one-third of the way from one-quarter to the top.

A process block 82A then binarizes the filtered image using the calculated threshold. The pixels in the split image that correspond to the pixels in the filtered image with brightnesses below the threshold are given a value of 1. The pixels in the split image that correspond to the pixels in the filtered image with brightnesses above the threshold are given a value of 4. The binarization process is identical to that of process block 82 (Figure 3A), and thus a split image is obtained in a similar manner. However, process block 82A operates on selected pixels rather than the entire image. Process block 82A operates only on the pixels selected by process block 502 and the eight nearest neighbors for each of those pixels.

The split image serves as input to edge extractor 32A. Since edge extractor 32A is identical to edge extractor 32 (Figure 2), the processing already described in connection with Figures 6A and 6B is performed, except that only a small portion of the image is affected. Process block 32A operates on the same group of pixels as process block 82A.

The confirmed edge map generated by edge extractor 32A is further processed by reference model generator 34A. The operation of an alternative embodiment of reference model generator 34 (Figure 2) is shown in Figure 9A. Reference model generator 34A performs the processing of Figure 9A with one exception. Process block 356 (Figure 9A) in which the list pairs of pixels along the edges are prepared is not performed. The list pairs with their corresponding correlation thresholds retained in process block 376 (Figure 9A) are simply provided as outputs to reference model generator 34A in Figures 9C and 9D.

A decision block 510 compares the correlation thresholds of any list pair generated by the reference model generator 34A to the correlation threshold of the list pair selected by the process block 500. If no list pairs were generated by the reference model generator 34A, processing proceeds to decision block 512. If the correlation thresholds of all newly generated list pairs are below the correlation threshold of the list pair selected by the process block 500, no further processing is performed on the newly generated list pairs. Processing then continues at decision block 512 with the list pair selected by the process block 500, effectively ignoring the newly generated list pairs. If it is determined at decision block 510 that any of the newly generated list pairs has a correlation threshold that exceeds the correlation threshold of the list pair selected by the process block 500, exceeds , processing continues at decision block 516. Decision block 516 determines whether more than one pair of lists was generated by reference model generator 34A. If only one pair of lists was generated, the pair of lists selected by process block 500 is deleted from the model at process block 518 and the newly generated list is added to the model.

If multiple list pairs were generated by reference model generator 34A, decision block 520 determines whether all of the newly generated list pairs have correlation thresholds that exceed the correlation threshold of the list pair selected by process block 500. If all of the newly generated list pairs have correlation thresholds above the correlation threshold of the list pair selected by process block 500, process block 522 retains all of the list pairs to use as part of the model for further consideration. Process block 518 replaces the list pair selected by process block 500 with one of the newly generated list pairs. The other newly generated list pairs are stored by process block 522. These newly created list pairs are referred to as "spin-off list pairs" and are treated the same as the list pairs retained in process block 376 (Figure 9A).

If all of the newly generated list pairs do not have correlation thresholds that exceed the correlation threshold of the list pair selected by process block 500, decision block 524 examines the largest correlation threshold of all of the newly generated list pairs. If this correlation threshold is sufficiently greater than the correlation threshold of the list pair selected by process block 500, all of the newly generated list pairs are processed in process blocks 522 and 518 as described above. Otherwise, none of the newly generated list pairs are processed and processing continues at decision block 512. In the present embodiment, decision block 524 determines that a newly generated correlation threshold is sufficiently greater than another if it is at least 5% greater or at least 0.1% greater.

If any of the conditions described above occur such that process block 518 replaces the list pair selected by process block 500, processing returns to process block 502. The processing described above is then repeated with the new list pair used in place of the list pair previously selected by process block 500. However, until some limits are imposed, processing may repeatedly return to process block 502. Decision block 526 determines how many times processing returns to process block 502. If process block 518 has been executed three times without executing decision block 512, decision block 526 prevents processing from returning to process block 502. Instead, decision block 512 is executed.

Recall that process block 502 selects the pixels in the EPS image corresponding to the pixels in the high brightness list of the list pair selected by process block 500. The low brightness list of the selected list pair may be processed in a similar manner. Decision block 512 determines whether to process the low brightness list of the selected list pair. In this case, the selected list pair may be the list pair selected by process block 500 or the replacement created for that list pair by process block 518. If the selected list is a list pair stored by process block 522 or is a replacement list pair stored by process block 518, the low brightness list of the pair is processed only if the correlation threshold of the selected list pair is less than the correlation threshold of the original list pair. If the low brightness list has been processed or does not need to be processed, processing proceeds to decision block 530 where it is checked whether the correlation threshold of the selected list pair exceeds a minimum value, in this case 1.25. If the correlation threshold is below this value, the list pair is discarded by process block 532 and does not become part of the model. Processing then continues at decision block 528 which returns processing to process block 500 where another list pair is selected and the entire processing is repeated.

When processing the low intensity list of the selected list pair, process block 502 selects the pixels in the EPS image corresponding to the pixels in the low intensity list. The processing of these selected pixels is performed in the same manner as the processing of the pixels selected using the high intensity list. The only difference is that decision block 505 and process block 506 operate on the pixel three-quarters of the way from the bottom of the sorted list, which is opposite the pixel one-quarter of the way from the bottom.

Once decision block 528 determines that all retained list pairs have been processed, the model preparation process is completed. The list pairs with their corresponding correlation thresholds then form the model that is the output of reference model generator 34 (Figure 2).

It should be noted that the model generated by the alternative embodiment of the reference model generator 34 (Figure 2) comprises a model having a different frequency from that generated by the preferred embodiment of the reference model generator 34 (Figure 2). Therefore, a modification of the edge model correlator (Figure 2) is required.

Figure 10 shows the processing performed by the alternate embodiment of the edge model correlator 42 (Figure 2). A process block 300A, in conjunction with decision block 326A, moves a local window over the target region image in the same manner as process block 300 and decision block 326 (Figure 8).

A process block 550 selects one of the list pairs that make up the model. Each list pair is processed in turn such that the process block 550 selects a list pair that has not been previously processed. A decision block 552 ensures that the selected list pair has entries in both the high brightness and low brightness lists. If either list has no entry, that list pair is skipped. Otherwise, processing continues at process block 554.

Process block 554 calculates two thresholds for the selected list pair. The thresholds THRP and THRM are calculated according to the following equations:

THRP = (SUMP/NPOS)/K

THRM = K (SUMN/NNEG)

where

K is the correlation threshold for this list pair;

SUMP the sum of the brightnesses of all pixels in the local window of the target area image corresponding to the pixels in the high brightness list;

SUMN the sum of the brightnesses of all pixels in the local window of the target area image corresponding to the pixels in the low brightness list;

NPOS the number of pixels in the high brightness list; and

NNEG is the number of pixels in the low brightness list.

A process block 556 uses the thresholds THRP and THRM to form two binary image lists, labeled SENB+ and SENB-. The SENB+ list has the same number of entries as there are pixels in the high brightness pixel list. The SENB- list has as many entries as there are pixels in the low brightness pixel list. For each pixel in the high brightness list, the corresponding pixel in the local window of the target area image is identified. If the brightness of that pixel exceeds THRM, the entry in the SENB+ list corresponding to the pixel in the high brightness list is set to 1. Otherwise, the entry in the SENB+ list is set to -1. Similarly, for each pixel in the low brightness list, the corresponding pixel in the local window of the target area image is identified. If the intensity of this pixel exceeds THRP, the corresponding entry in SENB is set to 1. Otherwise, it is set to -1.

A process block 558 updates the variables R1, R2, S1, S2, H11, H12, H21, and H22 based on the values assigned to the entries in the SENB+ and SENB- lists. The variables, which are initially set to 0 prior to the start of processing in Figure 10, are updated according to the following equations:

R�1 is increased by the number of pixels in the low brightness list;

R2 is increased by the number of pixels in the high brightness list;

S1 is increased by the number of positive entries in SENB+ and SENB-;

S2 is increased by the number of negative entries in SENB+ and SENB-;

H�1;�1 is increased by the number of positive entries in SENB-;

H₁₂ is increased by the number of negative entries in SENB-;

H₂₁ is increased by the number of positive entries in SENB+; and

H₂₂ is increased by the number of negative entries in SENB+.

Once the variables for a list pair have been updated, decision block 560 determines if any list pairs remain to be processed. If so, processing returns to process block 550. The next list pair is selected and processed. When all list pairs have been processed, decision block 560 redirects processing to process block 322A. Note that process blocks 322A and 324A, as well as decision block 326A, are identical to process blocks 322, 324, and decision block 326 (Figure 8). Thus, the results from the alternate embodiment of edge model correlator 42 (Figure 2) are similar in form to the results produced by the preferred embodiment of edge model correlator 42 (Figure 2).

Claims (4)

1. A method for matching a first image and a second image of a same scene, wherein both the first and the second image have a pixel array, comprising the steps: forming (25, 30, 32) a reference map of a portion of the first image, the reference map comprising an array of pixels each corresponding to a corresponding pixel of the portion of the first image, and the values of the pixels of the reference map indicating the position of one or more maps represented in that portion of the first image, the reference map being successively compared (42) with the portions of the second image and a mark being calculated (42) for each portion of the second image, the mark indicating the probability that the pixels in the portion of the second image corresponding to the pixels in the reference map have values indicating one or more edges in the second image; and Selecting (42) the portion of the second image which produces the highest mark to correspond to the portion of the first image, characterized in that the reference map is formed by the steps of: forming (28) a first map of the portion of the first image, wherein the first map comprises an array of pixels each corresponding to a corresponding pixel of the portion of the first image and having a value indicating whether the pixel represents an edge in a local window (120,122,124,126) surrounding the corresponding pixel in the first image; forming (30) a second map of the portion of the first image, wherein the second map consists of an array of pixels each corresponding to a corresponding pixel of the portion of the first image and having a value indicating whether the corresponding pixel in the first image has a value that exceeds a threshold; and combining (32) the first map with the second map, thereby forming the reference map. 2. Method according to claim 1, characterized in that the first image represents an area in the vicinity of a target (T) and is generated by a first SAR (15A) in an aircraft (14) carrying a guided missile (16) and the second image is generated by a second SAR (18B) in the missile (16) after the missile (16) has been fired from the aircraft (14); and that the tracking (44) of the guided missile (16) is carried out on the basis of the selected section of the second image which produces the highest mark. 3. Method according to claim 1 or 2, characterized in that the reference map is processed (34) for comparison with the sections of the second image so that a model is created. 4. Method according to claim 3, characterized in that the model has a list pair of pixel brightnesses that represent the edges.