GB2134742A - Investigation of surface topography from stereo images - Google Patents

Abstract

In the investigation of surface topography estimates of elevation are derived from stereo images 13, 14 of a surface 10 presented photographically or displayed by a cathode ray tube. The space 17 containing the surface 10 is modelled as a three-dimensional array of volume elements each defining a unique plan position and elevation. The projection geometry of the images being known, a calculation is made, for a selected plan position 18, of the positions of the pair of image points conjugate to each element of elevation 20 and the brightness of each point is measured. The pair 22, 23 most closely similar in brightness has the highest probability of correspondence to a single point on the surface 10, thus identifying the relevant element 21 and elevation. <IMAGE>

Description

SPECIFICATION Investigation of surface topography from stereo images The invention relates to the investigation of the three-dimensional topography of a surface for which stereo images are available.

In the technique of photogrammetry it is well known that the surface topography of a terrain can be studied conveniently from a stereographic pair of air photographs for which the viewing geometry is known. The usual procedure involves the manual identification of a particular feature which is present in both photographs and must represent a single ground position. From a knowledge of the angular and linear separation of the cameras and of the conjugate positions in the photographs the plan position and elevation of the ground position can be calculated. Repetition by an operator of this matching procedure for a large number of features is both time consuming and liable to error as a result of visual fatigue.This has led to attempts to automate the procedure by the use of pattern correlation techniques but substantial problems remain since the acceptance of a single false match not only produces a false ground position but must confuse subsequent matchings.

Furthermore, in such correlation techniques there can be no certainty that any range of potential matches which may be considered will include a valid match.

The same problem arises in industrial applications of stereoscopic optical microscopy and electron microscopy, for example in the development and production of microfabricated integrated circuits.

In accordance with the invention a method for the investigation of the three-dimensional topography of a surface represented by a pair of stereo images comprises the operations of modelling the space containing the surface by designating an array of volume elements in a coordinate a system such that a relative plan position and a relative elevation may be identified for each element, determining for each element in a sequence of elevations at a selected plan position the positions of the conjugate pair of points in the images, deriving from the images for each such pair of points a level of probability that the points of that pair correspond to a single physical feature of the surface, determining the pair of points for which the level of probability is a maximum, and assigning to a representation of the surface at the selected plan position the value of elevation of the element to which that pair of points is conjugate.

The level of probability may be derived from the values of a predetermined brightness parameter for each conjugate pair of points.

The brightness parameter may be a direct measure of brightness, observed in a manner appropriate to the image medium which may be a photographic transparency or print or the screen of a cathode ray tube.

Alternatively the brightness parameter may be a linear gradient or other function relating the brightness of each conjugate point and of the surrounding points in the respective image. It will be understood that the term 'point' is used for convenience in relation to an image to denote an elemental area, the actual size of which will be limited by the resolution of the imaging and brightness measurement systems.

The derivation of the level of probability of correspondence for a pair of points may include the operation of determining the value of a normalised probability distribution function corresponding to the difference between the brightness parameter values. It is necessary to normalise the function in terms of the noise level in the images under investigation either by direct measurement or by applying a noise factor which may have been established from experience as characteristic of the class of image.

It will now be appreciated as an advantage of the method that the geometrical procedure precisely defines a set of pairs of conjugate points for a specified plan position and that the set must include the pair which is a valid match for the element on the surface. For well contested images that pair is readily identified. The resultant assigned elevation must then be correct. The further development of the method is therefore directed to overcoming possible errors arising from less favourable image conditions. For example, unexpected noise in the image might produce an anomalous measured level of brightness but steps can be taken to remove such an error for any image subject which is known to present a substantially continuous surface within the limit of resolution of the image recording system.

Thus the method may further include the operation of eliminating a pre-determined range of high-frequency components from a signal which represents the level of probability of correspondence. Such a smoothing or filtering operation may be performed if the value of the probability level first derived for a volume element at a designated plan position is modified for consistency with the values derived for elements which are adjacent in the x, y and z directions before determining the element for which the value is a maximum at that plan position.

The value first derived may be so modified by arithmetical averaging.

In an image pair having extended regions of substantially uniform brightness the levels of probability derived from pairs of conjugate points within such regions may appear invariant with elevation. This situation may be resolved by weighting the probability level first derived for a volume element at a designated plan position, by the use of a relaxation procedure, in dependence on the values of probability level derived for elements at adjacent plan positions, such that values of elevation assigned within the region are rendered consistent with those bounding the region.

A further situation in which probability levels may appear inconsistent with prevailing indications as to the nature of the surface is that of occlusion, when a surface feature is visible in only one image of the stereo pair. Thus, for any plan position and known stereo geometry, by comparison of the surface elevation probability level with the levels for surrounding plan positions, a further value of probability may be derived that the surface element at that position is occluded. A level of such further value may be set above which that plan position is omitted from the operation of assigning a value of elevation.

In situations which provide more than two views of a single area the results from the investigation of a first pair of images may of course be confirmed or refined by repeating the method for any other pair in the same coordinate system and combining the resultant probability levels.

In a preceding paragraph the general advantage of the method based on precise geometrical data has been emphasised. The method however also enables a more conclusive investigation to be made than would otherwise be possible when the projection geometry is only approximately known.

For this purpose the method may include the operations of deriving a set of probability levels relating to a set of plan positions for each of a range of values of the projection parameters and determining the average value of each set, the elevation to be assigned at each plan position being derived from the set of probability levels for which the average value is a maximum.

The method may be performed advantageously by providing means under the control of a suitably programmed computer for automatically deriving and storing image data related to selected volume elements.

In such automatic operation the means for deriving data from photographic images may comprise cathode-ray tube means for illuminating the image by a digitally advanced scanning beam and photo-detection means for measuring the image intensity at each step, the output signal from the photo-detection means being directed to a store addressed in synchronism with the scanning rate.

Suitably programmed computing means may further be arranged for automatically deriving from the stored image data that value which determines the elevation to be assigned to a selected plan position.

The method may further include the automatic printing or display of a representation of the surface in response to the production of the assigned surface elevation value for each selected plan position.

A method of carrying out the invention wil be described by way of example with reference to the accompanying drawings in which: Figure 1 represents diagrammatically the recording of stereo images in aerial mapping and illustrates the principle of the method of the invention; Figure 2 represents schematically the arrangement of an automatic system for the investigation of surface topography from the images of Figure 1 in accordance with the invention.

The basic concept of the invention will be explained with reference to Figure 1 which illustrates the aerial mapping of an area shown in vertical cross-section and having an undulating surface 10. Two cameras are used having respective viewpoints effectively located at points 11 and 12. The associated images are recorded in planes 13, 1 4 and the projection geometry is such that both images inlcude a view of the surface 1 0 between points 1 5 and 1 6. It is to be assumed that only the broad limits of variation in elevation of the surface 10 are known and that the profile as drawn is schematic. The known limits can however be contained within a space model bounded by a frame 1 7 in the plane of the crosssection.Locations within the model are denoted by an x, y and z coordinate system in which x represents horizontal displacement, z represents vertical displacement, andy represents the extension of the model through the plane of the paper. The scale of the coordinate system must be chosen to be consistent with the resolution available in the image, taking into account the distance of the cameras from the ground. If a plan position 1 8 is now selected for investigation the value x, y is known but z is unknown. Dependent on the chosen coordinate scale the space along the z axis at position 1 8 is sub-divided into a number of equal volume elements 20. Any one of such elements may, so far as we know, lie at the surface 10 and all of the elements must be tested.

For example, an element 21 is in fact at the surface 10 and is imaged at a point 22 in plane 13 and at a point 23 in plane 14. A second element 25 is in fact in mid-air but is notionally imaged at a point 26 in plane 13 and at point 27 in plane 14.

Similarly an element 30 is in fact underground but is notionally imaged at point 31 in plane 13 and at point 32 in plane 14. On examining the photographs the positions of the postuiated image points can be precisely identified from the projection geometry. Measurements of brightness can then be made at these image points and comparisons made between points 22 and 23, 26, 27, 31 and 32. It is established that two images of a single surface feature have a high probability of being closely similar in brightness and that images of different features have a much lower probability of similarity. As a result of the brightness comparison measurements, a level of probability is now deduced for each pair of image points that they derive from a common source. A high level is predictable for the pair 22, 23 relating to element 21 in the surface and low levels for the pairs 26, 27 and 31,32 relating to elements remote from the surface. For example, points 26, 27 in fact relate to surface positions 35, 36 respectively, these being remote from plan position 18 and from each other so that confusion is improbable, although fortuitously possible. By testing all elements 20 and assigning to the surface 10 the element for which the probability level is a maximum the risk of an invalid selection is minimised. For elements near to the surface the points actually imaged are close together on the surface and would be expected to show fairly high probability levels.A true maximum will therefore be quite broadly based whilst a false match, e.g. between positions 35 and 36 will produce a spike of high probability for element 25 which is recognisable and can be rejected.

It may be noted that in the cross-correlation method which is known in the art, such a false match, identifying image points 26 and 27, would carry no warning of the error. The greater risk in cross-correlation however lies in the absence of any guidance as to the regions of the images which are suitable for cross-correlation. Regions of the surface containing sharp variations in elevation will not produce similar patterns in the images and will not correlate well. A false match may also be produced if a region is visible to one of the imaging systems but not to the other.

Clearly if an image point 22 should thus become associated with a point remote from point 23 in plane 14, then a substantial error in the computed plan position and elevation becomes inevitable.

in the same way that point 21 has been established on surface 10 at plan position 1 8 it is necessary to follow the procedure either for all possible plan positions in the space model or for all those which are to be taken into account. Some part of the viewing area may be less critical and may be treated with coarser resolution or omitted.

There is complete freedom to select the area for analysis since it is clear that correspondingly only those points in the images which are relevant to that area need to be selected for brightness measurement.

The procedure can be carried out automatically under computer control in the manner to be described with reference to Figure 2. Brightness measurements are made by exposing a pair of photographic transparencies 40, 41 to a scanning light-beam projected from a cathode-ray tube 42.

A precisely positioned scanning action is proudced by a digital deflection unit 43 which is controlled by a control unit 44. At each desired point of each transparency the intensity of the transmitted light is measured by means of respective photomultiplier units 45, 46.

Control unit 44 (an operating unit of a computer 47) holds details of the space model coordinate system so that any volume element from x,y,z, to xnynzn can be located. Values of the projection geometry parameters are entered and unit 44, as programmed, calculates the conjugate positions to be observed in transparencies 40, 41 for each volume element of interest. The required control signals are then passed to deflection unit 43.

As was explained with reference to Figure 1 the invetigation proceeds by considering all plan positions in turn and all the elements z0 to z, at each plan position. Thus for coordinates xayb the conjugates of z0 are determined and the brightness measured at those points. The two intensity signals from photomultipler units 45, 46 are input to a comparator 48 which operates to measure the difference between the two inputs and to establish a level of probability that the difference is compatible with a common physical origin for the two image points. This requires comparison of the difference value with a normal probability distribution function for which zero deviation corresponds to a maximum level of probability.The standard deviation of the function must be normalised to the noise level of the image system and this process will involve specific measurement of the sytem until sufficient experience of it is accumulated.

The value of probability level derived in this way from comparator 48 relates to the element at xaybzO and values are derived similarly for elements zr to z,. The sequence of values is held in a store 51 and scanned from z0 to Zn to detect a maximum in the distribution of probability levels.

The element at the maximum represents the elevation of the surface at xa, yb. The procedure is repeated for all the required x, y positions, the value of relative surface elevation at each position being scaled to true height by computer 47 for output to a visual display unit 52 and/or printer 53. Under the control of computer 47 the display or print-out can be arranged for example as a contour map, a vertical cross-section or for the isolation of any desired feature with colour or brightness emphasis.

In practice it is preferred to modify the procedure described in the preceding paragraph in order to reduce or eliminate the effect of chance matchings of brightness which would produce a spike level of probability in a region of generally low levels. The modification involves the operation of filtering high-frequency components from the comparator output signal by means of a spatial filter 50, before the resultant probability value is entered in store 51. The filtering operation is based on an assumption of continuity of the surface within a small region which thus requires continuity within the volume containing that region of the surface. In the modified procedure therefore the discrete value of probability level derived from comparator 48 for any one of the elements 20 is averaged with the values for the adjacent elements in the x, y and z directions for entry in store 51.Since such an average value is required for each element, comparator output data is accumulated for all the elements of the space model (or for a sufficient number of them to define a region of interest) before the filtering process is applied.

An on-line measurement system has been described but it will be clear that the method of the invention would permit the initial storage of data from photomultiplier units 45, 46 and the subsequent analysis of the stored data to provide the required surface topography.

In either mode of operation the measurements can be made to take account not only of point brightness but of the brightness distribution in the surrounding area. Thus the brightness gradient may be measured in one or more directions, providing a smoothed value which tends to eliminate anomalous readings.

Information may also be introduced from more than two views of the area of interest. Provided that the projection geometry in the same coordinate system is known for each case the comparison operation can be performed successively for selected pairs. The combination of the resultant levels of probability should provide confirmaation or refinement of the resolution obtained with one stereo pair.

A further possibility, of advantage where the original observations cannot be closely controlied, lies in the application of the method when the projection geometry is not accurately known. On the assumption of approximate values for the projection parameters, the derived levels of probability will be generally low giving only a poor indication of a maximum. This maybe sufficient to adjust the parameters for a further attempt, the process being continued until satisfactory resolution is achieved. Alternatively the average value of probability level is calculated for each of a succession of trials with varying parameters until a maximum is found in the range of average values.

Two further situations arise from the original viewing conditions in which images may yield anomalous values of elevation. In the first case, a particular surface profile viewed from a particular angle may result in occlusion of part of the surface for one of the cameras. For such a surface element only one of the calculated conjugate points is a true image and the level of probability derived for the two points can only fortuitously lead to a correct value for elevation. Particularly where anomalies are suspected the situation can be investigated by comparing the surface elevation probability level at the suspect position with the levels for surrounding plan positions so as to derive a value of probability of occlusion.If, given the elevations of surrounding plan positions and the geometry of the stereo viewing system, the most probable case for a plan position is that its surface element is occluded, then no value of elevation is assigned to that plan position. The occurrence of a value of probability of occlusion which has been designated as significantly high can be signalled to an operator or an appropriate limiting value can be incorporated in the part of the program for computer 47 at which assignment normally occurs.

In the second case the surface is undifferentiated over an extended region so that the images contain large areas of approximately uniform brightness. All of the normal range of elevation elements at certain plan positions may then have conjugate points within the uniform areas so that all the derived probability values are closely similar. lathe absense of a maximum value it is not possible to determine elevation. The situation can be resolved by an iterative procedure, involving the computation of mutual probabilities in which an estimated probable value of elevation at each plan position is progressively weighted in dependence on surrounding values.In the absence of an abrupt step at the edge of the uniform region there is an assumed continuity of surface between the boundary region for which good values of elevation are established and the region of ambiguity. Close to the boundary estimated elevations consistent with the established values are heavily weighted whilst the probability level for an inconsistent estimate is reduced. The weighting process is applied progressively across the whole region and repeated as necessary until no further changes occur. The weighting values must be set to satisfy the particular characteristics of the type of surface to be investigated.

The general procedure, not hitherto known to be used for stereo image investigation, is referred to in the literature as "relaxation labelling." Particular applications are described in a paper by Rosenfled et al. (Scene Labelling by Relaxation Operations, IEEE Trans. SMC-6, No. 6, 420-433, 1976).

The programming of computer 47 for the data handling and control functions which have been described will be understood by those skilled in the art and will depend on the type of facility which is provided. It is considered that one of the range of commerciaily available minicomputers will be most generally suitable as providing reasonable speed of operation at moderate cost.

Alternatively the new type of computer system known as SIMD (single instruction, multiple data stream) and consisting of a large number of simple computers, operating simultaneously, is also suitable.

As an example of the application of the present inventive method of investigation, reference has been made to images produced in aerial mapping.

It should be noted however that the physical scale of the subject imposes no limitation on the use of the method. A production process such as microfabrication in which progressive changes in surface structure can be viewed stereoscopically by optical or eletron microscopy is an equally suitable subject. Monitoring or control of the process in response to such changes is made possible by the use of the method to a degree of precision dependent on the quality and resolution of the images produced.

Claims (16)

1. A method for the investigation of the threedimensional topography of a surface represented by a pair of stereo images compdses the operations of modelling the space containing the surface by designating an array of volume elements in a coordinate system such that a relative plan position and a relative elevation may be identified for each element, determining for each element in a sequence of elevations at a selected plan position the positions of the conjugate pair of points in the images, deriving from the images for each such pair of points a level of probability that the points of that pair correspond to a single physical feature of the surface, determining the pair of points for which the level of probability is a maximum, and assigning to a representation of the surface at the selected plan position the value of elevation of the element to which that pair of points is conjugate.

2. A method according to claim 1 in which the level of probability is derived from the values of a pre-determined brightness parameter for each conjugate pair of points.

3. A method according to claim 2 in which the parameter is a direct measure of brightness.

4. A method according to claim 2 in which the parameter is a function of the relative values of brightness of each conjugate point and of the surrounding points in the respective image.

5. A method according to claim 4 in which the function is a linear gradient.

6. A method according to any of claims 2 to 5 in which deriving the level of probability includes the operation of determining that value of a normalised probability distributed function which corresponds to the difference between the brightness parameter values for the points of a pair.

7. A method according to any preceding claim in which the value of the probability level first derived for a volume element at a designated plan position is modified for consistency with the values derived for elements which are adjacent in the x, y and z directions before determining the element for which the value is a maximum at that plan position.

8. A method according to claim 7 in which the first derived value of probability level is modified by arithmetical averaging with the values of such adjacent elements.

9. A method according to any of claims 2 to 6 in which, for an image having an extended region of substantially uniform brightness, the value of of the probability level first derived from such a region for a volume element at a designated plan position is weighted by the use of a relaxation procedure in dependence on the values of probability level derived for elements at adjacent plan positions, such that values of elevation assigned within the region are rendered consistent with those bounding the region.

1 0. A method according to any preceding claim in which, in order to exclude any occluded surface element, the values of probability level corresponding to the elements of each plan position are compared with values for surrounding plan positions, to derive by reference to the geometry of the stereo viewing system a level of probability that the surface element at such a plan position is occluded, the operation of assigning a value of elevation, being omitted on the occurrence of a pre-determined level of probability of occlusion.

11. A method according to any preceding claims in which, when the surface is represented by more than two images which may be related to a common coordinate system, the values of probability level derived for the volume elements at a designated plan position from a first pair of images are modified by comparison with values derived for that position from further pairs of images.

12. A method according to any preceding claim, for use where the projection geometry is only approximately known, in which a set of probability levels is derived, related to a set of plan positions, for each of a range of values of the projection parameters, the elevation to be assigned at each plan position being derived from the set of probability levels for which the average value is a maximum.

1 3. A method according to any preceding claim in which means is provided under the control of a suitably programmed computer for automatically deriving and storing image area related to selected volume elements.

14. A method according to claim 13 in which the means for deriving data from photographic images compirses cathode-ray tube means for illuminating the images by a digitally advanced scanning beam and photo-detection means for measuring the image intensity at each step, the output signal from the photo-detection means being directed to a store addressed in synchronism with the scanning rate.

1 5. A method according to claim 13 or claim 14 in which there is provided suitably programmed computing means for automatically deriving from the stored image data that value which determines the elevation to be assigned to a selected plan position.

16. A method according to claim 1 5 in which there is provided suitably programmed computing means for controlling the automatic production of a representation of the surface in response to the production of the assigned surface elevation value for each selected plan position.