METHOD AND APPARATUS FOR PERFORMING OPERATIONS ON THREE-DIMENSIONAL SURFACES . This invention relates to the production of three-dimensional reproductions of various objects, for example sculptures of human heads, reproductions of human teeth and the like, and machine parts, and to apparatus for putting the method into effect. The invention also relates to certain types of imaging apparatus, which is useful in conjunction with the said method and apparatus.
In recent years, machine tools have become increasing sophisticated, and many tools are presently available which do not require continual manual operation. For the continuous reproduction of machinable parts, modern machine tools, for example lathes, milling machines, and the like may be programmed using a digital signal representative of mutually orthogonal (ie. x,y,z) coordinates of points on a surface to be machined, and, once programmed, the machine will produce a three-dimensional representation of the surface.
I have realised that, by providing a machine, for example a machine tool with a control signal derived from a stereoscopic photograph or the like imaging means, it is possible to operate directly on a three-dimensional representation of an imaged surface,
OMPI
for example to use a machine tool to produce a three- dimensional reproduction of the surface, without the need for complex physical measuring operations, but simply by deriving a control signal for the machine
05 from an electronic signal representative of the relative spatial positions of points on the surface to be reproduced, using the stereoscopic image. As an alternative to producing a reproduction of the surface, the control signal may be applied to a
10 machine to control a robot arm or the like, for example to apply paint or the like material to a machine part.
Apparatus for processing stereoscopic images is widely used, for example, in the processing of aerial
15 stereoscopic photographs, such as those taken from satellites, in cartography. The technology of such machines is well known, and present day photogrammetry machines are capable of producing from a pair of stereoscopic photographs a digital signal
20. representative of x,y and z coordinates of points on the surface represented in the photographs.- These may be expressed as contour lines on a surface. The method of the invention seeks to utilise such signals directly to operate on, and in a preferred embodiment,
25 to produce therefrom a three-dimensional reproduction
of a surface, for example using a machine tool as outlined above.
According to a first aspect of the invention, there is accordingly provided a method of performing an operation on a three-dimensional surface, which method comprises deriving from at least one stereo¬ scopic image pair of the surface an electrical signal representative of the relative spatial position of points on the surface, processing the electrical signal so as to produce a control signal for example a control signal for a machine, and operating a machine, for example a machine tool using the control signal, so as to perform an operation on a three-dimensional surface, for example so as to produce a three-dimensional reproduction of the surface from which the image pair was derived, or to perform a welding or pointing operation on the surface from which the image pair was produced.
By the term "stereoscopic image pair" as used herein is meant a pair of images, for example photographic images, of a surface, which are preferably, but not essentially when the surface is inanimate, obtained simultaneously, and which are such as to enable information relating to the three dimensional form of the surface to be extracted therefrom, by comparison of the images.
OMPI
The invention also provides apparatus for producing a reproduction of a three-dimensional surface, which apparatus comprises means for processing at least one stereoscopic image pair of the surface to produce therefrom an electrical signal representative of the relative spatial position of points on the surface, means for processing the said signal so as to produce a control signal for a machine for example a machine tool, and a machine connected to the said processing means, to receive the control signal and to operate on a three-dimensional reproduction of the surface under the control of the control signal.
In a preferred embodiment, the electronic signal representative of relative spatial position of- points on the surface is a digital signal indicative of displacement in 3 mutually orthogonal directions, and is produced by scanning the said images.
In order to provide the desired degree of spatial information, the electrical signal is preferably. derived from at least two stereoscopic image pairs, which may be constituted by four separate images, for example produced by two stereoscopic camera pairs or may be constituted by only three images, of which one image is common to both image pairs. The method of the invention may be employed using a single
stereoscopic camera pair to produce a three- dimensional reproduction of the surface, for example a mask from a human face. However, in a preferred embodiment, at least- four and preferably seven pairs of stereoscopic imaging means are provided spaced around an object, thereby to produce stereoscopic images from all sides of the object, and thus to reproduce an entire three-dimensional object.
When two or more stereoscopic image pairs are employed, it is desirable that at least two common datum points should be included in each image, preferably externally of the surface in question, to facilitate registration of data from the respective image pairs. The datum points may be two points on a plumbline, which may preferably be damped to minimise oscillation.
In one embodiment the imaging means may comprise a plurality of stereoscopic camera pairs spaced around the object. In an alternative embodiment, the imaging means may take the form simply of an appropriate number of mirrors spaced around the object, orientated to produce an image on a single surface, for example a light-sensitive plate or film or a television camera. Using this method, it is possible to produce in a sim- pie and effective way simultaneous stereoscopic images of overlapping areas of the object, at a single
OMPI
camera. Thus, if the object is a human head, a stereoscopic camera pair may be provided directly in front of the head, and mirrors provided slightly to the rear on each side, and above the head, and arranged to provide an image of various back portions of the head on the plate of the front stereoscopic camera pair in a single flash exposure.
The images of the surface need not be optical images, and images formed by focussing, for example charged particles such as electrons may also be employed. Thus, the method of the invention may be utilised to form a 3-dimensional representation of, an article subjected to electron spectroscopy, for "example active sites in a chemical molecule. In one embodiment of the method of the invention, a two-stage process may be employed, in which one or more stereoscopic photographs is first made of an object. Such photographs may be either positive or negative photographs. The stereoscopic photographs may then be removed from the machine producing them and processed elsewhere by apparatus of the kind first set forth above, so that apparatus for producing the stereoscopic photograph may be in a place remote from that in which the reproduction is produced. According to a further aspect of the invention there is therefore provided apparatus for producing a
OMPI
stereoscopic image of an object comprising at least two pairs of stereoscopic imaging means, means for supporting the imaging means in spaced relation around an object, and means for obtaining from the imaging means at least two stereoscopic image pairs.
The means for supporting the imaging means may preferably include a rigid frame supporting the desired number of pairs of imaging means, for example still or TV cameras or mirrors. The frame may also include means for supporting the object to be reproduced in the desired position, and one or more datum features, for example plumb lines, to enable data derived from one stereoscopic pair of images to be brought into register with data produced by a second pair. The plumb lines may be damped to? minimise oscillation.
In a particularly preferred embodiment, the desired number of stereoscopic camera pairs, preferably at least four pairs, and in a particular embodiment seven pairs, are supported by the frame in the desired array, and are triggered simultaneously using an electronic flash arrangement. It is particularly important that at least two stereoscopic image pairs are used, when the object is living tissue, for example a human face or bust.
OMPI
The stereoscopic imaging means may take the form of a generally conventional photographic camera for producing a permanent record of the images on an imaging medium, for example photographic film, or in an alternative embodiment, may, as suggested above, include a television type camera producing an elec¬ tronic signal from a non-permanent image, by a raster scan technique. The camera may be of a colour or black-and-white type, and, if colour, additional information indicative of colour may be passed to the machine tool to cause the machine to produce corres¬ ponding coloured regions in the three-dimensional reproduction.
According to a further particularly preferred embodiment of the invention, there is provided^ a method of producing a reproduction of a three- demensional surface, which method comprises producing at least two stereoscopic pairs of photographic images of the surface, combining the images of each pair so as to produce for each pair a set of data for each of a plurality of points on the surface, indicative of displacement in three mutually orthogonal directions from a datum position, storing the data for each of the said points for each of the pairs in memory locations in a computer, identifying for each of the said pairs a set of data for each of the said pairs
MPI
- a - corresponding to at least two datum points to enable corresponding sets of data for each of the image pairs to be correlated, computing from the said sets of data a control signal to operate a numerically controlled machine, and operating a numerically controlled machine with the control signal to produce a reproduction of the surface.
There is no particular -limitation on the type of machine tool which may be utilised in the method of the invention, and in general most numerical control type machines will be suitable. It is however particularly preferred that the machine head itself is driven by an air rotor rather than conventional mechanical or hydraulic means, since this in general enables higher rotation speeds for example upj'to
250,000 r.p.m., and thus greater precision of detail, to be achieved in the three-dimensional reproduction. Spark erosion machines may also be favourably utilised. As an alternative to a machine tool the control signal may be used control a robot arm or the like, performing an operation for example painting a reproduction of the surface.
A preferred embodiment of the invention will now be described with reference to the accompanying drawings, in which:-
Figure 1 is a contour map of a human head in side view;
Figure 2 is a similar contour map from above, Figure 3 is a similar contour map in front view, and
Figure 4 shows a preferred camera arrangement, Figure 5 shows a side view of an arrangement for locating an object and
Figure 6 is a plan view of the arrangement of Figure 5.
In the arrangement illustrated in Figure 4, three stereoscopic camera pairs are equispaced (i.e. are spaced at approximately 120* along a circle centred on the head to be photographed) and are positioned at approximately 10* below the horizontal
of the chin of the subject, so as to deal with undercut portions. The fourth camera pair is vertically above the head. For more complex shapes, a further three camera pairs above the horizontal level may be provided. In general in order to provide the required amount of data on each point of a surface, a sufficient number of image viewpoints is required so that each point on the surface is viewed by at least one stereoscopic image pair. Figures 1 to 3' show the kind of contour maps which may be obtained using conventional stereoscopic analysing machines to analyse such stereoscopic
OMPI
photographs of a human face. A rigid frame (not shown in Figure 4) is set up carrying the desired number of stereoscopic camera pairs. All of the cameras are then simultaneously exposed using a flash technique, and each pair of photograph processed separately and thereafter analysed. The contour lines of Figures 1 to 3 illustrate clearly the kinds of detail which can be obtained using even very coarse analysis techniques. The contours in Figures 1 to 3 are marked in m.m. from an arbitrary datum line, and it can be seen that two internal datum points 2 and 4 are noted on each of the Figures. To obtain such contour maps, information derived from the pairs of stereo photographs is first converted to digital coordinate (xyz) form. A number of commercially available machines are able to carry out the processing of a stereoscopic image pair to produce data in digiital coordinate form. Examples are the PLANICOMP C100 produced by Zeiss, which normally operates on a pair - of stereophotographs, each photograph of the pair having a size of up to 23 cm x 23 cm. The photographs are scanned successively in the "y" direction as the "x" coordinate is incremented in steps, and thus a "z* coordinate is obtained for each of a plurality of points along each "y" scan line. Spatial resolution of the surface in question may be improved for a particular pair of photographs by decreasing the distance between the parts at which readings are taken. To obtain high resolution data for large surfaces requires the storage of quite significant amounts of data, some of which may be
superfluous because it relates to parts of the surface under study which are relatively flat. Thus, in a preferred-embodiment, separate scans at higher resolution may be performed for parts of the surface which are very detailed or have a high radius of curvature, and the data obtained from such scans may be matched to corresponding data obtained from a lower resolution scan to provide higher resolution of a particular area. - There are three main ways in which the spatial resolution of the data points obtained may be increased, when using an image analysing machine of the PLANICOMP type. The first method is, as indicated above, to decrease the inter-point distance at which data is extracted from the stereo photograph pjai'r. A second method of increasing resolution for a particular pair of photographic negatives is to produce increasingly enlarged prints or other forms of image from the negative pair, and to perform the scanning operation on a thus enlarged image. The increase in resolution which can be obtained by the second method is limited by the quality of the original photographic negative. Thirdly, an additional stereoscopic negative pair at closer range or using a lens of higher power may be used to produce a more detailed negative of the area in question. In order to obtain the correct registration
between image areas scanned initially at different resolution or between adjacent overlapping areas at the same resolution, a number of techniques may be employed. In the preferred method one or more accurate datum indicator, for example optical grids or rulers, are included within the image. The datum indicator may preferably take the form of a glass plate having a grid marked thereon, to enable accurate location of the subject in three dimensions relative to the datum indicators for various camera positions.
A sufficient number of such datum indicators are -preferably provided to enable location in the x,y and z directions, and the intervals between datum markings on the datum indicators may be chosen -in accordance with the degree of accuracy of measurement desired for the particular application. For example in the reproduction of a machine part, interval markings as close as .001 mm could be selected. Such accuracy would not be required, for example, in the reproduction of a human head, where spacings between datum markings of as large as tens of centi¬ meters may be used, provided that their relative position is known to a sufficiently high degree of accuracy.
Provision is preferably also made for causing a second grid to be marked directly on the image formed,
OMPI
for example on the photographic negative. This may be done by sandwiching a photographic plate or film between two optically flat glass plates for example RESEAU plates, of which one has a grid marked thereon, such that both this grid, and the grid resulting from the image of the datum indicator, are reproduced on the negative. The two grids may then be used to relate accurately relative dimensions on the image to absolute dimensions of the object. In an alternative embodiment, a computer may be utilised, programmed so as to examine points of the area scanned at high resolution in such a manner as to disregard some of them, such that the three- dimensional mapping initially obtained corresponds to that obtained from a low-resolution scan. Th t-a low resolution scan is then examined for an area which corresponds to the area of the high resolution scan. When the correct registration has been found, the additional data points of the low resolution scan are made available.
The coordinate data obtained from the image processing machine may be stored electrically or magnetically for example in a computer RAM (Random Access) type memory, or on magnetic tape or disk. Once the data is stored, operations may be performed on it to present it in any desired form.
OMPI
For example the data obtained from a coarse scan may be numerically combined with the data from a more ~~ detailed scan as outlined above. By a similar technique data obtained from overlapping stereophoto- graphic pairs may be compared in the region of overlap, to bring the data from such pairs into register. Thus, data obtained from camera pairs placed at different angles may be combined to produce a full reproduction of a solid object, or a machine part. The same technique may be used in the processing of multiple images of the same object on a single photographic plate, produced for example using a plurality of mirrors, and a simple flash exposure. Views such as this may be combined in pairs so as to produce the desired coordinate data from a single photograph exposure. In a preferred embodiment however, two such multiple exposures may be produced, each corresponding view being from a slightly different angle and corresponding views may be scanned to generate the coordinate data.
Various operations in the nature of matrix transformation may be carried, out on the data obtained, to facilitate its use in a particular operation. For example, using the scanning method described the data is generated in the form of a series of pairs of "yz" coordinate for each of a
series of planes in the "x" direction. This may be considered as a series of successive two-dimensional contour maps in the "yz" plane for each of a plurality of different values of "x". This format is very desirable where it is desired that a tool following or reproducing the surface should similarly be arranged to cut in the "z" direction as "y" is caused to vary, for each of a plurality of planes spaced along the "x" axis. Where it is desired that a different machine direction should be utilised, a matrix transformation may be carried out on the original data, to produce the data in the desired form, for example as a series of pairs of "xy" coordinates for each of a plurality of valves of "z".
Conventional curve-smoothing algorithms may be used to interpolate data between the datum points measured. Conventional averaging technique may be utilised to improve the accuracy of the data obtained. For example an initial averaging may be carried out to identify extreme and therefore unlikely readings, and then those extreme reading may be discarded and a further averaging technique performed.
The sets of coordinates may then be fed to a electronically controlled machine tool to produce a three-dimensional copy of the original article, for
example the head, without the need for operator intervention. The apparatus for producing the three- dimensional reproduction may be remote from the means for processing the two images, which may itself be remote from the apparatus for producing the images. Thus, the frame and cameras may be provided at one location, may send appropriate stereo photo pairs to an image processing station for processing, and the data obtained from the image processing may be converted into a suitable control signal for a machine tool either at the time of generating the electrical image signal, by suitable computing means, or else may be stored for example on magnetic tape-or disc, or transmitted along landlines to a machine tool, which generates the necessary eontrol signal from the image signal by means of appropriate integral circuitry. Alternatively one or more pairs of video cameras may be utilised and the signal obtained from the video camera may be passed to a remote location before processing.
Means are preferably provided in the machine tool for examining data corresponding to the area immediately surrounding the area on which the tool is operating at any particular time, and for calculating the optimum orientation of the tool head to ensure the
OMPI
cutting operation does not interfere with the desired final surface form of preceding or following cuts.
The ratio of the distance between cameras of a stereoscopic pair and the distance from the image playing to the object is preferably of the order of 1:5, since this provides maximum three-dimensional resolution.
The cameras utilised may be for example Rolleiflex SLX RESEAU cameras, provided with RESEAU plate film supports of conventional form. These consist of glass plates provided with regular grid markings which be on both sides of the film to enhance the film flatness, and facilitate registration between exposures taken at different times. When the mirror system described above is/ utilised to provide multiple images on a single plate it is desirable that any aberrations in the mirror surfaces should be measured accurately before the images to be analysed are produced. Mathematical compensation may then be made for such aberrations. When mirrors are utilised in this way, the optical path length will be different for each image formed. Care must the'refore be taken to ensure that the depth of field of the optical system is sufficient to enable focussing of the various images on the single plate, and allowance must be made during
comparison of the stereoscopic image pair obtained for the different apparent distances from the focal plane of the subject for the different images.
As indicated above, in order to obtain the maximum resolution and in particular in order to obtain registration between stereoscopic images from different angles, it is important that a datum indicator, preferably an accurate scale of some kind, is provided adjacent the object of which the surface is to be reproduced or otherwise operated upon. This "external" datum can then be accurately related to a scale provided in the plane of the image (for example the photographic film or plate). The external datum may be, for example, a photographic studio, provided at suitable points with accurately positioned linear scales or grids, preferably of a material with a very low temperature expansion coefficient. The positioning of such grids or plate with respect to each other may be measured, for example optical means such as by the use of a laser beam, and the position¬ ing should be checked at regular intervals, for example intervals of one week or less. Conventional survey apparatus may be used for the measuring opera¬ tion. The position of points on the object itself with respect to the surroundings may be determined by
employing method, which is illustrated by reference to Figures 5 and 6.
Figures 5 and 6 illustrate an arrangement for accurately locating and photographing an object. The arrangement of Figures 5 and 6 comprises a cell, comprising an immovable base 54, having mounted thereon and respectively secured to each other glass plates, 51, 52,53, and 54. Each of the glass plates 51,52,53, and 54 has marked -thereon highly accurate vertical and horizontal grid markings 55 and 56 respectively. The object to be photographed 57 rests on the base 50, and two laser or other light rays 58 and 59 are caused to pass through glass plate 51. The beams 58 and 59 impinge on the object 57 at point 60 and 61 respectively. Photographic record is ^aken as the points at which the beams 58 and 59 exit from the glass plate 51, and of the points 60 and 61. The object 57 is then removed, and the points 62 and 63 noted, again by taking a photographic record, at which the beams 58 and 59 impinge upon plate 52, in the absence of the object 57. It is then a relatively simple matter to determine the absolute position of the object 57, and in particular the position of the points 60 and 61, with respect to the frame constituted by the plates 51,52,53 and 54. Because measurements of the point of impact of the light beams on the inside surfaces of the plates 51 and 52 are
PI
taken, no account need be taken of refraction of the light beam in the glass plates. A microscope may be utilised to measure the points at which the beams 58 and 59 impinge on the respective glass plates, by reference to the grid markings 55 and 56.
It will be appreciated that the accuracy to which datum points within the image must be located will depend on the particular task which it is desired to carry out. Thus, for very accurate measurements, and in particular where fine detail is required in a particular area using the image combination technique, a laser-measured datum system as described above may be desirable or essential. When the resolution required is less demanding, non-laser light sources may be utilised. When the required resolution^ is less
I demanding still, mechanical measuring methods may be used to locate the datum points, for example micrometers, dial gauges or the like. Computer-aided design drawings, and computer data derived therefrom, may also be utilised as a supplement to whatever physical datum measurement is provided.
The images used in the method of process of this invention may be, for example, images produced on a photographic plate by the action of X-rays, and such images may be used to perform operations on internal surfaces of engineering parts. In general it will be
OMPI
necessary also to provide optical images, against which such X-ray photographs may calibrated.
I have also appreciated that, by utilising the technique of stereoscopic image scanning outlined above, and in particular by utilising the techniques of determinin the registration between pairs of stereoscopic images, it is possible to produce an engineering drawing directly from one and preferably two or more stereoscopic image pairs of an engineering item or the like, for example a machine part.
Furthermore, since the co-ordinate data can represent information about the whole of the viewable surfaces of an .object, the same data may be utilised to provide a sectional drawing at any desired angle or displacement, merely by determining, for example using a computer to intersection the desired section plane, and the information stored which represents the object sur ce.
In an alternative aspect of the invention, there is therefore provided a method of producing a drawing of an object, which method comprises deriving from at least two images of a three-dimensional image from different view points an electrical signal representative of the relative spatial position of points on the surface, processing the electrical signal so as to produce a control signal for a plotting machine, and operating the plotting machine
using the control signal so as to produce a drawing of at least a part of the object.
Any or all of the techniques of resolution enhancement and image processing may similarly be applied to the method of producing drawings according to the invention.
The method of the invention has very wide ranging fields of application, from the production of sculpted heads, as outlined above, to the automatic copying of machine parts from a model or original machine parts. Such parts may be produced in any size by simple scaling of the data in a particular set, and thus the method may be used to reproduce either enlarged or reduced size models of- an original article, or scale drawings of an original article. Similarly a Machine tool can be programmed to produce a negative reproduction of the surface of an article and thus the method is particularly suited to producing moulds and dies, for example for the plastics or metal working industries, directly from full size or scale models of the articles which the moulds or dies are intended to reproduce.
A particular application which is envisaged is the production of false teeth from plaster models or the like.
Clearly a very wide range of alternative detailed methods of operation are intended to be comprehended within the scope of the appended claims.