Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
  • G01B11/00—Measuring arrangements characterised by the use of optical techniques
  • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
  • G01B11/00—Measuring arrangements characterised by the use of optical techniques
  • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
  • G01B11/245—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures using a plurality of fixed, simultaneously operating transducers
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
  • G01B11/00—Measuring arrangements characterised by the use of optical techniques
  • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
  • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
  • G01B11/2545—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with one projection direction and several detection directions, e.g. stereo
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/50—Depth or shape recovery
  • G06T7/55—Depth or shape recovery from multiple images
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/97—Determining parameters from multiple pictures
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/10—Image acquisition
  • G06V10/12—Details of acquisition arrangements; Constructional details thereof
  • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
  • G06V10/147—Details of sensors, e.g. sensor lenses
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
  • G06V2201/12—Acquisition of 3D measurements of objects

Definitions

• the present mvent.on relates to a scanning apparatus and method for acquirin o the 5 three- ⁇ .mens i onal shape , size or other three-dimensional surface features of an object such as colour for example.
• the invention relates particularly but not exclusively to hand-held or other freely movable 3D scanners.
• Overlappino l i nes obta i ned in a similar manner following rotation of the scanned object are comb i ned by correlating their overlapping regions to derive a closed loop which def i nes one cross-section of the object's surface and a multiplicity of such cross- sect i onal loops are derived and combined to form a wire frame description of the 0 object s surface.
• the ob j ect is a large forging which is rotated and the cameras are fixed and spaced one metre apart. There is no reference to a hand-held scanner.
• each camera comprises a focuss i ng lens L of focal length a and a photosensitive imaging plane 1 and the optical 99/06950
• each camera 2 axis of each camera is spaced apa ⁇ from the Z axis b ⁇ a distance S.
• An arbitrary point (X,Y.Z) on the object in the field of view of both cameras is imaged onto the image plane 1 of each camera as illustrated by rays 10, 10" and 20, 20 " .
• Camera 1 images the point (X,Y,Z ) at a point (xj t -y) in local coordinates of the camera system and camera 2 images the point at a point (-x2--y) in the local coordinates.
• the coordinates X, Y and Z can be determined from x ] , X2 and y.
• a problem which arises in the case of objects of complex shapes is that it is difficult to determine whether a particular point on one camera's image plane corresponds to a particular point on the other camera ' s image plane (ie whether the points are both conjugate points of the same point on the object's surface). For example, overhanging - 5 regions of an object might obscure an underlying region so that it only appears in one camera ' s filed of view. This problem is particularly likely to arise when the spacing of the cameras is relatively large compared to their distance from the object. However if the spacing is reduced relative to the distance from the object, the geometrical accuracy is compromised.
• WO91/15732 discloses an arrangement in which a laser scanner projects a 3 series of stripes onto the scanned object and left and right cameras to detect the distorted stripes from the object. It is recognised that a given bright point in the image plane of one camera cannot be simply correlated with an illuminated point on the surface of the object because it is not known which stripe illuminates that point.
• an arbitrary pixel in the stripe in one camera's image plane is selected and a line drawn through the centre of the camera lens projecting this line out into 35 space.
• This line is then projected onto the image plane of the other camera and the resulting epi-polar line in the other camera's image plane cuts a number of stripes also imaged on its image plane. Any one of these points of intersection could in principle correspond to the arbitrary pixel mentioned above. The particular point which corresponds is found by projecting all the points of intersection back into space 99/06950
• the above arrangement has the disadvantage that a projected pattern is required and that some uncertainty in the correlation of points may arise if any of the last- mentioned projected points cut or nearly cut more than one laser stripe. '
• One object of the present invention is to provide scanner arrangements which do not require projected patterns.
• Another object of certain embodiments is to provide scanner arrangements in which movement of the scanner relative to the scanned object is determined by processing ] ⁇ the image of the object and does not require hardware such as inertial sensors (although the output of inertial sensors can be used to supplement such processing).
• the invention provides an arrangement as claimed in claim 1 for acquiring the 3D shape of an object.
• the invention provides an arrangement as claimed in claim 8 for acquiring the 3D shape of an object.
• the invention provides a method of processing overlapping images as claimed in claim 1 1.
• the invention provides an image processing arrangement as claimed in claim 17.
• the invention provides an arrangement for acquiring the 3D shape of an object as claimed in claim 24.
• the '"viewpoints" of the cameras can be characterised by differences in position and/or in orientation of the camera relative to the object.
• Figure 1 is a ra ⁇ diagram show ing the camera arrangement of one embodiment in plan view
• Figure 2 is a ray diagram showing the camera arrangement of Figure 1 in elevation
• Figure 3 is a sketch perspective ray diagram of the embodiment of Figures 1 and 2 showing the generation of epi-polar lines of the corners of one image region ABCD and their intersection at a common point adjacent the corresponding image region in the other camera ' s image plane.
• Figure 4 is an end elevation of the above embodiment looking through the image planes towards the object region and show ing the geometrical construction of the above epi-polar lines.
• Figure 5 is a ray diagram in plan view showing the above embodiment and illustrating the uncertainty the object position w ithout correlation of image points in the left and right cameras' image planes.
• Figure 6 is a front ele ⁇ ation of the above ray diagram show ing how epi polar lines can be used to assist in the correlation of image points
• Figure 7 is a flow diagram showing the process of correlating the image points in the above embodiment
• Figure 8 is a sketch perspective view of a variant of the above embodiment utilising inertial sensors
• Figure 9 is a sketch perspective view of another embodiment utilising onlv a single camera
• Figure 10 is a sketch perspective view show ing a rav diagram of the embodiment of Figure 9 being used to track its position relat ⁇ e to an object
• Figure 1 1 is an elevation of a ray diagram showing the correlation of image points between the images of the respective cameras in the first embodiment or of the camera m different known positions in the second embodiment
• Figure 12 is a ray diagram which is a section taken on XII-XII of Figure 1 1 and illustrates the derivation of the object position relativ e to the camera from the correlated images,
• Figure 13 is a flow diagram showing the process of correlating the image points in the embodiment of Fisures 9 and 10
• Figure 14 is a flow diagram showing a process for obtainin g a complete 3d surface description using the embodiment of Figures 10 and 1 1 and image correlation software:
• Figure 15A is an illustration of a projected fractal pattern for use in an embodiment of the invention:
• Figure 15B is an illustration of the distortion of the fractal pattern by the inclination of the camera relative to the surface
• Figure 16 is a sketch perspective view of a further embodiment of the invention.
• Figure 17 is a diagrammatic transverse cross-section of an endoscope head in accordance with another aspect of the invention:
• Figure 18 is a diagrammatic side view of the endoscope head of Figure 17, and
• Figure 19 is a diagrammatic representation of a further endoscope in accordance with the present invention.
• Figures 1 and 2 have already been referred to and are applicable to the optics of the first embodiment.
• a given point x ] , y in one camera's image plane 1 can be correlated with an image point x 2
• y in the other camera's image plane ie both points are conjugate points of a common point X, Y, Z on the object ' s surface
• this correlation cannot be performed without some processing of the two images in the respective image planes of the two cameras 1 and 2.
• Figure 3 illustrates a preliminary step in searching for points A2, B2. C2 and D2 in the image plane I of camera 2 which correlate with given points A l , B l . C l and DI in the image plane of camera 1.
• Points A, B, C and D on the surface of the object define a surface region S and are imaged by both cameras as the above sets of points.
• Undeviated ray lines a, b, c and d connect A to A 1 , B to B 1 , C to C 1 and D to D 1 and when imaged onto image plane I of camera 2 define four epi-polar lines EP a to EPd respectively which meet at a point X2'. This is the conjugate point of the centre XI of lens L of camera 1.
• the position of these epi-polar lines is independent of the position and orientation of the object and the vertices of the image region A2, B2. C2, D2 of camera 2 must lie on them.
• Figure 4 illustrates the construction of the epi-polar lines.
• a construction line CONST is constructed so as to join the optical centres X I and X2 of the lenses L. It can then be seen that line a projects onto epi-polar line EP a w hose points A2 and X2 " define with point X2 a triangle which is geometrically similar to triangle A. X I . X2.
• the other epi-polar lines can be constructed similarly and it will be noted that they intersect at a point X2 * which is a distance 2S (the spacing between X I and X2) from
• the true position of the object can be found by an algorithm which selects various rectangular image regions whose vertices lie on the respective epi-polar lines EP and compares them with image region 3, 3, 4, 4.
• the image region of camera 2 lying on these epi-polar lines which gives the best match is taken to be that w hich truly correlates with the image region 3, 3, 4, 4 of camera 1 and thereby defines the position of the object with reference to the camera arrangement and also enables the 3D coordinates of each point on image region S to be determined by the geometrical procedure of Figures 1 and 2.
• Gruen's algorithm is an adaptive least squares correlation algorithm in which two image patches of typically 15 x 15 to 30 x 30 pixels are correlated (ie selected from larger left and right images in such a manner as to give the most consistent match between patches) by allowing an affine geometric distortion between coordinates in the images (ie stretching or compression in which originally adjacent points remain adjacent in the transformation) and allowing an additive radiometric distortion between the grey levels of the pixels in the image patches, generating an over-constrained set of linear equations representing the discrepancies between the correlated pixels and finding a least squares solution which minimises the discrepancies.
• the Gruen algorithm is essentially an iterative algorithm and requires a reasonable approximation for the correlation to be fed in before it will converge to the correct solution.
• the Otto and Chau region-growing algorithm begins with an approximate match between a point in one image and a point in the other, utilises Gruen ' s algorithm to produce a more accurate match and to generate the geometric and radiometric distortion parameters, and uses the distortion parameters to predict approximate matches for points in the region of the neighbourhood of the initial matching point.
• the neighbouring points are selected by choosing the four adjacent points on a grid having a grid spacing of eg 5 or 10 pixels in order to avoid running Gruen ' s algorithm for every pixel.
• the first pair of points used to generate the initial approximate match can be found eg by searching for patterns or features in each image which match approximately and choosing pairs of clearly defined points within the pairs of matching patterns or features. This can be done by appropriate software or firmware w ithout the need for human intervention.
• the algorithm can be made to converge much more quickly and with less uncertainty. This is an important advantage.
• a pattern eg 3, 3, 4, 4 ( Figure 6) is selected in one camera ' s image plane and its vertices located and identified.
• the shape of the pattern boundary is preferably but not necessarily a rectangle, a parallelogram, an equilateral triangle, a regular hexagon or some other figure hich can be repeated to cov er substantial] ) the entire area of the photodetector array
• step 1 10 the epi polar lines of these vertices (eg EP in Figure 6) are pro j ected onto the other camera ' s image plane These epi-polar lines converge to a point (which is not necessa ⁇ ly within the photosensitiv e detector of the other camera but whose coordinates in the other camera ' s image plane can be found by extrapolation if necessary) and define a range of pattern bounda ⁇ es having a similar shape to the selected pattern boundary of step 100
• step 120 sets of possible corresponding pattern vertices in the other camera's image plane are determined by selecting a pattern shape corresponding to the shape selected in step 100 (eg rectangular in the embodiment described above) and fitting its vertices to the above epi-polar lines A range of eg rectangular patterns of different elongation and size results
• each of these patterns is compared with the pattern selected in step 100 and the pattern with the closest match (in eg image densitv distribution, colour distribution, contrast distribution (ie distribution of rate of change of image density) or any w eighted average of the above parameters) is selected ( using eg Gruen s algonthm or a similar algonthm) as the pattern which best correlates with the pattern of step 100.
• the individual pixels within the matching patterns are then correlated ith each other and the 3D coordinates of these pixels relative to the scanner can then be determined, as will be shown below ith reference to Figures 12 and 13
• step 140 the abo e steps 100 to 130 are repeated for all other patterns in the first camera ' s image plane and hence all the pixels in each camera ' s photodetector array are correlated
• step 120 none of the patterns selected in step 120 will correspond to the pattern selected in step 100, eg as a result of overhang of a region of the object which prevents a region of the object surface from being viewed by both cameras In such a case the processor will report that no correlation is possible and will select a new pattern (step 100)
• the size of the pattern selected in step 100 is not critical but should preferably be smaller than most surface features of the object in order to minimise the possibility of some pixels but not others in the selected pattern correlating w ith a pattern in the other camera ' s field of view, possibly leading to incorrect matching of the tw o patterns
• the size could be selected by the user
• Hand-held scanner 13 is provided w ith an inertial sensor arrangement comprising a 3-ax ⁇ s vibratory gyroscope arrangement 1 1 (which detects rate of rotation about mutually perpendicular axes ⁇ x , ⁇ v and ⁇ z ) and an accelerometer 12 (which detects acceleration along the corresponding axes X, Y and Z).
• a 3-ax ⁇ s vibratory gyroscope arrangement 1 1 which detects rate of rotation about mutually perpendicular axes ⁇ x , ⁇ v and ⁇ z
• an accelerometer 12 which detects acceleration along the corresponding axes X, Y and Z.
• the output signals from these sensors are processed by a microprocessor arrangement ⁇ P which is similar to that shown in Figure 2 of our granted patent GB 2,292,605B (whose entire disclosure is hereb y incorporated by reference) and the resulting position and orientation information is stored in a memory (such as a miniature hard disc M) which is optionall y removable)
• the position and onentation information can also be output from the processor to a computer PC by a wire, radio or optical link via a bidirectional output port
• Computer PC is provided witTi a conventional RAM, hard disk and microprocessor and is arranged to perform the processing already referred to in connection w ith Figures 3 to 7 as well as the processing desc ⁇ bed subsequently in connection with Figures 10 to 15.
• the arrangement is powered from a rechargeable battery B and the acquisition of data from the cameras, accelerometer and gyroscope arrangement is controlled by a hand-operated tngger button TR
• the position and onentation data from the microprocessor can be used to guide the software which carnes out the processing of Figure 7, particularlv in the search for a matching pattern in step 130.
• the position and onentation of the object O with reference to the scanner 13 is defined.
• This position and onentation is likelv to be more accurate than that obtained from the gyroscopes and accelerometers because the latter are subject to drift.
• Scanner 13 also carnes a light source LS (eg a stroboscopic light) for illuminating the object O and optionally a supplementary laser arrangement LA which can be used to derive additional depth information or other surface coordinate information by 50
• a light source LS eg a stroboscopic light
• LA supplementary laser arrangement
• laser arrangement LA is a triangulation arrangement
• the optical axes of the cameras are arranged to cross at the centre of such a projected pattern, at a distance corresponding to the centre of the depth of field of triangulation. and the cameras are used to acquire a true monochrome or colour representation of the object either simultaneously with triangulation or alternateh .
• This can be superimposed on the 3D image acquired by triangulation to allow the profile obtained by triangulation to be rendered using the image data from the cameras. In this manner surface features other than profile , eg surface printing and colours can be captured.
• the light source LS can be used to provide consistent incident light for the object O in the majoritv of ambient lighting conditions. Jt can be synchronised to image capture by the cameras in order to prevent interference with the triangulation process.
• the position and attitude data obtained from the gyroscopes and accelerometers can be tagged to the image data from the cameras and triangulation arrangement to enable the data from the cameras and triangulation a ⁇ angement to be processed either in real time or off line, allowing areas which are poorly described by the triangulation signals to be to be corrected by the image data from the cameras, or vice versa.
• surface features detected by the cameras ie any distinct pattern of pixels
• groups of such surface features can be tracked as the scanner 13 is moved relative to the object O and the trend of movement and/or distortion of such features and/or the trend in the movement or spacing of such surface features can be used to predict the next position and orientation of the scanner and hence to guide the software in correlating the images from the two cameras.
• This information can also be used to correct acceleration and rate of rotation data from the accelerometers and gyroscopes.
• the post processing methods disclosed in our PCT/GB95/01994 can be used to combine separately acquired surface regions obtained from the outputs of either a triangulation arrangement or a Moire anangement or the cameras 1 and 2. Such methods can also be used to provide position and orientation data for use in any of the processes described above requiring such methods.
• a common processor is used a) to process stereoscopic data (eg by the process of Figure 7 of the present application) from the two cameras 1 and 2 b) to predict the position and/or orientation of the scanner relative to the object by tracking groups of features (which could be any distinct groups of pixels) between frames and c) to process triangulation data (eg as obtained by the optical arrangement of Figure 1 or Figure 3 of our PCT/GB95/01994).
• These processes, designated S, F and T respectively can be carried out sequentially in ratios varying with the frame rate and the processing power of the processor. For example at 60 frames/second the sequence could be:
• Figure 9 shows a further embodiment 13' which utilises only a single camera and a microprocessor ⁇ P to track an object O and to determine its shape.
• This embodiment utilises stereoscopic image processing analogous to that outlined in Figures 1 and 2 and utilised by the embodiment of Figures 1 to 8, but stereoscopically combines the images acquired at different positions and orientations of the scanner 13' relative to the object, using information on the position and orientation of the scanner acquired by tracking the image in its image plane I.
• This processing is carried out by a computer PC which receives the output data from the scanner.
• the object O moves relative to the scanner from position O to position O ' and the face ABC is projected onto image plane 1 first as image abc (position O) and then as image a ' b ' c ' (position O ' ).
• Bb' and Cc' can be constructed merely from the position of the image a ' b'c ' and a knowledge of the camera geometry.
• the triangle ABC of defined size and shape can be fitted to these ray lines in only one position and orientation beyond the optical centre of the lens L. Hence the position and orientation of the object can be tracked.
• the above analysis assumes that the size and orientation of the face ABC of the object is initially known. However this is not essential in order to track the movement of the scanner relative to the object. Thus the ray lines Aa, Bb and Cc are consistent with a smaller object having a face A'B'C (as shown) located nearer the camera. However (assuming its initial orientation is as shown) the difference in detected orientation of the object between positions O' and O will be the same and the distance of movement of the scanner will be scaled by the assumed object size. The shape of the object will not be affected.
• the initial position O of the object is not the only position consistent with ray lines Aa, Bb and Cc.
• these ray lines would also be consistent with a face A ' BC.
• the orientation and position of hypothetical face A 'BC in relation to (equally hypothetical ) face ABC would also remain unchanged and the relative movement of the object between positions O and O ' as determined by tracking movement of the image abc/a ' b ' c ' would not be affected.
• the points abc are derived from corners of the object.
• any set of three or more clearly defined points or groups of points on the object O can be tracked to track its movement and rotation relative to the scanner 13 ' .
• the scanner would normally move and the object would be stationary. Since the tracked points will gradually move off the edge of the photosensitive array of the camera, it is desirable to track at least four points in order to track the movement of the object relative to the scanner over distances which are large in comparison with the size of the photodetector array.
• the movement M shown in Figure 12 is a rectilinear movement in the direction of the image plane of the camera, and illustrates the similarity in the analysis of a stationary stereoscopic camera arrangement and a moving monocular camera arrangement. In principle however the position and orientation of the surface could be determined just as easily from a known, non-rectilinear movement of the scanner.
• step 200 The overall process of determining the shape of the object is illustrated in the flow diagram of Figure 13.
• step 200 four or more points in the image plane are selected. Alternatively, in some cases it may not be necessary to select more than three points.
• step 210 adjacent ones of the four or more points are connected to form a network of at least two triangles. (Alternatively it may be possible to rely on only three such points to form one triangle in certain cases).
• One such triangle abc has already been referred to in connection with Figure 10.
• the second triangle enables the network to be tracked as one point moves off the edge of the photosensitive array, but this will not be necessary on certain cses
• the points are tracked (step 220).
• the undeviated ray lines are then projected from the tracked points (step 230).
• step 240 the network of four or more points ( in some cses three or more points) is then fitted to the projected ray lines to define the new position of the object (eg O' in Figure 10).
• step 250 the point(s) which have moved out of the field of view are discarded and a new network is constructed (ie steps 200 and 210 are repeated). This step will not be necessary in certain cases. At the same time, the calculated position and orientation data are output.
• step 260 successive views are compared (cf Figures 1 and 2 and Figure 12) and, using the position and orientation output of step 250, used to obtain the 3D coordinates of the surface portion (eg S in Figure 12) of the object which is common to the two views.
• Step 260 (which is illustrated in Figures 1 1 and 12) is elaborated in the flow diagram of Figure 14. This flow diagram is equally applicable to the stereoscopic and monocular camera anangements.
• step 300 a pattern (eg the left hand pattern P in Figure 14) is selected.
• a corresponding pattern (eg the right hand pattern P in Figure 11) is searched for, using variable scaling factors eg to compress/expand the image in the X and Y directions and possibly also to apply a linear correction to the image density in order to take into account oblique illumination of the object.
• variable scaling factors eg to compress/expand the image in the X and Y directions and possibly also to apply a linear correction to the image density in order to take into account oblique illumination of the object.
• step 320 the above step is repeated for other patterns and the points in the corresponding pa t terns are correlated (cf the mapping illustrated bv lines 25 in F. o ure 1 1).
• step 330 the correlated points are used to construct the 3D surface region of the ob j ect region common to both cameras ' fields of view ( cf Figure ⁇ usin c the position and/or orientation data of step 250 of Figure 13. Additionall y gy ⁇ Tdata and/or accelerat i on data and/or geometncal data defining the geometry of the stereoscop i c camera arrangement of Figures 1 to 8 (if used ) can be utilised in this step.
• step 340 the process is repeated for other surface regions and the process then continues with step 270 ( Figure 13).
• a fractal pattern in which the small-scale structure and the large-scale structure share a common element ) is provided (eg by optical project i on ) on the scanned object and the pattern is viewed from different angles to derive d i fferent images which are correlated to derive the 3-D surface coordinates of the region of the object on which the pattern is formed is illustrated by way of example only in Figures 15 and 15A.
• a fractal pattern 500 ( obtained bv forming a cross and then supe ⁇ mposing a cross of half the size on each tip, and repeating this process in respect of each prev ious cross) is shown in Figure 1 5 after three iterations of the above process ( in practice more iterations may be used to increase the detail and the density of coverage) and a pattern region P is selected.
• the pattern is projected orthogonally onto a fiat region of the object so that the image shown in F i gure 15 is an undistorted representation of the original pattern.
• FIG. 16 A completely different arrangement for determining depth information, at least approximately, is disclosed in Figure 16.
• An array of point illumination sources LS projects light spots R onto the surface S of the object and a lens L images the illumination pattern formed by the spots on a photosensitive image plane 700.
• the size and/or shape of each imaged spot is measured by appropriate image-processing software.
• the size of a spot R is directly proportional to the distance of the corresponding illuminated surface region from the conesponding light source LS and the shape (in this case the deviation from circularity) gives information about the local inclination and curvature of the surface.
• the shape in this case the deviation from circularity
• the light sources could suitably be optical fibres and the arrangement is suitable for miniaturisation in eg an endoscope.
• the light sources need not be point sources but could for example be line sources, whereby the width of each illumination stripe on the surface S is related to the distance from the light source.
• dark region(s) could be projected onto the surface S using appropriate mask(s).
• an endoscope arranged to provide a 3-D representation of a body canal or cavity having a head having a diameter about 10mm in diameter with a central region (of about 6 mm diameter) in which two CCD photosensitive arrays 701 and 702 are disposed.
• the endoscope is also provided with four regularly circumferentially distributed channels 704 for accommodating surgical instruments and the like, as is standard. Cables 703 are disposed regularly about the peripheral region of the endoscope and are used for bending the endoscope to guide it through a body canal, as is standard.
• a fibre-optic bundle 710 terminates at the front face of the head 705 and carries a light beam from an illumination source at the proximal end of the endoscope (not shown).
• the beam is projected onto the surface S of the region to be viewed and appears as an array of overlapping circular areas, only one of which is shown.
• the size and distortion from circularity of these illuminated areas can be detected by the detector arrays 701 and 702 to provide an initial estimate of the 3-D coordinates of the surface region S (indeed in some cases satisfactory- information could be provided by this technique alone, so that only one photo sensitive array would be needed).
• the illuminated region S is imaged separately by the two photosensitive arrays and the images are correlated as described above with reference to Figure 14 to find the accurate 3-D coordinates of the interior of the body canal.
• the image is focussed by a graded index (GRIN) rod or fibre 71 1.
• GRIN graded index
• the illumination source is coupled to one or more selected fibres and at least these fibres in bundle 710 are provided with lenses 708 (or GRIN portions) which focus an optical beam as a spot on the interior surface of the body canal as shown.
• the focussed spot can be scanned eg in raster fashion either by coupling the illumination source to adjacent fibres in succession to displace the beam relative to the exit face of the fibre bundle in eg raster fashion or, less preferably, by moving the optic fibre bundle by any suitable means, eg piezoelectrically.
• the resulting spot can then be tracked and imaged by the photosensitive arrays and the coordinates on the respective arrays used to derive the 3-D coordinates of the surface S. as explained with reference to Figures 1 and 2.
• a single photodetector could be employed, as in our PCT/GB95/01994, preferably satisfying the Scheimpflug condition
• a conventional monocular rigid endoscope 401 having an objective lens 402 at its distal tip and an ocular 403 at its proximal end is combined with a video camera 404 which focusses light exiting from ocular 403 onto the photosensitive image plane I of a photosensitive detector (such as a CCD for example) by means of a focussing lens L.
• a photosensitive detector such as a CCD for example
• lens L will normally be a multi-element lens and the exposure will normally be controlled by an iris (not shown).
• the camera 404 may be a cine camera, in which case the light from lens L is focussed onto the photosensitive image plane of cine film.
• the distal head of the endoscope is provided with miniature inertial sensors 1 1 and 12 similar to those shown in Figure 8 which measure angular rotation about three orthogonal axes and linear acceleration along the three orthogonal axes respectively.
• the signals from these sensors are integrated as in the embodiment of Figure 8 to derive the instantaneous position and orientation of the endoscope head.
• the lens L is used to obtain stereoscopic views by alternately occluding the light exiting from the left and right regions of the ocular 403 with a shutter means 405 preferably at a rapid rate such as 50 times per second (for video), under the control of a signal from processing circuitry 408.
• the shutter means 405 may be provided in front of lens L as shown, between different lens elements of a multi-element lens (not illustrated) or may be located between the lens L and photosensitive image plane I, for example.
• the shutter may be a LCD shutter printed on a surface of lens L.
• the rays blocked by shutter means 405 are preferably parallel as shown but may alternatively converge or diverge.
• the shutter should preferably be located close to the lens.
• the stereoscopic image pairs can be tagged w ith position information and orientation information as in the embodiment of Figure 8 in order to obtain a 3-D representation of a body canal in which the endoscope is inserted, as previously described in connection with Figure 8
• the shutter 405 is omitted and the endoscope head is manipulated to change its position and/or onentation the resulting vie s being tagged w ith position and/or onentation information in order to enable the required 3-D representation to be obtained
• a hood (not shown ) is provided at the interface of camera 404 and 15 endoscope 401 to prevent stray light ente ⁇ ng the video camera, or the camera and endoscope are integral
• the endoscope may be a laparoscope, a borescope, a cystoscope or an arthroscope for example.
• the user may pull focus or zoom (assuming the lens has this facility ) without affecting the stereoscopic imaging
• a switching output (synchronised with the shutter 405) and a video output may be fed to a standard stereoscopic v iew ing arrangement as show n in Figure 1 of WO 95/14952
• the endoscope aspect of the invention is also applicable to the arrangements shown in GB-A-2,268.283. US 5.222.477 and US 5.12,650 for example It is not limited to any of the image processing methods disclosed herein and can be used in conjunction with eg the scanned pattern and depth detection anangements of our co-pending PCT/GB95/01994 for example, using eg an optic fibre array (not show n) in the endoscope head to project the pattern

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Vascular Medicine (AREA)
• Multimedia (AREA)
• Length Measuring Devices By Optical Means (AREA)
• Image Processing (AREA)
• Image Analysis (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (2)

Family

ID=10816794

Family Applications (1)

Country Status (6)

Cited By (6)

Families Citing this family (30)

Citations (9)

Family Cites Families (3)

• 1997
  • 1997-07-31 GB GBGB9716240.8A patent/GB9716240D0/en active Pending
• 1998
  • 1998-07-31 CA CA002299426A patent/CA2299426A1/en not_active Abandoned
  • 1998-07-31 JP JP2000505603A patent/JP2001512241A/ja active Pending
  • 1998-07-31 AU AU86362/98A patent/AU8636298A/en not_active Abandoned
  • 1998-07-31 GB GB9816756A patent/GB2328280B/en not_active Expired - Fee Related
  • 1998-07-31 EP EP98937633A patent/EP1000318A2/en not_active Withdrawn
  • 1998-07-31 WO PCT/GB1998/002307 patent/WO1999006950A2/en not_active Application Discontinuation

Patent Citations (9)

Also Published As

Similar Documents

Legal Events