WO1995030126A1

WO1995030126A1 - Assessment of distance

Info

Publication number: WO1995030126A1
Application number: PCT/GB1995/000969
Authority: WO
Inventors: Adrian Francis Clark; Seng Way Chan
Original assignee: University Of Essex
Priority date: 1994-04-29
Filing date: 1995-04-27
Publication date: 1995-11-09
Also published as: JPH07312611A; GB9408533D0; JP3178645B2

Abstract

In a method for performing the assessment of the distance to an object from a camera, a mirror is arranged on the optical axis of the camera so as to be inclined thereto and the mirror is then rotated about the optical axis. The camera produces images in the form of electronic signals and two successive images are captured and compared in order to allow the assessment of the distance. Prior to the comparison of the images, compensation for the rotation of the mirror is performed, either by optical processing of the light reflected by the mirror, or by electronic processing of the images from the camera. Common features in the two compensated images may be identified and compared to allow the distance to the object to be assessed.

Description

ASSESSMENT OF DISTANCE

This invention relates to methods of and apparatus for assessing the distance to an object from a particular location at which a camera may be mounted.

Depth perception is one of the more important functions of the human visual sys¬ tem. Although several cues are involved — such as shading and relative size — stereo plays a crucial role. Stereoscopic vision gives the observer the opportunity to assess subjectively the distance from his eyes to the object. Alternatively, using computer techniques operating on stereoscopic images, an assessment of the distance to an object may be derived.

Computational stereo vision is now in widespread use and mimics the human visual system by having two cameras viewing a three-dimensional scene. In recent years, single-camera approaches to computational stereo have become of interest and the most simple of these merely replaces the cameras with a mirror system or a glass plate, though more recent work has employed inclined mirrors. Such arrangements suffer from two disadvantages: they have a limited field of view and, in order to produce an acceptable baseline, must be physically large.

There is a demand for inspection equipment which is able to provide stereoscopic vision and yet which is relatively small. For example, to be able to view the interior of certain kinds of machinery without dismantling that machinery may save much time and labour; mention may be made here of aircraft engines, which have to be inspected in detail at frequent intervals. Also, medical endoscopy and related techniques require a small physical size and a panoramic field of view is an important factor.

It has been proposed to use a rotating camera in the construction of a stereoscopic viewing system and more recently there have been proposals to replace the rotating camera with a mirror [D. W. Murray and P. A. Beardsley: Range Recovery using Vzr- tua. Multi-camera Stereo, British Machine Vision Conference 1992, Springer- Verlag, September 1992]. This paper analyzed the case in which the axis of rotation of the mir¬ ror is perpendicular to the optical axis of the camera and reported simulated, though not experimental, results. It also gave a preliminary analysis of a case in which the axis of rotation of the mirror lies along the optical axis of the camera but concluded that the problem here was essentially insoluble.

The present invention stems from further research with a system for stereoscopic vision where a mirror is arranged to rotate about the optical axis of a camera. It has been determined that, provided either proper optical processing of the reflected light prior to capture by the camera or proper electronic processing of the images obtained from the camera is performed, a useful result may be obtained which allows stereo¬ scopic viewing of an object or, by further processing of images obtained by the system, a computed assessment of the distance from the camera to the object.

According to one aspect of the present invention, there is provided a method of per¬ forming distance assessment to an object, comprising the steps of:

• arranging a camera and a mirror to allow the camera to view the object with light reflected by the mirror, the mirror lying on but inclined with respect to the optical axis of the camera and being arranged for rotation about that axis and the camera providing images in the form of electronic signals;

• rotating the mirror about the camera optical axis;

• obtaining from the camera images of the object taken at different rotational pos¬ itions of the mirror and assessing the distance of the object from the camera by considering the images;

• wherein compensation for the rotation of the mirror is performed before said con¬ sidering of the images, either by optical processing of the light relected by the mirror or by electronic processing of the images from the camera; and

• two rotationally-compensated images both containing said object are obtained from two different mirror positions and are compared, the distance of the object from the camera being assessed from the comparison. It will be appreciated that, in the method of the present invention, the images of the object from two angularly-spaced positions of the rotating mirror are processed by effecting a rotation of these images in order to compensate for the rotation of the mirror. This compensation may be performed optically, on the reflected light before it reaches the camera, or electronically on the images obtained from the camera. Then, the rotationally-compensated images may be appropriately compared in order to give the required assessment of distance. Such assessment may be subjective, by an observer essentially simultaneously viewing the two rotationally-compensated images, one image with each eye. Alternatively, computational techniques may be employed by operating on one of the images to dete_τrιine features in that image and then comparing the determ¬ ined features with corresponding features in the other image. Conveniently, this may be performed by using the determined features of one of the images as a template for the other image; for example, a known correlation tracker may be used to locate the features in the second image.

According to a second aspect of the present invention, there is provided apparatus for assessing distance to an object, comprising:

• means for mounting at some required location a camera which provides images in the form of electronic signals;

• means mounting a mirror on the optical axis of a mounted camera but inclined to that axis, said mounting means being arranged to effect controlled rotation of the mirror about the camera optical axis;

• means monitoring the angular position of the mirror during the rotation thereof;

• means to capture images from the camera output, said images being taken at suc¬ cessive angular positions of the rotating mirror;

• compensation means to compensate for the rotation of the mirror, the compensa¬ tion means acting either optically on light reflected by the mirror or electronically on the images produced by the camera; and • comparison means to compare the rotation-compensated images to permit an assess¬ ment of the distance to the object from a mounted camera.

The apparatus of this invention is configured to perform methods of distance assess¬ ment (including stereoscopy or depth assessment) as defined hereinbefore as a first aspect of the present invention.

Further preferred aspects and features of the present invention will become appar¬ ent from the following analysis of the principles of the inventive concept. Moreover, two preferred arrangements for performing methods of this invention, and apparatus therefor, will also be described but solely by way of example. In the drawings:

• Figure 1 is a schematic diagram showing a mirror arrangement for use in a method of this invention;

• Figure 2 is a side elevation of an optical system for use in said method;

• Figure 3 illustrates the effective imaging system, after removal of the effect of rotation;

• Figures 4(a) to 4(d) show the rotation-compensation process;

• Figure 5 is a plan view of a system with a virtual camera;

• Figure 6 shows in block diagram form a stereoscopic viewing system arranged in accordance with the present invention that processes the images electronically;

• Figure 7 shows in block diagram form a stereoscopic viewing system arranged in accordance with the present invention that processes the images optically;

• Figure 8 shows a dove prism and the path of light through it; and

• Figure 9 is a comparison of directly-measured and calculated distances, obtained with an experimental example of apparatus of the present invention. The basic arrangement of the system is shown schematically in Figure 1 and in side elevation in Figure 2. Light from the scene is reflected by an mirror positioned at the origin and inclined at an angle θ onto a camera whose lens is centred at the point

( ^C\

C = 0 (1)

\ 0 J i.e., the camera is placed a distance C away from mirror along the _c-axis. The mirror is made to rotate about the _c-axis by an angle φ. We proceed by considering a 'virtual' camera: this occupies a position equivalent to the real camera but 'behind' the mirror; i.e„ it is the position at which the real camera would have to be located if the mirror were not present.

From Figure 2, we observe that the surface normal, π, is given by

\smθcos φJ In order to find the location of the virtual camera, C__v , we first consider the perpendicular projection of the real camera position onto the mirror plane (see Figure 2):

Now, C_m is in the mirror plane, so C„. • h — 0. Hence,

(C - ή) - ή = Q (4) and consequently = C cos θ. The position of the virtual camera may be written as

from which one may calculate Q__v:

As the mirror rotates, φ varies. The -component of Q_υ is unaffected by such rotation, as we would expect, while the y- and z-components undergo circular motion in the x = 0 plane.

The above derivation allows us to determine the movement of the centre of the vir¬ tual camera as the mirror moves but it does not allow us to determine the movement of an arbitrary point on the image. To do that, let us consider a point

i.e., displaced by amounts δx, δy and δz from C_ along the x-, y-, and z-axes respect¬ ively. This will have projection

C = C a n (8) on the mirror plane, from which we obtain

a' = C - ή. (9)

The corresponding virtual position is

= C' - 2(C! - n)n. (10)

Now,

C! • ή = cos Θ(C + δx) + sin θ sin φδy + sin θ cos φδz (11) and, substituting (11) into (10), we obtain

— cos 20(C + δx) — sin 2Θ sin φδy — sin 2Θ cos φδz \ = - sin 2Θ sin φ(C + δx) - (2 sin²0 sin² φ - l)δy - 2 sin²0 sin cos φδz

V - sin 20 cos (C -I- &r) - 2 sin²0 sin φ cos < 5y - (2 sin²0 cos² - l)δz )

(12) which simplifies to (6) when δx = δy — δz = 0.

A comparison of (6) and (12) shows that the motion of a point that does not he on the optical axis consists of two components: the circular motion of the virtual camera around the origin and a rotation of C around the optical axis. This realization allows the motion of images recorded by the camera system to be decomposed into a more useful form, as detailed in the next section.

Decomposition of Rotations

We may write (6) and (12) in the form

C_υ = T - C Ω!_V = T - C'

respectively for some transformation which maps from real to virtual coordinates. The difference between these positions is

C!_υ - Q_υ = Z - (C! - C) (13) from which we obtain

/ cos 20 sin 20sin <

T = - sin 20 sin φ 2 sin²0 sin² φ — 1 . (14)

Vsin 20sin r/> 2 sin²0 sin φ cos φ

This expression for _ must contain the two rotational components mentioned above in addition to any other transformations due to the imaging process. In particular, it must contain a rotation through an angle φ about the τ-axis, R_ψ, and a modification of the axes' directions due to the mirror. Taking both rotation and reflection into account, we may determine ', the transformation without these effects, from _' = R , where

assuming a right-handed coordinate system. Performing the matrix multiplication, we obtain

This expression shows that, after removing the effect of the rotation of a point C , around C__υ, the optical axis of the virtual camera, there is still a transformation present that depends simply upon φ: in effect, the imaging system has been reduced to a planar system with a rotating camera (see Figure 3).

The effect of this simple rotation of the image thus changes a difficult stereo match¬ ing problem, one which would require the matching of image features that are affected by translation and rotation between frames, to a much simpler one in which the fea¬ ture matches must lie on the scan lines. The following section describes how one may estimate the distance to points in the far field from measurements obtained from these rotation-compensated images.

Depth Estimation from Rotation-Compensated Images

Some images recorded from a system of the type of Figure 1 are shown in Figure 4 along with their rotation-compensated versions. Figures 4a and b show a pair of images recorded from the camera and Figures 4c and d show the same images after electronic rotation compensation. (In our example apparatus, the mirror rotates clockwise when viewed from above; the rotation-compensated images then survey the surroundings in an anti-clockwise manner.) Close inspection of these images will show that disparities may be observed: this means that one may use the images to estimate the distance from the observer using an approach similar to that for conventional computational stereo. Indeed, it is not difficult to derive equations that permit depth to be estimated from these rotation-compensated images. In doing so however, it is necessary to consider the map¬ ping from the raster and scan directions of the image onto the world coordinate system:

• the scan direction (_c-axis) of the image is coincident with the world y-axis after undoing the rotation;

• the raster direction (y-axis) of the image lies along the world rc-axis.

We shall also make the usual assumption that the camera may be accurately described by a 'pin-hole' model. Let us consider a point

in the object (Figure 5) that is imaged to P_r in the back focal plane of the real camera. Let us write

By similar triangles, we have x τn_x = — = f X - C Applying the transformation _' from real to virtual coordinates yields

- cos φY + sin φZ m_x = (16) cos 2ΘX + sin 20 sin φY + sin 20 cos φZ + C In the case when 0 = π/4, (16) may be simplified to _■ 771 — ___ —____ c__o__s___ φ__Y____ +____ s_i_n___ φ__Z____ ^x sin φY + cos φZ + C which may be re-arranged to give

x ≡ Cm_x = ( — m_x sin φ — cos φ (17)

Now, (17) gives an equation in two unknowns (Y and ) that relates to a single image measurement, x. However, we are free to vary φ: doing so builds up multiple 'instances' of (17) that refer to the same Y and Z, and we may then use numerical tech¬ niques to produce estimates of Y and Z.

Experimental Considerations

On the basis of the theoretical treatment presented above, one can construct practical systems that incorporate image capture, rotation compensation, stereoscopic display, and distance assessment. One such system is illustrated in block diagram form in Fig¬ ure 6: it captures images and then performs the rotation-compensation step on the res¬ ulting digital image.

An alternative configuration is shown in Figure 7: this differs by performing the rotation-compensation step optically before the images are captured. The latter config¬ uration is more suitable for real-time operation, though at the cost of increased mechan¬ ical complexity; it will yield higher-quality results since rotation and interpolation of a digital image are inherently inaccurate in the region of object boundaries. Optical rota¬ tion compensation may be accomplished by means of an appropriate optical system, such as a dove prism (Figure 8, known per se) rotated at half the rate of the mirror.

As Figure 4 shows, these images may be viewed directly, in order to obtain a visual impression of distance, or they may be used to calculate depth by matching features in the two images, estimating the disparity, and using d e formulation presented above. Estimates of the mirror orientation are required for the image rotation and distance cal¬ culation operations. It is anticipated that the camera-mirror separation and focal length of the camera lens will be estimated by calibration; indeed, the mirror rotation φ (or, rather, its speed of rotation dφ/dt) may also be estimated by calibration, as we shall see below. However, it is anticipated that direct measurement of φ will yield more accurate results.

A non-real-time, proof-of-concept apparatus was constructed in accordance with the arrangement shown in block form in Figure 6. This system used a plane mirror of about 10 x 10 cm which was rotated at 16 revolutions/minute; this allowed a reason¬ able overlap between consecutive frames and indeed there was still an overlap between every second frame. A video camera was aligned such that its optical axis (i.e., the centre of captured images) was aligned with the centre of rotation of the mirror. Images were captured in real time on a digital disc recorder using techniques known in the art and the images were then transferred to conventional computer equipment for pro¬ cessing in non-real time.

An important experimental consideration concerned the camera. The frame rate of a video camera is sufficiently slow that the mirror rotation caused blurring. As a con¬ sequence, a shuttered CCD video camera was employed but, even so, rotational blur¬ ring was apparent between the fields of a frame. Consequently, only one field of each frame was employed. A consequence of the use of a shuttered camera was that captured frames were rather dark but there was still enough contrast for matching and disparity measurements to succeed. In a more mature system, the rotating mirror could also be used to illuminate the scene by placing a light source near to the camera.

The rotational rate of the mirror was determined and used in the subsequent pro¬ cessing of the images. The camera was aligned with the rotational origin and calib¬ rated to determine the location in the image of the centre of rotation. The focal length of the camera was taken from the manufacturer's data sheet. The distance, C, between the rotation centre on the mirror and lens centre was then measured and the angle of rotation between consecutive frames estimated as described in the previous section. A sequence of images was then captured and processed to estimate the distance to these objects. The focal length of the camera, the size and rotational rate of the mirror, and the distance C may be varied to optimize operation.

The captured images were initially rotated about the rotation origin on the image with interpolation. The rotation-compensated images were passed through an interest operator to identify distinct features. The Moravec operator [H. P. Moravec: "Rover visual obstacle avoidance" in Proceedings of the 7— International Conference on Arti¬ ficial Intelligence pages 785-790, University of British Columbia, Vancouver, Canada, August 1981] was used for simplity: this attempts to select good features over small windows by returning areas that have local maxima of a directional variance measure, calculated by finding the sums of squares of differences of pixels adjacent in each of four directions over each window.

Each feature found in the first frame was extracted and used as a template for the second image. A correlation tracker was used to track the feature in the second image. The accuracy of feature location was enhanced by interpolating correlation values in the neighbourhood of the peak; it is estimated that the resulting accuracy is about ±0.5 pixels. The located features were then inserted into (17) and least-square optimization used to estimate values for Y and Z.

Experimental results, comparing the distances calculated using periscopic stereo and direcdy-measured values, are shown graphically in Figure 9. Ideally, the points should he on die line of unit gradient: taking errors into account, these results are close to this ideal. The errors indicated assume a 0.5-pixel error in the location of features in each frame, giving a one-pixel error in the disparity. They do not include other factors (e.g., error in the measurement of C or the focal length) and are dierefore under-estimates of the probable errors.

This specification has addressed the problem of 'periscopic stereo' and found that, although the problem initially appears to be a very difficult one, a simple decomposition of d e rotations present greatly simplifies it. This suggests a procedure for producing stereo images, both for direct viewing and for the estimation of depd . The experimental measurements have proven the viability of the technique.

This technique may be used to provide a panoramic distance measurement capabil¬ ity for, say, mobile robots. However, with careful design, it will be possible to make the sensor very compact. Periscopic stereo may thus be regarded as being complementary to, rather than a competitor of, conventional stereo: the former is well suited to view¬ ing the outside of objects, while periscopic stereo may well find a niche in exploring the inside of objects such as aircraft engines; it may also be able to provide a com¬ pact stereoscope and distance estimation system for endoscopy, which could be greatly beneficial for medical techniques such as keyhole surgery.

Claims

CLAEMS

1. A method of performing distance assessment to an object, comprising the steps of:

• rotating the mirror about the camera optical axis;

• obtaining from the camera images of the object taken at different rotational positions of the mirror and assessing d e distance of the obj ect from the cam¬ era by considering the images;

• wherein compensation for the rotation of the mirror is performed before said considering of the images, either by optical processing of the light relected by the mirror or by electronic processing of the images from the camera; and

• two rotationally-compensated images botii containing said object are obt¬ ained from two different mirror positions and are compared, the distance of the object from the camera being assessed from die comparison.

2. A method as claimed in claim 1 and in which optical processing is performed, said optical processing comprising rotating an optical element located on die optical patii between the mirror and the camera.

3. A method as claimed in claim 2, wherein the optical element comprises a dove prism and the element is rotated in synchronism with but at half the rate of the rotation of the mirror.

4. A method as claimed in claim 1 and in which electronic processing is performed, wherein the instantaneous position of the rotating mirror is monitored and said instantaneous position is processed together widi the image obtained at tiiat pos¬ ition in order to compensate for the rotation of the image consequent upon mirror rotation.

5. A metiiod as claimed in claim 4, in which the processing of the images is .per¬ formed by rotating the images about the rotation origin on the image with inter¬ polation.

6. A method as claimed in any of d e preceding claims, in which the comparison of two rotationally-compensated images is performed by essentially simultaneous display of die two images to the two eyes respectively of an observer, and d e assessment of the distance is performed subjectively.

7. A method as claimed in any of die preceding claims, in which the comparison of two rotationally-compensated images is performed by analysing the differences between the two images and computing therefrom the apparent distance to an object in both images.

8. A method as claimed in any of the preceding claims, in which two successive images are compared to obtain the assessment of distance, which said images are obtained witii angularly-spaced mirror positions of the order of 3° to 5°.

9. A method as claimed in any of die preceding claims, in which the mirror is planar and is mounted witii its plane at 45° to the optical axis of the camera.

10. A method as claimed in any of die preceding claims, in which the processing of the images obtained from the camera is performed in real time.

11. A metiiod as claimed in any of the preceding claims, in which the object is illu¬ minated with light obtained from a light source positioned at or adjacent die cam¬ era and directed on to the mirror for reflection on to die object.

12. Apparatus for assessing distance to an object, comprising: • means for mounting at some required location a camera which provides images in the form of electronic signals;

• means mounting a mirror on the optical axis of a mounted camera but inclined to that axis, said mounting means being arranged to effect con¬ trolled rotation of the mirror about the camera optical axis;

• means monitoring the angular position of the mirror during d e rotation diereof;

• means to capture images from the camera output, said images being taken at successive angular positions of die rotating mirror;

• compensation means to compensate for the rotation of d e mirror, the com¬ pensation means acting eitiier optically on light reflected by die mirror or electronically on the images produced by the camera; and

• comparison means to compare the rotation-compensated images to permit an assessment of the distance to die object from a mounted camera.

13. Apparatus as claimed in claim 12, wherein the compensation means comprises an optical element mounted on the optical axis and means to rotate d e optical element at a controlled rate synchronised to the rotation of the mirror.

14. Apparatus as claimed in claim 13, wherein the optical element comprises a dove prism, and said element is rotated at one half of die rate of rotation of die mirror.

15. Apparatus as claimed in claim 12, wherein the compensation means comprises electronic means to process said images in association witii the respective mon¬ itored angular position of the mirror to effect rotation of said images tiiereby to compensate for the rotation of die mirror.

16. Apparatus as claimed in any of claims 12 to 15, wherein the comparison means comprises two display devices arranged to display die two rotation compensated images for simultaneous viewing by the two eyes of an observer.

17. Apparatus as claimed in any of claims 12 to 15, wherein the comparison means comprises means to identify distinct features in one of the images, and means to correlate an identified feature in one image with the corresponding feature in the otiier image.

18. Apparatus as claimed in claim 14, wherein the output of the correlating means is used to compute the distance from a mounted camera to the identified feature.