GB2250604A

GB2250604A - Small standoff one-camera-stereo adaptor

Info

Publication number: GB2250604A
Application number: GB9124117A
Authority: GB
Inventors: Peng Seng Toh; Thomas Stuart Frank Nichols; Yean Joo Chong
Original assignee: Individual
Current assignee: Individual
Priority date: 1990-11-10
Filing date: 1991-11-11
Publication date: 1992-06-10
Anticipated expiration: 2011-11-11
Also published as: GB2250604B; GB9124117D0

Abstract

This invention relates to an apparatus and a method for automatically acquiring optimal stereo images for stereoscopic processing. The invention consists of a beam splitter 1, a mirror 2 mounted on a rotatable stage, shutters 6, 7 and a compensating zoom lens 9. These components are controlled by a processor and enclosed in a light-protective casing with means for attaching to a camera 4 or camera lenses. Stereo images are acquired by a pre-stereoscopic processing algorithm which controls the mirror in executing the correct vergence movement until maximally overlapped stereo images are obtained. Maximally overlapped stereo images are optimal for machine stereoscopic processing. The processor also controls the movement of the compensating zoom lens in conjunction with the vergence movement so that the two stereoscopic images are of the same optical magnification. With the incorporation of this compensating zoom lens, this invention permits the acquisition of stereo images for object to be positioned at a shorter distance than is possible otherwise. <IMAGE>

Description

Small Standoff One-Camera-Stereo Adaptor This invention relates to a method of and an apparatus for allowing the use of one stationary camera to acquire stereo images of the same scene with maximum overlap and provides means of compensating for the differences in image magnification of the stereo images obtained through different optical paths.

Stereoscopic or binocular cue is one of the most reliable and accurate visual cue for deriving the three-dimensional information of a scene. Two or more images with overlapping areas are required for stereoscopic processing. These images are obtained by employing multiple cameras or by moving a single camera to different viewing positions.

The acquisition of stereoscopic images is very important to the successful derivation of three-dimensional information.

The invention is an example of an active image acquisition method (to be distinguished from using active sensors such as laser or ultrasonic ranging). The primary objective of active vision is to provide a better posed image that is algorithmically simpler to be processed by a computer. It is especially important for the purpose of deriving threedimensional information that images be acquired in an active way. This is because it is required that the stereo images be maximally overlapped and simultaneously focused Different methods of acquiring stereo images are outlined in the following: 1. Parallel setup (the optical axes of two or more cameras are arranged in parallel); 2. Cross-looking setup (the optical axes of two or more cameras are converged at a fixed point).

3. Fine constant displacement setup (the camera is displaced by a fine constant displacement to obtain more than one image. The displacement is not adaptable to the depth of the scene).

4. Fixed locus setup (the camera moves in a fixed path either predetermined or by use of a guide. The displacement may be varied. Looming is a form of fixed locus setup); 5. Adaptive displacement setup (the direction and displacement of the camera is adapted to the scene).

6. Adaptive vergence setup (the optical axis of one or more cameras can verge, ie.

converge or diverge, so as to fixate on the point of interest).

Methods 1 to 4 can be classified as passive because they are not capable of adapting to the scene where optimal stereo images can be acquired. Passive setup suffers from the fundamental difficulty of obtaining overlapped images. Methods 5 and 6 are active because they are adaptive to the scene. This invention belongs to method 6.

Cost and spacing are major factors in choosing the number of cameras. Although multiple cameras have the advantage of acquiring stereoscopic images simultaneously, they cost more and take up more space. There are cases where it is simply not possible to use more than one camera because of space limitadon. It is obvious when obtaining images each separated by a minute displacement that only one camera is allowed. Moving a camera to acquire images form several view points can also suffer from positional error and consequenfly each camera position has to be re-calibrated individually. In contrast, this invention enables the use of a single, stationary camera to capture a maximum overlapped stereo image with reduced cost and spacing.

One aspect of this invention is the use of only one video source, usually a video camera, together with other optical components to capture stereoscopic images. This arrangement does not require to move the video camera to different positions. The optical components include a beam splitter, a mirror and a compensating zoom lens, all enclosed in a lightprotective casing, herein collectively called the adaptor The adaptor has provision to attach to the video camera with a proper and rigid alignment During operation, the optical axis of the video camera is first aligned to the object of interest. Moving the video camera to align with the object of interest will also move the entire adaptor along.The beam splitter is mounted at 45 degrees to the optical axis of the video camera such that it can view objects located straight ahead (along its pptical axis) as well as objects that are located at right angle to the straight ahead direction For the sake of clarity, the straight ahead view seen by the video camera is herein called the direct view. Since it is the stereoscopic images that is located in front of the video camera is of interest to the present invention, the rotatable mirror is positioned on the right angle path for bending images that is located in front of the video camera. This constitutes the side view and formed a stereo image pair with the direct view. By employing shutters, the video camera can select to view the straight ahead view or the side view seen through the mirror.In this way, two images of the same scene can be acquired. The optical axes of the video camera, the beam splitter, the mirror and the compensating zoom lens are all orientated on the same plane, herein called the horizontal meridional plane.

According to another aspect of the present invention, the stereoscopic image acquired should have a maximum overlapping area so that it is optimum for stereoscopic processing.

Maximum overlapping area, in the case of the human visual system, is obtained by the use of vergence eye-movement which involve the coordinated movement of the two eyes to fixate on the object of interest. In the present invention, vergence movement is achieved by rotating only the mirror. This greatly reduces the complexity of the controlling mechanism which is achieved by the use of a computer. The rotation of the mirror is activated by an electrical motor (either a DC servo motor or a stepper motor). The degree to the which the mirror should be rotated is computed by comparing the stereoscopic image pair. Optimum mirror rotation is obtained when the side view is most similar to the direct view. The common methods of image comparison are for examples, normalised cross-correlation coefficient, sum of squared difference, absolute sum of difference.For those skilled in the art, the implementation of these image comparison methods are straight forward The accuracy in estimating depth in the scene is dependent largely on the distance between the mirror and the beam splitter which essentially formed the baseline of the present invention. However, the different in the optical path length between the direct view and the side view will increase with the baseline. Consequently, the magnification of these two images will be difference. The present invention uses a compensating zoom lens inserted between the mirror and the beam splitter to correct for the different in magnification. The adjustment of the compensating zoom lens is coordinated with the rotation of the mirror such that it is adaptive to different fixation distance.The nearer the fixation point, the larger is the difference in magnification between the direct view and the side view. When the fixation point is far away, the difference in magnification will be very small and compensation will not be required and the compensating zoom lens can be temporarily removed The field of view of the side view seen through the mirror will always be smaller than that of the direct view. The present invention only uses the minimum field of view that is common to both the direct and the side view. A window that is large enough to accommodate this field of view is assigned as the useful area to both views. Stereoscopic processing such as image correlation, image matching and disparity estimation are carried out within this window.

In order for a stereoscopic machine vision system to operate autonomously, other than vergence control, focus and aperture (iris) control are also essential. Automatic focus and aperture control are well established techniques for those who are skilled in the art and require no detailed explanation here.

Patent 1 566 187 entitled "stereoscopic cine-camera and projector system" described a apparatus for capturing stereo images for cine-camera However, it had not stated the method whereby the side mirror can be controlled in order to bring the left and right images into registration. Also, the difference in image magnification between the left and right images, due to the difference in optical path length, has not been addressed. Stereo images having different magnification made registration and matching more difficult. US patent 4480893 entitled "Optical and viewing system for stereoscopic photography" described a method where the two optical axes of the stereoscopic views are rigidly set in parallel. PCT )HU83A)0041 described a 3 mirrors and an "electro-mirror" design.UK patent .GB 2 135 470A described a method for 3D photography employing polarizing filters for stereoscopic image recording. Patent 1 575 851 entitled "Apparatus for stereoscopic photographic" described a method using colour filter for the encoding of stereoscopic images.

The present invention is described in detail by way of a preferred embodiment with the following accompanying drawings: Fig. 1 is a block diagram of the preferred embodiment of the present invention.

Fig. 2 and 3 illustrate the effective field of view of the side view when the mirror is rotated to different angular positions.

Fig. 4a illustrates the area in which correlation computation will be carried out.

Fig. 4b shows the relationship between the translational offset and angular offset.

Fig. 5 shows the arrangement of the different optical components.

Fig. 6 illustrates the different in magnification between the straight ahead view and the side view due to the different in optical path length Fig. 7 shows the method in which a compensating zoom lens can be used to correct the different in magnification.

A preferred embodiment of the present invention is shown in Fig. 1. The essential components are, a beam splitter 1, a mirror 2 mounted on a rotatable stage 3 driven by a electrical motor-21 and a compensating zoom lens 9 which is also electrical motor 22 driven. The beam splitter 1 has an equal transmit/receive ratio and is positioned at 450 to the video camera 4 optical axis. These components are fitted into a single casing 5 to provide structural rigidity and to block out extraneous light Shutters 6 and 7 are used to multiplex the acquisition of the left and right images. The left image corresponds to the straight ahead view seen through the beam splitter 1. The right image corresponds to the side view that is reflected by the mirror 2, through the compensating zoom lens 9, then reflected by the beam splitter 1 into the video camera 4.A translational rail 20 is used to guide the movement of the compensating zoom lens 9. The casing has mounting 8 to fit onto the video camera 4. The output of the video camera, that is the image captured by the video camera, is sent to a computer/processor 23. A frame-grabber installed in the computer/processor 23 is used to receive the image captured from the video camera 4. The motors 21 and 22 are both controlled by the computer/processor 23. The coordination of the operation of the shutters 6 & 7 are both controlled by the computer/processor 23.

The optical axes of the video camera 4, the beam splitter 1, the mirror 2 and the compensating zoom lens 9 are all orientated on the same horizontal meridional plane. For the purpose of achieving automatic vergence control in which the stereoscopic images will have maximum overlapping areas, the rotation of the mirror 2 driven by the electrical motor 21 has to be properly controlled by the computer/processor 23. In the present invention, the vergence movement has only one degree-of-freedom, ie. the rotation of the mirror 2. In conjunction with the mirror vergence movement, the compensating zoom lens 9 is also simultaneously adjusted The computation of the movement of the compensating zoom lens 9 will be explained later.

The sequence of operation for the present invention is described in the following. The first step is essentially for the present invention to align the optical axis of the video camera 4 with the object of interest. The system begins with the acquisition of the image directly ahead of the video camera 4 seen through the beam-splitter 1. In order to acquire the direct view, shutter 6 is opened whereas shutter 7 is closed to prevent interference from the side image. Once the desired object of interest is aligned with the optical axis of the video camera 4, the image acquired is indeed the left image. The second step is the acquisition of stereoscopic image, that is the right image will now be captured The left image that has been acquired are stored in the computer/processor 23 image buffers.For the acquisition of the right image, shutter 7 opens and shutter 6 closes which then allows the mirror reflected image to be acquired The right image is also stored in the computer/processor's buffer for processing. The computer/processor will then carry out a correlation between the right and the left images. The detail in which the correlation between the two images are carried out will be described later. Based on the result of correlation computation, the computer/processor 23 will then command the motor 21 to rotate the mirror will by an appropriate amount. A new right image will be captured. The correlation computation between the newly captured right image and the left image will be performed.The computer/processor will again command the motor to rotate the mirror to a new angular orientation. The appropriate right image is acquired when the correlation between the left and the right image exit a certain threshold defined by the user.

The present invention have two modes of operation, manual and automatic. When operating this device in a known environment such as acquiring the images of objects on a conveyor belt at a fixed distance, it can be operated in the manual mode. The vergence angle of the mirror and the position of the zoom lens can be pre-determined and hence manually preset before operation begins. The other parameter such as the camera focus and aperture can also be preset.

For autonomous operation, the incorporation of an automatic vergence and automatic focusing controls are essential. The focal length of the first image is used to estimate the vergence angle of the mirror. Subsequently, after the mirror has been rotated to the new position controlled by the computer/processor 23, computation is carried out to estimate the fine angular displacement from the fixation point using correlation method.

The image that is obtained directly through the beam-splitter is aligned in the direction of the camera optical axis which is also aimed at the point of fixation. The direct image is focused and this focus setting is also used for acquiring the side view which is reflected by the mirror, through the zoom lens and reflected by the beam-splitter into the video camera It should be noted that the direct image has a larger field of view 10 whereas the effective side image is smaller as shown in 11 of Fig 2 and 3. The largest side image will appear in the camera receptor when the mirror 2 is orientated at a=9(P and towards 0 will result in a smaller view. The useful range of the mirror orientation angle where fixation is ensured is between cm45 to (900 - arctan Ti/b) where 11 is the minimum standoff distance and b is the baseline. In Fig. 5, b is the distance between 0203.

For the purpose of focusing and vergence control, a window representing the region of interest within the image is selected The size of the window within the effective side view is decided by the permissible vergence rotation. The smallest size of this window can be determined when the mirror is positioned at a=450.

The focus control of the direct view is based on a monocular attentive focusing technique so as to obtain a common focus for both images. This means that only the portion of the image contains within the attentive window 13 is focused such as illustrated in Fig 4a The focus setting obtained from the direct image provides a rough estimate of the vergence angle to which the mirror 2 is to be rotated using the depth-from-focus method. The largest possible aperture diameter is to be set so that the depth of field is very small. The effect of a small depth of field enables a more accurate estimation of the object distance.

Fine vergence movement is achieved by computing the cross-correlation of the direct and side images until the maximum coefficient is reached. This is tantamount to an image registration process where a window within the direct view is established as the reference.

The window within the direct image, extracted as a mask, is used to crosseorrelate with the side view along the horizontal or x direction 14. The location of the maximum coefficient generated by the cross-correlation computation process is used by the computer/processor 23 to estimate the fine vergence movement. Maximum correlation coefficient indicates that the most similar sub-image within the side view has been located.

The motor 21 then drives the rotatable stage 3 to the desired angular position. A new mirror-reflected view is acquired that replaces the previous one. This view is now converged more closely on the fixation point and hence achieves acquisition of an optimally overlapped stereo image pair.

Computation of the normalised cross-correlation coefficient of the stereo image pair is as follows:

where wl(x,y) is the window extracted from the direct image corresponding to 13 as shown in Fig. 4a It has a mean value denoted by wl. w2(x,y) is the window situated within the side image. The mean value of w2(x,y) is w2(x,y). wl slides across the side image searching for the most similar window defined by the displacement 8. Since the y axis of the two images are aligned, the search is one dimensional along the x-axis. Bigger window is chosen to ensure the reliability of the cross-correlation computation.

The maximum coefficient (Cmax) occurs when wl(x,y) is most similar to w2(x-o,y), where 6 is the vergence displacement as illustrated in Fig 4b. The sign of the vergence displacement 8 signifies whether to converge or to diverge. 8 is easily convertible to the vergence rotation angle û as illustrated in Fig 4b and is related by o = tan -1 o/l where I is the image distance which can be accurately determined by several methods, one of which is to use a position encoder to register the focusing lens position. Note that l is a variable due to the changes in image distance for different scenes. l is denoted by 1 in Fig 5.

The use of the compensating zoom lens is to correct the different in magnification caused the different in optical path lengths between the direct view and the side view The direct view is the image seen by the camera through the beam splitter. The side view is the image of the object reflected from the mirror, the lens then the beam splitter into the camera.

Hence, the side view has a longer optical path length than the direct view. As a result, the side image will appear smaller than the direct image.

Fig. 6 (a) and (b) show that the direct view has a optical path length L1 (or 0204) and the side view L2 (or 02050304); the entrance pupil E, the distance of the zoom lens from E is P (or 0205) and has a focal length f (power of the zoom lens). A technique for calculating the value f and P given other parameters such as L1 and Lz have been selected will be described Fig. 7 illustrates the optical path of the side image with the zoom lens incorporated.The visual angle seen by the side image is to be retained as large as the direct view for an object positioned at the point of fixation. Within a reasonably small visual angle and using raytracing, an object of height 1.0 unit positioned at the point of fixation is to sustain a visual angle u at the entrance pupil of the camera (inclusive of its lenses assembly) 1 (1) -Lr From simple geometrical optics: P (2) h=L11 1 then

Given that u' = u - hk (4) where k is the power of the zoom lens, ie. k = then

manipulating (5),

then

P has to be smaller than the baseline (b = 0203) and the thickness of the zoom lens assembly has to be considered.

We could also solve for P given k has been selected,

Note that minimum (Ll) is the standoff distance of the system and

From eqn (8) and (9)

L1 is related to b, the baseline and i as shown in Fig. 5 L1=b tan 2i Substituting (12) into (11)

Eqn (12) shows that P is a variable that changes with the vergence angle 2i and therefore a means of automatically controlling theses two variables can be achieved.

From the above description, the present invention allows the use of one single camera to acquire maximally overlapped stereo images that are optimal for stereoscopic processing.

Claims

1. The small standoff one-camera-stereo adaptor is an apparatus of and a method for acquiring maximum overlapped stereo images using only one video source, consisting of a beam splitter, a mirror, a compensating zoom lens, shutters, all enclosed in a light-protective casing attachable to the video source; the beam splitter is oriented at 45 degree to the optical axis of the video source such that the video source is able to capture scene along its optical axis and also scene that is at right angle to its optical axis; the mirror is positioned along the right angle path of the beam splitter separated by a distance; the compensating zoom lens is positioned in between the mirror and the beam splitter;

2.The small standoff one-camera- stereo adaptor as claimed in claim 1, the optical axes of the video source, the beam splitter, the mirror and the compensating zoom lens are all aligned on a same plane.

3. The small standoff one-camera - stereo adaptor as claimed in claim 2, the stereo image pair acquired correspond to the scene directly ahead of the video source along its optical axis and the scene reflected by the mirror, through the compensating zoom lens then again reflected by the beam splitter into the video source.

4. The small standoff one-camera - stereo adaptor as claimed in claim 3, the mirror is rotatable by an electrical motor controllable by a computer whereby the amount of rotation is computed by correlating the stereo images acquired.

5. The small standoff one-camera stereo adapter as claimed in claim 4, the compensating zoom lens can be engaged and disengaged from the optical path and when it is -engaged it is controlled by the computer and is coordinated with the rotation of the mirror such that the magnification factor for the stereo images will be the same.

5. The small standoff one-camera stereo adaptor as claimed in claim 4, the compensating zoom lens is also controlled by the computer and is coordinated with the rotation of the mirror such that the magnification factor for the stereo images is the same.

6. The small standoff one-camera - stereo adaptor as claimed in claim 5, the shutters are under computer control such that at any one time only one view will be seen by the video source.

Amendments to the claims have been filed as follows 1. The small standoff one-camera stereo adapter is an apparatus of and a method for acquiring maximum overlapped stereo images using only one video source; it consists of a beam splitter, a mirror, a compensating zoom lens, shutters, all enclosed in a light protective casing attachable to the video source; the beam splitter is oriented at 45 degrees to the optical axis of the video source such that it is able to allow scene positioned along the video source optical axis and also scene that is perpendicular to it to be captured by the video source; the mirror is positioned along the perpendicular optical path of the video source and the beam splitter separated with the beam splitter by a distance; the ComDensatinq zoom lens is positioned in between the mirror and the beam splitter; the optical axes of the video source, the beam splitter, the mirror and the compensating zoom lens are all aligned on a same plane.

2. The small standoff one-camera stereo adapter as claimed in claim 1, the stereo images acquired by the video source are stored in a computer where one of the stereo images corresponds to the scene seen by the video source along its optical axis through the beam splitter; and the other image being the scene reflected by the mirror, through the compensating zoom lens and reflected by the beam splitter into the video source.

3. The small standoff one-camera stereo adapter as claimed in claim 2, the mirror is rotated by an electrical actuator controllable by a computer where the amount of rotation is computed by correlating the stereo images to achieve maximum overlap.

4. The small standoff one-camera stereo adapter as claimed in claim 4, the shutters are under computer -control such that at any one time only one view will be seen by the video source.