WO2002019009A2

WO2002019009A2 - Omnidirectional imaging attachment

Info

Publication number: WO2002019009A2
Application number: PCT/GB2001/003892
Authority: WO
Inventors: Lee Scott Friend
Original assignee: Lee Scott Friend
Priority date: 2000-08-31
Filing date: 2001-08-31
Publication date: 2002-03-07
Also published as: AU2001284211A1; WO2002019009A3

Abstract

The invention relates to the field of capturing panospherical image information and according to one aspect, provides an adapter, or a camera, with means to select either an omnidirectional or a normal field of view. According to another aspect, both fields of view are provided. According to a third aspect, the apparatus for capturing omnidirectional information comprises a plurality of planar reflecting surfaces arranged, for example, in a pyramid structure, to reflect light from an annular section of the scene onto a common imaging plane.

Description

OMNI IRECTONAL IMAGING ATTACHMENT

This mvention relates generally to omnidirectional imaging of a panoramic scene. One aspect of the invention provides the option of a panospherical field of view as well as the normal field of view. Another aspect of the invention provides both panospherical and normal fields of view continuously. A further aspect of the invention relates to an omnidirectional image capturing device which employs planar reflecting surfaces. A yet further aspect relates to adapting a fisheye lens system to provide an optically scalable view. These different aspects of the invention can be used independently or in combination.

The first aspect of this invention can be used with an imaging device, such as a CCD camera to provide the option of a panospherical field of view, as well as the normal field of view and it can facilitate rapid selective viewing of a part of a panoramic scene to enable a user to view and preferably to zoom into the area of interest without optical distortion. It can be used, for example, in the field of surveillance, where it can greatly increase the amount of information captured by a camera without re-locating the camera. However, this aspect of the invention can be applied more broadly wherever the normal field of view of the lens system of a camera needs to be supplemented by omnidirectional image information.

The second aspect of the invention can be used to provide continuously both image information from a panospherical and normal fields of view. This is also useful in the field of surveillance, especially where continuous information needs to be recorded in a surveillance zone. It also enables a user to observe an area of interest and preferably zoom into the area in the normal field of view.

The third aspect of the mvention relates to the use of planar reflecting surfaces (in contrast to the use of wide angle lenses or mirrors, such as fisheye lenses or parabolic convex reflectors) for producing a panospherical field of view. This second aspect can be used broadly wherever there is a need to capture omnidirectional image information.

Reference herein to my co-pending GB Patent Applications is intended to include any of the following:-

GB 0006396.6 "Imaging Apparatus"

GB 0018017.4 "Imaging Apparatus" GB 0019850.7 "Imaging Apparatus"

GB 0021433.8 "Omnidirectional Imaging Apparatus"

GB 0023786.7 " Omnidirectional Imaging Apparatus "

References herein to a "panospherical field of view" should be taken to include a field of view extending over a hemisphere, besides a sphere itself, and any field of view which is major part of either such field of view. For example, a known fisheye or panoramic lens can provide a hemispherical field of view of a panoramic scene. Some parts of the hemispherical image may be very distorted due to optical effects near the edges of such wide angle lenses . When back-to-back fisheye lenses are used to provide a spherical field of view, this distortion is worse and there is also a need to seam the two hemispherical images together. Parabolic convex mirrors can be used to provide similar hemispherical and spherical fields of view. In some of my co-pending GB Patent Applications, I describe planar reflecting surfaces which can be used to achieve the same effects. Accordingly, any suitable means can be employed to capture omnidirectional information (in the first and second aspects of the invention).

Generally speaking, optical information captured over at least a major part of a hemisphere provides for 180°x 360° viewing, whereas that captured over the major part of a sphere is for 360°x 360° viewing.

Whilst embodiments of the invention can be described with reference to surveillance equipment, this is by way of example only, because the principles of the invention can be used by those skilled in the art in other fields of use.

In the field of surveillance, a video camera is usually located in a position where it can be panned, tilted and rotated in order to aim at a point in the surveillance zone where some activity needs to be observed. The camera is also fitted with variable power (zoom) so that, after scanning around the total zone of surveillance, to watch activity occurring anywhere in that zone, an operator can zoom in to observe the activity more closely. The scanning zone could cover, for example, a large car park whereby a camera scans around to watch for any suspicious activity and then, if such activity is seen in a smaller area of the zone, the camera is aimed at this area and zoomed in to observe the activity more closely (e.g. someone attacking a car door). Whilst such equipment has proved to be of considerable advantage to those involved with security, they have the disadvantage that, during a standby phase, the camera may be pointing in the wrong direction in order to observe initially some suspicious activity. Another problem involved with surveillance, is that permission is usually necessary before erecting a camera on a mast, so there is often a conflict of interest between those wanting more cameras for better security and those refusing permission to instal say another camera for reasons of privacy. This can be a problem where say one camera has only a very limited field of view and it is necessary to try to find a suitable location for another camera, besides the additional costs involved in its installation and operation.

The first aspect of the present invention seeks to solve these and other problems in the surveillance field, by providing apparatus which can, for example, be simply attached to an existing camera in order to improve its image capture. The use of the invention can substantially reduce wear on pan, tilt and zoom servomotors by reducing the required rate of scanning around the total zone of surveillance. For example, the apparatus may take the form of an adapter which can be fitted to the lens system of the existing camera. Alternatively, the camera can effectively incorporate the adapter, in which case it can be used as a new camera to replace an existing conventional camera. However, this first aspect of the invention has other advantages for 180° x 360° or 360° x 360° image gathering as will be more apparent from the following description of background art.

US-A-6002430 discloses a 360° image gathering system using back to back fisheye lenses attached to a camera with computer processed output of a navigable plane image. This appears to have been developed specifically for the Film Entertainment and Internet industry where constant 360° image gathering is required for constant image mapping and continuous interactive image navigation by a remote user. There is no facility for an undistorted optical enlargement of a selected area and no facility to retract plane mirrors or correcting optics.

US-A-6118474 discloses a separate camera mirror and parabolic catadioptric reflector system to achieve the separate use of Omnidirectional viewing . However it requires the camera to be moved rather than a retracting plane mirror or prism. The system is not compact and is not practical for outdoor use as it does not address the problems cause by rain. Further it does not show a capability of 360° image gathering on a vertical axis. Where some provision is made for optical zoom, this would merely amplify any optically reflected aberrations if an optical zoom were applied to the convex parabolic reflector shown in this reference. The problems of resolution for the purpose of enlargement in catadioptric reflector system employing parabolic, hyperbolic, or ellipsoidal reflectors is covered in an Internet publication entitled Analysis of a Defocus Blur, by Simon Baker in paper published on Columbia Universities website. US-A-6118474 attempts to resolve resolution problems by use of telecentric methods to filter out a substantial portion of principle rays that are not orthographically reflected it does not provide a perfect solution to defocus blur for the purpose of enlargement. However much this may assist in digital scaling, it does not resolve the ability of the system to produce an optical enlargement, because any enlargement of the camera's view would merely increase the size of the central area until one is only left with an enlarged image reflection of the camera lens itself, and its optical anomalies.

US-A-5185667 and US-A-5359363 advocate the use of Fisheye lenses to replace PTZ systems for camera navigation, both of these methods suffer from the inability to perform an optical scaling (zoom) in a selected area, nor is the method seen as an omnidirectional attachment.

Computer mapping of the resulting panospherical images is described in my copending GB Applications . Computer mapping from spherical or panospherical means to a plane image is also well described in the textbook Space Image Processing, e.g. where Julio Sanchez mentions that skilled workers at NASA have developed over 30 algorithms to translate a spherical image to a flat plane view. Sanchez describes Haversine Solutions and in a chapter entitled Map Projections, projecting a sphere into a developable surface, such as Cylindrical, conical and plane. Furthermore he refers to the mapping of cylindrical projections into regular, traverse and oblique formats which have been in common usage for producing world maps since 1569. In A Simplified Approach to Image Processing, Randy Crane describes computer-processing methods of spatial transform such as Affine and bilinear transforms that can remap images to the flat plane and preserve perspective. Further reference can be made to image spatial transformation techniques and scan line algorithms in Digital Image Warping by George Wolberg ISBN 080868944-7. The use of Cartesian look up tables to assist in real time processing is also described in my co-pending GB Applications. Reference is also made to US-A-5563650.

US-A-4012126 and US-A-484801 show hyperbolic reflectors which attempt to resolve Petzvtal curvatures introduced by such reflectors by specific lens groupings on a 360° horizontal and up to 180° vertical axis. The annular image recorded is intended for projection back through the assembly to artificially negate the distortion effects to the viewer on a spherical projection screen. There is no facility for an optical zoom or computer mapping nor does it provide for a 360° Vertical image gathering.

None of the above noted prior art teaches a satisfactory method of tracking a moving object in a panoramic scene (or surveillance zone), since there may be some distortion in parts of the panoramic image, which may mean that the object cannot be clearly seen in these parts, or there may be other limitations on the nature or quality of the image information available.

The first aspect of the invention solves these prior art problems of improving image quality by enabling an operator to use an omnidirectional imaging device to track or to detect visually or through computer analysis, targets moving within the entire horizontal and vertical field of view. The operator may then switch to standard, variable power (zoom) or telephoto lens to make an accurate undistorted identification. Hence, the invention can be embodied so that an omnidirectional lens or mirror need not be removed to resume normal lens and camera function. It can also be employed to resolve problems of poor image quality of omnidirectional devices and the inability to scale or zoom either optically or digitally on a selected area without granularity or without exaggerating severe spherical or chromatic or other optical aberrations inherent within such systems. The capability to scale or zoom or enlarge is important in relation to surveillance, security and defence applications. However in Sport or Film and Entertainment applications it would save the use of a separate camera to provide close up, tracking or zoom enlargements.

The first aspect of the invention provides apparatus for use with a camera having a variable power (zoom) lens system with a normal (limited) field of view. The apparatus can provide an optional panospherical field of view, and it can be embodied either as an attachment for the camera, or as part of the camera itself. The apparatus includes:

(a) panospherical image capturing means for capturing omnidirectional information from around a panoramic scene in an omnidirectional viewing mode;

(b) optical means for capturing image information from part of the panoramic scene in a normal viewing mode; and (c) selecting means for selecting either the omnidirectional, or the normal viewing mode, by causing light from either the panospherical means (a), or the optical means (b) to be incident on an image plane of the camera.

An advantage of the latter apparatus is that the optical device or camera can be simply switched from omnidirectional to normal viewing, or vice versa, where part of a scene can be viewed (preferably by zoom) without distortion introduced by omnidirectional image gathering. This is made possible by having both the panospherical and the optical image capturing means available on the same camera so that, for example, a user can switch from one to the other by causing various means to move, such as a mirror which (in one position) reflects an omnidirectional image from say a fisheye lens onto the image plane of the camera, or (in another position) enables the image plane from the camera to receive an image in a normal field of view of a standard, or preferably standard variable power (zoom) lens system. Thus, the omnidirectional image can provide continuous panoramic scene viewing, (where some spherical distortion may exist but is tolerable), whilst the normal field of view provides a non- distorted image in an area of interest within the zone of the panoramic scene.

The second aspect of the present invention provides apparatus for providing a panospherical field of view and a normal field of view, the apparatus being provided either in the form of an adapter, or as part of a camera, the apparatus including:

(a) panospherical image capturing means for capturing omnidirectional information from around a panoramic scene and for supplying it to a first image plane;

(b) optical means for capturing image information from part of the panoramic scene within a normal field of view and having a second imaging plane;

whereby both the omnidirectional information, as well as the information from the normal field of view is available from both the first and second imaging planes.

According to the second aspect of the invention, these omnidirectional and normal fields of view are available at the same time. In this case there is no need for moveable means to divert images from one lens system or another to cast either an omnidirectional image or a normal image onto a single image plane of a camera. Instead, there are two image planes respectively for omnidirectional and normal fields of view. A practical example of this can employ a divided screen display where (e.g.) the full frame shows a panospherical image in a flat viewing plane, and an inset vignette shows the normal field of view (e.g. zoom).

Apart from increasing the field of view, both the first and second aspects of the invention can provide for a fail safe system should there be any servo motor failure in cameras directed remotely by a mechanical Pan Tilt and Zoom (PTZ) system. Should a partial or full failure in any of the servo guidance motors or a communications failure occur the operator can still observe through the omnioptics the complete field of view if the PTZ system leaves the camera looking the wrong way. Although this still has limited scaling, it is very useful in a security situation where cameras are often controlled remotely via wireless links, and jamming of camera guidance communications has been known to occur. The cameras internal electronics can be programmed to switch to omnidirectional automatically when the PTZ guidance signals are lost. As a panoramic scene is being continuously monitored by the apparatus, there is less need to use the PTZ servo motors to continuously scan the scene, thereby reducing servo motor wear.

Accordmg to the third aspect of the invention, the panospherical imaging device comprises a plurality of planar reflecting surfaces arranged to reflect light from a panoramic scene into corresponding light guiding means; optical means for combhiing image information received from the light guiding means so as to provide a composite image of at least an annular section of the panoramic scene.

The planar surfaces are preferably sides of a pyramid, the sides being triangular, quadrilateral or other shapes.

Preferably, each light guiding means is arranged to view an area which not only includes the respective part of the planar reflecting surface, but also a portion of adjacent reflecting surfaces, whereby overlapping optical image information is captured for the purposes of joining images from the adjacent planar surfaces when producing the composite image.

Embodiments of the invention in general can also resolve problems caused by environmental conditions such as rain and dust and also lens flare. For example, the means to provide a large field of view result in optics having a large exposed and complex portal areas which require protection. Without some form of environmental protection these devices would be of no use. Whilst the portals can be protected to some degree by using known chemical compounds to assist in water removal, they could be rendered useless in heavy downpours or snow or in desert conditions. In preferred embodiments of these aspects of the invention, means are provided for the camera devices to be continually positively pressurised by low-pressure air, e.g. from a compressor which will also supply means to treat the air for temperature and humidity. This prevents dust or water ingress; prevents the build up condensation; avoids flexing of the reflective components; and reduces the complexity of seals.

Embodiments of the invention can also convert e.g. a CCD camera to a pure 360° by 360° motion detection system by utilising known methods of computer image analysis. For example, the system described in the publication Handbook of Image and Video Processing by Al Bovik detailed in chapter 3.10 entitled Motion Detection and Estimation. This removes the requirement for a separate device and lens system.

The omnidirectional means, i.e. the lens systems or reflector systems can be mounted on the top and bottom of a housing or in a side by side configuration, to suit individual operational conditions. In an embodiment of the invention, this can be achieved by a locking ring on the adapter mounting plate and by remounting the front housing containing the frontal protection window and wiper assembly.

Preferably, the panospherical image capturing means includes any one or more of fish eye lenses, convex reflectors, and a plurality of planar reflectors. A suitable omnidirectional optical means is a fish-eye lens and this can be used, back-to-back, as shown in US-A-4256373, but other lens assemblies of 180° field of view +/- may be utilised instead. US-A-5631778 is an example of a catadioptric system that attempts to achieve wide angles of view utilising Cassagrain effect principles and is another suitable fitting for the invention. Other suitable lens include (but not exclusively) The Nikon Nikkor 6.3 mm f2.8, The Pentax Fisheye 17mm F4 or The Ziess F-Distagon 16mm F2.8. Further suitable examples can be see in Merte USP 2126126 Muller USP 4647161 or designs by Myamoto Josa. The Optical assembly described in US-A- 5627675 could also be modified for use.

The panospherical image capturing means may comprise a fish eye lens and a plurality of secondary lenses for focussing light from the fish eye lens on to the image plane. At least two of the secondary lenses may be selectively moveable to vary the magnification of information captured by the panospherical image capturing means. This is an important feature of the present invention, and may be provided separately. Accordingly, the present invention also provides omnidirectional imaging apparatus for capturing omnidirectional information from around a panoramic scene and for supplying it to an image plane, the apparatus comprising a fish eye lens and a plurality of secondary lenses for focussing light from the fish eye lens on to the image plane, at least two of the secondary lenses being selectively moveable to vary the magnification of captured information.

Any wide angle lens, dioptric or catadioptric or cataptric especially those that reduce the length or depth of the back workings are preferable as the ideal embodiment attempts to bring the lens assemblies as close together as possible to reduce any blind area in the centre of the field of view.

My co pending International application No. PCT/GB01/01115 resolves the problems of lens flare or diffraction from the large lens or reflector surfaces by the use of FOLED or TOLED or transparent LCD coatings, on portal covers using dynamic shading and this can be a feature of embodiments of the invention.

Embodiments of the invention can be employed to provide for a retractable lens relay or correction system to enable stock cameras and lenses to be used in conjunction. Where such cameras have a macro or close focussing capability the retractable relay or correction lens may be replaced by a ground glass screen to project the reflected scenes.

Retracting prisms as well as retracting plane mirrors can be used to select either the normal or omnidirectional viewing mode and the choice will suit varying optical embodiments, or a more robust assembly, but use of prisms increases the size of the back workings requiring more space for retraction.

The use of a single retracting beam splitter rather than two mirrors is also an option in a more simplified approach, but it is a larger solid component and needs more space when retracted, besides creating a larger distance between the two lenses or reflectors. A dual lens or reflector configuration can include a mechamsm to cover an aperture on the opposite side of the retracted component, thereby preventing stray light from entering the assembly during normal lens function. Some embodiments of the invention use drive means for moving the image plane of the camera, so that (in one position) the image plane receives a focussed omnidirectional image, and (in another position), the image plane receives a focussed image from the normal field of view, (preferably from a variable power lens system). This can greatly simplify the mechanisms required for moving any of the optical means when the same image plane is shared by both a panospherical lens and the standard (e.g. variable power) lens systems.

Preferably, in the first aspect of the invention, the selecting means includes a beam splitter and inhibiting means which enable the beam splitter to reflect light or to pass light to the image plane of the camera, depending on whether the omnidirectional or the normal viewing mode is selected.

For general environmental protection the devices utilise a combination of chemical water repellent coatings, wiper blades and centrifugal methods. On the fisheye arrangements no wipers are shown. Preferred arrangements utilise centrifugal and water repellent coatings only, as the use of wipers restricts the field of view. Wipers shown on all omni portal coverings are as optional for severe environmental conditions.

A front portal screen or window can have varying angles from 90° to 45° with wiper fitting to suit varying operating condition. A preferred embodiment has a glass window parallel to the image plane. In heavy rain or certain covert applications where stray reflections from the glass may be undesirable a 45° embodiment may be preferred.

All mirrors and reflectors are first surface reflectors, e.g. preferably using vacuum deposited materials and coatings.

For use in surveillance of a zone in which the apparatus captures image information from a panoramic scene, the invention comprises an imaging device having a variable power lens system, means for mounting the imaging device on an axis of rotation so that it can be rotated to a selected azimuthal position, whereby the variable power lens system can capture image information from an area in the panoramic scene within its field of view, the imaging device further including panospherical image capturing means for capturing an omnidirectional image from said scene, and selecting means which are operable so as to capture either image information from the variable power lens system of the imaging device, or from the panospherical image capturing means, or both, depending on a selected mode of operation. The latter apparatus preferably includes image processing and displaying means for processing the image information from the camera and the panospherical image capturing means to enable respective images to be displayed on a flat viewing plane.

An advantage of the latter apparatus is that whichever radial direction the imaging device is pointed to observe and zoom into the detail in the normal field of view, the panospherical image capturing means will continuously capture omnidirectional information around the scene. This could be an all round panoramic image or a part of the scene. Alternatively, or in addition, the azimuthal position required for viewing a particular area in the zone can be determined with respect to the panoramic scene seen by the panospherical image capturing means. For example, where the panospherical image is in the form of a disc or discs, and where activity is seen in a radial position around the disc(s), the image display means can provide azimuthal position data and/or drive signals to cause drive means to pan the imaging device about the axis so that it looks in the right direction for tilting and zooming onto the activity. This could be done by observing the area on the disc as it rotates to a reference position, i.e to pan the imaging device correspondingly.

Another aspect of the invention relates to omnidirectional imaging apparatus which employs optical imaging directly onto a CCD imaging surface or other image recording device. The invention significantly reduces, or substantially eliminates complex and bulky storage and processing of image data signals when mapping panoramic optical image information onto a flat plane as a composite image of a panoramic scene. Embodiments of the invention enable full 360° x 360° image capture where zones of the spherical field of view are separately and directly imaged onto devices such as CCDs with flat (or curved) image planes with substantially no spherical distortion.

Thus, this aspect of the present invention seeks to reduce as far as possible the use of image transformation for distortion correction, as bad data input can be irretrievably distorted. For example, it employs optical and/or mechanical means for correcting spherical distortion to montage or to compose a viewable image directly on a screen. Thus, the image of the scene is optically gathered and the image components can be assembled with far less image processing. Whilst some mapping can be used to improve image quality, a substantial portion of this prior art process is replaced by the optical corrections made by the invention. This reduces the mapping burden especially with high resolution CCD imaging. Moreover, well known processing techniques can be used to join the seams of image components of the montage, in order to provide a continuous image (where this is required), although this is not essential in certain applications (such as surveillance). In any event, the invention can be employed to significantly reduce or substantially eliminate the complex and bulky prior art process of mapping image data signals, derived from wide angle optics such as fish-eye lenses, onto a flat image plane, which is used in the prior art to compensate for spherical optical image distortions, and to provide an easily navigable image. Processing of video signals to join the seams is required where, for example, the viewed image needs to be of high quality as a continuous montage. In surveillance equipment, such picture quality is not required and each image component can contain adjacent edge strips of similar image information. If seaming is required, various known prior art video processing techniques can be used, such as introducing Gaussian blur, or signal blending to soften the joined edges. Whilst this involves computer processing of the digital image data, the task of applying Gaussian blur along a constant given track, threshold or matrix is negligible compared with mapping. Other techniques of photo montage, such as those described in the internet paper entitled "An introduction to image mosaicing" by Sevket Gumustekin, July 1999 and Digital Image Warping by George Wolberg, and using constructing tools such as Apple's Quicktime NR (RTM), which have been well known in the computer field for a very long time, can be employed for accurate seaming of the image components, e.g. to correct for misalignment. Moreover, many more simple techniques become realistic with the way in which the invention can be used initially to correct, by optical means, for spherical distortions. Some means of calibrating the seams could also be used to montage components accurately into a continuous flat image so that seams are not then required.

In cases where some mapping is carried out, the invention is used to remove a substantial proportion of the spherical distortion introduced by (e.g.) fish eye lenses, the remainder being dealt with by a less complex mapping algorithm. In this sense, the invention can be used to improve the prior art omnidirectional cameras which depend on the use of mapping techniques.

According to this aspect of the invention, omnidirectional imaging apparatus comprises: a plurality of optical imaging devices having optical axes which are directed to respective fields of view for forming images from a hemisphere or sphere surrounding a panoramic scene; each said device having a lens and an imaging surface for receiving optical images from the lens, each said device also converting the optical images into video signals; and processing means for processing the video signals and for reproducing components of the scene,, as a montage, in a navigable viewing plane; characterised in that image distortion reducing means are provided, between each lens and a respective zone on the imaging surface, so as to conduct light from different areas of the optical image produced by each lens to respective zones on said imaging surface, said zones being selected so that any image distortion, which could otherwise be caused by using fisheye lenses or conical reflectors is avoided, or substantially eliminated.

The present invention also provides omnidirectional stereoscopic imaging apparatus comprising two imaging systems displaced on the same optical axis, one of which systems is larger than the other by a factor depending on a predetermined interocular distance. Preferably, the imaging systems include two fisheye lens systems, one of which is larger than the other by a size factor determined by the amount of interocular distance required. Preferably, each lens system is arranged to focus an image of a scene on to respective image sensing means.

The present invention further provides omnidirectional imaging apparatus comprising first optical means having an annular field of view, second optical means having a field of view extending over a segment of a hemisphere and means for optically combining light received from both fields of view for the purpose of generating a video signal output, the first and second optical means being co-axially disposed so that their combined fields of view are hemispherical. Preferably, the first optical means comprises a first convex reflector and a first planar or concave reflector for providing the annular field of view; and wherein the second optical means comprises a concave reflector and a second convex reflector for reflecting light from a scene to provide the segmental field of view. The apparatus preferably includes an aperture or lens for directing light from the segmental field through the aperture in the apex of the convex reflector together with the light from the annular field. Preferably, each reflector and or lens is arranged on a common optical axis. Preferably, the optical combining means comprises any one of a prismatic device, light guide means, reflecting means, or any combinations thereof, for directing light from the aperture in the apex of the convex reflector to an imaging surface of a device for converting optical images into video signals. The apparatus may include means for attaching the apparatus to a camera to provide omnidirectional imaging. In the case, convex reflectors are preferably arranged back to back and a laterally extending member provides the camera attachment. Embodiments of the invention will now be described with reference to some of the accompanying schematic drawings, in which:

Figs 1 and 2 are respectively elevational and plan views of a prior art system in which a video camera views an area A in a surveillance zone Z,

Fig. 3a is an elevational view of a camera modified in accordance with an embodiment of the invention, the camera being mounted on a mast in the zone Z for 360° horizontal and up to 180° vertical movement;

Fig 3b is an elevational view of a similar camera to Fig. 3a for substantially 360° horizontal x 360° vertical viewing;

Fig. 4 is a sectional elevation view through a conventional camera and an adapter for providing an optional omnidirectional view for 360° horizontal x 180° vertical viewing;

Fig. 5 is a cross-section in plan view of a similar embodiment but with two fish eye lenses for 360° x 360° vertical viewing with side by side hemispherical views;

Fig. 6 is an embodiment similar to Fig. 5 but intended for 360° horizontal x 360° vertical viewing top and bottom hemispherical views;

Fig. 7 is another similar embodiment which uses prisms;

Fig. 8 is a sectional elevation through an embodiment which employs a primary convex parabolic reflecting surface for providing an omnidirectional view and a secondary plane or concave secondary reflector for reflecting light through an aperture in the convex reflector to the mirror system;

Fig. 9 shows a similar arrangement with top and bottom primary convex hyperbolic reflectors and secondary planar or curved reflectors for 360° vertical axis viewing;

Fig. 10 shows another embodiment for 360° vertical axis viewing with top and bottom omnidirectional devices each having a primary convex reflector and a lens system;

Fig. 11 shows a simplified embodiment where a single convex reflector reflects light directly onto a moveable mirror;

Fig. 12 shows an embodiment where the lens system (la) has been separated from the CCD imaging part (lb) of the CCD device (la, lb) normally found together in the camera 1;

Figs. 13 to 18 show embodiments with generally similar features to those of corresponding preceding embodiments except that the lens system (la) is shown separated from the CCD imaging part (lb) of the CCD device normally used in combination (la, lb);

Fig. 19 shows a first simplified embodiment of an omnidirectional imaging device which employs planar reflecting surfaces (and no fisheye lens, nor convex paraboloid reflectors);

Figs. 20a and 20b are respectively cross-sectional elevation and plan views of a similar device using planar reflectors but also having Goertz prisms;

Figs. 20c to 20e show respectively the scene capture analysis, the recorded input data and the plane output transformation;

Fig. 21a shows a similar embodiment but with a lens assembly for receiving light from an upper segment of a panoramic scene via the upper lens system to improve blind spot viewing;

Figs. 21b and 21c show similar embodiments to Fig. 21a with a variable power lens insert to selectively enlarge part of the panoramic scene;

Figs. 22 to 24 show embodiments including the planar mirror arrangement for omnidirectional viewing;

Fig. 25 shows the use of various relay lens arrangements;

Figs. 26a and 26b show arrangements with moveable CCD devices;

Figs. 27 to 30 show modular arrangements of the apparatus with variable power lens removed; Figs. 31 and 32 show an embodiment having a worm drive for advancing and withdrawing a CCD so as to provide focussing in the omnidirectional and normal viewing modes;

Fig. 33 shows an embodiment where two cameras are used, one for omnidirectional viewing and receiving light from a solid N-shaped reflector, the other camera being used for the normal field of view, giving continuous viewing from all lenses/reflectors;

Fig. 34 shows an embodiment which employs a stationary beam splitter;

Figs. 35 and 36 illustrate the selective masking of one of the panospherical and normal images incident on a beam splitter;

Figs. 37 to 40 show further embodiments which employ a stationary beam splitter;

Fig. 41 to 43 show embodiments of a spherical six lens imaging device;

Figs. 44 to 47 show embodiments in which a standard, preferably variable power, lens system captures a central zone of a panospherical image captured by a fisheye lens system;

Figs 48 to 50 show embodiments of a stereoscopic omnidirectional imaging system;

Figs. 51 and 52 illustrate arrangements for providing variable power or zoom to a fisheye lens system;

Fig 53 is a section though an omnidirectional imaging apparatus which employs convex reflecting surfaces;

Fig. 54 shows an internal prism arrangement within the convex reflecting surfaces of Fig. 53 for directing images captured by the apparatus to an attachment to a camera;

Fig. 55 shows an alternative arrangement for directing the captured images to the camera; and

Fig 56 shows a front view of part of the apparatus of Fig. 53 attached to a camera. The drawings are not to scale and are provided only to show principles of the invention. Focal lengths or focal plane distances from lenses or reflectors are not shown in correct scale and reference is made to the prior art as mentioned herein and elsewhere in order to provide the details for the optical constructions. For example, US-A-4012126 shows more details of an arrangement which is similar to that shown in Fig. 8, except that the prior art reference does not deal with the processing of image signals from a CCD in order to derive a montage of different segments of panoramic scene in a flat viewing plane. Instead, this US reference deals with a device simply used for collecting light from an omnidirectional field of view, passing it through a lens system and projecting it on a hemispherical viewing plane, whereby spherical distortion is not a problem. (It will also be noted that the transparent outer cover described and illustrated in US-A-4012126 should have a complimentary symmetrical shape in order to avoid introducing effects due to coma or a stigmatism).

Most lens assemblies are not detailed, references can be made to Modern Optical Engineering by Warren Smith, Modern Lens Design by the same author, or any of the other books listed above. Further reference can be found in Visual Instrumentation by Pantaziz Mouroulis or Practical Computer aided lens design by Gregory Hallock Smith. Such lens configurations would be chosen to minimise the Petzval curvature introduced by the reflectors in the catadioptric embodiments. Fisheye lens elements are reverse telephoto strongly negative meniscus lenses and include the use of lenses described in US-A-4,647, 161 , US-A-4,256,373 and a design from Miyamoto Josa in 1964. These lenses are shown for diagrammatic purposes only and are not exclusive to the present invention. Biconcave, anamorphic or relay lenses can be located as appropriate to relay ray paths from a negative meniscus lens to a practical image plane.

Lens, mirror and prisms mountings are not fully detailed; references can be found in publications such as Design and Mounting Of Prisms and Small Mirrors in Optical Instruments by Paul Yader. Seals and bearings and other supplementary fittings are also not described. Where mirrors are shown, they may be replaced by prismatic reflectors, the variance of the angle may differ and the form of the mirror may be such as to improve the ray path transmissions. Diaphragms, irises, stops and baffles are, unless otherwise specified, not drawn or described in detail, and would be positioned appropriate to the final specification.

Turning now to the drawings, Figs. 1 and 2 show a prior art system, where a conventional TV camera C is mounted on a post P, usually high enough to enable the camera to look down on to an area A within a much larger surveillance zone Z (shown circular in Fig. 2). Whilst the camera C can be tilted and panned so that it can look at any part of the zone Z, it must be controlled by the security personnel who need to use initiative in moving the camera to observe activity somewhere in zone Z.

Camera C is replaced by an imaging device having an omnidirectional field of view (e.g. a camera with a fisheye lens which looks downwardly on zone Z), whilst this would capture image information continually from zone Z, it is difficult to spot some activity occurring in a small area of this zone. The fisheye lens and CCD combination provides image information, for example, in the form of a circular image, which needs to be processed, by known means, to assist in removing spherical distortion before assembling a flat image from different parts of the panoramic scene (see for example my co-pending UK Patent Applications). Therefore, such an omnidirectional imaging device needs to be used in conjunction with a conventional camera, which can be aimed and zoomed onto a much smaller selected area within the zone. However, no means are then available to aid the conventional camera to point in the right direction when some activity is observed in a particular area in the zone and the conventional camera C independently since it will often be looking in the wrong direction. There is also the disadvantage of cost if both the conventional camera system, as well as an omnidirectional imaging unit are used for surveillance of the same zone.

Fig. 3a shows a camera or imaging device 1 , modified in accordance with the invention by having a demountable adapter 2 which provides either a normal limited field of view N, or an omnidirectional field of view O. The drawing is schematic because the camera 1 would normally be tilted downwardly so that the normal field of view will see an area A on the "floor" of the zone Z, and the omnidirectional field of view will see the zone Z. This will be more apparent with regard to the following more detailed description of Fig. 4. Figure 3b shows a similar camera with "back to back" omnidirectional fields of view O and O' to provide substantially 360° horizontal by 360° vertical viewing.

Referring to Fig. 4, this shows the conventional CCD camera or imaging device 1 with the demountable adapter 2 attached thereto by a screw coupling 4. Camera 1 has CCD imaging means la, lb, which includes a standard lens system la with a normal field of view, preferably having a zoom facility (and optionally a wide angle view) and a CCD device lb which converts optical image information into video signals. The "normal field of view" is more limited than that of the omnidirectional field of view. By suitably processing digital image signals from the CCD device lb, the normal field of view can be observed on a flat viewing image plane (on the screen of a monitor). As there is reduced or little distortion of the image, it can be viewed with a standard (or non-standard) aspect ratio. However, the fisheye lens 20 will introduce spherical distortion. While the omnidirectional field of view can also be observed on a flat image plane of the monitor, the initial image from the fisheye lens may be disc shaped (since it circumferentially depicts a panoramic scene) . This needs to be developed first into e.g. a cylindrical or annular image where spherical distortion is reduced as far as possible. This annular image can then be developed into a continuous navigable strip, for example by the use of scan line algorithms, spatial look up tables and image registration techniques, whereby parts of the strip image can then be montaged into a suitable aspect ratio for viewing on the monitor.

The lens system la focuses image information entering window 3 (along axis a-a shown in chain and dot line) onto the sensitive screen of the CCD lb, for generating the video signals, which are processed (as known in the art), to show (on the monitor) the image from the normal field of view. The lens and CCD combination la, lb can also receive light from the omnidirectional field of view of the fisheye lens system 20 and mirror 7 , whereby video signals are processed to reduce spherical perspective distortion before viewing the panoramic scene on the monitor. The camera can be used to select either a normal field of view, or an omnidirectional field of view by tilting the mirror 7 as will now be described in more detail below.

The adapter 2 comprises a planar mirror 7, mounted on an opaque backing plate 8, which can be tilted about pivot 9 so that it can be positioned either at 45° with respect to axis a-a (as shown in the drawing), or it can be folded down into a storage position (not shown) where the leading edge 7e of the mirror and backing plate abuts a recess 14 in the corner of housing 15. When the mirror 7 is in the folded or stored position, the barrel 16 is clear, device 17 also then being in a retracted position (see below). When the barrel 16 is clear, light enters window 3 and passes unimpeded towards the lens system la of the lens/CCD combination la, lb. (As the barrel 16 provides a rather elongated entry port to the CCD device la, lb, it can restrict the normal field of view, but in embodiments described below, the lens system la is moved along the axis a-a so as to occupy position, for example, between window 3 and mirror 7.)

Window 3 is provided with a wiper blade 6, driven by motor 5 in housing 5a in order to provide a clear view in rainy weather. Mirror 7 and plate 8 are fast with pulley 10, which is connected by belt 11 to pulley 12, which is driven by motor 13 in order to flip mirror 7 into its omnidirectional or stored position. Other drive means can be employed such as cogs and wheels, etc. , or this can also be accomplished by some mechanical linkage operated by hand.

Device 17 can be any one or more of (i) a relay or correcting lens system; (ii) a ground glass screen; or (iii) a CCD device. Device 17 can be raised as shown in Fig. 4 (across the barrel 16) or lowered into a storage position within housing 15, by any suitable drive means, such as rack and pinion 17, 18 which includes a belt drive from a motor to the toothed pinion.

In the omnidirectional mode, light from a panoramic scene enters transparent dome 24 over a wide angle (as indicated by arrows 21a,21b) and is received by the fisheye lens system 20 whereby light from the lens system is incident on mirror 7 and thereby reflected onto device 17. As indicated above, this can be a relay or correcting lens system so that the omnidirectional image information is correctly focussed onto the lens system la of the lens and CCD combination la, lb. Alternatively, if device 17 is a ground glass screen, then a focussed image is received on this screen, whereby lens system la can focus an image of the screen on the CCD surface of lb for the same purpose. Alternatively, device 17 may be a CCD which receives a focussed image from mirror 7. Clearly, the choice of device 17 will depend on the design of the optics and the application. For example, the prior art mentioned herein can be used to design suitable optics within the concepts of the invention.

The apparatus is equipped with means (not shown) for providing a stream of heated dried pressurised air via tube 22 to the chamber 23 in which the lens system 20 is mounted. This not only prevents dust or water ingress, but also deals with condensation and prevents movement of optical components (due to steady temperature control) which can otherwise flex and cause defocusing. It will also be appreciated that the various components of the adapter are connected together with seals to keep out dust and rain, but these need not be so complex when pressurised air is continuously pumped into the adapter interior, since this will keep dust and moisture out of the system. As shown in Fig. 4, the omnidirectional image is that which is obtained by looking downwardly on the panoramic scene. Hemispherically shaped transparent dome 24 is designed to avoid flare and optical aberration, so that light from the panoramic scene passes through the shield 24 without any problem. The dome 24 is coupled, via circular rack and pinion 25, to motor 26 so that the dome 24 can be spun to throw ff rain by centrifugal force.

Where the lens system la of the camera 1 includes a zoom facility, this can be used, in the normal field of view, to zoom into part of the area seen by the camera. In the omnidirectional field of view, if the CCD device lb is of high resolution, parts of the panoramic image can be inspected more closely by scaling techniques used when processing the digital signals, so as to enlarge a portion of the image on the CCD. These digital processing techniques are well known in the art. Moreover, as described below with reference to Fig. 5, where two fisheye lens systems are used back-to-back in order to provide 360° x 360° viewing, known processing techniques can be used for seaming together images from the upper and lower (or side by side) hemispherical fields of view.

Instead of providing CCD devices and video signals which are processed for display on a monitor, the camera can be modified for use with film, e.g. to produce either still or moving images, in which case known film transport mechanisms are employed.

Fig. 5 shows an embodiment in which similar components have been used, the main difference being the inclusion of another fisheye lens assembly 20a, the former fisheye lens assembly being shown as 20b. This arrangement is intended to provide two hemispherical fields of view, one on either side of (e.g.) a vertical mast. The adapter 2 also comprises two mirrors 7a,7b, which can be pivoted towards each other, in order to join at an apex on the optical axis of barrel 16 and lens/CCD device la, lb. Also devices 17a, 17b (similar to any of those described above with regard to device 17) can be moved out of storage positions in housings 15a, 15b, so as to meet on the optical axis (as shown) where they act as relay lenses, ground glass screens, or CCD devices according to requirements. Panospherical image information in the form of video signals can be processed by known means in order to transform images to remove spherical distortions and also to provide seams between the hemispherical images whereby a spherical image can be constructed from the optical information captured by both fisheye lens systems. When used in the normal field of view, the mirrors 7a,7b are tilted into their storage positions (similar to that shown in Fig. 4) and devices 17a, 17b are retracted into their housings 15a, 15b. The lens/CCD combination la, lb then has an unrestricted view of the window 3.

Fig. 6 is an embodiment with generally similar features to that of Fig. 5 except that this arrangement is intended for use when viewing upper and lower hemispheres of a panoramic scene. A motor 5 provides a drive for a wiper blade 6 for weather protection.

Fig. 7 shows an embodiment similar to that of Fig. 5, except that prisms 30a,30b replace planar reflector mirrors 7a,7b. Whilst prisms provide improved reflections, more space is required within the cylindrical housing 2a,2b to accommodate movement when the prisms are pivoted from the positions shown in the drawing, to storage positions. In other words, the side housings 2a,2b need to be longer.

Fig. 8 illustrates an adapter which is based on using a panospherical image capturing system similar to that disclosed in US0-A-4012126 (Rosendahl). This replaces the fisheye lens and it is capable of capturing image information over a very wide angle of view. This panospherical image capturing means comprises hyperbolic mirrors 30,31 arranged opposite one another and in confocal relationship as explained in US-A- 4012126. The primary and convex hyperbolic mirror 30 reflects light from the panoramic scene onto the concave hyperbolic surface 31, which in turn reflects light into a lens assembly 32, which is also described in detail in US-A-4012126. This lens system provides a strong positive contribution to Petzval curvature of the system which assists in correcting image aberrations and provides correction for axial colour and colour magnification. However, this prior art system employs a camera or projector positioned to receive light directly from the lens assembly 32, since it does not deal with CCD imaging and the processing of video signals to provide transformations to remove spherical distortions. Instead, Rosendahl records an annular image intended for projection back through the lens assembly 32 artificially to negate distortion effects to the viewer on a spherical projection screen. There is no facility for optical zoom or computer mapping, nor provision for 360° degree vertical image gathering.

One of the problems faced by the invention is to provide panospherical images which are transformed into flat viewing planes with reduced spherical distortions and this is not addressed by Rosendahl.

Referring again to Fig. 8, convex parabolic primary and secondary reflectors may also be used, depending on the intended operational parameters. A transparent cover 33 has outer and inner surfaces formed symmetrically with respect to the near focal point of the primary mirror 30 to allow primary rays to pass normally through the surfaces of the cover 33 without introducing a stigmatism or coma. A central post may be employed to negate internal reflections. The concave secondary reflector 31 may also, in some applications, be a plane reflector (see for example, my co-pending GB Patent Applications). A wiper blade 34 can be moved into contact with the surface of the transparent cover 33, so that, when the cover is rotated by drive system 25,26, the blade clears raindrops. Alternatively, the cover may be spun so as to remove rain by centrifugal force.

As an alternative to using the teachings in US-A-4012126 (Rosendahl) use can be made of the lens groupings disclosed in US-A-44844801 (Cox).

Fig. 9 shows an embodiment based on that shown in Fig. 8, where two of the panospherical imaging devices 33a,33b, are provided for 360° x 360° viewing. These are arranged on opposite sides of the barrel 16 of the adapter.

In Fig. 8, mirror 8 can be flipped up or down, (by the drive system including motor 13) between the position shown in the drawing and a storage position where the barrel 16 is clear. Device 17 can also be moved up into the position shown, or retracted into a storage position. In Fig. 9, planar mirrors 7a,7b, can be rotated into the positions shown in the drawing, where they meet at an apex, or rotated into storage positions (not shown). Likewise, devices 17a, 17b, can be extended or retracted.

The embodiment of Fig. 10 is of much simpler construction since it dispenses with a secondary reflector and can use a more simple lens system. In this embodiment, the primary convex reflectors 36a,36b reflect light from the panoramic scene directly onto the respective lens systems 37a,37b, which directs light onto respective mirrors 7a,7b. These mirrors 7a, 7b, can be moved into the position shown, or retracted into storage positions as before. Also, device 17 can be moved into the position shown, or retracted into a storage position.

Fig. 11 shows a more simplified arrangement for 180° x 360° image capture and it employs a convex reflector 36 which reflects light directly onto mirror 7, which in turn reflects light onto device 17. Mirror 7 is movable from the position shown in the drawing, to a storage position, and device 17 can be retracted into housing 15 by drive system 18, in order to provide the normal field of view. Weather protection is provided by a transparent dome section which is designed so as to follow the symmetry of the reflector 36 thereby reducing coma and/or a stigmatism (similar to that described above with reference to Figs. 8 and 9). In the embodiment of Fig. 12, the components are similar to those shown in the embodiment of Fig. 4 except for the main difference of separating the lens system la from the CCD device lb. The lens system la preferably includes variable power and is situated between the mirror 7 and the window 3. This is closer to the inlet aperture of the adapter and hence provides a better "normal" field of view. In the position shown in the drawing, the mirror 7 is flipped up so as to reflect light from the fisheye lens system 20 onto the CCD device lb. In this case, device 17 (similar to that described above) is shown in its storage position, since it may or may not be required in either the normal or omnidirectional mode of operation. For example, it may be a relay or correcting lens which is raised across the barrel 16 in order to correct for focussing in the normal field of view, when mirror 7 is in its storage position. Alternatively, it might be a ground glass screen or a CCD which receives image information from the fisheye lens when the mirror 7 is in the position shown.

The embodiment of Fig. 13 is similar to that of Fig. 5, except for separating the lens system la from the CCD device lb. This has the same advantages as described above with reference to Fig. 12. The operation of this embodiment will be clear from preceding embodiments, except that devices 17a and 17b are shown in their storage positions, since they can be extended across the barrel 16 in order to act as relay or correcting lenses when light is received from the lens system la and the mirrors 7a and 7b are in their storage positions.

Similarly, Fig. 14 shows an embodiment like that of Fig. 8, except for separating the lens system la from the CCD device lb as shown in the drawing. Otherwise, the operation will be clear by analogy with the preceding embodiments where the lens and CCD combination la, lb, is separated.

Fig. 15 is another embodiment in which the lens system la is separated from the CCD device lb, but otherwise similar to the embodiment of Fig. 9. However, it will also be noticed that device 17a is in the storage position, whilst device 17b is extended across the path between the mirrors 7a,7b, and the CCD device lb. These could be different relay or correcting lens systems which are used for focussing image information on the CCD surface, lb due to the difference between the optics of the omnidirectional and normal fields of view. In the embodiment of Fig. 16, the lens assembly la is also separated from the CCD device lb (as in Fig. 15) but this embodiment is similar to that shown in Fig. 10. Likewise, the embodiment of Fig. 17 shows lens assembly la separated from CCD device lb, but otherwise similar to the embodiment of Fig. 11. However, device 17 is shown in the stored position because it could be a relay or correcting lens which is lowered across the CCD device lb when the mirror 7 is down and the device is used with the normal field of view. A lens assembly 39 may also be provided for correcting focus where light is reflected from the convex reflector 36 onto mirror 7 and then onto the CCD lb, although, as shown in the embodiment of Fig.18, ths lens assembly may be omitted and device 17 used for correcting focus.

The embodiments of Figs. 19-25 will now be described but it will be noted that they each include a panospherical image capturing device which employs six, and preferably eight inclined planar reflecting surfaces 40 (although any suitable number of reflecting surfaces may be employed as required). This device can be used with the preceding aspect of the invention, but can also be used independently. Before describing the device in detail, reference is made to US-A-5016109 and US-A-6115176. Both of these references employ four and eight-sided pyramids used as reflectors, but require multiple cameras positioned over the mirror surfaces. The use of multiple cameras is an expensive and cumbersome arrangement and is not practical in respect of some of the aspects of this invention, which require an image of the scene to be relayed to the centre of the structure.

Generally speaking, both fisheye and parabolic reflectors create maximum distortion or image compression/focal aberrations at the periphery or edge of the field of view. As this periphery is utilised for 360° x 360° viewing, it is the desirable section of the panoramic scene and this aspect of the invention can utilise plain or flat mirrors to image that part of the scene which will otherwise be distorted using the prior art wide angle lenses or parabolic reflectors.

Referring to the embodiment of Fig. 19, light from the surrounding panoramic scene is incident on each of six planar reflecting surfaces 40. These planar surfaces 40 are at +45° to the optical axis. Six outriggers 47 are arranged above the planar surfaces and each of these contains a relay lens 41 and a mirror arrangement or right angled prism 42,43, for turning the light back towards the centre of the structure into a lens 44, which slightly enlarges the image seen by the respective planar mirror and which contains vanes or light guides which crop or mask part of the overlap of the panoramic scene viewed by adjacent planar mirrors. The resulting image is then transmitted through relay and field lenses 45,46, to below the structure where the image information is used by the camera device (as described in more detail below). It is to be noted that the upper part 47' is opaque, since it represents the outrigger 47 which is seen at the side of the cross-sectional view and that the light path through the lens 44' is that of the outrigger seen behind the cross-sectional view.

In the embodiment of Fig. 20a, the planar mirrors 42,43, are replaced by modified Goertz prisms 50. It will again be noted that the prism 50' is seen end on, compared with the side view of prisms 50, in the drawing. This prism arrangement allows a much more compact profile which is useful when using a central lens arrangement 53 (as described below with reference to Fig. 21a). Apart from the prisms, the construction of the device shown in Fig. 20a will be apparent from the embodiment of Fig. 19. The use of prisms allows for a lens assembly to be placed in the centre of the apparatus.

Fig. 20b is a plan view of the arrangement shown in Fig. 20a. Here it can be seen that the outriggers 47 extend over part of the incline planar mirrors 40, i.e. looking down on the same. The lenses 41 (see Fig. 20a) have a field of view which not only covers most of the planar mirror 40 over which they are located, but also a portion of the adjacent planar mirrors, one on each side. This overlap is used in optically assisting in composing the omnidirectional image in the planar viewing field (as described below).

Fig. 20c - 20e shows how the panoramic image is composed in the flat viewing plane. Fig. 20c is similar to the plan view of Fig. 20b, except for showing regions which have been numbered and lettered. The numbers 1-8 represent the respective parts of the scene reflected by the planar mirrors 40. The letters A-H represent sectors in an upper part "9" of the arrangement shown in the plan view of the Fig. 20c. This will include a lens 53, as shown in Fig. 21a, which provides a wide angle view of the upper part (blind spot) of the scene and directs light downwardly through the lens system 45,46, towards, e.g. a CCD device, which also receives the light from the planar reflectors via lenses 44. Fig. 20d shows these optical image sectors and segments 1-8 and A-H in more detail.

Optical image sectors and segments enable a composite image to be composed at least partly within the camera (in contrast to the digital signal transformation techniques mentioned by Nalwa in US 6115176). This facilitates the processing of the video signals in order to display the omnidirectional image in the flat viewing plane (as disclosed for example, in my co-pending GB Application 0021433.8). Light guides, fibre optics, optical glass rods, reflecting surfaces, lenses, montaging baffles and/or masking matrixes can be used to compose the optical image sectors and segments on the face of a common CCD, for example, so as to position the respective parts of the panoramic scene side by side, during the process of assembling the composite image for viewing in the flat viewing plane. Some processing of the digital image signals can also be carried out to create a cropped image in the flat viewing plane (if this is required). This will be simplified by any of the optical composing steps carried out within the camera. However, in some cases, it may not be necessary to create an image in the flat viewing plane, especially for movement detection purposes. Fig. 20e shows how the sectors and segments are composed side by side (optically and/or by known digital spatial transformation techniques) after removing distortions, so as to provide a matrix of columns and rows of rectangular image portions A-H and 1-8 corresponding with those in Fig. 20c. Where digital processing is required known techniques can be used in order to achieve this.

With further reference to Fig. 21a, a wide angle lens 53 can be inserted in the centre of the outriggers 47 to view the central 90° area and this is composed into the central ring or circle of Fig. 20d. Alternatively, with reference to Fig. 21b, a standard or variable power (zoom) lens system 54 can be inserted in the centre of the outriggers 47. A pan and tilt system can direct the apparatus to a part of the omnidirectional scene requiring closer examination, and the image captured by the lens system 54, having a longer effective focal length, can be scrutinised. The lens system 54 illustrated in Fig. 21b is Nikon Patent S 53 131853, although this system is shown for illustration only and it will be appreciated that any other suitable lens system may be used. Fig. 21c illustrates a "back to back" version of the apparatus of Fig. 21b.

Figs. 22 shows how the arrangement of Fig. 21a can be used in a 360° x 180° camera arrangement. Similar components are being given similar reference numerals. In Fig. 22, lens system la and CCD device lb are located together, and device 17 is in the raised position, e.g. to act as a relay lens for the omnidirectional device. When the mirror is stored, device 17 is also stored, since the lens/CCD combination la, lb, has a normal field of view. Fig. 23 shows a similar embodiment, but the lens system la is separated from the CCD device lb and placed in front of the mirror 7, and device 17 is lowered into its stored position and a correction lens 55 is employed for correcting the focus of the omnidirectional imaging device. In the normal field of view, the mirror is stored and device 17 is raised into its operating position to correct the focussing. Other parts of the construction and operation of the embodiments of Figs. 22 and 23 will be clear from the preceding description.

Fig. 24 shows the planar reflecting device used in a 360° x 360° panoramic imaging embodiment which otherwise employs folding mirrors 7a,7b and devices 17a, 17b, besides the lens and CCD combination la, lb. A similar embodiment is shown in Fig. 25, except the lens system la is separated from the CCD device lb to obtain an increased field of view. However, it will also be noted that other relay or correcting lenses 57,58 are provided. Lens 57 is fixed across the optical path to the CCD device lb, whereas lens 58 is attached to the backing plate of mirror 7a so that, when the mirrors 7a,7b, are moved into their stored positions, the lens 58 is positioned across the optical path between the lens system la and the CCD device lb so that it cooperates with lens 57 to correct for focussing or relay.

In either Fig. 24 or 25, the devices 17a, 17b, can be any of the devices mentioned above. Moreover, they may provide different prescriptions for the omnidirectional field of view, i.e. either device 17a or 17b being moved into the optical path between the mirrors and the CCD device lb. However, they may also be part of the focussing correction in the normal field of view when the mirrors are in their stored positions.

Fig. 26a shows a detailed modification where mirrors 7a,7b, are in their operating positions for reflecting light from omnidirectional imaging devices onto a combination of devices 17c, 17d. Device 17c can be, for example, a high resolution CCD which is used for enlarging or scaling up a portion of the scene in the omnidirectional field of view. Device 17d can be an image intensifier which is useful for detecting movement in a surveillance zone at night, e.g. by detecting movement of a thermal body such as an intruder. Devices 17c and 17d may be used in combination or separately. Also, device 17a can be moved from its storage position into an operational position to provide other facilities, such as correcting focus, etc. In this modification, the lens system la is adjacent the CCD device lb, but they may also be separated as described above with respect to other embodiments.

Fig. 26b shows an arrangement where device 17a is positioned (as shown) across the optical path to the lens/CCD combination la, lb (or CCD lb) in order to correct focussing of the omnidirectional image capturing means and wherein the device 17b is raised into the same operating position (after storing device 17a) to correct for focussing in the normal field of view. Alternatively, device 17b may be an alternative sensor, for example, an image intensifier.

Figs. 27 to 30 illustrate embodiments which demonstrate the modular potential of the apparatus, in which imaging systems, similar to those described above, may be used in combination or in isolation as required. This enables the systems to be stand alone, or converted into a pure omnidirectional imaging device by removal of the variable power standard lens system and mirror servos, and fitting an internal CCD and blanking plates. The internal housing is common to these embodiments, and enables the use of a combination of different omnidirectional image gathering devices in the same system, for example, a fisheye lens system in one hemisphere and a catadioptric in another.

With reference to Fig. 27, in these embodiments, systems may be mounted for side to side, or top to bottom hemispherical views of a panoramic scene and may be suspended from a mast 70, or located on top of a mast, depending on how the apparatus is to be used. In this case, folding mirrors 7a,7b, can be moved manually into their operating position (as shown in the drawing) or into stored positions where they move from being inclined to being vertical. Access is available through an inspection plate 71. This can be a CCD for converting panospherical image information into video signals, or it could be a film holder, either for still or motion photography. Device 17 can also be selectively adjusted or replaced via access through the inspection cover 71 and after folding back mirrors 7a, 7b.

Generally speaking, Fig. 27 provides a low-cost alternative by providing conversion plates to enable the system to be customised as required, especially by enabling inexpensive panospherical image capturing means to be used in combination with more expensive devices. For example, Fig. 28 shows a modification where the fisheye lenses are replaced by the omnidirectional image capturing means shown in Fig. 9. In this case, the inspection cover 71 is located in the position otherwise occupied by window 3. Fig. 29 shows another modification, based on the embodiment of Fig. 10 and also having an inspection cover 71. Fig. 30 shows a modification with the planar reflecting omnidirectional devices, similar to the embodiment of Fig. 24. This also has an inspection cover 71.

Figs. 31 and 32 illustrate embodiments which employ back-to-back fisheye lens systems 20a,20b (similar to the embodiments of Figs. 5 and 6), but where the lens/CCD combination la, lb, has been further modified to simplify the construction. In this case, the lens system la is permanently located between the window 3 and the movable mirrors 7a,7b. These mirrors are shown in the omnidirectional viewing position in Fig. 31, which also shows CCD device lb in a retracted position at which the omnidirectional image information reflected from the mirrors is focussed on the surface of the CCD device. The CCD lb is mounted on a backing plate 71, which is threadably coupled to a screw 72 that can be rotated by motor 73. Motor 73 is operable in forward and reverse modes whereby screw 72 can be rotated either clockwise, or anticlockwise to move the CCD device lb and backing plate 71 either forwards or backwards so as to correct for the focal distance of either the omnidirectional (e.g. fisheye) lens 20a,20b or the standard lens la.

Fig. 32 shows the normal field of view, i.e. where mirrors 7a,7b, have been pivoted into their storage positions and where CCD lb and backing plate 71 have been advanced towards the window 3. Hence, the CCD device lb can be moved to the correct position for receiving a focussed image from the lens system la in the normal field of view.

The embodiments of Figs. 31 and 32 can greatly simplify the construction of the apparatus, since it disposes with device 17 and its drive systems, as well as simplifying the structure with regard to the use of relay and correcting lens systems.

Fig. 33 shows an embodiment where back-to-back fisheye lenses are also used for panospherical image gathering, but in this case, the mechanisms are comparatively fixed for omnidirectional and normal fields of view. For example, the lens system la has a normal field of view via window 3 and is arranged to focus images on CCD 17e. Mirrors 7a,7b are fixed in the positions shown, in order to reflect light from the back- to-back fisheye lens systems 20a,20b onto CCD device lb. This embodiment provides a low-cost alternative, i.e. where mechanisms for moving various components within the camera device are no longer required. It also has the advantage of enabling continuous image capture in a dual viewing mode from both the normal field of view (3,la,17e) as well as the omnidirectional field of view (20a,7a,20b,7b,lb).

Another alternative is to provide panospherical image capturing means (e.g. a fisheye lens) and a standard lens on a movable member, such as a slide or turret, whereby either one may be moved into position for giving a normal or omnidirectional field of view to the same image plane of the camera. Such a device could be housed in a waterproof transparent enclosure which is pressurised with warm dry air. Figure 34 shows a modification of the embodiment shown in Figure 8, the main difference being that a beam-splitter 80 is used instead of movable planar mirror 8. This has the important advantage of avoiding the need to move the mirror 8 and hence avoiding the need for servomotors, which suffer from wear due to continued use. The beam splitter can be any device which enables image information to be gathered on each of the incident light paths of the optics for the omnidirectional and normal (or standard) fields of view and which combines them on a common incident path to the imaging system (CCD). An example of a beam splitter is found in US 5140151 (Weiner). Preferably a beam-splitting cube (as illustrated in Fig. 34) can be used, but it can be a semi-silvered transparent substrate, such as a partially reflective/partially transparent mirror. For example, dichroic cube beam-splitters having two precision right angle prisms cemented together, with an appropriate interference coating on the surface for reducing absorption loss, can be used. The irises in the omnidirectional and normal image capturing systems can be used to black out the ray path from the lens not required, or an alternative iris can be provided.

Alternatively, pellicle beam-splitters can be used, wherein the pellicles are very thin covered cellulose membranes bonded to lapped aluminium frames. Further details of such beam-splitting devices are provided in the catalogue of Edmund Optical Company as available, for example, from www.edmondoptics.com., the contents of which are incorporated by cross-reference. These and other beam splitters in the state of the art can be components in cameras, or attachments, which embody the invention. These are characterised in that they can be fixed, yet still provide the facility to gather omnidirectional and normal images. In the case of the cube 80, light is either reflected from the semi-silvered surface (i.e. ray path 81,82), or it passes straight through the semi-silvered surface (i.e. ray path 83,82). Thus, as shown in Figure 34, the panospherical image captured by convex mirror 30, which is then reflected from the planar or curved mirror31, is directed through lens system 32 onto the beam-splitting cube 80 where it is reflected onto the imaging device lb, whereas light in the normal field of view passes through (zoom) lens system la, and then directly through the beam-splitting cube 80, where it is received by the same CCD device lb. The focussing of the optics enables these images to be imaged on the focal plane of the camera. This would result in two simultaneous images on the focal plane and whilst these could be separated by using differently polarised light for each view and suitable filters, or other optical or digital processing means for separating the images for viewing either separately or simultaneously the panospherical or partial scene, it is preferred to include further means which optically switches either image on or off on the camera's focal plane. This can be achieved by inhibiting or blanking out the unwanted field of view by using stops, irises, shields, or filters to blank out one or other image. For example, in Fig. 34, the lens systems la and 32 are each provided with an iris diaphragm 84a, 84b and either can be "stopped down" so that the diameter of the hole ("the stop") is reduced to nil, whereby the entire circular interleaving components effectively block light passing to the semi-silvered beam splitter 80. Where circular iris diaphragms are not provided, iris components or simple shields having other shapes (such as rectangular or rectilinear, square, and crosshatching interleaving components such as overlapping blind formations) can be used. The Edmund Optical Company (see above cross-reference describes circular neutral density filters of simple , variable and stepped construction; 3 M Light Control film; polarising filters and variable density filters which can all be used for these purposes.

As irises have moving mechanical parts, these are subject to wear, and they can be avoided by using, for example, some of these controllable filters, and devices such as a transparent (active matrix) LCD (TLCD) or similar display material such as, but not exclusively, transparent organic light emitting diodes (TOLED), which can generate a 100 per cent black mask for either the normal or omnidirectional field of view. These can be called "dynamic" or "LCD blinds". The use of stops or irises can thus effectively switch the field of view on the beam splitter from omnidirectional to normal or vice versa, and can remove any ghost or overlaying duplicate image from the unwanted field of view before it is received by the CCD imaging device lb..

With the iris 84a stopped down, and iris 84b open, the semi silvered cube 80 will allow lOOper cent reflectance of the omnidirectional field of view. With iris 84b stopped down and iris 84a open, light passes directly through the cube 80. As a further alternative, a shield or stop can be switched or moved either to block, or to pass light, but this has the disadvantage of requiring a movable part.

A controllable device is, for example, a TLCD, which is a display having an active matrix of pixels that can be switched either to pass, or to block all red, green and blue light. The matrix display then appears either transparent or black respectively (since switching the red, green and blue light effectively passes white light, or blocks all light).

Use of polarising materials to reduce or to pass the light transmitted to the semi silvered beam splitter 80 can include polarising filters used in pairs or individually rotated about an axis to control the light output from either the omnidirectional or normal field of view. Such devices are also available from Edmund Optical Company as noted above.

Stepped or linear density filters can be used in a variable filter translation mount, or circular variable density filters can also be used to mask or blind the undesired optic (and these are also available from the Edmund Optical Company) or to control light output.

As shown in Fig 34, the devices 84a and 84b, for blanking out one or other lens system la or 32, can be simply attached to the lens system, but preferably, they are combined with a filter or shield built into the semi-silvered beam splitting cube or glass sheet. This is shown in Figs. 35 and 36, where a TLCD or mechanically operated filter or shield 90b is incorporated onto the 45 degree surface of the prism 91 (or glass sheet with semi silvered surface, not shown), and/or a second filter or shield 90a is attached to the horizontal surface. When the filter 90b on the 45 degree surface is activated to mask the horizontal beam, this results in an opaque mirror finish which assist in diverting the vertical or lower beam through 90 degrees to the image sensor. Conversely, when the filter 90a on the horizontal surface is activated, and the 45 degree surface deactivated, the 45 degree surface would be clear so that the horizontal ray would penetrate to the sensor, while the lower optic would be masked. Both lens systems for the omnidirectional and normal fields of view also have an iris or a TLCD blind 84a, 84b. An iris can be operated by a servo, and the lens may have one anyway. These TLCD blinds could act as a means for controlling the exposure, as disclosed in my copending International Application No . PCT/GBO 1/01115, and could save the cost of a mechanical iris.

In practice, a TLCD blind could have a visible matrix (or dot screen) when it is transparent and in focus to the sensor, so it would best be positioned to avoid being in sharp focus on the sensor.

These devices which employ a beam splitter, can be embodied in the other arrangements which employ movable elements, such as mirrors, for example in the embodiments described above with regard to Figs 7.12,17 and 22.

Alternatively, the image information gathered in the omnidirectional and normal fields of view can be directed to respective separate autonomous image sensors, whereby some form of switching enables either the panoramic scene or the partial scene to be selected for display on a viewing screen. This can also be achieved by multiplexing and/or by selective processing of the information in order to display either the panoramic or partial scene. The same advantage is still obtained in using separate optics for the image information gathered in the omnidirectional and normal fields of view, even though it is directed to respective image sensors, because better image resolution of part of the panoramic scene is achieved with (say) an ordinary variable power lens, compared to scaling up a portion of the image information gathered by the omnidirectional optics. A simple way to achieve this alternative, is to use either respective CCD imaging chips for the omnidirectional and normal fields of view, or to use respective parts of a common CCD imaging chip for the same purpose. It is then only necessary to switch from one to the others so as to view the required scene. This could be a lower cost alternative than using means for moving mirrors or other elements, depending on the cost of the chip.

Figs 37-40 show a similar modification to previous embodiments, i.e. where the beam splitter 80 and inhibiting means 84a, 84b, 84c (such as irises, blinds or TLCD's are used to effect selectively the passage of light through, or reflection of light from the semi silvered surface of the beam splitter, as explained above with respect to Figs 34- 36.

Whilst various embodiments have been described in detail, the components used in these embodiments may be interchanged, in some cases, in order to modify the examples described. Moreover, other changes can be made within the broad concepts of the various aspects of this invention. For example, some embodiments illustrate only the capture of omnidirectional information from a single hemispherical field of view. It will appreciated that these embodiments may be provided "back to back" to capture omnidirectional information from a spherical field of view.

Referring to Figure 41, this illustrates a spherical omnidirectional camera having six lenses 125. Each lens has a circular periphery and is supported in a split mounting 134 which has opaque spherical sides 135. Figure 42 shows the camera in an exploded view, where respective hemispherical halves 135a, 135b of the mounting are shown separated vertically in order to illustrate each of the two horizontally and oppositely directed lenses. Each of the six lenses captures light over 120° solid angle of the panoramic scene. Each lens may be optically coupled to a lens splitter or baffle 126 so that respective components of the optical image from the lens 125 can be split into bundles of optical fibres or optical glass rods 127 which optically couple these components to a montage lens matrix, or montaging matrix, 128, shown in more detail in Figure 43. The montaging matrix 128 composes the images directly onto the CCD array 130 via a masking/cropping matrix 129. As the top and bottom lens images need to be composed along the length of the sides of the CCD the image is split in the lens housing into four sectors and projected onto the array through having been divided and rotated.

Each lens is shown schematically, since it will be of a more complex construction. Each lens may be a variable power fisheye/wide angle lens system, such as those described below with reference to Figures 51 and 52, which provides each segment with optical scaling, thereby removing the need to provide a PTZ system to manoeuvre the optical system

Referring to Figures 43a and 43b, this illustrates in more detail how each of the light guides or fibres 127 from each of the six lenses 125 is coupled to respective direct imaging zones on the CCD 130. A lens splitter or baffle and/or screen arrangement is mounted below each of the wide angle lenses, whereby light from the lens 125 is divided into respective optic fibres 127, which are gathered into cables for insertion into the fan shaped mounting assembly in the montaging matrix 128. Each end of each fibre can include a screen (not shown) to enable optical coupling, the lower end of each being shown(in Figure 43c) optically coupled to a lens which provides direct optical imaging, via respective aperture in the masking baffle 129, onto a respective zone of the imaging surface of the CCD array 130. References to montaging matrices can be found in my co-pending UK applications.

Fig. 44 shows an embodiment in which the standard lens system la, preferably having a zoom facility, is mounted within a central meniscus void formed in the strongly negative meniscus element, or "fisheye lens" , 20a of the fisheye lens system 20. Light focussed by the lens system la passes through a void formed in secondary lens 20b of the fisheye lens system into lens grouping 140 for subsequent capture in the centre of the CCD lb. Light focussed by the lenses 20a, 20b of the fisheye lens system 20 also passes into lens grouping 140 to be captured on the same CCD lb. As in the embodiment shown in Fig. 34, this would result in two simultaneous images in the common focal plane. Optical or digital processing means may be provided for separating the images for viewing either separately or simultaneously the panospherical or normal field of view. Alternatively, as shown in Figure 44, a separate CCD/sensor lc may be mounted within the centre of an surrounding sensor lb to capture separate the image captured by the standard lens system la.

Fig. 45 shows an embodiment which, similar to that shown in Fig. 34, utilises a beam splitter 80. The beam splitter 80 receives light focussed by fisheye lens 20a, secondary lens 20b and bi-concave, or anamorphic, lens 20c of the fisheye lens system 20. Lens 20c assists in ray path relay. The beam splitter 80 reflects the panospherical image captured by the fisheye lens system 20 on to imaging device lc via lens grouping 140, whilst enabling the central zone of the panospherical image to pass to standard lens system la for imaging on imaging device lb. The standard lens system la preferably includes a variable power (zoom) lens system for magnifying the central zone of the panospherical image. As discussed above, the beam splitter may take any convenient form; whilst the beam splitter 80 shown in Fig. 45 is a partially reflective/partially transparent mirror, it may take any of the alternative forms described above, such as a prismatic or cubic beam-splitter. It may be semi-silvered in only a central part thereof, or completely over the surface of the reflecting plane. Although the mirror is shown as plane and at 45° to the principal axis of the fisheye lens system 20, it may be shaped into any suitable form for directing the captured image as appropriate to the lens grouping 140' and to improve on the distortional characteristics of the reflected image.

Fig. 46a shows a similar arrangement to that shown in Fig. 45, except that, as in the arrangement shown in Fig. 44, part of the standard, or variable power, lens system la is mounted within a central meniscus void formed in the fisheye lens 20a of the fisheye lens system 20. This part of the lens system la may continue through the meniscus void to the secondary lens 20b using tubular baffles or relay lenses, or the ray path can be projected through that void without a surrounding collar. Fig. 46b shows a "back to back" embodiment of the arrangement shown in Fig. 46a for full spherical image capture.

Fig. 47 shows a similar arrangement to that shown in Fig. 46a except that it is the image from the central zone of the panospherical image, as captured by part of the standard lens system la, which is reflected by the beam splitter 80 on to the imaging device lb, whilst the remainder of the panospherical image is passed on to lens grouping 140 for imaging on the imaging device lc.

In the arrangement shown in Fig. 48a, the standard lens system la is replaced by a smaller fisheye lens system 20' comprising lenses 20a', 20b' and 20c', and lens grouping 140' . As described in my co-pending International application no PCT/GB01/03251, the contents of which are incorporated herein by reference, the fisheye lens system 20 is larger than the other lens system 20' by a factor related to the interocular distance of 65 mm. This figure may vary for the purpose of hyperstereoscopy / hypostereoscopy for subject of varying distance. There would be a 65mm difference in size of the two fisheye lenses in the apparatus from the common central axis to the edges. The recorded scene strikes the larger lens from a wider or horizontally displaced viewpoint allowing image artefacts unseen by the smaller lens with its narrower viewing arc to be recorded. The recorded signals of the different lens systems are recorded and displayed at the same size and resolution, however there is clear visible differences in the scene giving the output genuine three dimensional relief.

In this arrangement, the fisheye lens 20a' and parts 20b' and 20c' of the secondary lens grouping of the smaller lens system are mounted on fisheye lens 20a of the larger lens system such that parts 20b' and 20c' of the secondary lens grouping of the smaller lens system lies within a meniscus void 145 formed in the fisheye 20a of the larger lens system. Opaque material 150 is located between the lens systems in order to prevent light cross over between the lens systems. Light focussed by the lenses 20a', 20b 'and 20c' passes through the void and secondary lens 20b of the larger lens system and passes through mirror 80 having an aperture, on to lens grouping 140' for focussing on imaging device lb. Light focussed by the lenses 20a, 20b is redirected by the beam splitter on to separate imaging device lc via lens grouping 140. As in Figure 45, mirror 80 may also be replaced by a beam splitter to enable the complete image to be seen by both imaging devices lb and lc. Each imaging device may comprise a CCD array with a plane, convex or concave image receiving surface to reduce optical distortion. Fig. 48b shows a back to back embodiment of the arrangement shown in Fig. 48a for full 360° x 360° image capture.

Fig. 49 shows a similar arrangement to Fig. 48a except that part of the lens grouping 140' of the smaller lens system is located within a meniscus void in the secondary lens 20b of the larger lens system, and that the beam splitter 80 reflects the image captured by the smaller lens system onto imaging device lb via the remainder of the lens grouping 140' whilst passing the image captured by the larger lens system on to imaging device lc via lens grouping 140. In the back to back arrangement shown in Fig. 50, lens grouping 140 is common to both the larger and smaller lens systems, the images from the two lens systems being captured on a common imaging device lb. Optical or digital processing means may be provided for separating the images. Alternatively,, as shown in Figure 44, a separate CCD/sensor lc may be mounted within the centre of an surrounding sensor lb to capture separate the image captured by the standard lens system la.

In any of the above described embodiments in which the panospherical image capturing apparatus is provided by a fisheye lens system and a group of secondary lenses, at least two of the secondary lenses may be moveable relative to one, or both, of the fisheye lens and the image plane in order to vary the magnification of the omnidirectional image captured by the lens system. For example, as illustrated schematically in Fig. 51, in an optically compensated zoom system two alternate secondary lenses, for example, biconcave lenses 20c and 20e, are moveable together along the optical axis, for example, using servo motors and appropriate couplings, relative to the lenses 20a, 20b, 20d, lens grouping 140 and imaging device lb to provide more than two magnifications, in this embodiment four magnifications, at which the captured image is in focus. Fig. 52 illustrates a mechanically compensated zoom system, in which nonlinear motion of secondary biconcave lens 20c and biconvex lens 20d is effected by a rotating cam arrangement 160 to provide compensating shift of the lenses. Such zoom systems can be used in combination with a beam splitter or reflecting surface as described above to reduce backworkings.

The above described embodiments can enable the use of varying and separate lens materials such as germanium to be used in the central lens assemblies to gather image properties on the same axis and perspective. This would enable infra red wavelengths to be collected and directed to a separate sensor, and overlaid on the visible wavelength display, or displayed separately. Where "back to back" (top and bottom, or side to side) omnidirectional embodiments have been described, one of the omnidirectional assemblies may contain elements and materials such as germanium and prescriptions suited to the gathering of infra red information. A retractable or permanent infra red image intensifier can be utilised.

Figs. 53 and 54 illustrate an arrangement in which omnidirectional imaging apparatus 200 is in the form of an attachment to a camera. With reference first to Fig. 53, the apparatus 200 comprises upper and lower parabolic reflectors 202 arranged back to back with respect to the horizontal axis a-a. Above the reflector 202 is situated a planar or concave reflecting surface 204 arranged back-to -back with a convex parabolic reflector 205. A lens system 206 is located centrally of the reflecting surface 204 below an upper convex reflector 207 (which is preferably hemispherically parabolic). The various optical elements are all arranged on a central vertical axis b-b. With reference to Fig. 54, situated inside each parabolic reflector 202 is a known Goerz prism assembly 208. This collects light from both reflecting system 202,204, and reflecting system 205,207, and directs it to a tubular member 210 coupled to camera 212 by screw thread connector 213. As illustrated in Fig. 55, alternatively a prism 214 may be used to direct light to the tubular member 210, which may contain relay lenses to focus light on the plane of the sensor of the camera. The above assembly may also be applied to the pyramidal ad fish eye lens systems described herein.

In use, the apparatus 200 is arranged to view an all round panoramic scene. Light from the "sides" of this scene is reflected from the parabolic reflector 202 onto the reflecting surface 204, where it is reflected through lens 203 into the prism 208 and hence directed to camera 212. This is represented by the ray paths in the upper section of the apparatus shown in Fig. 53. This reflecting system 202,204 therefore collects light from an annular field of view, i.e. from the sides of the scene.

Light from the upper parts of the scene (e.g. the sky) is reflected from reflector 205 onto the convex reflector 207, which in turn reflects it through lens system 206, lens system 203 and into the prism arrangement 208. This system 205,207 therefore collects light from an upper segment of a hemispherical field of view, i.e. from the upper (sky) part of the scene. Thus, the reflective system 205,207 images the upper segment, whereas the reflective surface combination 202,204 images the annular segment of the hemispherical field of view, 180° x 360°. The lower reflecting systems (below the axis a-a) are constructed and operate in the same way, whereby they contribute to the images received from the sides and the bottom of the hemispherical scene.

As described in my previous International application no PCT/GB01/01115, variable power lenses may be provided in the parabolic system shown in Fig. 53, and the convex reflector 207 omitted, to enable variable power optical scaling.

With reference to Fig. 56, the camera can be a still camera which is fitted with grips 216 to help in keeping it steady, or it can be a video camera which is supported on a boom (not shown) for making video films. In these arrangements, suitable correcting lenses, filters, ground glass screens, or other optical means 218 are provided to enable the optical images captured by the apparatus 200 to be transferred to the optical system of the camera 212 to provide for digital imaging. These digital images can be stored for subsequent processing, e.g. by a home based pc, or immediate processing on a mobile laptop, so that the 360° x 36,0° image can be shown as a flat montage on the display of a screen. Such processing is known to those skilled in the art and is also described in more detail in my earlier GB patent Applications referenced above.

Claims

1. Apparatus for providing an optional panospherical field of view in a camera having a lens system with a normal field of view, being provided either in the form of an adapter, or as part of a camera; the apparatus including:

(a) panospherical image capturing means for capturing omnidirectional information from around the panoramic scene in an omnidirectional viewing mode;

(b) optical means for capturing image information from part of the panoramic scene in a normal viewing mode; and

(c) selecting means for selecting either the omnidirectional, or the normal viewing mode, by causing light from either the panospherical image capturing means, or the optical means with the normal field of view, to be incident on an image plane of the camera.

2. Apparatus according to Claim 1, wherein the selecting means is operable so that: in the omnidirectional viewing mode, only the omnidirectional image information from the panospherical image capturing means is incident on the image plane of the camera; and in the normal viewing mode, only the image information from the optical means with the normal field of view is passed to the image plane of the camera.

3. Apparatus according to Claim 2 , wherein the selecting means includes reflecting means which reflect omnidirectional image information from the panospherical means to the image plane of the camera and which also prevent the image information from the optical means, with the normal field of view, from reaching the image plane of the camera.

4. Apparatus according to Claim 2, wherein the selecting means includes reflecting means which, in the normal viewing mode, passes the image information from the optical means to the image plane of the camera, and which prevent the omnidirectional image information from reaching the image plane of the camera.

5. Apparatus according to Claim 3 or 4 , wherein the reflecting means includes one or more reflectors movable between first and second positions in which the respective information is either reflected onto, or prevented from reaching, the image plane of the camera.

6. Apparatus according to Claim 1, including a dual operating mode in which omnidirectional information, as well as image information from the optical means with the normal field of view are passed to the image plane of the camera and further including means for distinguishing between the respective image information captured by the panospherical and optical means.

7. Apparatus according to any preceding claim wherein the camera includes a movable element defining the image plane of the camera, and drive means for moving said element between positions where either omnidirectional information or information from the optical means with the normal field of view is in focus on the image plane of the camera.

8. Apparatus according to Claim 7 , wherein said movable element includes a CCD device.

9. Apparatus according to Claim 8, wherein said movable element is coupled to reflecting means, whereby movement of said element results in either information from the normal field of view or omnidirectional image information being reflected from said image plane of the camera.

10. Apparatus according to any preceding claim, wherein said optical means comprises a barrel portion which can either be demountably attached, or is fixed to the camera, said barrel portion having a window with means for dispersing atmospheric drops or particles.

11. Apparatus according to Claim 10, wherein said barrel portion has at least one branch portion in which respective panospherical image capturing means is mounted.

12. Apparatus according to any preceding claim, including means for maintaining optical surfaces clean and dry.

13. Apparatus according to Claim 12, wherein said means for maintaining optical surfaces clean and dry includes any one of wiper blades; centrifugally driven surfaces; and/or means for pressurising the apparatus with heated dry air.

14. Apparatus according to any preceding claim, including a screen for receiving an omnidirectional image from the panospherical means, whereby an image on the screen means can then be picked up on the image plane of the camera.

15. Apparatus according to any preceding claim, including a CCD device for capturing the respective image information in either the omnidirectional or normal viewing modes.

16. Apparatus according to any of Claims 1 to 14, including an auxiliary imaging means, which can be moved into or out of an optical path to the image plane of the camera, for receiving either the omnidirectional information, or the optical information from the normal field of view so as to provide auxiliary image information.

17. Apparatus according to any preceding claim, wherein the panospherical image capturing means includes back-to-back hemispherical image capturing means.

18. Apparatus according to Claim 17, further including one or more reflecting surfaces which move into a position in order to reflect respective hemispherical images onto the image plane of the camera, and into a different position to enable the image plane to see the normal field of view.

19. Apparatus according to any preceding claim, wherein said panospherical image capturing means includes any one or more of fish eye lenses, convex reflectors, and a plurality of planar reflectors.

20. Apparatus according to Claim 19, wherein said panospherical image capturing means comprises a fish eye lens and a plurality of secondary lenses for focussing light from the fish eye lens on to the image plane.

21. Apparatus according to Claim 20, wherein at least two of the secondary lenses are selectively moveable towards or away from the fish eye lens to vary the magnification of information captured by the panospherical image capturing means.

22. Apparatus according to any preceding claim, wherein said image plane also provides a support for film whereby the camera can be used for still or motion photography.

23. Surveillance apparatus for capturing image information from a panoramic scene , comprising an imaging device having a standard and preferably variable power lens system; means for mounting the imaging device on an axis of rotation so that it can be rotated to a selected azimuthal position, whereby the variable power lens system can capture image information from an area in the panoramic scene within its field of view, the imaging device further including panospherical image capturing means for capturing an omnidirectional image from said scene; and selecting means which are operable so as to capture either image information from the standard lens system of the imaging device, or from the panospherical image capturing means, or both, depending on a selected mode of operation.

24. Apparatus according to Claim 23, wherein the imaging device includes means for converting optical information into video signals, and further including image processing and display means for displaying either the image information from the variable power lens, or the omnidirectional information from the panospherical image capturing means.

25. Apparatus according to Claim 24 , further including means coupled between the imaging device and the image display means for providing azimuthal position data as the imaging device is rotated about said axis, whereby activity observed in an area within the panospherical scene can be viewed by rotating the optical device to the respective azimuthal position.

26. Apparatus according to Claim 25 , wherein the imaging device is coupled to the processing and display means so that the imaging device can be automatically guided to observe activity in an area within the panospherical image.

27. Apparatus according to any of Claims 23 to 26, wherein motion sensing means provides pan, tilt and/or elevation data for controlling the position of the imaging device.

28. Apparatus according to any of Claims 23 to 27 wherein the panospherical image capturing means includes optical means for capturing hemispherical images, and wherein the standard lens system has its normal field of view between the means for capturing the hemispherical images.

29. Apparatus according to Claim 28, wherein the hemispherical image means are mounted either side by side to provide omnidirectional information on each side of a central zone, the standard lens then having said central zone as its normal field of view, or one above the other to omnidirectional information from regions above and below a scene, the standard lens then having a peripheral part of the regions as its normal field of view.

30. Apparatus for providing a panospherical field of view and a normal field of view, the apparatus being provided either in the form of an adapter, or as part of a camera, the apparatus including:

31. Apparatus according to Claim 30, wherein the panospherical image capturing means comprises means for capturing a hemispherical field of view, and the optical means has as its normal field of view a central zone of the hemispherical field of view.

32. Apparatus according to Claim 31, wherein at least part of the optical means is located within a central aperture of the panospherical image capturing means.

33. Apparatus according to any of Claims 30 to 32, in which omnidirectional information, as well as image information from the optical means with the normal field of view are passed to a common image plane of the camera and further including means for distinguishing between the respective image information captured by the panospherical image capturing means and the optical means.

34. Apparatus according to any of Claims 30 to 32, comprising a beam splitter for reflecting the image information from one of the panospherical image capturing means and the optical means to its imaging plane, and for passing the image information from the other, of the panospherical image capturing means and the optical means to its imaging plane.

35. Apparatus according to any of Claims 30 to 34, wherein said optical means comprises a variable power lens system.

36. Apparatus according to any of Claims 30 to 35, wherein said panospherical image capturing means comprises a fish eye lens and a plurality of secondary lenses for focussing light from the fish eye lens on to the first image plane.

37. Apparatus according to Claim 36, wherein at least two of the secondary lenses are selectively moveable towards or away from the fish eye lens to vary the magnification of information captured by the panospherical image capturing means.

38. Panospherical imaging apparatus comprising a plurality of planar reflecting surfaces arranged to reflect light from a panoramic scene into corresponding light guiding means; and optical means for combining image information received from the light guiding means so as to provide a composite image of at least an annular section of the panoramic scene.

39. Apparatus according to Claim 38, wherein the planar reflecting surfaces are sides of a pyramidal structure, said sides being triangular or quadrilateral in shape and inclined with respect of the annular section of the scene.

40. Apparatus according to Claim 38 or Claim 39, wherein each light guiding means is arranged to view an area including at least a the respective part of the planar reflecting surface, and preferably also a portion of adjacent planar reflecting surfaces, whereby overlapping optical information is captured for the purpose of joining images from the adjacent planar surfaces, when producing the composite image.

41. Apparatus according to any of Claims 38 to 40, wherein the light guiding means comprise planar reflecting surfaces or prisms.

42. Apparatus according to any of Claims 38 to 41, wherein there are at least six planar reflecting surfaces for each pyramid and preferably eight.

43. Apparatus according to any of Claims 38 to 42 , including a standard lens system located centrally of the planar reflecting surfaces at the apex of a pyramid for receiving light directly incident from the other part of the panoramic scene than the annular section.

44. Apparatus according to Claim 43 , wherein the lens system is situated at the apex of a pyramid structure for directing light inwardly of the structure.

45. Apparatus according to any of Claims 1 to 4, wherein the selecting means includes a beam splitter which either passes or reflects image information from either the normal or omnidirectional field of view to the image plane of the camera, and inhibiting means which either (a) operates either to prevent the image information from reaching, or to pass the image information to the beam splitter, from either the omnidirectional or normal field of view, or (b) operates so as to cause the beam splitter to reflect or to pass the image information from either the omnidirectional or normal field of view.

46. Apparatus according to Claim 45, wherein the inhibiting means includes respective stops, filters, irises or blinds which block at least a major part of the image information incident on, or passing through either the panospherical, or optical image capturing means, whereby the beam splitter either passes, or reflects incident light onto the image plane of the camera.

47. Apparatus according to Claim 45 or 46, wherein the beam splitter includes a semi reflective surface which passes and reflects image information from the normal or omnidirectional field of view, or vice versa, to the image plane of the camera.

48. Apparatus according to Claim 47, wherein the inhibiting means includes means on, or proximate to the semi reflective surface, for selectively preventing the image information from reaching the beam splitter according to whether the omnidirectional or normal field of view is selected.

49. Apparatus according to any of Claims 45 to 48, wherein the inhibiting means includes a transparent display material having a matrix or mask of optically active elements that can be switched (a) either cause the beam splitter to pass or to reflect the image information from the normal or omnidirectional field of view, or (b) to block or to pass the image information from the omnidirectional or normal field of view.

50. Apparatus according to any of Claims 1 to 4, wherein respective image sensing means are provided for the image information from the normal or omnidirectional fields of view and wherein the selecting means includes display means, and means for switching, or for processing the information to enable either field of view to be displayed.

51. Omnidirectional imaging apparatus comprising:

a plurality of optical imaging devices having optical axes which are directed to respective fields of view for forming images from a hemisphere or sphere surrounding a panoramic scene; each device having a lens and an imaging surface for receiving optical images from the lens, each device also converting the optical images from the lens into video signals; and

processing means for processing the video signals and for reproducing components of the scene, as a montage, in a flat viewing plane;

characterised in that image distortion reducing means are provided, between each lens and a respective zone on the imaging surface, so as to conduct light from different areas of the optical image produced by each lens to respective zones on said imaging surface, said zones being selected so that any image distortion, which would otherwise be caused by fisheye lenses or conical reflectors is avoided, or substantially eliminated.

52. Apparatus according to Claim 51, wherein the lens of each optical device focuses light on to optical image forming means which is coupled to a plurality of light guides for conducting components of the image to respective zones on said imaging surface.

53. Apparatus according to Claim 52, including a focussing and masking arrangement which divides the imaging surface into respective zones on which components of optical images from each of said lenses is focussed.

54. Apparatus according to any of Claims 51 to 53, wherein said optical imaging devices are arranged on axes perpendicular to optical axes of a spherical camera arrangement having six lens systems arranged on orthogonal axes.

55. Apparatus according to Claim 54, wherein each lens system comprises a fish eye lens and a plurality of secondary lenses.

56. Apparatus according to Claim 55, wherein at least two of the secondary lenses are selectively moveable towards or away from the fish eye lens to vary the magnification of information captured by the lens system.

57. Apparatus for providing an optional panospherical field of view in a camera having a lens system with a normal field of view substantially as herein described with reference to any of Figures 1 to 40 the accompanying drawings.

58. Surveillance apparatus for capturing image information from a panoramic scene substantially as herein described with reference to any of the accompanying drawings.

59. Apparatus for providing a panospherical field of view and a normal field of view substantially as herein described with reference to any of Figures 1 to 40 and 44 to 52 of the accompanying drawings.

60. Panospherical or omnidirectional imaging apparatus substantially as herein described with reference to any of the accompanying drawings.