GB2354390A - Wide-angle image capture apparatus - Google Patents

Abstract

Disclosed is apparatus for wide-angle image capture comprising a pair of image capture device 301,302, each image capture device being arranged to capture an image from one plane mirror; and a pair of said plane mirrors, each plane mirror being arranged to reflect a portion of a scene to one of said image capture devices, said pair of plane mirrors being arranged in planes which intersect at an edge and form a reflective wedge 303, said edge being between said scene and a line between optical centres of said image capture devices. In the apparatus, the image capture devices have their virtual optical centres substantially lying on a common plane which intersects said edge, further characterised in that said virtual optical centres of said image capture devices are located substantially on a median plane of said wedge. Preferably, the field of view overlap is less than 0.1 and preferably less than 0.06 degrees. The devices can be cameras or charge-coupled devices. The mirrors can be rectangular or trapezoidal. Preferably a plurality of devices (601,603, Fig 6 or 703,704, Fig 7) are provided with either a further wedge mirror (605, Fig 6) or a pair of intersecting wedges (701,702, Fig 7).

Description

2354390 A WIDE-ANGLE IMAGE CAPTURE APPARATUS This invention relates to the

field of image capture apparatus, such as cameras, and particularly to wide-angle and immersive image capture technology.

In the field of image capture technology, there has long been a need for wide-angle images, to increase the illusion of realism given to the viewer by producing the effect of being immersed in a scene rather than of looking through a small,window". For example, in television coverage of some sporting events, it is desirable have a wide field of view to avoid the rapid panning and confusion involved in, for example, tracking the motion of a ball. Several technologies have conventionally been available; for example, various forms of optically distorting lenses and mirrors. Thus, for example, fisheye lenses and catadioptric mirror arrangements have conventionally been used for this purpose. These both achieve wide fields of view, but the images are highly distorted, and require a lot of processing before they can be interpreted and recognised by the human eye. Also, resolution of such images varies, sometimes considerably, across the field of view, giving poor resolution and loss of light values at the edges of the image. It would require considerable re-computation, for example, to take a small section of these images, and project them undistorted onto a conventional TV screen.

Fisheye lenses and catadioptric systems also have the effect of using a single viewpoint to view the entire image. This implies using a single sensor (for example, a CCD chip). Using a single sensor has the advantage of simplifying the setup procedure, as there aLre no colour-matching and registration issues. However, if the very-wide-angle image is required to have a resolution greatly in excess of what current CCD chips can achieve, then the use of multiple sensors is essential, and in turn this requires the ability to superimpose multiple sensors at the same viewpoint.

Another approach is to avoid or obscure the problem: for example, one camera can take multiple images from the same viewpoint, looking in different directions, but at different times. The time discontinuity can be safely ignored for views like countryside scenery, but this cannot be applied to views containing any live action, as for example, moving athletes, motor vehicles, dancers or the like.

in cases where time -discontinuities cannot be tolerated, one trade off is to accept spatial -viewpoint discontinuity instead. in this approach, synchronised cameras, positioned in proximity, are used, and parallax-distortion occurs along the stitching-edges where the separate images are stitched together. This may be acceptable for distant views, but it is unacceptable where distant objects and close-up objects are 2 both present along the stitching edge; that is, where the scene contains objects with large depth -variations.

The problems are particularly apparent when very wide-angle images are made by stitching a number of smaller images together. When the images contain moving objects, that is, they are not of the still-life photo-studio type, the images to be joined need to be taken at the same instant in time. This requires two or more synchronised cameras.

Clearly, the cameras cannot occupy the same physical space, so each image ends up being taken from a different viewpoint. This leads to parallax errors when joining the two-dimensional images, and these parallax errors cannot be ef f ectively corrected, as the necessary 3 -dimensional depthof - objects information has been discarded during the capture of the original two-dimensional images.

If none of the known techniques of avoiding or obscuring the problem is suitable, and if the processing required to correct the effects of distorting lenses and mirrors becomes computationally burdensome, the user is left with stitching and parallax problems.

For the purpose of creating wide and deep views, as for example in immersive television applications or in creating realistic panoramas, there are no current technologies that solve these problems without the associated computational burdens. The closest approach to a solution is that proposed by Nalwa for the purpose of creating a 360-degree view of an area, for use in video teleconferencing or the like, which uses a reflective pyramid and a set of four cameras to produce an all-round viewing capability. Nalwa, in US Patent Number 5,745,305, discloses the positioning of four cameras disposed about a pyramid-shaped element having reflective sides, in which each camera receives a reflected image from one of the reflective sides of the pyramid. The cameras are arranged so that they share a common virtual optical centre. The arrangement proposed by Nalwa solves the problem of providing 360-degree image capture, but does so at the expense of the appearance of realism of the imaged scene, in that only approximately 50 degrees of vertical field of view appear on the resulting image. This is clearly insufficient for the purposes of producing a realistic immersive wide-angle view, for example for use in immersive television, in which it is desirable to have a field of view approaching 180 degrees horizontally and approaching 120 degrees vertically.

The present invention accordingly provides apparatus for wide-angle image capture comprising: a pair of image capture devices, each image capture device being arranged to capture an image from one plane mirror; and a pair of said plane mirrors, each plane mirror being arranged to reflect a portion of a scene to one of said image capture devices, said 3 pair of plane mirrors being arranged in planes which intersect at an edge and form a reflective wedge, said edge being between said scene and a line between optical centres of said image capture devices.

The apparatus is preferably further characterised in that said image capture devices have their virtual optical centres substantially lying on a common plane which intersects said edge.

The apparatus is preferably further characterised in that said virtual optical centres of said image capture devices are located substantially on a median plane of said wedge.

The apparatus is preferably further characterised in that said plane mirrors comprise reflective external surfaces of said wedge. In an alternative, the apparatus may be further characterised in that said plane mirrors comprise reflective internal surfaces of a prism.

Preferably, in the apparatus as described, a field of view overlap between said image capture devices has an angle of less than 0.1 degrees; said angle is preferably 0.06 degrees.

The apparatus is preferably further characterised in that said image capture devices comprise charge-coupled devices.

In the apparatus as described, said plane mirrors may preferably be substantially rectangular, or substantially trapezoidal.

The apparatus of the present invention may be used to produce optical systems comprising a plurality of the apparatus as described.

The present invention addresses the parallax problem by using reflective optics to create "virtual cameras", where a virtual camera is a reflection of a real camera. As these virtual cameras are intangible, they can be superimposed at the same point in space, and this allows us to capture plural images using synchronised cameras which all share the same viewpoint. The requirement to have some overlap between cameras to allow for the subsequent stitching process means in practice that the viewpoint matching between the cameras is not perfect, but the invention is so close to the ideal geometry that the misalignment is so extremely small that it can be ignored for practical purposes.

The invention therefore reduces the scale of the parallax problem so that it is effectively eliminated, and the plural images can be stitched together at the edges with few or no undesired distortions, to provide the required very wide-angle view.

4 Note that because the view-planes of the cameras are not co-planar, there will be foreshortening effects in the pairs of images to be stitched, and these effects are preferably corrected before the images are stitched. The correction to be applied, which is not relevant to the present invention, depends on what geometrical model (flat, cylindrical, spherical, and so on) the final image is to be mapped onto.

For immersive or otherwise realistic image viewing applications, the invention is optically restricted to achieving a theoretical maximum field of view of 180 degrees horizontally and 180 degrees vertically. An attempt to exceed these limits would mean that the cameras would image themselves. (There are, of course, applications in which this does not matter -- an example is wide-angle surveillance, in which the aesthetic effect of the image is irrelevant.) In practice, because of the width of the lenses and of their surrounding mountings, the practical maximum is somewhat less than the theoretical maximum. Nevertheless, the invention advantageously achieves wide angles considerably in excess of those achievable with conventional optical systems. The use of multiple CCDs arranged in a non-coplanar manner also reduces effects such as vignetting (the fall-off in illumination at the extreme edges) that is a problem when attempting wide angle views with single cameras.

A preferred embodiment of the present invention will now be described by way of example, with reference to the drawings in which:

Figure 1 shows a schematic view of a single camera of known type and its horizontal field of view, as seen from above.

Figure 2 shows the optically ideal positioning of a pair of cameras to solve the problem of gaining a wide angle f ield of view.

Figure 3 shows an apparatus according to a preferred embodiment, as seen from above.

Figure 4 shows a schematic view of the optical lines of a part of the apparatus of Figure 3, showing the relationship between a real and a virtual camera and their respective clear fields of view. Again the apparatus is seen from above.

Figure 5 shows the combined horizontal clear fields of view of the pair of cameras comprised in the apparatus of the preferred embodiment.

Figure 6 shows a four-camera apparatus according to an alternative embodiment to give an increased vertical field of view in addition to the increased horizontal field of view.

Figure 7 shows an apparatus according to a further alternative embodiment in which the viewing space is partitioned by a pair of intersecting reflective wedges.

Figure 1 shows a schematic view of a single camera (100) of a known type and its horizontal field of view, as seen from above. The camera of the preferred embodiment is, for example, a conventional 35mm camera with an 18mm focal length lens. This gives the camera a field of view of 90 degrees horizontally, which is, practically, the widest achievable field of view using non-distorting optical equipment. The camera has a viewpoint (101) positioned approximately at the centre of lens (102) The dotted lines (103, 104) indicate the field of view of the camera.

In Figure 2 there is shown an ideal positioning of a pair of the cameras as shown in Figure I to give a wide angle field of view. The figure shows the optical lines of an ideal camera arrangement as seen from above. A first camera has field of view (200), while a second camera has field of view (201). The fields of view (200, 201) overlap by some small amount shown in the figure as angle (202), to allow for the stitching- together of the two images with properly corrected colour balance, and so on. The geometry of this ideal arrangement requires that the two cameras have their viewpoints superimposed at a point (203); this is clearly physically impossible: the viewpoint of each camera is positioned ef f ectively at the centre of its lens (as shown in Figure 1), so to superimpose them would require both cameras to be positioned in the same place at the same time.

Figure 3 shows an apparatus of a preferred embodiment. The apparatus comprises a pair of cameras (301,302) of the type shown in Figure 1, arranged facing opposing sides of a reflective wedge (303).

The reflective wedge (303) might advantageously be a wedge-shaped arrangement of two face-silvered mirrors. Face-silvering is a preferred technique, as it prevents the distortions that arise from the multiple shallow-angle internal reflections that occur in conventional silvered glass mirrors. Preferably, the face-silvering of the two reflective planes that comprise the wedge is carried to the edge at which they intersect to avoid production of artefacts and to improve the optical quality of the image in the vicinity of the stitching region. However, provided that the reflective planes are located in planes that intersect, some discontinuity at or about the intersection of the mirrors may not affect the final image in practical terms. It is also preferable to have the virtual camera images offset very slightly from the median plane of the wedge composed of the plane mirrors. In practice, the angle can be made very small, as a column of no more than two pixels in width may be sufficient to allow for the stitching of the images. with charge-coupled 6 devices of pixel width 3072, for example, an overlap of 2 pixels in an image capture device with a 90 degree field of view gives an angle of:

(2/3072) 90 = 0.06 degrees The parallax errors given by an angular displacement of the virtual cameras of 0.06 degrees is, for practical purposes, negligible.

As will be clear to those skilled in the art, the reflective surfaces used to reflect portions of a scene may be of many types. It is possible, for example, to achieve the optical performance required by using external mirrors on a wedge, as in the face-silvered example described above. Equally well, it is possible, for example, to use one or more optical prisms, arranged with the image capture devices, to achieve the optical performance required. In that case, it is the internal reflections of the prism or prisms that give the image capture devices the benefits of having substantially coincident virtual optical centres.

Figure 4 shows a schematic view of the optical lines of a part of the apparatus of Figure 3, showing the relationship between a real and a virtual camera and their respective clear fields of view. Again the apparatus is shown as if seen from above. For the sake of clarity, the optics of only one of the pair of cameras is shown. Camera (400) is positioned facing one side of reflective wedge (401). A virtual camera (402) is a reflected image of camera (400) having a virtual viewpoint (403) beneath the surface of the reflective wedge (401), offset slightly from the line (404) representing the median plane of the reflective wedge (401). The offset is to give an area of image overlap (405) so that the images from the pair of cameras can be stitched correctly; the overlap (405) allows for brightness and colour adjustments to be applied to give the illusion of continuity across the join-line of the pair of images.

The ideal field of view in this apparatus would comprise the field of view equivalent to the combination of field of view angle (406) and field of view angle (407), but field of view angle (407) is obscured by the body of camera (400). This reduces the practical maximum field of view by a small amount from the theoretical maximum, and means that the practical unobscured field of view is that represented by field of view angle (406). Nonetheless, when combined with the second camera of the preferred embodiment, the apparatus gives an advantageously wide horizontal field of view without significant loss of vertical field of view.

In Figure 5, the apparatus of the preferred embodiment is shown to illustrate the effect of combining the fields of view of the cameras.

The combined field of view made possible by the apparatus of the

7 preferred embodiment is shown as field of view angle (502), and the image overlap area required for image stitching is shown as overlap angle (501).

Figure 6 shows a four-camera apparatus according to an alternative embodiment to give an increased vertical field of view in addition to the increased horizontal field of view. A first pair of cameras (601) is disposed adjacent to a first reflective wedge (602) to form the apparatus of the preferred embodiment, and a second pair of cameras (603) is disposed about a second reflective wedge (604) to form the apparatus of the preferred embodiment. These are then disposed about a third, preferably larger, reflective wedge (605) to form an alternative embodiment giving an increased vertical field of view as well as an increased horizontal field of view. In this alternative embodiment, the third wedge (605) has the effect of "splitting" the world-view into two portions: an upper and a lower portion. Each of these portions is then imaged using the apparatus of the preferred embodiment. Of course, the third wedge (605) may be oriented to split the world view at any convenient angle.

Further alternative arrangements of cameras and reflective wedges are also possible, as shown in Figure 7, which illustrates an apparatus according to a further alternative embodiment a pair of intersecting reflective wedges (701, 702) is used to partition the world-view for image capture by pairs of cameras (703, 704).

8

Claims (12)

1. Apparatus for wide-angle image capture comprising:

a pair of image capture devices, each image capture device being arranged to capture an image from one plane mirror; and a pair of said plane mirrors, each plane mirror being arranged to reflect a portion of a scene to one of said image capture devices, said pair of plane mirrors being arranged in planes which intersect at an edge and form a reflective wedge, said edge being between said scene and a line between optical centres of said image capture devices.

2. Apparatus as claimed in claim 1, further characterised in that said image capture devices have their virtual optical centres substantially lying on a common plane which intersects said edge.

3. Apparatus as claimed in claim 1 or claim 2, further characterised in that said virtual optical centres of said image capture devices are located substantially on a median plane of said wedge.

4. Apparatus as claimed in any preceding claim, further characterised in that said plane mirrors comprise reflective external surfaces of said wedge.

5. Apparatus as claimed in any preceding claim, further characterised in that said plane mirrors comprise reflective internal surfaces of a prism.

6. Apparatus as claimed in any preceding claim, further characterised in that a field of view overlap between said image capture devices has an angle of less than 0.1 degrees.

7. Apparatus as claimed in claim 6, further characterised in that said 35 angle is 0.06 degrees.

8. Apparatus as claimed in any preceding claim, further characterised in that said image capture devices comprise charge-coupled devices.

9. Apparatus as claimed in any preceding claim, further characterised in that said plane mirrors are substantially rectangular.

10. Apparatus as claimed in any of claims 1 to 8, further characterised in that said plane mirrors are substantially trapezoidal.

1

11 9 11. An optical system comprising a plurality of apparatus as claimed in any preceding claim.

12. Apparatus substantially as described herein with reference to any of Figures 3 to 7.