WO2017221007A1

WO2017221007A1 - Imaging capturing apparatus and image reconstruction method for reducing a blind spot

Info

Publication number: WO2017221007A1
Application number: PCT/GB2017/051814
Authority: WO
Inventors: Gareth Edwards
Original assignee: Observant Technology Limited
Priority date: 2016-06-21
Filing date: 2017-06-21
Publication date: 2017-12-28
Also published as: GB201610786D0

Abstract

An image capturing apparatus comprises an image capture device for capturing an image, a first mirror disposed so as to reflect light from a scene onto a field of view of the image capture device, and one or more second mirrors disposed so as to be adjacent the first mirror in the field of view of the image capture device. The one or more second mirrors are arranged to reflect light from a region of the scene located in a blind spot of the first mirror onto the image capture device. An image of the scene can be reconstructed by extracting separate images of the first mirror and the one or more second mirrors from an image captured by the image capture device, de-warping the separate images based on known curvatures of the first mirror and the one or more second mirrors, and combining the de-warped separate images to reconstruct an image of the scene by replacing a blind spot in the de-warped image of the first mirror with at least part of the de-warped image of the one or more second mirrors. In this way, a blind spot in the reconstructed image can be reduced, and in some cases eliminated entirely.

Description

Imaging Capturing Apparatus and Image Reconstruction Method for Reducing a Blind Spot

Technical Field

The present invention relates to imaging capturing apparatus and image reconstruction methods for reducing a blind spot in a captured image. More particularly, but not exclusively, the present invention relates to apparatus and methods for panoscopic, multiscopic, or stereoscopic image reconstruction. Background

Panoramic cameras have been developed which use a folded catadioptric design to capture a 360 degree panoramic view in real-time, using a single camera. One such example of a prior art panoramic camera is the "Observer 360" ceiling-mounted surveillance camera developed by Observant Innovations Ltd. This prior art imaging apparatus uses a convex parabolic mirror to capture light from a hemi-spherical field of view. A primary mirror situated in front of the convex mirror reflects an image of the surface of the convex mirror back onto a digital camera. This is known as a folded optical system. The image of the convex mirror can then be de-warped to reconstruct a 360 degree, hemi-spherical field of view.

A drawback of this folded configuration is that a significant blind spot exists behind the primary mirror. Objects situated behind the primary mirror are obscured from the field of view of the main parabolic mirror, and will not be visible in the reconstructed hemi-spherical field of view. It would therefore be desirable to provide an imaging apparatus with improved visibility.

The invention is made in this context.

Summary of the Invention

According to a first aspect of the present invention, there is provided an image capturing apparatus comprising: an image capture device for capturing an image; a first mirror disposed so as to reflect light from a scene onto a field of view of the image capture device; and one or more second mirrors disposed so as to be adjacent the first mirror in the field of view of the image capture device, wherein the one or more second mirrors are arranged to reflect light from a region of the scene located in a blind spot of the first mirror onto the image capture device.

In some embodiments, the image capturing apparatus further comprises: an aperture formed in a surface of the first mirror, wherein the image capture device is disposed behind the aperture; and a primary mirror facing the first mirror, wherein the primary mirror is configured to direct light from the first mirror and the one or more second mirrors through the aperture and onto the image capture device. The blind spot of the first mirror may be a region obscured by the primary mirror. For example, when the primary mirror is circular, the blind spot of the first mirror is a cone of view defined by a circular section of the primary mirror and an apex at a nodal point of the image capture device.

The one or more second mirrors may comprise two second mirrors disposed on opposite sides of the first mirror.

The first mirror and/or the one or more second mirrors maybe convex mirrors. In other embodiments, planar mirrors may be used. The first mirror and/or the one or more second mirrors may have a cross section equal to a whole or part of a conic section. In some embodiments, the first mirror and/or the one or more second mirrors are hyperboloidal.

The image capturing apparatus may further comprise an image processing unit configured to receive the image captured by the image capture device, extract separate images of the first mirror and the one or more second mirrors from the captured image, dewarp the separate images based on known curvatures of the first mirror and the one or more second mirrors, and combine the de-warped separate images to reconstruct an image of the scene. When combining the de-warped separate images, the image processing unit is configured to replace a blind spot in the de-warped image of the first mirror with at least part of the de-warped image of the one or more second mirrors. The reconstructed image may, for example, be a panoscopic, multiscopic, or

stereoscopic image. The image processing unit may be further configured to generate one or more views of the scene from the combined de-warped images, and output the generated one or more views to a display unit. According to a second aspect of the present invention, there is provided a method of reconstructing an image of a scene captured using the image capturing apparatus of the first aspect, the method comprising: extracting separate images of the first mirror and the one or more second mirrors from an image captured by the image capture device; de-warping the separate images based on known curvatures of the first mirror and the one or more second mirrors; and combining the de-warped separate images to reconstruct an image of the scene, by replacing a blind spot in the de-warped image of the first mirror with at least part of the de-warped image of the one or more second mirrors. The method may further comprise: generating one or more views of the scene from the combined de-warped images; and outputting the generated one or more views to a display unit.

According to a third aspect of the invention, there is provided a non-transitory computer-readable storage medium arranged to store computer program instructions which, when executed, perform a method according to the second aspect. The computer program instructions may further be configured to perform any other method steps disclosed herein. Brief Description of the Drawings

Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

Figure 1 illustrates an image capturing apparatus, according to an embodiment of the present invention;

Figure 2 illustrates a cross-section of the image capturing apparatus of Fig. 1, according to an embodiment of the present invention;

Figure 3 illustrates a region of the scene that can be imaged using the second mirrors of the image capturing apparatus of Fig. 1, according to an embodiment of the present invention;

Figure 4 illustrates an example of an image captured using the image capturing apparatus of Fig. 1, according to an embodiment of the present invention; and Figure 5 is a flowchart showing a method of reconstructing and displaying a view of the scene, according to an embodiment of the present invention.

Detailed Description

Figures 1 and 2 schematically illustrate an image capturing apparatus, according to an embodiment of the present invention. Figure 1 illustrates the apparatus in a 3D perspective view, while Fig. 2 illustrates the mirror arrangement used in the apparatus in cross-section. Figures 1 and 2 are provided for illustrative purposes only and are not to scale. As such, the relative size, position and curvature of the mirrors shown in Figs. 1 and 2 should not be taken as limiting, and other geometries may be used in other embodiments.

As shown in Figs. 1 and 2, the image capturing apparatus 100 of the present

embodiment comprises a primary mirror 101, a first mirror 102, and two second mirrors 103. The primary mirror may also be referred to as a First Surface Mirror

(FSM), and the first mirror 102 and second mirrors 103 may be referred to as secondary mirrors. Embodiments of the invention are not limited to two second mirrors 103 as shown in Figs. 1 and 2. In general, one or more secondary mirrors 103 may be provided in other embodiments. The first and second mirrors 102, 103 are located so as to be viewed in part, or entirely, by the reflected Field of View (FOV) of the primary or subsequent surface mirrors. In the present embodiment the first and second mirrors 102, 103 are convex. Preferably, the first and second mirrors 102, 103 are configured to have a cross-section which is equal to a conic section, for example a hyperbola. When a conic section is used, a simplified dewarping algorithm can be used to transform the distorted image to the correct proportions. However, more complex dewarping operations are possible when non-conic section mirrors are used. Depending on the particular embodiment, each of the primary mirror 101, first mirror 102, and second mirrors 103 may be planar or curved. The apparatus 100 further comprises an image capture device 210, for example a digital camera for capturing still images or video. Preferably, the image capture device 210 comprises a digital camera with a prime (fixed focal length) lens. The image capture device 210 is disposed behind the aperture, and the nodal point of the lens is aligned so that the whole or part of the primary mirror 101 lies within the FOV of the image capture device 210. The second mirrors 103 are disposed so as to be adjacent the first mirror 102 in the FOV of the image capture device 210. The primary mirror 101 faces the first mirror 102, and is configured to direct light from the first mirror 102 and the second mirrors 103 through an aperture 104 formed in the surface of the first mirror 102. In this way, the first mirror is arranged to reflect light from the scene onto the FOV of the image capture device 210, via the primary mirror. This arrangement is known as a folded geometry.

A drawback of conventional folded systems is that a significant blind spot exists behind the primary mirror. In embodiments of the present invention, as shown in Fig. 2, the use of one or more second mirrors 103 enables a region of the scene located in the blind spot 240 of the first mirror 102 to be imaged. The second mirrors 103 are arranged to reflect light from a region of the scene located in the blind spot 240 onto the image capture device 210, via the primary mirror 101. Although a folded geometry is used in the present embodiment, in other embodiments the primary mirror 101 and aperture 104 may be omitted and the first and second mirrors 102, 103 may be imaged directly. For example, an image capture device located along the central axis of the apparatus, below the first mirror 102 in the orientation shown in Fig. 2, could be used to directly capture an image of the first and second mirrors 102, 103. In such embodiments a blind spot would still exist, since the image capture device would itself obscure any objects disposed directly behind the image capture device. By providing one or more second mirrors 103 as shown in Figs. 1 and 2, objects in this blind spot could still be imaged. In the present embodiment, two second mirrors 103 are provided, disposed on opposite sides of the first mirror 102. In another embodiment a single second mirror could be used. To provide full coverage of the blind spot, the single second mirror would need to extend further away from the edge of the first mirror, increasing the physical size of the image capture apparatus. Additionally, the camera would need to be configured to capture a large FOV in order to image the whole surface of the larger second mirror. This would mean imaging a larger area around the first mirror, increasing the number of wasted pixels which would be discarded during processing, and therefore decreasing the effective resolution of the final reconstructed image. Hence more than one second mirror is preferred. By using mirrors disposed on opposite sides of the first mirror, the overall physical size of the apparatus can be minimised. In the present embodiment, the apparatus 100 further comprises an image processing unit 220 configured to reconstruct an image of the scene by dewarping and combining images of the first and second mirrors 102, 103, generate one or more views of the scene from the combined dewarped images, and output the generated one or more views to a display unit 230. In other embodiments, an operator may choose to view the image captured by the image capture device 210 directly without further image processing.

Referring now to Fig. 3, a region of the scene that can be imaged using the second mirrors of the image capturing apparatus of Fig. 1 is illustrated, according to an embodiment of the present invention. Figure 3 illustrates the blind spot cone due to the presence of the primary mirror 101. In Fig. 3 the dashed lines indicate the outline of the blind spot cone. Objects within the blind spot cone will not be visible in the reflection of the scene in the first mirror 102, as a result of being obscured by the primary mirror 101. The shaded areas in Fig. 3 illustrate the regions of the scene that are visible in the surface reflections from the left and right second mirrors 103 respectively. Together, the second mirrors 103 provide full coverage of the blind spot.

As well as reflecting light from the blind spot region 240, the second mirrors 103 are arranged to reflect light onto the image capture device 210 from part of the scene that is also visible to the image capture device 210 in the reflection from the first mirror 102. This arrangement ensures that there will be a region of overlap in the dewarped images of the first and second mirrors 102, 103. To put it another way, the first and second mirrors 102, 103 and the image capture device 210 are arranged such that the same part of the scene will be present in the dewarped image of one of the second mirrors

103 and the dewarped image of the first mirror 102. The overlap between the dewarped images of the first and second mirrors 102, 103 can be used to correctly rotate, scale and position the dewarped images relative to one another when stitching the separate images together into a composite image. For example, an algorithm for generating a composite image from the individual dewarped images can determine that the dewarped images of the first mirror 102 and one of the second mirrors 103 are correctly aligned when there is a high degree of correlation between the overlapped regions.

As shown by the shaded regions in Fig. 3, in the present example each second mirror 103 is arranged to reflect light from a part of the scene 303a, 303b within the blind spot that is not imaged by other mirrors in the configuration. In addition, each second mirror 103 is arranged to have a region of overlap 302a, 302b with part of the scene that is imaged by the first mirror 102, allowing the dewarped images of the second mirrors 103 to be stitched together with the image of the first mirror 102. Furthermore, in the present embodiment the second mirrors 103 are also arranged to have a region of overlap 303c with each other, such that another part of the scene is visible in the images of both second mirrors 103, enabling the individual images of the second mirrors 103 to be stitched together into a composite image of the blind spot. The composite image of the blind spot can then be combined with the dewarped image of the first mirror 103 to obtain the final complete image of the scene. In other

embodiments the dewarped images may be combined simultaneously or in a different order, without first obtaining a composite blind spot image as described above.

As described above, in the present embodiment the blind spot arises due to the use of a primary mirror. However, a blind spot may still exist when a non-folded geometry is used, as objects directly behind the viewpoint from which an image of the first mirror is captured will not be visible in the reflection from the first mirror. In such

embodiments, one or more second mirrors may be used as described above, to reduce or eliminate the blind spot in the reconstructed image of the scene. When the object which causes the blind spot is circular, for example when a circular primary mirror is used as shown in Fig. 1, the blind spot of the first mirror 102 is a cone of view defined by a circular section of the primary mirror and an apex at a nodal point of the image capture device 210, as illustrated in Fig. 3. In other embodiments the blind spot may have a different geometry, depending on the particular shape and position of the obscuring object.

Referring now to Fig. 4, an example of an image of the primary mirror captured by the image capture device is illustrated, according to an embodiment of the present invention.

To reconstruct an image of the scene, the image processing unit 220 may be configured to receive the image captured by the image capture device, as shown in Fig. 4, and extract separate images of the first mirror and the one or more second mirrors from the captured image. The size, shape and location of the areas in the image which contain the reflections of the first and second mirrors 102, 103 can be known in advance and pre-programmed in to the image processing unit 220, based on a known geometry of the imaging apparatus 100. In some embodiments, the separate images could be extracted by using an edge-detection algorithm to automatically detect the edges of the first and second mirrors 102, 103 in the image. This approach will enable the system to cope with unexpected changes in the geometry during use.

After extracting the separate images of the first and second mirrors 102, 103, the image processing unit 220 is configured to dewarp the separate images based on known curvatures of the first mirror and the one or more second mirrors, and combine the dewarped separate images to reconstruct an image of the scene. A blending algorithm may be used when combining the images. As explained above, the dewarping algorithm can be simplified by using first and/or second mirrors 102, 103 with hyperboloidal geometry, or other conic sections. When combining the dewarped separate images, the image processing unit 220 is configured to replace a blind spot in the dewarped image of the first mirror with at least part of the dewarped image of the one or more second mirrors.

Various types of image may be reconstructed in embodiments of the present invention, for example panoscopic, stereoscopic, multiscopic, or inverted multiscopic images. In an embodiment of the invention, a panoscopic image can be created by transforming the First Surface Mirror Reflection (FSMR) of an axially aligned circular curved surface into a panoramic view. In another embodiment, a stereoscopic image can be created by transforming the FSMR of two identical curved surfaces, located with a radial offset providing the required inter-ocular separation, into left/right views. In a stereoscopic image, the left/right eye viewpoints are displayed to provide a 3-dimensional (3D) effect.

In another embodiment of the invention, multiscopic images can be created in a similar manner to a stereoscopic image, but using multiple mirror surfaces (planar, curved, annular etc.). The term 'multiscopic' is used herein to refer to a display or image that simultaneously provides two or more views of the same scene. In a 3D multiscopic image, multiple angles are displayed at once to allow a viewer to move around the 3D subject and view it from different angles.

In some embodiments, a plurality of reconstructed images may be used for multiscopic photogrammetry, which is a method for enabling exact positions and motion pathways of reference points to be tracked. The reference points maybe on the surface of a moving object, on its components, and/or in the adjacent environment. In some embodiments, images captured from a plurality of panoscopic, stereoscopic, multiscopic and/or inverted multiscopic non-co-located image capturing apparatuses, similar to those described above, can be integrated into a single spatially coherent network photogrammetry solution.

In yet another embodiment, an inverted multiscopic image can be constructed. An inverted 3D multiscopic view is comprised of multiple angle (typically vertical) sections/slices within a single image. An inverted Multiscopic view can be created using an imaging assembly which comprises an axially aligned annular curved mirror disposed around an axially aligned circular curved mirror.

For stereoscopic or multiscopic image reconstruction, additional mirror assemblies such as the one shown in Figs. 1 and 2 may be provided to enable additional viewpoints to be captured. Depending on the embodiment, a separate image capture device may be used to capture images from each mirror assembly, or a single image capture device may be used. For example, when two mirror assemblies are provided to enable stereoscopic image reconstruction, the primary mirrors of both assemblies may be configured to direct light from the first and second mirrors towards the same image capture device, so that both mirror assemblies are visible in the FOV of the image capture device.

In the embodiment shown in Figs. 1 and 2, two separate second mirrors 103 are provided on opposite sides of the first mirror 102, resulting in an image similar to the one shown in Fig. 4 being captured by the image capture device 210. In an alternative embodiment, the second mirrors 103 illustrated in Figs. 1 and 2 can be replaced by one or more full or partial annular curved surfaces around the axially aligned circular first mirror 102. In such an embodiment, the blind spot view can be digitally extracted from the image captured by the image capture device, which may be referred to as a FSM "assembly" image, and composited into the centre of the FSM image so as to reconstruct a complete view of the scene.

Referring now to Fig. 5, a flowchart is illustrated showing a method of reconstructing and displaying a view of the scene, according to an embodiment of the present invention. The method steps in Fig. 5 correspond to the functions performed by the image processing unit 220 in the embodiment of Fig. 2. For the sake of brevity, a detailed description of similar functions will not be repeated here. Depending on the embodiment, any of the method steps illustrated in Fig. 5 may be implemented using dedicated hardware, or by software instructions executed on a processor. In software- based embodiments, the image processing unit 220 can include a suitable non- transitory computer-readable storage medium arranged to store computer program instructions for performing the method.

First, in step S501 the image processing unit 220 receives an image from the image capture device 210. In other embodiments, the method may be performed on archived images retrieved from storage rather than being performed in real-time on an image directly received from the image capture device 210. Then, in step S502 separate images of the first and second mirrors 102, 103 are extracted, and in step S503 the separate images are de-warped based on known curvatures of the first and second mirrors 102, 103. Next, in step S504 the de-warped images are combined to reconstruct an image of the scene, by replacing a blind spot in the de-warped image of the first mirror 102 with at least part of the de-warped image of the one or more second mirrors 103. In this way, a full image of the scene can be reconstructed without a blind spot. In the present embodiment, the combined image is then used to generate one or more views of the scene in step S505, and in step S506 the generated one or more views are outputted to a display unit 230. For example, when a mirror assembly such as the one shown in Figs. 1 and 2 is used, the combined image can be mapped onto a virtual hemispheric surface. One or more virtual cameras can then be positioned within the virtual hemisphere, and a view from each virtual camera can be generated and displayed on the display unit 230.

Embodiments of the invention have been described in which one or more second mirrors are provided to enable a blind spot to be reduced or eliminated in a

reconstructed image of a scene. In some embodiments, tertiary mirrors can be provided to extend the FOV of the reconstructed image further, for example to provide a 'look-up' capability in a surveillance camera embodiment such as the one shown in Figs. 1 and 2. For a 'look-up' capability, one or more tertiary mirrors may be positioned so as to be visible in a FOV of the first or second mirrors 102, 103. As with the primary, first and second mirrors 101, 102, 103, any tertiary mirrors in an embodiment of the present invention may be planar or curved. In some embodiments of the invention, an extrinsic calibration process can be performed to enable the image processing unit 220 to generate an accurate three- dimensional reconstruction of a scene, as follows. The image capturing apparatus 100 can be constructed such that at a given distance from a plane, for example the floor of a room in which the apparatus 100 is located, two or more of the second mirrors 103 are arranged to reflect light that shares a precisely calculated convergent point directly below the axial centre of the apparatus 100. An object located at this convergent point will therefore appear in a known location in each of the images captured from the second mirrors 103.

If the image capturing device 100 is then used to acquire a view in which a feature is located at this convergent point, that is to say, on the plane and at the given distance directly below the axial centre of the apparatus, the dimensions of this feature can then be extrinsically calculated based on the known geometry of the system. In addition, the distance of another feature from this feature can be extrinsically calculated.

Alternatively, if the apparatus is not located at the given distance from any feature in the scene when a view is acquired, then extrinsic calibration may be performed by either placing a planar graphic such as a grid of known intervals on a suitable planar surface, or by identifying features on a suitable planer surface that are present in the reflections from at least two second mirrors 103. The suitable planar surface may be any planar surface which is perpendicular to the axial centre of the apparatus, for example, the floor in a room in which the imaging apparatus 100 is located. Since the dimensions and scale of the planar graphic is known, the distance of the image capturing apparatus from the planar surface can then be calculated.

By placing a suitable planar graphic in various measurable positions and orientations within the scene imaged by the first and second mirrors, embodiments of the invention can enable complex three-dimensional (3D) spatial reconstructions of the scene to be extrinsically calculated. For example, the extrinsic calculation process to reconstruct a 3D model of the scene can be carried out by performing 3D corner point extrinsic triangulation, followed by 3D least squares extrinsic calculation, and then applying non-linear extrinsic calculations based on 3D spatial constraints. Whilst certain embodiments of the invention have been described herein with reference to the drawings, it will be understood that many variations and modifications will be possible without departing from the scope of the invention as defined in the

accompanying claims.

Claims

1. An image capturing apparatus comprising:

an image capture device for capturing an image;

a first mirror disposed so as to reflect light from a scene onto a field of view of the image capture device; and

one or more second mirrors disposed so as to be adjacent the first mirror in the field of view of the image capture device, wherein the one or more second mirrors are arranged to reflect light from a region of the scene located in a blind spot of the first mirror onto the image capture device.

2. The image capturing apparatus of claim 1, further comprising:

an aperture formed in a surface of the first mirror, wherein the image capture device is disposed behind the aperture; and

a primary mirror facing the first mirror, wherein the primary mirror is configured to direct light from the first mirror and the one or more second mirrors through the aperture and onto the image capture device.

3. The image capturing apparatus of claim 2, wherein the blind spot of the first mirror is a region obscured by the primary mirror.

4. The image capturing apparatus of claim 3, wherein the primary mirror is circular, and the blind spot of the first mirror is a cone of view defined by a circular section of the primary mirror and an apex at a nodal point of the image capture device.

5. The image capturing apparatus of any one of the preceding claims, wherein the one or more second mirrors comprise two second mirrors disposed on opposite sides of the first mirror.

6. The image capturing apparatus of any one of the preceding claims, wherein the first mirror and/or the one or more second mirrors are convex mirrors.

7. The image capturing apparatus of any one of the preceding claims, wherein the first mirror and/ or the one or more second mirrors have a cross section equal to a whole or part of a conic section.

8. The image capturing apparatus of claim 7, wherein the first mirror and/ or the one or more second mirrors are hyperboloidal.

9. The image capturing apparatus of any one of the preceding claims, further comprising:

an image processing unit configured to receive the image captured by the image capture device, extract separate images of the first mirror and the one or more second mirrors from the captured image, de-warp the separate images based on known curvatures of the first mirror and the one or more second mirrors, and combine the de- warped separate images to reconstruct an image of the scene,

wherein when combining the de-warped separate images, the image processing unit is configured to replace a blind spot in the de-warped image of the first mirror with at least part of the de-warped image of the one or more second mirrors.

10. The image capturing apparatus of claim 9, wherein the reconstructed image is a panoscopic, multiscopic, or stereoscopic image.

11. The image capturing apparatus of claim 9 or 10, wherein the image processing unit is further configured to generate one or more views of the scene from the combined de-warped images, and output the generated one or more views to a display unit.

12. A method of reconstructing an image of a scene captured using an image capture apparatus comprising an image capture device for capturing an image, a first mirror disposed so as to reflect light from a scene onto a field of view of the image capture device, and one or more second mirrors disposed so as to be adjacent the first mirror in the field of view of the image capture device, wherein the one or more second mirrors are arranged to reflect light from a region of the scene located in a blind spot of the first mirror onto the image capture device, the method comprising:

extracting separate images of the first mirror and the one or more second mirrors from an image captured by the image capture device;

de-warping the separate images based on known curvatures of the first mirror and the one or more second mirrors; and

combining the de-warped separate images to reconstruct an image of the scene, by replacing a blind spot in the de-warped image of the first mirror with at least part of the de-warped image of the one or more second mirrors.

13. The method of claim 12, further comprising:

generating one or more views of the scene from the combined de-warped images; and

outputting the generated one or more views to a display unit.

14. A non-transitory computer-readable storage medium arranged to store computer program instructions which, when executed, perform the method of claim or 13.