GB2316255A

GB2316255A - Improvements in image stabilisation

Info

Publication number: GB2316255A
Application number: GB9616767A
Authority: GB
Inventors: John Mansbridge; George Malcolm Swift Joynes
Original assignee: Roke Manor Research Ltd
Current assignee: Roke Manor Research Ltd
Priority date: 1996-08-09
Filing date: 1996-08-09
Publication date: 1998-02-18
Anticipated expiration: 2016-08-09
Also published as: GB9616767D0; GB2316255B

Abstract

An image stabilisation method for substantially reducing the effects of camera shake comprises the steps of (i) capturing a first frame representative of the image captured by the camera at a first time, (ii) storing the first frame in a frame store, (iii) capturing a second frame representative of the image captured by the camera at a second time, (iv) measuring an image shift appertaining to a shift of the image captured at the second time with respect to the image captured at the first time, and (v) forming a composite frame by over writing parts of the first frame stored in the frame store with common parts of the second frame in accordance with the measured image shift, thereby ensuring that the image represented by the composite frame will appear stable when viewed.

Description

IMPROVEMENTS IN OR RELATING TO IMAGE STABILISATION The present invention relates to methods for stabilising images captured by cameras.

Furthermore, the present invention relates to apparatus for providing a camera with a means for stabilising an image.

Images captured by video cameras and the like, suffer from a problem known as camera shake, wherein vibrations induce rotation and movement of the video camera, thereby causing a captured image to shake. In particular, but not exclusively, camera shake is a problem for hand held cameras and for cameras mounted on non-rigid bodies, where camera shake is inherent.

Known systems for reducing the effects of camera shake include apparatus for electronically manipulating an image captured by the camera on a display of the camera such that the viewer sees a stable image despite instability in the image captured by the camera. To this end, known electronic stabilisation systems operate to measure an image shift of the image captured by the camera with respect to sensors within the camera. An image shift measurement may be made in various ways, including using sensors to measure an angular movement of the camera. Furthermore, with knowledge of the focal length of the camera lens, a corresponding image shift on a camera sensor may be calculated. Alternatively, features of an image captured by the camera can be tracked using image processing techniques.

Once image shift has been measured, prior art stabilisation systems thereafter operate to display each of a plurality of frames of the image shifted in accordance with the measured image shift to compensate for the shift in the image caused by camera shake, thereby providing a viewer with a displayed scene wherein at least some part of the image appears stable.

It is hereby stated for clarity that an image formed by a camera is a representation of a scene within the field of view of the camera in accordance with light emanating from the scene and passing into the camera, whereas a frame is a signal or an amount of data representative of the image captured at a particular time.

The effects of known electronic image stabilisation systems is illustrated in Figure 1. In Figure l(a) an example of a displayed image effected by camera shake is presented, wherein a plurality of superimposed frames of an image representative of a scene of a car 1, and a lamp post 2, are shown to move to various positions within the displayed image by a display means. The superposition of frames representative of the image of the scene captured at different positions of the camera, has an effect of obscuring visual information of the scene of the car 1 and lamp post 2. Figure l(a) therefore presents a representation of an unstable image of a car 1 and a lamp post 2.

In Figure l(b) there is presented a representation of a displayed image after the known electronic stabilisation system has operated on the unstable image presented in Figure l(a). In Figure l(b), the stabilised image produced by the electronic stabilisation means may now be seen to be centrally located and stable within the display. The electronic stabilisation means operates to shift each frame representing the image of the car 1 and lamp post 2, such that the car 1, and lamp post 2, appear to be stable in the display. However, as the stabilisation process operates to shift the entire image of the car 1 and lamp post 2, part of the display image will not contain any displayed information and will therefore appear blank. Furthermore, as indicated in Figure l(b), solid lines 3, 4, 5, 6, 7, 8, which are representative of the edges of the frames of the displayed image, the edge of the image will be continuously be moving as the electronic stabilisation means operates to present a stable image of the car 1 and lamp post 2 within the scene.

In practice, the known electronic stabilisation system, hereinbefore described, suffers a disadvantage in that although the stabilisation system operates to stabilise an image within the centre of a display, the peripheral region of the displayed image will be continuously moving, which is distracting to the viewer.

To alleviate this problem known systems operate to blank out or crop the moving part of the image, thereby providing a stable representation of the scene within a centre portion of the captured images. Thereafter, the cropped image is expanded to fill a display, such that the blanked border is removed. Figure l(c) therefore provides a representation of a cropped image wherein the edge of the scene 9, has been blanked off, whereas Figure l(d) presents the stabilised image of the scene 10, expanded to fill the display. The known prior art image stabilisers therefore suffer a disadvantage in that although a stable image is produced, the effective image area of a scene provided by the camera in combination with the image stabiliser is substantially reduced.

More especially, when viewing an object of interest, the electronic image stabiliser operates to stabilise an image of the object but peripheral information at the regions of the object appertaining to contextual information will be lost, thereby greatly reducing the value of the image. Therefore, to produce an image of the same scene, a camera must be provided with a shorter focal length which has a further disadvantage that the image will suffer a reduction in spatial resolution.

It is therefore an object of the present invention to provide a method of stabilising an image captured by a camera, wherein the disadvantages of the aforementioned prior art are substantially reduced.

It is a further object of the present invention to provide an image stabiliser for a camera wherein the disadvantages of known stabilisation systems are substantially mitigated.

According to the present invention there is provided a method of stabilising images captured by a camera, comprising steps of; (i) capturing a first frame representative of the image captured by the camera at a first time, (ii) storing the first frame in a frame store, (iii) capturing a second frame representative of the image captured by the camera at a second time, (iv) measuring an image shift appertaining to a shift of the image captured at the second time with respect to the image captured at the first time, and (v) forming a composite frame by over writing parts of the first frame stored in the frame store with common parts of the second frame in accordance with the measured image shift, thereby providing a substantially stable image, represented by the composite frame, when viewed.

By over writing the first frame with common parts from the second frame, any contextual information present in peripheral regions of the image will be retained when displaying the over written composite frame. Thereafter, when further images are captured at subsequent times and frames representative of these captured images are used to over write the information in the frame store, such that common parts only are over written, the composite frame produced will be provided with peripheral contextual information generated from a plurality of captured images each of which was captured at a different tune, and which may correspond to a different position of the camera as the camera has moved in accordance with camera shake. The method of stabilising an image therefore provides a stabilised image of an object with contextual information at the periphery of a scene in which the object lies, without a need for increasing the focal length of the camera, and retains all information captured by the camera.

A displayed image produced in accordance with the method of stabilising images produced by a camera, therefore consists of a last frame of an image captured by the camera, super imposed on a mosaic of parts of previous frames of images. As a result camera shake may result in some areas of a scene not being refreshed for many frames, which may further result in a situation wherein parts of the displayed composite frame are no longer consistent with a latest image captured by the camera.

This situation may occur when a scene is changing rapidly, for example, where cars or people are moving into or out of the scene.

The method of stabilising an image may further comprise a step of high-pass filtering the measured image shift before step (v).

To reduce the impact of inconsistencies in the composite image, the measured image shift may be high-pass filtered so that the stabilisation method does not act to remove camera shake at frequencies below a predetermined cut-off frequency of the filter.

As a result, the stabilised image will appear to move in response to camera movements representative of image shift frequencies below the cut-off. If the cut-off frequency is low enough this slow movement of the image will not be distracting. If it is required that the selected image feature be stable for shake frequencies below the cut-off it will be necessary to ensure that the camera follows the selected object. At low frequencies this will be easy to achieve.

According to an aspect of the present invention there is provided an image stabiliser for stabilising a displayed image captured by a camera, the said camera having means for representing the image as a plurality of frames, each of which frames being representative of the image captured by the camera at different times, the image stabiliser comprising a frame store for storing data representative of the frames, an image shift measurer which operates to measure an image shift appertaining to a shift of the image from one time to another time, and a control unit connected to the image shift measurer and to the frame store which operates to overwrite parts of the frame store which parts are common to the captured image after the captured image has been shifted in accordance with the image shift.

One embodiment of the present invention will now be described by way of example only, with reference to the accompanying drawing, wherein, Figure 2 is a schematic and somewhat conceptual block diagram of an image stabilising system.

In Figure 2 a video camera 20 is positioned such that a scene of interest is within a field of view 26, bounded by solid lines 22 and 24. The video camera 20 may be positioned on an observation platform high above the ground in order that the scene of interest falls within the field of view 26. As such the observation platform may suffer from vibrations caused by actions of wind and other vibrations, thereby causing shaking in the camera 20.

The camera 20 operates to generate frames of information representative of an image captured by the camera 20 of the scene of interest within the field of view 26. Frames that are generated in accordance with analogue signals are communicated from the camera 20 to an analogue to digital converter 28, which operates to generate digital representations of the analogue frames. Digital representations of the analogue frames are thereafter fed to a frame store 30, under control of a control unit 32. The control unit 32 is comprised of an image shift measurer 34 and a control processor 36. The image shift measurer 34 operates to measure a shift in an image captured at a particular time represented as a digital frame generated by the analogue to digital converter in combination with the camera 20, with respect to an image captured at an earlier time. The control processor 36, operates in accordance with the image shift measurer to write the frames representative of images captured by the camera 20 in a frame store 30. Each digital frame fed from the analogue to digital converter 28 is thereby written into the frame store 30 at a position wherein an image or object of interest within an image captured by the camera 20 appears to be stable and wherein common parts of an image are over written by common parts of a subsequent image, thereby providing that the composite image comprises common parts and composite parts generated from frames representative of earlier images. The composite frame stored in the frame store 30, is thereafter fed to an image display means 38, which thereafter communicates the image to a screen 40.

As illustrated in Figure 2, the displayed image on the screen 40, corresponding to the composite frame is comprised of the common parts 42 and the composite parts 44. The image displayed on the screen 40 is represented as a stable image with contextual information from the camera 20 at a periphery of the image 42 being comprised of parts from earlier frames. In this case, the object of interest is the car 1 and the lamp post 2 which appear substantially in the centre of the screen 40. Objects surrounding the car 1 and lamp post 2 are thereby presented at the edges and periphery of the scene which are generated by frames captured from earlier images.

Where the scene displayed by the image stabilising system, shown in Figure 2, is changing rapidly, objects will be moving into and out of the scene. In this case a situation can occur wherein inconsistencies appear in the image displayed by the display means 40, wherein objects remain within the displayed image after they have in reality left the scene. To mitigate inconsistencies in the displayed image the output of the control processor 36 is high-pass filtered so that the stabilisation method does not act to remove camera shake at frequencies below a predetermined cut-off frequency of the filter. If this is implemented the shifted image will not remain in one area of the screen for a long period of time. Instead the shifted image will slowly move back to the centre and the inconsistencies in the image will be overwritten. The cut-off frequency is set so that changes in the scene have a statistically low probability of generating inconsistencies in the displayed image.

To effect panning (i.e. deliberate movement of the image) an image shift appertaining to movement of the image in accordance with the intended panning could be removed from the measured image shift so that the stabilised image moves smoothly, as intended. To this end, a panning signal representative of movement of the camera in correspondence with intended movement of the image, may be provided by a camera control system. Such a camera control system would operate to provide remotely controlled movement of the camera in accordance with control signals generated by an operator, which control signals could furthermore be used to generate the panning signal.

Alternatively the panning signal could be generated by a movement sensor, in response to physically sensed movement of the camera By subtracting the panning signal from the image shift signal a resultant shift signal would be used by the control unit to shift the image in the frame store in sympathy with the intended panning.

The camera 20 shown in Figure 2, may be a video camera, or it may be an infra red camera generating infra red images.

Furthermore, the camera 20 may be provided with a means for generating images in accordance with microwave signals received thereby.

As will be appreciated by those skilled in the art, various modifications may be made to the stabilisation method and apparatus hereinbefore described without departing from the scope of the present invention. In particular, effects of jitter caused when a camera is tracking (panning or tilting) may also be mitigated using this apparatus. In this case the image stabilising system operates to arrange for the images captured in accordance with a tracking motion of the camera to appear to be smooth and relatively still, whilst tracking is being performed. Furthermore, the stabilisation system can be applied to cameras to stabilise images for security purposes or for broadcast applications wherein the camera is used for a sporting event and is mounted on, for example, a grandstand or on a vehicle which is subject to vibration.

Claims

1. A method of stabilising an image captured by a camera, comprises the steps of; (i) capturing a first frame representative of the image captured by the camera at a first time, (li) storing the first frame in a frame store, (iii) capturing a second frame representative of the image captured by the camera at a second time, (iv) measuring an image shift appertaining to a shift of the image captured at the second time with respect to the image captured at the first time, and (v) forming a composite frame by over writing parts of the first frame stored in the frame store with common parts of the second frame in accordance with the measured image shift, thereby providing a substantially stable image, represented by the composite frame, when viewed.

2. A method of stabilising an image as claimed in Claim 1, wherein the measured image shift if filtered before step (v).

3. A method of stabilising an image as claimed in Claim 2, wherein the measured image shift is high-pass filtered before step (v).

4. A method of stabilising an image as claimed in any preceding claim, further including the steps of generating a panning signal representative of a shift of the image in correspondence with a panning movement of the camera, and subtracting the panning signal from the image shift before step (v).

5. An image stabiliser for stabilising a displayed image captured by a camera, the said camera having means for representing the image as a plurality of frames, each of which frames being representative of the image captured by the camera at a particular time, the image stabiliser comprises a frame store for storing data representative of the said frames, an image shift measurer which operates to measure an image shift appertaining to a shift of the image from one time to another time, and a control unit connected to the image shift measurer and to the frame store which operates to overwrite parts of the frame store which parts are common to the captured image after the captured image has been shifted in accordance with the image shift.

6. An image stabiliser as claimed in Claim 5, wherein the image shift measurer comprises at least one sensor which operates to measure angular movement of the camera.

7. An image stabiliser as claimed in Claim 6, wherein the said at least one sensor operates to measure the image shift in accordance with a focal length of the camera lens.

8. An image stabiliser as claimed in Claim 6 or 7, wherein the said image shift measured by the image shift measurer is also representative of a rotation of the image.

9. An image stabiliser as claimed in any one of the Claims 5 to 8, further comprising an analogue to digital converter being connected to the camera which operates to convert analogue frames generated by the camera into digital frames, which digital frames are stored in the frame store.

10. An image stabiliser as claimed in any one of the Claims 5 to 9, further comprising a display means which operates to display images appertaining to the frames stored in the frame store in communication therewith.

11. An image stabiliser as claimed in any one of the Claims 5 to 10, wherein the control unit further comprises a comparator which operates to compare the image shift with a predetermined threshold and wherein the control unit operates to refresh the store with the frame captured at the said another time, where the image shift exceeds the predetermined threshold.

12. An image stabiliser as claimed in any one of Claims 5 to 10, further includes a filter being connected to the image shift measurer which filter operates to filter the image shift before being communicated to the control unit.

13. An image stabiliser as claimed in Claim 12, wherein the filter is a high-pass filter.

14. An image stabiliser as claimed in any preceding claim, wherein the control unit operates to subtract a panning signal representative of movement of the image in correspondence with a panning movement of the camera from the image shift.

15. An image stabiliser as claimed in Claim 14, wherein the panning signal is generated from control signals generated by a camera control system, which camera control system operates to remotely move the said camera in response to the said control signals.

16. An image stabiliser as claimed in Claim 14, wherein the panning signal is generated by a movement sensor in accordance with a sensed movement of the said camera.

17. An image stabiliser as claimed in any one of the Claims 5 to 16, wherein the image is captured by the camera in accordance with microwave, millimetre or ultrasonic signals received thereby.

18. An image stabiliser as claimed in any one of the Claims 5 to 16, wherein the image is captured by the camera in accordance with infra red radiation received thereby.

19. A camera including an image stabiliser as claimed in any one of the Claims 5 to 18.

20. An image stabiliser as hereinbefore described with reference to Figure 2.