WO1998034400A1

WO1998034400A1 - Scene-motion based dynamic shutter speed adjustment system and method

Info

Publication number: WO1998034400A1
Application number: PCT/JP1998/000388
Authority: WO
Inventors: Muhammed Ibrahim Sezan
Original assignee: Sharp Kabushiki Kaisha
Priority date: 1997-01-31
Filing date: 1998-01-30
Publication date: 1998-08-06
Also published as: DE69809191D1; EP0956695B1; JP2001511961A; EP0956695A1; DE69809191T2

Abstract

A shutter speed adjustment system adjusts shutter speed based on scene motion. A video camera equipped with the system includes an image pick up element (12) therein, which is set to operate at a pre-set shutter speed. A capturing mechanism (16) captures plural images from the image pick up element. A motion sensor/estimator (18) detects relative motion between the plural images and a scene motion parameter is defined (19). A shutter speed controller (20) sets a new shutter speed (22) as a function of the pre-set shutter speed (14) and the scene motion parameter. The method of adjusting the shutter speed based on motion within a scene includes the steps of: capturing plural images at a pre-set shutter speed; determining a scene-motion parameter based on the relative scene motion between the plural images; and setting a new shutter speed as a function of the pre-set shutter speed and the scene-motion parameter.

Description

DESCRIPTION

SCENE-MOTION BASED DYNAMIC SHUTTER SPEED ADJUSTMENT SYSTEM

AND METHOD

Field of the Invention This invention relates to a scene-motion based dynamic shutter speed adjustment that determines a quantitative scene-motion parameter and adjusts the shutter speed in the camera to minimi7.fi blur and to optimize capture of the scene on video tape. Background of the Invention

There are a number of still and video cameras that are known that will automatically adjust shutter speed based on a detected motion component of the image being captured. Some of these known devices utilize an auto-focus mechanism to determine scene motion, which, unfortunately, limits the detected motion to motion along the optical axis. While such a limitation is advantageous for some purposes when used in a still camera, it does little to eliminate blur in the case of a video camera trying to capture motion moving with a perpendicular component to the optical axis. Other known devices adjust shutter speed, but do not relate the newly adjusted shutter speed to the existing shutter speed. All known devices are operable only to increase shutter speed. There are many instances when it is desirable to provide an automatic decrease in shutter speed, along with a decrease in lens aperture, thereby to provide a greater depth-of-field.

It is desirable to utilize as high a shutter speed as possible to capture a moving object, particularly when an object is moving perpendicular to the optical axis of the camera lens. At the same time, it is desirable to maintain, under normal circumstances, a small aperture to maintain a larger depth of field so that as much of the image as possible will be in focus. These two opposing criteria are generally accommodated by some form of automatic exposure control which relates the shutter speed, aperture, and the lighting conditions to optimize the image, usually by providing a small aperture and a low shutter speed. Such features are generally combined with an automatic focusing mechanism. On many video, and some still, cameras, an image stabilization system is provided to detect camera motion and to compensate for such motion. As is well known to those of skill in the art, as the shutter speed increases, the aperture must also increase to admit the requisite amount of light, resulting in a decrease in depth of field, which also decreases the image sharpness.

It is an object of this invention to provide a dynamic shutter speed adjustment mechanism, and method for dynamically adjusting the shutter speed, such that the shutter speed is increased as scene motion increases.

It is another object of this invention to provide a dynamic shutter speed adjustment mechanism, and method for dynamically adjusting the shutter speed, such that the shutter speed is decreased as the scene motion decreases.

Another object of the invention is to provide a dynamic shutter speed adjustment mechanism and method which adjusts shutter speed based on the dominant motion in a scene.

Yet another object of the invention is to provide a dynamic shutter speed adjustment mechanism that is based on the current shutter speed.

A further object of the invention is to provide a dynamic shutter speed adjustment mechanism and method which adjusts shutter speed based on scene motion regardless of whether that motion is perpendicular or parallel to the optical axis of the camera lens.

Summary of the Invention

The shutter speed adjustment system, or mechanism, of the invention adjusts shutter speed based on scene motion. A video camera equipped with the system of the invention includes an image pick up element therein, which is set to operate at a current shutter speed. A capturing

mechanism captures plural images from the image pick up element. A motion sensor/estimator detects and estimates relative motion between the plural images and determines a scene motion parameter, which is a quantitative indication of scene motion content A shutter speed controller sets a new shutter speed and aperture as a function of the current shutter speed and the scene motion parameter. The mechanism may operate in various modes.

The method of the invention adjusts the shutter speed based on motion within a scene and includes the steps of: capturing plural images at a current shutter speed; determining a scene- motion parameter based on the relative scene motion between the plural images; and setting a new shutter speed as a function of the current shutter speed and the scene-motion parameter.

These and other objects and advantages of the invention will become more fully apparent as the description which follows is read in conjunction with the drawings. Brief Description of the Several Views of the Drawing Fig. 1 is a block diagram depicting the shutter speed adjustment system of the invention.

Fig. 2 is a flow chart depicting the method of the invention.

Detailed Description of the Preferred Embodiment

The shutter speed adjustment system and method of the invention is primarily intended for use with video cameras, and specifically for use with consumer digital video recording equipment

(DVCR), however, the system and method may also be suitable for use with conventional, analog video recording equipment, and with still cameras, whether film-based, advanced photo system or digital.

Tiirning initially to Rg. 1, a block diagram representing the shutter speed adjustment

system of the invention is depicted generally at 10. It should be appreciated that all of system 10 is incorporated within a video camera. The video camera includes an image pick up element 12, which may be some form of charge-coupbd device (CCD). Image pick up element 12 is operating at what is referred to herein as a current, or pre-set, shutter speed (S_PRE) 14.

A capturing mechanism 16 is operable to capture plural images, designated I_α and I₂ from image pick up element 12. There may be several components of the video camera which operate as a capturing mechanism, including the video tape or some other form of memory device, however, as used herein, "capturing mechanism" designates a memory device, which is most likely some form of solid state integrated circuit

Data from capturing mechanism 16 is transmitted to a motion sensor/estimator 18, also referred to herein as a motion sensor, which performs analysis on the plural images, I_lt I₂, and determines what is referred to herein as a scene-motion parameter (M). Sensor/estimator 18 includes a motion estimator, 18a, which estimates motion on the basis of real motion vectors, as determined between successive, usually sequential, images. Sensor/estimator 18 then defines M, block 19, which is a quantitative indication of the dominant motion content of the scene. Motion sensor 18 may be part of an image stabilization system, or may be a separate camcorder component

A shutter speed controller 20 analyzes pre-set shutter speed 14 and scene-motion parameter M to determine a new shutter speed (SNE _I/) 22. New shutter speed 22 then becomes the current shutter speed which operates with image pick up element 12. As the scene-motion parameter M changes as a result of motion within the scene detected by image pick up element 12, the shutter speed controller continues to change the shutter speed, depending on the operation mode, which will be described later herein. Changes in the shutter speed are, of course, constrained by lighting and available aperture, and appropriate safeguards are built into shutter speed controller speed 20 to limit the increase of shutter speed so that

is not so fast as to cause the image to become too dark, and also to limit any decrease in shutter speed so that the image does not become overexposed. In this regard, shutter speed control 20 also regulates the lens aperture.

Video cameras, or camcorders, utilize an electronic shutter mechanism. This mechanism determines the amount of time that an image is allowed to remain on image pick up element 12 before it is transferred to the recording mechanism of the camcorder, be that recording mechanism video tape or some form of integrated circuit memory. Examples of shutter speeds are shown in Table 1, both as for S_PRE and S_lfEW. At low shutter speeds, i.e., l/60th, 1/lOOth, l/250th, the image will suffer from motion blur if there is significant motion in the scene.

The extent of the motion blur will depend on the amount of motion and the shutter speed. Scene motion that is perpendicular to the optical axis of the lens is particularly susceptible to motion blur. However, any motion that has a component perpendicular to the optical axis may be blurred if its motion is relatively fast If the shutter speed is low, the blur will limit the visual quality of the video. This is especially true if a still image is generated from the video. The invention described herein analyzes the motion in the scene and, if appropriate, increases the shutter speed to eliminate blur. It is difficult to remove motion blur via post-processing without introducing objectionable, spurious artifacts, such as edge ringing, to the final image. In one mode of operation, when the amount of motion within the scene decreases, the shutter speed is reset to its original value. In the second mode of operation, when the scene motion decreases, the shutter speed may be reset to its original value, or, if appropriate, to a lower value.

Video cameras utilize a number of mechanisms for capmring an image. Additionally, video may be interlaced or non-interlaced. In the case of interlaced video, a first field provides an image in alternating rasters of the image pick up element and the next image fills in the rasters which were not filed during the first capture. The images in this case are referred to as image fields, with two successive image fields being required to form a complete image. In the case of non-interlaced video, all of the rasters are filled in a single pass, making what is referred to as a "frame," not unlike a frame of movie film. Although the motion sensor could select any fields or frames for analysis of the scene motion parameter, the most practical technique is to analyze two sequential, successive fields or frames to determine the scene motion parameter. As used herein, I_x designates the first field or frame, and I₂ designates the second frame or field.

The motion content of the scene may be determined in a number of ways. The preferred method of this invention is to utilize the dominant motion component of the scene in deternύning the motion content Dominant motion is in general due to camera movement i.e., panning, shaking, or camera zoom. Dominant motion may also be due to an object whose size occupies a substantial portion of the image field and which is in rapid movement By far the main concern, particularly for DVCR, is with dominant motion that is the result of a fast moving camera, or rapid zoom.

Dominant motion may be modeled using a four parameter rotation-zoom-translation model:

(1)

here which has the parameters a_v α₂, α₃, and α₄. The coordinates [xy] at frame I_j are related to coordinates [uv] in frame I₂ using the four-parameter motion transformation. All of the points in the

[xy]-plane are rotated by an angle theta about the origin, scaled (zoom) by k, where k is a constant and translated by [α₃ αj. In that case, β_j = fccos θand α₂ = -fcsinθ .

Dominant motion may also be modeled by a six-parameter affine model that adds directional scaling and sheer to the four parameter model, resulting in:

u = α „x + a + a 13 (2)

v = α„ ₂₁xx ++ αα^y + «_β (₃)

Parameters of the dominant motion model (rotation-zoom-translation or affine) may be estimated by using a multi-resolution, iterative algorithm. J. Bergen et al., "A Three-Frame Algorithm for Estimating Two-Component Image Motion, " IEEE Trans. Pattern Anal. & Mach. Intel., Vol. 14, No. 9, pp886-896, September 1992. It is also possible to use higher order models, such a 8-parameter perspective models. S. Mann et al., "Video Orbits of the Projective Group: A New Perspective on Image Mosaicing, " MIT Media Laboratory Perceptual Computing Section Technical Report No. 338, 1995.

In one embodiment of this invention, dominant motion is modeled by the rotation- zoom-translational model In this embodiment only the translational parameters are estimated, while the motion content is determined on the basis of the magnitude of the translational component i.e., 1/2

M (αj + α _f ² ₄\_/ (4)

In other embodiments, both translation and rotation components may be used in deriving the scene motion parameter. An appropriate function of the parameters α _1? α₂, α₃, and α₄ may be defined as an indicator of the motion content i.e.,

M - / (α, , α₂ , α₃ , α₄) = (αj +

. Other motion estimation schemes may be used, such as tieing the motion sensor to an image stabilization system, whether electronic or optical. Such an image stabilizer provides an estimate of the motion, which is usually some type of dominant motion estimation. The output of the motion estimator of the image stabilizer may be then directed, along the current shutter speed, to the shutter speed controller. The automatic shutter speed adjustment mechanism of the invention has two modes of operation: in the first mode of operation, the shutter speed may only increase beyond the speed that is manually set by the user, or which is the default setting of the camera, referred to herein as the preset shutter speed. In the second mode of operation, the shutter speed continuously changes according to the scene motion parameter, M. regardless of the initial setting. Referring now to Table 1, the first mode of operation will be described. As previously noted, in this mode, the shutter speed only increases above its initial, or pre-set value, and, as also previously noted, is constrained by the light available to capture the scene. In this mode, when recording ceases, the shutter speed is reset to the original, pre-set value. Unless there is sufficient motion in the scene to trigger a shutter speed increase, the shutter speed is not changed. In Table 1, the shutter speed values along the left side of the Table are the pre-set values, S_PRE, and the values along the top are the new shutter speed values after the adjustment mechanism of the invention has

been implemented, S_NE_W-

SNEW (seconds)

°PRE

(Sec.)

Table 1

In Table 1, ai, bi, ci, di, ei and fi are threshold values for the scene motion content parameter M, where i is an integer between 1 and 6 for this particular set of shutter speeds. Different cameras may have different shutter speed capabilities. Examples of how the adjustment mechanism of the invention will adjust the shutter speed follow:

If the pre-set shutter speed is 1/60 second, for 0 < M < al, the shutter speed does not change. If a2 < M < a3, the shutter speed increases to 1/250 sec. If the motion content parameter increases further, such that a5 < M < a6, the shutter speed increases to 1/7000 sec. If the motion content parameter decreases such that 0 < M < al, the shutter speed will decrease to the original preset value of 1/60 sec.

If the current, or pre-set shutter speed, is 1/1000 sec, and dl < M < d2, then the shutter speed will be increased to 1/4000 sec. If the scene motion content decreases to 0 < M < dl, the shutter speed will return to its original, pre-set value of 1/1000. In this mode, the shutter speed will never decrease below its original, pre-set value. If, in this mode, the shutter speed is set to 1/10000 sec, the speed will not change regardless of the scene motion content parameter.

It should be appreciated that in the first mode of operation, the adjustment mechanism allows for conservative up-shifting as the pre-set value increases. This is accomplished by having bi > a(i+l), for i = 1, 2, 3, 4, 5; ci > b(i+l), for i = 1, 2, 3, 4; di > c(i+l), for i = 1, 2, 3; ei > d(i+l), for i = 1, 2; and fl > e2. This schema is used to insure that the adjustment mechanism is sufficiently robust to prevent the overestimation of scene motion content, which, in practice, may result in darker images. This mode also provides protection against image blur when the motion momentarily exceeds the level at which the pre-set value may not be sufficiently large.

Table 2 depicts the second mode of operation, in which the shutter speed continuously changes according to the scene motion parameter M, regardless of the initial setting. This mode may be implemented using the constraints of Table 2 between the value of M and the shutter speed S, which is allowed to constantly change, and which is not limited to a lowest speed of the initial, or preset value.

Table 2

Referring now to Fig. 2, the method of the invention, depicted generally at 30, will

be summarized. Initially, images I_j, I₂ are captured at a current or pre-set shutter speed, block 32. Motion sensor/estimator 18 then estimates scene motion, block 34, on the basis of real motion vectors as determined, in the preferred embodiment between sequential images. The scene motion parameter, M, is then defined, block 35, based on the scene motion between I_x and I₂. The pre-set shutter speed 14 is read, block 36, and shutter speed controller 20 sets a new shutter speed, block 38. Analysis of scene motion continues, as previously described, to further increase or decrease the shutter speed, depending on scene motion and lighting. Thus, an automatic shutter speed adjustment system, and a method for automatically adjusting the shutter speed, has been disclosed. The system and method of the invention provide for an appropriate increase or decrease of shutter speed to match scene motion as captured by a image pick up element in a video camera.

Although a preferred embodiment of the invention, and a variation thereof have been disclosed herein, it should be appreciated that further variations and modifications may be made to the invention without departing from the scope of the invention as defined in the appended claims.

Claims

1. A shutter speed adjustment system for adjusting shutter speed based on scene motion,

comprising: a video camera having an image pickup element therein and which is set to operate at a pre-set shutter speed (S_PRE); a capturing mechanism for capturing plural images (I_1? I₂) from said image pickup element; a motion sensor for estimating motion between I_x and I₂, and for determining a scene- motion parameter (M); and a shutter speed controller for setting a new shutter speed (S^^) as a function of S_PRE and M.

2. The shutter speed adjustment system of claim 1 wherein said motion sensor detects the dominant motion in a scene.

3. The shutter speed adjustment system of claim 1 wherein said motion sensor includes a motion estimator for estimating motion on the basis of real motion vectors within the scene.

4. The shutter speed adjustment system of claim 1 wherein said motion sensor is an image stabilization system.

5. The shutter speed adjustment system of claim 1 wherein said plural images are

sequential images.

6. The shutter speed adjustment system of claim 1 wherein said plural images are

successive images.

7. The shutter speed adjustment system of claim 1 wherein I_: and I₂ are successive fields in an interlaced video.

8. The shutter speed adjustment system of claim 1 where Ij and I₂ are successive frames in a non-interlaced video.

9. A method of adjusting shutter speed based on motion within a scene, comprising: capturing plural images ( I₂) at a pre-set shutter speed (S_PRE); determining a scene-motion parameter (M) based on the scene motion between Ii and

I₂; and setting a new shutter speed (SHE_W) ^ & function of S_PRE and M.

10. The method of claim 9 wherein said capturing plural images includes capturing sequential images.

11. The method of claim 9 wherein said captiiring plural images includes capturing

successive images.

12. The method of claim 9 wherein said determining M is based on the dominant motion

within a scene.

13. The method of claim 12 wherein the dominant motion is determined by a four parameter rotation-zoom-translation model:

where [x, y] of I_j relate to [u, v] of I₂ via the four parameter rotation-zoom-translation model.

14. The method of claim 12 wherein the dominant motion is determined by a six parameter affine model:

u = ╬▒_ux + ╬▒₁₂y + ╬▒ 13

" ^tt┬½^{X +} *2? ^{+ tt}23

15. The method of claim 9 wherein said determining M is based on output from an image stabilizer.

16. The method of claim 9 wherein I_j and I₂ are successive fields in an interlaced video.

17. The method of claim 9 wherein and I₂ are successive frames in non-interlaced video.