DE102007049740A1 - Method for determining two-dimensional motion vector fields in an image sequence and image processing device for this purpose - Google Patents

Abstract

A method of detecting two-dimensional motion vector fields in an image sequence comprises the steps of: - taking at least one image of a scene with a first exposure time; Recording the same scene with a second exposure duration that is longer than the first exposure duration, such that the image of the scene captured with the second exposure duration has a greater motion blur than the at least one image of the same scene taken with the first exposure duration, and determining two-dimensional motion vector field from the longer exposure image of the scene with higher motion blur taken with the second exposure duration with respect to the temporally associated with this image of the scene at least one captured with the first exposure time image of this scene with first exposure time.

Description

• – Aufnehmen mindestens eines Bildes einer Szene mit einer ersten Belichtungsdauer;
• – Aufnehmen derselben Szene mit einer zweiten Belichtungsdauer, die länger als die erste Belichtungsdauer ist, so dass das mit der zweiten Belichtungsdauer aufgenommene Bild der Szene eine größere Bewegungsunschärfe hat als das mit der ersten Belichtungsdauer aufgenommene, mindestens eine Bild derselben Szene.

The invention relates to a method for determining two-dimensional motion vector fields in an image sequence with the steps:

• - taking at least one image of a scene with a first exposure time;
• Recording the same scene with a second exposure duration which is longer than the first exposure duration, such that the image of the scene recorded with the second exposure duration has a greater motion blur than the at least one image of the same scene taken with the first exposure duration.

The The invention further relates to an image processing device with at least one camera for capturing a sequence of images Scene, with a control unit to control the at least a camera and with an image evaluation unit.

conventional Video recording methods provide time-separated snapshots a recorded dynamic scene. To view a view of any dynamic scene from a fixed viewpoint to any one Time between a fixed defined start and end time photorealistic Being able to produce is a reconstruction of the time continuous course of the recorded scene required. This is done conventionally by reconstruction of the so-called "optical flow" by means of image processing methods. It is based on individual snapshots on the two-dimensional Motion vector frame of the scene closed

Theobalt, J. Carranza, MA Magnor, H.-P. Seidel: "Enhancing Silhouette-based Human Motion Capture with 3D Motion Fields", in: Proc. IEEE Pacific Graphics 2003, Canmore, Canada, pp. 185-193, October 2003 discloses a method for detecting motion in images with three-dimensional motion fields. From the calculations of the optical flow in two-dimensional space for a plurality of viewpoints and an object model, a three-dimensional motion field is derived.

The determination of the optical flow in image sequences is, for example, in Sukhan Lim and Abbas EI Gamal: Optical Flow Estimation Using High Frame Rate Sequences, in: Proc. Int Conf. Image Processing, Vol. 2, 2001, pp. 925-928 described. For this purpose, a special camera with a high frame rate is used and the optical flux between the individual images is calculated by means of a known method according to Lukas-Kanade.

The motion estimation in digital image sequences is known inter alia from image compression technology. In this case, successive images of a sequence of displacement vectors are determined for image blocks, which describe a shift of image blocks having the least change in the image content. Such Bewegungsschätzverfahnen are, for example, in the WO 98/14010 . EP 0 614 318 B1 known.

US 2006/0290821 A1 discloses a video rendering process based on the interpretation of global image motion information and local motion in image blocks. To describe the local image motion, optical flux vectors are calculated.

M. Ben Ezra and SK Nayar: "Motion-Based Motion Deburring" in: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 6, June 2004 describes a method for processing videos significantly degraded by a camera movement. In order to obtain the required motion information of the camera, a second image detector with a lower resolution than the video recording camera is coupled to the video camera. While the video to be recorded and post-processed is taken with the high-resolution video camera with long exposure time and low temporal resolution, the second image detector operates with higher temporal resolution, shorter exposure times and lower spatial resolution. The motion of the video cameras is determined on the basis of the image sequence recorded with the second image detector. The method is to sharpen the long-exposed image by compensating for camera motion while the scene is assumed to be static. For this purpose, the optical flow between the images with the low spatial resolution, which are exposed only briefly calculated. Then, an approximation to the motion field is calculated, which corresponds to the movement of the camera during the long exposure time. The long-exposed, motion-blurred image is then sharpened using this approach.

The Coupling of video camera and second image detector may be in the process Ben-Ezra and Nayar either made by the second Image detector attached to a housing of the video camera and focused on the same object. It can also be one Beam splitters are used, leaving video camera and second image detector use the same look. Described is also the use of a special Image capture chips with two image detectors on the same chip.

The reconstruction of the optical flow by means of image processing methods in the known manner has the disadvantage that the reconstruction of the "Optical flow" is an underdeveloped problem that needs to be solved by other marginal assumptions that are not generally true: the optical flow also fails in scene regions that are not visible in both images, such as occlusions and occlusions also not necessarily identical to the true movement of the scene, for example a spinning spiral, Non-linear movements (acceleration, deceleration) can not be determined correctly by means of the optical flow.

outgoing It is an object of the present invention to provide an improved Method for determining the optical flow in an image sequence create that is simple and improved and more robust results supplies.

The Task is solved by the method of the type mentioned by determining a two-dimensional motion vector field the longer recorded with the second exposure time Illuminated image of the scene with the higher motion blur in relation to the temporally related to this image of the scene at least one recorded with the first shorter exposure time Picture of the scene.

It Thus, it is suggested that the movement information both from the images of a scene with a short exposure time, as well as off the same scene with a long exposure time equally exploit and correlate with each other to get out of the combination both images of the same scene with long and short exposure time to derive the optical flow. The two-dimensional motion vector field is the projection of the actual three-dimensional Movement in the recorded scene on the picture plane.

With the long-exposed shots of the scene are additional Information about the true course of the scene in motion available, so that the solution space for reconstruction of the optical flow can be significantly restricted. As a result, the consideration of the additional information leads from the long-exposed shots to much improved and more robust results compared to previous methods, at which did not show a correlation of longer and shorter Recordings done.

at The method is exploited that the shorter exposed Pictures depict a dynamic scene in sharp snapshots while the long-exposed images motion blur demonstrate. The at least one long-exposed image is combined with short-exposed images used to create the complex 2D motion field robust determination of a dynamic scene.

For example can by means of a long-exposed recording and the two temporally nearest exposed exposures the motion field be reconstructed a scene in which is exploited that the long exposed image of a convolution of the two short exposed Images with a nucleus that depends on the optical flow, equivalent.

In In a preferred embodiment, a convolution occurs of at least two recorded with the first exposure time Pictures of the scene and exploitation of one with the second exposure time taken picture of the same scene.

The Determining the course of the movement can be improved if that Taking the at least one image at the scene with the first temporal Resolution takes place with the aid of a first image acquisition unit, which provides an amplification factor for image acquisition control which is higher than the gain of a second image pickup unit for recording the same scene with a second exposure time is. So the scene will be both short Exposure time and high gain as well with as long exposure time and small gain factor added. The amplification factors have to be included be chosen so that the short-exposed images are not are underexposed while the long-exposed images not overexposed. The maximum exposure time depends on the refresh rate of the cameras.

alternative for image capture with two independent cameras is also conceivable that recording the scene with a camera is such that temporally one behind the other images of the scene with different from each other Exposures are taken to take pictures of a scene to get with different motion blur. Of the Course of movement then becomes with different exposure durations recorded images of the same scene. Again, it is beneficial if the only video camera alternately pictures with short and long exposure time simultaneously with correspondingly varying gain factors so that when taking pictures with long exposure time a small gain and when shooting with short exposure time a large gain factor is selected.

The object of the present invention is furthermore an improved image processing device having at least one camera for recording a sequence of images and a scene, with a control unit for controlling the at least one image Camera and with an image evaluation unit to create.

The Task is with the image processing device of the aforementioned Art solved by the fact that the control unit is set up is to capture at least one image of a scene with a first exposure time and record the same scene with a second exposure time, which is less than the first exposure time, so that with the second exposure time recorded image of the scene a larger one Motion blur has, than that with the first exposure time recorded, at least one image of the same scene, and that the image evaluation unit for determining the two-dimensional motion vector field from the Image taken with the second temporal resolution the scene with greater motion blur in relation to the temporally related to this image of the scene, at least one image of this scene with the first exposure time is set up.

advantageous Embodiments are in the subclaims described.

The Invention will become apparent from exemplary embodiments below with the accompanying drawings by way of example closer explained. Show it:

1 Block diagram of an image processing device with two cameras and an image evaluation unit for determining the 2D motion vector field in an image sequence;

2 Block diagram of an image processing device with a camera with a beam splitter and an image evaluation unit connected to two image sensors of the camera;

3 Exposure time diagram for displaying the image recording capabilities for determining the 2D motion history in an image sequence;

4 Sketch of an image sequence recorded with short and long exposure times over time and image B ₀ derived from a first approximation of the optical flow.

1 shows a block diagram of an image processing device 1 recognize the first camera 2a to record a scene 3 with a first exposure time. This first camera 2a is to a second camera 2 B coupled, on the same scene 3 is aligned to this scene 3 record with a second exposure time. These are the first and second camera 2a . 2 B by associated control units 4a . 4b controlled, which may be formed separately or as a unit. The cameras 2a . 2 B are controlled in such a way that the first camera 2a the scene 3 with short exposure time and high gain (gain), while the second camera 2 B the same scene 3 with the longest possible exposure time and a small amplification factor. The control units 4a . 4b as well as the cameras 2a . 2 B are themselves set so that the short-exposed images are not underexposed, while the long-exposed images are not the scene 3 overexposed. This is done by appropriate selection of the gain factors.

The maximum exposure time itself is the frame rate of the cameras 2a . 2 B dependent.

The short exposure time through the first camera 2a recorded images as well as the at least one long exposure time recorded image of the same scene 3 become an image evaluation unit 5 fed, in the short and long exposed images of the same scene 3 the two-dimensional motion vector field BV is calculated, which consistently converts the short-exposed images into each other and at the same time reconstructs the long-exposed image.

2 lets another embodiment of one of the two cameras 2a . 2 B be operated synchronously. The two cameras 2a . 2 B own by an optical beam splitter 6 same optical axis and use the same optics 7 , Again, with the first camera, the scene 3 with short exposure time and high gain with the second camera 2 B the same scene 3 recorded with a long exposure time and a small amplification factor.

An alternative technical realization is a single video camera 2 to use alternatingly taking pictures with short and long exposure time at simultaneously corresponding to varying gain factors.

3 shows an exposure time diagram for displaying the image recording possibilities for determining the image movement progression, ie the 2D motion vector field BV in an image sequence.

The conventional methods a) have provided the optical Flow by viewing two consecutive short-time exposures determine.

In an embodiment b) of the method for determining the course of motion, ie of 2D motion vector fields in an image sequence, a regular switching of the exposure time to alternately briefly exposed exposures A, C, E, G and long exposures B, D, F, H takes place to ma chen.

In another method c) can with a two-camera system short exposed images A, C, E, G and long exposed images B, D, F, H are recorded simultaneously.

Another alternative d) provides, with a correspondingly modified camera 2 record alternately briefly exposed images A, C, E, H and long exposures B, D, F, H at minimum dead times of equal length.

By recording a scene 3 with long and short exposed images A to H, much more information is available than in the known methods for determining the 2D motion vector field. This information is related to the image formation by the following model:
If v: R ² × [t ₁ , t ₂ ] → R ² , (x →, t) ↦ v _t (x →) is the 2D movement of a point of the image plane at time t, then the long-time-exposed image B ₀ from the briefly exposed image A, calculate as follows:

mit einem geeigneten Kern k: Ω² → R möglich.A similar formulation can be found for the short-exposed image C. Another description of the image formation process is also by means of the integral equation

with a suitable kernel k: Ω ² → R possible.

It is exploited that the short-exposed images A, C, E, G the dynamic scene 3 in sharp snapshots, while the long exposed images B, D, F, H show motion blur. On the basis of a long exposed shot B and the two temporally closest located short exposures A, C, the field of motion of the scene 3 be reconstructed. For this purpose, use is made of the fact that the long-exposed image corresponds to a convolution with the two short-exposed images A, C as the nucleus and the 2D motion vector field to be determined as the definition region of the convolution.

to Determining the 2D motion vector field BV from this information Different approaches are possible, such as Development approaches, approaches to variation, analysis-by-synthesis approaches Etc.

schnell simuliert. Dies hat die Wirkung, dass die kurz belichteten Bilder A und C entlang eines initialen Bewegungsfeldes v⁰ verschmiert und gewichtet gemittelt werden. Das auf diese Weise erzeugte Ergebnisbild B₀ wird anschließend mit lang belichteten Aufnahme B verglichen. Zum Vergleich genügt ein einfaches Fehlermaß, z. B. das aufsummierte Fehlerquadrat SSD. Mit Hilfe eines geeigneten Optimierungsverfahrens, z. B. nach Powell wird das Bewegungsfeld vⁿ iterativ solange variiert, bis der Fehlerwert e minimal ist. 4 shows a sketch of short exposures A, C and a long exposure photograph B. With the help of, for example, the analysis-by-synthesis approach, the image formation process of the long-exposure image B becomes according to the above formula

quickly simulated. This has the effect that the briefly exposed images A and C are smeared and weighted along an initial motion field v ⁰ . The result image B ₀ produced in this way is subsequently compared with long exposure B. For comparison, a simple error measure, z. B. the accumulated error square SSD. With the help of a suitable optimization method, z. B. Powell is ⁿ iteratively varying the motion field v until the error value e is minimal.

Both the optical flux between the two short-exposed images A and C are available as the initial motion field v ⁰ , and the optical flux from the long-term exposure B is available.

QUOTES INCLUDE IN THE DESCRIPTION

This list The documents listed by the applicant have been automated generated and is solely for better information recorded by the reader. The list is not part of the German Patent or utility model application. The DPMA takes over no liability for any errors or omissions.

Cited patent literature

• WO 98/14010 [0006]
• - EP 0614318 B1 [0006]
• US 2006/0290821 A1 [0007]

Cited non-patent literature

• - Chr. Theobalt, J. Carranza, MA Magnor, H.-P. Seidel: "Enhancing Silhouette-based Human Motion Capture with 3D Motion Fields", in: Proc. IEEE Pacific Graphics 2003, Canmore, Canada, pp. 185-193, October 2003 [0004]
• Sukhan Lim and Abbas EI Gamal: "Optical Flow Estimation Using High Frame Rate Sequences", in: Proc. Int Conf. Image Processing, Vol. 2, 2001, pp. 925-928 [0005]
• - M.-Ben-Ezra and SK Nayar: "Motion-Based Motion Deburring" in: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 6, June 2004 [0008]

Claims (14)

Method for determining two-dimensional Motion vector fields in a picture sequence with the steps: - Take up at least one image of a scene having a first exposure duration; - Take up same scene with a second exposure, the longer as the first exposure time, so that with the second Exposure time recorded image of the scene a greater motion blur has, at least, as the exposure taken with the first exposure a picture of the same scene, marked by - Determine of the two-dimensional motion vector field from that with the second Exposure time taken longer exposed image the scene with higher motion blur in relation at the time associated with this image of the scene at least one image taken with the first exposure time this scene with first exposure time. Method according to claim 1, characterized by folding at least one image taken with the first exposure time and taking advantage of the recorded with the second exposure time Picture of the same scene. A method according to claim 1 or 2, characterized by capturing the at least one image of a scene with a first short exposure time with the aid of a first image acquisition unit, which has a gain factor for exposure control, which is higher than the gain of a second one Image recording unit for recording the same scene with a second one Exposure duration is. Method according to claim 3, characterized in that the first and second image acquisition units are independent of one another Cameras are that the scene from approximately the same Take a perspective. Method according to claim 3, characterized that the first and second image acquisition unit via a common beam splitters are aligned on the same scene to to record the scene with identical perspective. Method according to one of claims 1 to 3, characterized by taking pictures of the scene temporally in a row with a camera with different from each other Exposure time to take pictures of the scene with different To get motion blur. Method according to claim 6, characterized by picking up of at least to be recorded with a first short exposure period an image of a scene with a gain control factor for exposure control, the higher than that for recording the same scene with one second exposure period provided gain is. Image processing device with at least one Camera for taking a sequence of images of a scene, with a Control unit for controlling the at least one camera and with an image evaluation unit, characterized in that the Control unit is set up to take at least one picture of a Scene with a first exposure duration and the same scene with one second exposure time, which is less than the first Exposure time is, so that with the second exposure time taken picture of the scene a higher motion blur has at least as the first exposure taken an image of the same scene, and that the image evaluation unit for Determination of the two-dimensional motion vector field from the with the second exposure time recorded image of the scene with higher Motion blur in relation to the time with this Image of the scene contiguous, at least one image This scene is set up with the first exposure time. Image processing device according to claim 8, characterized characterized in that the image evaluation unit for convolution of at least two pictures of a sequence taken with the first exposure from pictures of the same scene. Image processing device according to claim 8 or 9, characterized by a first for taking a picture of a Scene with a first exposure period equipped image recording unit, the has a gain factor for exposure control, which is higher than the gain of a second one Image recording unit for recording the same scene with a second one Exposure duration is. Image processing device according to claim 10, characterized characterized in that the first and second image pickup unit from each other Independent cameras are those that are approximate to capture the scene aligned with the same perspective. Image processing device according to claim 10, characterized characterized in that the first and second image acquisition unit via a common beam splitter are aligned on the same scene, to record the scene with identical perspective. Image processing device according to one of claims 8 or 9, characterized in that the image processing device exactly one camera for recording the scene, and the control unit is arranged to record temporally successive images of the scene with mutually different exposure times with the camera in order to obtain images with different motion blur from a scene. Image processing device according to claim 13, characterized characterized in that the control unit is arranged to the scene to be recorded with a first exposure duration with a Pick up gain for exposure control, the higher than that for recording the same scene with one second exposure period provided gain is.