WO2008072024A1

WO2008072024A1 - Video stabilization system with motion estimation error correction

Info

Publication number: WO2008072024A1
Application number: PCT/IB2006/003618
Authority: WO
Inventors: Markku Vehvilainen; Marius Tico
Original assignee: Nokia Corporation; Nokia Inc.
Priority date: 2006-12-14
Filing date: 2006-12-14
Publication date: 2008-06-19
Also published as: WO2008072024A9

Abstract

A method is presented for solving the quality/complexity trade-off in a video stabilizer and uses multiple motion estimators of incremental quality and complexity. The motion is estimated with the lowest complexity estimator following to be re-estimated with the next higher quality estimator only if the motion is detected as potentially erroneous. A confidence interval is established for determining whether the motion estimation should be considered erroneous in order to decide whether or not it is necessary to re-estimate the motion with the higher complexity motion estimator.

Description

VIDEO STABILIZATION SYSTEM WITH MOTION ESTIMATION ERROR

CORRECTION

TECHNICAL FIELD

The present invention relates generally to video stabilization and deals more specifically with the detection and correction of motion estimation errors in a video stabilization system.

LIST OF ABBREVIATIONS

ME=Motion Estimator

MED = Motion Error Detector

MF = Motion Filter

MC=Motion Correction

BACKGROUND OF THE INVENTION

The ongoing development and miniaturization of consumer devices that have video acquisition capabilities increases the need for robust and efficient video stabilization solutions to remove the effect of unwanted motion fluctuations from the video data to provide a much less irritating viewer experience. In the context of hand held video cameras such unwanted motion fluctuations are typically caused by undesired shakes or jerking of the hand during video capturing session. The continued development and demand for mobile phones and mobile phone systems with video communication capabilities further increases the need for the development of a more robust video stabilization system that ensures a high video stabilization quality in any conditions with minimal computational cost.

What is needed therefore is a way to ensure a high video stabilization quality in any video acquisition condition with minimal computational cost and to improve the video stabilization quality by detecting and correcting motion estimation errors to reduce their effect on the video data.

i SUMMARY OF THE INVENTION

In accordance with a broad aspect of the invention, a method is presented for solving the quality/complexity trade-off in a video stabilizer uses multiple motion estimators of incremental quality and complexity. The motion is estimated with the lowest complexity estimator following to be re-estimated with the next higher quality estimator only if the motion is detected as potentially erroneous. This strategy ensures a judicious usage of the motion estimators to preserve the stabilization quality in any video acquisition condition with minimal computational cost. In accordance with a further aspect of the invention, a confidence interval is established for determining whether the motion estimation should be considered erroneous in order to decide whether or not it is necessary to re-estimate the motion with a higher complexity motion estimator.

BRIEF DESCRIPTION OF THE DRAWINGS

Additional features, objects and advantages of the present invention will become readily apparent from the following written description taken in conjunction with the drawings wherein:

Figure 1 is a functional block diagram of a video stabilization system as it may be utilized for example in the context of a video acquisition system.

Figure 2 is a flowchart showing the major functional operations of a video stabilization algorithm that are carried out for stabilizing one video frame.

Figure 3 is a flowchart showing the major functional operations of a video stabilization algorithm embodying the present invention.

Figures 4A and 4B are graphic plot representations of an example of motion estimation error detection in accordance with the present invention.

Figure 5 is a functional block diagram showing the major functional elements of a video stabilization system embodying the present invention. Figure 6 is a functional block diagram of a signal processor for carrying out the invention.

Figure 7 is a functional block diagram of a mobile phone with video communication capabilities embodying the video stabilization system of the present invention. DESCRIPTION OF PREFERRED EMBODIMENTS

A number of approaches to provide video stabilization have been proposed and rely on the use of a single motion estimator. In one approach, the single motion estimator achieves high quality but is too complex due to multiple Fourier transforms operations and re-sampling to polar coordinates that is required and is therefore too complex for a mobile phone implementation. In another approach, the single motion estimator is a compromise of a trade-off between quality and complexity resulting in a low complexity but less robust motion estimator. In another approach, the motion estimator is not complex but it is less robust and is highly sensitive to the presence of moving objects in front of the camera.

Turning now to Figure 1 , a functional block diagram of a video stabilization system such as described above relying on a single motion estimator is illustrated therein and generally designated 10. The video stabilization system is illustrated schematically in Figure 1 for example as it may be utilized in the context of a video acquisition system The video stabilization system 10 may be included between the image pipeline shown in the dash lined box 12 and the video encoder shown in the dash lined box 14 in a well known and suitable manner as understood by those skilled in the art. The video stabilization system 10 comprises three main functional components: Motion Estimation (ME) 16, Motion Filtering (MF) 18, and Motion Correction (MC) 20 and a suitable memory 22 all of which components operate in a manner well understood by those skilled in the art.

The "Motion Estimation" (ME) 18 component functions to estimate the camera motion by matching the video data information from consecutive video frames. The "Motion Filtering" (MF) 18 component functions to identify the unwanted motion component from the estimated motion. The "Motion Correction" (MC) 20 component functions to warp the current video frame in such a way to attempt to cancel the effect of unwanted camera motion on the video being viewed. The motion estimator ME 16 component is, by far the most computationally intensive part of the video stabilizer system 10. Driven by the requirements to reduce the overall computational load, the design of the motion estimator ME 16 component is done in such a way to achieve an acceptable trade-off between quality and complexity. In attempting to arrive at an acceptable compromise, several assumptions are often made, for example, the video frame has a small noise level; there are small variations in the illumination between frames; there are no large moving objects in the scene; there are sufficient details in the scene for motion estimation determination, and other assumptions well known to those skilled in the art. The problem is the motion estimator fails as soon as any of these factual assumptions do not hold true and as a result the output of the video stabilizer is compromised. This means that, the "stabilized" video output jitters even more than the input video due to erroneous motion correction such that the user would prefer the video that would result without stabilization. On the other hand, a motion estimator that takes into consideration all possible problems is too complex for a real time implementation in a mobile device.

A flowchart generally designated 30 is shown in Figure 2 and illustrates the sequence of the major functional operations that are carried out in the video stabilization algorithm for the video stabilization system shown in Figure 1 for stabilizing one video frame. The sequence starts at operation 32 in which the current video frame and selected data from the previous video frame are compared. The current and previous video frames are compared in the motion estimation operation 34 and filtered or smoothed in the motion filtering operation 36. Based on the results of the motion estimation and filtering, a motion correction factor is determined in the motion correction operation 38. The motion correction determined in operation 38 is warped in the current video frame in accordance with corrective motion parameters well known and understood by those skilled in the art as shown in operation 40. These operations are then repeated for each video frame in the sequence. Turning now to Figure 3, a flowchart generally designated 50 is shown therein and illustrates the major functional operations that are carried out in the video stabilization algorithm in a video stabilization system embodying the present invention. In accordance with the invention, multiple motion estimators of different increasing complexities and qualities are used to correct the motion estimation errors detected by the system.

Still referring to Figure 3, the sequence begins at operation 52 in which the current video frame and selected data from the previous video frame are compared. The motion estimation error determination sequence is carried on for a suitable number of iterations depending on the number K of motion estimators in the system from k to K beginning with k set equal to "1" (one) in operation 54. The current video frame and the previous video frame are compared in the motion estimation operation 56 using the first motion estimator ME_k where k = 1. A Motion Error Detector (MED) analyzes in operation 58 the estimated motion determined in operation 56. If it is determined in operation 60 that the estimated motion is potentially erroneous and would cause video jitter, the system decides to re-estimate the motion between the two video frames with the next higher complexity estimator ME_k+i and increments the counter in operation 62. If it is determined in operation 64 that the next higher complexity motion estimator ME_k+i has not yet been used, the estimated motion between the video frames are again determined in operation 56 using the next higher complexity motion estimator.

The Motion Error Detector again analyzes in operation 58 the estimated motion determined in operation 56 with the motion estimator ME_k+i. If it is determined in operation 60 that the estimated motion is potentially erroneous and would cause video jitter, the system decides to again re- estimate the motion between the two video frames with the next higher complexity estimator and increments the counter in operation 62. If it is determined in operation 64 that the next higher complexity motion estimator ME_k+i has not yet been used, the estimated motion between the video frames are again determined in operation 56 using the next higher complexity motion estimator.

This sequence of operations continues until it is either determined in operation 60 that the estimated motion is not potentially erroneous and forwards that value to the motion filtering operation 68 or there are no further higher complexity motion estimators available, that is, MEK has been used and there are no others left to be used.

If all the motion estimators are used without passing the MED error test then the motion between the two frames is set to the most probable value in operation 66 which value is then forwarded to the motion filtering operation 68. The most probable value might be set, for example: (i) to the median of motions estimated by the different complexity motion estimators, or (ii) to a value closer to the average of correct estimated motions in previous frames. The motion estimation value is filtered or smoothed in the motion filtering operation 68 and the motion correction is determined in the motion correction operation 70. The motion correction is warped in the current video frame in accordance with corrective motion parameters well known and understood by those skilled in the art as shown in operation 72.

In accordance with the invention, an adaptive algorithm for motion error detection such as utilized in operation 58 in the flowchart shown in Figure 3 is described in the following.

The estimated motion between the Frame_n and the Frame_n-i of the video sequence is defined by a number of parameters well known to those skilled in the art such as for example: horizontal translation, vertical translation, rotation, scale, etc. At the current time, the majority of the practical implementations consider only the two translational parameters, that is, horizontal translation and vertical translation. In contrast to the majority of current day implementations, the present invention is not limited only to the horizontal translation and vertical translation parameters, but may be applied to any motion parameter for example, rotation, scale, etc.

In the following discussion, V_n denotes one such motion parameter, which may represent, for instance, the horizontal displacement or the vertical displacement between two video frames, or the rotational displacement between the two video frames, etc. The motion parameters are estimated by any one of the motion estimators ME_k based on the data available from the last captured frame (i.e. Frame_n) and from the previously captured frame (Frames). The objective of the algorithm is to determine whether V_n is a potentially erroneous estimate of the motion between the successive frames in the video sequence. To determine whether V_n is a potentially erroneous estimate, the algorithm defines a confidence interval within which a correct estimate should belong. The confidence interval is defined by two variables whose values are updated at each step. These variables are: T_n which is a positive threshold value, and M_n which is an estimate of the mean inter-frame motion (i.e. M_n =E[V_n]). The confidence interval at step n is given by [M_n.-) -T_n-1, M_n-i+T_n-i], and the estimated motion V_n is considered as a potentially erroneous estimate if the value is not within this interval. The algorithm updates the threshold value T_n and the mean estimate M_n at each estimation step n based on their respective previous values T_n-i and M_n-i, in such a way to adapt to the changes in the camera motion that occurs during video capturing.

The video stabilization algorithm with motion estimation error correction in accordance with the invention is set forth below as follows:

1. estimate motion value V_n using MEi

2. while | V_n- M_π-i|> T_n-I

3. do //The estimated motion is potentially erroneous a. if k < K b. then i. k=k+1 ii. re-estimate the motion with ME^ c. else i. set the inter-frame motion V_n to a probable value, e.g.

4. M_n = M_n-1 + b* (V_n - M_n-1)

5. T_n = T_n-1 + b* [q* I V_n- M_n| - T_n-1]

Some typical ranges for the constants b and q used in the above algorithm are [0.01 ,0.15] for b, and [4,16] for q. For efficiency both of these constants b and q can be expressed as powers of 2 in the corresponding ranges.

Graphic plot representations of an example of motion estimation error detection in accordance with the invention are shown in Figures 4A and 4B in which figure 4A represents horizontal motion and Figure 4B represents vertical motion. The mean M_n is represented by the graphic plot 80 in Figure 5A and 82 in Figure 5B. The borders of the motion estimation confidence interval [M_n-T_n ,M_n+T_n] are shown by the graphic plots 84, 86 respectively in Figure 4A and 88, 90, respectively in Figure 4B. It is to be noted that all vertical motion estimation error spikes 92 and horizontal estimation spikes 94 are clearly detected as they extend outside the confidence interval set by the respective borders 84, 86 and 88, 90 defining the respective confidence intervals.

Figure 5 is a functional block diagram showing the major functional elements of a video stabilization system embodying the present invention and is generally designated 100. The video stabilization system is illustrated schematically in Figure 5 for example as it may be utilized in the context of a video acquisition system for example a mobile device such as a mobile telephone. The video stabilization system 100 may be included between an image pipeline shown in the dash line box 102 and a video encoder shown in the dash line box 104 in a well known and suitable manner as understood by those skilled in the art. The stabilization system 100 comprises four main functional components: a Motion Estimator component 108, a Motion Error Detector component 110, a Motion Filtering component 112, a Motion Correction component 114 and a suitable memory. It should be noted that in contrast to the stabilization system shown in Figure 1 as discussed above, the video stabilization system 100 embodying the invention includes a Motion Error Detector component 110 in the feedback path between the Motion Estimator component 108 and the Motion Filtering component 112.

The interactions between the major logical functions should be obvious to those skilled in the art for the level of detail needed to gain an understanding of the video stabilization system embodying the present invention. It should be noted that the algorithm of the invention may be implemented with an appropriate signal processor such as shown in Figure 6, a digital signal processor or other suitable processor to carry out the intended function of the invention,

Turning now to Figure 7, a schematic functional block diagram of a mobile device with video capabilities for example a mobile telephone is illustrated therein showing the major operational functional components which may be required to carry out the intended functions of the mobile telephone and implement the video stabilization method of the invention. A processor such as the signal processor of Figure 6 carries out the computational and operational control of the mobile telephone in accordance with one or more sets of instructions stored in a memory. A suitable camera module operates under the control of the controller to provide video capabilities for the mobile device. A user interface may be used to provide alphanumeric input and control signals such as camera shutter activation by a user and is configured in accordance with the intended function to be carried out. A display sends and receives signals from the controller that controls the graphic, text and video representations shown on a screen of the display in accordance with the function being carried out.

The controller controls a transmit/receive unit that operates in a manner well known to those skilled in the art. The functional logical elements for carrying out the video stabilization with motion estimation error correction operational functions such as described above in connection with Figure 5 are suitably interconnected with the controller to carry out the video stabilization system as contemplated in accordance with the invention. An electrical power source such as a battery is suitably interconnected within the mobile terminal to carry out the functions described above. It will be recognized by those skilled in the art that the mobile telephone may be implemented in other ways other than that shown and described.

In a further example, the present invention is embodied in a computer program carried on a storage medium and having a set of instructions executable by a processor in a mobile device with video capabilities for stabilizing a video image in accordance with an algorithm by estimating with a plurality of different motion estimators the motion difference between a current video frame and a preceding video frame in response to detecting a potential error in the motion estimation for determining a motion correction value for warping with the current video frame to remove the motion difference between the current video frame and the preceding video frame.

In another example, the present invention is embodied in a computer program product and includes a computer readable storage structure embodying computer program code thereon for execution by a computer processor, wherein the computer program code comprises instructions for stabilizing a video image by estimating with a plurality of different motion estimators a motion value representative of the motion difference between a current video frame and a preceding video frame, and in response to detecting a potential error in the motion estimation determining a probable estimated motion value after all the available plurality of motion estimators have been used for determining a motion correction value for warping the current video frame with the motion correction value to remove the motion difference between the current video frame and the preceding video frame. A video stabilization system with motion estimation error correction has been presented above in several examples. It will be recognized and appreciated that numerous changes and modifications may be made by those skilled in the art without departing from the invention as disclosed. Accordingly, one or more aspects, embodiments and/or features in isolation and in all various combinations (whether or not specifically claimed in that combination or isolation) are considered to be within the present disclosure.

Claims

The invention claimed is:

1. Method, comprising: stabilizing a video image by estimating with a plurality of different motion estimators a motion value representative of the motion difference between a current video frame and a preceding video frame in response to detecting a potential error in the motion estimation for determining a motion correction value, and warping the current video frame with the motion correction value to remove the motion difference between the current video frame and the preceding video frame.

2. The method as defined in claim 1 further comprising stabilizing the video image by iteratively estimating the motion value representative of the motion difference between a current video frame and a preceding video frame.

3. The method as defined in claim 1 further comprising estimating with a plurality of different incrementally increasingly complex motion estimators.

4. The method as defined in claim 1 further comprising determining a probable motion value in response to detecting a potential error in the estimated motion value after all the available plurality of motion estimators have been used.

5. The method as defined in claim 4 further comprising determining the probable motion value to be the median of the motion values estimated by the plurality of different motion estimators.

6. The method as defined in claim 4 further comprising determining the probable motion value to be the average value of the correctly estimated motion values in previous video frames.

7. The method as defined in claim 1 further comprising defining a confidence interval within which a correctly estimated motion value should belong and considering an estimated motion value not within the confidence interval as a potentially erroneous estimate.

8. The method as defined in claim 7 further comprising defining the confidence interval to have a positive threshold variable value T_n and a mean inter-frame motion variable value M_n such that the confidence interval at the estimation step n is defined by the interval [M_n.-) - T_n.i , M_n-i + T_n-i].

9. The method as defined in claim 8 further comprising updating the threshold value T_n and the mean inter-frame motion value M_n for each estimation step n based on their respective previous values T_n-1 and M_π-i.

10. Apparatus, comprising: a suitable signal processor for carrying out a set of computational operations for stabilizing a video image by estimating with a plurality of different motion estimators the motion value representative of the motion difference between a current video frame and a preceding video frame in response to detecting a potential error in the motion estimation for determining a motion correction value for warping with the current video frame to remove the motion difference between the current video frame and the preceding video frame.

11. The apparatus as defined in claim 10 further configured for iteratively estimating the motion value representative of the motion difference between a current video frame and a preceding video frame.

12. The apparatus as defined in claim 10 further comprising said plurality of different motion estimators being configured as incrementally increasingly complex motion estimators.

13. The apparatus as defined in claim 10 further comprising a digital signal processor.

14. The apparatus as defined in claim 10 further comprising a memory for storing for retrieval the operational steps in an algorithm for determining a confidence interval within which a correctly estimated motion value should be to not be considered erroneous.

15. Apparatus, comprising: signal processing means, for carrying out a set of computational operations for stabilizing a video image in response to motion estimation means for estimating with different values and qualities the motion difference between a current video frame and a preceding video frame, and motion estimation error detection means, for detecting a potential error in the motion estimation and in response thereto, determining a probable motion difference for warping with the current video frame to remove the motion difference between the current video frame and the preceding video frame to stabilize the video image.

16. Computer program carried on a storage medium and executable by a processor in a mobile device with video capabilities for stabilizing a video image in accordance with an algorithm by estimating with a plurality of different motion estimators the motion difference between a current video frame and a preceding video frame in response to detecting a potential error in the motion estimation for determining a motion correction value for warping with the current video frame to remove the motion difference between the current video frame and the preceding video frame.

17. A computer program product comprising a computer readable storage structure embodying computer program code thereon for execution by a computer processor, wherein said computer program code comprises instructions for performing a method according to claim 1.