WO2007066268A2 - Method and device for the selection of a stabilization method - Google Patents

Method and device for the selection of a stabilization method Download PDF

Info

Publication number
WO2007066268A2
WO2007066268A2 PCT/IB2006/054558 IB2006054558W WO2007066268A2 WO 2007066268 A2 WO2007066268 A2 WO 2007066268A2 IB 2006054558 W IB2006054558 W IB 2006054558W WO 2007066268 A2 WO2007066268 A2 WO 2007066268A2
Authority
WO
WIPO (PCT)
Prior art keywords
energy
determining
gmv
video sequence
image
Prior art date
Application number
PCT/IB2006/054558
Other languages
French (fr)
Other versions
WO2007066268A3 (en
Inventor
Stéphane Auberger
Jean Gobert
Sébastien LAVAL
Carolina Miro
Yann Picard
Original Assignee
Koninklijke Philips Electronics N.V.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics N.V. filed Critical Koninklijke Philips Electronics N.V.
Publication of WO2007066268A2 publication Critical patent/WO2007066268A2/en
Publication of WO2007066268A3 publication Critical patent/WO2007066268A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/262—Analysis of motion using transform domain methods, e.g. Fourier domain methods

Definitions

  • the invention relates to the evaluation and the selection of the stabilization quality of a video sequence stabilization method applied on a video sequence, and to a corresponding stabilization device incorporating said evaluation and selection method.
  • Video footage from hand-held camcorders is typically jerky due to small, unwanted camera movements. Removal of those undesired movements requires video stabilization techniques. As its name suggests, video stabilization is the process of generating a compensated video sequence where image motion by the camera's undesirable shake or jitter is removed. Digital image processing techniques are often used to perform such a task and are preferred over mechanical or optical video stabilization approaches since modern VLSI techniques allow a more compact camera design.
  • the Interframe Transformation Fidelity (ITF) method can, for example, be used. This method computes the Pick Signal to Noise Ratio (PSNR) between consecutive images of the stabilized sequence.
  • Another possible method is the Global Transformation Fidelity (GTF) method. This method calculates the PSNR between the current image and a reference image, for example the first image when the stabilization starts.
  • the invention relates to a method to evaluate and select the stabilization quality of at least a stabilization method applied to an original video sequence composed of several images to obtain a stabilized video sequence, the evaluation method being adapted to assign a quality factor (Q) to the stabilization method according to the quality of the obtained stabilized video sequence, wherein the evaluation method comprising the following steps:
  • step of determining the quality factor (Q) includes a step of determining a motion component, the motion component being proportional to a weighted sum of the energy differences (W d (a), W d (b), W d (c)) of the set of energy differences (W d (a), W d (b), W d (c)).
  • the quality factor is not influenced by the global motion of the sequence.
  • the invention relates to a device to evaluate and select the stabilization quality of at least a stabilization method applied to an original video sequence composed of several images to obtain a stabilized video sequence, the evaluation method being adapted to assign a quality factor (Q) to the stabilization method according to the quality of the obtained stabilized video sequence, wherein the device comprises:
  • - motion estimation means adapted to extract a set of initial global motion vectors (GMV° 2 , GMV° 4 ) from the original video sequence and a set of final global motion vectors (GMV o, GMV 12 ) from the stabilized video sequence ;
  • - energy analysis means adapted to determine a set of initial energy quantities (W°(a), W°(b), W°(c)) by predefined frequency band (a, b, c), of the energy spectrum of the set of initial global motion vectors (GMV°2, GMV°4) and a set of final energy quantities (W f (a), W f (b), W f (c)) by predefined frequency band (a, b, c) of the energy spectrum of the set of final global motion vectors (GMV f io, GMV f i 2 );
  • - computing means adapted to determine a set of energy differences (W d (a), W d (b), W d (c)) by frequency band (a, b, c), each energy difference (W d (a), W d (b), W d (c)) resulting from the subtraction of a final energy quantity (W (a), W (b), W (c)) of a defined frequency band (a, b, c) to the initial energy quantity (W°(a), W°(b), W°(c)) of the same frequency band (a, b, c); and
  • the computing means being adapted to determine a quality factor (Q) characterizing the evaluated stabilization method, wherein the step of determining the quality factor (Q) includes a step of determining a motion component, the motion component being proportional to a weighted sum of the energy differences (W d (a), W d (b), W d (c)) of the set of energy differences (W d (a), W d (b), W d (c)).
  • the invention also relates to a device for digital image stabilization for removing unwanted camera movements, called jitter, from an original sequence of images generated by said camera and obtaining a stabilized sequence of images, said device incorporating a stabilization quality metric based on a previous evaluation and selection of stabilization quality which are performed in a device according to claim 12.
  • FIG. 1 is a schematic block diagram of a stabilization method to be tested
  • FIG. 2 is a block diagram of a device according to the invention, which allows to estimate the stabilization quality of a stabilization method
  • FIG. 3 is a flow chart of a method according to the invention, to estimate the stabilization quality of a stabilization method
  • FIG. 4 is a block diagram of an example of a set of motion vectors associated to macroblocks of an image of an original video sequence
  • FIG. 5 is a graph representing the energy spectrum of the set of initial global motion vectors of the original video sequence.
  • This stabilization method also referenced 1 is applied to a set of consecutive images 2, 4, 6, etc, of an original video sequence 8 (obtained at the output of a camera) to obtain a set of consecutive images 10, 12, 14, etc, of a stabilized video sequence 16.
  • a device 15 is adapted to evaluate the stabilization method 1. It comprises a motion estimation block 17 connected to an energy analysis block 19, and an image analysis block 21, and the energy analysis block 19 and to the image analysis block 21 are themselves connected to a computing block 23.
  • the device 15 is adapted to receive both the original sequence 8 and the stabilized sequence 16 in order to evaluate the quality of the stabilization method 1 and to deliver a quality factor Q featuring this quality.
  • the highest quality factor is assigned to the stabilization method having an image quality similar to that obtained when the camera is stable or unshaken.
  • the device 15 performs an evaluation method 18 (see Fig.3) according to the invention.
  • the evaluation method 18 is applied to video sequences 8, 16 comprising between 500 and 700 images.
  • the evaluation method 18 comprises a first step 20 of extracting a set of initial global motion vectors GMV° 2 , GMV° 4 , GMV° 6 , GMW° 8 , etc, from the original video sequence 8. This step is performed in Fig.2 by the motion estimation block 17. To obtain an initial motion vector of the two first images 2, 4 of the original sequence 8, different methods can be used.
  • each motion vector Vk represents the movement of the scene from one image 2 to the subsequent image 4, in each macroblock 24.
  • each macroblock 24 comprises 16x16 pixels of the image.
  • the median GMV°2 of the set of motion vectors Vk is determined.
  • this median motion vector is the initial global motion vector GMV° 2 and represents the global movement of the scene realized between images 2 and 4.
  • the sub-steps 22 and 26 are repeated for all images of the original sequence 8 to obtain a set of initial global motion vectors GMV°2, GMV° 4 , etc. They are repeated for the 700 images of the original sequence to obtain a set of 699 global motion vectors, and more generally for all images of the original sequence, where the sequence have more or less frame than 700.
  • Each initial global motion vector comprises an horizontal u° 2 and a vertical v° 2 components which are the projection on the horizontal and respectively vertical axis of the global motion vector GMV°2.
  • GMV°4 (u°4, v° 4 )
  • the evaluation method 18 affects to each tested stabilization method 1 a quality factor Q that comprises a motion component and an image component.
  • the motion component of the quality factor is calculated by performing an analysis of the energy spectrum of the original 8 and the stabilized 16 sequences.
  • an energy analysis is performed by the energy analysis block 19, on both horizontal u° 2 and vertical v° 2 components of the set of initial global motion vectors
  • each component of the initial global motion vectors GMV° 2, GMV° 4 is made for example with a Cooley-Tukey Fast Fourier Transform algorithm (FFT) using the Danielson-Lanczos Lemma.
  • FFT Cooley-Tukey Fast Fourier Transform algorithm
  • V(F) is the FFT transform of the vertical component v° ;
  • Vdb 10 log V.
  • Figure 5 shows the graph representing the evolution of the energy in function of the frequency, namely the spectrum W°(F), obtained from the FFT transform of the set of the initial global motion vectors.
  • a set of initial energy quantities W°(a), W°(b), W°(c) are computed from the energy spectrum W°(F).
  • step 32 the stabilization method 1 to be evaluated, is applied to the original sequence 8 to obtain the stabilized sequence 16.
  • a set of final global motion vectors GMV f io, GMV f i 2 are extracted from the stabilised sequence 16 by the motion estimation block 17 and using the same method as the one used at step 20.
  • a set of final energy quantities W (a), W (b), W (c) is determined from the energy spectrum W f (F) of the set of final global motion vectors GMV f io, GMV f i2 etc, using the same method as the one explained at steps 28 and 30.
  • a set of energy differences W d (i) is computed. Each energy difference results from the subtraction of a final energy quantity W (i) of a defined frequency band i to the initial energy quantity W°(i) of the same frequency band i, as shown
  • W d (a) W°(a) - W f (a)
  • the motion component of the quality factor Q is proportional to a weighted sum of the set of energy differences W d (a), W d (b) , W d (c) over each predefined frequency band a, b, c.
  • a set of frequency-based weighting coefficients of the motion components are determined by the computing block 23.
  • a couple comprising an inside band energy coefficient "Energy_in_i” and an outside band energy coefficient "Energy_out_i" is computed for each frequency band i.
  • the frequency band a is considered.
  • the inside band energy coefficient "Energy_in_a” of the band a is equal to the final energy quantity W°(a) of the frequency band a.
  • the outside band energy coefficient "Energy_out_a” of the band a is equal to the sum of the initial energy quantity W°(i) of all frequency bands i of the energy spectrum W°(F), at the exception of the band a.
  • Energy_out_a W°(b) + W°(c).
  • the inside band energy coefficient and the outside band energy coefficient are calculated for all predefined frequency bands a, b, c to obtain a couple of coefficients per predefined frequency band a, b, c.
  • Each couple is associated to a frequency band and is one of the frequency -based weighting coefficients.
  • a set of relative inside "AJnJ” and the set of relative outside "A_out_F band energy coefficients are determined by the computing block 23, for each frequency band i, using the following equation :
  • Total_energy is the integral over all frequency band i of the energy spectrum W(F) of the set of initial global motion vectors and where "Energy_in_i" and
  • Energy_out_i are the inside band and respectively the outside band energy coefficients.
  • the image component of the quality factor Q is calculated by the image analysis block 21, performing image analysis on the original sequence 8.
  • a global inner motion coefficient IM is computed.
  • This coefficient IM constitutes an image-based weighting coefficient and represents the global motion within the scene. This coefficient is computed, for example, by calculating the disparity of motion vectors V k inside the image using the following equation :
  • - GMV° j is the global motion vector of the image j computed at step 26, by the motion estimation block 17;
  • V k is the motion vector of one macro block 24 of the image j of the original sequence 8 computed at sub-step 22, by the motion estimation block 17 ;
  • n is a number of macro blocks 24 in an image j which corresponds to the number of motion vectors in this image
  • a mean texture rate of each image 2, 4, 6 is determined using a classical texture analysis method.
  • An average TI of the mean of the texture rates determined for all images is computed.
  • the average texture rate TI constitutes an image-based weighting coefficient.
  • an Interframe Transformation Fidelity is computed.
  • This index is a standard Pick Signal to Noise Ratio (PSNR). It is determined on the luminance data between two consecutive images.
  • - a y and b y are the luminance data of each pixel of the current image and of the corresponding pixel of the consecutive image, - "# pixels" is the number of pixels in the image.
  • the ITF is given by :
  • ITF y (PSNR(l K ,l K+1 ))
  • the ITF is computed for each image of the original sequence 8 and the median value of the computed ITF is used for the determination of the quality factor Q.
  • the quality factor Q is computed from the following equation:
  • - ITF is the median value of the computed Interframe Transformation Fidelity index
  • - IT is the average for all images of the mean of the texture rates determined.
  • the factors ⁇ 1; ⁇ li, ⁇ 2 b ⁇ 3 b ⁇ 4 1; ⁇ 5j and CtO 1 , CtI 1 , a2 u ⁇ 3 1; ct ⁇ are determined empirically, during a training phase, by viewing several stabilized sequences 16 obtained from performing a stabilization method 1 on original sequences 8 showing different types of scene. Once the factors determined, they are fixed and used to determine the quality factor Q of other stabilization method to evaluate.
  • the method 18 according to the invention can be used in a particular case where the energy spectrum W°(F) is only divided into two energy bands i, one for high frequency, for example above 1 Hz, and one for low frequency, for example below 1 Hz. In such cases, the above equation is given by:
  • This method can be implemented in hardware on a micro-controller or in software on acommun support executed by a microprocessor from a non volatile memory of the ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory) kind or an equivalent.
  • ROM Read Only Memory
  • EEPROM Electrically Erasable Programmable Read Only Memory
  • the method of the invention can also be carried out by a computer program for a processing unit comprising a set of instructions, which, when loaded into said processing unit, causes the processing unit to carry out the method described above.
  • a computer program for a processing unit comprising a set of instructions, which, when loaded into said processing unit, causes the processing unit to carry out the method described above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)

Abstract

The invention relates to a method (18) allowing to evaluate the stabilization quality of a stabilization method and comprising the steps of : - determining (30) a set of initial energy quantities per predefined frequency band, of the energy spectrum of a set of initial global motion vectors obtained from the original video sequence ; - determining (36) a set of final energy quantities per predefined frequency band, of the energy spectrum of the set of final global motion vectors obtained from the stabilized video sequence ; - determining (38) a set of energy differences per predefined frequency band, each energy difference resulting from the subtraction for each predefined frequency band of a final energy quantity to the initial energy quantity ; and - determining (50) the quality factor proportional to a weighted sum of the energy differences of the set of energy differences.

Description

METHOD AND DEVICE FOR THE SELECTION OF A STABILIZATION METHOD
FIELD OF THE INVENTION
The invention relates to the evaluation and the selection of the stabilization quality of a video sequence stabilization method applied on a video sequence, and to a corresponding stabilization device incorporating said evaluation and selection method.
BACKGROUND OF THE INVENTION
Video footage from hand-held camcorders is typically jerky due to small, unwanted camera movements. Removal of those undesired movements requires video stabilization techniques. As its name suggests, video stabilization is the process of generating a compensated video sequence where image motion by the camera's undesirable shake or jitter is removed. Digital image processing techniques are often used to perform such a task and are preferred over mechanical or optical video stabilization approaches since modern VLSI techniques allow a more compact camera design.
To estimate the quality of different stabilization methods, stabilized versions obtained from the application of different stabilization methods to the same original sequence, are watched by viewers who decide which one they prefer. The decision criterion is the visual comfort. This method allows a fine ranking of the stabilization methods. However, subjective assessment of viewer is highly time consuming and expensive.
Evaluation of the perceptual quality of stabilization method is also performed with the help of computers. The Interframe Transformation Fidelity (ITF) method can, for example, be used. This method computes the Pick Signal to Noise Ratio (PSNR) between consecutive images of the stabilized sequence. Another possible method is the Global Transformation Fidelity (GTF) method. This method calculates the PSNR between the current image and a reference image, for example the first image when the stabilization starts. These Interframe Transformation Fidelity (ITF) and the Global Transformation Fidelity (GTF) methods are simple to implement and to understand.
However, they are biased when local motion exists in the image or when the images differ with lighting conditions. They are also biased when jitters are simultaneous with an intentional global motion during, for example, a zoom movement of the camera. In addition, they cannot differentiate the quality of a motion extraction and a motion compensation. For example, the use of different interpolation filters to generate the stabilized sequence would lead to different PSNR, which are not necessarily linked to the quality of the stabilization method. Further, the ITF and the GTF methods do not give an indication about the smoothness of the global motion of the sequence from image to image.
Therefore, it is desirable to develop a new evaluation method to estimate and select a stabilization method among the existing stabilization methods and to provide an estimation quality factor as close as possible to human perception that is not influenced by the global motion of the image. SUMMARY OF THE INVENTION
Accordingly, it is an object of the invention to provide an improved method to evaluate and select the quality of a stabilization method.
To this end, the invention relates to a method to evaluate and select the stabilization quality of at least a stabilization method applied to an original video sequence composed of several images to obtain a stabilized video sequence, the evaluation method being adapted to assign a quality factor (Q) to the stabilization method according to the quality of the obtained stabilized video sequence, wherein the evaluation method comprising the following steps:
- extracting a set of initial global motion vectors (GMV°2, GMV0*) from the original video sequence ;
- determining a set of initial energy quantities (W°(a), W°(b), W°(c)) per predefined frequency band (a, b, c), from the energy spectrum of the set of initial global motion vectors (GMV°2, GMV°4) ;
- stabilizing the original video sequence using the stabilization method, to obtain the stabilized video sequence ;
- extracting a set of final global motion vectors (GMVfio, GMVfi2) from the stabilized video sequence ;
- determining a set of final energy quantities (Wf(a), Wf(b), Wf(c)) per predefined frequency band (a, b, c), from the energy spectrum of the set of final global motion vectors (GMVfi0, GMVf 12);
- determining a set of energy differences (W (a), W (b), W (c)) per predefined frequency band (a, b, c), each energy difference (Wd(a), Wd(b), Wd(c)) resulting from the subtraction for each predefined frequency band (a, b, c) of a final energy quantity (Wf(a), Wf(b), Wf(c)) to the initial energy quantity (W°(a), W°(b), W°(c)); and - determining the quality factor (Q) featuring the stabilization method, wherein the step of determining the quality factor (Q) includes a step of determining a motion component, the motion component being proportional to a weighted sum of the energy differences (Wd(a), Wd(b), Wd(c)) of the set of energy differences (Wd(a), Wd(b), Wd(c)).
Since the evaluation and selection method computes the energy difference of the energy spectrums of the original video sequence and of the stabilized video sequence, the quality factor is not influenced by the global motion of the sequence.
Other features and advantages of the method are recited in the dependent claims. In addition, the invention relates to a device to evaluate and select the stabilization quality of at least a stabilization method applied to an original video sequence composed of several images to obtain a stabilized video sequence, the evaluation method being adapted to assign a quality factor (Q) to the stabilization method according to the quality of the obtained stabilized video sequence, wherein the device comprises:
- motion estimation means adapted to extract a set of initial global motion vectors (GMV°2, GMV°4) from the original video sequence and a set of final global motion vectors (GMV o, GMV 12) from the stabilized video sequence ;
- energy analysis means adapted to determine a set of initial energy quantities (W°(a), W°(b), W°(c)) by predefined frequency band (a, b, c), of the energy spectrum of the set of initial global motion vectors (GMV°2, GMV°4) and a set of final energy quantities (Wf(a), Wf(b), Wf(c)) by predefined frequency band (a, b, c) of the energy spectrum of the set of final global motion vectors (GMVfio, GMVfi2);
- computing means adapted to determine a set of energy differences (Wd(a), Wd(b), Wd(c)) by frequency band (a, b, c), each energy difference (Wd(a), Wd(b), Wd(c)) resulting from the subtraction of a final energy quantity (W (a), W (b), W (c)) of a defined frequency band (a, b, c) to the initial energy quantity (W°(a), W°(b), W°(c)) of the same frequency band (a, b, c); and
- the computing means being adapted to determine a quality factor (Q) characterizing the evaluated stabilization method, wherein the step of determining the quality factor (Q) includes a step of determining a motion component, the motion component being proportional to a weighted sum of the energy differences (Wd(a), Wd(b), Wd(c)) of the set of energy differences (Wd(a), Wd(b), Wd(c)). The invention also relates to a device for digital image stabilization for removing unwanted camera movements, called jitter, from an original sequence of images generated by said camera and obtaining a stabilized sequence of images, said device incorporating a stabilization quality metric based on a previous evaluation and selection of stabilization quality which are performed in a device according to claim 12.
These and other aspects of the invention will be apparent from the following description, drawings and from the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
- Figure 1 is a schematic block diagram of a stabilization method to be tested;
- Figure 2 is a block diagram of a device according to the invention, which allows to estimate the stabilization quality of a stabilization method;
- Figure 3 is a flow chart of a method according to the invention, to estimate the stabilization quality of a stabilization method;
- Figure 4 is a block diagram of an example of a set of motion vectors associated to macroblocks of an image of an original video sequence; and
- Figure 5 is a graph representing the energy spectrum of the set of initial global motion vectors of the original video sequence. DETAILED DESCRIPTION
Referring to Figure 1, a method and device for the evaluation and the selection of a stabilization method is illustrated. This stabilization method also referenced 1 is applied to a set of consecutive images 2, 4, 6, etc, of an original video sequence 8 (obtained at the output of a camera) to obtain a set of consecutive images 10, 12, 14, etc, of a stabilized video sequence 16.
Referring to Figure 2, a device 15 according to the invention is adapted to evaluate the stabilization method 1. It comprises a motion estimation block 17 connected to an energy analysis block 19, and an image analysis block 21, and the energy analysis block 19 and to the image analysis block 21 are themselves connected to a computing block 23.
The device 15 is adapted to receive both the original sequence 8 and the stabilized sequence 16 in order to evaluate the quality of the stabilization method 1 and to deliver a quality factor Q featuring this quality. The highest quality factor is assigned to the stabilization method having an image quality similar to that obtained when the camera is stable or unshaken. To this end, the device 15 performs an evaluation method 18 (see Fig.3) according to the invention. To obtain efficient result, the evaluation method 18 is applied to video sequences 8, 16 comprising between 500 and 700 images.
As illustrated in Figures 2 and 3, the evaluation method 18 comprises a first step 20 of extracting a set of initial global motion vectors GMV°2, GMV°4, GMV°6, GMW°8 , etc, from the original video sequence 8. This step is performed in Fig.2 by the motion estimation block 17. To obtain an initial motion vector of the two first images 2, 4 of the original sequence 8, different methods can be used.
One of them consists in considering the image 2 and its subsequent image 4 during a sub-step 22 and determining a set of motion vectors Vk of macroblocks 24 of these images. As illustrated in Figure 4, each motion vector Vk represents the movement of the scene from one image 2 to the subsequent image 4, in each macroblock 24. Typically, each macroblock 24 comprises 16x16 pixels of the image.
In a sub-step 26, the median GMV°2 of the set of motion vectors Vk is determined. At the first iteration, this median motion vector is the initial global motion vector GMV°2 and represents the global movement of the scene realized between images 2 and 4.
Once the initial global motion vector GMV°2 of the considered images 2, 4 is determined, the sub-steps 22 and 26 are repeated for all images of the original sequence 8 to obtain a set of initial global motion vectors GMV°2, GMV°4, etc. They are repeated for the 700 images of the original sequence to obtain a set of 699 global motion vectors, and more generally for all images of the original sequence, where the sequence have more or less frame than 700.
For reason of simplification, in the following the method is described only in connection to the global motion vectors GMV°2 and GMV°4 but it is applied to the 699 global motion vectors GMW°i with 1=2, 4, 6, 8, etc, determined in the sequence.
Each initial global motion vector comprises an horizontal u°2 and a vertical v°2 components which are the projection on the horizontal and respectively vertical axis of the global motion vector GMV°2.
Figure imgf000007_0001
GMV°4 = (u°4, v°4)
The evaluation method 18 affects to each tested stabilization method 1 a quality factor Q that comprises a motion component and an image component. During steps 28 to 42, the motion component of the quality factor is calculated by performing an analysis of the energy spectrum of the original 8 and the stabilized 16 sequences.
In the step 28, an energy analysis is performed by the energy analysis block 19, on both horizontal u°2 and vertical v°2 components of the set of initial global motion vectors
Figure imgf000008_0001
The transformation of each component of the initial global motion vectors GMV°2, GMV°4, from the time domain to the frequency domain is made for example with a Cooley-Tukey Fast Fourier Transform algorithm (FFT) using the Danielson-Lanczos Lemma.
From the FFT transform of the horizontal u°2 and the vertical v°2 components of the initial global motion vectors, the energy spectrum is obtained using the following relation:
W°(F) = Re (U(F))2 + Im (U(F))2 + Re ((V(F))2 + Im (V(F))2
where:
- U(F) is the FFT transform of the horizontal component u0 ;
- V(F) is the FFT transform of the vertical component v° ; and
- "Re" is the real part and "Im" the imaginary part of the complex numbers resulting of the FFT transform.
Thereafter, W°(F) (as well as all other energy values) is expressed in decibels, and V is a non-negative value, its value in decibels Vdb is given by: Vdb = 10 log V.
Figure 5 shows the graph representing the evolution of the energy in function of the frequency, namely the spectrum W°(F), obtained from the FFT transform of the set of the initial global motion vectors.
During step 30, a set of initial energy quantities W°(a), W°(b), W°(c) are computed from the energy spectrum W°(F). Each initial energy quantity W°(i) corresponds to the integral of the energy of the energy spectrum W°(F) over a predefined frequency band i= a, b, c.
Besides, during step 32, the stabilization method 1 to be evaluated, is applied to the original sequence 8 to obtain the stabilized sequence 16.
At step 34, a set of final global motion vectors GMVfio, GMVfi2 are extracted from the stabilised sequence 16 by the motion estimation block 17 and using the same method as the one used at step 20. At step 36, a set of final energy quantities W (a), W (b), W (c) is determined from the energy spectrum Wf(F) of the set of final global motion vectors GMVfio, GMVfi2 etc, using the same method as the one explained at steps 28 and 30.
At step 38, a set of energy differences Wd(i) is computed. Each energy difference results from the subtraction of a final energy quantity W (i) of a defined frequency band i to the initial energy quantity W°(i) of the same frequency band i, as shown
for example with the following expressions :
Wd(a) = W°(a) - Wf(a)
Wd(b) = W°(b) - Wf (b)
As explained later in the description, the motion component of the quality factor Q is proportional to a weighted sum of the set of energy differences Wd(a), Wd(b) , Wd(c) over each predefined frequency band a, b, c.
During steps 40 and 42, a set of frequency-based weighting coefficients of the motion components are determined by the computing block 23.
At step 40, a couple comprising an inside band energy coefficient "Energy_in_i" and an outside band energy coefficient "Energy_out_i" is computed for each frequency band i.
For example, the frequency band a is considered. The inside band energy coefficient "Energy_in_a" of the band a, is equal to the final energy quantity W°(a) of the frequency band a. The outside band energy coefficient "Energy_out_a" of the band a, is equal to the sum of the initial energy quantity W°(i) of all frequency bands i of the energy spectrum W°(F), at the exception of the band a. In the above mentioned example, Energy_out_a = W°(b) + W°(c).
The inside band energy coefficient and the outside band energy coefficient are calculated for all predefined frequency bands a, b, c to obtain a couple of coefficients per predefined frequency band a, b, c. Each couple is associated to a frequency band and is one of the frequency -based weighting coefficients.
At step 42, a set of relative inside "AJnJ" and the set of relative outside "A_out_F band energy coefficients are determined by the computing block 23, for each frequency band i, using the following equation :
Energyjnj _ Energy_ouji
-iπJ ~ Total_enegy -0UtJ ~ TotaTenegy where the "Total_energy" is the integral over all frequency band i of the energy spectrum W(F) of the set of initial global motion vectors and where "Energy_in_i" and
"Energy_out_i" are the inside band and respectively the outside band energy coefficients.
During steps 44 to 50, the image component of the quality factor Q is calculated by the image analysis block 21, performing image analysis on the original sequence 8.
At step 44, a global inner motion coefficient IM is computed. This coefficient IM constitutes an image-based weighting coefficient and represents the global motion within the scene. This coefficient is computed, for example, by calculating the disparity of motion vectors Vk inside the image using the following equation :
m 1 n
=— Σ -∑abs(V k - GMV .° ; where:
- GMV°j is the global motion vector of the image j computed at step 26, by the motion estimation block 17;
- Vk is the motion vector of one macro block 24 of the image j of the original sequence 8 computed at sub-step 22, by the motion estimation block 17 ;
- "abs" is the absolute value;
- "n" is a number of macro blocks 24 in an image j which corresponds to the number of motion vectors in this image; and
- "m" is a number of images.
At step 46, a mean texture rate of each image 2, 4, 6 is determined using a classical texture analysis method. An average TI of the mean of the texture rates determined for all images is computed. The average texture rate TI constitutes an image-based weighting coefficient.
At step 48, an Interframe Transformation Fidelity (ITF) is computed. This index is a standard Pick Signal to Noise Ratio (PSNR). It is determined on the luminance data between two consecutive images.
2
255
PSNR(lk,lk + i)=10. |og1 0
#pixels 4Λa'i b'i where
- ay and by are the luminance data of each pixel of the current image and of the corresponding pixel of the consecutive image, - "# pixels" is the number of pixels in the image.
The ITF is given by :
.. nb_image-1
ITF = y (PSNR(lK,lK+1 ))
(nb_image - 1 ) ^ v κ> κ.+i " where "nb_image" is the number of images in the sequence 8.
The ITF is computed for each image of the original sequence 8 and the median value of the computed ITF is used for the determination of the quality factor Q.
At step 50, the quality factor Q is computed from the following equation:
Q = V UxO1 + (Xl1 * A + α2j * Aout l + α3 j * IM + α41 * TI U lTF
ie frequency_ bands V /
TpO1 + βlx * Energy_in_ i H- β21 * Energy_out _i ^j
+ Σ + PS ^ A111 1 + β41 * Aout l + β51 * IM + β61 * TI * w d(i)
IG frequency_ bands I J
where:
- the factors βθ1; βli, β2b β3b β41; βS^re constant function of the frequency band i;
- the factors CtO1, CtI1, α2b α31; ct^ are constant function of the frequency band i;
- "Energy_in_i" is the inside band energy coefficient for the frequency band i;
- "Energy_out_i" is the outside band energy coefficient for the frequency band i;
- "AJnJ" is the relative inside band energy coefficient (= Energy_in_z / Total energy);
- uA_outJ" is the relative outside band energy coefficient (= Energy_out_z / Total energy);
- ITF is the median value of the computed Interframe Transformation Fidelity index;
- IM is the global inner motion coefficient; and
- IT is the average for all images of the mean of the texture rates determined.
The factors βθ1; βli, β2b β3b β41; β5j and CtO1, CtI1, a2u α31; ct^ are determined empirically, during a training phase, by viewing several stabilized sequences 16 obtained from performing a stabilization method 1 on original sequences 8 showing different types of scene. Once the factors determined, they are fixed and used to determine the quality factor Q of other stabilization method to evaluate. The method 18 according to the invention can be used in a particular case where the energy spectrum W°(F) is only divided into two energy bands i, one for high frequency, for example above 1 Hz, and one for low frequency, for example below 1 Hz. In such cases, the above equation is given by:
Q = (αO + αl * A Below + α2 * A Above + α3 * IM + α4 * Tl) * ITF
+ (βO + βl * A Below + βl * AAbove + β2 * IM + β3 * Tl ) * redHF
with + ^° + γl * ABelow + βl * A Above + γ2 * IM + γ3 * TI^ * redLF
_ HF_Energy _ LF_Energy
Above " HF_Energy+ LF_Energy Below " HF_Energy+ LF_Energy " " Above and "HF_energy", "LF_energy" are absolute levels of high and low frequency energy, and redHF is the energy reduction in the high frequency band and redLF is the energy reduction in the low frequency band.
This method can be implemented in hardware on a micro-controller or in software on a logiciel support executed by a microprocessor from a non volatile memory of the ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory) kind or an equivalent.
The method of the invention can also be carried out by a computer program for a processing unit comprising a set of instructions, which, when loaded into said processing unit, causes the processing unit to carry out the method described above. There are indeed numerous ways of implementing functions by means of items of hardware or software, or both. In this respect, the drawings are very diagrammatic, each representing only one possible embodiment of the invention. Thus, although a drawing shows different functions as different blocks, this by no means excludes that a single item of hardware or software carries out several functions. Nor does it exclude that an assembly of items of hardware or software or both carry out a function.
The remarks made herein before demonstrate that the detailed description, with reference to the drawings, illustrates rather than limits the invention, and that numerous alternatives, which fall within the scope of the appended claims, are possible. Any reference sign in a claim should not be construed as limiting the claim. The word "comprising" does not exclude the presence of other elements or steps than those listed in a claim. The word "a" or "an" preceding an element or step does not exclude the presence of a plurality of such elements or steps.

Claims

CLAIMS :
1. A method (18) to evaluate and select the stabilization quality of at least a stabilization method (1) applied to an original video sequence (8) composed of several images (2, 4, 6) to obtain a stabilized video sequence (16), the evaluation method (18) being adapted to assign a quality factor (Q) to the stabilization method (1) according to the quality of the obtained stabilized video sequence (16), wherein the evaluation method (18) comprising the following steps:
- extracting (20) a set of initial global motion vectors (GMV°2, GMV°4) from the original video sequence (8);
- determining (30) a set of initial energy quantities (W°(a), W°(b), W°(c)) per predefined frequency band (a, b, c), from the energy spectrum of the set of initial global motion vectors (GMV°2, GMV°4) ;
- stabilizing (32) the original video sequence (8) using the stabilization method (1), to obtain the stabilized video sequence (16);
- extracting (34) a set of final global motion vectors (GMVfio, GMVfi2) from the stabilized video sequence (16);
- determining (36) a set of final energy quantities (Wf(a), Wf(b), Wf(c)) per predefined frequency band (a, b, c), from the energy spectrum of the set of final global motion vectors (GMVfi0, GMVf 12);
- determining (38) a set of energy differences (Wd(a), Wd(b), Wd(c)) per predefined frequency band (a, b, c), each energy difference (Wd(a), Wd(b), Wd(c)) resulting from the subtraction for each predefined frequency band (a, b, c) of a final energy quantity (Wf(a), Wf(b), Wf(c)) to the initial energy quantity (W°(a), W°(b), W°(c)); and
- determining (50) the quality factor (Q) featuring the stabilization method (1), wherein the step (50) of determining the quality factor (Q) includes a step (40, 42, 50) of determining a motion component, the motion component being proportional to a weighted sum of the energy differences (W (a), W (b), W (c)) of the set of energy differences (Wd(a), Wd(b), Wd(c)).
2. A method according to claim 1, wherein the step (40, 42, 50) of determining the motion component comprises the following steps:
- determining (40, 42) a set of frequency- based weighting coefficients, (Energy_in_i, Energy_out_i, A_in_i, A_out_i) which are function of the predefined frequency bands (a, b, c); and - weighting (50) each energy difference (Wd(a), Wd(b), Wd(c)) for each frequency band (a, b, c), by a combination of the determined frequency-based weighting coefficient (Energy_in_i, Energy_out_i, A_in_i, A_out_i) of the corresponding frequency band (a, b, c).
3. A method according to claim 2, wherein the step (40, 42) of determining the set of frequency-based weighting coefficients (Energy_in_i, Energy_out_i, A_in_i, A_out_i) comprises the following steps :
a) considering (40) a frequency band (a, b, c) among the predefined frequency bands (a, b, c);
b) determining (40) an inside band energy coefficient (Energy_in_i) which is proportional to the initial energy quantity (W°(a), W°(b), W°(c)) of the considered frequency band (a, b, c);
c) determining (40) an outside band energy coefficient (Energy_out_i) which is proportional to the sum of all initial energy quantities (W°(a), W°(b), W°(c)) at the exception of the initial energy quantity of the considered frequency band (a, b, c); and d) repeating (40) steps a) to c) for all predefined frequency bands (a, b, c) to obtain a couple of frequency-based weighting coefficients per predefined frequency band (a, b, c), each couple comprising an inside band energy coefficient (Energy_in_i) and an outside band energy coefficient (Energy_out_i).
4. A method according to claim 3, wherein the step (40, 42) of determining the set of frequency-based weighting coefficients (Energy_in_i, Energy_out_i, A-in_i, A_out_i) comprises the following step :
- adding (42) all initial energy quantities (W°(a), W°(b), W°(c)) corresponding to all predefined frequency bands (a, b, c) to obtain a total energy (Total_energy);
- determining (42) a set of relative inside band energy coefficients (A_in_i) by dividing each inside band energy coefficient (Energy_in_i) by the total energy (Total_energy), the frequency-based weighting coefficients (Energy_in_i, Energy_out_i, A_in_i, A_out_i) comprising the set of relative inside band energy coefficients (A_in_i);
- determining (42) a set of relative outside band energy coefficients (A_out_i) by dividing each outside band energy coefficient (Energy_out_i) by the total energy (Total_energy), the frequency-based weighting coefficients (Energy_in_i, Energy_out_i, A_in_i, A_out_i) comprising the set of relative outside band energy coefficients (A_out_i).
5. A method according to any of the preceding claims, wherein the step of determining the quality factor (Q) includes a step (44, 46) of determining a set of image- based weighting coefficients (IM, TI), the step (44, 46) of determining the set of image- based weighting coefficients (IM, TI) comprises the following steps:
a) considering (24) two consecutive images (2, 4, 6) in the original video sequence (8); b) determining (24) a set of motion vectors (Vk) of macro blocks (24) of the considered images (2, 4, 6);
c) determining (26) the global motion vector (GMV°2, GMV0^) of the set of motion vectors (Vk);
d) determining (44) an inner motion coefficient by calculating the difference between each motion vector and the global motion vector (GMV°2, GMV°4) of the set of motion vectors (Vk); and
e) repeating (44) steps a) to d) for all images (2, 4, 6) of the original video sequence (8) and calculating the average of the set of determined inner motion coefficients to obtain a inner global motion coefficient (IM) ; the image- based weighting coefficients (IM, TI) comprising the inner global motion coefficient (IM).
6. A method according to any of the preceding claims 1 to 5, wherein the step of determining the quality factor (Q) includes a step (44, 46) of determining a set of image- based weighting coefficients (IM, TI), the step (44, 46) of determining the set of image- based weighting coefficients (IM, TI) comprises the following steps:
a) considering (46) an image (2, 4, 6) in the original video sequence (8);
b) determining (46) a texture rate of the considered image (2, 4, 6) using a texture analysis method; and
c) repeating (46) steps a) and b) for all images (2, 4, 6) of the original video sequence (8) and calculating the average of the set of determined texture rates to obtain an average texture rate (TI), the image-based weighting coefficients (IM, TI) comprising the average texture rate (TI).
7. A method according to claims 2 and 5 or 6 in combination, wherein the step (40, 42, 50) of determining the motion component of the quality factor (Q) comprises a step (50) of multiplying each image-based weighting coefficient (IM, TI) to the sum of all final energy quantities (Wf(a), Wf(b), Wf(c)).
8. A method according to any of the preceding claims, wherein the step of determining the quality factor (Q) comprising the following steps:
- determining (44, 46, 48) an image component;
- adding (50) the image component to the motion component; and wherein the step of determining the image component (44, 46, 48) comprises the following steps:
a) considering (48) two consecutive images (2, 4, 6) in the original video sequence (8); b) computing (48) the inter-frame transformation fidelity (ITF) index of the considered images (2, 4, 6); and
c) repeating (48) steps a) and b) for all images (2, 4, 6) of the original video sequence (8) and calculating the mean of the set of Inter-frame Transformation Fidelity index (ITF) computed, the image component being proportional to the mean inter-frame
transformation fidelity (ITF).
9. A method according to claims 8 and 5 or 6 in combination, wherein the step (44, 46, 48) of determining the image component comprises a step of multiplying each image- based weighting coefficients (IM, TI) to the mean Inter-Frame Transformation fidelity index (ITF).
10. A method according to claims 4 and 8 or 9 in combination, wherein the step (44, 46, 48) of determining the image component comprises the following steps :
- computing (50) a weighted sum of the relative outside band energy coefficients (A_out_i) and of the relative inside band energy coefficients (A_in_i) over all frequency bands (a, b, c) ; and
- multiplying the computed sum by the mean Inter-frame Transformation Fidelity index (ITF).
11. A method according to claims 2 and 5 in combination, wherein the step (44, 46, 48) of determining the quality factor (Q) comprises the following steps:
- determining (50) empirically factors by viewing stabilized sequences (16) obtained from different original sequences (8) showing different scenes, and
- multiplying (50) each determined factor to each frequency and image-based weighting coefficients.
12. A device (15) to evaluate and select the stabilization quality of at least a stabilization method (1) applied to an original video sequence (8) composed of several images (2, 4, 6) to obtain a stabilized video sequence (16), the evaluation method (18) being adapted to assign a quality factor (Q) to the stabilization method (1) according to the quality of the obtained stabilized video sequence (16), wherein the device (15) comprises: - motion estimation means (17) adapted to extract a set of initial global motion vectors (GMV°2, GMV°4) from the original video sequence (8) and a set of final global motion vectors (GMVflo, GMVfi2) from the stabilized video sequence (16);
- energy analysis means (19) adapted to determine a set of initial energy quantities (W°(a), W°(b), W°(c)) by predefined frequency band (a, b, c), of the energy spectrum of the set of initial global motion vectors (GMV°2, GMV°4) and a set of final energy quantities (Wf(a), Wf(b), Wf(c)) by predefined frequency band (a, b, c) of the energy spectrum of the set of final global motion vectors (GMVfio, GMVfi2);
- computing means (23) adapted to determine a set of energy differences (Wd(a), Wd(b), Wd(c)) by frequency band (a, b, c), each energy difference (Wd(a), Wd(b), Wd(c)) resulting from the subtraction of a final energy quantity (Wf(a), Wf(b), Wf(c)) of a defined frequency band (a, b, c) to the initial energy quantity (W°(a), W°(b), W°(c)) of the same frequency band (a, b, c); and
- the computing means (23) being adapted to determine a quality factor (Q) characterizing the evaluated stabilization method (1), wherein the step (50) of determining the quality factor (Q) includes a step of determining a motion component, the motion component being proportional to a weighted sum of the energy differences (Wd(a), Wd(b), Wd(c)) of the set of energy differences (Wd(a), Wd(b), Wd(c)).
13. A device for digital image stabilization for removing unwanted camera movements, called jitter, from an original sequence of images generated by said camera and obtaining a stabilized sequence of images, said device incorporating a stabilization quality metric based on a previous evaluation and selection of stabilization quality which are performed in a device according to claim 12.
14. A computer program for a processing unit comprising a set of instructions which, when loaded into said processing unit, causes the processing unit to carry out the steps of the method as claimed in any of the claims 1 to 11.
PCT/IB2006/054558 2005-12-05 2006-12-01 Method and device for the selection of a stabilization method WO2007066268A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP05301005 2005-12-05
EP05301005.4 2005-12-05

Publications (2)

Publication Number Publication Date
WO2007066268A2 true WO2007066268A2 (en) 2007-06-14
WO2007066268A3 WO2007066268A3 (en) 2007-09-27

Family

ID=38039777

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2006/054558 WO2007066268A2 (en) 2005-12-05 2006-12-01 Method and device for the selection of a stabilization method

Country Status (1)

Country Link
WO (1) WO2007066268A2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108040225A (en) * 2017-12-11 2018-05-15 梁金凤 A kind of intelligent monitoring machine people

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
AUBERGER S ET AL: "Digital Video Stabilization Architecture for Low Cost Devices" IMAGE AND SIGNAL PROCESSING AND ANALYSIS, 2005. ISPA 2005. PROCEEDINGS OF THE 4TH INTERNATIONAL SYMPOSIUM ON SEPTEMBER 15-17, 2005, PISCATAWAY, NJ, USA,IEEE, 15 September 2005 (2005-09-15), pages 474-479, XP010843828 ISBN: 953-184-089-X *
MATSUSHITA Y ET AL: "Full-Frame Video Stabilization" COMPUTER VISION AND PATTERN RECOGNITION, 2005. CVPR 2005. IEEE COMPUTER SOCIETY CONFERENCE ON SAN DIEGO, CA, USA 20-26 JUNE 2005, PISCATAWAY, NJ, USA,IEEE, 20 June 2005 (2005-06-20), pages 50-57, XP010817414 ISBN: 0-7695-2372-2 *
MORIMOTO C ET AL: "Evaluation of image stabilization algorithms" ACOUSTICS, SPEECH AND SIGNAL PROCESSING, 1998. PROCEEDINGS OF THE 1998 IEEE INTERNATIONAL CONFERENCE ON SEATTLE, WA, USA 12-15 MAY 1998, NEW YORK, NY, USA,IEEE, US, vol. 5, 12 May 1998 (1998-05-12), pages 2789-2792, XP010279442 ISBN: 0-7803-4428-6 *
UOMORI K ET AL: "ELECTRONIC IMAGE STABILIZATION SYSTEM FOR VIDEO CAMERAS AND VCRS" SMPTE JOURNAL, SMPTE INC. SCARSDALE, N.Y, US, vol. 101, no. 2, February 1992 (1992-02), pages 66-75, XP000252752 ISSN: 0036-1682 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108040225A (en) * 2017-12-11 2018-05-15 梁金凤 A kind of intelligent monitoring machine people

Also Published As

Publication number Publication date
WO2007066268A3 (en) 2007-09-27

Similar Documents

Publication Publication Date Title
EP3800878B1 (en) Cascaded camera motion estimation, rolling shutter detection, and camera shake detection for video stabilization
Tan et al. A multi-metric objective picture-quality measurement model for MPEG video
KR970009469B1 (en) Interlace/sequential scan conversion apparatus and method for facilitating double smoothing function
KR101830804B1 (en) Digital image stabilization method with adaptive filtering
US8805121B2 (en) Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames
JP5038483B2 (en) Video data deinterlacing
US20110109765A1 (en) Visual processing device, visual processing method, program, display device, and integrated circuit
US9635308B2 (en) Preprocessing of interlaced video with overlapped 3D transforms
Van Roosmalen et al. Correction of intensity flicker in old film sequences
JP2004343715A (en) Frame interpolating method at frame rate conversion and apparatus thereof
WO2005060272A1 (en) Image and video quality measurement
Ghadiyaram et al. A no-reference video quality predictor for compression and scaling artifacts
JP2000502549A (en) Noise filter processing
US20040013196A1 (en) Quantization control system for video coding
JP2004518337A (en) Apparatus and method for providing a useful metric based on coded information for video enhancement
US7711044B1 (en) Noise reduction systems and methods
WO2007066268A2 (en) Method and device for the selection of a stabilization method
US20060181643A1 (en) Spatial image conversion
JPH1098695A (en) Image information converter and its device and product sum arithmetic unit
WO2006016336A1 (en) A unit for and method of image conversion
Oelbaum et al. Building a reduced reference video quality metric with very low overhead using multivariate data analysis
CN111369459B (en) Method and device for optimizing global tone mapping contrast
JP5084755B2 (en) Noise reduction apparatus and program thereof
JP2007179211A (en) Image processing device, image processing method, and program for it
Tenze et al. Design and real-time implementation of a low-cost noise reduction system for video applications

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application
NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 06832050

Country of ref document: EP

Kind code of ref document: A2