WO2020081018A1

WO2020081018A1 - Video processing-based image stabilization method

Info

Publication number: WO2020081018A1
Application number: PCT/TR2018/050598
Authority: WO
Inventors: Cevahir ÇIĞLA
Original assignee: Aselsan Elektroni̇k Sanayi̇ Ve Ti̇caret Anoni̇m Şi̇rketi̇
Priority date: 2018-10-17
Filing date: 2018-10-17
Publication date: 2020-04-23

Abstract

The invention is a method which enables to eliminate the horizontal and vertical motion on live footage depending on the vibration of the camera; comprising the process steps of detecting sub blocks to be used for shift detection, detecting the shift values based on the optimization process through phase correlation between each consecutive frame for the sub blocks and by accumulating the detected shift values on the time- scale axis; providing stabilization by means of applying the cumulative shifts on the recent frame in the reverse direction.

Description

Video Processing-Based Image Stabilization Method Technical Field

The invention is related to a method which enables to eliminate the horizontal and vertical motion on live footage depending on the vibration of the camera by utilizing image processing methods.

Prior Art

Today, vibrations of the camera depending on external factors cause an inconvenience while watching live footage and the vibrations hinder the use of the camera.

Some of the methods widely used in the prior art for preventing vibrations detect the shifting of the images in the vertical and horizontal direction by looking at the correlation between sequential frames. In almost all of these methods, shifts between consecutive frames are calculated over the entire image or by using all the blocks. This approach works at reasonable levels for the scenes which have enough intensity modulation and texture. At this point, most of the image should have textural areas. However, considering the usage area of the surveillance cameras, this intensity modulation distribution may not always be at the desired level. At this level, the shift values to be calculated over the entire image may not correspond to real vibration values. Especially for the scenes in which the areas having non-textural and similar color distribution are predominant, the pixel vibration values to be calculated over the entire image will be misleading.

The above mentioned similar problem is the same with the methods using feature points. For the scenes in which noise and non-textural areas are in the majority, inconsistencies will be formed during the matching of the feature points to be established. Especially the corner points which will appear depending on the noise are sensitive to mismatching.

Projection (profile) based methods widely used for eliminating vibration do not appear to be the optimal choice, especially for the cases in which moving targets are large or numerous. The moving targets may cause irregularities during the extraction of profiles that result in errors in the estimation of the camera vibration movements. The other type of approach is the optical flow-based stabilization that has high computational complexity since it estimates a motion value for each pixel. Besides, calculation of consistent optical flow values for the images in which there is not enough texture difference does not give reliable estimates.

As a result, due to the problems described above and the insufficiency of the present solutions on the subject, an innovation is required in the related technical field.

The Purpose of the Invention

The invention has been drawn up by being inspired from the present situations and aims to solve the above-mentioned problems.

The main purpose of the invention is to eliminate the vibrations in the image by only processing the images obtained from the camera, without adding additional sensors to the camera assembly. The method realized by the system is designed such that it shall process within the video frame rate the camera generates and shall not cause any disruption in live video streaming. By this means, a more user friendly video stream which will not cause any problems during observation or processing will be provided by means of eliminating camera vibrations to occur depending on any kind of external factors.

Another purpose of the invention is to enable to reduce false alarms that occur during vibrations (moving target detection, target tracking) for any automatic video processing algorithm, besides enhancing the video watching experience by the method that has been developed. This method provides significant contribution in terms of reducing false alarms that will be received from the cameras installed for security purposes.

In order to fulfill the above-mentioned purposes, the invention is a method which enables to eliminate horizontal and vertical motion on live footage depending on the vibration of the camera; comprising the procedure steps of detecting sub blocks to be used for shifting detection, calculation of the shift values with the optimization process through phase correlation between each consecutive frame for the sub blocks and by accumulating the detected shift amount on the time-scale axis; providing the stabilization by means of applying the cumulative shift values on the recent frame in the reverse direction. The structural and characteristic features and the advantages of the invention will be understood more clearly from the figures given below and the detailed description written by means of referring to these figures, and for this reason, the evaluation should be made in consideration of these figures and the detailed description.

Figures Illustrating the Invention

Figure 1 is the view of the flow diagram of the method subject to the invention.

Figure 2 is the view of the approach in which the image is divided into sub blocks.

Figure 3 is the graphical view of the exemplary sharpness calculation on a single dimensional correlation.

Figure 4 is the view of the exemplary auto-correlation maps for the blocks which are not suitable for shift estimation and the ones that are.

Detailed Description of the Invention

In this detailed description, the preferred embodiments of the video processing-based image stabilization method subject to the invention are described in order to enable better understanding of the subject.

The image stabilization method proposed by the invention is basically composed of three stages. In the first stage, the sub blocks to be used for shift detection are determined. Since the application area, in principle, is surveillance cameras, the alteration amount in the video is below a certain level under normal conditions. Therefore, the sub block detection process is performed at certain intervals (every other M frame) rather than on each frame and the determined sub blocks are used for a short period of time for sequential frames. At the second stage, the global pixel shifts are detected with the optimization process through phase correlation between each consecutive frame for the sub blocks. The estimated shifts are accumulated on the time-scale axis; and the stabilization is provided by means of applying the cumulative shift values calculated on the last frame depending on the reference image, to the recent frame in reverse direction. The basic stages of the method are shown in the flow diagram in Figure 1.

Determining Sub Blocks for Shift Detection The two basic features of the invention which differentiates the invention from similar methods is that the phase correlation is performed through a specific number of reliable sub blocks which have passed the compliance test, instead of performing it on the entire frame and that the intensity modulation values are used vertically and horizontally by providing edge detection instead of the intensity values in the images. By this means, sturdy optimal correlation values are obtained and the computational complexity is reduced.

For the detection of sub blocks, first, depending on the image resolution, as shown in Figure 2, a specific number of frames are divided into blocks. At this stage, blocking may be provided by means of blocks overlapping at certain proportions. During all of these processes, use of different pixels around by using vertical and horizontal intensity modulation amounts instead of camera output intensity values is provided. Conformity of each block to the phase correlation that will be used in the shift calculation is determined by auto correlation. For this reason, by taking the phase correlation of each block with itself, a measurement on the texture structure it comprises is performed.

As shown in Equation 5, the phase correlation is obtained by applying the Fourier transform (F) to any block (B,), with the inverse Fourier transform (F¹) of its multiplication with its conjugate (F^*). The auto correlation maps {AC obtained for each block is analyzed according to two criteria. The maximum value ( max_corr ) in the auto correlation maps is expected at the center since no shifting is in question for the block. In this context, the first criterion is chosen as the ratio between the maximum value at the center and the maximum value ( max_corr_exclude ) outside a specific region surrounding the center (AC ^¾:). The typical representation for the central region and the region outside is determined by the limit formed by the dashed line in the correlation maps of Figure 3. At this point, the width of the central region may differ in proportion to the block dimension used. In general, values between 5-20% of the block width, yields reasonable results.

In Equation 1 given below, the first criterion indicated as the sharpness value {Pi) indicates how high the global maximum value is in comparison to the maximum value outside the specified center.

(max eorr— min corr)

F, - - - = - = - - -

(m x_ i?rr_e i£ae‘ - mii _corr) (Equation 1 ) raax.corr = maxfAC (Equation 2) vam_c&rr = min (/I ) (Equation 3) max_carr_e?:¾; ¾ = mm ^ Ί (Equation 4) (Equation 5)

(Equation 6)

The sharpness value (P) determined for each sub block being high shows that the block can be separated from its surroundings, irrespective of the different shift values and that it is convenient for shift detection. Low sharpness values (P) indicate that the block is unreliable and can lead to false shift detection by depending on local maximum values. In Figure 3, the sharpness calculation approach is exemplified through the one dimensional correlation signal and the same processes may be provided by extending to two dimensional for calculation over the image. By disregarding a specific width around the determined maximum value ( max_corr ), a second maximum value ( max_corr_exclude ) is found and the processes provided in Equation 1 are performed. As shown in Figure 4, the phase correlation map for the blocks convenient for shift detection should be collected at the center.

The second criterion for the block is the total energy amount at the center of the phase correlation map (.4C_E). This value should be high for a reliable block. The main reason for this is that high correlation indicates that there are enough texture changes within the block.

At this stage, in order to prevent the errors that will result from the moving objects in the scene, a moving target detection algorithm depending on any background modelling should be run. As shown in Figure 1 , a moving target detection block output is used during sub block detection. The sharpness value (P) of the blocks comprising the possible targets that will be received from motion detection and the center energy values are initialized and these blocks are disregarded. Accumulating the outputs from the motion detection at certain intervals enables to cover the areas which are likely to feature motion and secures a more optimal elimination. Thereafter, the blocks are limited according to the calculated sharpness value {Pi) and the central energy values. This stage is chosen for N number of block phase correlations having the highest central energy value in the one-third fraction from among the blocks that have been limited initially, according to the sharpness value Pi). At this stage, blocks having sufficient texture changes which can be separated from their surroundings are determined.

Calculatina the Shifts

The stage following the detection of the blocks suitable for shift calculation, is to determine the correlation maps between two consecutive frames {t, t+1 ) over the blocks. In this context, the phase correlation (FK) is performed as below, through the intensity modulation values received from two different frames for each block region (index number i in Equation 7). At this point, attention should be paid that no movement is present within the sub blocks. Since sub block selection is performed at certain intervals, a moving object appearing in any region can cause an alteration in the related sub blocks. In this case, in order to provide a reliable shift detection, each sub block in which correlation will be performed is checked and blocks which do not comprise any motion are determined. i e {l, N} (Equation 7)

Correlation maps calculated for each block are added and the cumulative correlation (Equation 8) is found. At this point, different conjugation approaches may also be used. For example, blocks providing a similar shift amounts are grouped and the outputs obtained from the incorrect sub blocks can be disregarded. Another alternative is to perform (w_s) weighted addition by calculating the correlation reliability coefficient for each block. Weight calculation can be provided by using the sub block sharpness ratio used in Figure 1 or by similar approaches. The maximum point of the (Equation 8) correlation map (FK) obtained by weighted or cumulative addition provides the vertical and horizontal plane shift amount ({D.c, Dn}*).

FA'* = åf_{= 1 Wj}FK? (Equation 8)

{ ^c, Dn}* = argmax^ lFK* } (Equation 9) The shift values obtained in Equation 9 correspond to the modulation between two consecutive frames. Since image stabilization will be performed according to the reference frame for a global stabilization, for n number of frames, these values are accumulated cumulatively and the calculation of how much of a correction is to be applied for the frame at that second is carried out.

(Equation 10)

At the last stage, the vertical and horizontal shift amounts determined for the / second are applied to the image at that second and stabilization is provided. In order to ensure that the system works for moving cameras, filtration is provided over the shift values measured instantaneously and the usual average movement (Ax, Ay) the camera comprises is detected. The camera vibration amount can be determined by subtracting these average values from the cumulative value.

(Equation 1 1 )

At this point, an alternative solution exists apart from the shift amount accumulation. Following the selection of sub blocks in a reference frame, the effective shift values of that moment can be calculated over the shift calculation by the reference frame, for each new frame. In this approach, since shift values are calculated according to a specific reference, timewise accumulation is not needed.

By means of using a limited number of sub blocks, two advantages are obtained. The first advantage is that the camera resolutions increase together with the improving surveillance systems and fast processing ability is provided, which is necessary for real-time systems depending on the increase in frames per second. In comparison with the phase correlation to be calculated over the entire image, at least 20 times of speed increase is provided. Another advantage is that with the detection of suitable sub blocks which are suitable for shift calculation in terms of content, an optimal shift detection of the image can be obtained which is independent from the general texture distribution. Shift detection approaches to be realized in the areas which do not have texture do not operate properly in any way; hence, any shift value to be calculated over the entire image for the scenes in which non-textural areas are in the majority will probably be incorrect. Detecting the suitable sub blocks provides a useful solution to this problem. Using the moving objects as a criterion in sub block detection prevents the use of sub blockss that include moving objects and are not proper, and provides a scene-sensitive video stabilization.

Claims

1. A method which eliminates horizontal and vertical motion on live footage depending on the vibration of the camera, characterized in that it comprises the process steps of;

• detecting the sub blocks (B,) to be used for shift detection at certain intervals,

• detecting the shift amounts by means of the optimization process via phase correlation between each consecutive frame for the sub blocks (B,),

• providing stabilization by means of applying the cumulative shift values which is calculated on the last frame depending on the reference image, to the image in reverse direction by collecting the detected shift amount on the time-scale axis.

2. Method according to Claim 1 , characterized in that sub block (B,) detection comprises the process steps of,

• determining the phase correlation map {AC by carrying out the Fourier transform (F(Bi)) of each sub block (B,), and the inverse Fourier transform (F^_1) of its multiplication with its conjugate (F^*(Bi)),

• calculating the high value ( max_corr ) and low value (min_corf of the phase correlation map (AC_t).

• calculating the ratio of the phase correlation map (A ) to the phase correlation central value (AC,^center) and calculating the maximum value ( max_corr_exclude ) excluded from a specific region surrounding the center,

• calculating the sharpness value ( P i) with the ratio of the difference between the high value ( max_corr ) and the low value ( min corr ) to the difference between the maximum value ( max_corr_exclude ) excluded from a specific region surrounding the center and the low value ( min corr ),

• calculating the sharpness value ( P ,) suitable for the shift detection process.

3. Method according to Claim 1 , characterized in that the shift amount detection comprises the process steps of,

• calculating the phase correlation (FK) by the inverse Fourier transform (F¹) of the multiplication of the conjugate (F^*(Bi’)) of the Fourier transform of the sub block (Bi) at t frame and the Fourier transform (F^*(Bi^t_1 )) of the sub block (Bi) at t-1 frame, where f and t+1 are two consecutive frames,

• calculating the cumulative correlation by adding the correlation maps (FK) calculated for each sub block (Bi),

• determining the maximum point of the correlation map (FK) in order to detect the shifting amount ({D,c_,AnU) on the vertical and horizontal frame,

• providing the cumulative sum {Ax, Ay} of the shift amounts ({άc,DgU) for n number of frames,

• calculating the usual average values (Si,ly) the camera has by providing filtration over the shift values which are instantaneously measured for moving cameras,

• calculating the camera vibration amount by subtracting the average values (Ex_r&y) from the cumulative values {Ax, Ay}.