CN111355881A - A Video Stabilization Method to Eliminate Rolling Shutter Artifacts and Jitters Simultaneously - Google Patents

Abstract

The invention discloses a video stabilization method for simultaneously eliminating rolling shutter artifacts and jitter, comprising the following steps: 1) gridding video frames; 2) estimating motion between frames; 3) constructing data fidelity terms and motion smoothing rules 4) Construct a joint optimization model for intra-frame motion; 5) Estimate intra-frame motion; 6) Set adaptive sliding window; 7) Calculate adaptive weights; 8) Solve restoration transformation; 9) Simultaneously remove rolling shutter pseudo Shadow stabilized video generation. By gridding each frame of the video, the present invention uses homography transformation to simulate inter-frame motion, rigid transformation simulates intra-frame motion, establishes a joint model of inter-frame and intra-frame motion, and directly solves the restoration of rolling shutter artifacts and de-jittering. Transform; this method can simultaneously achieve video rolling shutter artifact removal and video stabilization, avoiding the problem of over-smoothing in the video de-shake process, and can be widely used in mobile phone shooting, drone shooting, vehicle navigation, etc. using CMOS cameras types of video stabilization.

Description

A Video Stabilization Method to Eliminate Rolling Shutter Artifacts and Jitters Simultaneously

technical field

The present invention relates to the stabilization technology of shaking video, in particular to a video stabilization method for simultaneously eliminating rolling shutter artifacts and shaking.

Background technique

In the field of video processing and display, video signals captured by CMOS cameras through vehicle-mounted camera platforms, UAV or ship camera systems, handheld camera equipment, etc. often have rolling shutter artifacts due to the camera's progressive imaging, and the shooting The random disturbance of the process is also easy to cause video jitter. On the one hand, these video degradations can easily cause visual fatigue of video observers and affect the content understanding of video images, leading to misjudgment or omission by observers; Subsequent processing of video, such as tracking, recognition and pattern analysis, etc.

At present, there have been many methods for rolling shutter artifacts and video jitter separation processing, such as robust mesh restoration methods [Yeong Jun Koh, Chulwoo Lee, and Chang-Su Kim. 2015. Video StabilizationBased on Feature Trajectory Augmentation and Selection and Robust Mesh GridWarping. IEEE Transactions on Image Processing 24, 12(2015), 5260-5273.] and subspace methods [Feng Liu, Michael Gleicher, Jue Wang, Hailin Jin and AseemAgarwala. 2011. Subspace video stabilization. ACM Transactions on Graphics 30 , 1 (2011), 4.]. The robust mesh repair method adopts the framework of separation processing, which firstly estimates the motion trajectories without rolling shutter artifacts, and then performs motion smoothing on the corrected feature trajectories. In contrast, the subspace method treats rolling artifact as a set of structured noises and implicitly removes the rolling artifact during video stabilization.

However, both the robust mesh repair method and the subspace method are based on dealing with rolling shutter artifacts or video jitter respectively, and the algorithm process is complicated.

SUMMARY OF THE INVENTION

The object of the present invention is to provide a video stabilization method that simultaneously eliminates rolling shutter artifacts and jitter.

The technical solution that realizes the object of the present invention is: a kind of video stabilization method that simultaneously eliminates rolling shutter artifact and jitter, and the method comprises the following steps:

并定义同一网格列下相邻网格之间的曝光时间为单位时间；Step 1. Gridded video frames: Assume that the observed jittering video sequence with rolling shutter artifacts is: {I _t |t∈[1,N]}, where N represents the number of frames of the video sequence, for each frame The video image is defined as an 8×8 grid, then the grid of the i-th row and the j-th column in the _t -th frame It can be expressed as

And define the exposure time between adjacent grids under the same grid column as unit time;

其中i,j∈[1,8],t∈[1,N]；Step 2. Estimate inter-frame motion: apply feature points to the grids corresponding to adjacent frames in the video sequence to obtain motion feature points, and use the random sampling consistency method to calculate the rigid transformation matrix and homography of each frame of video image. matrix, then the rigid transformation matrix and the homography matrix of the grid at the ith row and the jth column of the t-th frame can be expressed as:

where i,j∈[1,8],t∈[1,N];

的在单位时间内的 帧内运动为

利用当前帧I_t的前后帧I_t-1与I_t+1；Step 3. Construct data fidelity term and motion smoothing regular term: define grid

The intra-frame motion in unit time is

_Utilize the frame _It-1 and _It+1 before and after the current frame It;

According to the fidelity between the sum of the intra-frame motion and the inter-frame motion under the same grid column, construct the inter-frame intra-frame motion data fidelity term

According to the similarity of intra-frame motion under the condition of dense sampling density, construct the smoothing regular term of intra-frame motion

Step 4. Build a joint optimization model for intra-frame motion: According to the constraints constructed in Step 3, build a joint optimization model for intra-frame motion: argmin _{F} P(F)+λQ(F), where the regularization parameter λ>0;

Step 5. Estimate the motion in the frame: According to the additivity of the angle θ, the horizontal displacement x and the vertical displacement y in the rigid transformation matrix parameters, the joint model in Step 4 is optimized for the three parameters, and finally solved separately. The three parameters of the angle θ, horizontal displacement x and vertical displacement y of the intra-frame motion are combined to solve the intra-frame motion matrix

与步骤5中得到的帧内运动矩阵

s的取值范围为[0,30]的整数；Step 6. Set the adaptive sliding window: adopt windowing processing for each grid, and set the window size to s, then the inter-frame motion matrix of the _t -th frame It obtained in step 2

with the intra-frame motion matrix obtained in step 5

The value range of s is an integer in the range of [0,30];

其中G(·)表 示高斯函数，最终可获得一组权重向量

并为同一帧的网格估计一个统一的 权重向量：

||·||₁为矩阵的L1范数；Step 7. Calculate the adaptive weight: Calculate the time distance and spatial distance between the current grid and the kth frame global shutter point grid, then the weight

where G( ) represents the Gaussian function, and finally a set of weight vectors can be obtained

and estimate a uniform weight vector for the grid of the same frame:

||·|| ₁ is the L1 norm of the matrix;

可求解第t帧I_t中第i行第j列的网格 的复原变化，其中w_t,k为自适应权重，

为窗口中网格

到网格

之间的帧间 运动的累积，

为第k帧i行j列网格到该网格列全局快门点的帧内运动累积， 则

可表示当前网格到第k帧全局快门点的总运动；Step 8. Solve the restoration transformation: according to the adaptive weight obtained in step 7, according to the relationship

It can solve the restoration change of the grid of the i-th row and the j-th column in the _t -th frame It, where w _t,k is the adaptive weight,

grid in the window

to grid

The accumulation of inter-frame motion between,

is the accumulation of the intra-frame motion from the grid to the global shutter point of the kth frame i row j column, then

It can represent the total motion from the current grid to the global shutter point of the kth frame;

对每一帧视频图像的每一网格进行重新绘制，最终生成去卷帘伪影的稳 定的视频图像序列。Step 9. Simultaneously remove the rolling shutter artifacts and stabilize the video generation: according to the transformation matrix

Each grid of each frame of video image is redrawn, and finally a stable video image sequence with rolling shutter artifacts is generated.

的帧内运动将与同一 网格列中的其他网格

共享，可得到性质

等 式左侧为网格

经过8个单位时间帧内运动的累积，右侧为该网格对应的帧间 运动，依据此性质可设计帧内运动之和与帧间运动之间的数据保真项P(F)。Further, when constructing the data fidelity item in step 3, according to the grid

The intra-frame motion will be shared with other grids in the same grid column

shared, available

The left side of the equation is the grid

After the accumulation of motion in 8 unit time frames, the right side is the inter-frame motion corresponding to the grid. According to this property, the data fidelity term P(F) between the sum of intra-frame motion and inter-frame motion can be designed.

的帧内运动

其与同一网格列下一网格行 的网格

的帧内运动

应具有相似性，则可设计平滑正则项

Further, when constructing the motion smoothing regular term in step 3, according to the similarity that the intra-frame motion should have under high-frequency sampling conditions, for the grid

Intra-frame motion

its grid with the same grid column next grid row

Intra-frame motion

should have similarity, then a smooth regular term can be designed

其中θ、x与y分别表示两两相邻网格之间的旋 转角度、水平位移与垂直位移；该变换具有的可加性，即

Further, the rigid transformation in step 5 is defined as a three-degree-of-freedom transformation

where θ, x and y represent the rotation angle, horizontal displacement and vertical displacement between two adjacent grids, respectively; the transformation has additivity, that is,

Further, according to the additive property of rigid transformation in step 5, the optimization model in step 4 can be transformed into an optimization model with three degrees of freedom. For horizontal displacement, f represents the horizontal displacement within the frame, and r represents Horizontal displacement between frames, then the data fidelity term and motion smoothing term in step 3 are transformed into:

垂直位移与旋转角度求解方法同水平位移；该模型以矩阵-矢量形式 可表示为：The optimization model is transformed into argmin _{f} P(f)+λQ(f), and the horizontal displacement of each grid can be obtained through the new model

The solution methods of vertical displacement and rotation angle are the same as horizontal displacement; the model can be expressed in matrix-vector form as:

时，定义网格

到网格

的时 间距离与空间距离分别为|t-k|与

则为两个网 格之间的水平距离，

为两个网格之间的垂直距离。Further, when constructing the adaptive weight in step 7

, define the grid

to grid

The temporal and spatial distances are |tk| and

is the horizontal distance between the two grids,

is the vertical distance between the two grids.

为第k帧第i行j列网格到全局快门点的帧内运 动累积，若以第4网格行作为全局快门点，则第t帧I_t中第k行第j列的网格的帧内 运动复原矩阵可表示为：

通过

可去除每一网 格的卷帘伪影。Further, as defined in step 8

It is the accumulation of the intra-frame motion from the grid of the i-th row and _j column of the k-th frame to the global shutter point. If the 4th grid row is used as the global shutter point, the grid of the k-th row and the j-th column in the t-th frame It is The intra-frame motion restoration matrix can be expressed as:

pass

Rolling shutter artifacts can be removed for each grid.

Compared with the prior art, the present invention has the following significant advantages: (1) For the degradation problem of rolling-shutter artifact and jitter appearing at the same time in low-quality video, the prior art often requires two separate methods for rolling-shutter artifact and jitter removal. preprocessing process, and the present invention establishes an optimization model of joint processing, which can overcome the under-smoothing and over-smoothing in video stabilization through intra-frame and inter-frame motion estimation of gridded video frames and full-time adjustment of adaptive sliding window. smoothing problem; (2) the present invention makes full use of the inter-frame and intra-frame motion correlation, and adaptively adjusts the weight parameters, so that the method of the present invention has good effects in both video de-jitter and video rolling-shutter artifact removal at the same time. (3) The present invention utilizes homography transformation to simulate inter-frame motion, and rigid transformation simulates intra-frame motion through each frame of gridded video, establishes a joint model of inter-frame intra-frame motion, and directly solves the rolling shutter artifact and The recovery transformation of de-shake can be widely used in various types of video stabilization, such as mobile phone shooting, drone shooting, and car navigation using CMOS cameras.

Description of drawings

FIG. 1 is a flow chart of the video stabilization method for simultaneously eliminating rolling shutter artifacts and jitter according to the present invention.

Figure 2(a) is the first frame of the first test video.

Figure 2(b) is the second frame of the first test video.

Figure 2(c) is the residual map between the original two frames.

Fig. 2(d) is the residual image between two frames after only using this method to de-jitter.

Figure 2(e) is a residual map between two frames after processing by the robust mesh repair method.

Figure 2(f) is the residual image between two frames after simultaneous de-jitter and rolling-shutter artifact removal by this method.

Figure 3(a) is the original feature point trajectory map.

Figure 3(b) is the feature point trajectories processed by the robust mesh repair method.

Figure 3(c) is the feature point trajectory after processing by the subspace method.

Fig. 3(d) is a trajectory diagram of the feature points processed by the method of the present invention.

Figure 4(a) is the original image of three randomly selected frames in the test video.

Figure 4(b) is the result of the three-frame image processed by the robust mesh repair method.

FIG. 4( c ) is the result of processing the three-frame image by the subspace method.

FIG. 4(d) is a result diagram of the three-frame image processed by the method of the present invention.

(1)-(10) in FIG. 5 are 10 test video images including jitter and rolling shutter artifacts used in the experiment of the present invention, respectively.

FIG. 6 is a graph showing the results of video visual evaluation for 35 users.

Detailed ways

Aiming at the video shot by the CMOS camera, the invention realizes the effect of simultaneously removing the rolling shutter artifacts and the jitter in the video, and proposes a joint optimization for simultaneously eliminating the rolling shutter artifacts and the jitter by establishing an estimation model for the motion between frames and frames. method. Below in conjunction with Fig. 1, the implementation process of the present invention is described in detail:

并定义同一网格列下相邻网格之间的曝光时间为单位时间。Step 1. Gridded video frames: Assume that the observed jittering video sequence with rolling shutter artifacts is: {I _t |t∈[1,N]}, where N represents the number of frames of the video sequence, for each frame The video image is defined as an 8×8 grid, then the grid of the i-th row and the j-th column in the _t -th frame It can be expressed as

And define the exposure time between adjacent grids under the same grid column as unit time.

其中i,j∈[1,8],t∈[1,N]。Step 2. Estimate inter-frame motion: use feature points to detect the grids corresponding to adjacent frames in the video sequence to obtain dense motion feature points, and use the random sampling consistency method to calculate the rigid transformation matrix of each frame of video image and the single frame. The rigid transformation matrix and the homography matrix of the grid at the ith row and the jth column of the t-th frame can be expressed as:

where i,j∈[1,8],t∈[1,N].

的在单位时间内的 帧内运动为

利用当前帧I_t的前后帧I_t-1与I_t+1。Step 3. Construct data fidelity term and motion smoothing regular term: define grid

The intra-frame motion in unit time is

Use the frames _It-1 and _{It+1 before} and after the current frame It.

According to the fidelity between the sum of the intra-frame motion and the inter-frame motion under the same grid column, construct the inter-frame intra-frame motion data fidelity term

According to the similarity of intra-frame motion under the condition of dense sampling density, construct an intra-frame motion smoothing regular term

的帧内运动将与同一网格列中的 其他网格

共享，可得到性质

等式左侧为网 格

经过8个单位时间帧内运动的累积，右侧为该网格对应的帧间运动，依据 此性质可设计帧内运动之和与帧间运动之间的数据保真项P(F)。Step 3.1. When constructing data fidelity items, according to the grid

The intra-frame motion will be shared with other grids in the same grid column

shared, available

The left side of the equation is the grid

After the accumulation of motion in 8 unit time frames, the right side is the inter-frame motion corresponding to the grid. According to this property, the data fidelity term P(F) between the sum of intra-frame motion and inter-frame motion can be designed.

的帧内运动

其与同一网格列下一网格行的网格

的帧内运动

应具有相似性，则可设计平滑正则项

Step 3.2. When constructing the motion smoothing regular term, according to the similarity that the motion in the frame should have under the condition of high frequency sampling, for the grid

Intra-frame motion

its grid with the same grid column next grid row

Intra-frame motion

should have similarity, then a smooth regular term can be designed

Step 4. Build a joint optimization model for intra-frame motion: According to the constraints constructed in Step 3, build a joint optimization model for intra-frame motion: argmin _{F} P(F)+λQ(F), where the regularization parameter λ>0.

Step 5. Estimate the motion in the frame: According to the additivity of the angle θ, the horizontal displacement x and the vertical displacement y in the rigid transformation matrix parameters, the joint model in Step 4 is optimized for the three parameters, and finally solved separately. The three parameters of the angle θ, horizontal displacement x and vertical displacement y of the intra-frame motion are combined to solve the intra-frame motion matrix

其 中θ,x与y分别表示两两相邻网格之间的旋转角度，水平位移与垂直位移。依据该 变换具有的可加性，即

Step 5.1, the rigid transformation is defined as a three-degree-of-freedom transformation

where θ, x and y represent the rotation angle, horizontal displacement and vertical displacement between two adjacent grids, respectively. According to the additivity of this transformation, that is

Step 5.2. According to the additivity of rigid transformation, the optimization model in step 4 can be transformed into an optimization model with three degrees of freedom. For example, for horizontal displacement, f represents the horizontal displacement within the frame, and r represents the inter-frame horizontal displacement. Horizontal displacement, the data fidelity term and motion smoothing term in step 3 are transformed into:

并推广到垂直位移与旋转角度上。该模型以矩阵-矢量形式可表示为：The optimization model is transformed into argmin _{f} P(f)+λQ(f), and the horizontal displacement of each grid can be obtained through the new model

And generalized to vertical displacement and rotation angle. The model can be represented in matrix-vector form as:

与步骤5中得到的帧内运动矩阵

s的取值范围为[0,30]的整数。Step 6. Set the adaptive sliding window: adopt windowing processing for each grid, and set the window size to s, then the inter-frame motion matrix of the _t -th frame It obtained in step 2

with the intra-frame motion matrix obtained in step 5

The value range of s is an integer in the range [0,30].

其中G(·)表 示高斯函数，最终可获得一组权重向量

并为同一帧的网格估计一个统一的 权重向量：

为矩阵的L1范数。Step 7. Calculate the adaptive weight: Calculate the time distance and spatial distance between the current grid and the kth frame global shutter point grid, then the weight

where G( ) represents the Gaussian function, and finally a set of weight vectors can be obtained

and estimate a uniform weight vector for the grid of the same frame:

is the L1 norm of the matrix.

时，定义网格

到网格

的时间距离与空间距离分 别为|t-k|与

则为两个网格之间的水平距离，

为两个网格之间的垂直距离。When constructing adaptive weights

, define the grid

to grid

The temporal and spatial distances are |tk| and

is the horizontal distance between the two grids,

is the vertical distance between the two grids.

可求解第t帧I_t中第i行第j列的网格 的复原变化，其中w_t,k为自适应权重，

为窗口中网格

到网格

之间的帧间 运动的累积，

为第k帧i行j列网格到改网格列全局快门点的帧内运动累积， 则

可表示当前网格到第k帧全局快门点的总运动。Step 8. Solve the restoration transformation: according to the adaptive weight obtained in step 7, according to the relationship

It can solve the restoration change of the grid of the i-th row and the j-th column in the _t -th frame It, where w _t,k is the adaptive weight,

grid in the window

to grid

The accumulation of inter-frame motion between,

is the intra-frame motion accumulation from the grid of the kth frame i row j column to the global shutter point of the new grid column, then

It can represent the total motion of the current grid to the global shutter point of the kth frame.

为第k帧i行j列网格到改网格列全局快门点的帧内运动累积，若以 第4网格行作为全局快门点，则第t帧I_t中第k行第j列的网格的帧内运动复原矩阵 可表示为：

通过

可去除每一网格的卷帘伪影。definition

It is the intra-frame motion accumulation from the grid in the i row and _j column of the kth frame to the global shutter point of the changed grid column. The intra-frame motion restoration matrix of the grid can be expressed as:

pass

Rolling shutter artifacts can be removed for each grid.

对每一帧视频图像的每一网格进行重新绘制，最终生成去卷帘伪影的稳 定的视频图像序列。Step 9. Simultaneously remove the rolling shutter artifacts and stabilize the video generation: according to the transformation matrix

Each grid of each frame of video image is redrawn, and finally a stable video image sequence with rolling shutter artifacts is generated.

The effect of the present invention can be further illustrated by the following simulation experiments:

(1) Simulation conditions

Ten groups of jitter video data containing rolling shutter artifacts are used in the simulation experiments. The simulation experiments are all completed under the Windows 7 operating system using Matlab R2012. The processor is a Xeon W3520 CPU (2.66GHz), and the memory is 4GB. The initialization values of each parameter in the simulation experiment are: the regularization parameter λ is set to 1, the standard deviations of the two Gaussian functions are 6 and 30, respectively, and the window length is 2s+1 and s=30.

The performance of the method of the present invention is analyzed by qualitative evaluation of the user's visual subjective experience. In the experiment, 35 participants subjectively scored the resulting videos of different stabilization methods. For fair testing, participants selected videos with better human visual perception when the specific methods were unknown.

(2) Simulation content

The present invention adopts real shaking video data to check the de-shaking performance of the algorithm, and the test video is a shaking video containing rolling shutter artifacts. In order to test the performance of the algorithm of the present invention, the proposed video stabilization method is compared with the current international mainstream method. The comparison methods include: robust mesh repair method and subspace method.

(3) Analysis of simulation results

Figures 2(a) to 2(f) are the residuals of two frames of images obtained by different methods of removing the rolling shutter artifact, and Figures 3(a) to 3(d) are the feature points processed by different video stabilization methods. Trajectory diagrams, Figure 4(a)~Figure 4(d) are the results obtained by different methods of simultaneously removing rolling shutter artifacts and jitter, Figure 5 is 10 test videos, Figure 6 is 35 users on 10 test videos visual evaluation results.

In Figures 2(a) to 2(f), the frame difference method is used to visualize the restoration effect. Figure 2(a) is the first frame of the first test video, Figure 2(b) is the second frame of the first test video, and Figure 2(c) is the residual image between the original two frames, Figure 2(d) is the residual image between the two frames after de-jittering with this method, Figure 2(e) is the residual image between the two frames after the robust mesh repair method, and Figure 2(f) is the residual map between the two frames after simultaneous de-jittering and rolling-shutter artifact removal by this method. It can be clearly observed that using this method to remove jitter and rolling shutter artifacts at the same time obtains the smallest residual between the two frames of images, which can well remove the rolling shutter artifacts and jitter effects contained in the video.

Figure 3(a) is the original feature point trajectory diagram, Figure 3(b) is the feature point trajectory diagram processed by the robust grid repair method, and Figure 3(c) is the feature point trajectory diagram processed by the subspace method, Fig. 3(d) is a trajectory diagram of the feature points processed by the method of the present invention. It can be observed that the method of the present invention also has a good effect when dealing with video jitter. Due to the adaptive weight design, when dealing with the jitter of the step motion type, the phenomenon of over-smoothing is not easy to occur, which shows that the method of the present invention is effective in It also has excellent results in video de-shaking.

Figure 4(a) is the original image of three frames randomly selected from the test video, Figure 4(b) is the result image of the three-frame image processed by the robust grid repair method, and Figure 4(c) is the image of the three-frame image after sub-processing Figure 4(d) is the result of the three-frame image processed by the method of the present invention. As can be seen from Figure 4(a)~Figure 4(d), the subspace method only treats the rolling shutter artifact as a kind of structured noise, which is implicitly processed in the process of video debounce, resulting in spatial distortion. phenomenon, while both the robust mesh repair method and the method of the present invention have a specific step to deal with the rolling shutter artifact, so the object can be corrected to a certain extent. Furthermore, our results preserve more image information than the other two methods in the case of fast motion due to the adaptive weights employed in the present invention.

Figure 6 shows a graph of video visual evaluation results for 35 users. Since people may have different standards, it is difficult to devise a metric to quantitatively assess the effects of stabilization and shutter correction. Therefore, we conducted a user survey of 35 participants for qualitative comparisons. Fig. 5 is a randomly selected test video. For each test video, when the user does not know the three repair methods (robust grid repair method, subspace method and the method of the present invention), he chooses the best visual effect. 's video. In the test cases of the 7th, 9th, and 10th videos, no participant chose the subspace method, because of the obvious geometric distortion, the visual experience was greatly reduced. In addition, more users are more inclined to the method of the present invention, they believe that the method of the present invention achieves a better balance between motion smoothing and information preservation, and therefore believe that the method of the present invention is superior to the other two state-of-the-art methods, achieving Simultaneously removes video jitter and removes rolling shutter artifacts.

Claims (7)

1. a kind of video stabilization method that eliminates rolling shutter artifact and shaking simultaneously, it is characterised in that the method comprises the following steps: Step 1. Gridded video frames: Assume that the observed jittering video sequence with rolling shutter artifacts is: {I _t |t∈[1,N]}, where N represents the number of frames of the video sequence, for each frame The video image is defined as an 8×8 grid, then the grid of the i-th row and the j-th column in the _t -th frame It can be expressed as

i, j∈[1,8], t∈[1,N], and define the exposure time between adjacent grids under the same grid column as unit time; Step 2. Estimate inter-frame motion: apply feature points to the grids corresponding to adjacent frames in the video sequence to obtain motion feature points, and use the random sampling consistency method to calculate the rigid transformation matrix and homography of each frame of video image. matrix, then the rigid transformation matrix and the homography matrix of the grid at the ith row and the jth column of the t-th frame can be expressed as:

where i, j∈[1,8], t∈[1,N]; Step 3. Construct data fidelity term and motion smoothing regular term: define grid

The intra-frame motion in unit time is

_Utilize the frame _It-1 and _It+1 before and after the current frame It; According to the fidelity between the sum of the intra-frame motion and the inter-frame motion under the same grid column, construct the inter-frame intra-frame motion data fidelity term

According to the similarity of intra-frame motion under the condition of dense sampling density, construct an intra-frame motion smoothing regular term

Step 4. Build a joint optimization model for intra-frame motion: According to the constraints constructed in step 3, build a joint optimization model for intra-frame motion: arg min _{F} P(F)+λQ(F), where regularization parameter λ>0; Step 5. Estimate the motion in the frame: According to the additivity of the angle θ, the horizontal displacement x and the vertical displacement y in the rigid transformation matrix parameters, the joint model in Step 4 is optimized for the three parameters, and finally solved separately. The three parameters of the angle θ, horizontal displacement x and vertical displacement y of the intra-frame motion are combined to solve the intra-frame motion matrix

Step 6. Set the adaptive sliding window: adopt windowing processing for each grid, and set the window size to s, then the inter-frame motion matrix of the _t -th frame It obtained in step 2

with the intra-frame motion matrix obtained in step 5

The value range of s is an integer in the range of [0, 30]; Step 7. Calculate the adaptive weight: Calculate the time distance and spatial distance between the current grid and the kth frame global shutter point grid, then the weight

where G( ) represents the Gaussian function, and finally a set of weight vectors can be obtained

and estimate a uniform weight vector for the grid of the same frame:

is the L1 norm of the matrix; Step 8. Solve the restoration transformation: according to the adaptive weight obtained in step 7, according to the relationship

i, j∈[1,8], t∈[1,N] can solve the restoration change of the grid in the i-th row and j-th column in the t-th frame I _t , where w _{t, k} are the adaptive weights,

grid in the window

to grid

The accumulation of inter-frame motion between,

is the intra-frame motion accumulation from the grid to the global shutter point of the kth frame i row j column, then

It can represent the total motion from the current grid to the global shutter point of the kth frame; Step 9. Simultaneously remove the rolling shutter artifacts and stabilize the video generation: according to the transformation matrix

Each grid of each frame of video image is redrawn, and finally a stable video image sequence with rolling shutter artifacts is generated. 2. the video stabilization method of eliminating rolling shutter artifact and shaking simultaneously according to claim 1, is characterized in that: when constructing data fidelity item in step 3, according to grid

The intra-frame motion will be shared with other grids in the same grid column

shared, available

The left side of the equation is the grid

After the accumulation of motion in 8 unit time frames, the right side is the inter-frame motion corresponding to the grid. According to this property, the data fidelity term P(F) between the sum of intra-frame motion and inter-frame motion can be designed. 3. the video stabilization method that eliminates rolling shutter artifact and jitter simultaneously according to claim 1, is characterized in that: when constructing motion smoothing regular term in step 3, should have under high-frequency sampling condition according to intra-frame motion. similarity, for grids

Intra-frame motion

its grid with the same grid column next grid row

Intra-frame motion

should have similarity, then a smooth regular term can be designed

4. the video stabilization method of eliminating rolling shutter artifact and shaking simultaneously according to claim 1, is characterized in that: in step 5, rigid transformation is defined as three degrees of freedom transformation

where θ, x and y represent the rotation angle, horizontal displacement and vertical displacement between two adjacent grids, respectively; the transformation has additivity, that is,

5. the video stabilization method that eliminates rolling shutter artifact and jitter simultaneously according to claim 1, is characterized in that: in step 5, according to rigid transformation has the property of additivity, then the optimization model in step 4 can be transformed is the optimization model on three degrees of freedom. For horizontal displacement, f represents the horizontal displacement within the frame, and r represents the horizontal displacement between frames. Then the data fidelity term and motion smoothing term in step 3 are transformed into:

The optimization model is transformed into arg min _{f} P(f)+λQ(f), and the horizontal displacement of each grid can be obtained through the new model

The solution methods of vertical displacement and rotation angle are the same as horizontal displacement; the model can be expressed in matrix-vector form as:

. 6. the video stabilization method that eliminates rolling shutter artifact and shaking simultaneously according to claim 1, it is characterized in that: when constructing self-adaptive weight in step 7

, define the grid

to grid

The temporal and spatial distances are |tk| and

is the horizontal distance between the two grids,

is the vertical distance between the two grids. 7. the video stabilization method that eliminates rolling shutter artifact and jitter simultaneously according to claim 1, it is characterized in that: define in step 8

It is the accumulation of the intra-frame motion from the grid of the i-th row and _j column of the k-th frame to the global shutter point. If the 4th grid row is used as the global shutter point, the grid of the k-th row and the j-th column in the t-th frame It is The intra-frame motion restoration matrix can be expressed as:

pass

Rolling shutter artifacts can be removed for each grid.

Images