CN107516300B - Hybrid jitter correction method for unmanned aerial vehicle video acquisition - Google Patents

Hybrid jitter correction method for unmanned aerial vehicle video acquisition Download PDF

Info

Publication number
CN107516300B
CN107516300B CN201710719877.XA CN201710719877A CN107516300B CN 107516300 B CN107516300 B CN 107516300B CN 201710719877 A CN201710719877 A CN 201710719877A CN 107516300 B CN107516300 B CN 107516300B
Authority
CN
China
Prior art keywords
image
kalman
observation
filter
aerial vehicle
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710719877.XA
Other languages
Chinese (zh)
Other versions
CN107516300A (en
Inventor
吴占雄
吴东南
杨宇翔
曾毓
何志伟
高明煜
黄继业
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Dianzi University
Original Assignee
Hangzhou Dianzi University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Dianzi University filed Critical Hangzhou Dianzi University
Priority to CN201710719877.XA priority Critical patent/CN107516300B/en
Publication of CN107516300A publication Critical patent/CN107516300A/en
Application granted granted Critical
Publication of CN107516300B publication Critical patent/CN107516300B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a mixed jitter correction method for unmanned aerial vehicle video acquisition, which comprises the following steps:acquiring a pre-acquired image from an unmanned aerial vehicle video image; calculating a system state transition matrix A of the Kalman filter by using characteristic points of image rotation generated by jitter in a pre-acquired image; constructing a Kalman filter system equation and a process observation equation; calculating the estimated value of the system state of the pre-collected image at the time k according to the system noise variance Q and the observation noise variance R
Figure DDA0001384712770000011
And an estimate of a state observation
Figure DDA0001384712770000012
(ii) a Calculating Kalman gain and calculating characteristic parameters of Kalman evaluation images by combining the Kalman gain; luminance matrix of Kalman evaluation image by using discretization Gaussian filter
Figure DDA0001384712770000013
And carrying out low-pass filtering to filter system noise and observation noise to obtain a corrected image without jitter.

Description

Hybrid jitter correction method for unmanned aerial vehicle video acquisition
Technical Field
The invention relates to the field of video image processing, in particular to a mixed jitter correction method for unmanned aerial vehicle video acquisition.
Background
With the continuous development of the unmanned aerial vehicle technology, the application of the unmanned aerial vehicle in aerial shooting or video monitoring is gradually wide, and the unmanned aerial vehicle is gradually applied to the high-precision detection fields of surveying and mapping, detection and the like. The unmanned aerial vehicle can shake when moving under the influence of factors such as air flow and motor self vibrations to make the image that the machine carried camera was shot there be phenomenons such as rocking, vibrations and distortion, this can cause the video interframe unstability that unmanned aerial vehicle shot, and interframe viewpoint transform angle is big, motion blur. The task execution is seriously influenced, and the difficulty is brought to the subsequent processing.
In addition, for the small unmanned aerial vehicle, the unmanned aerial vehicle has small volume and high flying speed, and is more easily interfered, so that the scout video sometimes has large jitter, and particularly, when a tracking target is converted, the edge and detail distortion is serious. Compared with the aerial photo shooting, the aerial photo shooting period is relatively long, the speed is relatively slow, and the obtained image is relatively stable, so that for the same ground object, the resolution difference between the aerial photo and the reconnaissance video is large, the viewpoint conversion and the angle change between frames of the reconnaissance video are large, and the motion blur problem and the noise problem exist at the same time.
Disclosure of Invention
The invention provides a mixed jitter correction method for unmanned aerial vehicle video acquisition, aiming at solving the problem that the phenomena of shaking, vibration, distortion and the like exist in a shot image due to jitter during image acquisition of the existing unmanned aerial vehicle.
In order to achieve the above object, the present invention provides a mixed jitter correction method for unmanned aerial vehicle video acquisition, comprising the following steps:
acquiring a pre-acquired image from an unmanned aerial vehicle video image;
calculating a system state transition matrix A of the Kalman filter by using characteristic points of image rotation generated by jitter in the pre-acquired image:
Figure GDA0002437350600000021
where θ is the rotation angle of the adjacent images, TxFor the translation of adjacent images on the horizontal axis, TyThe amount of translation of adjacent images on the vertical axis;
constructing a Kalman filter system equation and a process observation equation according to a system state transition matrix A of the Kalman filter;
calculating the estimated value of the system state of the pre-collected image at the time k according to the system noise variance Q and the observation noise variance R
Figure GDA0002437350600000022
And an estimate of a state observation
Figure GDA0002437350600000023
Figure GDA0002437350600000024
Figure GDA0002437350600000025
Based on system state estimates
Figure GDA0002437350600000026
And state observation estimates
Figure GDA0002437350600000027
Calculating Kalman gain and calculating characteristic parameters of a Kalman evaluation image by combining the Kalman gain: :
Kg＝Pk-1HT(HPk-1HT+R)-1
Figure GDA0002437350600000028
Figure GDA0002437350600000029
Figure GDA00024373506000000210
wherein, KgAs Kalman gain, xkEvaluating the system state value, P, of an image for KalmankTo evaluate the state observations of the images for kalman,
Figure GDA00024373506000000211
luminance matrix for Kalman evaluation of images, H is cardA system observation matrix in the process observation equation of the Kalman filter;
luminance matrix of Kalman evaluation image by using discretization Gaussian filter
Figure GDA00024373506000000212
And carrying out low-pass filtering to filter system noise and observation noise to obtain a corrected image without jitter.
In an embodiment of the present invention, constructing the kalman filter system equation according to the system state transition matrix a of the kalman filter is:
Xk+1＝AXk+Wk
wherein, XkIs the system state at time k, Xk+1The system state at time k +1, A is the system state transition matrix, WkIs the system noise.
In an embodiment of the present invention, the constructed kalman filter process observation equation is:
Zk＝HXk+Vk
wherein Z iskIn order to be a state observation value,
Figure GDA0002437350600000031
Vkto observe noise.
In an embodiment of the present invention, the filter function of the discretized gaussian filter is:
Figure GDA0002437350600000032
where σ is 4, the filter image frame number r is 10, and k is [ -r, r ].
In an embodiment of the invention, the feature points of the pre-captured image that generate image rotation due to shaking are four corner points of the image or a reference object in the image.
In summary, the hybrid shake correction method for unmanned aerial vehicle video acquisition provided by the invention constructs a kalman dynamic motion model capable of actually describing inter-frame motion caused by airborne camera motion by pre-acquiring feature points of image rotation generated by shake in an image. An evaluation image containing system noise and observation noise is obtained through a recursive estimation method, and finally the system noise and the observation noise are filtered by a discretized Gaussian filter, so that a good jitter correction effect can be achieved, an over-processing phenomenon can be avoided, and the problem of inaccurate parameter change trend prediction can be avoided. Furthermore, the Kalman filter is an algorithm with small calculation amount and flexibility, so that the mixed jitter correction method for unmanned aerial vehicle video acquisition provided by the invention has the advantages of high image processing speed and good real-time property.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
Fig. 1 is a flowchart illustrating a hybrid jitter correction method for unmanned aerial vehicle video capture according to an embodiment of the present invention.
Detailed Description
As shown in fig. 1, the hybrid shake correction method for drone video capture provided by the present embodiment starts with step S1, obtaining a pre-capture image from within a drone video image. In this embodiment, the video image collected by the unmanned aerial vehicle is transmitted to the cloud storage, so that the pre-collected image can be obtained in the cloud storage. However, the storage or transmission mode of the images of the unmanned aerial vehicle is not limited in any way.
After the pre-collected image is obtained, a Kalman filter is adopted to estimate the jitter parameters, and the Kalman filter is an algorithm with small calculation amount and flexibility. A large number of experiments prove that the Kalman filter is adopted to process the pre-acquired images of the unmanned aerial vehicle, and the interframe motion of the video images formed by the jitter of the airborne camera can be well described. The small and flexible algorithm enables the hybrid shake correction method for unmanned aerial vehicle video acquisition provided by the embodiment to have fast correction speed, the real-time performance is very good, and the image acquisition speed of the existing unmanned aerial vehicle can be well met.
Specifically, the method comprises the following steps: step S2 is executed to calculate the system state transition matrix a of the kalman filter using the feature points of the pre-captured image where the image rotation is caused by the shake. The image that unmanned aerial vehicle machine carried camera can make when the shake and shoot takes place to rotate, and the rotatory characteristic point of representation image that can be fine on the image is four corners of image, and four corner regions once have a small rotation homoenergetic to embody, so in this embodiment, the characteristic point that produces the image rotation because of the shake is four corner points of image. However, the present invention is not limited thereto. In other embodiments, when the image contains a reference object with a large area, the rotation caused by the shake is also very obvious on the reference object, and in this case, a point on the reference object can be used as a characteristic point of the image rotation caused by the shake.
In this embodiment, the system state transition matrix a of the kalman filter is calculated by the following formula.
Figure GDA0002437350600000041
Where θ is the rotation angle of the adjacent images, TxFor the translation of adjacent images on the horizontal axis, TyIs the amount of translation of the adjacent images on the vertical axis.
After the system state transition matrix a of the kalman filter is obtained, step S3 is executed to construct a system equation and a process observation equation of the kalman filter according to the system state transition matrix a of the kalman filter. In this embodiment, the kalman filter system equation is:
Xk+1＝AXk+Wk
wherein, XkIs the system state at time k, Xk+1The system state at time k +1, A is the system state transition matrix, WkIs the system noise.
The Kalman filter process observation equation is:
Zk＝HXk+Vk
wherein Z iskIn order to be a state observation value,
Figure GDA0002437350600000051
Vkto observe noise.
Because the system state transition matrix A is obtained by calculation according to the characteristic points on the pre-collected image, the Kalman filter system equation and the process observation equation established according to the system state transition matrix A can accurately describe the interframe motion of the video image caused by the jitter of the airborne camera.
Next, step S4 is executed to calculate an estimated value of the system state of the pre-captured image at time k based on the system noise variance Q and the observation noise variance R
Figure GDA0002437350600000052
And an estimate of a state observation
Figure GDA0002437350600000053
Figure GDA0002437350600000054
Figure GDA0002437350600000055
Obtaining an estimate of the state of the system
Figure GDA0002437350600000056
And an estimate of a state observation
Figure GDA0002437350600000057
Then, step S5 is executed to obtain the kalman gain by using a recursive estimation method and calculate the feature parameters of the kalman estimated image by combining the kalman gain. The specific calculation formula is as follows, wherein KgAs Kalman gain, xkEvaluating the system state value, P, of an image for KalmankTo evaluate the state observations of the images for kalman,
Figure GDA0002437350600000058
luminance matrix for Kalman evaluation imagesAnd H is a system observation matrix in a Kalman filter process observation equation.
Kg＝Pk-1HT(HPk-1HT+R)-1
Figure GDA0002437350600000061
Figure GDA0002437350600000062
Figure GDA0002437350600000063
Luminance matrix of an elman evaluation image
Figure GDA0002437350600000064
The method comprises the characteristic parameters of representing the rotation of the pre-acquired image, such as a system state value, a system state observation value, Kalman gain and the like of the pre-acquired image.
Step S6, adopting discretization Gaussian filter to evaluate the brightness matrix of the image to Kalman
Figure GDA0002437350600000065
And carrying out low-pass filtering to filter system noise and observation noise to obtain a corrected image without jitter. The filter function of the discretized gaussian filter is:
Figure GDA0002437350600000066
where σ is 4, the filter image frame number r is 10, and k is [ -r, r ].
In summary, the hybrid shake correction method for unmanned aerial vehicle video acquisition provided by the invention constructs a kalman dynamic motion model capable of actually describing inter-frame motion caused by airborne camera motion by pre-acquiring feature points of image rotation generated by shake in an image. An evaluation image containing system noise and observation noise is obtained through a recursive estimation method, and finally the system noise and the observation noise are filtered by a discretized Gaussian filter, so that a good jitter correction effect can be achieved, an over-processing phenomenon can be avoided, and the problem of inaccurate parameter change trend prediction can be avoided. Furthermore, the Kalman filter is an algorithm with small calculation amount and flexibility, so that the mixed jitter correction method for unmanned aerial vehicle video acquisition provided by the invention has the advantages of high image processing speed and good real-time property.
Although the present invention has been described with reference to the preferred embodiments, it should be understood that various changes and modifications can be made therein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (2)

1. A hybrid shake correction method for unmanned aerial vehicle video acquisition is characterized by comprising the following steps:
acquiring a pre-acquired image from an unmanned aerial vehicle video image;
calculating a system state transition matrix A of the Kalman filter by using characteristic points of image rotation generated by jitter in the pre-acquired image:
Figure FDA0002437350590000011
where θ is the rotation angle of the adjacent images, TxFor the translation of adjacent images on the horizontal axis, TyThe amount of translation of adjacent images on the vertical axis;
constructing a Kalman filter system equation and a process observation equation according to a system state transition matrix A of the Kalman filter;
calculating the estimated value of the system state of the pre-collected image at the time k according to the system noise variance Q and the observation noise variance R
Figure FDA0002437350590000012
And an estimate of a state observation
Figure FDA0002437350590000013
Figure FDA0002437350590000014
Figure FDA0002437350590000015
Based on system state estimates
Figure FDA0002437350590000016
And state observation estimates
Figure FDA0002437350590000017
Calculating Kalman gain and calculating characteristic parameters of a Kalman evaluation image by combining the Kalman gain:
Kg＝Pk-1HT(HPk-1HT+R)-1
Figure FDA0002437350590000018
Figure FDA0002437350590000019
Figure FDA00024373505900000110
wherein, KgAs Kalman gain, xkEvaluating the system state value, P, of an image for KalmankTo evaluate the state observations of the images for kalman,
Figure FDA00024373505900000111
a brightness matrix of the Kalman evaluation image is used, and H is a system observation matrix in a Kalman filter process observation equation;
using dispersionLuminance matrix of Gaussian filter pair Kalman evaluation image
Figure FDA0002437350590000023
Low-pass filtering is carried out to filter system noise and observation noise, and a corrected image without jitter is obtained;
constructing a Kalman filter system equation according to a system state transition matrix A of the Kalman filter as follows:
Xk+1＝AXk+Wk
wherein, XkIs the system state at time k, Xk+1The system state at time k +1, A is the system state transition matrix, WkSystem noise;
the constructed Kalman filter process observation equation is as follows:
Zk＝HXk+Vk
wherein Z iskIn order to be a state observation value,
Figure FDA0002437350590000021
Vkto observe noise;
the filter function of the discretized gaussian filter is:
Figure FDA0002437350590000022
where σ is 4, the filter image frame number r is 10, and k is [ -r, r ].
2. The hybrid shake correction method for unmanned aerial vehicle video capture according to claim 1, wherein the feature points within the pre-captured image that cause image rotation due to shake are four corner points of the image or a reference within the image.
CN201710719877.XA 2017-08-21 2017-08-21 Hybrid jitter correction method for unmanned aerial vehicle video acquisition Active CN107516300B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710719877.XA CN107516300B (en) 2017-08-21 2017-08-21 Hybrid jitter correction method for unmanned aerial vehicle video acquisition

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710719877.XA CN107516300B (en) 2017-08-21 2017-08-21 Hybrid jitter correction method for unmanned aerial vehicle video acquisition

Publications (2)

Publication Number Publication Date
CN107516300A CN107516300A (en) 2017-12-26
CN107516300B true CN107516300B (en) 2020-05-22

Family

ID=60723475

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710719877.XA Active CN107516300B (en) 2017-08-21 2017-08-21 Hybrid jitter correction method for unmanned aerial vehicle video acquisition

Country Status (1)

Country Link
CN (1) CN107516300B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110401784B (en) * 2018-04-24 2021-09-21 展讯通信(天津)有限公司 Motion smoothing method and system for adaptively adjusting filtering intensity and video equipment

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103873743A (en) * 2014-03-24 2014-06-18 中国人民解放军国防科学技术大学 Video de-noising method based on structure tensor and Kalman filtering
CN104135598A (en) * 2014-07-09 2014-11-05 清华大学深圳研究生院 Method and device of stabilizing video image

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103873743A (en) * 2014-03-24 2014-06-18 中国人民解放军国防科学技术大学 Video de-noising method based on structure tensor and Kalman filtering
CN104135598A (en) * 2014-07-09 2014-11-05 清华大学深圳研究生院 Method and device of stabilizing video image

Also Published As

Publication number Publication date
CN107516300A (en) 2017-12-26

Similar Documents

Publication Publication Date Title
Bloesch et al. Robust visual inertial odometry using a direct EKF-based approach
JP7078139B2 (en) Video stabilization methods and equipment, as well as non-temporary computer-readable media
KR102563750B1 (en) Method and Device of Image Deblurring
Hee Park et al. Gyro-based multi-image deconvolution for removing handshake blur
CN105611116B (en) A kind of global motion vector method of estimation and monitor video digital image stabilization method and device
CN109618103B (en) Anti-shake method for unmanned aerial vehicle image transmission video and unmanned aerial vehicle
CN103139568B (en) Based on the Video Stabilization method of degree of rarefication and fidelity constraint
CN109743495B (en) Electronic stability augmentation method and device for video image
JP7151488B2 (en) Moving object detection device, moving object detection method and program
US9838604B2 (en) Method and system for stabilizing video frames
US9406108B2 (en) Deblurring of an image from a sequence of images
CN104282003B (en) Digital blurred image blind restoration method based on gradient screening
WO2018053994A1 (en) Adaptive motion filtering in an unmanned autonomous vehicle
CN107222662A (en) A kind of electronic image stabilization method based on improved KLT and Kalman filtering
CN109015633B (en) Visual servo tracking method, device and system
WO2018053809A1 (en) Adaptive image processing in an unmanned autonomous vehicle
CN106550187A (en) For the apparatus and method of image stabilization
JP2019046434A (en) System and method for image motion compensation
CN108900775B (en) Real-time electronic image stabilization method for underwater robot
CN114979489A (en) Gyroscope-based heavy equipment production scene video monitoring and image stabilizing method and system
CN107360377B (en) Vehicle-mounted video image stabilization method
CN107343145A (en) A kind of video camera electronic image stabilization method based on robust features point
CN107516300B (en) Hybrid jitter correction method for unmanned aerial vehicle video acquisition
KR101538923B1 (en) Real-time video stabilizer for tracking for region of interest
KR101202642B1 (en) Method and apparatus for estimating global motion using the background feature points

Legal Events

Date Code Title Description
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant