CN114095659B - Video anti-shake method, device, equipment and storage medium - Google Patents
Video anti-shake method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114095659B CN114095659B CN202111429727.8A CN202111429727A CN114095659B CN 114095659 B CN114095659 B CN 114095659B CN 202111429727 A CN202111429727 A CN 202111429727A CN 114095659 B CN114095659 B CN 114095659B
- Authority
- CN
- China
- Prior art keywords
- shake
- path
- video
- thumbnail
- calculating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 239000011159 matrix material Substances 0.000 claims abstract description 94
- 239000013598 vector Substances 0.000 claims abstract description 65
- 230000009466 transformation Effects 0.000 claims abstract description 22
- 238000009499 grossing Methods 0.000 claims abstract description 17
- 230000001629 suppression Effects 0.000 claims abstract description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 27
- 238000004364 calculation method Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 abstract description 22
- 230000006870 function Effects 0.000 description 7
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 238000009792 diffusion process Methods 0.000 description 4
- 238000006073 displacement reaction Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6811—Motion detection based on the image signal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/21—Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
Abstract
The invention discloses a video anti-shake method, a device, equipment and a storage medium, which comprise the following steps: downsampling the video image frames to obtain corresponding thumbnail images; performing multi-scale block search by using the thumbnail, and calculating a motion vector of each pixel point between two image frames; calculating a homography transformation matrix of multiple degrees of freedom by using the motion vector of each pixel point; traversing the video, and calculating a homography transformation matrix of each frame, namely a motion path; the motion path is subjected to smoothing treatment to obtain an optimized path; performing anti-shake suppression processing based on the optimized path to calculate an optimal path; and the optimal path is utilized to perform anti-shake processing on each image frame of the video, so that the whole processing scheme not only can obtain a higher-precision anti-shake effect, but also has no short board in terms of operation efficiency, thereby realizing the effect of real-time operation of low-end mobile equipment.
Description
Technical Field
The present invention relates to the field of video anti-shake technology, and in particular, to a video anti-shake method, and a video anti-shake apparatus, a device and a computer readable storage medium using the method.
Background
Video shot by a handheld mobile device is difficult to avoid due to video picture jitter caused by camera shake. Some mobile devices shoot videos and have anti-shake functions, but the problem of shake cannot be thoroughly solved, and some mobile devices do not have anti-shake capability, so that anti-shake processing is needed to be performed on the videos by designing an anti-shake algorithm, and the processed videos are better, stable and smooth in playing effect.
The current video anti-shake technology is roughly divided into mechanical anti-shake, optical anti-shake, electronic anti-shake and digital anti-shake. Wherein:
mechanical anti-shake: the shooting effect is improved mainly by additionally arranging external stabilizing equipment, the anti-shake effect of mechanical anti-shake is good, but the anti-shake physical equipment is usually large, is unfavorable for carrying about, is usually applied to the professional shooting field, and has high cost.
Optical anti-shake: the principle is that a micro displacement is detected by a gyroscope in a lens, a displacement signal is transmitted to a microprocessor, and after the microprocessor calculates the displacement amount to be compensated, the camera lens is reversely moved to compensate according to the shaking direction and the displacement amount, so that anti-shaking is realized. The disadvantage is that additional equipment is required, the cost is high, the anti-shake device is only used in high-end equipment at present, the anti-shake amplitude is not large, and only small-amplitude shake can be eliminated.
Electronic anti-shake: the image on the sensor is analyzed and collected mainly through the program, and when the photo is pasted, the edge image is used for compensating the pasting part, so that the anti-shake effect is realized. This technique has a major disadvantage in that it causes a reduction in picture quality and an increase in noise. The pure electronic anti-shake can also cut out a stretched picture, and if the shake is large, the stretched picture can be deformed.
Digital anti-shake: the method mainly comprises the steps of carrying out digital image processing operation on the acquired image, and realizing anti-shake through image feature matching, motion information extraction and image conversion modes.
At present, the anti-shake method on the market generally adopts the combination of more than two of the anti-shake techniques to achieve better anti-shake effect. Under the condition that the hardware technology is basically unchanged, the improvement of the digital anti-shake algorithm is particularly important. However, the existing digital anti-shake algorithm is complex as a whole, has low operation efficiency, and cannot meet the requirement of real-time operation on low-end mobile equipment; meanwhile, the limit anti-shake capability is not strong enough, residual shake exists after the anti-shake for a severely-shake video, and if the video cutting proportion is increased, the content is easy to be lost, so that the precision requirement and the efficiency requirement of the anti-shake effect cannot be considered.
Disclosure of Invention
The invention mainly aims to provide a video anti-shake method, a device, equipment and a storage medium, which can further obtain a higher-precision anti-shake effect on the basis of guaranteeing anti-shake efficiency.
In order to achieve the above object, the present invention provides a video anti-shake method, which includes the following steps:
downsampling is carried out on target image frames and reference image frames of the video to obtain corresponding target thumbnail and reference thumbnail;
performing multi-scale block search by using the target thumbnail and the reference thumbnail, and calculating a motion vector of each pixel point between two image frames;
calculating a homography transformation matrix of multiple degrees of freedom by using the motion vector of each pixel point;
traversing the video, and calculating a homography transformation matrix of each frame, namely a motion path;
the motion path is subjected to smoothing treatment to obtain an optimized path; performing anti-shake suppression processing based on the optimized path to calculate an optimal path;
and carrying out anti-shake processing on each image frame of the video by utilizing the optimal path.
The scheme based on the thumbnail is more beneficial to efficiently carrying out multi-scale search and multi-degree-of-freedom matrix operation, so that the whole processing scheme not only can obtain a higher-precision anti-shake effect, but also has no short plate in the aspect of operation efficiency, thereby realizing the effect of real-time operation of low-end mobile equipment. In addition, the optimal path algorithm of the invention can greatly improve the smoothness and optimize the anti-shake effect.
Preferably, gray level conversion processing is further carried out on each thumbnail to obtain a corresponding single-channel thumbnail; the gray conversion formula is as follows:
Gray＝(RED+GREEN+BLUE)/3；
and the Gray is the pixel value of the pixel point of the single-channel thumbnail, and RED, GREEN, BLUE is the pixel value of the pixel point corresponding to the three-channel thumbnail.
The invention further carries out gray level conversion on the thumbnail to obtain the single-channel thumbnail, and adopts the single-channel thumbnail to carry out subsequent operation, thereby further reducing the calculated amount and improving the operation efficiency.
Preferably, the multi-scale block search includes the steps of:
dividing the target thumbnail and the reference thumbnail to obtain an image block;
calculating motion vectors between the image blocks;
further dividing the image block to obtain sub-blocks;
calculating motion vectors between the sub-blocks;
and so on until a motion vector is obtained for a block of pixels that is 1 in both width and height.
The invention uses a multi-scale mode to search blocks, calculates the motion vector of each pixel from the image block to the sub-block and then to the pixel level from the thickness to the fineness, can detect a large-scale jitter offset, can improve the detection precision, and can save the calculation time.
Preferably, the searching is further performed by adopting a central diffusion searching strategy: searching within a first preset grid range to obtain a position corresponding to a minimum mean square error value within the first preset grid range as a first search center; searching in a second preset grid range by taking the first search center as a reference to obtain a position corresponding to a minimum mean square error value in the second preset grid range as a second search center; and traversing the total preset search range of the thumbnail by analogy to obtain a final target image block, a target sub-block and a target pixel block.
Preferably, the calculating of the image block includes the following steps:
calculating a mean square error value of an image block in the target thumbnail corresponding to an image block in the reference thumbnail, and taking the image block in the target thumbnail corresponding to a designated image block in the reference thumbnail as a first search center;
searching an image block with the minimum mean square error value in a first preset grid range in the target thumbnail by taking the first search center as a reference, and taking the image block as a second search center;
searching an image block with the minimum mean square error value in a second preset grid range in the target thumbnail by taking the second search center as a reference, and the like until the image block with the minimum mean square error value in the target thumbnail is searched out, wherein the image block is used as a target image block;
calculating a motion vector between the target image block and the designated image block;
similarly, a motion vector for a target sub-block is calculated within the range of the target image block, and a motion vector for the target pixel block is calculated within the range of the target sub-block.
The block searching strategy of the invention adopts a central diffusion strategy, thereby avoiding all units in the calculation grid and greatly reducing the calculation amount.
Preferably, the calculation formula of the motion vector is:
Motion＝(BlockM[x]–Block[x]，BlockM[y]–Block[y])；
wherein, block represents the appointed image Block in the reference thumbnail, block [ x ], block [ y ] represent the coordinate value of the appointed image Block; the blockM represents a target image block in the target thumbnail, and the blockM [ x ] and the blockM [ y ] represent coordinate values of the target image block; motion represents a Motion vector between the target image block relative to the specified image block.
Preferably, the step of calculating the homography transformation matrix includes:
eliminating outlier motion vectors from motion vectors of all pixel points by using a RANSAC algorithm to obtain a target motion vector;
and iteratively calculating a homography matrix of 3*3 by using the target motion vector and using a least square method.
Traversing the video, and calculating a homography matrix of each frame to obtain a motion path.
The 3x3 matrix in the invention is a homography matrix, which is also called a projection matrix, and the homography matrix contains 8 degrees of freedom in 9 variables, so that the anti-shake effect can be greatly improved, the anti-shake precision is improved, and the water ripple phenomenon is avoided.
Preferably, the method for calculating the optimal path includes:
performing a first round of Gaussian blur smoothing processing on the motion path by using a one-dimensional Gaussian blur algorithm to obtain an optimized path;
calculating a cutting proportion, and comparing and judging the cutting proportion with a preset threshold value:
if the cutting ratio is greater than the preset threshold, performing anti-shake suppression processing on the path points of which the cutting ratio exceeds the preset threshold: carrying out Gaussian filtering treatment on the anti-shake matrix of the path points exceeding a preset threshold value; the anti-shake matrix is equal to the homography matrix of the optimized path at the current node minus the homography matrix of the motion path at the current node;
if the cutting proportion of all the path points is smaller than a preset threshold value, continuing the Gaussian blur smoothing processing and the anti-shake suppression processing of the next round; and (3) circulating in this way, and obtaining an optimal path after meeting the convergence condition.
The optimal path algorithm adopts a plurality of simple algorithm combinations such as a Gaussian blur algorithm, a threshold algorithm, an anti-shake algorithm and the like, and is simple and efficient; and the whole path is globally processed and smoothed, so that the anti-shake degree can be maximized within a limited cutting size, and the best anti-shake effect is achieved on the premise of retaining more video information.
Preferably, the anti-shake processing is performed on each image frame of the video by using the optimal path, including the following steps:
calculating an anti-shake matrix corresponding to the target image frame by using the optimal path and the motion path; the anti-shake matrix is equal to the homography matrix of the optimal path at the current node minus the homography matrix of the motion path at the current node;
calculating corresponding offset coordinates of original coordinates of each pixel point of each image frame of the video by using the anti-shake matrix;
and calculating the pixel value of the current pixel after the anti-shake by using a bilinear interpolation formula based on the offset coordinate to obtain the image frame and the video after the anti-shake.
Corresponding to the video anti-shake method, the present invention provides a video anti-shake apparatus, comprising:
the scaling processing module is used for carrying out downsampling processing on the target image frame and the reference image frame of the video to obtain a corresponding target thumbnail and a corresponding reference thumbnail;
the vector processing module is used for carrying out multi-scale block search by utilizing the target thumbnail and the reference thumbnail, and calculating the motion vector of each pixel point between two image frames;
a path calculation module for calculating a homography transformation matrix of multiple degrees of freedom using the motion vector of each pixel point; traversing the video, and calculating a homography transformation matrix of each frame, namely a motion path; the motion path is subjected to smoothing treatment to obtain an optimized path; performing anti-shake suppression processing based on the optimized path to calculate an optimal path;
and the anti-shake processing module is used for carrying out anti-shake processing on each image frame of the video by utilizing the optimal path.
In addition, the invention also provides electronic equipment, which comprises image pickup equipment, wherein the image pickup equipment comprises the video anti-shake device.
The invention also provides a computer device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps of any video anti-shake method when executing the computer program.
Furthermore, the present invention provides a computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the steps of the video anti-shake method according to any one of the above.
The beneficial effects of the invention are as follows:
(1) According to the method, the motion vector is calculated through the thumbnail converted from the gray level, so that the operation time is greatly reduced;
(2) The method and the device can quickly and stably calculate the inter-frame motion trail of the strong-shake video by using a multi-scale block search technology;
(3) The invention calculates the motion vector of each pixel from coarse to fine, not only can detect the jitter offset in a large range, but also can improve the detection precision and save the calculation time
(4) The invention uses the homography matrix with multiple degrees of freedom calculated by the motion vector as the transformation matrix, thus improving the final anti-shake precision, avoiding the generation of water ripple phenomenon and effectively solving the water ripple phenomenon;
(5) The invention obtains an optimized path by carrying out smoothing treatment on the motion path; anti-shake suppression processing is carried out based on the optimized path, and under the condition of effective anti-shake, the cutting proportion is ensured to be as small as possible, so that video with richer content is obtained;
(6) The invention can better identify the shaking in a large range by adopting multi-scale search, so that the optimal path algorithm of the invention can greatly improve the smoothness and optimize the anti-shaking effect; simultaneously iterating to the calculation of the motion of each pixel point, enabling the granularity to be finer, fully utilizing the characteristics of the multi-degree-of-freedom matrix, obtaining a motion matrix with higher precision, improving the effect of eliminating small jitter, and removing the water ripple phenomenon;
(7) Compared with the existing anti-shake algorithm in the market, the method has the advantages of being capable of achieving the effects of being minimum in cutting, strongest in anti-shake, minimum in water ripple phenomenon and fastest in running, and being more suitable for low-end mobile equipment.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention discloses a video anti-shake method, which comprises the following steps:
downsampling is carried out on target image frames and reference image frames of the video to obtain corresponding target thumbnail and reference thumbnail;
performing multi-scale block search by using the target thumbnail and the reference thumbnail, and calculating a Motion vector Motion of each pixel point between two image frames;
calculating a homography transformation matrix of multiple degrees of freedom by using the Motion vector Motion of each pixel point;
traversing the video, and calculating a homography transformation matrix of each frame, namely a motion path;
the motion path is subjected to smoothing treatment to obtain an optimized path; performing anti-shake suppression processing based on the optimized path to calculate an optimal path;
and carrying out anti-shake processing on each image frame of the video by utilizing the optimal path.
The scheme based on the thumbnail is more beneficial to efficiently carrying out multi-scale search and multi-degree-of-freedom matrix operation, so that the whole processing scheme not only can obtain a higher-precision anti-shake effect, but also has no short plate in the aspect of operation efficiency, thereby realizing the effect of real-time operation of low-end mobile equipment. In addition, the optimal path algorithm of the invention can greatly improve the smoothness and optimize the anti-shake effect.
In this embodiment, gray level conversion processing is further performed on each thumbnail to obtain a corresponding single-channel thumbnail; the gray conversion formula is as follows:
Gray＝(RED+GREEN+BLUE)/3；
and the Gray is the pixel value of the pixel point of the single-channel thumbnail, and RED, GREEN, BLUE is the pixel value of the pixel point corresponding to the three-channel thumbnail.
The invention further carries out gray level conversion on the thumbnail to obtain the single-channel thumbnail, and adopts the single-channel thumbnail to carry out subsequent operation, thereby further reducing the calculated amount and improving the operation efficiency.
In this embodiment, the multi-scale block search includes the following steps:
dividing the target thumbnail and the reference thumbnail to obtain an image block;
calculating motion vectors between the image blocks;
further dividing the image block to obtain sub-blocks;
calculating motion vectors between the sub-blocks;
and so on until a motion vector is obtained for a block of pixels that is 1 in both width and height.
The invention uses a multi-scale mode to search blocks, calculates the motion vector of each pixel from the image block to the sub-block and then to the pixel level from the thickness to the fineness, can detect a large-scale jitter offset, can improve the detection precision, and can save the calculation time.
In this embodiment, a search policy of central diffusion is further adopted to perform a search: searching within a first preset grid range to obtain a position corresponding to a minimum mean square error value within the first preset grid range as a first search center; searching in a second preset grid range by taking the first search center as a reference to obtain a position corresponding to a minimum mean square error value in the second preset grid range as a second search center; and traversing the total preset search range of the thumbnail by analogy to obtain a final target image block, a target sub-block and a target pixel block.
In this embodiment, the calculation of the image block includes the following steps:
calculating a mean square error value of an image block in the target thumbnail corresponding to an image block in the reference thumbnail, and taking the image block in the target thumbnail corresponding to a designated image block in the reference thumbnail as a first search center;
searching an image block with the minimum mean square error value in a first preset grid range in the target thumbnail by taking the first search center as a reference, and taking the image block as a second search center;
searching an image block with the minimum mean square error value in a second preset grid range in the target thumbnail by taking the second search center as a reference, and the like until the image block with the minimum mean square error value in the target thumbnail is searched out, wherein the image block is used as a target image block;
calculating a motion vector between the target image block and the designated image block;
similarly, a motion vector for a target sub-block is calculated within the range of the target image block, and a motion vector for the target pixel block is calculated within the range of the target sub-block.
In this embodiment, the width and height of the thumbnail may be set to 512 and 384, the thumbnail is divided into 16 times 12 image blocks, each image Block has a width and height of 32, each image Block searches within a grid range of 5 times 5, the searching strategy is a central diffusion strategy, that is, the mean square error value MSE of 9 image blocks within a range of 3 times 3 of the searching center is calculated first, the image Block with the minimum MSE is set as the searching center, then 9 image blocks within a range of 3 times 3 of the current searching center are calculated, the image Block with the minimum MSE is the matching Block m of the current round of searching, the central points of the Block and the Block m are linked together to form a Motion vector, and in this embodiment, the calculation formula of the Motion vector Motion is as follows:
Motion＝(BlockM[x]–Block[x]，BlockM[y]–Block[y])；
wherein, block represents the appointed image Block in the reference thumbnail, block [ x ], block [ y ] represent the coordinate value of the appointed image Block; the blockM represents a target image block in the target thumbnail, and the blockM [ x ] and the blockM [ y ] represent coordinate values of the target image block; motion represents a Motion vector between the target image block relative to the specified image block.
The MSE calculation formula is as follows:
wherein M is an average number, and the calculation formula is as follows:
after calculating the motion vector of the image block, the motion vectors of the sub-block and the pixel block are further calculated, and the embodiment specifically includes the following steps:
and splitting each image block into 4 sub-blocks, wherein the length and the width of each sub-block are 16, the Motion vector Motion of the previous layer is used as a search center, searching is performed within a grid range of 5 times 5, the mean square error value MSE is calculated from sub-blocks, the sub-block with the minimum MSE is used as a matching block, and the Motion vector Motion of the current sub-block is calculated. Similarly, the Motion vector Motion is calculated layer by layer until the width and height of the block are 1, namely 1 pixel, and finally the Motion vector Motion of each pixel is obtained.
In this embodiment, the step of calculating the homography transformation matrix includes:
eliminating outlier motion vectors from motion vectors of all pixel points by using a RANSAC algorithm to obtain a target motion vector;
and iteratively calculating a homography matrix of 3*3 by using the target motion vector and using a least square method.
Traversing the video, and calculating a homography matrix of each frame to obtain a motion path.
The 3x3 matrix in the invention is a homography matrix, which is also called a projection matrix, and the homography matrix contains 8 degrees of freedom in 9 variables, so that the anti-shake effect can be greatly improved, the anti-shake precision is improved, and the water ripple phenomenon is avoided.
The homography matrix (Homography Matrix) is a matrix in which matching points in two images are aligned with one image to the other image through rotation, transformation and the like. The aligned coordinates homogeneous coordinate of points on two images of different perspectives can be expressed in terms of a projective transformation (projective transformation), namely: x1=hx2, and H is the homography matrix.
The RANSAC algorithm, also known as a random sample consensus algorithm (random sample consensus, RANSAC), is an iterative method of estimating parameters of a mathematical model from a set of observed data containing outliers. The calculation process is as follows:
(1) taking a model with a minimum sampling set potential of n (n is the minimum sample number required for initializing model parameters) and a sampling set P, randomly extracting a subset S of P containing n samples from the P to initialize the model M, wherein the sample number # of the set P (P) > n;
(2) the sample set of the remainder sc=p\s with an error from the model M smaller than a certain set threshold t and S constitute S. S is considered as the inner point set, which constitutes a consistent set of S;
(3) if # (S) is greater than or equal to N, the correct model parameters are considered to be obtained, and a set S is utilized to recalculate a new model M by adopting a least square method and the like; and randomly extracting new S again, and repeating the process.
(4) After a certain sampling times are completed, the maximum consistency set obtained after sampling is selected to judge the inner point and the outer point, and the algorithm is ended.
In this embodiment, the method for calculating the optimal path includes:
performing first-round Gaussian blur smoothing on the motion path by using a one-dimensional Gaussian blur algorithm to obtain an optimized path OptPath; in this embodiment, the blur radius is 3;
calculating a cutting proportion, and comparing and judging the cutting proportion with a preset threshold value:
if the cutting ratio is greater than the preset threshold T, performing anti-shake suppression processing on the path points of which the cutting ratio exceeds the preset threshold: carrying out Gaussian filtering treatment on the anti-shake matrix of the path points exceeding a preset threshold value; the anti-shake matrix is equal to the Homography of the optimized path at the current node minus the Homography of the motion path at the current node;
if the cutting proportion of all the path points is smaller than a preset threshold value, continuing the Gaussian blur smoothing processing and the anti-shake suppression processing of the next round; and (3) circulating in this way, and obtaining an optimal path after meeting the convergence condition.
The convergence conditions adopted in this embodiment are:
if the cutting proportion of all the path points is smaller than the threshold value T, continuing the Gaussian blur smoothing of the next round, and then carrying out anti-shake suppression. From the second round, each round of optimization calculates the error from the previous round, and when the average error is less than 0.0001, the convergence condition is considered to be reached, and the optimal path BestPath is obtained. The threshold T is a parameter set by a user, and the smaller the T is, the smaller the cutting proportion of the anti-shake result is, and the larger the reserved information amount is.
The optimal path algorithm adopts a plurality of simple algorithm combinations such as a Gaussian blur algorithm, a threshold algorithm, an anti-shake algorithm and the like, and is simple and efficient; and the whole path is globally processed and smoothed, so that the anti-shake degree can be maximized within a limited cutting size, and the best anti-shake effect is achieved on the premise of retaining more video information.
In this embodiment, the anti-shake processing is performed on each image frame of the video by using the optimal path, including the following steps:
calculating an anti-shake Matrix corresponding to the target image frame by using the optimal Path BestPath and the motion Path Path; the anti-shake matrix is equal to the homography matrix of the optimal path at the current node minus the homography matrix of the motion path at the current node;
calculating corresponding offset coordinates of original coordinates of each pixel point of each image frame of the video by using the anti-shake matrix;
and calculating the pixel value of the current pixel after the anti-shake by using a bilinear interpolation formula based on the offset coordinate to obtain the image frame and the video after the anti-shake.
The calculation formula of the offset coordinates is as follows:
x’＝(x*Matrix[0]+y*Matrix[1]+Matrix[2])/(x*Matrix[6]+y*Matrix[7]+Matrix[8])
y’＝(x*Matrix[3]+y*Matrix[4]+Matrix[5])/(x*Matrix[6]+y*Matrix[7]+Matrix[8])
wherein x ', y' are offset coordinates of the pixel point, x and y are original coordinates of the pixel point, matrix represents an anti-shake Matrix, matrix [0], matrix [1], matrix [2], matrix [3], matrix [4], matrix [5], matrix [6], matrix [7], matrix [8] respectively represent values of 9 different degrees of freedom of the homography transformation Matrix, such as translation, rotation, scaling and the like.
Corresponding to the video anti-shake method, the present invention provides a video anti-shake apparatus, comprising:
the scaling processing module is used for carrying out downsampling processing on the target image frame and the reference image frame of the video to obtain a corresponding target thumbnail and a corresponding reference thumbnail;
the vector processing module is used for carrying out multi-scale block search by utilizing the target thumbnail and the reference thumbnail, and calculating the motion vector of each pixel point between two image frames;
a path calculation module for calculating a homography transformation matrix of multiple degrees of freedom using the motion vector of each pixel point; traversing the video, and calculating a homography transformation matrix of each frame, namely a motion path; the motion path is subjected to smoothing treatment to obtain an optimized path; performing anti-shake suppression processing based on the optimized path to calculate an optimal path;
and the anti-shake processing module is used for carrying out anti-shake processing on each image frame of the video by utilizing the optimal path.
In addition, the invention also provides electronic equipment, which comprises image pickup equipment, wherein the image pickup equipment comprises the video anti-shake device.
The invention also provides a computer device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps of any video anti-shake method when executing the computer program.
The apparatus comprises: a device with a photographing function such as a mobile phone, a digital camera or a tablet computer, or a device with a video acquisition function or a video importing function. The device may include a memory, a processor, an input unit, a display unit, a power source, and the like.
The memory may be used to store software programs and modules, and the processor executes the software programs and modules stored in the memory to perform various functional applications and data processing. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (e.g., an image playing function, etc.) required for at least one function, and the like; the storage data area may store data created according to the use of the device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide access to the memory by the processor and the input unit.
The input unit may be used to receive input digital or character or image information and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. Specifically, the input unit of the present embodiment may include a touch-sensitive surface (e.g., a touch display screen) and other input devices in addition to the camera.
The display unit may be used to display information entered by a user or provided to a user as well as various graphical user interfaces of the device, which may be composed of graphics, text, icons, video and any combination thereof. The display unit may include a display panel, and alternatively, the display panel may be configured in the form of an LCD (Liquid Crystal Display ), an OLED (Organic Light-Emitting Diode), or the like. Further, the touch-sensitive surface may overlay the display panel, and upon detection of a touch operation thereon or thereabout, the touch-sensitive surface is communicated to the processor to determine the type of touch event, and the processor then provides a corresponding visual output on the display panel based on the type of touch event.
Furthermore, the present invention provides a computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the steps of the video anti-shake method according to any one of the above. The computer-readable storage medium may be a computer-readable storage medium contained in the memory in the above-described embodiment; or may be a computer-readable storage medium, alone, that is not assembled into a device. The computer readable storage medium has stored therein at least one instruction that is loaded by a processor and that performs the video anti-shake method. The computer readable storage medium may be a read-only memory, a magnetic disk or optical disk, etc.
It should be noted that, in the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described as different from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other. For the device embodiments, the apparatus embodiments and the storage medium embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.
Also, herein, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
While the foregoing description illustrates and describes the preferred embodiments of the present invention, it is to be understood that the invention is not limited to the forms disclosed herein, but is not to be construed as limited to other embodiments, but is capable of use in various other combinations, modifications and environments and is capable of changes or modifications within the scope of the inventive concept, either as described above or as a matter of skill or knowledge in the relevant art. And that modifications and variations which do not depart from the spirit and scope of the invention are intended to be within the scope of the appended claims.
Claims (12)
1. The video anti-shake method is characterized by comprising the following steps of:
downsampling is carried out on target image frames and reference image frames of the video to obtain corresponding target thumbnail and reference thumbnail;
performing multi-scale block search by using the target thumbnail and the reference thumbnail, and calculating a motion vector of each pixel point between two image frames;
calculating a homography transformation matrix of multiple degrees of freedom by using the motion vector of each pixel point;
traversing the video, and calculating a homography transformation matrix of each frame, namely a motion path;
the motion path is subjected to smoothing treatment to obtain an optimized path; performing anti-shake suppression processing based on the optimized path to calculate an optimal path;
performing anti-shake processing on each image frame of the video by utilizing the optimal path;
the method for calculating the optimal path comprises the following steps:
performing a first round of Gaussian blur smoothing processing on the motion path by using a one-dimensional Gaussian blur algorithm to obtain an optimized path;
calculating a cutting proportion, and comparing and judging the cutting proportion with a preset threshold value:
if the cutting ratio is greater than the preset threshold, performing anti-shake suppression processing on the path points of which the cutting ratio exceeds the preset threshold: carrying out Gaussian filtering treatment on the anti-shake matrix of the path points exceeding a preset threshold value; the anti-shake matrix is equal to the homography matrix of the optimized path at the current node minus the homography matrix of the motion path at the current node;
if the cutting proportion of all the path points is smaller than a preset threshold value, continuing the Gaussian blur smoothing processing and the anti-shake suppression processing of the next round; and (3) circulating in this way, and obtaining an optimal path after meeting the convergence condition.
2. The video anti-shake method according to claim 1, further comprising performing gray level conversion processing on each thumbnail to obtain a corresponding single-channel thumbnail; the gray conversion formula is as follows:
Gray = （RED + GREEN + BLUE）/ 3；
and the Gray is the pixel value of the pixel point of the single-channel thumbnail, and RED, GREEN, BLUE is the pixel value of the pixel point corresponding to the three-channel thumbnail.
3. The video anti-shake method according to claim 1, wherein the multi-scale block search comprises the steps of:
dividing the target thumbnail and the reference thumbnail to obtain an image block;
calculating motion vectors between the image blocks;
further dividing the image block to obtain sub-blocks;
calculating motion vectors between the sub-blocks;
and so on until a motion vector is obtained for a block of pixels that is 1 in both width and height.
4. A video anti-shake method according to claim 3, further comprising searching using a central-diffused search strategy: searching within a first preset grid range to obtain a position corresponding to a minimum mean square error value within the first preset grid range as a first search center; searching in a second preset grid range by taking the first search center as a reference to obtain a position corresponding to a minimum mean square error value in the second preset grid range as a second search center; and traversing the total preset search range of the thumbnail by analogy to obtain a final target image block, a target sub-block and a target pixel block.
5. The video anti-shake method according to claim 4, wherein the calculating of the image block includes the steps of:
calculating a mean square error value of an image block in the target thumbnail corresponding to an image block in the reference thumbnail, and taking the image block in the target thumbnail corresponding to a designated image block in the reference thumbnail as a first search center;
searching an image block with the minimum mean square error value in a first preset grid range in the target thumbnail by taking the first search center as a reference, and taking the image block as a second search center;
searching an image block with the minimum mean square error value in a second preset grid range in the target thumbnail by taking the second search center as a reference, and the like until the image block with the minimum mean square error value in the target thumbnail is searched out, wherein the image block is used as a target image block;
calculating a motion vector between the target image block and the designated image block;
similarly, a motion vector for a target sub-block is calculated within the range of the target image block, and a motion vector for the target pixel block is calculated within the range of the target sub-block.
6. The video anti-shake method according to claim 5, wherein the motion vector has a calculation formula:
Motion = （BlockM[x] – Block[x]， BlockM[y] – Block[y]）；
wherein, block represents the appointed image Block in the reference thumbnail, block [ x ], block [ y ] represent the coordinate value of the appointed image Block; the blockM represents a target image block in the target thumbnail, and the blockM [ x ] and the blockM [ y ] represent coordinate values of the target image block; motion represents a Motion vector between the target image block relative to the specified image block.
7. The video anti-shake method according to claim 1, wherein the step of calculating the homography transformation matrix includes:
eliminating outlier motion vectors from motion vectors of all pixel points by using a RANSAC algorithm to obtain a target motion vector;
iteratively calculating a homography matrix of 3*3 by using the target motion vector and a least square method;
traversing the video, and calculating a homography matrix of each frame to obtain a motion path.
8. The video anti-shake method according to claim 1, wherein the anti-shake processing is performed on each image frame of the video by using the optimal path, comprising the steps of:
calculating an anti-shake matrix corresponding to the target image frame by using the optimal path and the motion path; the anti-shake matrix is equal to the homography matrix of the optimal path at the current node minus the homography matrix of the motion path at the current node;
calculating corresponding offset coordinates of original coordinates of each pixel point of each image frame of the video by using the anti-shake matrix;
and calculating the pixel value of the current pixel after the anti-shake by using a bilinear interpolation formula based on the offset coordinate to obtain the image frame and the video after the anti-shake.
9. A video anti-shake apparatus, comprising:
the scaling processing module is used for carrying out downsampling processing on the target image frame and the reference image frame of the video to obtain a corresponding target thumbnail and a corresponding reference thumbnail;
the vector processing module is used for carrying out multi-scale block search by utilizing the target thumbnail and the reference thumbnail, and calculating the motion vector of each pixel point between two image frames;
a path calculation module for calculating a homography transformation matrix of multiple degrees of freedom using the motion vector of each pixel point; traversing the video, and calculating a homography transformation matrix of each frame, namely a motion path; the motion path is subjected to smoothing treatment to obtain an optimized path; performing anti-shake suppression processing based on the optimized path to calculate an optimal path;
the anti-shake processing module is used for carrying out anti-shake processing on each image frame of the video by utilizing the optimal path;
the method for calculating the optimal path comprises the following steps:
performing a first round of Gaussian blur smoothing processing on the motion path by using a one-dimensional Gaussian blur algorithm to obtain an optimized path;
calculating a cutting proportion, and comparing and judging the cutting proportion with a preset threshold value:
if the cutting ratio is greater than the preset threshold, performing anti-shake suppression processing on the path points of which the cutting ratio exceeds the preset threshold: carrying out Gaussian filtering treatment on the anti-shake matrix of the path points exceeding a preset threshold value; the anti-shake matrix is equal to the homography matrix of the optimized path at the current node minus the homography matrix of the motion path at the current node;
if the cutting proportion of all the path points is smaller than a preset threshold value, continuing the Gaussian blur smoothing processing and the anti-shake suppression processing of the next round; and (3) circulating in this way, and obtaining an optimal path after meeting the convergence condition.
10. An electronic device, characterized in that the electronic device comprises an image pickup apparatus including the video anti-shake apparatus according to claim 9.
11. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the video anti-shake method of any one of claims 1 to 8 when the computer program is executed.
12. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor implements the steps of the video anti-shake method according to any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111429727.8A CN114095659B (en) | 2021-11-29 | 2021-11-29 | Video anti-shake method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111429727.8A CN114095659B (en) | 2021-11-29 | 2021-11-29 | Video anti-shake method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114095659A CN114095659A (en) | 2022-02-25 |
| CN114095659B true CN114095659B (en) | 2024-01-23 |
Family
ID=80305320
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111429727.8A Active CN114095659B (en) | 2021-11-29 | 2021-11-29 | Video anti-shake method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114095659B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115379115A (en) * | 2022-07-29 | 2022-11-22 | 黑芝麻智能科技(成都)有限公司 | Video shooting method and device and electronic equipment |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010093343A (en) * | 2008-10-03 | 2010-04-22 | Nikon Corp | Camerawork optimization program, imaging apparatus, and camerawork optimization method |
| CN102348046A (en) * | 2010-07-30 | 2012-02-08 | 富士通株式会社 | Video shake removing method and video shake removing device |
| CN102474568A (en) * | 2009-08-12 | 2012-05-23 | 英特尔公司 | Techniques to perform video stabilization and detect video shot boundaries based on common processing elements |
| JP2015084186A (en) * | 2013-10-25 | 2015-04-30 | キヤノン株式会社 | Information processing device, control method therefor, and program |
| CN105791705A (en) * | 2016-05-26 | 2016-07-20 | 厦门美图之家科技有限公司 | Video anti-shake method and system suitable for movable time-lapse photography and shooting terminal |
| CN110047091A (en) * | 2019-03-14 | 2019-07-23 | 河海大学 | One kind is based on the estimation of camera track and the matched digital image stabilization method of characteristic block |
| CN110753181A (en) * | 2019-09-29 | 2020-02-04 | 湖北工业大学 | Video image stabilization method based on feature tracking and grid path motion |
| KR102173244B1 (en) * | 2019-12-10 | 2020-11-03 | (주)인펙비전 | Video stabilization system based on SURF |
| CN113132560A (en) * | 2019-12-31 | 2021-07-16 | 武汉Tcl集团工业研究院有限公司 | Video processing method, computer equipment and computer readable storage medium |
| WO2021218694A1 (en) * | 2020-04-27 | 2021-11-04 | 华为技术有限公司 | Video processing method and mobile terminal |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9374532B2 (en) * | 2013-03-15 | 2016-06-21 | Google Inc. | Cascaded camera motion estimation, rolling shutter detection, and camera shake detection for video stabilization |
-
2021
- 2021-11-29 CN CN202111429727.8A patent/CN114095659B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010093343A (en) * | 2008-10-03 | 2010-04-22 | Nikon Corp | Camerawork optimization program, imaging apparatus, and camerawork optimization method |
| CN102474568A (en) * | 2009-08-12 | 2012-05-23 | 英特尔公司 | Techniques to perform video stabilization and detect video shot boundaries based on common processing elements |
| CN102348046A (en) * | 2010-07-30 | 2012-02-08 | 富士通株式会社 | Video shake removing method and video shake removing device |
| JP2015084186A (en) * | 2013-10-25 | 2015-04-30 | キヤノン株式会社 | Information processing device, control method therefor, and program |
| CN105791705A (en) * | 2016-05-26 | 2016-07-20 | 厦门美图之家科技有限公司 | Video anti-shake method and system suitable for movable time-lapse photography and shooting terminal |
| CN110047091A (en) * | 2019-03-14 | 2019-07-23 | 河海大学 | One kind is based on the estimation of camera track and the matched digital image stabilization method of characteristic block |
| CN110753181A (en) * | 2019-09-29 | 2020-02-04 | 湖北工业大学 | Video image stabilization method based on feature tracking and grid path motion |
| KR102173244B1 (en) * | 2019-12-10 | 2020-11-03 | (주)인펙비전 | Video stabilization system based on SURF |
| CN113132560A (en) * | 2019-12-31 | 2021-07-16 | 武汉Tcl集团工业研究院有限公司 | Video processing method, computer equipment and computer readable storage medium |
| WO2021218694A1 (en) * | 2020-04-27 | 2021-11-04 | 华为技术有限公司 | Video processing method and mobile terminal |
Non-Patent Citations (1)
| Title |
|---|
| 基于Android平台的数字视频稳像算法;郭亨;《电子科技大学硕士论文》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114095659A (en) | 2022-02-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI381719B (en) | Full-frame video stabilization with a polyline-fitted camcorder path | |
| US10887519B2 (en) | Method, system and apparatus for stabilising frames of a captured video sequence | |
| JP6658791B2 (en) | Method, apparatus, device, and computer-readable storage medium for processing panoramic video | |
| Hu et al. | Joint depth estimation and camera shake removal from single blurry image | |
| Guilluy et al. | Video stabilization: Overview, challenges and perspectives | |
| CN106878612B (en) | A kind of video stabilizing method based on the optimization of online total variation | |
| US10121262B2 (en) | Method, system and apparatus for determining alignment data | |
| EP2898473A1 (en) | Systems and methods for reducing noise in video streams | |
| JP7253622B2 (en) | Image stabilization method for panorama video and portable terminal | |
| EP2733923A2 (en) | Multiresolution depth from defocus based autofocus | |
| US20200160560A1 (en) | Method, system and apparatus for stabilising frames of a captured video sequence | |
| Xu et al. | Panoramic video stitching from commodity HDTV cameras | |
| WO2010151215A1 (en) | Real time video stabilization | |
| JP7253621B2 (en) | Image stabilization method for panorama video and portable terminal | |
| CN114095659B (en) | Video anti-shake method, device, equipment and storage medium | |
| WO2021139764A1 (en) | Method and device for image processing, electronic device, and storage medium | |
| Ma et al. | Effective video stabilization via joint trajectory smoothing and frame warping | |
| WO2023241495A1 (en) | Photographic method and apparatus | |
| Wu et al. | Locally low-rank regularized video stabilization with motion diversity constraints | |
| Zhen et al. | Multi-image motion deblurring aided by inertial sensors | |
| CN111199514B (en) | Image background blurring method, device and equipment and readable storage medium | |
| CN113873160B (en) | Image processing method, device, electronic equipment and computer storage medium | |
| CN117541507A (en) | Image data pair establishing method and device, electronic equipment and readable storage medium | |
| CN114612837A (en) | Video processing method and device and video stabilizing method | |
| Liu et al. | Real-time Video Stabilization Algorithm based on SuperPoint |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |