CN115209030B - Video anti-shake processing method and device, electronic equipment and storage medium - Google Patents
Video anti-shake processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115209030B CN115209030B CN202110379627.2A CN202110379627A CN115209030B CN 115209030 B CN115209030 B CN 115209030B CN 202110379627 A CN202110379627 A CN 202110379627A CN 115209030 B CN115209030 B CN 115209030B
- Authority
- CN
- China
- Prior art keywords
- video
- frame
- smoothing
- image frames
- track
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000009499 grossing Methods 0.000 claims abstract description 95
- 238000012545 processing Methods 0.000 claims abstract description 80
- 238000000034 method Methods 0.000 claims abstract description 55
- 230000008569 process Effects 0.000 claims abstract description 29
- 230000009466 transformation Effects 0.000 claims description 35
- 239000011159 matrix material Substances 0.000 claims description 30
- 238000004590 computer program Methods 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 9
- 230000000694 effects Effects 0.000 abstract description 12
- 208000012788 shakes Diseases 0.000 description 56
- 238000010586 diagram Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 210000000697 sensory organ Anatomy 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2622—Signal amplitude transition in the zone between image portions, e.g. soft edges
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30241—Trajectory
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Studio Devices (AREA)
Abstract
The embodiment of the disclosure relates to a video anti-shake processing method, a device, electronic equipment and a storage medium, wherein the method comprises the following steps: the method comprises the steps of tracking characteristic points among different image frames in a video, and determining a moving track of a shooting position of the video; respectively carrying out smoothing treatment on shooting positions of different image frames in a moving track based on smoothing radiuses corresponding to different image frames in a video to obtain a smoothing track; the method comprises the steps that the smooth radius corresponding to different image frames in a video is determined based on the number of feature points used for tracking of the different image frames in the video; and deforming the video based on the difference between the smooth track and the moving track to obtain the video subjected to the anti-shake processing. According to the embodiment of the disclosure, in the video anti-shake processing process, the effect of adaptively adjusting the smooth radius corresponding to the image frames according to the number of the characteristic points on the image frames in the video to be processed is realized, the processing effect of video anti-shake is optimized, and the video quality is effectively improved.
Description
Technical Field
The disclosure relates to the technical field of video processing, and in particular relates to a video anti-shake processing method, a video anti-shake processing device, electronic equipment and a storage medium.
Background
With the advent of short video, video capture has become increasingly important. Currently, users generally use handheld devices to shoot video, which easily causes video jitter, resulting in poor video quality. Therefore, how to perform anti-shake processing on video and improve video quality is still a current problem to be solved.
Disclosure of Invention
In order to solve the technical problems described above or at least partially solve the technical problems described above, embodiments of the present disclosure provide a video anti-shake processing method, apparatus, electronic device, and storage medium.
In a first aspect, an embodiment of the present disclosure provides a video anti-shake processing method, including:
determining a moving track of a shooting position of a video by tracking characteristic points among different image frames in the video, wherein the moving track is used for indicating the shooting position of the different image frames in the video;
respectively carrying out smoothing treatment on shooting positions of different image frames in the moving track based on smoothing radiuses corresponding to the different image frames in the video to obtain a smoothing track; the method comprises the steps that the smooth radius corresponding to different image frames in a video is determined based on the number of feature points used for tracking of the different image frames in the video;
and deforming the video based on the difference between the smooth track and the moving track to obtain the video subjected to anti-shake processing.
In a second aspect, an embodiment of the present disclosure further provides a video anti-shake processing apparatus, including:
the mobile track determining module is used for determining a mobile track of a shooting position of the video by tracking characteristic points among different image frames in the video, wherein the mobile track is used for indicating the shooting position of the different image frames in the video;
the smooth track determining module is used for respectively carrying out smooth processing on shooting positions of different image frames in the moving track based on smooth radiuses corresponding to the different image frames in the video to obtain a smooth track; the method comprises the steps that the smooth radius corresponding to different image frames in a video is determined based on the number of feature points used for tracking of the different image frames in the video;
and the video anti-shake processing module is used for deforming the video based on the difference between the smooth track and the moving track so as to obtain the video subjected to anti-shake processing.
In a third aspect, an embodiment of the present disclosure further provides an electronic device, including a memory and a processor, where the memory stores a computer program, and when the computer program is executed by the processor, the electronic device is caused to implement any one of the video anti-shake processing methods provided in the embodiments of the present disclosure.
In a fourth aspect, the embodiments of the present disclosure further provide a computer readable storage medium, where a computer program is stored, where the computer program, when executed by a computing device, causes the computing device to implement any one of the video anti-shake processing methods provided by the embodiments of the present disclosure.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has at least the following advantages: in the embodiment of the disclosure, based on the number of the feature points used for tracking on different image frames in the video to be processed, the smooth radius corresponding to the different image frames in the video is dynamically determined, that is, the embodiment of the disclosure achieves the effect of adaptively adjusting the smooth radius corresponding to the different image frames in the video to be processed based on the number of the feature points on the image frames, then carries out smooth processing on the shooting positions of the different image frames in the moving track of the shooting position of the video based on the smooth radius corresponding to the different image frames in the video, and finally carries out deformation on the video to be processed based on the difference between the smooth track and the moving track, thereby achieving anti-shake processing of the video and optimizing the anti-shake processing effect of the video. Compared with the unified value of the smooth radius, the embodiment of the disclosure effectively avoids the phenomena of shaking and abnormal deformation of a video picture caused by inaccurate motion estimation of a shooting device for shooting video, namely inaccurate movement trend estimation of a video shooting position when the number of feature points for tracking on an image frame is small (or the number of feature points for feature matching is small, namely insufficient), and effectively improves the video quality.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments of the present disclosure or the solutions in the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
Fig. 1 is a flowchart of a video anti-shake processing method according to an embodiment of the disclosure;
FIG. 2 is a flowchart of another video anti-shake processing method according to an embodiment of the disclosure;
fig. 3 is a schematic structural diagram of a video anti-shake processing apparatus according to an embodiment of the disclosure;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, a further description of aspects of the present disclosure will be provided below. It should be noted that, without conflict, the embodiments of the present disclosure and features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced otherwise than as described herein; it will be apparent that the embodiments in the specification are only some, but not all, embodiments of the disclosure.
Fig. 1 is a flowchart of a video anti-shake processing method according to an embodiment of the present disclosure, which may be suitable for a case of performing anti-shake processing on a video. The method can be performed by a video anti-shake processing apparatus, which can be implemented in software and/or hardware, and can be integrated on any electronic device with computing power, such as a terminal or a server.
In the embodiment of the disclosure, the video to be processed may be a video in shooting or a video after shooting, that is, the embodiment of the disclosure may perform anti-shake processing on the shot video in real time in the video shooting process, or may perform anti-shake processing on the video after the video shooting is completed, so as to achieve the effect of improving the video quality.
As shown in fig. 1, the video anti-shake processing method provided by the embodiment of the disclosure may include:
s101, tracking characteristic points among different image frames in the video, and determining a moving track of a shooting position of the video, wherein the moving track is used for indicating the shooting position of the different image frames in the video.
After the video to be processed is acquired, feature point tracking (for example, feature point extraction and tracking on each frame of image are sequentially performed from the first frame of image in the video) can be performed between different image frames in the video by using any available feature point extraction and tracking technology, matching feature points (which refer to feature points for the same shooting object in different image frames, the number of the matching feature points can be determined according to the situation) between different image frames are determined, and then shooting positions or relative variation of the shooting positions of different image frames in the video are calculated based on the matching feature points, so as to obtain a moving track of the shooting position of the video (or a moving track of a shooting device for shooting the video). And determining the moving track of the video shooting position, namely determining the jitter trend of the video shooting position.
In an alternative implementation mode, the moving track is represented by a transformation matrix, and different transformation matrixes in the moving track respectively represent shooting positions of different image frames in the video; accordingly, by tracking feature points among different image frames in the video, determining a moving track of a shooting position of the video comprises the following steps: and tracking characteristic points among different image frames in the video, determining a transformation matrix of shooting positions among the different image frames in the video, and determining a moving track of the shooting positions of the video based on the transformation matrix of the shooting positions among the different image frames in the video.
The transformation matrix may include a homography (homography) matrix, an affine transformation (affine transformation) matrix, or a similarity transformation (similarity transformation) matrix, among others. The degrees of freedom corresponding to different transformation matrixes are different, the motion fitting capacity corresponding to the transformation matrix with higher degrees of freedom is stronger, and the type of the transformation matrix can be flexibly selected according to the requirements in the actual processing process. For a specific calculation of the transformation matrix reference is made to prior art implementations.
In the process of determining the moving track based on a plurality of transformation matrixes, a frame of image can be selected in the video to serve as a reference frame (specifically, the moving track can be flexibly determined), then the transformation matrixes of the shooting positions of each frame of image relative to the reference frame are respectively determined based on the transformation matrixes of the shooting positions of different image frames, and finally the required moving track is obtained based on the transformation matrixes of the shooting positions of each frame of image relative to the reference frame.
Taking a reference frame as a first frame image in a video, different image frames in the video refer to two adjacent frame images as an example, and determining a moving track of a shooting position of the video by tracking feature points among the different image frames in the video comprises the following steps:
the method comprises the steps of tracking characteristic points of two adjacent frames of images in a video, determining matched characteristic points between the two adjacent frames of images, and calculating a transformation matrix of shooting positions between the two adjacent frames of images based on the matched characteristic points;
determining a transformation matrix of the shooting position of each frame of image in the video relative to the shooting position of the first frame of image in the video based on the transformation matrix of the shooting position between two adjacent frames of images;
and determining the moving track of the shooting position of the video based on a transformation matrix of the shooting position of each frame of image in the video relative to the first frame of image in the video.
For example, suppose that video V contains n frames of images in total, let the i-th frame of image be f i Then video v= { f 1 ，f 2 ，...，f n-1 ，f n (ii) each frame of image of the video is processed in turn:
1) For the ith frame image f i Extracting feature points, denoted as p i 。
2) In the (i+1) th frame image f i+1 Characteristic point p of last frame image i Tracking, wherein the tracked characteristic point is marked as p i～i+1 I.e. p i And p i～i+1 Respectively the ith frame image f i And the (i+1) th frame image f i+1 Matching successfully feature points;
3) According to p i And p i～i+1 Fitting the corresponding relation of the ith frame image f i To the (i+1) th frame image f i+1 Is denoted as T i ；
4) Will be the i frame image f i And the transformation matrix between each two adjacent frame images before the ith frame image is accumulated, for example, the accumulated multiplication is performed (which may be determined according to actual processing) to obtain the ith frame image f i The transformation matrix of the shooting position relative to the first frame image is expressed as follows:
a transformation matrix of the shooting position of each frame image relative to the first frame image in the video is sequentially obtained, and then the moving track of the shooting position of the video can be expressed as C= { C 1 ，C 2 ，…，C n-1 ，C n N represents the number of image frames included in the video.
S102, respectively carrying out smoothing treatment on shooting positions of different image frames in a moving track based on smoothing radiuses corresponding to the different image frames in the video to obtain a smoothing track; the smooth radius corresponding to different image frames in the video is determined based on the number of feature points used for tracking by the different image frames in the video.
The number of feature points in the video that are used for tracking by different image frames (or feature points on the subsequent frame image that match successfully with the previous frame image) is related to the subject on each frame image. Taking two adjacent frames of images as face images, a complete face area is displayed on a front frame of image, the number of feature points included is large, only half of the face area is displayed on a rear frame of image, the number of feature points included is small, and further the feature points used for tracking on the rear frame of image in the two adjacent frames of images can be only part of feature points corresponding to five sense organs in the face feature points.
In the embodiment of the disclosure, a correspondence between the number of feature points and the smoothing radius is preset, so that the smoothing radius corresponding to the image frame can be dynamically determined according to the correspondence and the number of feature points used for tracking on the image frame, and the value of the smoothing radius determines the number of image frames participating in smoothing processing. When the number of feature points used for tracking is larger (i.e. more features) on an image frame, the more accurate the motion estimation of a shooting device used for shooting video is based on the image frame, therefore, a larger smoothing radius can be set for the image frame, so that the shooting position of the image frame is smoothed based on more adjacent image frames, and larger accumulated errors are not introduced; when the number of feature points on an image frame is smaller (i.e. features are smaller), the accuracy of motion estimation of a shooting device for shooting video based on the image frame is relatively lower, so that a smaller smoothing radius can be set for the image frame, so that the shooting position of the image frame is smoothed based on fewer adjacent image frames, larger accumulated errors are avoided being introduced, and the video anti-shake processing effect is finally optimized.
Optionally, the smooth radius corresponding to the different image frames in the video is determined based on the number interval to which the number of feature points used for tracking by the different image frames in the video belongs and the correspondence between the number interval and the smooth radius. Illustratively, each frame of image f in the video i Corresponding smooth radius r i The following correspondence may be used for determination:
wherein m is 1 ＞m 2 ＞m 3 The values can be flexibly set, for example, m 1 Can take the value of 50, m 2 Can take the value of 25, m 3 Can take a value of 10; n is n i Representing an image f of each frame i R relative to the number of feature points used for tracking of the previous frame image 0 The initial value of the smooth radius is represented, and the value can be flexibly determined. In the above formula, for different feature point number intervals, the radius r is smoothed i Is a value of (1) and an initial value r 0 As an example only, r may be flexibly set according to the processing requirements in actual processing i And r 0 The value ratio relation between the two.
The corresponding relation between the number of the feature points and the value of the smooth radius is preset, so that the efficiency of determining the smooth radius corresponding to each frame of image is improved.
In the embodiment of the present disclosure, the smoothing algorithm may be any algorithm available in the art that can achieve a smoothing effect, such as a gaussian smoothing algorithm or the like. The smoothed trajectory obtained by smoothing can be expressed, for example, asThe smoothed trajectory is used to indicate the shooting position of different image frames in the smoothed video, e.g +.>The position of the image of the nth frame in the video after the smoothing process is indicated.
Optionally, based on smooth radii corresponding to different image frames in the video, smoothing processing is performed on shooting positions of different image frames in the moving track to obtain a smooth track, which includes:
determining images of a preset frame number participating in each smoothing process in the video based on the smoothing radius corresponding to each frame of image in the video;
and carrying out smoothing processing on the shooting position of each frame of image in the moving track based on the shooting position of the image of the preset frame number participating in each smoothing processing, so as to obtain a smoothed track. For example, it is possible to perform weighted summation calculation of the shooting position of the image of the preset frame number in the moving trajectory, which participates in each smoothing process, and to use the calculation result as a smoothing result of the shooting position of the current frame image.
S103, deforming the video based on the difference between the smooth track and the moving track to obtain the video subjected to the anti-shake processing.
Will smooth the trackComparing with the movement track C before smoothing to determine the adjustment parameter W= { W of the movement track 1 ，W 2 ，...，W n-1 ，W n Each sub-value in the tuning parameter W +.>n represents the number of image frames included in the video; then, according to the corresponding relation between each sub-value in the adjustment parameters and each frame of image in the video, the corresponding image frame is deformed based on each sub-value in the adjustment parametersAnd obtaining the video after the anti-shake processing. In the deformation processing, the processing of rotation, translation, scaling or clipping of the specific frame image is involved, and can be performed according to the actual processing requirements. That is, optionally, in the embodiment of the present disclosure, the deforming the video based on the difference between the smooth track and the moving track to obtain the video subjected to the anti-shake processing includes: determining an adjustment parameter based on a difference between the smoothed trajectory and the movement trajectory; and deforming the video by utilizing the adjustment parameters to obtain the video subjected to the anti-shake treatment.
In the embodiment of the disclosure, based on the number of feature points used for tracking on different image frames in a video to be processed, the smooth radius corresponding to the different image frames in the video is dynamically determined, that is, the embodiment of the disclosure achieves the effect of adaptively adjusting the smooth radius corresponding to the different image frames in the video based on the number of feature points on the image frames, then respectively carries out smooth processing on the shooting positions of the different image frames in the moving track of the shooting position of the video based on the smooth radius corresponding to the different image frames in the video, and finally carries out deformation on the video based on the difference between the smooth track and the moving track, thereby achieving anti-shake processing of the video and optimizing the processing effect of anti-shake of the video. Compared with the unified value of the smooth radius, the embodiment of the disclosure effectively avoids the phenomena of shaking and abnormal deformation of a video picture caused by inaccurate motion estimation of a shooting device for shooting video, namely inaccurate movement trend estimation of a video shooting position when the number of feature points for tracking on an image frame is small (or the number of feature points for feature matching is small, namely insufficient), and effectively improves the video quality.
Fig. 2 is a flowchart of another video anti-shake processing method according to an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution, and may be combined with the above various optional embodiments.
As shown in fig. 2, the video anti-shake processing method provided by the embodiment of the disclosure may include:
s201, tracking characteristic points among different image frames in the video, and determining a moving track of a shooting position of the video, wherein the moving track is used for indicating the shooting position of the different image frames in the video.
S202, determining images of preset frames participating in each smoothing process in the video based on the smoothing radius corresponding to each frame of image in the video.
Exemplary one, a previous frame image of a first preset frame number before each frame image can be determined in the video based on a smoothing radius corresponding to each frame image in the video;
determining each frame of image and the previous frame of image with the first preset frame number as images with the preset frame number participating in each smoothing process in the video; at this time, the value of the first preset frame number may be a value 1 of the smooth radius.
The embodiment of the disclosure supports the anti-shake processing of the video while the video is shot, and the video shooting is completed, and the anti-shake processing of the video is also completed, so that the shot video with better quality is obtained.
Exemplary two, a previous frame image of a second preset frame number before each frame image and a subsequent frame image of the second preset frame number after each frame image can be determined in the video based on the smoothing radius corresponding to each frame image in the video;
determining each frame of image, a previous frame of image with a second preset frame number and a subsequent frame of image with the second preset frame number as images with preset frame numbers which participate in each smoothing process in the video; at this time, the value of the second preset frame number may be the value of the smooth radius.
The embodiment of the disclosure also supports anti-shake processing of the shot video, and takes a certain number of frames of images before and after each frame of image to participate in the smoothing processing of the shooting position of each frame of image in the moving track, thereby achieving the effect of improving the video quality.
S203, carrying out weighted summation calculation on the shooting positions based on the shooting positions of the images with the preset frame numbers in the moving track, which participate in each smoothing process, so as to obtain the smoothing position of each frame of image.
Taking the case where the moving trajectory includes a plurality of transformation matrices, that is, the photographing position of each frame of image in the video may be represented by one transformation matrix as an example, the smoothing position of each frame of image (that is, the photographing position after the smoothing process) may be represented in the form of a matrix, which will be hereinafter referred to as a smoothing matrix. The weights in the weighted sum calculation process may be adaptively valued, and embodiments of the present disclosure are not particularly limited.
Taking a certain number of frames of images before each frame of images according to the corresponding smooth radius of each frame of images as an example, taking the smoothing processing of the shooting position of each frame of images as participation, and taking the example of each frame of images f i The smoothing matrix of (c) may be represented as follows:
taking the example of taking images with the same frame number before and after each frame image according to the corresponding smoothing radius of each frame image and participating in the smoothing processing of the shooting position of each frame image as an example, each frame image f i The smoothing matrix of (c) may be represented as follows:
in the above two examples, r is the smoothing radius corresponding to each frame of image, C t Representing the shooting position (or referred to as transformation matrix) of each frame of image participating in smoothing processing in the moving trajectory C, w i～t The value may be adaptively set for the weight of each frame image participating in the smoothing process. After obtaining the smoothing matrix corresponding to each frame of image in the video, the smoothing track can be expressed as follows:
the smooth position corresponding to each frame of image is obtained by carrying out weighted summation calculation on the shooting positions of the images with the preset frame number in the moving track, so that the calculation mode is simple and efficient, the real-time processing requirement can be met, and the smooth processing effect of the moving track of the video shooting position can be ensured.
S204, obtaining a smooth track based on the smooth position of each frame of image.
S205, deforming the video based on the difference between the smooth track and the moving track to obtain the video subjected to the anti-shake processing.
In the embodiment of the disclosure, the smooth radius corresponding to each frame of image is dynamically determined according to the number of the characteristic points for tracking each frame of image in the video to be processed in the video anti-shake processing process, and then a certain number of image frames are dynamically determined to participate in the smooth processing of the moving track of the video shooting position, so that the video anti-shake processing effect is optimized. Compared with the unified value of the smooth radius, the embodiment of the disclosure effectively avoids the phenomenon that when the number of feature points for tracking on the image frames in the video is less, the motion estimation of the shooting device for shooting the video is inaccurate, namely the motion trend estimation of the shooting position of the video is inaccurate, and further the shaking and abnormal deformation of the video picture are caused, and the video quality is effectively improved.
Fig. 3 is a schematic structural diagram of a video anti-shake processing apparatus according to an embodiment of the present disclosure, which may be suitable for the case of performing anti-shake processing on a video. The apparatus may be implemented in software and/or hardware and may be integrated on any electronic device having computing capabilities, such as a terminal or server, etc.
As shown in fig. 3, the video anti-shake processing apparatus 300 provided by the embodiment of the present disclosure may include a movement track determining module 301, a smooth track determining module 302, and a video anti-shake processing module 303, where:
the movement track determining module 301 is configured to determine a movement track of a shooting position of the video by tracking feature points between different image frames in the video, where the movement track is used to indicate the shooting position of the different image frames in the video;
the smooth track determining module 302 is configured to perform smoothing processing on shooting positions of different image frames in the moving track based on smooth radii corresponding to different image frames in the video, so as to obtain a smooth track; the method comprises the steps that the smooth radius corresponding to different image frames in a video is determined based on the number of feature points used for tracking of the different image frames in the video;
the video anti-shake processing module 303 is configured to deform the video based on the difference between the smooth track and the moving track, so as to obtain an anti-shake processed video.
Optionally, the smooth radius corresponding to the different image frames in the video is determined based on the number interval to which the number of feature points used for tracking by the different image frames in the video belongs and the correspondence between the number interval and the smooth radius.
Optionally, the smooth trajectory determination module 302 includes:
an image frame number determining unit, configured to determine, based on a smoothing radius corresponding to each frame of image in the video, a preset frame number of images participating in each smoothing process in the video;
and the smoothing processing unit is used for carrying out smoothing processing on the shooting position of each frame of image in the moving track based on the shooting position of the image of the preset frame number participating in each smoothing processing to obtain a smoothed track.
Optionally, the image frame number determining unit includes:
a first determining unit, configured to determine, in the video, a previous frame image of a first preset frame number before each frame image based on a smoothing radius corresponding to each frame image in the video;
a second determining unit, configured to determine each frame of image and a previous frame of image of a first preset frame number as images of preset frame numbers participating in each smoothing process in the video; or alternatively
A third determining unit, configured to determine, in the video, a preceding frame image of a second preset frame number before each frame image, and determine a following frame image of the second preset frame number after each frame image, based on a smoothing radius corresponding to each frame image in the video;
and a fourth determining unit configured to determine each frame of image, a preceding frame of image of a second preset frame number, and a following frame of image of the second preset frame number as images of preset frame numbers participating in each smoothing process in the video.
Optionally, the smoothing processing unit includes:
a smoothing position determining subunit, configured to perform weighted summation calculation of the shooting positions based on the shooting positions of images of preset frames participating in each smoothing process in the moving track, so as to obtain a smoothing position of each frame of image;
and the smooth track determining subunit is used for obtaining a smooth track based on the smooth position of each frame of image.
Optionally, the moving track is represented by a transformation matrix, and different transformation matrixes in the moving track respectively represent shooting positions of different image frames in the video;
the movement track determining module 301 is specifically configured to:
and tracking characteristic points among different image frames in the video, determining a transformation matrix of shooting positions among the different image frames in the video, and determining a moving track of the shooting positions of the video based on the transformation matrix of the shooting positions among the different image frames in the video.
Optionally, the video anti-shake processing module 303 includes:
an adjustment parameter determining unit configured to determine an adjustment parameter based on a difference between the smooth trajectory and the moving trajectory;
and the video deformation unit is used for deforming the video by utilizing the adjustment parameters so as to obtain the video subjected to the anti-shake processing.
The video anti-shake processing device provided by the embodiment of the disclosure can execute any video anti-shake processing method provided by the embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of the executing method. Details of the embodiments of the apparatus of the present disclosure that are not described in detail may refer to descriptions of any of the embodiments of the method of the present disclosure.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure, which is used to exemplarily illustrate an electronic device implementing a video anti-shake processing method according to an embodiment of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), car terminals (e.g., car navigation terminals), and the like, as well as stationary terminals such as digital TVs, desktop computers, smart home devices, wearable electronic devices, servers, and the like. The electronic device shown in fig. 4 is merely an example and should not be construed to limit the functionality and scope of the disclosed embodiments.
As shown in fig. 4, electronic device 400 includes one or more processors 401 and memory 402.
The processor 401 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities and may control other components in the electronic device 400 to perform desired functions.
Memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory can include, for example, random Access Memory (RAM) and/or cache memory (cache) and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on a computer readable storage medium, and the processor 401 may execute the program instructions to implement the video anti-shake processing method provided by the embodiments of the present disclosure, and may also implement other desired functions. Various contents such as an input signal, a signal component, a noise component, and the like may also be stored in the computer-readable storage medium.
The video anti-shake processing method provided by the embodiment of the disclosure may include: determining a moving track of a shooting position of the video by tracking characteristic points among different image frames in the video, wherein the moving track is used for indicating the shooting position of the different image frames in the video; respectively carrying out smoothing treatment on shooting positions of different image frames in a moving track based on smoothing radiuses corresponding to different image frames in a video to obtain a smoothing track; the method comprises the steps that the smooth radius corresponding to different image frames in a video is determined based on the number of feature points used for tracking of the different image frames in the video; and deforming the video based on the difference between the smooth track and the moving track to obtain the video subjected to the anti-shake processing. It should be appreciated that electronic device 400 may also perform other alternative implementations provided by the method embodiments of the present disclosure.
In one example, the electronic device 400 may further include: an input device 403 and an output device 404, which are interconnected by a bus system and/or other forms of connection mechanisms (not shown).
In addition, the input device 403 may also include, for example, a keyboard, a mouse, and the like.
The output device 404 may output various information to the outside, including the determined distance information, direction information, and the like. The output device 404 may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, etc.
Of course, only some of the components of the electronic device 400 that are relevant to the present disclosure are shown in fig. 4, with components such as buses, input/output interfaces, etc. omitted for simplicity. In addition, electronic device 400 may include any other suitable components depending on the particular application.
In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product comprising a computer program or computer program instructions which, when executed by a processor, cause the computing apparatus to implement any of the video anti-shake processing methods provided by the embodiments of the present disclosure.
The computer program product may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device and partly on the remote electronic device, or entirely on the remote electronic device.
In addition, the embodiments of the present disclosure may further provide a computer readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause a computing device to implement any of the video anti-shake processing methods provided by the embodiments of the present disclosure.
The video anti-shake processing method provided by the embodiment of the disclosure may include: determining a moving track of a shooting position of the video by tracking characteristic points among different image frames in the video, wherein the moving track is used for indicating the shooting position of the different image frames in the video; respectively carrying out smoothing treatment on shooting positions of different image frames in a moving track based on smoothing radiuses corresponding to different image frames in a video to obtain a smoothing track; the method comprises the steps that the smooth radius corresponding to different image frames in a video is determined based on the number of feature points used for tracking of the different image frames in the video; and deforming the video based on the difference between the smooth track and the moving track to obtain the video subjected to the anti-shake processing. It should be understood that the computer program instructions, when executed by a processor, may also cause a computing device to implement other alternative implementations provided by the method embodiments of the present disclosure.
A computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
The above is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (9)
1. A video anti-shake processing method, comprising:
determining a moving track of a shooting position of a video by tracking characteristic points among different image frames in the video, wherein the moving track is used for indicating the shooting position of the different image frames in the video;
respectively carrying out smoothing treatment on shooting positions of different image frames in the moving track based on smoothing radiuses corresponding to the different image frames in the video to obtain a smoothing track; the method comprises the steps that the smooth radius corresponding to different image frames in a video is determined based on the number of characteristic points used for tracking by the different image frames in the video and the corresponding relation between the number of the characteristic points and the smooth radius;
deforming the video based on the difference between the smooth track and the moving track to obtain a video with a shooting position located on the smooth track, wherein the video with the shooting position located on the smooth track refers to the video subjected to anti-shake processing;
the smoothing processing is performed on the shooting positions of different image frames in the moving track based on the smoothing radii corresponding to the different image frames in the video, so as to obtain a smoothing track, including:
determining images of a preset frame number which participate in each smoothing process in the video based on the smoothing radius corresponding to each frame of image in the video;
and carrying out smoothing processing on the shooting position of each frame of image in the moving track based on the shooting position of the image of the preset frame number participating in each smoothing processing, so as to obtain a smoothed track.
2. The method of claim 1, wherein the smoothing radius corresponding to the different image frames in the video is determined based on a number interval to which the number of feature points used for tracking by the different image frames in the video belongs and a correspondence between the number interval and the smoothing radius, respectively.
3. The method of claim 1, wherein determining the number of images of the video that participate in each smoothing process based on the smoothing radius corresponding to each frame of image in the video comprises:
determining a previous frame image of a first preset frame number before each frame image in the video based on a smooth radius corresponding to each frame image in the video;
determining each frame of image and the previous frame of image with the first preset frame number as images with the preset frame number which participate in each smoothing process in the video; or alternatively
Determining a second preset frame number of previous frame images before each frame image and a second preset frame number of subsequent frame images after each frame image in the video based on the smooth radius corresponding to each frame image in the video;
and determining each frame of image, the previous frame of image with the second preset frame number and the subsequent frame of image with the second preset frame number as images with preset frame numbers which participate in each smoothing process in the video.
4. The method according to claim 3, wherein smoothing the photographing position of each frame of image in the moving trajectory based on the photographing position of the image of the preset number of frames participating in each smoothing process to obtain a smoothed trajectory, comprises:
based on shooting positions of images of preset frames participating in each smoothing process in the moving track, carrying out weighted summation calculation of the shooting positions to obtain a smoothing position of each frame of image;
and obtaining the smooth track based on the smooth position of each frame of image.
5. The method according to claim 1, wherein the moving track is represented by a transformation matrix, and different transformation matrices in the moving track respectively represent shooting positions of different image frames in the video;
the method for determining the moving track of the shooting position of the video by tracking the characteristic points among different image frames in the video comprises the following steps:
and tracking characteristic points among different image frames in the video, determining a transformation matrix of shooting positions among the different image frames in the video, and determining a moving track of the shooting positions of the video based on the transformation matrix of the shooting positions among the different image frames in the video.
6. The method of claim 1, wherein deforming the video based on the difference between the smooth trajectory and the moving trajectory to obtain an anti-shake processed video, comprises:
determining an adjustment parameter based on a difference between the smoothed trajectory and the movement trajectory;
and deforming the video by utilizing the adjustment parameters to obtain the video subjected to anti-shake processing.
7. A video anti-shake processing apparatus, comprising:
the mobile track determining module is used for determining a mobile track of a shooting position of the video by tracking characteristic points among different image frames in the video, wherein the mobile track is used for indicating the shooting position of the different image frames in the video;
the smooth track determining module is used for respectively carrying out smooth processing on shooting positions of different image frames in the moving track based on smooth radiuses corresponding to the different image frames in the video to obtain a smooth track; the method comprises the steps that the smooth radius corresponding to different image frames in a video is determined based on the number of characteristic points used for tracking by the different image frames in the video and the corresponding relation between the number of the characteristic points and the smooth radius;
the video anti-shake processing module is used for deforming the video based on the difference between the smooth track and the moving track to obtain a video with a shooting position located on the smooth track, wherein the video with the shooting position located on the smooth track refers to the video subjected to anti-shake processing;
the smooth trajectory determination module includes:
an image frame number determining unit, configured to determine, based on a smoothing radius corresponding to each frame of image in the video, a preset frame number of images participating in each smoothing process in the video;
and the smoothing processing unit is used for carrying out smoothing processing on the shooting position of each frame of image in the moving track based on the shooting position of the image of the preset frame number participating in each smoothing processing to obtain a smoothed track.
8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the electronic device to implement the video anti-shake processing method of any of claims 1-6.
9. A computer readable storage medium, characterized in that the storage medium has stored therein a computer program, which when executed by a computing device, causes the computing device to implement the video anti-shake processing method of any one of claims 1 to 6.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110379627.2A CN115209030B (en) | 2021-04-08 | 2021-04-08 | Video anti-shake processing method and device, electronic equipment and storage medium |
| PCT/CN2022/085382 WO2022214001A1 (en) | 2021-04-08 | 2022-04-06 | Video image stabilization method and apparatus, electronic device, and storage medium |
| US18/472,001 US20240013347A1 (en) | 2021-04-08 | 2023-09-21 | Method for video anti-shake processing, electronic apparatus, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110379627.2A CN115209030B (en) | 2021-04-08 | 2021-04-08 | Video anti-shake processing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115209030A CN115209030A (en) | 2022-10-18 |
| CN115209030B true CN115209030B (en) | 2024-02-27 |
Family
ID=83545997
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110379627.2A Active CN115209030B (en) | 2021-04-08 | 2021-04-08 | Video anti-shake processing method and device, electronic equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115209030B (en) |
| WO (1) | WO2022214001A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116797497B (en) * | 2023-08-24 | 2023-11-14 | 摩尔线程智能科技(北京)有限责任公司 | Image processing method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108632501A (en) * | 2017-03-23 | 2018-10-09 | 展讯通信（上海）有限公司 | Video anti-fluttering method and device, mobile terminal |
| WO2018223381A1 (en) * | 2017-06-09 | 2018-12-13 | 厦门美图之家科技有限公司 | Video shake-prevention method and mobile device |
| CN110519507A (en) * | 2019-07-23 | 2019-11-29 | 深圳岚锋创视网络科技有限公司 | A kind of camera lens smoothing processing method, device and portable terminal |
| CN111901678A (en) * | 2020-07-31 | 2020-11-06 | 成都云格致力科技有限公司 | Anti-jitter smoothing method and system for TCP real-time video stream |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8872928B2 (en) * | 2010-09-14 | 2014-10-28 | Adobe Systems Incorporated | Methods and apparatus for subspace video stabilization |
| WO2014092193A1 (en) * | 2012-12-14 | 2014-06-19 | 株式会社コンセプト | Image processing device, image processing method, image processing program, program, and mobile terminal device with camera |
| CN105409196B (en) * | 2013-07-23 | 2018-11-30 | 微软技术许可有限责任公司 | Adaptive Path for video stabilization is smooth |
| CN108564554A (en) * | 2018-05-09 | 2018-09-21 | 上海大学 | A kind of video stabilizing method based on movement locus optimization |
| CN108596858B (en) * | 2018-05-10 | 2021-03-09 | 中国科学技术大学 | Traffic video jitter removal method based on characteristic track |
| CN110047091B (en) * | 2019-03-14 | 2022-09-06 | 河海大学 | Image stabilization method based on camera track estimation and feature block matching |
| CN110139031B (en) * | 2019-05-05 | 2020-11-06 | 南京大学 | Video anti-shake system based on inertial sensing and working method thereof |
-
2021
- 2021-04-08 CN CN202110379627.2A patent/CN115209030B/en active Active
-
2022
- 2022-04-06 WO PCT/CN2022/085382 patent/WO2022214001A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108632501A (en) * | 2017-03-23 | 2018-10-09 | 展讯通信（上海）有限公司 | Video anti-fluttering method and device, mobile terminal |
| WO2018223381A1 (en) * | 2017-06-09 | 2018-12-13 | 厦门美图之家科技有限公司 | Video shake-prevention method and mobile device |
| CN110519507A (en) * | 2019-07-23 | 2019-11-29 | 深圳岚锋创视网络科技有限公司 | A kind of camera lens smoothing processing method, device and portable terminal |
| CN111901678A (en) * | 2020-07-31 | 2020-11-06 | 成都云格致力科技有限公司 | Anti-jitter smoothing method and system for TCP real-time video stream |
Non-Patent Citations (1)
| Title |
|---|
| 视频监控图像去抖动视觉监测算法优化仿真;尹雅楠;;计算机仿真(第09期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022214001A1 (en) | 2022-10-13 |
| CN115209030A (en) | 2022-10-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115209031B (en) | Video anti-shake processing method and device, electronic equipment and storage medium | |
| WO2020228522A1 (en) | Target tracking method and apparatus, storage medium and electronic device | |
| CN109766925B (en) | Feature fusion method and device, electronic equipment and storage medium | |
| JP2022508988A (en) | Compression for Face Recognition-Extended Depth Directional Convolutional Neural Network | |
| US20220130025A1 (en) | Picture optimization method device, terminal and corresponding storage medium | |
| CN109902588B (en) | Gesture recognition method and device and computer readable storage medium | |
| CN112561269A (en) | Advisor recommendation method and device | |
| CN107368181B (en) | Gesture recognition method and device | |
| CN115209030B (en) | Video anti-shake processing method and device, electronic equipment and storage medium | |
| CN111951192A (en) | Shot image processing method and shooting equipment | |
| CN111985556A (en) | Key point identification model generation method and key point identification method | |
| CN112991151B (en) | Image processing method, image generation method, apparatus, device, and medium | |
| CN111966473B (en) | Operation method and device of linear regression task and electronic equipment | |
| CN114390201A (en) | Focusing method and device thereof | |
| CN111815748B (en) | Animation processing method and device, storage medium and electronic equipment | |
| WO2024022301A1 (en) | Visual angle path acquisition method and apparatus, and electronic device and medium | |
| CN112183946A (en) | Multimedia content evaluation method, device and training method thereof | |
| WO2024021504A1 (en) | Facial recognition model training method and apparatus, recognition method, and device and medium | |
| CN116934591A (en) | Image stitching method, device and equipment for multi-scale feature extraction and storage medium | |
| CN116012418A (en) | Multi-target tracking method and device | |
| US20240013347A1 (en) | Method for video anti-shake processing, electronic apparatus, and storage medium | |
| CN112967351A (en) | Image generation method and device, electronic equipment and storage medium | |
| CN110443746B (en) | Picture processing method and device based on generation countermeasure network and electronic equipment | |
| CN110222777B (en) | Image feature processing method and device, electronic equipment and storage medium | |
| CN111866609B (en) | Method and apparatus for generating video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |