WO2022214001A1

WO2022214001A1 - Video image stabilization method and apparatus, electronic device, and storage medium

Info

Publication number: WO2022214001A1
Application number: PCT/CN2022/085382
Authority: WO
Inventors: 杨松; 刘宇龙
Original assignee: 北京字跳网络技术有限公司
Priority date: 2021-04-08
Filing date: 2022-04-06
Publication date: 2022-10-13
Also published as: CN115209030A; CN115209030B

Abstract

Embodiments of the present disclosure relate to a video image stabilization method and apparatus, an electronic device, and a storage medium. The method comprises: determining a movement trajectory of shooting positions of a video by means of performing feature point tracking between different image frames in the video; respectively performing smoothing on shooting positions of different image frames in the movement trajectory on the basis of smoothing radii corresponding to the different image frames in the video, and obtaining a smoothed trajectory, wherein the smoothing radii corresponding to the different image frames in the video are respectively determined on the basis of the number of feature points used for tracking of different image frames in the video; and performing shape transformation on the video on the basis of a difference between the smoothed trajectory and the movement trajectory, so as to obtain a video that has undergone image stabilization. An embodiment of the present disclosure achieves an effect of adaptively adjusting smoothing radii corresponding to image frames during video image stabilization according to the number of feature points on image frames in a video to be processed, a video image stabilization effect is optimized, and video quality is effectively improved.

Description

Video anti-shake processing method, device, electronic device and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of the Chinese patent application filed on April 8, 2021 with the application number of 202110379627.2 and the invention titled "video anti-shake processing method, device, electronic device and storage medium". The entire content of the application is approved by Reference is incorporated in this application.

technical field

The present disclosure relates to the technical field of video processing, and in particular, to a video anti-shake processing method, device, electronic device and storage medium.

Background technique

With the rise of short videos, video shooting has become more and more important. At present, users generally use handheld devices to shoot video, which can easily cause video jitter, resulting in poor video quality. Therefore, how to perform anti-shake processing on the video and improve the video quality is still a problem to be solved at present.

SUMMARY OF THE INVENTION

In order to solve the above technical problem or at least partially solve the above technical problem, the embodiments of the present disclosure provide a video anti-shake processing method, apparatus, electronic device, and storage medium.

In a first aspect, an embodiment of the present disclosure provides a video anti-shake processing method, including:

By tracking feature points between different image frames in the video, the movement trajectory of the shooting position of the video is determined, wherein the movement trajectory is used to indicate the shooting position of different image frames in the video;

Based on the smoothing radii corresponding to different image frames in the video, the shooting positions of the different image frames in the moving trajectory are respectively smoothed to obtain a smooth trajectory; wherein the smoothing radii corresponding to the different image frames in the video are based on the respective smoothing radii of the different image frames in the video Determine the number of feature points used for tracking in different image frames in the video;

Based on the difference between the smooth trajectory and the moving trajectory, the video is deformed to obtain an anti-shake processed video.

In a second aspect, an embodiment of the present disclosure further provides a video anti-shake processing device, including:

A movement trajectory determination module, configured to determine the movement trajectory of the shooting position of the video by tracking feature points between different image frames in the video, wherein the movement trajectory is used to indicate the shooting of different image frames in the video Location;

A smooth trajectory determination module, configured to perform smooth processing on the shooting positions of different image frames in the moving trajectory based on the smoothing radii corresponding to different image frames in the video, respectively, to obtain a smooth trajectory; wherein, the different image frames in the video The corresponding smooth radius is determined based on the number of feature points used for tracking in different image frames in the video respectively;

A video anti-shake processing module, configured to deform the video based on the difference between the smooth track and the moving track to obtain a video that has undergone anti-shake processing.

In a third aspect, embodiments of the present disclosure further provide an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the electronic device is made to The device implements any of the video anti-shake processing methods provided in the embodiments of the present disclosure.

In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a computing device, the computing device enables the computing device to implement the embodiment of the present disclosure Any of the provided video anti-shake processing methods.

In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program/instruction, when the computer program/instruction is executed by a processor, the video anti-shake processing according to any one of the embodiments of the present disclosure is implemented. method.

Compared with the prior art, the technical solutions provided by the embodiments of the present disclosure have at least the following advantages: in the embodiments of the present disclosure, based on the number of feature points used for tracking on different image frames in the video to be processed, the The smoothing radius corresponding to the image frame, that is, the embodiment of the present disclosure realizes the effect of adaptively adjusting the smoothing radius corresponding to the different image frames in the video to be processed based on the number of feature points on the image frame, and then based on the corresponding smoothing radius of the different image frames in the video. Smooth radius, respectively smooth the shooting positions of different image frames in the moving track of the video shooting position, and finally deform the video to be processed based on the difference between the smooth track and the moving track, realize the anti-shake processing of the video, and optimize the video Anti-shake processing effect. Compared with the uniform value of the smooth radius, the embodiment of the present disclosure effectively avoids a small number of feature points used for tracking (or called feature points used for feature matching) on the image frame through the dynamic value of the smooth radius. When the number is small, i.e. insufficient features), the motion estimation of the video shooting device used to shoot the video is inaccurate, that is, the estimation of the movement trend of the video shooting position is inaccurate, which in turn causes the phenomenon of video screen shaking and abnormal deformation, which effectively improves the video quality.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the accompanying drawings that are required to be used in the description of the embodiments or the prior art will be briefly introduced below. In other words, on the premise of no creative labor, other drawings can also be obtained from these drawings.

1 is a flowchart of a video anti-shake processing method provided by an embodiment of the present disclosure;

2 is a flowchart of another video anti-shake processing method provided by an embodiment of the present disclosure;

3 is a schematic structural diagram of a video anti-shake processing apparatus according to an embodiment of the present disclosure;

FIG. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments may be combined with each other under the condition of no conflict.

Many specific details are set forth in the following description to facilitate a full understanding of the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, and Not all examples.

FIG. 1 is a flowchart of a video anti-shake processing method provided by an embodiment of the present disclosure, which can be applied to a situation of performing anti-shake processing on a video. The method can be performed by a video anti-shake processing apparatus, which can be implemented by software and/or hardware, and can be integrated on any electronic device with computing capability, such as a terminal or a server.

In the embodiment of the present disclosure, the video to be processed may be a video being shot or a video that has been shot, that is, the embodiment of the present disclosure may perform anti-shake processing on the shot video in real time during the video shooting process, or the video may be Anti-shake processing is performed on the video after shooting, which can improve the quality of the video.

As shown in FIG. 1 , the video anti-shake processing method provided by the embodiment of the present disclosure may include:

S101. Determine the movement track of the shooting position of the video by tracking feature points between different image frames in the video, where the movement track is used to indicate the shooting position of different image frames in the video.

After acquiring the video to be processed, any available feature point extraction and tracking technology can be used to track the feature points between different image frames in the video (for example, starting from the first frame of the video, the feature points on each frame of the image are sequentially performed. point extraction and tracking), determine the matching feature points between different image frames (referring to the feature points of the same object in different image frames, the number of matching feature points can be determined according to the situation), and then calculate the video based on the matching feature points. The shooting positions of different image frames or the relative variation of the shooting positions in the video are obtained to obtain the movement track of the shooting position of the video (or the motion track of the shooting device used for shooting the video). The movement track of the video shooting position is determined, that is, the shaking trend of the video shooting position is determined.

In an optional implementation manner, the movement track is represented by a transformation matrix, and different transformation matrices in the movement track respectively represent the shooting positions of different image frames in the video; correspondingly, by tracking feature points between different image frames in the video , determining the movement trajectory of the shooting position of the video, including: by tracking the feature points between different image frames in the video, determining the transformation matrix of the shooting position between different image frames in the video, and based on the shooting between different image frames in the video. The transformation matrix of the position determines the movement trajectory of the shooting position of the video.

The transformation matrix may include a homography (homography) matrix, an affine transformation (affine transformation) matrix or a similarity transformation (similarity transformation) matrix. Different transformation matrices correspond to different degrees of freedom. A transformation matrix with a higher degree of freedom has a stronger motion fitting ability. In the actual processing process, the type of transformation matrix can be flexibly selected according to requirements. The specific calculation of the transformation matrix can be implemented with reference to the prior art.

In the process of determining the movement trajectory based on multiple transformation matrices, one frame of image in the video can be selected as the reference frame (specifically, it can be determined flexibly), and then based on the transformation matrix of the shooting position between different image frames, each frame of image can be determined separately. The transformation matrix relative to the shooting position of the reference frame, and finally the required movement trajectory is obtained based on the transformation matrix of each frame of image relative to the shooting position of the reference frame.

Taking the reference frame as the first frame image in the video, and different image frames in the video refer to two adjacent frames of images as an example, by tracking the feature points between different image frames in the video, the movement trajectory of the shooting position of the video is determined, include:

By tracking the feature points of two adjacent frames of images in the video, the matching feature points between the two adjacent frames of images are determined, and based on the matched feature points, the transformation matrix of the shooting position between the two adjacent frames of images is calculated;

Based on the transformation matrix of the shooting position between two adjacent frames of images, determine the transformation matrix of the shooting position of each frame of image in the video relative to the shooting position of the first frame of image in the video;

Based on the transformation matrix of each frame of image in the video relative to the shooting position of the first frame of image in the video, the movement track of the shooting position of the video is determined.

For example, assuming that the video V contains n frames of images in total, and denote the i-th frame image as f _i , then the video V={f ₁ ,f ₂ ,...,f _n-1 ,f _n, }, and sequentially for each frame of the video The image is processed as follows:

1) For the i-th frame image f _i , extract feature points, denoted as p _i .

2) Track the feature point p _i of the previous frame image on the i+1th frame image f _i+1 , and the tracked feature points are recorded as p _i～i+1 , namely p _i and p _{i～i+ 1} are the successfully matched feature points on the i-th frame image f _i and the i+1-th frame image f _i+1 ;

3) According to the corresponding relationship of p _i and p _i～i+1 , fit the transformation matrix of the shooting position of the i-th frame image f _i to the i+1-th frame image f _i+1 , and denote it as T _i ;

4) Accumulate the transformation matrix between the ith frame image f _i and the adjacent two frame images before the ith frame image, for example, perform cumulative multiplication calculation (specifically, it can be determined according to the actual processing), and obtain the ith frame. The transformation matrix of the image f _i relative to the shooting position of the first frame image is expressed as follows:

The transformation matrix of the shooting position of each frame of image in the video relative to the first frame of image is obtained in turn, then the movement trajectory of the shooting position of the video can be expressed as C={C ₁ ,C ₂ ,...,C _n-1 ,C _n }, n represents the number of image frames included in the video.

S102. Based on the smoothing radii corresponding to different image frames in the video, perform smoothing processing on the shooting positions of different image frames in the moving trajectory respectively, to obtain a smooth trajectory; wherein, the smoothing radii corresponding to different image frames in the video are based on the different image frames in the video respectively. The number of feature points used for tracking is determined.

The number of feature points used for tracking in different image frames in the video (or called feature points that are successfully matched on the image of the subsequent frame and the image of the previous frame) is related to the photographed object on each frame of image. Taking two adjacent frames of images as face images as an example, the complete face area is displayed on the previous frame image, including a large number of feature points, while only half of the face area is displayed on the next frame image, including the The number of feature points is small, and the feature points used for tracking on the subsequent frame image in the two adjacent frames of images may only be part of the feature points corresponding to the facial features in the facial feature points.

In the embodiment of the present disclosure, the corresponding relationship between the number of feature points and the smoothing radius is preset, so that the smoothing radius corresponding to the image frame can be dynamically determined according to the corresponding relationship and the number of feature points used for tracking on the image frame. The value determines the number of image frames involved in smoothing. When the number of feature points used for tracking on the image frame is more (that is, there are more features), the more accurate the motion estimation of the shooting device used for shooting video is based on the image frame. Therefore, for the image frame, you can set a larger , so that the shooting position of the image frame is smoothed based on more adjacent image frames, and no larger cumulative error is introduced; when the number of feature points on the image frame is less (that is, less Then, based on the image frame, the accuracy of the motion estimation of the shooting device used for shooting the video is relatively low. Therefore, for the image frame, a smaller smoothing radius can be set, so that the image frame is based on fewer adjacent image frames. The shooting position is smoothed to avoid introducing large cumulative errors, so as to finally optimize the processing effect of video anti-shake.

Optionally, the smoothing radii corresponding to different image frames in the video are respectively determined based on the number interval to which the number of feature points used for tracking of the different image frames in the video belong, and the corresponding relationship between the number interval and the smoothing radius. Exemplarily, the smooth radius r_i corresponding to each frame of image f_i in the video can be determined using the following correspondence:

Among them, m ₁ >m ₂ >m ₃ , each value can be set flexibly, for example, m ₁ can be 50, m ₂ can be 25, m ₃ can be 10; n _i represents each frame of image f _i is relative to the number of feature points used for tracking in the previous frame image, r ₀ represents the initial value of the smooth radius, and its value can also be flexibly determined. It should be noted that, in the above formula, for different feature point quantity intervals, the relationship between the value of the smooth radius _ri and the initial value r ₀ is just an example, and _ri can be flexibly set according to processing requirements in actual processing. The value proportional relationship with r ₀ .

By presetting the corresponding relationship between the feature point quantity interval and the value of the smoothing radius, it is helpful to improve the efficiency of determining the smoothing radius corresponding to each frame of image.

In this embodiment of the present disclosure, the smoothing algorithm may be any available algorithm in the prior art that can achieve a smoothing effect, such as a Gaussian smoothing algorithm and the like. The smooth trajectory obtained by smoothing can be expressed as, for example,

The smoothing trajectory is used to indicate the shooting positions of different image frames in the smoothed video, such as

Indicates the position where the shooting position of the nth frame image in the video has been smoothed.

Optionally, based on the smoothing radii corresponding to different image frames in the video, smoothing is performed on the shooting positions of different image frames in the moving track respectively, to obtain a smooth track, including:

Based on the smoothing radius corresponding to each frame of image in the video, determine the images in the video that participate in the preset number of frames in each smoothing process;

Based on the shooting positions of the images of the preset number of frames participating in each smoothing process in the moving track, smoothing is performed on the shooting position of each frame of the image in the moving track to obtain a smooth track. For example, a weighted sum calculation may be performed on the shooting positions of the images of the preset number of frames involved in each smoothing process in the moving track, and the calculation result may be used as the smoothing result of the shooting positions of the current frame image.

S103 , deforming the video based on the difference between the smooth trajectory and the moving trajectory to obtain a video that has undergone anti-shake processing.

will smooth the trajectory

Comparing with the moving trajectory C before smoothing, the adjustment parameter W={W ₁ , W ₂ ,..., W _n-1 , W _n } of the moving trajectory can be determined, and each sub-value in the adjustment parameter W can be determined.

n represents the number of image frames included in the video; then, according to the corresponding relationship between each sub-value in the adjustment parameter and each frame of image in the video, the corresponding image frame can be deformed based on each sub-value in the adjustment parameter, so as to obtain anti-shake processed video. In the deformation processing process, it involves the rotation, translation, scaling or cropping of a specific frame image, which can be performed according to actual processing requirements. That is, in the embodiment of the present disclosure, deforming the video based on the difference between the smooth trajectory and the moving trajectory to obtain an anti-shake processed video includes: based on the difference between the smooth trajectory and the moving trajectory, Determine the adjustment parameters; use the adjustment parameters to deform the video to obtain a video that has undergone anti-shake processing.

In the embodiment of the present disclosure, based on the number of feature points used for tracking on different image frames in the video to be processed, the smooth radius corresponding to the different image frames in the video is dynamically determined, that is, the embodiment of the present disclosure realizes the The number of feature points, adaptively adjust the effect of the smoothing radius corresponding to different image frames in the video, and then based on the smoothing radius corresponding to different image frames in the video, respectively smooth the shooting positions of different image frames in the moving trajectory of the shooting position of the video. Finally, based on the difference between the smooth track and the moving track, the video is deformed to realize the anti-shake processing of the video, and the processing effect of the video anti-shake is optimized. Compared with the uniform value of the smooth radius, the embodiment of the present disclosure effectively avoids a small number of feature points used for tracking (or called feature points used for feature matching) on the image frame through the dynamic value of the smooth radius. When the number is small, i.e. insufficient features), the motion estimation of the video shooting device used to shoot the video is inaccurate, that is, the estimation of the movement trend of the video shooting position is inaccurate, which in turn causes the phenomenon of video screen shaking and abnormal deformation, which effectively improves the video quality.

FIG. 2 is a flowchart of another video anti-shake processing method provided by an embodiment of the present disclosure, which is further optimized and expanded based on the foregoing technical solution, and may be combined with the foregoing optional implementation manners.

As shown in FIG. 2 , the video anti-shake processing method provided by the embodiment of the present disclosure may include:

S201. Determine the movement track of the shooting position of the video by tracking feature points between different image frames in the video, where the movement track is used to indicate the shooting position of different image frames in the video.

S202 , based on the smoothing radius corresponding to each frame of the image in the video, determine the images in the video with a preset number of frames participating in each smoothing process.

Exemplary one, based on the smooth radius corresponding to each frame of image in the video, the previous frame image of the first preset number of frames before each frame of image can be determined in the video;

Determine each frame of image and the previous frame image of the first preset number of frames as the image of the preset number of frames participating in each smoothing process in the video; at this time, the value of the first preset number of frames may be smoothing The value of the radius is 1.

The embodiments of the present disclosure support anti-shake processing on the video while the video is being captured. After the video is captured, the video anti-shake processing also ends, thereby obtaining a captured video with better quality.

Exemplary two, based on the smooth radius corresponding to each frame of image in the video, determine the previous frame image of the second preset number of frames before each frame of image in the video, and determine the second preset number of frames after each frame of image. the image in the later frame;

Each frame of image, the previous frame image of the second preset number of frames, and the subsequent frame image of the second preset number of frames are determined as the images of the preset number of frames participating in each smoothing process in the video; at this time, the The value of the second preset number of frames may be the value of the smoothing radius.

The embodiment of the present disclosure also supports the anti-shake processing of the video after shooting, and takes a certain number of images before and after each frame of image to participate in the smooth processing of the shooting position of each frame of image in the moving trajectory, which also improves the video quality. Effect.

S203 , based on the shooting positions of the images of the preset number of frames participating in each smoothing process in the movement track, perform weighted summation calculation of the shooting positions to obtain the smoothing positions of each frame of images.

Taking the movement trajectory including multiple transformation matrices, that is, the shooting position of each frame of image in the video can be represented by a transformation matrix as an example, the smoothing position of each frame image (that is, the shooting position after smoothing processing) can be expressed in the form of a matrix, the following is called a smoothing matrix. The weight in the weighted sum calculation process may take adaptive values, which is not specifically limited in the embodiment of the present disclosure.

Exemplary one, according to the smoothing radius corresponding to each frame of image, take a certain number of images before each frame of image, and participate in the smoothing processing of the shooting position of each frame of image as an example, the smoothing matrix of each frame of image f _i can be expressed as follows: :

Exemplary 2, take images of the same number of frames before and after each frame of image according to the smoothing radius corresponding to each frame of image, and participate in the smoothing processing of the shooting position of each frame of image as an example, the smoothing matrix of each frame of image f _i It can be expressed as follows:

In the above two examples, r is the smoothing radius corresponding to each frame of image, C _t represents the shooting position (or called transformation matrix) of each frame of image participating in the smoothing process in the moving track C, and w _i～t is the participating smoothing The weight of each frame of image processed, the value can be set adaptively. After obtaining the smoothing matrix corresponding to each frame of image in the video, the smoothing trajectory can be expressed as follows:

The smooth position corresponding to each frame of image is obtained by weighting and summing the shooting positions of the images with a preset number of frames in the moving trajectory. The smoothing effect of the movement trajectory of the position.

S204 , obtaining a smooth trajectory based on the smoothed position of each frame of the image.

S205 , deforming the video based on the difference between the smooth trajectory and the moving trajectory to obtain a video that has undergone anti-shake processing.

In the embodiment of the present disclosure, the smooth radius corresponding to each frame of image is dynamically determined according to the number of feature points used for tracking in each frame of image in the video to be processed during the video anti-shake processing process, and then a certain number of image frames is dynamically determined, In order to participate in the smooth processing of the movement trajectory of the video shooting position, the processing effect of video anti-shake is optimized. Compared with the case where the smooth radius is uniformly valued, the embodiment of the present disclosure effectively avoids, through the dynamic value of the smooth radius, that when the number of feature points used for tracking on the image frame in the video is small, it is The motion estimation of the shooting device is inaccurate, that is, the estimation of the movement trend of the video shooting position is inaccurate, which further causes the phenomenon of video picture shaking and abnormal deformation, which effectively improves the video quality.

FIG. 3 is a schematic structural diagram of a video anti-shake processing apparatus according to an embodiment of the present disclosure, which can be applied to a situation of performing anti-shake processing on a video. The apparatus can be implemented by software and/or hardware, and can be integrated on any electronic device with computing capability, such as a terminal or a server.

As shown in FIG. 3 , the video anti-shake processing apparatus 300 provided by the embodiment of the present disclosure may include a movement trajectory determination module 301, a smooth trajectory determination module 302, and a video anti-shake processing module 303, wherein:

The movement trajectory determination module 301 is used to determine the movement trajectory of the shooting position of the video by tracking the feature points between different image frames in the video, wherein the movement trajectory is used to indicate the shooting position of different image frames in the video;

The smooth trajectory determination module 302 is configured to perform smooth processing on the shooting positions of different image frames in the moving trajectory based on the smoothing radii corresponding to different image frames in the video, respectively, to obtain a smooth trajectory; wherein, the smoothing radii corresponding to different image frames in the video are respectively Determined based on the number of feature points used for tracking in different image frames in the video;

The video anti-shake processing module 303 is configured to deform the video based on the difference between the smooth track and the moving track, so as to obtain a video that has undergone anti-shake processing.

Optionally, the corresponding smoothing radii of different image frames in the video are respectively determined based on the number interval to which the number of feature points used for tracking by different image frames in the video belongs, and the corresponding relationship between the number interval and the smoothing radius.

Optionally, the smooth trajectory determination module 302 includes:

an image frame number determination unit, configured to determine the images of the preset number of frames participating in each smoothing process in the video based on the smoothing radius corresponding to each frame of image in the video;

The smoothing processing unit is configured to perform smoothing processing on the shooting position of each frame of images in the moving trajectory based on the shooting positions of the images of the preset number of frames participating in each smoothing processing in the moving trajectory to obtain a smoothed trajectory.

Optionally, the image frame number determination unit includes:

The first determining unit is used to determine the previous frame image of the first preset frame number before each frame image in the video based on the smooth radius corresponding to each frame image in the video;

a second determining unit, configured to determine each frame of image and the previous frame image of the first preset number of frames as the image of the preset number of frames participating in each smoothing process in the video; or

The third determining unit is configured to determine, based on the smooth radius corresponding to each frame of image in the video, the previous frame image of the second preset number of frames before each frame of image in the video, and to determine the second preset number of frames after each frame of image The following frame image of the frame number;

The fourth determining unit is used to determine each frame of image, the previous frame image of the second preset frame number and the subsequent frame image of the second preset frame number as the preset number of frames participating in each smoothing process in the video Image.

Optionally, the smoothing processing unit includes:

The smoothing position determination subunit is used to perform the weighted sum calculation of the shooting position based on the shooting position of the image of the preset number of frames participating in each smoothing process in the moving track, so as to obtain the smoothing position of each frame of image;

The smoothed trajectory determination subunit is used to obtain a smoothed trajectory based on the smoothed position of each frame of image.

Optionally, the movement track is represented by a transformation matrix, and different transformation matrices in the movement track respectively represent the shooting positions of different image frames in the video;

The movement trajectory determination module 301 is specifically used for:

By tracking feature points between different image frames in the video, the transformation matrix of the shooting position between different image frames in the video is determined, and the movement trajectory of the shooting position of the video is determined based on the transformation matrix of the shooting position between different image frames in the video. .

Optionally, the video anti-shake processing module 303 includes:

an adjustment parameter determination unit for determining an adjustment parameter based on the difference between the smooth trajectory and the moving trajectory;

The video deformation unit is used to deform the video by using the adjustment parameters to obtain the video that has undergone anti-shake processing.

The video anti-shake processing apparatus provided by the embodiment of the present disclosure can execute any video anti-shake processing method provided by the embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method. For the content that is not described in detail in the apparatus embodiment of the present disclosure, reference may be made to the description in any method embodiment of the present disclosure.

Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure, and is used to exemplarily describe an electronic device that implements the video anti-shake processing method provided by the embodiment of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (eg, Mobile terminals such as car navigation terminals), etc., and stationary terminals such as digital TVs, desktop computers, smart home devices, wearable electronic devices, servers, and the like. The electronic device shown in FIG. 4 is only an example, and should not impose any limitation on the functions and occupancy scope of the embodiments of the present disclosure.

As shown in FIG. 4 , electronic device 400 includes one or more processors 401 and memory 402 .

Processor 401 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in electronic device 400 to perform desired functions.

Memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and/or cache memory, among others. Non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may execute the program instructions to implement the video anti-shake processing method provided by the embodiments of the present disclosure, and may also implement other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

The video anti-shake processing method provided by the embodiment of the present disclosure may include: by tracking feature points between different image frames in the video, determining a movement trajectory of the shooting position of the video, wherein the movement trajectory is used to indicate different images in the video. The shooting position of the frame; based on the smoothing radii corresponding to different image frames in the video, the shooting positions of different image frames in the moving track are smoothed respectively to obtain a smooth track; wherein, the smoothing radii corresponding to different image frames in the video are based on the The number of feature points used for tracking in different image frames is determined; based on the difference between the smooth trajectory and the moving trajectory, the video is deformed to obtain an anti-shake processed video. It should be understood that the electronic device 400 may also perform other optional implementations provided by the method embodiments of the present disclosure.

In one example, the electronic device 400 may also include an input device 403 and an output device 404 interconnected by a bus system and/or other form of connection mechanism (not shown).

In addition, the input device 403 may also include, for example, a keyboard, a mouse, and the like.

The output device 404 can output various information to the outside, including the determined distance information, direction information, and the like. The output devices 404 may include, for example, displays, speakers, printers, and communication networks and their connected remote output devices, among others.

Of course, for simplicity, only some of the components in the electronic device 400 related to the present disclosure are shown in FIG. 4 , and components such as buses, input/output interfaces, and the like are omitted. Besides, the electronic device 400 may also include any other appropriate components according to the specific application.

In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be computer program products, which include computer programs or computer program instructions, which, when executed by a processor, cause a computing device to implement what is provided by the embodiments of the present disclosure. Any video anti-shake processing method.

When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device.

The computer program product may write program code for performing operations of embodiments of the present disclosure in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., as well as conventional procedural programming language, such as "C" language or similar programming language. The program code may execute entirely on the user electronic device, partly on the user electronic device, as a stand-alone software package, partly on the user electronic device and partly on the remote electronic device, or entirely on the remote electronic device execute on.

In addition, the embodiments of the present disclosure may further provide a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by the processor, the computer program instructions enable the computing device to implement any video anti-shake processing provided by the embodiments of the present disclosure method.

The video anti-shake processing method provided by the embodiment of the present disclosure may include: by tracking feature points between different image frames in the video, determining a movement trajectory of the shooting position of the video, wherein the movement trajectory is used to indicate different images in the video. The shooting position of the frame; based on the smoothing radii corresponding to different image frames in the video, the shooting positions of different image frames in the moving track are smoothed respectively to obtain a smooth track; wherein, the smoothing radii corresponding to different image frames in the video are based on the The number of feature points used for tracking in different image frames is determined; based on the difference between the smooth trajectory and the moving trajectory, the video is deformed to obtain an anti-shake processed video. It should be understood that, when the computer program instructions are executed by the processor, the computer program instructions can also cause the computing device to implement other optional implementations provided by the method embodiments of the present disclosure.

A computer-readable storage medium can employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses or devices, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

It should be noted that, in this document, relational terms such as "first" and "second" etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these There is no such actual relationship or sequence between entities or operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article, or device that includes the element.

The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not to be limited to the embodiments herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

A video anti-shake processing method, comprising:

By tracking feature points between different image frames in the video, the movement trajectory of the shooting position of the video is determined, wherein the movement trajectory is used to indicate the shooting position of different image frames in the video;

Based on the smoothing radii corresponding to different image frames in the video, the shooting positions of the different image frames in the moving trajectory are respectively smoothed to obtain a smooth trajectory; wherein the smoothing radii corresponding to the different image frames in the video are based on the respective smoothing radii of the different image frames in the video Determine the number of feature points used for tracking in different image frames in the video;

Based on the difference between the smooth trajectory and the moving trajectory, the video is deformed to obtain an anti-shake processed video.
The method according to claim 1, wherein the smoothing radii corresponding to different image frames in the video are based on the number interval to which the number of feature points used for tracking of the different image frames in the video belong, and the number interval and The correspondence of the smoothing radius is determined.
The method according to claim 1, wherein, based on the smoothing radii corresponding to different image frames in the video, smoothing is performed on the shooting positions of different image frames in the moving track respectively to obtain a smooth track, comprising:

Based on the smoothing radius corresponding to each frame of image in the video, determine the image of the preset number of frames participating in each smoothing process in the video;

Based on the shooting positions of the images of the preset number of frames participating in each smoothing process in the moving track, smoothing is performed on the shooting position of each frame of the image in the moving track to obtain a smooth track.
The method according to claim 3, wherein, based on the smoothing radius corresponding to each frame of image in the video, determining the images of the preset number of frames participating in each smoothing process in the video, comprising:

Based on the smooth radius corresponding to each frame of image in the video, determine the previous frame image of the first preset number of frames before each frame of image in the video;

Determining each frame of image and the previous frame image of the first preset number of frames as the image of the preset number of frames participating in each smoothing process in the video; or

Based on the smooth radius corresponding to each frame of image in the video, determine the previous frame image of the second preset number of frames before each frame of image in the video, and determine the second preset number of frames after each frame of image in the video. image in later frame;

Determining each frame of image, the previous frame image of the second preset frame number, and the subsequent frame image of the second preset frame number as the number of preset frames participating in each smoothing process in the video. image.
The method according to claim 3, wherein the shooting position of each frame of images in the moving track is smoothed based on the shooting positions of the images of the preset number of frames participating in each smoothing process in the moving track processing to obtain a smooth trajectory, including:

Based on the shooting positions of the images of the preset number of frames participating in each smoothing process in the moving track, a weighted sum calculation of the shooting positions is performed to obtain the smoothing position of each frame of image;

The smoothed trajectory is obtained based on the smoothed position of each frame of image.
The method according to claim 1, wherein the movement track is represented by a transformation matrix, and different transformation matrices in the movement track respectively represent the shooting positions of different image frames in the video;

By tracking feature points between different image frames in the video, the movement trajectory of the shooting position of the video is determined, including:

By tracking the feature points between different image frames in the video, the transformation matrix of the shooting position between different image frames in the video is determined, and the transformation matrix of the shooting position between different image frames in the video is determined based on the transformation matrix of the shooting position between the different image frames in the video. The movement trajectory of the shooting position of the video.
The method according to claim 1, wherein the video is deformed based on the difference between the smooth trajectory and the moving trajectory to obtain an anti-shake processed video, comprising:

determining an adjustment parameter based on the difference between the smooth trajectory and the moving trajectory;

The video is deformed by using the adjustment parameter to obtain an anti-shake processed video.
A video anti-shake processing device, comprising:

A movement trajectory determination module, configured to determine the movement trajectory of the shooting position of the video by tracking feature points between different image frames in the video, wherein the movement trajectory is used to indicate the shooting of different image frames in the video Location;

A smooth trajectory determination module, configured to perform smooth processing on the shooting positions of different image frames in the moving trajectory based on the smoothing radii corresponding to different image frames in the video, respectively, to obtain a smooth trajectory; wherein, the different image frames in the video The corresponding smooth radius is determined based on the number of feature points used for tracking in different image frames in the video respectively;

A video anti-shake processing module, configured to deform the video based on the difference between the smooth track and the moving track to obtain a video that has undergone anti-shake processing.
An electronic device, characterized by comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the electronic device is made to implement claims 1-7 The video anti-shake processing method described in any one.
A computer-readable storage medium, characterized in that, a computer program is stored in the storage medium, and when the computer program is executed by a computing device, the computing device is made to implement any one of claims 1-7. The video anti-shake processing method.
A computer program product, comprising a computer program/instruction, characterized in that, when the computer program/instruction is executed by a processor, the video anti-shake processing method according to any one of claims 1-7 is implemented.