WO2018223381A1 - Video shake-prevention method and mobile device - Google Patents

Abstract

Disclosed is a video shake-prevention method. The method comprises: acquiring tri-axial angular velocity data of a gyroscope and photographed video frame image data of a mobile device during a photographing process; according to the axial angular velocity of the gyroscope within adjacent time intervals, obtaining, by means of calculation, a rotation angle of a corresponding axis; for each photographed video frame, calculating a tri-axial rotation angle corresponding to the video frame; calculating a first motion trajectory of the video frame according to the tri-axial rotation angle of the video frame and a camera calibration matrix; according to at least one reference frame adjacent to the video frame, smoothing the first motion trajectory of the video frame, so as to obtain a motion trajectory of the video frame; carrying out block segmentation processing on the video frame, and calculating a motion trajectory of each sub-block according to the motion trajectory of the video frame; and according to the motion trajectory of each sub-block, adjusting image data within each sub-block, so as to output the video frame after being subjected to shake-prevention processing. Also disclosed is a corresponding mobile device.

Description

Video anti-shake method and mobile device Technical field

The present invention relates to the field of image processing technologies, and in particular, to a video anti-shake method and a mobile device.

Background technique

With the rapid development of science and technology, various mobile devices are continuously enriched and convenient for the public life. Because of its advantages of convenience, saving resources, etc., it has become an indispensable part of people's lives.

Among them, mobile devices with cameras such as smart phones and tablet computers enable users to take video images they want at any time and increase the user experience. However, during the video shooting process, due to environmental factors or other factors, for example, during shooting during walking or driving, due to the irregular shaking of the shooting target caused by the following movement of the device, the captured video image is shaken, thereby reducing User's perception.

It can be seen that in order to improve the shooting effect, it is necessary to perform anti-shake processing on the acquired video. There are two main types of video anti-shake technology: one is to use the motion sensor to detect the motion vector of the camera and then convert it into the motion amount of the image to perform anti-shake; the other is to directly use the digital image processing technology to determine the amount of motion between the images. The compensation is achieved to achieve the purpose of anti-shake. The anti-shake method based on digital image processing technology depends on the quality of the image, and it is prone to a situation in which the estimation error of the motion amount is large, and the anti-shake effect is not ideal under the conditions of large rapid movement and dim shooting environment.

Therefore, how to effectively reduce the video jitter caused by the instability of mobile devices during video shooting is an urgent problem to be solved.

Summary of the invention

To this end, the present invention provides a video anti-shake method and mobile device in an effort to solve or at least alleviate at least one of the problems present above.

According to an aspect of the present invention, a video anti-shake method is provided for performing anti-shake processing on a video captured by a mobile device, the method comprising the steps of: obtaining a three-axis angular velocity of the gyroscope of the mobile device during shooting Data and captured video frame image data; calculating the rotation angle of the corresponding axis according to the angular velocity of the gyroscope in the adjacent time interval; for each video frame captured: calculating the three-axis rotation angle corresponding to the video frame Calculating a first motion trajectory of the video frame according to a three-axis rotation angle of the video frame and a camera calibration matrix; and smoothing the first motion trajectory of the video frame according to at least one reference frame adjacent to the video frame to obtain the video The motion track of the frame; the video frame is subjected to block processing, and the motion track of each block is calculated according to the motion track of the video frame; and the image data in each block is adjusted according to the motion track of each block, and the image stabilization is output. The processed video frame.

Optionally, in the video anti-shake method according to the present invention, the step of acquiring the triaxial angular velocity data of the gyroscope during the photographing process of the mobile device further comprises: constraining the acquired gyroscope triaxial angular velocity value into a predetermined interval; And using a predetermined kernel function to smooth the corresponding axial angular velocity of the current moment gyroscope and the corresponding axial angular velocity of the plurality of gyroscopes in the preceding and following time periods, to obtain the triaxial angular velocity of the gyroscope.

Optionally, in the video image stabilization method according to the present invention, the step of calculating the rotation angle of the corresponding axis according to the angular velocity of the gyroscope in the adjacent time interval further comprises: time stamp information in the adjacent time interval of the gyroscope Timestamp information characterizing the three-axis rotation angle is determined as the first time.

Optionally, in the video anti-shake method according to the present invention, the video frame image data is acquired, and the system time corresponding to the video frame is also acquired as the second time.

Optionally, in the video anti-shake method according to the present invention, the step of calculating a three-axis rotation angle corresponding to the video frame comprises: matching the correspondence between the first time and the three-axis rotation angle by using the second time of the video frame The three-axis rotation angle of the video frame.

Optionally, in the video anti-shake method according to the present invention, the method further includes the step of correcting the second time in advance: correcting the second time of the video frame according to the exposure time of the video frame, and obtaining the corrected second time. .

Optionally, in the video anti-shake method according to the present invention, the second time after the correction is:

Frame_time2=frame_time1+base_val+(0.03-exp_Time)×0.5

Where frame_time1 is the second time of the video frame, and frame_time2 is the corrected video frame. The second time, base_val is the reference correction value, and exp_Time is the exposure time of the video frame. When the exposure time of the video frame cannot be obtained, exp_Time is 0.

Optionally, in the video anti-shake method according to the present invention, the step of matching the three-axis rotation angle of the current video frame according to the correspondence between the first time and the three-axis rotation angle comprises: finding whether there is correction from the first time The second time after the second time, if present, the three-axis rotation angle corresponding to the first time found is taken as the three-axis rotation angle of the video frame; and if not, the three-axis rotation of the video frame is calculated according to a predetermined condition. angle.

Optionally, in the video image stabilization method according to the present invention, the step of calculating a three-axis rotation angle of the video frame according to a predetermined condition comprises: extracting from the first time two before and after the corrected second time Time; and calculating a three-axis rotation angle of the video frame based on the two times taken and their corresponding three-axis rotation angles.

Optionally, in the video anti-shake method according to the present invention, the three-axis rotation angle θ _i of the video frame is:

Where i=x, y, z represent the three coordinate axes x, y, z, respectively, and gyro(k)_time and gyro(k+1)_time represent two times before and after the second time frame_time2 after correction, Gyro(k)_θ _i and gyro(k+1)_θ _i represent the three-axis rotation angles corresponding to the two times.

Optionally, in the video anti-shake method according to the present invention, the camera of the mobile device is calibrated by using the Zhang Zhengyou calibration algorithm, and the camera focal length is obtained.

Optionally, in the video anti-shake method according to the present invention, the first motion trajectory H=KR,

Where K is the camera calibration matrix and R is the rotation matrix.

Where θ _x , θ _y , and θ _z represent the rotation angles of the three axes x, y, and z, respectively.

Optionally, in the video anti-shake method according to the present invention, the smoothing processing of the first motion trajectory includes:

Where t is the current image frame and r is the front/back reference frame adjacent to t. H(t) represents the motion trajectory of the t-th frame, H(r) represents the motion trajectory of the r-th frame, and P(t) represents the motion trajectory of the t-th frame after the trajectory smoothing. Gt represents the weight of the rth frame at the frame sequence level for the tth frame, and Gm represents the weight of the rth frame for the tth frame at the motion trajectory level.

Optionally, in the video anti-shake method according to the present invention, the step of smoothing the first motion trajectory of the video frame to obtain a motion trajectory of the video frame comprises: smoothing the first motion trajectory to obtain a second motion Tracking, adjusting the second motion trajectory to obtain a motion trajectory of the video frame, that is, calculating a difference value between the first motion trajectory and the second motion trajectory; if the difference value is greater than the threshold value, adjusting the second motion trajectory according to the difference value Until the difference value is less than the threshold, the adjusted second motion trajectory is taken as the motion trajectory of the video frame; and if the difference value is smaller than the threshold value, the second motion trajectory is used as the motion trajectory of the video frame.

Optionally, in the video image stabilization method according to the present invention, the calculating the difference value of the first motion trajectory and the second motion trajectory comprises: defining initial coordinates of four corner points of the effective region according to image data of the video frame Calculating a first coordinate set of four corner points under the first motion trajectory; calculating a second coordinate set of the four corner points under the second motion trajectory; and calculating a rectangular area respectively determined by the first coordinate set and the second coordinate set The area difference is used as the difference value between the two motion trajectories.

Optionally, in the video anti-shake method according to the present invention, the first coordinate X _t = H _t X of the corner point of the first motion track, and the second coordinate X _s = H _s X of the corner point of the second motion track, wherein H _t is the homography matrix of the first motion trajectory, H _s is the homography matrix of the second motion trajectory, and X is the initial coordinate of the corner point.

Optionally, in the video anti-shake method according to the present invention, the difference value diff=Area _t -Area _s ,

Where Area _t and Area _s represent the area of the rectangular area respectively determined by the first coordinate set and the second coordinate set.

Optionally, in the video anti-shake method according to the present invention, the step of adjusting the second motion trajectory according to the difference value comprises: calculating an interpolation ratio according to the difference value; and linearly according to the interpolation ratio The interpolation method adjusts the second motion trajectory.

Alternatively, in the video anti-shake method according to the present invention, the interpolation ratio radio=Area _t /diff.

Optionally, in the video anti-shake method according to the present invention, the threshold is:

Threshold=cos(atan2(min(width,height)/2,f)),

Where width and height represent the width and height of the video frame, respectively, and f is the nominal camera focal length.

Optionally, in the video anti-shake method according to the present invention, the step of performing block processing on the video frame comprises: segmenting the video frame in order from top to bottom according to the characteristics of the rolling shutter.

Optionally, in the video anti-shake method according to the present invention, the step of calculating each block motion trajectory according to the motion trajectory of the video frame comprises: calculating the system time of each block according to the shutter time; calculating according to the system time of each block The three-axis rotation angle corresponding to each block; the initial motion trajectory of each block is calculated according to the three-axis rotation angle of each block and the camera calibration matrix; and the initial motion trajectory of each block and the motion track of the video frame are calculated Obtain the motion trajectory of each block.

Optionally, in the video anti-shake method according to the present invention, the system time of each partition is defined as:

t(y)=frame_time2+t _s *y/height,

Where y represents the row index number of each partition and t _s represents the shutter time.

Optionally, in the video anti-shake method according to the present invention, the motion trajectory of each block is:

H'(y)=H'*H(y) ^-1 ,

Where H' is the motion trajectory of the video frame, and H(y) is the initial motion trajectory of each block.

Alternatively, in the video image stabilization method according to the present invention, the predetermined interval is [-4, 4].

According to still another aspect of the present invention, a mobile device is provided, comprising: a camera subsystem adapted to acquire video image data; a gyroscope; one or more processors; a memory; one or more programs, wherein the one Or a plurality of programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods described above.

According to still another aspect of the present invention, a computer readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a mobile device, cause the mobile device to perform the above Any of the methods described.

According to the video anti-shake scheme of the present invention, a gyroscope is used to obtain a mobile device during a video capture process. The three-axis rotation angle, and then obtain the motion trajectory of a certain frame of the mobile device; and then use the characteristics of the rolling shutter to block the video frame and transform each block according to the motion trajectory of the block And finally output a stable video. The scheme is not only simple in algorithm, fast in calculation speed, but also does not depend on the quality of the video image itself, and is particularly suitable for scenes in which the quality of the captured video image itself is not high, for example, anti-shake processing of a video shot in an indoor low-light environment.

DRAWINGS

In order to achieve the above and related objects, certain illustrative aspects are described herein in conjunction with the following description and the accompanying drawings. Within the scope of the claimed subject matter. The above as well as other objects, features and advantages of the present invention will become more apparent from the Detailed Description Throughout the disclosure, the same reference numbers generally refer to the same parts or elements.

FIG. 1 shows a schematic configuration of a mobile device 100 in accordance with one embodiment of the present invention;

FIG. 2 shows a flow chart of a video anti-shake method 200 in accordance with one embodiment of the present invention.

detailed description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While the embodiments of the present invention have been shown in the drawings, the embodiments Rather, these embodiments are provided so that this disclosure will be more fully understood and the scope of the disclosure will be fully disclosed.

FIG. 1 shows a schematic configuration of a mobile device 100 in accordance with one embodiment of the present invention. Referring to FIG. 1, mobile device 100 includes a memory interface 102, one or more data processors, an image processor and/or central processing unit 104, and a peripheral interface 106. Memory interface 102, one or more processors 104, and/or peripheral interface 106 can be either discrete components or integrated into one or more integrated circuits. In mobile device 100, various components may be coupled by one or more communication buses or signal lines. Sensors, devices, and subsystems can be coupled to the peripheral interface 106 to help implement a variety of functions. For example, motion sensor 110, light sensor 112, and distance sensor 114 can be coupled to peripheral interface 106 to facilitate functions such as orientation, illumination, and ranging. Other sensors 116 are the same It may be coupled to a peripheral interface 106, such as a positioning system (e.g., a GPS receiver), a temperature sensor, a biometric sensor, or other sensing device, thereby helping to implement related functions. According to an implementation of the present invention, the other sensor 116 includes at least one angular velocity sensor, that is, a gyroscope, which is different from the acceleration sensor (G-sensor), and the gyroscope is mainly used to measure the rotational angular velocity of the mobile device 100 when the deflection and tilt occur. . According to an embodiment of the present invention, the gyroscope is arranged as a three-axis gyroscope, and the three-axis rotational angular velocity data can be simultaneously measured.

Camera subsystem 120 and optical sensor 122 may be used to facilitate implementation of camera functions such as recording photos and video clips, where the camera subsystem and optical sensor may be, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) optics. sensor. In the present invention, the video image is acquired by the camera subsystem 120, and the video data with good anti-shake effect is obtained after post-processing.

Communication functions may be facilitated by one or more wireless communication subsystems 124, which may include radio frequency receivers and transmitters and/or optical (e.g., infrared) receivers and transmitters. The particular design and implementation of wireless communication subsystem 124 may depend on one or more communication networks supported by mobile device 100. For example, the mobile device 100 may include a network is designed to support GSM, GPRS communication network, EDGE network, Wi-Fi or WiMax network, and the network subsystem 124 Bluebooth ^TM. The audio subsystem 126 can be coupled to the speaker 128 and the microphone 130 to assist in implementing voice-enabled functions such as voice recognition, voice replication, digital recording, and telephony functions.

I/O subsystem 140 may include touch screen controller 142 and/or one or more other input controllers 144. Touch screen controller 142 can be coupled to touch screen 146. For example, the touch screen 146 and the touch screen controller 142 can detect contact and movement or pause with any of a variety of touch sensing technologies, including but not limited to capacitive, Resistive, infrared and surface acoustic wave technology. One or more other input controllers 144 may be coupled to other input/control devices 148, such as one or more buttons, rocker switches, thumb wheels, infrared ports, USB ports, and/or pointing devices such as styluses . One or more buttons (not shown) may include up/down buttons for controlling the volume of the speaker 128 and/or the microphone 130.

Memory interface 102 can be coupled to memory 150. The memory 150 can include high speed random access memory and/or nonvolatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and/or flash memory (eg, NAND, NOR). The memory 150 can To store an operating system 152, such as an operating system such as Android, IOS, or Windows Phone. The operating system 152 can include instructions for processing basic system services and performing hardware dependent tasks. The memory 150 can also store the application 154. These applications, when operational, are loaded from the memory 150 onto the processor 104 and run on top of the operating system already run by the processor 104, and utilize the interface provided by the operating system and the underlying hardware to implement various user desired functions. Such as instant messaging, web browsing, photo management, and so on. Applications can be either independent of the operating system or native to the operating system. In some implementations, the application 154 can be one or more programs.

The present invention provides a video anti-shake solution that achieves the above functionality by storing a corresponding one or more programs in memory 150 of mobile device 100. It should be noted that the mobile device 100 referred to in the present invention may be a mobile phone, a tablet, a camera, or the like having the above configuration.

FIG. 2 shows a flow chart of a video anti-shake method 200 in accordance with one embodiment of the present invention. 2, the method begins at step S210, the mobile device 100 acquired triaxial gyro angular velocity data during recording (referred to as _{w x, w y, w z} ) and video frames of the photograph image data.

According to an embodiment of the present invention, the acquired triaxial angular velocity data of the gyroscope is further corrected and smoothed.

During the correction process, the gyroscope triaxial angular velocity is corrected according to the camera direction. Specifically, the triaxial angular velocity is adjusted according to the direction of the mobile device 100 (horizontal or vertical), and finally adjusted to w _x represents the angular velocity of the pitch, w _y represents the angular velocity of the heading angle (yaw), and w _z represents The angular velocity of the roll. That is, the acquired gyroscope triaxial angular velocity values are constrained to a predetermined interval. Generally, according to the empirical value, the predetermined interval is taken between [-4, 4].

Then, when performing the smoothing process, the corresponding angular velocity of the gyroscope at the current time and the corresponding angular angular velocity of the plurality of gyroscopes in the adjacent time period (that is, the gyro collected in a certain time interval) by using a predetermined kernel function The corresponding angular velocity of the instrument is smoothed. Taking the angular velocity data _{α of} a certain axis of the gyroscope at the moment _α as an example, the corresponding angular velocity of the plurality of gyroscopes in the adjacent time interval is:

{data _α-4 ,data _α-3 ,data _α-2 ,data _α-1 ,data _α ,data _α+1 ,data _α+2 ,data _α+3 },

According to the predetermined kernel function [Kernel0, Kernel1, Kernel2, Kernel3, Kernel4, Kernel5, Kernel6, Kernel7] smoothes it and outputs the triaxial angular velocity value as the triaxial angular velocity of the gyroscope.

According to an embodiment of the present invention, while acquiring the video frame image data, the system time corresponding to each video frame is also acquired as the second time, which is recorded as frame_time1.

Then in step S220, the rotation angle of the corresponding axis is calculated based on the angular velocity of the gyroscope in the adjacent time interval and the time interval. Optionally, the rotation angle of each axis is calculated by performing uniform motion calculation on the average value of the corresponding angular velocity of the gyroscope in the time interval.

At the same time, the timestamp information characterizing the three-axis rotation angle is determined as the first time by the timestamp information in the adjacent time interval of the gyroscope. Optionally, the first time takes the mean of the two timestamps in the adjacent time interval.

Thus, during the video capture, each of the calculated three-axis rotation angles has a corresponding first time corresponding thereto.

In the next step, each video frame in the captured video (ie, the image frame, for the unified description, the video frame referred to below refers to the image frame in the captured video) is subjected to anti-shake processing. .

In step S230, a three-axis rotation angle corresponding to the current video frame is calculated. In this step, it is necessary to use the second time of the current video frame (determined in step S210) to match the three-axis rotation of the video frame according to the correspondence between the first time and the three-axis rotation angle (determined in step S220). angle.

According to the embodiment of the present invention, before calculating the three-axis rotation angle of the video frame, it is necessary to correct the second time according to the exposure time of the video frame, and the corrected second time frame_time2 is:

Frame_time2=frame_time1+base_val+(0.03-exp_Time)×0.5

Where frame_time1 is the second time of the video frame (ie, obtained in step S210), frame_time2 is the second time of the corrected video frame, base_val is the reference correction value, and exp_Time is the exposure time of the video frame. In particular, when the exposure time of a video frame cannot be acquired, exp_Time is set to zero.

After the second time after the correction is obtained, it is searched from the first time whether there is a second time after the correction, and if present, the three-axis rotation angle corresponding to the first time found is used as the three axes of the video frame. Rotation angle.

If the second time after the correction is not found, the three-axis rotation angle of the video frame is calculated according to predetermined conditions. Specifically, two times before and after the corrected second time are taken out from the first time, and then the three-axis rotation angle θ of the video frame is calculated according to the two times taken and their corresponding three-axis rotation angles. _i , such as:

Where i=x, y, z represent the three coordinate axes x, y, z, respectively, and gyro(k)_time and gyro(k+1)_time represent two times before and after the second time frame_time2 after correction, Gyro(k)_θ _i and gyro(k+1)_θ _i represent the three-axis rotation angles corresponding to the two times.

According to this step, the three-axis rotation angle corresponding to each video frame can be calculated.

Then in step S240, the first motion trajectory of the video frame is calculated from the three-axis rotation angle of the video frame and the camera calibration matrix. According to an embodiment of the present invention, assuming that the camera performs a pure rotational motion, the first motion trajectory H is calculated as follows:

H=KR,

Where K is the camera calibration matrix, R is the rotation matrix, and

Where θ _x , θ _y , and θ _z represent the rotation angles of the three axes x, y, and z, respectively.

Optionally, the camera of the mobile device 100 is calibrated using the Zhang Zhengyou calibration algorithm, and the camera focal length is obtained, denoted as f. Since the Zhang Zhengyou calibration algorithm is an algorithm generally known to those skilled in the art and is not the focus of the present invention, the description will not be repeated here.

Then, in step S250, the first motion trajectory of the video frame is smoothed according to the transformation trend of the motion trajectory of the at least one reference frame adjacent to the video frame and the space-time distance of the reference frame from the current video frame (that is, Referring to the motion track of multiple frames before and after the current video frame, smoothing the current video frame, fitting the second motion track of the video frame, and then adjusting the second motion track to obtain the video frame. Movement track.

Optionally, smoothing the first motion trajectory includes:

Where t is the current image frame and r is the front/back reference frame adjacent to t. H(t) represents the motion trajectory of the t-th frame, H(r) represents the motion trajectory of the r-th frame, and P(t) represents the motion trajectory of the t-th frame after the trajectory smoothing. G _t denotes the weight of the rth frame at the frame sequence level for the tth frame, G _m denotes the weight of the rth frame for the tth frame at the motion trajectory level, and G _t makes the reference frame closer to the current video frame larger The weight, G _m ensures the change of the trajectory of the two video frames. For more details, please refer to the paper "MeshFlow: Minimum Latency Online Video Stabilization, S Liu, P Tan, L Yuan, et al, Springer International Publishing, 2016". Due to limited space, it will not be repeated here.

Then, the difference is compared according to the first motion trajectory and the second motion trajectory, and if the difference produces a black edge effect, the black edge suppression operation is performed, that is, the second motion trajectory is adjusted. It is judged whether a black edge effect is generated, that is, a difference value between the first motion trajectory and the second motion trajectory is calculated, and whether the difference value is compared within a threshold range is compared.

More specifically, the flow of calculating the difference value of the first motion trajectory and the second motion trajectory is:

a. Defining the initial coordinates of the four corner points of the active area based on the image data of the video frame.

b. Calculate the first coordinate of each of the four corner points under the first motion trajectory as the first coordinate set. Assuming that the initial coordinate of one of the four corner points is X(x, y), then the first coordinate X _t = H _t X of the corner point under the first motion trajectory, where H _t is the first motion The homography matrix of the trajectory.

In other forms, the formula for calculating the corner coordinate X _t (x', y') of the first motion trajectory is:

Where h _t1 , h _t2 , h _t3 represent the first, second, and third rows of the homography matrix H _t , and T represents the transpose of the matrix.

Similarly, the first coordinates of each of the four corner points can be calculated to obtain the first coordinate set.

c. Calculating a second coordinate of each of the four corner points under the second motion trajectory as a second coordinate set. In the same manner as in the above step b, the second coordinate X _s = H _s X of a corner point under the second motion trajectory, where H _s is the homography matrix of the second motion trajectory. That is, the calculation formula of the corner coordinate X _s (x", y") of the first motion locus is:

Where h _s1 , h _s2 , h _s3 represent the first, _second , and third rows of the homography matrix H _s , and T represents the transposition of the matrix.

Similarly, the second coordinates of the four corner points are calculated accordingly, and the second coordinate set is obtained.

d. According to the four corner points in the first coordinate set, a rectangle can be determined. Similarly, a rectangle can be determined according to the four corner points in the second coordinate set, and the area difference of the two rectangular areas is calculated as The difference between these two motion trajectories, ie,

The difference value diff=Area _t -Area _s ,

Where Area _t and Area _s represent the area of the rectangular area respectively determined by the first coordinate set and the second coordinate set.

So far, the difference values of the first motion trajectory and the second motion trajectory can be calculated through the above steps a, b, c, d, and then the magnitude of the difference value and the threshold value are compared to determine whether a black edge effect is generated.

According to an embodiment of the invention, the threshold is calculated from the camera focal length obtained by calibration:

Threshold=cos(atan2(min(width,height)/2,f)),

Wherein, width and height respectively represent the width and height of the video frame, f is the calibrated camera focal length, and f has been obtained in step S240.

If the comparison result is that the difference value is greater than the threshold, the second motion trajectory is adjusted according to the difference value until the difference value is smaller than the threshold value (ie, the black edge suppression operation), and the adjusted second motion trajectory is used as the motion of the video frame. Trajectory; conversely, if the comparison results in a difference value that is less than the threshold, the second motion trajectory is taken as the motion trajectory of the video frame.

Specifically, the step of adjusting the second motion trajectory according to the difference value may be divided into two steps: first, the interpolation ratio is calculated according to the difference value, and optionally, the interpolation ratio is defined as radio=Area _t /diff.

Then, according to the calculated interpolation ratio radio, the second motion trajectory is approached to the first motion trajectory by radio in a linear interpolation manner.

Then in step S260, the video frame is subjected to blocking processing, and according to the motion trajectory of the video frame Calculate the motion trajectory of each block. According to an embodiment of the present invention, video frames are segmented in order from top to bottom according to the characteristics of the rolling shutter, that is, the video frames are divided into rows.

According to an implementation, the step of calculating each block motion trajectory according to the motion trajectory of the video frame includes:

1) Calculate the system time of each block according to the shutter time. For example, the system time to define each partition is:

t(j)=frame_time2+t _s *j/height,

Where j denotes the row index number of each partition, t _s denotes the shutter time, and frame_time 2 has been calculated in step S230.

2) According to the system time of each block, the three-axis rotation angle corresponding to each block can be interpolated in the cumulatively recorded three-axis rotation angle of the gyroscope (obtained in the above step S220), optionally, linearly interpolated The method interpolates the corresponding three-axis rotation angle of each block, wherein the linear ratio is calculated based on the system time of each block and the system time of the corresponding gyroscope.

3) Calculating the initial motion trajectory of each block according to the triaxial rotation angle of each block and the camera calibration matrix, and calculating the initial motion trajectory H(j) of each block is the same as step S240, and details are not described herein again.

4) Combining the initial motion trajectory H(j) of each block and the motion trajectory of the video frame (determined by step S250), the motion trajectory of each block is calculated, and the motion trajectory of each block is defined as:

H'(j)=H'*H(j) ^-1 ,

Where H' is the motion trajectory of the video frame, and H(j) is the initial motion trajectory of each block.

So far, the final motion trajectory of each block of the video frame is determined according to the final motion trajectory of the video frame.

Then, in step S270, the image data in each of the blocks is adjusted according to the motion track H'(j) of each block, and a stable video frame is output. Optionally, the image pixels of the block are affine or projected transformed according to the obtained motion trajectory of each block, and the frame data after the anti-shake is output.

In summary, the video anti-shake solution uses the gyroscope to obtain the three-axis rotation angle of the mobile device 100 during the video acquisition process, and then acquires the motion trajectory of a certain video frame of the mobile device, and obtains the defense according to the trajectory smoothing algorithm and the suppression black-side operation. The motion track of the shaken video frame; then, using the characteristics of the rolling shutter, the video frame is divided into blocks, and each block is transformed according to the motion track of the block, The final output is a stable video.

The video anti-shake scheme according to the present invention not only has a simple algorithm, a fast calculation speed, but also does not depend on the quality of the video image itself, and is particularly suitable for a scene in which the quality of the captured video image itself is not high, for example, shooting in an indoor low-light environment. The video is anti-shake.

The various techniques described herein can be implemented in conjunction with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions of the methods and apparatus of the present invention, may take the form of program code embedded in a tangible medium, such as a floppy disk, CD-ROM, hard disk drive, or any other machine-readable storage medium. The form of (i.e., instruction), wherein when a program is loaded into, and executed by, a machine such as a computer, the machine becomes a device embodying the present invention.

Where the program code is executed on a programmable computer, the computing device typically includes a processor, a processor readable storage medium (including volatile and nonvolatile memory and/or storage elements), at least one input device, And at least one output device. Wherein the memory is configured to store program code; the processor is configured to perform the video image stabilization method of the present invention in accordance with instructions in the program code stored in the memory.

The computer readable media includes computer storage media and communication media by way of example and not limitation. Computer readable media includes both computer storage media and communication media. Computer storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. Combinations of any of the above are also included within the scope of computer readable media.

It is to be understood that the various features of the present invention are sometimes grouped together in a single embodiment, figure, or In the description of it. However, the method disclosed is not to be interpreted as reflecting the intention that the claimed invention requires more features than those recited in the claims. Rather, as the following claims reflect, inventive aspects reside in less than all features of the single embodiments disclosed herein. Therefore, the claims following the specific embodiments are hereby explicitly incorporated into the embodiments, and each of the claims as a separate embodiment of the invention.

Those skilled in the art will appreciate the modules or singles of the devices in the examples disclosed herein. The elements or components may be arranged in a device as described in this embodiment, or alternatively may be located in one or more devices that are different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into a plurality of sub-modules.

Those skilled in the art will appreciate that the modules in the devices of the embodiments can be adaptively changed and placed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and further they may be divided into a plurality of sub-modules or sub-units or sub-components. In addition to such features and/or at least some of the processes or units being mutually exclusive, any combination of the features disclosed in the specification, including the accompanying claims, the abstract and the drawings, and any methods so disclosed, or All processes or units of the device are combined. Each feature disclosed in this specification (including the accompanying claims, the abstract and the drawings) may be replaced by alternative features that provide the same, equivalent or similar purpose.

The invention also discloses:

A9. The method of A8, wherein the calculating the three-axis rotation angle of the video frame according to a predetermined condition comprises: extracting two times before and after the second time after the correction from the first time; The two times taken out and their corresponding three-axis rotation angles calculate the three-axis rotation angle of the video frame.

A10. The method of A9, wherein the three-axis rotation angle θ _i of the video frame is:

Where i=x, y, z represent the three coordinate axes x, y, z, respectively, and gyro(k)_time and gyro(k+1)_time represent two times before and after the second time frame_time2 after correction, Gyro(k)_θ _i and gyro(k+1)_θ _i represent the three-axis rotation angles corresponding to the two times.

The method of any one of A1-10, wherein the step of calculating a first motion trajectory of the video frame according to a three-axis rotation angle of the video frame and a camera calibration matrix further comprises: using a Zhang Zhengyou calibration algorithm on the mobile device The camera is calibrated and the camera focal length is determined.

A12. The method of A11, wherein the first motion trajectory H=KR,

Where K is the camera calibration matrix and R is the rotation matrix.

Where θ _x , θ _y , and θ _z represent the rotation angles of the three axes x, y, and z, respectively.

A13. The method of any of A1-12, wherein smoothing the first motion trajectory comprises:

Where t is the current image frame and r is the front/back reference frame adjacent to t. H(t) represents the motion trajectory of the t-th frame, H(r) represents the motion trajectory of the r-th frame, and P(t) represents the motion trajectory of the t-th frame after the trajectory smoothing. Gt represents the weight of the rth frame at the frame sequence level for the tth frame, and Gm represents the weight of the rth frame for the tth frame at the motion trajectory level.

The method of any one of A1-13, wherein the step of smoothing the first motion trajectory of the video frame to obtain a motion trajectory of the video frame comprises: smoothing the first motion trajectory to obtain a first a moving track, adjusting a second moving track to obtain a motion track of the video frame: calculating a difference value between the first motion track and the second motion track; and if the difference value is greater than the threshold, the second motion track according to the difference value The adjustment is performed until the difference value is smaller than the threshold, the adjusted second motion trajectory is taken as the motion trajectory of the video frame; and if the difference value is smaller than the threshold value, the second motion trajectory is used as the motion trajectory of the video frame.

A15. The method of A14, wherein calculating the difference value of the first motion trajectory and the second motion trajectory comprises: defining initial coordinates of four corner points of the effective region according to image data of the video frame; calculating the first motion a first coordinate set of the four corner points under the track; a second coordinate set of four corner points under the second motion track; and an area difference of the rectangular area respectively determined by the first coordinate set and the second coordinate set respectively As the difference value of these two motion trajectories.

A16. The method of A15, wherein the first coordinate X _t = H _t X of the corner point of the first motion trajectory, wherein H _t is a homography matrix of the first motion trajectory, and X is an initial coordinate of the corner point .

A17. The method of A16, wherein the second coordinate X _s = H _s X of the corner point of the second motion trajectory, wherein H _s is a homography matrix of the second motion trajectory, and X is an initial coordinate of the corner point .

A18. The method of any of A15-17, wherein the difference value diff=Area _t -Area _s , wherein Area _t and Area _s represent a rectangular area respectively determined by the first coordinate set and the second coordinate set Area.

A19. The method of A18, wherein the adjusting the second motion trajectory according to the difference value comprises: calculating an interpolation ratio according to the difference value; and adjusting the second motion trajectory by a linear interpolation method according to the interpolation ratio.

A20. The method of A19, wherein the interpolation ratio is radio=Area _t /diff.

The method of any of A14-20, wherein the threshold is:

Threshold=cos(atan2(min(width,height)/2,f)), where width and height represent the width and height of the video frame, respectively, and f is the nominal camera focal length.

A22. The method of any of A1-20, wherein the step of performing a block processing on the video frame comprises: segmenting the video frame in a top-to-bottom order according to characteristics of the rolling shutter.

The method of any of A1-2, wherein the step of calculating each of the block motion trajectories according to the motion trajectory of the video frame comprises: calculating a system time of each block according to a shutter time; Calculating the corresponding three-axis rotation angle of each block; calculating the initial motion trajectory of each block according to the three-axis rotation angle of each block and the camera calibration matrix; and combining the initial motion trajectory of each block and the motion of the video frame The trajectory calculation obtains the motion trajectory of each block.

A24. The method of A23, wherein the system time of each partition is:

t(y)=frame_time2+t _s *y/height, where y represents the row index number of each partition and t _s represents the shutter time.

A25. The method of A24, wherein the motion trajectory H'(y)=H'*H(y) ^{-1 of} each block, wherein H' is a motion trajectory of the video frame, and H(y) is each The initial motion trajectory of the block.

A26. The method of any of A1-25, wherein the predetermined interval is [-4, 4].

In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features that are included in other embodiments and not in other features, combinations of features of different embodiments are intended to be within the scope of the present invention. Different embodiments are formed and formed. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

Moreover, some of the described embodiments are described herein as being operative by a processor of a computer system or A method or combination of method elements implemented by other means performing the described functions. Accordingly, a processor having the necessary instructions for implementing the method or method elements forms a means for implementing the method or method elements. Furthermore, the elements described herein of the device embodiments are examples of means for performing the functions performed by the elements for the purpose of carrying out the invention.

As used herein, the use of the ordinal "first", "second", "third", etc., to describe a generic object merely means a different instance referring to a similar object, and is not intended to imply such The objects being described must have a given order in time, space, ordering, or in any other way.

While the present invention has been described in terms of a limited number of embodiments, it will be understood by those skilled in the art that In addition, it should be noted that the language used in the specification has been selected primarily for the purpose of readability and teaching, and is not intended to be interpreted or limited. Therefore, many modifications and changes will be apparent to those skilled in the art without departing from the scope of the invention. The disclosure of the present invention is intended to be illustrative, and not restrictive, and the scope of the invention is defined by the appended claims.

Claims (10)

1. A video anti-shake method for performing anti-shake processing on a video captured by a mobile device, the method comprising the steps of:

   Obtaining three-axis angular velocity data of the gyroscope and captured video frame image data of the mobile device during shooting;

   Calculating the rotation angle of the corresponding axis according to the angular velocity of the gyroscope in the adjacent time interval;

   For each video frame taken:

   Calculating a three-axis rotation angle corresponding to the video frame;

   Calculating a first motion trajectory of the video frame according to a three-axis rotation angle of the video frame and a camera calibration matrix;

   Smoothing a first motion trajectory of the video frame according to at least one reference frame adjacent to the video frame, to obtain a motion trajectory of the video frame;

   Performing block processing on the video frame, and calculating a motion trajectory of each block according to the motion trajectory of the video frame;

   The image data in each block is adjusted according to the motion track of each block, and the video frame after the anti-shake process is output.
2. The method of claim 1 wherein said step of obtaining three-axis angular velocity data of the gyroscope during the photographing of the mobile device further comprises:

   Constraining the acquired gyroscope triaxial angular velocity values to a predetermined interval;

   The predetermined angular function of the gyroscope at the current time and the corresponding angular velocity of the plurality of gyroscopes in the preceding and following time periods are smoothed by a predetermined kernel function to obtain the triaxial angular velocity of the gyroscope.
3. The method according to claim 1 or 2, wherein the step of calculating the rotation angle of the corresponding axis based on the angular velocity of the gyroscope in the adjacent time interval further comprises:

   The timestamp information characterizing the three-axis rotation angle is determined as the first time by the timestamp information in the adjacent time interval of the gyroscope.
4. The method of claim 3, wherein the step of acquiring video frame image data further comprises:

   Get the system time corresponding to the video frame as the second time.
5. The method of claim 4 wherein said step of calculating a corresponding three-axis rotation angle of the video frame comprises:

   The three-axis rotation angle of the video frame is matched according to the second time of the video frame according to the correspondence between the first time and the three-axis rotation angle.
6. The method of claim 5 further comprising the step of correcting the second time in advance:

   The second time of the video frame is corrected according to the exposure time of the video frame, and the corrected second time is obtained.
7. The method of claim 6 wherein the corrected second time is:

   Frame_time2=frame_time1+base_val+(0.03-exp_Time)×0.5

   Where frame_time1 is the second time of the video frame, frame_time2 is the second time of the corrected video frame, base_val is the reference correction value, and exp_Time is the exposure time of the video frame. When the exposure time of the video frame cannot be obtained, exp_Time is 0. .
8. The method according to claim 6 or 7, wherein the step of matching the three-axis rotation angle of the current video frame according to the correspondence between the first time and the three-axis rotation angle comprises:

   Finding from the first time whether there is a second time after the correction, if present, the three-axis rotation angle corresponding to the first time found is taken as the three-axis rotation angle of the video frame;

   If not, the three-axis rotation angle of the video frame is calculated according to predetermined conditions.
9. A mobile device comprising:

   a camera subsystem adapted to capture video image data;

   Gyro;

   One or more processors;

   Memory

   One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including for performing according to claim 1 -8 instructions of any of the methods described.
10. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a mobile device, cause the mobile device to perform the method of claims 1-8 Any of the methods.

Images