CN114339101A

CN114339101A - Video recording method and device

Info

Publication number: CN114339101A
Application number: CN202011057023.8A
Authority: CN
Inventors: 孙思佳; 朱聪超; 王宇; 卢圣卿
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-09-29
Filing date: 2020-09-29
Publication date: 2022-04-12
Anticipated expiration: 2040-09-29
Also published as: CN114339101B

Abstract

The embodiment of the application provides a video recording method and video recording equipment, relates to the technical field of electronics, and can obtain motion information of a camera by combining motion information of a shot main body and attitude sensor data under a video recording scene, so that anti-shake processing is performed according to the motion information of the camera, anti-shake is realized on the motion main body and a background at the same time, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved. The specific scheme is as follows: the method comprises the steps that after a video recording function is started, an electronic device collects an original image; acquiring motion information of a main body according to the collected multi-frame original images; acquiring the motion information of the camera according to the attitude sensor data corresponding to the multiple frames of original images and the motion information of the main body; calculating a stable image transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in a plurality of frames of original images; and carrying out image transformation on the first original image according to the image stabilization transformation matrix to obtain a target image. The embodiment of the application is used for video anti-shaking.

Description

Video recording method and device

Technical Field

The embodiment of the application relates to the technical field of electronics, in particular to a video recording method and video recording equipment.

Background

With the development of shooting technology, the requirements of users on video recording effects are higher and higher. In the video recording process, electronic equipment such as a mobile phone or a tablet computer can perform anti-shake processing on an image according to gyroscope (gyro) data. This anti-shake processing scheme can realize better anti-shake effect to the background of shooting within range, but is difficult to realize the anti-shake to the motion main part of shooting within range, and the user shoots and experiences relatively poorly.

Disclosure of Invention

The embodiment of the application provides a video recording method and video recording equipment, which can obtain motion information of a camera by combining motion information of a shot main body and attitude sensor data under a video recording scene, so that anti-shake processing is performed according to the motion information of the camera, anti-shake is realized on the motion main body and a background at the same time, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

in one aspect, an embodiment of the present application provides a video recording method, which is applied to an electronic device, where the electronic device includes a camera, and the method may include: the electronic equipment collects an original image after the video recording function is started. The electronic equipment acquires the motion information of the main body according to the collected multi-frame original images. And then, the electronic equipment obtains the motion information of the camera according to the posture sensor data corresponding to the multi-frame original image and the motion information of the main body. The electronic equipment calculates a stable image transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in a plurality of frames of original images. And the electronic equipment performs image transformation on the first original image according to the image stabilization transformation matrix to obtain a target image.

In the scheme, the electronic equipment can obtain the motion information of the camera according to the motion information of the shot main body and the attitude sensor data under the video recording scene, so that the image stabilization transformation matrix is obtained according to the motion information of the camera, and then the motion compensation and the anti-shake are realized on the motion main body and the background through the image stabilization transformation matrix, the image stabilization effect of the video image is improved, and the shooting experience of a user is improved.

In one possible design, the acquiring, by the electronic device, the original image after the video recording function is turned on may include: the electronic equipment starts a video recording function and acquires an original image after detecting a shooting operation of a user. The method may further comprise: after the electronic equipment detects that the shooting operation of the user is stopped, a video file is generated according to the video image, and the video image is a target image.

That is to say, in the video shooting process, the electronic device can obtain the motion information of the camera by combining the motion information of the shot main body and the attitude sensor data, so that the original image is subjected to anti-shake processing according to the motion information of the camera to obtain a video image, the motion compensation and anti-shake are realized on the motion main body and the background at the same time, the image stabilizing effect of the video image is improved, and the shooting experience of a user is improved.

In another possible design, acquiring the original image after starting the video recording function may include: the electronic equipment starts a video recording function and acquires an original image after detecting a shooting operation of a user. The method may further comprise: the electronic equipment displays a recorded image on a shooting interface, wherein the recorded image is a target image.

That is to say, in the video shooting process, the electronic equipment can combine the motion information of the shot main body and the attitude sensor data to obtain the motion information of the camera, so that the anti-shake processing is carried out on the original image according to the motion information of the camera to obtain the recorded image on the shooting interface, the motion compensation and the anti-shake are simultaneously realized on the motion main body and the background, the image stabilization effect of the recorded image is improved, and the shooting experience of a user is improved.

In another possible design, the target image is a preview image, and the method may further include: the electronic device displays the preview image on the preview interface.

That is to say, in the preview state after the video recording function is started, the electronic device can obtain the motion information of the camera by combining the motion information of the photographed main body and the attitude sensor data, so that the preview image on the preview interface is obtained by performing anti-shake processing on the original image according to the motion information of the camera, and therefore motion compensation and anti-shake are simultaneously realized on the moving main body and the background, the image stabilizing effect of the recorded image is improved, and the user shooting experience is improved.

In another possible design, the motion information of the subject is a target motion curve of the subject, the target motion curve of the subject is obtained according to an original motion curve of the subject, and the original motion curve of the subject is obtained according to a position of a center of the subject on the multi-frame original image.

That is, the original motion curve of the subject and the target motion curve of the subject may be used to represent the motion of the subject.

In another possible design, the target motion profile of the subject is continuous; the target motion curve of the main body is first-order and second-order conductive, and the curvature is smaller than or equal to a first preset threshold; the distance between the center of the subject on the target motion curve of the subject and the center of the subject on the original motion curve of the subject is less than or equal to a first preset value.

That is, the motion track of the face center is smooth, and the change cannot be too severe; the change of the speed and the acceleration of the target motion curve of the human face of the subject is smooth and cannot be too violent; the face center on the target motion curve of the subject face cannot deviate too far from the original motion curve of the subject face, and the face center on the target motion curve of the subject face should be near the original motion curve of the subject face.

In another possible design, the motion information of the camera is a target motion curve of the camera, and the obtaining, by the electronic device, the motion information of the camera according to the posture sensor data corresponding to the multiple frames of original images and the motion information of the subject may include: the electronic equipment obtains an original motion curve of the camera according to the attitude sensor data corresponding to the multiple frames of original images. The electronic device obtains a target motion curve of the camera according to the original motion curve and the subject constraint of the camera. The subject constraint is related to a target motion curve of the subject, and the subject constraint is used for enabling the distance between the subject center on the target image and the corresponding subject center on the target motion curve of the subject to be smaller than or equal to a second preset value.

That is, the electronic device may combine the target motion curve of the subject and the raw motion curve of the camera to obtain the target motion curve of the camera.

In another possible design, the obtaining, by the electronic device, a target motion curve of the camera according to the original motion curve of the camera and subject constraints may include: the electronic equipment determines a corresponding main body constraint gear according to the motion amplitude of the main body. The electronic equipment gradually increases the main body constraint strength from the initial value to an upper bound value corresponding to the main body constraint gear. The electronic device obtains a target motion curve of the camera according to the original motion curve and the subject constraint strength of the camera.

Therefore, the electronic equipment can dynamically adjust the strength of the main body constraint according to the motion amplitude of the main body so as to balance the anti-shake effect of the background and the main body and ensure that the image stabilizing effect of the background and the main body is good. Moreover, when the electronic equipment dynamically adjusts the strength of the face constraint, the face constraint strength can be smoothly changed, so that sudden change of the anti-shake strength of the background or the main body is avoided, sudden change of the image effect of the background and the main body is avoided, the anti-shake effect of the background and the main body is smoothly transited, and the shooting experience of a user is improved.

In another possible design, the obtaining, by the electronic device, motion information of the subject according to the collected multiple frames of original images may include: and after the electronic equipment meets the preset condition, acquiring the motion information of the main body according to the collected multi-frame original image.

That is, the electronic device starts the anti-shake mode of the main body after a certain condition is satisfied, thereby acquiring the motion information of the main body, so as to acquire the motion information of the camera according to the motion information of the main body and the data of the attitude sensor, thereby performing anti-shake processing according to the motion information of the camera.

In another possible design, the preset condition may include: the motion amplitude of the electronic equipment is greater than or equal to a third preset value; the area ratio of the main body on the original image is greater than or equal to a fourth preset value; the number of frames in which the original images of the subject appear continuously is greater than or equal to a fifth preset value.

That is, when the motion amplitude of the electronic device is large, the ratio of the subject is large, and the subject continuously appears on the original image, the electronic device obtains the motion information of the camera according to the motion information of the subject and the data of the attitude sensor, so as to perform the anti-shake processing according to the motion information of the camera.

In another possible design, the obtaining, by the electronic device, the motion information of the camera according to the posture sensor data corresponding to the multiple frames of original images and the motion information of the subject may include: the electronic device obtains motion information of the camera according to the posture sensor data corresponding to the N frames of original images and the motion information of the subject, wherein N is an integer larger than 1, N is N1+ I + N2, N1 and I are positive integers, and N2 is a non-negative integer. The electronic equipment calculates an image stabilization transformation matrix of the first original image according to the motion information of the camera, and the image stabilization transformation matrix comprises the following steps: the electronic equipment calculates an image stabilization transformation matrix of the I frame original image according to the target pose of the camera corresponding to the N frame original image on the motion information of the camera, wherein the I frame original image is the first original image, the image stabilization transformation matrix of the I frame original image is used for obtaining the I frame target image, and the initial frame of the I frame original image corresponds to the (N1 + 1) th frame original image in the N frame original image.

That is to say, the electronic device can calculate the camera pose and the image stabilization transformation matrix corresponding to a certain original image according to the original image before and after the original image, so that the change between the camera poses corresponding to different original images is smoother, and the image stabilization effect is improved.

In another possible design, when the target image is a preview image or a captured image, N2 is 0.

Therefore, on the preview interface and the shooting interface, the electronic equipment does not need to calculate the camera pose and the image stabilization transformation matrix corresponding to a certain original image according to the original image behind the original image, so that the preview image and the recorded image corresponding to the original image can be processed and displayed in real time.

In another possible design, the method may further include: and if the preset condition is not met, the electronic equipment obtains the motion information of the camera according to the attitude sensor data corresponding to the multiple frames of original images. The electronic equipment calculates a stable image transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in a plurality of frames of original images; and carrying out image transformation on the first original image according to the image stabilization transformation matrix to obtain a target image.

In this scheme, if the subject anti-shake condition is not satisfied, the electronic device may obtain motion information of the camera from the posture sensor data without obtaining the motion information of the camera in combination with the motion information of the subject, thereby eliminating the need for subject anti-shake processing.

In another possible design, the method may further include: and if the preset condition is met, the electronic equipment prompts the user that the target anti-shake mode is started. And if the preset condition is not met, the electronic equipment prompts the user that the target anti-shake mode is not started.

In this way, the user can conveniently know whether to process the target anti-shake mode currently, and the target anti-shake mode can be a main body anti-shake mode.

In another possible design, before the electronic device calculates the image stabilization transformation matrix of the first original image according to the motion information of the camera, the method may further include: the electronic device calculates a Rolling Shutter (RS) compensation amount of the first raw image from data of the attitude sensor. The electronic device calculates an image stabilization transformation matrix of the first original image according to the motion information of the camera, and may include: the electronic device calculates a rotation compensation amount of the first original image according to the motion information of the camera. And the electronic equipment calculates the image stabilization transformation matrix of the first original image according to the rotation compensation quantity and the RS compensation quantity.

The RS compensation amount is used for RS correction, and the scheme can perform RS correction while anti-shake processing is performed according to the motion information of the camera so as to effectively correct the jelly effect of the background on the original image.

In another possible design, the electronic device may calculate an image stabilization transformation matrix of the first original image according to the rotation compensation amount and the RS compensation amount, and may include: if the electronic device determines to enter the panning mode, the electronic device decreases the RS compensation amount according to the panning amplitude. The panning amplitude is inversely related to the size of the angle between the principal axis of motion of the camera and the target coordinate axis. The electronic device calculates an image stabilization transformation matrix of the first original image according to the rotation compensation amount and the reduced RS compensation amount.

Because the stability of the background and the main body on the original image conflict with each other, the electronic device can adjust the RS compensation amount according to the panning amplitude when the RS correction is performed on the background, so that the distortion of the face of the main body is reduced or avoided as much as possible.

In another possible design, the subject is a human face.

In the scheme, the electronic equipment can obtain the motion information of the camera by combining the motion information of the main body face and the gesture sensor data, so that anti-shake processing is performed according to the motion information of the camera, anti-shake is realized on the main body face and the background, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved.

In another aspect, an embodiment of the present application provides a shooting device, which is included in an electronic device. The device has the function of realizing the behavior of the electronic equipment in any one of the above aspects and possible designs, so that the electronic equipment executes the video recording method executed by the electronic equipment in any one of the possible designs of the above aspects. The function can be realized by hardware, and can also be realized by executing corresponding software by hardware. The hardware or software includes at least one module or unit corresponding to the above functions. For example, the apparatus may comprise an acquisition unit, a processing unit, and the like.

In another aspect, an embodiment of the present application provides an electronic device, including: a camera for capturing an image; a screen for displaying an interface, one or more processors; and a memory having code stored therein. When executed by an electronic device, cause the electronic device to perform the video recording method performed by the electronic device in any of the possible designs of the above aspects.

In another aspect, an embodiment of the present application provides an electronic device, including: one or more processors; and a memory having code stored therein. When executed by an electronic device, cause the electronic device to perform the video recording method performed by the electronic device in any of the possible designs of the above aspects.

In yet another aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions, when the computer instructions are executed on an electronic device, cause the electronic device to perform the video recording method in any one of the possible designs of the foregoing aspect.

In yet another aspect, the present application provides a computer program product, which when run on a computer, causes the computer to execute the video recording method executed by an electronic device in any one of the possible designs of the above aspect.

In another aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a line; the interface circuit is used for receiving signals from a memory of the electronic equipment and sending the signals to the processor, and the signals comprise computer instructions stored in the memory; the computer instructions, when executed by the processor, cause the electronic device to perform the video recording method of any of the possible designs of the above aspects.

For the advantageous effects of the other aspects, reference may be made to the description of the advantageous effects of the method aspects, which is not repeated herein.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure;

fig. 2 is a flowchart of a video recording method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a set of interfaces provided by an embodiment of the present application;

fig. 4 is a schematic view of a face feature point according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram illustrating an effect of a constraint provided in an embodiment of the present application;

fig. 6A is a flowchart of adjusting human face constraint strength according to an embodiment of the present application;

fig. 6B is a flowchart of a face anti-shake processing provided in the embodiment of the present application;

FIG. 7A is a flowchart illustrating a fourth condition being satisfied according to an embodiment of the present disclosure;

FIG. 7B is a flowchart of a method for reducing the amount of RS compensation according to an embodiment of the present disclosure;

fig. 8A is a schematic view of a panning scene provided in an embodiment of the present application;

fig. 8B is a schematic timing relationship diagram of an original image and a preview image according to an embodiment of the present disclosure;

FIG. 9A is a schematic diagram of a set of preview interfaces provided by an embodiment of the present application;

FIG. 9B is a schematic diagram of another set of preview interfaces provided by an embodiment of the present application;

FIG. 9C is a schematic diagram of another set of preview interfaces provided by an embodiment of the present application;

FIG. 9D is a schematic diagram of another set of preview interfaces provided by an embodiment of the present application;

FIG. 10A is a schematic diagram of a set of capture interfaces provided in an embodiment of the present application;

fig. 10B is a schematic view of another set of shooting interfaces provided in the present embodiment of the application;

fig. 10C is a schematic view of another set of shooting interfaces provided in the embodiment of the present application;

fig. 10D is a schematic view of another group of shooting interfaces provided in the embodiment of the present application;

FIG. 11A is a schematic diagram of a set of video images provided by an embodiment of the present application;

FIG. 11B is a schematic diagram of another set of video images according to an embodiment of the present application;

FIG. 11C is a schematic diagram of another set of video images provided by an embodiment of the present application;

FIG. 11D is a schematic diagram of another set of video images provided by an embodiment of the present application;

fig. 12 is a schematic diagram of a prompt interface provided in an embodiment of the present application;

FIG. 13 is a flowchart of another video recording method according to an embodiment of the present application;

FIG. 14 is a flowchart of another video recording method according to an embodiment of the present application;

FIG. 15 is a flowchart of another video recording method according to an embodiment of the present application;

FIG. 16 is a schematic view of another interface provided by an embodiment of the present application;

fig. 17 is a flowchart of another video recording method according to the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present embodiment, "a plurality" means two or more unless otherwise specified.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to mean serving as examples, illustrations or descriptions. Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion.

When the electronic device is used for recording videos, the photographed images are easy to shake and blur due to the hand shake of a user, the shaking of the electronic device, the movement of a photographed object and the like. Particularly, when the shot moving body is closer to the camera of the electronic device, the relative movement of the moving body is larger, the shaking of the body image is more obvious, and the user experience is poorer. For example, in a video scene of a front-end self-timer, the face is closer to the camera of the electronic device, and the image of the face shakes more obviously when the face moves. In a technical scheme, electronic equipment carries out anti-shake according to gyroscope data to the image and handles, can realize better anti-shake effect to the background of shooing within range like this, but is difficult to realize the anti-shake to the motion main part of shooing within range, and the user shoots and experiences relatively poorly.

The embodiment of the application provides a video recording method and video recording equipment, which can be applied to electronic equipment and can be used for carrying out anti-shake processing by combining multi-source information such as motion information, attitude sensor data and the like of a shot main body under a video recording scene, so that anti-shake is realized on the motion main body and a background at the same time, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved.

The video recording method provided by the embodiment of the application can be used for a rear video recording scene and can also be used for a front video recording scene, and is not limited. After the photographing function is started, the electronic device may calculate motion information of the camera in combination with the motion information of the photographed body and the posture sensor data, thereby performing image anti-shake processing according to the motion information of the camera.

For example, the electronic device may be a mobile phone, a tablet computer, a wearable device (e.g., a smart watch), an in-vehicle device, an Augmented Reality (AR)/Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), or other mobile terminals, or may be a professional camera or other devices.

Fig. 1 shows a schematic structural diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a Subscriber Identification Module (SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It is to be understood that the illustrated structure of the embodiment of the present application does not specifically limit the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

The controller may be, among other things, a neural center and a command center of the electronic device 100. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution.

A memory may also be provided in processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to reuse the instruction or data, it can be called directly from memory. Avoiding repeated accesses reduces the latency of the processor 110, thereby increasing the efficiency of the system.

The electronic device 100 implements display functions via the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, video, and the like. The display screen 194 includes a display panel. The display panel may adopt a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-oeld, a quantum dot light-emitting diode (QLED), and the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, with N being a positive integer greater than 1. In an embodiment of the present application, the display screen 194 may be used to display a preview interface, a shooting interface, and the like in a video recording mode.

The electronic device 100 may implement a shooting function through the ISP, the camera 193, the video codec, the GPU, the display 194, the application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when a photo is taken, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converting into an image visible to naked eyes. The ISP can also carry out algorithm optimization on the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, the electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

Wherein, camera 193 can include the camera of different focal length, for example ultra wide-angle camera, wide-angle camera and tele camera that equivalent focal length changes from small to big etc.. The camera with the smaller equivalent focal length has a larger visual field range and can be used for shooting larger pictures such as landscapes and the like. The camera with the larger equivalent focal length has a smaller visual field range, can be used for shooting objects at a far position, and has a smaller shooting area.

In addition, the camera 193 may further include a depth camera for measuring an object distance of an object to be photographed, and other cameras. For example, the depth camera may include a three-dimensional (3D) depth-sensing camera, a time of flight (TOF) depth camera, a binocular depth camera, or the like.

The digital signal processor is used for processing digital signals, and can process digital image signals and other digital signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to perform fourier transform or the like on the frequency bin energy.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, and the like.

The NPU is a neural-network (NN) computing processor that processes input information quickly by using a biological neural network structure, for example, by using a transfer mode between neurons of a human brain, and can also learn by itself continuously. Applications such as intelligent recognition of the electronic device 100 can be realized through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, pattern recognition, machine self-learning, and the like.

The internal memory 121 may be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The storage data area may store data (such as audio data, phone book, etc.) created during use of the electronic device 100, and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like.

In the embodiment of the present application, the processor 110 may obtain the motion information of the camera by combining the motion information of the photographed main body and the attitude sensor in a video recording scene by operating the instruction stored in the internal memory 121, so as to perform anti-shake processing according to the motion information of the camera, so as to simultaneously realize anti-shake for the moving main body and the background, improve the image stabilization effect of the video image, and improve the user shooting experience.

The gyro sensor 180B may be used to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of electronic device 100 about three axes (i.e., x, y, and z axes) may be determined by gyroscope sensor 180B, i.e., motion information, such as rotation information, of electronic device 100 may be obtained. The gyro sensor 180B may be used for photographing anti-shake. For example, when the shutter is pressed, the gyro sensor 180B detects a shake angle of the electronic device 100, calculates a distance (i.e., a compensation amount) to be compensated for the lens module according to the shake angle, and allows the lens to counteract the shake of the electronic device 100 through a reverse movement, thereby achieving anti-shake. The gyroscope sensor 180B may also be used for navigation, somatosensory gaming scenes. The rotation information of the electronic device 100 may indicate rotation information of the camera 193 or rotation information of the camera module. Wherein the camera module comprises a camera 193. The camera module is hereinafter referred to simply as a camera.

The acceleration sensor 180E may detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. The method can also be used for recognizing the posture of the electronic equipment, and is applied to horizontal and vertical screen switching, pedometers and other applications.

A distance sensor 180F for measuring a distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, taking a picture of a scene, electronic device 100 may utilize range sensor 180F to range for fast focus.

The touch sensor 180K is also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is used to detect a touch operation applied thereto or nearby. The touch sensor can communicate the detected touch operation to the application processor to determine the touch event type. Visual output associated with the touch operation may be provided through the display screen 194. In other embodiments, the touch sensor 180K may be disposed on a surface of the electronic device 100, different from the position of the display screen 194.

In the embodiment of the present application, in a video recording scene, the camera 193 may be used to acquire an image; the display screen 194 may be used to display a preview interface, a capture interface, and the like; attitude sensor data such as a gyro sensor 180B (also referred to as a gyroscope) can be used to obtain rotation information of the camera; the processor 110 may obtain motion information of the photographed body by operating an instruction stored in the internal memory 121, thereby obtaining motion information of the camera by combining the motion information of the body and the attitude sensor data, and performing anti-shake processing according to the motion information of the camera, so as to simultaneously realize anti-shake for the moving body and the background, improve an image stabilization effect of a video image, and improve a user's photographing experience.

It is understood that the attitude sensor is used to detect the attitude of the electronic device 100 to obtain the rotation information of the electronic device 100. For example, the attitude sensor may be a gyroscope, a three-axis electronic compass, or the like, and the type of the attitude sensor is not limited in the embodiments of the present application.

The video recording method provided by the embodiment of the present application will be explained below by taking an example in which the electronic device is a mobile phone having a structure shown in fig. 1, the attitude sensor is a gyroscope, and the video recording scene is a front-end self-timer scene. As shown in fig. 2, the video recording method may include:

201. the mobile phone starts a shooting function and then enters a video recording mode, and original images are collected according to a preset frame rate in a preview state.

When a user wants to use the mobile phone to shoot an image, the shooting function of the mobile phone can be started. For example, the mobile phone may start a camera application, or start other applications with a photographing or video recording function (such as an AR application like a tremble or a river view cyberverse), so as to start the photographing function of the mobile phone.

In some embodiments, the video recording method provided by the embodiment of the present application may be applied to a video recording mode, and after the mobile phone starts the shooting function, the mobile phone may enter the video recording mode, so as to perform anti-shake processing in combination with the motion information of the shot subject and the gyroscope data in the video recording mode.

For example, after detecting an operation of clicking the camera icon 301 shown in (a) of fig. 3 by the user, the mobile phone starts a shooting function of the camera application, and displays a preview interface in a shooting mode as shown in (b) of fig. 3. After detecting that the user clicks the control 302 shown in (b) in fig. 3, the mobile phone enters the video recording mode as shown in (c) in fig. 3.

As another example, the mobile phone displays an interface of a desktop or a non-camera application, and after detecting a voice instruction of the user indicating to record a video, starts a shooting function, and enters a video recording mode as shown in (c) of fig. 3.

It should be noted that the mobile phone may also enter the video recording mode in response to other operations such as a user's touch operation, a voice instruction, or a shortcut gesture, and the operation of triggering the mobile phone to enter the video recording mode is not limited in the embodiment of the present application.

In other embodiments, in the video mode, the mobile phone does not automatically combine the motion information of the main body and the gyroscope data to perform anti-shake processing; after the mobile phone detects the preset operation 1 of the user, the mobile phone starts to perform anti-shake processing by combining the motion information of the main body and the gyroscope data. The preset operation 1 is used to instruct the mobile phone to perform anti-shake processing on the main body. Illustratively, in the video recording mode, the preview interface includes a main body anti-shake control, and after the mobile phone detects that the user clicks the main body anti-shake control, the mobile phone starts a main body anti-shake flow, so as to perform anti-shake processing by combining motion information of the main body and gyroscope data.

After the mobile phone enters a video recording mode, an original image is collected according to a preset frame rate in a preview state.

In other embodiments, the video recording method provided by the embodiment of the present application is applied to a specific target shooting mode other than the video recording mode, and the mobile phone may enter the target shooting mode after starting the shooting function, so as to perform anti-shake processing in combination with the motion information of the main body and the gyroscope data in the target shooting mode.

In the following embodiments, an example will be described in which a mobile phone performs anti-shake processing in a video recording mode by combining motion information of a main body and gyro data.

202. The mobile phone determines whether a first condition is met according to the S frames of original images, wherein S is an integer larger than 1. If the first condition is satisfied, the mobile phone performs step 203 and step 204; if the first condition is not satisfied, the handset performs step 205.

In the preview state of the video recording mode, after the mobile phone collects the S-frame original image, whether the first condition is satisfied can be determined according to the original image 1-S (i.e. according to the S-frame original image). For example, S is 2 frames, 20 frames, 45 frames, or the like. If the first condition is met, it can be determined that the mobile phone is moving, the ratio of the shot main body is large, the main body continuously appears on the original image and needs to be subjected to main body anti-shake processing, so that the mobile phone can enter a main body anti-shake mode to obtain the motion information of the main body and perform anti-shake processing on the main body by combining the motion information of the main body. If the first condition is not met, the main body anti-shaking processing is not needed, the mobile phone needs to enter a main body anti-shaking mode, and the mobile phone only carries out anti-shaking processing on the background. This first condition may also be referred to as a subject anti-shake condition.

In the embodiment of the present application, the subject to be photographed is an object or subject in which a user is interested in the photographing range. Other objects than the subject within the shooting range may be referred to as a background. Correspondingly, the image acquired by the mobile phone can also comprise a main body and a background.

For example, in some embodiments, the subject is a preset type of object. For example, the preset type of object is a human face or an animal.

In other embodiments, the subject is an object whose image on the preview image is located in the center region. The subject of interest to the user will typically be facing the camera and the image of the subject on the preview image will therefore typically be located in the central region.

In other embodiments, the subject is an object whose image on the preview image is close to the central region and whose area is larger than the preset threshold 1. The subject of interest to the user will typically be facing the camera and closer to the camera, so that the image of the subject on the preview image is close to the central region and has an area greater than the preset threshold 1.

In other embodiments, the subject is a preset type of object whose image on the preview image is near the center region.

In other embodiments, the subject is a preset type of object whose image on the preview image is near the center region and whose area is greater than a preset threshold 1.

In other embodiments, the subject is a preset type of object with the smallest depth near the center region of the image on the preview image.

In other embodiments, the subject is an object designated by a user through a preset touch operation, a preset gesture operation, or a preset voice instruction.

Wherein the body may be one object. Alternatively, the body may be an integral body formed of a plurality of objects held relatively stationary.

In the following embodiments of the present application, a main body is mainly taken as a face as an example for explanation.

When a plurality of faces appear in the shooting range, the main body can be a face closest to the camera, or a face with the largest area, or a face closest to the middle position on the image picture, or a face designated by the user, and the like.

For example, when the subject is a human face, the first condition may include any one of the following sub-conditions 1 to 3, or a combination of any two of the sub-conditions.

Wherein, sub-condition 1: the motion amplitude of the mobile phone is greater than or equal to a preset value 1. When the mobile phone determines that the motion amplitude of the mobile phone is larger through motion sensors such as an accelerometer or a gyroscope, the anti-shaking processing of the human face of the main body is carried out; therefore, the problem that the background moves along with the face of the main body due to anti-shake processing of the face of the main body when the mobile phone is static is avoided.

Sub-condition 2: the area ratio of the main face on the original image is greater than or equal to a preset value 2. Under the condition that the face occupation ratio of the main body is smaller than the preset value 2, the face area of the main body is smaller, and a user probably wants a better background effect, so that the first condition is not met, and the anti-shake of the face of the main body is not needed. When the ratio of the main face is greater than or equal to the preset value 2, the user may want to mainly shoot the main face, and the anti-shake processing of the main face may be performed.

Sub-condition 3: the number of frames of the original images in which the face of the subject continuously appears is greater than or equal to a preset value 3, and the preset value 3 is an integer greater than 1. For example, the preset value 3 is L, the subject face continuously appears on the original images of consecutive L frames, and the duration of the continuous appearance of the subject face on the original images is greater than or equal to the preset duration. That is to say, the mobile phone performs the anti-shaking processing of the main face only after the main face appears within a continuous period of time, so as to prevent the related problems caused by the frequent start or quit of the anti-shaking processing of the main face when the main face frequently enters or exits the shot picture.

If the sub-conditions 1-3 are met, the mobile phone determines that the first condition is met, which can indicate that the mobile phone moves, the area ratio of the main face is large, and the main face continuously appears on the original image, so that anti-shake processing of the main face is required. If any one of the sub-condition 1, the sub-condition 2 or the sub-condition 3 is not satisfied, the mobile phone determines that the first condition is not satisfied, so that whether other sub-conditions in the first condition are satisfied is no longer determined, and the mobile phone determines that the anti-shake processing of the face of the subject is not required.

In some embodiments, the mobile phone may determine whether sub-condition 1 is satisfied according to the S-frame original image, and if sub-condition 1 is satisfied, then determine whether sub-condition 2 and sub-condition 3 are satisfied to further determine whether the first condition is satisfied; if the sub-condition 1 is not satisfied, the mobile phone determines that the first condition is not satisfied.

In a preview state, if the frame number of the original image acquired by the mobile phone is greater than the S frame, the mobile phone determines whether the first condition is met according to the most recently acquired S frame original image. That is, after acquiring the S frame image, the mobile phone determines whether the first condition is satisfied according to the historical original image of the latest S frame every time the mobile phone acquires an original image of one frame. If the first condition is met and the main body anti-shake mode is entered currently, the mobile phone continues to perform anti-shake processing in the main body anti-shake mode. If the first condition is met and the mobile phone does not enter the main body anti-shake mode currently, the mobile phone enters the main body anti-shake mode. If the first condition is not met and the anti-shake mode is currently entered, the mobile phone exits the main body anti-shake mode.

In some embodiments, after the mobile phone enters/exits the main body anti-shake mode, the user may be prompted by displaying information, voice broadcasting, vibration, or the like.

203. If the first condition is met, the mobile phone acquires the motion information of the main face according to M frames of original images, wherein M is an integer larger than 1.

If the first condition is satisfied, it indicates that the anti-shake processing of the main face is required, so that the mobile phone can start the face anti-shake mode, and then execute step 203 and step 204, so as to perform the anti-shake processing by combining the motion information of the main face and the gyroscope data.

In other embodiments, if the mobile phone detects that the user starts the face anti-shake mode, it indicates that the anti-shake processing of the main face is required, so that the mobile phone can start the face anti-shake mode, and then execute step 203 and step 204, so as to perform the anti-shake processing by combining the motion information of the main face and the gyroscope data. For example, the preview interface includes a face anti-shake control, and the mobile phone starts a face anti-shake mode after detecting an operation of clicking the control by the user.

The motion information of the main face is used for representing the motion situation of the main face. For example, the motion information of the main face may be a target motion curve of the main face, and the mobile phone acquires the target motion curve of the main face according to the M frames of original images after meeting the first condition, so as to perform anti-shake processing by combining the target motion curve of the main face and gyroscope data.

The following description will take an example in which the mobile phone acquires a target motion curve of a subject face from an M-frame original image.

And the mobile phone detects the human face characteristic points of the M frames of original images according to a human face characteristic detection algorithm. Illustratively, referring to fig. 4, the face includes 71 feature points in total. The mobile phone takes the coordinate mean value of each feature point as the center of the face. The face center of the M frames of original images constitutes an original motion curve of the subject face, which may also be referred to as an original motion trajectory of the subject face or an original motion path of the subject face.

The mobile phone carries out smoothing processing on the original motion curve of the main face, so that a smooth target motion curve of the main face is obtained. The target motion curve of the subject face satisfies the following second condition:

(1) the target motion curve of the subject face is continuous everywhere. That is, the motion trajectory of the face center is smooth and the change cannot be too drastic.

(2) The target motion curve of the subject face is first-order and second-order conductive, and the curvature is as small as possible (e.g., less than or equal to a preset threshold 2). That is, the change of the speed and the acceleration of the target motion curve of the subject face is smooth and cannot be too severe.

(3) The distance between the face center on the target motion curve of the subject face and the face center on the original motion curve of the subject face is less than or equal to a preset value 4. That is, the face center on the target motion curve of the subject face cannot deviate too far from the original motion curve of the subject face, and the face center on the target motion curve of the subject face should be in the vicinity of the original motion curve of the subject face.

The mobile phone can adopt various methods to carry out smoothing processing on the original motion curve of the main face so as to obtain the target motion curve of the main face. For example, the mobile phone may obtain a smooth target motion curve of the subject face through a quadratic programming method.

The motion of the subject face represented on the original images of multiple frames can be represented by 2 groups of equally-spaced face coordinate sequences represented by formula 1:

assuming that the face center coordinate sequences are all on one curve, the mobile phone may determine a smooth target motion curve of the subject face satisfying the second condition by a quadratic programming method according to the target function 1 represented by formula 2. C 'is passed to a point on a target motion curve of the subject face'_kAnd (4) sequence representation.

Wherein r ═ r_u,r_v]^T∈R^2×1,c_k＝[u_k,v_k]^T,c'_k＝[u'_k,v'_k]^T。

In formula 2, s.t. represents "so as to satisfy", C'_kThe coordinates of the center of the face on the k frame of the original image in the target motion curve of the subject face after smoothing are expected to be shown. C_kAnd representing the coordinates of the face center on the k frame original image on the original motion curve of the subject face before smoothing. Omega₁，ω₂Is a preset hyper-parameter. r represents a difference threshold of the face center before and after smoothing, and ru and rv represent a line coordinate threshold and a vertical coordinate threshold of the difference, respectively.

Formula 2 can also be represented as formula 3:

minω₁J₁+ω₂J₂formula 3

Wherein, J₁The first derivative of the object motion curve representing the subject face,

J₂the second derivative of the object motion curve representing the subject face,

in step 203, the mobile phone obtains the target motion curve of the main face according to the M frames of original images after meeting the first condition, so as to perform anti-shake processing by combining the target motion curve of the main face and the gyroscope data. For example, after the mobile phone acquires the target motion curve of the main face according to the M frames of original images for the first time, each acquired frame of original image can be combined with the M frames of original images acquired recently to continuously acquire subsequent track points on the target motion curve of the main face.

204. The mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N frames of original images and the motion information of the main face, wherein N is an integer larger than 1.

Wherein the motion information of the camera is used for representing the motion situation of the camera. Because the camera module comprises the camera, the motion condition of the camera is the motion condition of the camera, and the camera module is usually fixed on the mobile phone and moves along with the mobile phone, so the motion condition of the camera is the motion condition of the mobile phone. For example, the motion information of the camera is a target motion curve of the camera, which may also be referred to as a target motion trajectory or path of the camera, that is, a target motion curve of the mobile phone or a target motion trajectory or path of the mobile phone. The motion information of the camera is used for calculating an image stabilization transformation matrix of the original image subsequently, and the image stabilization transformation matrix is used for performing motion compensation on the original image through deformation (warp) transformation, so that the anti-shake and image stabilization effects of the image are achieved.

The following description will take the motion information of the camera as the target motion curve of the camera, and the mobile phone acquires the target motion curve of the camera as an example.

And after meeting the first condition, the mobile phone obtains an original motion curve of the camera according to the gyroscope data corresponding to the N frames of original images. And the gyroscope data corresponding to the N frames of original images is the gyroscope data corresponding to the N frames of original images in the acquisition time period. And the mobile phone processes the original motion curve of the camera according to the first constraint condition so as to obtain a target motion curve of the camera. Wherein the first constraint condition is associated with motion information of the subject face. That is, the mobile phone obtains the target motion curve of the camera according to the gyroscope data and the motion information of the face of the subject. The target motion curve of the camera satisfies the following third condition:

(1) the target motion curve of the camera is continuous everywhere. That is, the motion trajectory of the camera is smooth, and the trajectory change cannot be too drastic.

(2) The target motion curve of the camera is first, second, and third order conductive, and the curvature is as small as possible (e.g., less than or equal to a preset threshold 3). That is, the change in the velocity, acceleration, and rate of change in acceleration (i.e., jerk) of the target motion profile of the camera is smooth and cannot be too abrupt.

(3) And calculating an image stabilization transformation matrix according to a target motion curve of the camera, so that when the motion compensation is performed on the original image according to the image stabilization transformation matrix, the compensated original image cannot exceed the boundary of the cropping (after crop) to generate a black edge.

(4) When motion compensation is carried out on an original image according to a target motion curve of a camera, the distance between the center of a face subjected to warp conversion on the original image and the center of the face on the target motion curve of a main face is smaller than or equal to a preset value 5. That is, the face center on the target motion curve of the subject face cannot deviate too far from the original motion curve of the subject face, and the face center on the target motion curve of the subject face should be in the vicinity of the original motion curve of the subject face.

The mobile phone can adopt various methods to carry out smoothing processing on the original motion curve of the camera to obtain the target motion curve of the camera. For example, the mobile phone may obtain a smooth target motion curve of the camera through a quadratic programming method.

Wherein, the point on the original motion curve of the camera represents the original pose of the camera, which can be represented by using 3 groups of rotation angle sequences with equal time intervals represented by formula 4:

wherein the content of the first and second substances,

representing the rotation angles in three directions integrated from the gyroscope, respectively. The mobile phone can determine a smooth target motion curve of the camera satisfying the third condition according to the target function 2 represented by formula 5 and the first constraint condition represented by formula 6 by a quadratic programming method.

min w₁J₁+w₂J₂+w₃J₃Formula 5

Similar to formula 3 above, J_iRepresenting the ith derivative of the target motion curve of the camera, as shown in equation 6, the first constraint may include:

after the first condition is met, the mobile phone processes the acquired N frames of original images according to the original poses of the cameras corresponding to the original images by adopting the algorithm provided by the embodiment of the application for the first time, and obtains the optimized target poses of the cameras corresponding to the I frames of original images (I is a positive integer), so that the transition between the target poses of the cameras corresponding to the I frames of original images is smooth. The optimized target pose of the camera forms a target motion curve of the camera, and the optimized target pose of the camera is a point on the target motion curve of the camera. And then, after acquiring the subsequent I frame original image, the mobile phone processes the acquired I frame original image according to the original pose of the camera corresponding to the recently acquired historical N frame original image, and obtains the optimized target pose of the camera corresponding to the I frame original image.

Wherein N is N1+ I + N2, N1 and I are positive integers, and N2 is a non-negative integer. According to the original pose of the camera corresponding to the N frames of original images, the optimized target pose of the camera corresponding to the I frames of original images behind the N1 frames of original images is output by the algorithm. For example, N is 45, N1 is 15, N2 is 27, and I is 3, the algorithm processes the original poses of the cameras corresponding to the 1 st to 45 th frames of original images for the first time, and outputs the optimized target poses of the cameras corresponding to the I frames of original images of 16 th, 17 th and 18 th frames of original images. That is to say, the mobile phone obtains the target pose of the optimized camera corresponding to the I frame original image according to the original poses of the cameras corresponding to the I frame original image, the previous N1 frame original image and the next N2 frame original image.

The sub-constraint condition (1) in equation 6 is used to take the target poses of the cameras corresponding to the optimized partial original images as input to optimize the target poses of the cameras corresponding to the adjacent subsequent frames of original images, so that the overall transition of the target poses of the optimized cameras is smoother, rather than only the local transition between the target poses of the cameras corresponding to the I-frame original images. For example, N is 45, N1 is 15, N2 is 27, I is 3, the algorithm processes the target poses of the optimized cameras corresponding to the 4 th to 48 th frames of original images for the second time, and the target poses of the optimized cameras corresponding to the 19 th, 20 th and 21 th frames of original images are output. In order to ensure that the target poses of the optimized cameras corresponding to the 18 th and 19 th frames of original images do not jump and the transition is smooth, the mobile phone can replace the original positions of the cameras corresponding to the 17 th and 18 th frames of original images used in the second optimization with the target poses of the optimized cameras corresponding to the 17 th and 18 th frames of original images after the first optimization based on the sub-constraint condition (1).

The sub-constraint condition (2) in equation 6 is used to make the original image not exceed the preset clipping boundary after passing through warp transformation based on the target motion curve of the camera. Illustratively, referring to fig. 5, a block 501 indicates a range of an original image, a block 502 indicates a clipping-reserved range in anti-shake processing, and preset clippingThe boundary may be P as shown in FIG. 5_wAnd P_hA defined boundary. For example, in FIG. 5, P_iRepresenting pixel points, P, in the original image before warp transformation_i' denotes a pixel point subjected to warp conversion according to motion information of the camera. Four corners of the edge of the original image, namely four vertices of the box 501, cannot exceed P after warp transformation_wAnd P_hThe boundary being defined, i.e. not being at P_wAnd P_hThe cropping represented by box 502 outside the defined boundary remains within the range so that the image resulting from the cropping does not leave a black border. As another example, the center P of the face on the original image_cP after warp conversion_c' must not exceed the boundary P_wAnd P_hI.e. the cropping reservation represented by block 503 cannot be exceeded, so that the image resulting from the cropping does not leave a black border.

In the sub-constraint (3) in equation 6, the function [ < i >, < i > x >_f，w，w_f) Coordinate X representing the center of a face on an original image_fCalculating the pose of the camera obtained according to the objective function 2 and the sub-constraint conditions (1) - (2) and a transformation matrix w to obtain the projection position of the center of the face transformed by warp; x is the number of_hRepresenting the ideal position of the face center on the target motion curve of the smoothed subject face; epsilon represents the coordinate offset threshold between the projected position of the face center and the ideal position. That is to say, the sub-constraint condition (3) in equation 6 is used to make the coordinate offset between the projection position of the face center and the ideal position smaller than epsilon, so that the position of the face on the adjacent original image is relatively stable, the face shake is relatively small, and the image stabilizing effect is relatively good. Epsilon determines the stability of the human face in the image picture after image stabilization.

For example, the center point P of block 504₀Is the face center on the smoothed subject face target curve. For the center P of the face on the original image_cFace center P obtained after warp conversion_c' falls within the scope of block 504 defined by ε in sub-constraint (3).

As can be seen from the sub-constraint (3), since the target motion curve of the camera is obtained based on the target motion curve of the subject face, the original image corresponding to the point on the target motion curve of the subject face should include the point on the target motion curve of the camera. Thus, in some embodiments, M may be greater than or equal to N.

After the mobile phone obtains the target motion curve of the camera according to the N frames of original images for the first time, each subsequent I frame of original images can be combined with the N frames of original images which are collected recently to obtain subsequent I points on the target motion curve of the camera.

After the mobile phone determines the motion information of the camera, the rotation compensation amount is determined according to the motion information of the camera, and therefore the image stabilization transformation matrix of the original image is calculated according to the rotation compensation amount. The image stabilization transformation matrix is a homography matrix of the original image subjected to deformation (warp) transformation. The image stabilization transformation matrix is used for performing motion compensation on the original image through warp transformation, so that the anti-shake and image stabilization effects of the image are achieved. For example, the mobile phone may perform warp transformation processing on the original image according to the image stabilization transformation matrix to obtain a preview image, and display the preview image on a preview interface. Therefore, when the mobile phone performs warp conversion according to the motion information of the camera, the motion information of the main face is combined with the motion information of the camera, so that the main face can be prevented from shaking when the background is prevented from shaking, the image stabilizing effect of the image is improved, and the shooting experience of a user is improved.

It is understood that the stable situation of the background and the subject on the original image is usually conflicting in the anti-shake process. If the anti-shake processing strength of the background is high, the anti-shake effect of the main body is poor; if the anti-shake processing strength of the main body is high, the anti-shake effect of the background is poor.

Therefore, in some embodiments, after the mobile phone starts the face anti-shake mode, the strength of the face constraint can be dynamically adjusted according to the motion amplitude of the main face, so as to balance the anti-shake effect of the background and the main face, and thus the image stabilization effect of the background and the main face is good. Moreover, when the strength of the face constraint is dynamically adjusted, the mobile phone can smoothly enhance and weaken the face constraint strength so as to avoid sudden change of the anti-shake strength of the background or the main face and sudden change of the image effect of the background and the main face, so that the anti-shake effect of the background and the main face is smoothly transited, and the shooting experience of a user is improved.

For example, in a possible implementation manner, referring to fig. 6A, after the face anti-shake mode is turned on, the process of adjusting the face constraint strength according to the face motion amplitude by the mobile phone may include:

204a, the mobile phone determines a corresponding face constraint gear 1 according to the motion amplitude of the main face, and each gear corresponds to an upper bound value of the face constraint intensity.

Wherein, the motion amplitude of the main face is measured by the variance of the center of the main face on the M frames of original images. The larger the variance, the larger the magnitude of the motion of the subject face. The larger the motion amplitude of the main human face is, the higher the corresponding human face constraint gear is, and the stronger the human face constraint is; the smaller the motion amplitude of the main human face is, the lower the corresponding human face constraint gear is, and the weaker the human face constraint is.

And the strength of the face constraint is controlled by epsilon in the sub-constraint condition (3). The larger the epsilon value is, the weaker the human face constraint strength is; the smaller the epsilon value, the stronger the face constraint strength.

In some embodiments, the upper bound values of the face constraint strengths for different gears may correspond to different values of ε.

204b, the mobile phone gradually increases the face constraint strength from the initial value to an upper bound value 1 of the gear 1. Then, the handset executes step 204c or step 204 d.

The initial value can be a preset numerical value, and the face constraint intensity corresponding to the initial value is small, so that after the mobile phone enters a face anti-shake mode, the face constraint intensity when face anti-shake processing is started is small, and image change before and after anti-shake is smooth.

204c, if the mobile phone determines that the motion amplitude of the main face is increased and corresponds to the face constraint gear 2, gradually increasing the face constraint strength from the upper bound value 1 corresponding to the gear 1 to the upper bound value 2 corresponding to the gear 2.

The face constraint strength corresponding to the gear 2 is greater than the face constraint strength corresponding to the gear 1, and the face constraint strength corresponding to the upper bound value 2 is greater than the face constraint strength corresponding to the upper bound value 1. Accordingly, the value of ε for upper bound value 2 is less than the value of ε for upper bound value 1.

That is to say, when the motion amplitude of the main face is increased (for example, the main object is changed from walking to running), the mobile phone can correspondingly enhance the face constraint intensity, so as to perform anti-shake processing with higher intensity on the main face with larger motion amplitude, and improve the image stabilization effect of the main face. Moreover, the mobile phone needs to gradually, slowly and smoothly enhance the strength of the face constraint, so that the image effect of the main face and the background after the anti-shake processing is performed is slowly changed, the sudden change of the anti-shake strength of the background or the main face is avoided, and the sudden change of the image effect of the background and the main face is avoided.

204d, if the mobile phone determines that the motion amplitude of the main face is reduced and corresponds to the face constraint gear 3, gradually weakening the face constraint strength from the upper bound value 1 corresponding to the gear 1 to the upper bound value 3 corresponding to the gear 2 by the mobile phone.

The face constraint intensity corresponding to the gear 3 is smaller than the face constraint intensity corresponding to the gear 2, and the face constraint intensity corresponding to the upper bound value 3 is smaller than the face constraint intensity corresponding to the upper bound value 2. Accordingly, the value of ε for upper bound value 3 is greater than the value of ε for upper bound value 2.

That is to say, when the motion amplitude of the main face is reduced, the mobile phone can correspondingly weaken the face constraint strength, so that the anti-shake processing with weaker strength is performed on the main face with smaller motion amplitude, and the image stabilizing effect of the background is also ensured while the image stabilizing effect of the main face is improved. Moreover, the mobile phone needs to gradually, slowly and smoothly weaken the strength of face constraint, so that the image effect of the main face and the background subjected to anti-shake processing is slowly changed, sudden change of the anti-shake strength of the background or the main face is avoided, and sudden change of the image effect of the background and the main face is avoided.

205. And if the first condition is not met, the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N frames of original images.

In step 205, if the first condition is not satisfied, the motion information of the camera is obtained according to the gyroscope data corresponding to the N frames of original images without combining the motion information of the face of the subject. For example, when the motion information of the camera is the target motion curve of the camera, the target motion curve of the camera is calculated according to the objective function 2 expressed by equation 5 and the second constraint condition. Wherein the second constraint condition does not include the above-described sub-constraint condition (3) associated with the motion information of the subject face, unlike the first constraint condition. For example, the second constraint may include the above-described sub-constraint (1) and sub-constraint (2). That is, the target motion curve of the camera is not obtained from the motion information of the subject's face. The motion information of the camera is used for calculating an image stabilization transformation matrix of the original image subsequently, and the image stabilization transformation matrix is used for performing motion compensation on the original image through warp transformation, so that the anti-shake and image stabilization effects of the image are achieved.

If the mobile phone determines that the first condition is not met, it may indicate that the anti-shake processing of the main face is not required, so that step 211 and step 214 may be performed to perform the anti-shake processing by using the motion information of the camera obtained according to the gyroscope data corresponding to the N frames of original images, instead of performing the anti-shake processing by combining the motion information of the main face.

It should be noted that, if the mobile phone has previously entered the face anti-shake mode, the mobile phone exits the face anti-shake mode after determining that the first condition is not met, so as to execute step 211 and 214, delete the face constraint item corresponding to the sub-constraint condition (3), and perform anti-shake processing by using the motion information of the camera obtained according to the gyroscope data.

That is to say, if the first condition is not satisfied, the subsequent mobile phone calculates an image stabilization transformation matrix of the original image according to the motion information of the camera, and the mobile phone performs warp transformation processing on the original image according to the image stabilization transformation matrix to obtain a preview image, and displays the preview image on a preview interface.

In the solution described in the above embodiment of the present application, a flowchart of the mobile phone performing the anti-shake adjustment on the face may be shown in fig. 6B. As shown in fig. 6B, when the mobile phone determines that the face anti-shake condition is satisfied according to multiple frames of original images, the mobile phone calculates the face motion amplitude; gradually increasing the face constraint to a corresponding constraint intensity upper bound according to the face motion amplitude; if the face motion amplitude changes, gradually increasing the face constraint to a corresponding new constraint intensity upper bound according to the face motion amplitude; and if the face anti-shake condition is not met, deleting the face constraint and exiting the face anti-shake mode.

It can be understood that the rolling shutter of the camera acquires an original image by line-by-line exposure, and when a photographed object moves at a high speed or the scanning speed of the camera is not fast enough, the object on the image may have phenomena of tilting, swinging, or partial exposure, which is called Rolling Shutter (RS) effect or jelly effect. In some embodiments, the mobile phone can also calculate an RS compensation amount to perform RS correction, so as to effectively correct the jelly effect of the background on the original image. Because the stability of the background and the main body on the original image conflict with each other, when the mobile phone performs RS correction on the background, the distortion of the human face of the main body is reduced or avoided as much as possible by adjusting the RS compensation amount. Based on this, after the above steps 204 and 205, the method may further include:

206. and the mobile phone calculates the RS compensation amount corresponding to the original image according to the data of the gyroscope.

Wherein, the RS compensation quantity is used for carrying out motion compensation on the original image during warp conversion so as to carry out RS correction. Also, the rolling shutter is exposed line by line, and thus the RS compensation amount includes compensation amounts corresponding to each exposure line, respectively.

The mobile phone can calculate the RS compensation amount corresponding to the original image according to the data of the gyroscope. For example, the RS compensation is not needed for the central line of the exposure line of the original image, i.e., the RS compensation amount corresponding to the central line is 0. The mobile phone can calculate the RS compensation amounts respectively corresponding to the other exposure rows according to the difference between the timestamps of the other exposure rows and the timestamp of the center row and the difference between the gyroscope data corresponding to the other exposure rows and the gyroscope data corresponding to the center row.

For another example, the mobile phone may calculate RS compensation amounts of the first line/last line of the original image according to data of the gyroscope, and calculate RS compensation amounts respectively corresponding to other exposure lines according to a difference between the time stamps of the other exposure lines and the time stamps of the first line/last line and a difference between gyroscope data corresponding to the other exposure lines and gyroscope data corresponding to the first line/last line.

207. The handset determines whether a fourth condition is satisfied. If the fourth condition is satisfied, the mobile phone performs step 208; if the fourth condition is not satisfied, the mobile phone performs step 211.

If the fourth condition is met, it can be shown that the face and the background of the main body have large relative motion, the mobile phone can enter a panning mode, so that the RS compensation amount is reduced, and an image stabilization transformation matrix of the original image is calculated by combining the reduced RS compensation amount and the motion information of the camera, so that warp transformation is performed.

208. If the fourth condition is met, the mobile phone determines a reduction factor according to the panning amplitude, and the reduction factor is used for reducing the RS compensation amount.

Wherein the reduction factor is used to reduce the amount of RS compensation per row. For example, the reduction factor may be a reduction ratio k, and the reduced RS compensation amount is equal to the original RS compensation amount (1-k). The raw RS compensation amount is the unreduced RS compensation amount obtained from the gyroscope data in step 206.

If the fourth condition is satisfied, it may indicate that the subject face and the background have a relatively large relative motion. At this time, if the RS compensation amount obtained according to the data of the gyroscope is used to perform warp transformation on the original image, distortion is likely to occur to the face of the subject, so that the mobile phone can enter a panning mode, and the RS compensation amount is reduced according to the panning amplitude, so as to minimize or avoid the distortion of the face of the subject due to RS correction.

In some embodiments, the panning mode refers to the camera and the subject remaining relatively still or moving at the same speed, and the body of the shooting user rotates to rotate the mobile phone and the camera, so that the background changes, and the face of the subject and the background have larger relative movement.

For example, referring to fig. 7A, the determination by the handset that the fourth condition is satisfied includes:

208a, the mobile phone determines a main shaft of the rotation motion of the camera according to the gyroscope data corresponding to the multi-frame original image.

The mobile phone calculates the angular velocity of the camera according to the gyroscope data corresponding to the period of collecting continuous multi-frame original images, integrates the angular velocity to obtain the rotation angles of three coordinate axes of x, y and z, and further obtains the main shaft of the rotation motion of the camera according to the rotation angles of the three coordinate axes of x, y and z.

208b, the mobile phone calculates an included angle between the main axis and the target coordinate axis.

The target coordinate axis is a preset coordinate axis and is related to the current horizontal/vertical screen state of the mobile phone. For example, if the mobile phone is in a vertical screen state at present, the target coordinate axis is the y axis in the same direction as the gravity direction; if the mobile phone is in the landscape mode, the target coordinate axis is the x axis in the horizontal direction perpendicular to the gravity direction. After the main axis of the camera rotation motion is determined, the mobile phone can calculate the included angle between the main axis and the target coordinate axis.

208c, if the included angle is smaller than or equal to the preset value 6, the mobile phone determines that the fourth condition is met.

If the included angle between the main shaft and the target coordinate axis is smaller than or equal to the preset value 6, it can be shown that the included angle between the main shaft and the target coordinate axis is smaller, the camera basically rotates around the target coordinate axis, the human face of the subject and the background have larger relative motion, and the fourth condition is met.

For example, the mobile phone may calculate a cosine value of an angle between the main axis and the target coordinate axis, and if the cosine value is greater than or equal to a preset value of 7, it may be indicated that the angle between the main axis and the target coordinate axis is small, the camera rotates around the target coordinate axis, and the face of the subject and the background have a large relative motion, which satisfies the fourth condition.

For example, when the mobile phone is in the portrait state, a schematic diagram of the mobile phone around the y-axis panning may be shown in fig. 8A.

And the mobile phone enters a panning mode after meeting the fourth condition, and determines a reduction factor according to the panning amplitude. Wherein the panning amplitude is used to represent the amplitude of the camera rotation about the target axis. For example, a smaller angle between the principal axis and the target axis or a larger rotational speed of the camera may indicate a larger amplitude of panning. The greater the amplitude of panning, the greater the reduction factor. In contrast, if the angle between the main axis and the target coordinate axis is larger or the rotational speed of the camera is smaller, it can be indicated that the panning amplitude is smaller and the reduction factor is smaller.

In the embodiment of the present application, when performing the face distortion correction, the processing flow of the mobile phone obtaining the reduced RS compensation amount may also refer to fig. 7B. As shown in fig. 7B, the mobile phone may cache gyroscope data corresponding to multiple frames of original images, calculate a motion main axis when a face appears in the original image, then calculate an included angle between the main axis and a target coordinate axis, calculate a reduction factor when the included angle is less than or equal to a preset value of 8, and reduce RS compensation amount according to the reduction factor.

209. And the mobile phone calculates an image stabilization transformation matrix of the original image according to the reduced RS compensation quantity and the motion information of the camera.

The mobile phone can calculate the rotation compensation amount according to the motion information of the camera, and the rotation compensation amount also comprises the compensation amount corresponding to each exposure line. The mobile phone can add the reduced RS compensation quantity and the rotation compensation quantity, thereby calculating the image stabilization transformation matrix of the original image.

For example, the motion information of the camera is a target motion curve of the camera, and the mobile phone can respectively calculate the rotation compensation amounts corresponding to the I-frame original images according to the target poses of the camera corresponding to the I-frame original images on the target motion curve of the camera. The mobile phone adds the rotation compensation quantity corresponding to each frame of original image in the I frame of original image and the reduced RS compensation quantity, thereby obtaining the image stabilization transformation matrix corresponding to each frame of original image in the I frame of original image.

For example, when N is 45, N1 is 15, I is 3, N2 is 27, the mobile phone processes the original poses of the cameras corresponding to the 1 st to 45 th frames of original images, outputs the target poses of the optimized cameras corresponding to the 16 th, 17 th and 18 th frames of original images, and calculates the rotation compensation amounts corresponding to the 16 th, 17 th and 18 th frames of original images according to the target poses of the optimized cameras corresponding to the 16 th, 17 th and 18 th frames of original images. The mobile phone adds the rotation compensation quantity corresponding to each frame of original image in the 16 th, 17 th and 18 th frames of original images and the reduced RS compensation quantity, thereby obtaining the image stabilization transformation matrix corresponding to each frame of original image in the 16 th, 17 th and 18 th frames of original images.

Then, the mobile phone processes the original poses of the cameras corresponding to the 4 th to 48 th frames of original images (wherein, the original poses corresponding to the 17 th and 18 th frames of original images are replaced by the target poses of the optimized cameras), outputs the target poses of the optimized cameras corresponding to the 19 th, 20 th and 21 th frames of original images, and calculates the rotation compensation quantities corresponding to the 19 th, 20 th and 21 st frames of original images according to the target poses of the optimized cameras corresponding to the 19 th, 20 th and 21 st frames of original images. The mobile phone adds the rotation compensation quantity corresponding to each frame of original image in the 19 th, 20 th and 21 st frames of original images with the reduced RS compensation quantity, thereby obtaining the image stabilization transformation matrix corresponding to each frame of original image in the 19 th, 20 th and 21 st frames of original images.

210. And the mobile phone performs image transformation on the original image according to the image stabilization transformation matrix to obtain a preview image, and displays the preview image on a preview interface.

The mobile phone performs warp conversion on the corresponding original image according to the image stabilization conversion matrix corresponding to each frame of original image in the I frame of original image to obtain a preview image, and displays the preview image on a preview interface.

For example, N is 45, N1 is 15, N2 is 27, and I is 3, the mobile phone performs warp transformation on the 16 th, 17 th, and 18 th frame original images according to the image stabilization transformation matrices corresponding to each frame original image in the 16 th, 17 th, and 18 th frame original images, so as to obtain 1 st, 2 nd, and 3 rd frame preview images, and sequentially displays the 1 st, 2 nd, and 3 rd frame preview images on the preview interface. The mobile phone performs warp conversion on the 19 th, 20 th and 21 st frame original images according to the image stabilization transformation matrixes respectively corresponding to each frame original image in the 19 th, 20 th and 21 st frame original images to obtain 4 th, 5 th and 6 th frame preview images, and sequentially displays the 4 th, 5 th and 6 th frame preview images on a preview interface. For example, when N is 45, N1 is 15, N2 is 27, and I is 3, the time sequence correspondence between the original image frame acquired by the mobile phone in the preview state and the displayed preview image can be seen in fig. 8B.

The difference value between the sequence number of the preview image frame and the sequence number of the original image frame displayed by the mobile phone is N1. That is, the mobile phone does not display the preview images of the 1 st to N1 th frames of original images after warp conversion on the preview interface, but starts to display the preview images of the N1+1 th frames of original images after warp conversion.

It can be understood that the mobile phone has a relatively high acquisition frame rate of the original image in the preview state, a certain time is required for switching the mobile phone from the other mode to the video recording mode, and the mobile phone does not display the preview image of the original image with 1 to N1 frames after warp conversion after switching the mobile phone to the video recording mode, so that the user does not have poor visual experience such as blocking or black screen.

And the mobile phone can obtain the target pose of the camera corresponding to the N1+1 frame original image according to the initial pose of the camera corresponding to the N1+ I + N2 frame original image, so that the image stabilizing transformation matrix corresponding to the N1+1 frame original image can be obtained according to the target pose of the camera corresponding to the N1+1 frame original image, and further the 1 st preview image corresponding to the N1+1 frame original image after warp transformation is performed is obtained. That is, the starting frame of the I-frame original image corresponds to the N1+1 th frame of the N-frame original images, and the preview image displayed by the mobile phone is delayed by at least N2 frames compared with the original image captured by the mobile phone.

In some other embodiments of the present application, in the process of acquiring the first N frames of original images by the mobile phone immediately after entering the preview state, each frame of original image in the first N1 frames of original images is displayed on the preview interface as a preview image. Subsequently, after the mobile phone acquires a new original image, a preview image corresponding to the I-frame original image is generated according to the N-frame original image by using the method described in the above embodiment, and the generated preview image is displayed on the preview interface.

In some other embodiments of the present application, in the preview state, the frame number of the preview image displayed on the preview interface by the mobile phone corresponds to the frame number of the original image, i.e. N2 is 0, and I is 1. That is, after acquiring a frame of original image, the mobile phone displays a preview image corresponding to the frame of original image. N1 may be small, for example, 5, 8, or 10. In the preview state, before the mobile phone acquires N1 frames of original images, the mobile phone generates and displays a preview image corresponding to the current original image in combination with the original images acquired in the preview state.

For example, just after entering the preview state, the mobile phone acquires the 1 st frame of original image, that is, the 1 st frame of original image is displayed on the preview interface as the 1 st frame of preview image. After the mobile phone acquires the 2 nd frame original image, a 2 nd frame preview image corresponding to the 2 nd frame original image is generated according to the 1 st to 2 nd frame original images, and the 2 nd frame preview image is displayed on a preview interface. After the 3 rd frame original image is collected by the mobile phone, a 3 rd frame preview image corresponding to the 3 rd frame original image is generated according to the 1 st to 3 rd frame original images, and the 3 rd frame preview image is displayed on a preview interface. After the mobile phone acquires the N1+1 th original image, the N1+1 th preview image is generated according to the 1 st to N1+1 th original images, and the N1+1 th preview image is displayed on a preview interface. Subsequently, after acquiring a new original image, the mobile phone generates a preview image corresponding to the new original image by combining the N1 frame original image before the new original image and the new original image, and displays the generated preview image on a preview interface.

Under the condition that the preview image is obtained by image transformation according to a stable image matrix calculated by the target pose on the smooth target motion curve of the camera, the smooth target motion curve of the camera meets the sub-constraint condition (1), so that the whole transition between the preview images obtained based on the target motion curve of the camera after warp transformation is smooth. The smooth target motion curve of the camera meets the sub-constraint condition (2), so that a warp-transformed preview image obtained based on the target motion curve of the camera does not exceed the cropping boundary. The smooth target motion curve of the camera meets the sub-constraint condition (3), so that the image stabilizing effect of the human face of the subject on the preview image after warp transformation obtained based on the target motion curve of the camera is better.

211. And if the fourth condition is not met, the mobile phone calculates the image stabilization transformation matrix of the original image according to the RS compensation quantity and the motion information of the camera. Then, the mobile phone performs the step 210.

If the fourth condition is not met, it can be shown that the human face and the background of the main body do not have large relative motion, and the RS compensation amount does not need to be reduced, so that the mobile phone can calculate the image stabilization transformation matrix of the original image according to the original RS compensation amount and the motion information of the camera, perform warp transformation according to the image stabilization transformation matrix, obtain a preview image, and display the preview image on a preview interface.

That is to say, in the preview state, the mobile phone can combine the motion information of the main body and the gyroscope data to perform anti-shake processing, so that anti-shake is realized for the motion main body and the background at the same time, the image stabilizing effect of the preview image is improved, and the shooting experience of the user is improved. In addition, the mobile phone can dynamically adjust the strength of the face constraint in the preview state according to the motion amplitude of the main face so as to balance the anti-shake effect of the background and the main face on the preview image presented to the user, so that the image stabilizing effect of the background and the main face on the preview image is better. Moreover, when the mobile phone performs RS correction on the background, the distortion of the main human face on the preview image can be reduced or avoided as much as possible by adjusting the RS compensation amount.

For example, in the preview state, when the mobile phone is in the panning mode, the effect diagrams of the preview image of the mobile phone without the anti-shake processing can be seen in (a) - (d) of fig. 9A. As shown in fig. 9A, both the background and the subject face are shaken during the preview process. For example, the position of the leaves in the background shakes up and down and rotates, and the position of the face also shakes up and down and rotates. The effect diagrams of the preview image subjected to the anti-shake processing when the face anti-shake mode is not turned on by the mobile phone can be seen in (a) - (d) of fig. 9B. The effect diagrams of the preview image that is subjected to the anti-shake processing after the face anti-shake mode is turned on and is not subjected to the face distortion correction can be seen in (a) - (d) of fig. 9C. The effect diagrams of the preview image after the mobile phone starts the face anti-shake mode, and performs the anti-shake processing and the face distortion correction can be seen in (a) - (D) of fig. 9D.

Then, the mobile phone enters a shooting process after detecting the shooting operation of the user. Referring to fig. 2, the method may further include step 212:

212. after the mobile phone detects the shooting operation of the user, an original image is collected according to a preset frame rate in the shooting process.

For example, after detecting an operation of the user clicking the shooting control 900 shown in (D) in fig. 9D, the mobile phone determines that the shooting operation of the user is detected, thereby entering a video shooting process. For another example, after detecting that the user indicates to start shooting operation by voice, the mobile phone determines that the shooting operation of the user is detected, and then enters a video shooting process.

It can be understood that there may be many other ways for triggering the mobile phone to enter the video shooting process, and the embodiment of the present application is not limited thereto.

It should be noted that the image data stream during shooting includes a preview stream during recording and a video stream. And the preview stream in the video is used for presenting the recorded image on a shooting interface by a user in the video recording process. The preview stream used to present preview images to the user in the preview state, as opposed to the preview stream in video recording, may be referred to as a pre-video preview stream. The video stream is used to generate video images in a video file.

In some embodiments, the preview stream in the video recording is the same data stream as the preview stream before the video recording. After the shooting process is started, the mobile phone continues to process the state, the anti-shake mode and the processing process in the preview state to generate a recorded image. For example, before entering the shooting process, the mobile phone is in a face anti-shake mode, and the mobile phone generates a 60 th frame preview image; and after the shooting process is started, the mobile phone is still in a face anti-shake mode, and a 61 st frame preview image is generated by combining the original image before the shooting process is started, wherein the 61 st frame preview image is the 1 st frame recording image on the shooting interface. If the mobile phone does not enter the face anti-shake mode when the shooting process is started, the mobile phone determines whether the face anti-shake mode is started or not after the shooting process is started.

In other embodiments, the preview stream in the video recording is not the same stream as the preview stream before the video recording. And after the shooting process is started, the mobile phone stops previewing streams before video recording and starts previewing streams in the video recording. Similarly, in a preview state, the mobile phone performs anti-shake processing according to the original image in the preview stream before video recording by combining the motion information of the human face and the gyroscope data so as to generate and display a preview image; the mobile phone can perform anti-shake processing according to original images in preview streams in videos in the shooting process and by combining motion information of human faces and gyroscope data, and then generates and displays the recorded images. And the mobile phone can judge whether to enter the face anti-shake mode again according to the original image collected in the shooting process, and execute the face anti-shake process after entering the face anti-shake mode. For example, before the shooting process is started, the mobile phone is in a face anti-shake mode, and the mobile phone generates a 60 th frame preview image; after the shooting process is started, the mobile phone determines whether the mobile phone enters the face anti-shake mode again, and generates a 1 st frame of shooting image according to the original image acquired in the shooting process and displays the frame of shooting image on the shooting interface after the mobile phone enters the face anti-shake mode. For example, referring to fig. 2, after step 212, for the preview stream in the video recording, the method may further include the following steps 213 and 222:

213. the mobile phone determines whether a first condition is met according to an S 'frame original image in the shooting process, wherein S' is an integer larger than 1. If the first condition is satisfied, the mobile phone performs step 214 and step 221; if the first condition is not satisfied, the mobile phone performs step 222-225.

Wherein S' and S may be the same or different.

214. If the first condition is met, the mobile phone acquires the motion information of the main face according to the M 'frame original image, wherein M' is an integer larger than 1.

Wherein M' and M may be the same or different.

215. The mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N 'frame original image and the motion information of the main face, wherein N' is an integer larger than 1.

Wherein N 'and N may be the same or different, N1' and N1 may be the same or different, I 'and I may be the same or different, and N2' and N2 may be the same or different.

216. And if the first condition is not met, the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N' frame original image.

217. And the mobile phone calculates the RS compensation amount corresponding to the original image according to the data of the gyroscope.

218. The handset determines whether a fourth condition is satisfied. If the fourth condition is satisfied, the mobile phone performs step 219; if the fourth condition is not satisfied, the mobile phone performs step 222.

219. If the fourth condition is met, the mobile phone determines a reduction factor according to the panning amplitude, and the reduction factor is used for reducing the RS compensation amount.

220. And the mobile phone calculates an image stabilization transformation matrix of the original image according to the reduced RS compensation quantity and the motion information of the camera.

221. And the mobile phone carries out image transformation on the original image according to the image stabilization transformation matrix to obtain a recorded image, and the recorded image is displayed on a preview interface.

222. And if the fourth condition is not met, the mobile phone calculates the image stabilization transformation matrix of the original image according to the RS compensation quantity and the motion information of the camera. Then, the mobile phone performs the step 221.

It should be noted that, for the description of the steps 213-222, reference may be made to the related description of the steps 202-211, which is not repeated herein.

Similar to the preview state, in the shooting process, the mobile phone can obtain the target pose of the camera corresponding to the N1+1 th frame of original image according to the initial pose of the camera corresponding to the N1+ I + N2 frame of original image, so as to obtain the stationary image transformation matrix corresponding to the N1+1 th frame of original image according to the target pose of the camera corresponding to the N1+1 th frame of original image, and further obtain the 1 st frame of recorded image corresponding to the N1+1 th frame of original image after warp transformation. That is, the starting frame of the I-frame original image corresponds to the N1+1 th frame of the N-frame original images, and the mobile phone displays the captured image delayed by at least N2 frames compared with the original image acquired by the mobile phone.

In other embodiments, in the process of acquiring the first N frames of original images by the mobile phone immediately after entering the shooting process, each frame of original image in the first N1 frames of original images is displayed on the shooting interface as a captured image. Subsequently, after the mobile phone acquires a new original image, the method described in the above embodiment is used to generate a captured image corresponding to the I-frame original image according to the N-frame original image, and the generated captured image is displayed on the shooting interface.

In other embodiments, during the shooting process, the frame number of the captured image displayed on the shooting interface by the mobile phone corresponds to the frame number of the original image, i.e., N2 is 0, and I is 1. That is, after acquiring a frame of original image, the mobile phone displays the recorded image corresponding to the frame of original image. N1 may be small, for example, 5, 8, or 10. In the shooting process, before the mobile phone collects N1 frames of original images, the mobile phone generates and displays a recording image corresponding to the current original image by combining the original images collected in the shooting process.

For example, just after the shooting process is started, the mobile phone acquires the 1 st frame of original image, that is, the 1 st frame of original image is displayed on the shooting interface as the 1 st frame of recorded image. After the mobile phone acquires the 2 nd frame original image, a 2 nd frame recording image corresponding to the 2 nd frame original image is generated according to the 1 st to 2 nd frame original images, and the 2 nd frame recording image is displayed on a shooting interface. After the 3 rd frame original image is collected by the mobile phone, a 3 rd frame recording image corresponding to the 3 rd frame original image is generated according to the 1 st to 3 rd frame original images, and the 3 rd frame recording image is displayed on a shooting interface. After the mobile phone collects the N1+1 th frame of original image, the N1+1 th frame of recording image is generated according to the 1 to N1+1 th frame of original image, and the N1+1 th frame of recording image is displayed on the shooting interface. Subsequently, after acquiring a new original image, the mobile phone generates a recording image corresponding to the new original image by combining the N1 frame original image before the new original image and the new original image, and displays the generated recording image on the shooting interface.

Under the condition that the recorded images are obtained by image transformation according to a stable image matrix calculated by the target pose on the smooth target motion curve of the camera, the smooth target motion curve of the camera meets the sub-constraint condition (1), so that the integral transition between the recorded images after warp transformation, which are obtained based on the target motion curve of the camera, is smooth. The smooth target motion curve of the camera meets the sub-constraint condition (2), so that a warp-transformed captured image obtained based on the target motion curve of the camera does not exceed the cropping boundary. The smooth target motion curve of the camera meets the sub-constraint condition (3), so that the image stabilization effect of the main human face on the warp-transformed recorded image obtained based on the target motion curve of the camera is better.

Like this, at the shooting in-process, the cell-phone can combine the motion information and the gyroscope data of main part to carry out anti-shake and handle to realize anti-shake to motion main part and background simultaneously, improve the steady image effect of taking the image, improve the user and shoot and experience. In addition, the mobile phone can dynamically adjust the strength of face constraint in the shooting process according to the motion amplitude of the main face so as to balance the anti-shake effect of the background on the recorded image and the main face presented to the user, so that the image stabilizing effect of the background on the recorded image and the main face is better. Moreover, when the mobile phone can perform RS correction on the background, the distortion of the main human face on the recorded image can be reduced or avoided as much as possible by adjusting the RS compensation amount.

For example, in the shooting process, when the mobile phone is in the panning mode, the schematic diagrams of the effect of the captured image without the anti-shake processing performed by the mobile phone can be seen in (a) - (d) of fig. 10A. As shown in fig. 10A, in the preview process, both the background and the subject face are shaken. For example, the position of the leaf in the background shakes up and down, and the position of the face also shakes up and down. The schematic diagrams of the effect of the captured image subjected to the anti-shake processing when the face anti-shake mode is not turned on in the mobile phone can be seen in (a) - (d) of fig. 10B. The schematic effect diagrams of the captured images that are subjected to the anti-shake processing after the face anti-shake mode is turned on and are not subjected to the face distortion correction can be seen in (a) - (d) of fig. 10C. The schematic effect diagrams of the captured image after the anti-shake mode of the face is turned on and the anti-shake processing is performed and the face distortion correction is performed can be seen in (a) - (D) of fig. 10D.

In the shooting process, the video stream and the preview stream in the video recording are processed in parallel and independently. For the video stream, the state and the processing process of the mobile phone before the shooting process are irrelevant, the mobile phone can judge whether to enter a face anti-shake mode according to an original image collected in the shooting process, and the face anti-shake process is executed after the face anti-shake mode is entered. For example, after the shooting process is started, the mobile phone determines whether to start a face anti-shake mode, and generates and stores the 1 st frame of video image according to the original image acquired in the shooting process in the face anti-shake mode, so as to generate a video file according to the stored video image after the shooting is stopped. For example, referring to fig. 2, after the above step 212, the method may further include the following steps 213 '-222' for the video stream:

213', the mobile phone determines whether the first condition is satisfied according to the S "frame original image in the shooting process, where S" is an integer greater than 1. If the first condition is satisfied, the mobile phone performs steps 214 '-221'; if the first condition is not satisfied, the handset performs steps 222 '-225'.

Wherein S "and S" may be the same or different.

214', if the first condition is satisfied, the mobile phone obtains the motion information of the face of the subject according to M "frames of original images, where M" is an integer greater than 1.

Wherein M "and M may be the same or different.

215 ' and the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N ' frame original image and the motion information of the main face, wherein N ' is an integer larger than 1.

Wherein N ' and N may be the same or different, N1 ' and N1 may be the same or different, I ' and I ' may be the same or different, and N2 ' and N2 may be the same or different.

216 'if the first condition is not met, the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N' frames of original images.

217', the mobile phone calculates the RS compensation amount corresponding to the original image according to the data of the gyroscope.

218', the handset determines whether the fourth condition is satisfied. If the fourth condition is satisfied, the mobile phone performs step 219'; if the fourth condition is not satisfied, the mobile phone performs step 222'.

219', if the fourth condition is satisfied, the handset determines a reduction factor for reducing the RS compensation amount according to the panning amplitude.

220' and the mobile phone calculates an image stabilization transformation matrix of the original image according to the reduced RS compensation quantity and the motion information of the camera.

221' and the mobile phone performs image transformation on the original image according to the image stabilization transformation matrix to obtain a video image and stores the video image.

222', if the fourth condition is not met, the mobile phone calculates a stable image transformation matrix of the original image according to the RS compensation quantity and the motion information of the camera. Then, the mobile phone performs the step 221'.

It should be noted that, for the description of the steps 213 "-222", reference may be made to the related description in the above steps 202-211, and details are not repeated here.

Similar to the preview image and the captured image, in the shooting process, the mobile phone can obtain the target pose of the camera corresponding to the N1+1 th frame of original image according to the initial pose of the camera corresponding to the N1+ I + N2 frame of original image, so as to obtain the stationary image transformation matrix corresponding to the N1+1 th frame of original image according to the target pose of the camera corresponding to the N1+1 th frame of original image, and further obtain the 1 st frame of captured image corresponding to the N1+1 th frame of original image after warp transformation. That is, the starting frame of the I-frame original image corresponds to the N1+1 th frame of the N-frame original images, and the mobile phone displays the captured image delayed by at least N2 frames compared with the original image acquired by the mobile phone.

The video image does not need to be presented for the user in real time in the shooting process, so the anti-shake processing time length and the processing delay can be longer, and the number of the original image frames can be more. To obtain better quality video images, S "may be greater than S, M" may be greater than M, N "may be greater than N, N1" may be greater than N1, I "may be greater than I, and/or N2" may be greater than N2. For example, for the preview stream, N2 or N2' may be 0 or a smaller integer (e.g., 2 or 3, etc.) to reduce latency; for video streams, N2 "may be a larger integer (e.g., 27 or 15, etc.) to improve the anti-shake processing effect.

In other embodiments, in the process of capturing the first N frames of original images by the mobile phone immediately after entering the shooting process, each frame of original image in the first N1 frames of original images is taken as a video image and stored. Subsequently, after the mobile phone acquires a new original image, the method described in the above embodiment is used to generate a video image corresponding to the I-frame original image according to the N-frame original image.

In other embodiments, during the shooting process, the frame number of the video image generated by the mobile phone corresponds to the frame number of the original image, i.e., N2 is 0 and I is 1. That is, after acquiring a frame of original image, the mobile phone displays a video image corresponding to the frame of original image. N1 may be small, for example, 5, 8, or 10. In the shooting process, before the mobile phone collects N1 frames of original images, the mobile phone generates and stores a video image corresponding to the current original image by combining the original images collected in the shooting process.

For example, just after the shooting process is started, the 1 st frame of original image is collected by the mobile phone, that is, the 1 st frame of original image is used as the 1 st frame of video image. After the mobile phone collects the 2 nd frame original image, the 2 nd frame video image corresponding to the 2 nd frame original image is generated according to the 1 st to 2 nd frame original images. After the 3 rd frame original image is collected by the mobile phone, a 3 rd frame video image corresponding to the 3 rd frame original image is generated according to the 1 st to 3 rd frame original images. After the mobile phone collects the N1+1 th frame of original image, the N1+1 th frame of video image is generated according to the 1 st to N1+1 th frame of original image. Subsequently, after the mobile phone acquires a new original image, a video image corresponding to the new original image is generated according to the N1 frame original image before the new original image and the new original image.

Under the condition that the video images are obtained by image transformation according to the image stabilization matrix calculated by the target pose on the smooth target motion curve of the camera, the whole transition between the video images after warp transformation obtained on the basis of the target motion curve of the camera is smooth because the target motion curve of the smooth camera meets the sub-constraint condition (1). The smooth target motion curve of the camera meets the sub-constraint condition (2), so that a warp-transformed video image obtained based on the target motion curve of the camera does not exceed the clipping boundary. The smooth target motion curve of the camera meets the sub-constraint condition (3), so that the image stabilization effect of the main human face on the warp-transformed video image obtained based on the target motion curve of the camera is better.

Like this, at the shooting in-process, the cell-phone can combine the motion information and the gyroscope data of main part to carry out anti-shake and handle to realize anti-shake to motion main part and background simultaneously, improve video image's steady image effect, improve the user and shoot and experience. In addition, the mobile phone can dynamically adjust the strength of face constraint in the shooting process according to the motion amplitude of the main face so as to balance the anti-shake effect of the background and the main face on the video image, and the image stabilizing effect of the background and the main face on the video image is better. Moreover, when the mobile phone can perform RS correction on the background, the distortion of the main human face on the video image can be reduced or avoided as much as possible by adjusting the RS compensation amount.

After the above step 222 and step 222', referring to fig. 2, the method may further include:

223. and after the shooting of the mobile phone is finished, generating a video file according to the video image.

For example, the mobile phone determines that the shooting is finished after detecting the user's operation of clicking the stop shooting control 1000 as shown in (D) of fig. 10D. It can be understood that the shooting stopping operation may also be other gesture operations or user voice instruction operations, and the operation of triggering the mobile phone to end the shooting process is not limited in the embodiment of the present application.

The video image in the video file generated after the shooting of the mobile phone is finished is obtained by combining the motion information of the main body and the gyroscope data to perform anti-shake processing, so that the image stabilizing effect of the background on the video image and the face of the main body is better, and the shooting experience of a user is better.

For an exemplary effect diagram of a video image without the anti-shake processing of the mobile phone, see (a) - (d) in fig. 11A. As shown in fig. 11A, both the background and the subject face shake during shooting. For example, the position of the leaf in the background shakes up and down, and the position of the face also shakes up and down. The schematic effect diagrams of the video images subjected to the anti-shake processing when the face anti-shake mode is not turned on by the mobile phone can be seen in (a) - (d) of fig. 11B. The schematic effect diagrams of the video image that is subjected to the anti-shake processing after the face anti-shake mode is turned on and is not subjected to the face distortion correction can be seen in (a) - (d) of fig. 11C. The schematic effect diagrams of the video image after the mobile phone starts the face anti-shake mode, and performs the anti-shake processing and the face distortion correction can be seen in (a) - (D) of fig. 11D.

In addition, as mentioned above, after entering/exiting the main body anti-shake mode, the mobile phone may prompt the user in a manner of displaying information, voice broadcasting, vibration, or the like. For example, referring to fig. 12, in the shooting process, after the mobile phone enters the face anti-shake mode, the user may be prompted on the shooting interface through text information: the face anti-shake has been turned on!

In the shooting process described in the above step 201-223, the flow of the mobile phone performing the face anti-shake processing can also refer to fig. 13. The process comprises the following steps: the mobile phone detects the human face characteristic points according to the multi-frame original images, obtains a smooth target motion curve of the main human face according to the human face characteristic points, and generates a first constraint condition according to the target motion curve of the main human face. The process further comprises: the mobile phone calculates a rotation angle according to gyroscope data corresponding to multiple frames of original images to obtain an original motion curve of the camera; and the mobile phone solves the target motion curve of the camera according to the original motion curve of the camera and the first constraint condition. The process further comprises: after the mobile phone enters a panning mode, determining the reduced RS compensation amount so as to correct the face distortion when the RS is corrected; the mobile phone calculates a motion compensation homography matrix according to the target motion curve and the reduced RS compensation amount; and the mobile phone performs motion compensation and RS correction on the original image according to the motion compensation homography matrix, so as to obtain a stable video image frame after anti-shake processing.

In the scheme described in step 201-223, the mobile phone can perform anti-shake processing in combination with the motion information of the main body and the gyroscope data in the preview state and the shooting process, so that anti-shake processing is simultaneously performed on the motion main body and the background, the image stabilizing effect of the preview image, the recorded image and the video image is improved, and the shooting experience of the user is improved.

Particularly, under the video recording scene that the motion main body such as the front-mounted self-timer is closer to the camera of the mobile phone, the relative motion of the motion main body is larger, and the shaking of the main body image is more obvious, the video recording method provided by the application is adopted to perform anti-shaking processing, the anti-shaking and image stabilizing effects of the image are better, and the user shooting experience is also better.

The above description is given by taking as an example that the method provided by the embodiment of the present application is used in both the preview state and the shooting process of the mobile phone in the video recording mode, and the anti-shake processing is performed by combining the motion information of the face of the subject and the gyroscope data. In other embodiments, the mobile phone does not perform anti-shake processing in the preview state of the video mode by using the method provided by the embodiment of the present application, but performs anti-shake processing in the shooting process by using the method provided by the embodiment of the present application. Referring to fig. 14, the steps 201-211 shown in fig. 2 can be replaced by the steps 201A:

the mobile phone starts a shooting function and then enters a video recording mode, an original image is collected according to a preset frame rate in a preview state, a preview image is generated according to the original image, and the preview image is displayed on a preview interface.

In the scheme, the mobile phone can be used for carrying out anti-shaking processing by combining the motion information of the main body and gyroscope data in the shooting process, so that anti-shaking processing is realized on the motion main body and the background, the image stabilizing effect of the recorded images and the video images is improved, and the shooting experience of a user is improved.

In some other embodiments, in the video recording mode, the mobile phone does not perform the anti-shake processing on the preview stream before video recording and the preview stream in the video recording by using the method provided by the embodiment of the present application, and performs the anti-shake processing on the video stream by using the method provided by the embodiment of the present application. Referring to fig. 15, the method may include the above-described step 201A, the following step 212A, and steps 213' -223.

212A, after the mobile phone detects the shooting operation of the user, acquiring an original image according to a preset frame rate in the shooting process, generating a recorded image according to the original image, and displaying the recorded image on a shooting interface.

In the scheme, the mobile phone can be used for carrying out anti-shaking processing by combining the motion information of the main body and the gyroscope data in the shooting process, so that anti-shaking processing is realized on the motion main body and the background at the same time, the image stabilizing effect of the video image is improved, and the shooting experience of a user is improved.

In some embodiments of the present application, the video file saved by the mobile phone and subjected to the main body anti-shake processing may be distinguished from other video files and identified in particular, so that a user can intuitively know the video file subjected to the main body anti-shake processing. Illustratively, referring to fig. 16, a text identifier 1600 of "rlfd" is displayed on the video file generated by the face anti-shake process.

With reference to the foregoing embodiments and accompanying drawings, another embodiment of the present application provides a shooting method, which may be implemented in an electronic device having a hardware structure shown in fig. 1, where the electronic device includes a camera, and the camera includes a camera head. As shown in fig. 17, the method may include:

1701. the electronic equipment collects an original image after the video recording function is started.

After the video recording function is started, the electronic equipment can continuously acquire the original image according to the preset acquisition frame rate.

1702. The electronic equipment acquires the motion information of the main body according to the collected multi-frame original images.

For example, the subject may be the subject face described above. The motion information of the subject may be used to represent the motion condition of the subject, for example, the motion information of the subject may be the target motion curve of the face of the subject.

1703. The electronic equipment obtains the motion information of the camera according to the posture sensor data corresponding to the multi-frame original image and the motion information of the main body.

For example, the gesture sensor may be a gyroscope, the motion information of the camera may be a target motion curve of the camera, and the target motion curve of the camera may be obtained according to the face constraint in the sub-constraint condition (3).

1704. The electronic equipment calculates a stable image transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in a plurality of frames of original images.

Wherein the image stabilization transformation matrix is used for performing motion compensation and warp transformation on the first original image.

1705. And the electronic equipment performs image transformation on the first original image according to the image stabilization transformation matrix to obtain a target image.

For example, the target image may be a preview image, a captured image, or a video image of a video recording scene.

The foregoing is described by taking a video recording scene as a front-end self-timer scene as an example, and the video recording scene related to the embodiment of the present application is not limited to this kind of scene.

The above description is given by taking an electronic device as an example of a mobile phone, and other electronic devices such as a mobile phone, a smart watch, a tablet computer, and the like may also perform anti-shake processing by using the above method, which is not described herein again.

It will be appreciated that in order to implement the above-described functions, the electronic device comprises corresponding hardware and/or software modules for performing the respective functions. The present application is capable of being implemented in hardware or a combination of hardware and computer software in conjunction with the exemplary algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, with the embodiment described in connection with the particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In this embodiment, the electronic device may be divided into functional modules according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module may be implemented in the form of hardware. It should be noted that the division of the modules in this embodiment is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

Embodiments of the present application also provide an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, the one or more memories storing computer program code comprising computer instructions which, when executed by the one or more processors, cause the electronic device to perform the associated method steps described above to implement the video recording method of the above embodiments.

Embodiments of the present application further provide a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed on an electronic device, the electronic device is caused to execute the above related method steps to implement the video recording method in the above embodiments.

Embodiments of the present application further provide a computer program product, which when running on a computer, causes the computer to execute the above related steps, so as to implement the video recording method executed by the electronic device in the above embodiments.

In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component or a module, and may include a processor and a memory connected to each other; when the device runs, the processor can execute the computer execution instruction stored in the memory, so that the chip can execute the video recording method executed by the electronic equipment in the above method embodiments.

The electronic device, the computer-readable storage medium, the computer program product, or the chip provided in this embodiment are all configured to execute the corresponding method provided above, so that the beneficial effects achieved by the electronic device, the computer-readable storage medium, the computer program product, or the chip may refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Through the description of the above embodiments, those skilled in the art will understand that, for convenience and simplicity of description, only the division of the above functional modules is used as an example, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or partially contributed to by the prior art, or all or part of the technical solutions may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A video recording method is applied to an electronic device, the electronic device comprises a camera, and the method comprises the following steps:

collecting an original image after starting a video recording function;

acquiring motion information of a main body according to the collected multi-frame original images;

acquiring the motion information of the camera according to the attitude sensor data corresponding to the multiple frames of original images and the motion information of the main body;

calculating an image stabilization transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in the multiple frames of original images;

and carrying out image transformation on the first original image according to the image stabilization transformation matrix to obtain a target image.

2. The method of claim 1, wherein the capturing the original image after the video recording function is turned on comprises:

the method comprises the steps of starting a video recording function and collecting an original image after detecting shooting operation of a user;

the method further comprises the following steps:

and after the shooting stopping operation of the user is detected, generating a video file according to the video image, wherein the video image is the target image.

3. The method of claim 1, wherein the capturing the original image after the video recording function is turned on comprises:

the method further comprises the following steps:

and displaying a recorded image on a shooting interface, wherein the recorded image is the target image.

4. The method of claim 1, wherein the target image is a preview image, the method further comprising:

and displaying the preview image on a preview interface.

5. The method according to any one of claims 1 to 4, wherein the motion information of the subject is a target motion curve of the subject, the target motion curve of the subject is obtained according to an original motion curve of the subject, and the original motion curve of the subject is obtained according to a position of a center of the subject on the original images of the plurality of frames.

6. The method of claim 5, wherein the target motion profile of the subject is continuous; the target motion curve of the main body is first-order and second-order conductive, and the curvature is smaller than or equal to a first preset threshold; the distance between the center of the subject on the target motion curve of the subject and the center of the subject on the original motion curve of the subject is less than or equal to a first preset value.

7. The method according to claim 5 or 6, wherein the motion information of the camera is a target motion curve of the camera, and the obtaining of the motion information of the camera according to the corresponding gesture sensor data of the plurality of frames of raw images and the motion information of the subject comprises:

acquiring an original motion curve of the camera according to the attitude sensor data corresponding to the multiple frames of original images;

and obtaining a target motion curve of the camera according to the original motion curve of the camera and subject constraints, wherein the subject constraints are related to the target motion curve of the subject, and the subject constraints are used for enabling the distance between the center of the subject on the target image and the center of the subject corresponding to the target motion curve of the subject to be smaller than or equal to a second preset value.

8. The method of claim 7, wherein obtaining a target motion curve of a camera from a raw motion curve and subject constraints of the camera comprises:

determining a corresponding main body constraint gear according to the motion amplitude of the main body;

gradually increasing the main body constraint strength from an initial value to an upper bound value corresponding to the main body constraint gear;

and obtaining a target motion curve of the camera according to the original motion curve of the camera and the subject constraint strength.

9. The method according to any one of claims 1 to 8, wherein the obtaining motion information of the subject according to the collected multiple frames of original images comprises:

and acquiring the motion information of the main body according to the collected multiple frames of original images after meeting a preset condition.

10. The method according to claim 9, wherein the preset condition comprises:

the motion amplitude of the electronic equipment is greater than or equal to a third preset value;

the area occupation ratio of the main body on the original image is greater than or equal to a fourth preset value;

the number of frames in which the original images of the subject appear continuously is greater than or equal to a fifth preset value.

11. The method according to any one of claims 1 to 10, wherein the obtaining of the motion information of the camera according to the corresponding posture sensor data of the plurality of frames of raw images and the motion information of the subject comprises:

obtaining motion information of the camera according to pose sensor data corresponding to N frames of original images and motion information of the subject, wherein N is an integer larger than 1, N is N1+ I + N2, N1 and I are positive integers, and N2 is a non-negative integer;

calculating an image stabilization transformation matrix of a first original image according to the motion information of the camera, comprising:

and calculating an image stabilization transformation matrix of the I frame original image according to the target pose of the camera corresponding to the N frame original image on the motion information of the camera, wherein the I frame original image is the first original image, the image stabilization transformation matrix of the I frame original image is used for obtaining the I frame target image, and the initial frame of the I frame original image corresponds to the (N1 + 1) th frame original image in the N frame original image.

12. The method according to claim 11, wherein when the target image is a preview image or a captured image, the N2 is 0.

13. The method according to any one of claims 9-12, further comprising:

if the preset condition is not met, obtaining the motion information of the camera according to the attitude sensor data corresponding to the multiple frames of original images;

14. The method of claim 13, further comprising:

if the preset condition is met, prompting a user that a target anti-shake mode is started;

and if the preset condition is not met, prompting a user that the target anti-shake mode is not started.

15. The method according to any of claims 1-14, wherein before said calculating an image stabilization transformation matrix for a first raw image from motion information of said camera, the method further comprises:

calculating the RS compensation quantity of the rolling shutter of the first original image according to the data of the attitude sensor;

the calculating of the image stabilization transformation matrix of the first original image according to the motion information of the camera comprises:

calculating a rotation compensation amount of the first original image according to the motion information of the camera;

and calculating an image stabilization transformation matrix of the first original image according to the rotation compensation quantity and the RS compensation quantity.

16. The method of claim 15, wherein the calculating an image stabilization transformation matrix of the first original image according to the rotation compensation amount and the RS compensation amount comprises:

if the camera enters the panning mode, reducing the RS compensation amount according to the panning amplitude, wherein the panning amplitude is negatively related to the size of an included angle between a motion main axis of the camera and a target coordinate axis;

and calculating an image stabilization transformation matrix of the first original image according to the rotation compensation quantity and the reduced RS compensation quantity.

17. The method of any one of claims 1-16, wherein the subject is a human face.

18. An electronic device, comprising:

a camera for capturing an image;

a screen for displaying an interface;

one or more processors;

a memory;

and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions which, when executed by the electronic device, cause the electronic device to perform the video recording method of any of claims 1-17.

19. A computer-readable storage medium comprising computer instructions which, when executed on a computer, cause the computer to perform the video recording method of any one of claims 1-17.

20. A computer program product, which, when run on a computer, causes the computer to perform the video recording method according to any one of claims 1-17.