CN114339101B - Video recording method and equipment - Google Patents
Video recording method and equipment Download PDFInfo
- Publication number
- CN114339101B CN114339101B CN202011057023.8A CN202011057023A CN114339101B CN 114339101 B CN114339101 B CN 114339101B CN 202011057023 A CN202011057023 A CN 202011057023A CN 114339101 B CN114339101 B CN 114339101B
- Authority
- CN
- China
- Prior art keywords
- image
- camera
- original image
- motion information
- main body
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The embodiment of the application provides a video recording method and equipment, which relate to the technical field of electronics and can acquire the motion information of a camera by combining the motion information of a shot main body and gesture sensor data under a video recording scene, so that anti-shake processing is performed according to the motion information of the camera, and anti-shake is realized on the motion main body and a background at the same time, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved. The specific scheme is as follows: the electronic equipment acquires an original image after starting a video recording function; acquiring motion information of a main body according to the acquired multi-frame original image; acquiring motion information of a camera according to gesture sensor data corresponding to a plurality of frames of original images and motion information of a main body; calculating an image stabilizing transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in a plurality of frames of original images; and carrying out image transformation on the first original image according to the image stabilizing transformation matrix to obtain a target image. The embodiment of the application is used for video anti-shake.
Description
Technical Field
The embodiment of the application relates to the technical field of electronics, in particular to a video recording method and video recording equipment.
Background
With the development of shooting technology, the requirements of users on video recording effects are also increasing. In the video recording process, electronic equipment such as a mobile phone or a tablet personal computer can perform anti-shake processing on images according to gyroscope (gyro) data. The anti-shake processing scheme can achieve a good anti-shake effect on the background in the shooting range, but is difficult to achieve anti-shake on a moving body in the shooting range, and shooting experience of a user is poor.
Disclosure of Invention
The embodiment of the application provides a video recording method and video recording equipment, which can acquire the motion information of a camera by combining the motion information of a shot main body and gesture sensor data in a video recording scene, so that anti-shake processing is performed according to the motion information of the camera, and thus the anti-shake is realized on the motion main body and the background at the same time, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved.
In order to achieve the above purpose, the embodiment of the application adopts the following technical scheme:
in one aspect, an embodiment of the present application provides a video recording method, applied to an electronic device, where the electronic device includes a camera, the method may include: the electronic device collects the original image after the video recording function is started. And the electronic equipment acquires the motion information of the main body according to the acquired multi-frame original image. And then, the electronic equipment obtains the motion information of the camera according to the gesture sensor data corresponding to the multi-frame original image and the motion information of the main body. The electronic equipment calculates an image stabilizing transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in a plurality of frames of original images. And the electronic equipment performs image transformation on the first original image according to the image stabilizing transformation matrix to obtain a target image.
In the scheme, the electronic equipment can obtain the motion information of the camera according to the motion information of the shot main body and the gesture sensor data under the video recording scene, so that the image stabilizing transformation matrix is obtained according to the motion information of the camera, and further, the motion compensation and the anti-shake are realized on the motion main body and the background through the image stabilizing transformation matrix, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved.
In one possible design, the electronic device may collect the original image after turning on the video recording function, and may include: the electronic equipment acquires an original image after starting a video recording function and detecting shooting operation of a user. The method may further comprise: after the electronic equipment detects that the shooting operation of the user is stopped, a video file is generated according to a video image, wherein the video image is a target image.
That is, in the video shooting process, the electronic device can combine the motion information of the shot main body and the gesture sensor data to obtain the motion information of the camera, so that the original image is subjected to anti-shake processing according to the motion information of the camera to obtain the video image, and therefore, the motion compensation and the anti-shake are simultaneously realized for the motion main body and the background, the image stabilizing effect of the video image is improved, and the shooting experience of a user is improved.
In another possible design, the capturing of the original image after the video recording function is turned on may include: the electronic equipment acquires an original image after starting a video recording function and detecting shooting operation of a user. The method may further comprise: the electronic device displays a recorded image on a shooting interface, wherein the recorded image is a target image.
That is, in the video shooting process, the electronic device can combine the motion information of the shot main body and the gesture sensor data to obtain the motion information of the camera, so that the original image is subjected to anti-shake processing according to the motion information of the camera to obtain the recorded image on the shooting interface, and therefore the motion compensation and the anti-shake are realized on the motion main body and the background at the same time, the image stabilizing effect of the recorded image is improved, and the shooting experience of a user is improved.
In another possible design, the target image is a preview image, and the method may further include: the electronic device displays the preview image on the preview interface.
That is, in the preview state after the video recording function is started, the electronic device can combine the motion information of the photographed subject and the gesture sensor data to obtain the motion information of the camera, so that the original image is subjected to anti-shake processing according to the motion information of the camera to obtain the preview image on the preview interface, and thus the motion compensation and the anti-shake are realized for the moving subject and the background at the same time, the image stabilizing effect of the photographed image is improved, and the photographing experience of the user is improved.
In another possible design, the motion information of the main body is a target motion curve of the main body, the target motion curve of the main body is obtained according to an original motion curve of the main body, and the original motion curve of the main body is obtained according to the position of the center of the main body on a plurality of frames of original images.
That is, the original motion profile of the subject and the targeted motion profile of the subject may be used to represent the motion profile of the subject.
In another possible design, the target motion profile of the subject is continuous; the target motion curve of the main body is first-order and second-order conductive, and the curvature is smaller than or equal to a first preset threshold value; the distance between the center of the subject on the subject's target motion profile and the center of the subject on the subject's original motion profile is less than or equal to a first preset value.
That is, the motion trail of the center of the human face is smoother, and the change cannot be too severe; the change of the speed and the acceleration of the target motion curve of the human face of the main body is smoother and cannot be too severe; the center of the face on the target motion curve of the subject face cannot deviate too far from the original motion curve of the subject face, and the center of the face on the target motion curve of the subject face should be near the original motion curve of the subject face.
In another possible design, the motion information of the camera is a target motion curve of the camera, and the electronic device obtains the motion information of the camera according to the gesture sensor data corresponding to the multi-frame original image and the motion information of the main body, and may include: and the electronic equipment obtains an original motion curve of the camera according to the posture sensor data corresponding to the multi-frame original image. The electronic device obtains a target motion curve of the camera according to the original motion curve of the camera and the subject constraint. The subject constraint is used for enabling the distance between the center of the subject on the target image and the corresponding center of the subject on the target motion curve of the subject to be smaller than or equal to a second preset value.
That is, the electronic device may combine the target motion profile of the subject with the original motion profile of the camera to obtain the target motion profile of the camera.
In another possible design, the electronic device obtains a target motion profile of the camera from an original motion profile of the camera and subject constraints, which may include: and the electronic equipment determines a corresponding main body constraint gear according to the motion amplitude of the main body. And the electronic equipment gradually increases the main body constraint intensity from the initial value to an upper limit value corresponding to the main body constraint gear. The electronic equipment obtains a target motion curve of the camera according to the original motion curve of the camera and the subject constraint intensity.
Therefore, the electronic equipment can dynamically adjust the constraint intensity of the main body according to the motion amplitude of the main body so as to balance the anti-shake effect of the background and the main body, and the image stabilizing effect of the background and the main body is good. And when the intensity of the face constraint is dynamically regulated, the electronic equipment can smoothly change the face constraint intensity so as to avoid mutation of the anti-shake intensity of the background or the main body, avoid mutation of the image effect of the background and the main body, smoothly transition the anti-shake effect of the background and the main body and improve the shooting experience of a user.
In another possible design, the electronic device may obtain motion information of the subject according to the acquired multi-frame original image, and the method may include: and after the electronic equipment meets the preset conditions, acquiring the motion information of the main body according to the acquired multi-frame original image.
That is, the electronic device starts the main body anti-shake mode after a certain condition is satisfied, so as to obtain motion information of the main body, so as to obtain motion information of the camera according to the motion information of the main body and the posture sensor data, and thus, anti-shake processing is performed according to the motion information of the camera.
In another possible design, the preset conditions may include: the motion amplitude of the electronic equipment is larger than or equal to a third preset value; the area occupation ratio of the main body on the original image is larger than or equal to a fourth preset value; the number of frames in which the original image of the subject continuously appears is greater than or equal to a fifth preset value.
That is, when the motion amplitude of the electronic device is large, the main body occupies a large area, and the main body continuously appears on the original image, the electronic device obtains the motion information of the camera according to the motion information of the main body and the posture sensor data, so that the anti-shake processing is performed according to the motion information of the camera.
In another possible design, the electronic device obtains motion information of the camera according to the gesture sensor data corresponding to the multi-frame original image and the motion information of the main body, and may include: the electronic equipment obtains the motion information of the camera according to the gesture sensor data corresponding to the N frames of original images and the motion information of the main body, N is an integer larger than 1, N=N1+I+N2, N1 and I are positive integers, and N2 is a non-negative integer. The electronic device calculates an image stabilizing transformation matrix of a first original image according to the motion information of the camera, and the image stabilizing transformation matrix comprises the following components: the electronic equipment calculates an image stabilizing transformation matrix of an I frame original image according to a target pose of the camera corresponding to the N frame original image on the motion information of the camera, wherein the I frame original image is a first original image, the image stabilizing transformation matrix of the I frame original image is used for obtaining an I frame target image, and a starting frame of the I frame original image corresponds to an N1+1st frame original image in the N frame original images.
That is, the electronic device may calculate the camera pose and the image stabilizing transformation matrix corresponding to the original image according to the original image before and the original image after the certain original image, so that the changes between the camera poses corresponding to different original images are smoother, and the image stabilizing effect is improved.
In another possible design, N2 is 0 when the target image is a preview image or a recorded image.
Therefore, on the preview interface and the shooting interface, the electronic equipment does not need to calculate the camera pose and the image stabilizing transformation matrix corresponding to the original image according to the original image after the original image, so that the preview image and the recorded image corresponding to the original image can be processed and displayed in real time.
In another possible design, the method may further include: if the preset condition is not met, the electronic equipment obtains the motion information of the camera according to the gesture sensor data corresponding to the multi-frame original image. The electronic equipment calculates an image stabilizing transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in a plurality of frames of original images; and carrying out image transformation on the first original image according to the image stabilizing transformation matrix to obtain a target image.
In the scheme, if the main body anti-shake condition is not met, the electronic equipment can obtain the motion information of the camera according to the attitude sensor data without combining the motion information of the main body to obtain the motion information of the camera, so that the main body anti-shake processing is not needed.
In another possible design, the method may further include: if the preset condition is met, the electronic equipment prompts the user to start the target anti-shake mode. If the preset condition is not met, the electronic equipment prompts the user to not start the target anti-shake mode.
Thus, the user can conveniently know whether the target anti-shake mode is currently processed, and the target anti-shake mode can be the main anti-shake mode.
In another possible design, before the electronic device calculates the image stabilization transformation matrix of the first original image according to the motion information of the camera, the method may further include: the electronic device calculates a Rolling Shutter (RS) compensation amount of the first original image based on data of the attitude sensor. The electronic device calculating an image stabilizing transformation matrix of the first original image according to the motion information of the camera may include: the electronic device calculates a rotation compensation amount of the first original image according to the motion information of the camera. The electronic device calculates an image stabilizing transformation matrix of the first original image according to the rotation compensation quantity and the RS compensation quantity.
The RS compensation amount is used for RS correction, and the scheme can be used for carrying out anti-shake processing according to the motion information of the camera and RS correction at the same time so as to effectively correct the jelly effect of the background on the original image.
In another possible design, the electronic device calculating the image stabilization transformation matrix of the first original image according to the rotation compensation amount and the RS compensation amount may include: if the electronic equipment determines to enter the panning mode, the electronic equipment reduces the RS compensation amount according to the panning amplitude. The panning amplitude is inversely related to the magnitude of the included angle between the main axis of motion of the camera and the target coordinate axis. The electronic device calculates an image stabilizing transformation matrix of the first original image according to the rotation compensation amount and the reduced RS compensation amount.
The background on the original image and the stability of the main body are mutually conflicting, so that the electronic device can adjust the RS compensation amount according to the panning amplitude when carrying out RS correction on the background, thereby reducing or avoiding the distortion of the human face of the main body as much as possible.
In another possible design, the subject is a human face.
In the scheme, the electronic equipment can acquire the motion information of the camera by combining the motion information of the face of the main body and the gesture sensor data, so that anti-shake processing is performed according to the motion information of the camera, and anti-shake is realized on the face of the main body and the background at the same time, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved.
On the other hand, the embodiment of the application provides a shooting device, and the shooting device is contained in electronic equipment. The device has the function of realizing the behavior of the electronic equipment in any one of the aspects and the possible designs, so that the electronic equipment executes the video recording method executed by the electronic equipment in any one of the possible designs of the aspects. The functions can be realized by hardware, and can also be realized by executing corresponding software by hardware. The hardware or software includes at least one module or unit corresponding to the functions described above. For example, the apparatus may comprise an acquisition unit, a processing unit, and the like.
In yet another aspect, an embodiment of the present application provides an electronic device, including: a camera for capturing images; a screen for displaying an interface, one or more processors; and a memory in which the code is stored. The code, when executed by an electronic device, causes the electronic device to perform the video recording method performed by the electronic device in any of the possible designs of the above aspects.
In yet another aspect, an embodiment of the present application provides an electronic device, including: one or more processors; and a memory in which the code is stored. The code, when executed by an electronic device, causes the electronic device to perform the video recording method performed by the electronic device in any of the possible designs of the above aspects.
In yet another aspect, embodiments of the present application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the video recording method of any one of the possible designs of the above aspects.
In yet another aspect, embodiments of the present application provide a computer program product which, when run on a computer, causes the computer to perform the video recording method performed by the electronic device in any of the possible designs of the above aspects.
In yet another aspect, an embodiment of the present application provides a chip system that is applied to an electronic device. The system-on-chip includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a circuit; the interface circuit is used for receiving signals from the memory of the electronic device and sending signals to the processor, wherein the signals comprise computer instructions stored in the memory; the computer instructions, when executed by a processor, cause the electronic device to perform the video recording method of any of the possible designs of the above aspects.
The corresponding advantages of the other aspects mentioned above may be found in the description of the advantages of the method aspects, and are not repeated here.
Drawings
Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application;
FIG. 2 is a flowchart of a video recording method according to an embodiment of the present application;
FIG. 3 is a set of interface schematic diagrams provided in an embodiment of the present application;
fig. 4 is a schematic view of a face feature point provided in an embodiment of the present application;
FIG. 5 is a schematic illustration of the effect of a constraint provided by an embodiment of the present application;
fig. 6A is a flowchart for adjusting the face constraint intensity according to an embodiment of the present application;
fig. 6B is a flowchart of face anti-shake processing provided in an embodiment of the present application;
FIG. 7A is a flowchart illustrating a determination that a fourth condition is met according to an embodiment of the present application;
FIG. 7B is a flowchart for reducing the RS compensation amount according to an embodiment of the present application;
fig. 8A is a schematic diagram of a panning scenario provided in an embodiment of the present application;
FIG. 8B is a schematic diagram of a timing relationship between an original image and a preview image according to an embodiment of the present disclosure;
FIG. 9A is a schematic diagram of a set of preview interfaces provided in an embodiment of the present application;
FIG. 9B is a schematic diagram of another set of preview interfaces provided by embodiments of the present application;
FIG. 9C is a schematic diagram of another set of preview interfaces provided by embodiments of the present application;
FIG. 9D is a schematic diagram of another set of preview interfaces provided by embodiments of the present application;
fig. 10A is a schematic diagram of a group of shooting interfaces according to an embodiment of the present application;
fig. 10B is a schematic view of another group of shooting interfaces according to an embodiment of the present disclosure;
fig. 10C is a schematic view of another group of shooting interfaces according to an embodiment of the present disclosure;
fig. 10D is a schematic view of another group of shooting interfaces according to an embodiment of the present disclosure;
FIG. 11A is a schematic diagram of a set of video images according to an embodiment of the present application;
FIG. 11B is a schematic view of another set of video images provided in an embodiment of the present application;
FIG. 11C is a schematic view of another set of video images provided in an embodiment of the present application;
FIG. 11D is a schematic view of another set of video images provided in an embodiment of the present application;
fig. 12 is a schematic diagram of a prompt interface provided in an embodiment of the present application;
FIG. 13 is a flowchart of another video recording method according to an embodiment of the present application;
FIG. 14 is a flowchart of another video recording method according to an embodiment of the present application;
FIG. 15 is a flowchart of another video recording method according to an embodiment of the present application;
FIG. 16 is a schematic illustration of another interface provided in an embodiment of the present application;
fig. 17 is a flowchart of another video recording method according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Wherein, in the description of the embodiments of the present application, "/" means or is meant unless otherwise indicated, for example, a/B may represent a or B; "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" means two or more than two.
The terms "first" and "second" are used below for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present embodiment, unless otherwise specified, the meaning of "plurality" is two or more.
In the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as examples, illustrations, or descriptions. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.
When recording video by using an electronic device, the shot image is easy to shake and blur due to the shake of a user's hand, the shake of the electronic device, or the movement of a shot object. Especially, when the photographed moving body is closer to the camera of the electronic device, the relative movement of the moving body is larger, the shake of the body image is more obvious, and the user experience is poor. For example, in a front-end self-timer video scene, the face is closer to the camera of the electronic device, and the shake of the face image is more obvious when the face moves. In one technical scheme, electronic equipment carries out anti-shake processing to the image according to gyroscope data, can realize better anti-shake effect to the background in the shooting scope like this, but be difficult to realize the anti-shake to the motion main part in the shooting scope, user's shooting experience is relatively poor.
The embodiment of the application provides a video recording method and equipment, which can be applied to electronic equipment, and can combine motion information of a shot main body, attitude sensor data and other multi-source information to perform anti-shake processing under a video recording scene, so that anti-shake is realized on the motion main body and a background, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved.
The video recording method provided by the embodiment of the application can be used for a rear video recording scene, can also be used for a front video recording scene, and is not limited. After the photographing function is started, the electronic device may calculate the motion information of the camera in combination with the motion information of the photographed subject and the posture sensor data, thereby performing an image anti-shake process according to the motion information of the camera.
For example, the electronic device may be a mobile terminal such as a mobile phone, a tablet computer, a wearable device (e.g. a smart watch), a vehicle-mounted device, an augmented reality (augmented reality, AR)/Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), or a professional camera, and the embodiment of the present application does not limit the specific type of the electronic device.
By way of example, fig. 1 shows a schematic diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.
It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.
The controller may be a neural hub and a command center of the electronic device 100, among others. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.
A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it may be called directly from memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.
The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
The display screen 194 is used to display images, videos, and the like. The display 194 includes a display panel. The display panel may employ a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED) or an active-matrix organic light-emitting diode (matrix organic light emitting diode), a flexible light-emitting diode (flex), a mini, a Micro led, a Micro-OLED, a quantum dot light-emitting diode (quantum dot light emitting diodes, QLED), or the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, N being a positive integer greater than 1. In embodiments of the present application, the display 194 may be used to display preview and capture interfaces, etc. in a video recording mode.
The electronic device 100 may implement photographing functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.
The ISP is used to process data fed back by the camera 193. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, so that the electrical signal is converted into an image visible to naked eyes. ISP can also optimize the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in the camera 193.
The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.
The cameras 193 may include cameras with different focal segments, such as ultra-wide-angle cameras, and tele cameras with equivalent focal lengths ranging from small to large. The camera with smaller equivalent focal length has larger visual field range and can be used for shooting larger pictures such as scenery and the like. The camera with larger equivalent focal length has smaller visual field range, can be used for shooting remote objects, and has smaller shooting area.
In addition, the camera 193 may also include a depth camera for measuring an object distance of an object to be photographed, and other cameras. For example, the depth camera may include a three-dimensional (3 d) depth camera, a time of flight (TOF) depth camera, a binocular depth camera, or the like.
The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, or the like.
Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: dynamic picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent awareness of the electronic device 100 may be implemented through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, pattern recognition, machine self-learning, and the like.
The internal memory 121 may be used to store computer-executable program code that includes instructions. The processor 110 executes various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data created during use of the electronic device 100 (e.g., audio data, phonebook, etc.), and so on. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like.
In the embodiment of the present application, the processor 110 may obtain the motion information of the camera by operating the instruction stored in the internal memory 121 in combination with the motion information of the photographed subject and the gesture sensor in the video recording scene, so as to perform anti-shake processing according to the motion information of the camera, so as to implement anti-shake for the moving subject and the background at the same time, improve the image stabilizing effect of the video image, and improve the photographing experience of the user.
The gyro sensor 180B may be used to determine a motion gesture of the electronic device 100. In some embodiments, the angular velocity of electronic device 100 about three axes (i.e., x, y, and z axes) may be determined by gyro sensor 180B, i.e., motion information, such as rotational information, of electronic device 100 may be obtained. The gyro sensor 180B may be used for photographing anti-shake. For example, when the shutter is pressed, the gyro sensor 180B detects the shake angle of the electronic device 100, calculates the distance (i.e., compensation amount) that the lens module needs to compensate according to the angle, and makes the lens counteract the shake of the electronic device 100 through the reverse motion, so as to realize anti-shake. The gyro sensor 180B may also be used for navigating, somatosensory game scenes. The rotation information of the electronic device 100 may be rotation information of the camera 193 or rotation information of the camera module. Wherein the camera module includes a camera head 193. Hereinafter, the camera module is simply referred to as a camera.
The acceleration sensor 180E may detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). The magnitude and direction of gravity may be detected when the electronic device 100 is stationary. The electronic equipment gesture recognition method can also be used for recognizing the gesture of the electronic equipment, and is applied to horizontal and vertical screen switching, pedometers and other applications.
A distance sensor 180F for measuring a distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, the electronic device 100 may range using the distance sensor 180F to achieve quick focus.
The touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is for detecting a touch operation acting thereon or thereabout. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type. Visual output related to touch operations may be provided through the display 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 at a different location than the display 194.
In embodiments of the present application, in video recording scenarios, camera 193 may be used to capture images; the display 194 may be used to display preview and capture interfaces, etc.; attitude sensor data such as a gyro sensor 180B (also referred to as a gyroscope) may be used to obtain rotation information of the camera; the processor 110 may obtain the motion information of the photographed subject by executing the instructions stored in the internal memory 121, thereby obtaining the motion information of the camera by combining the motion information of the subject and the posture sensor data, and performing anti-shake processing according to the motion information of the camera, so as to simultaneously implement anti-shake for the moving subject and the background, improve the image stabilizing effect of the video image, and improve the photographing experience of the user.
It is understood that the posture sensor is used to detect the posture of the electronic device 100 to obtain the rotation information of the electronic device 100. For example, the attitude sensor may be a sensor such as a gyroscope or a three-axis electronic compass, and the type of the attitude sensor is not limited in the embodiments of the present application.
The following will describe a video recording method provided in the embodiment of the present application by taking an example in which an electronic device is a mobile phone having a structure shown in fig. 1, an attitude sensor is a gyroscope, and a video recording scene is a front-mounted self-timer scene. As shown in fig. 2, the video recording method may include:
201. After the shooting function is started, the mobile phone enters a video recording mode, and an original image is acquired according to a preset frame rate in a preview state.
When the user wants to shoot an image by using the mobile phone, the shooting function of the mobile phone can be started. For example, the mobile phone may launch a camera application, or launch other applications with photographing or video recording functions (such as an AR application like tremble or river map cyberverse) to launch the photographing function of the mobile phone.
In some embodiments, the video recording method provided in the embodiments of the present application may be applied to a video recording mode, and after a shooting function is started, the mobile phone may enter the video recording mode, so as to perform anti-shake processing in the video recording mode by combining motion information of a subject to be shot with gyroscope data.
Illustratively, after detecting the user's click on the camera icon 301 shown in fig. 3 (a), the mobile phone starts the photographing function of the camera application and displays a preview interface in the photographing mode as shown in fig. 3 (b). After detecting the user clicking the control 302 shown in fig. 3 (b), the mobile phone enters a video recording mode as shown in fig. 3 (c).
For another example, the mobile phone displays a desktop or a non-camera application interface, starts a shooting function after detecting a voice command indicating video recording by a user, and enters a video recording mode as shown in (c) of fig. 3.
It should be noted that, the mobile phone may also enter the video mode in response to other touch operations, voice instructions, or shortcut gestures of the user, which are not limited in the embodiment of the present application.
In other embodiments, in the video mode, the mobile phone does not automatically combine the motion information of the main body with the gyroscope data to perform anti-shake processing; after detecting the preset operation 1 of the user, the mobile phone starts anti-shake processing by combining the motion information of the main body and the gyroscope data. The preset operation 1 is used for indicating the mobile phone to perform anti-shake processing on the main body. In an exemplary video mode, the preview interface includes a main body anti-shake control, and after detecting that a user clicks the main body anti-shake control, the mobile phone starts a main body anti-shake process, so that anti-shake processing is performed by combining motion information of the main body and gyroscope data.
After the mobile phone enters a video mode, an original image is acquired according to a preset frame rate in a preview state.
In other embodiments, the video recording method provided in the embodiments of the present application is applied to a specific target shooting mode other than the video recording mode, and the mobile phone may enter the target shooting mode after starting the shooting function, so as to perform anti-shake processing in combination with the motion information of the main body and the gyroscope data in the target shooting mode.
In the following embodiments, the anti-shake processing is described by taking the mobile phone as an example in which the motion information of the main body and the gyroscope data are combined in the video mode.
202. And the mobile phone determines whether the first condition is met according to the S frame original image, wherein S is an integer greater than 1. If the first condition is satisfied, the mobile phone executes steps 203-204; if the first condition is not satisfied, the mobile phone performs step 205.
In the preview state of the video mode, after the mobile phone collects the S-frame original image, it may be determined whether the first condition is satisfied according to the original image 1-S (i.e., according to the S-frame original image). For example, S is 2 frames, 20 frames, 45 frames, or the like. If the first condition is met, it can be determined that the mobile phone moves, the occupied area of the shot main body is large, the main body continuously appears on the original image, and main body anti-shake processing is needed, so that the mobile phone can enter a main body anti-shake mode, and therefore movement information of the main body is acquired, and the main body is subjected to anti-shake processing by combining the movement information of the main body. If the first condition is not met, the main body anti-shake processing is not needed, the main body anti-shake mode is needed to be entered, and the mobile phone only carries out the anti-shake processing on the background. This first condition may also be referred to as a subject anti-shake condition.
In the embodiment of the present application, the subject to be photographed is an object or subject of interest to the user within the photographing range. Other objects than the subject within the photographing range may be referred to as a background. Correspondingly, the image acquired by the mobile phone can also comprise a main body and a background.
For example, in some embodiments, the body is an object of a preset type. For example, the predetermined type of object is a human face or an animal, or the like.
In other embodiments, the subject is an object with an image on the preview image in a central region. The subject of interest to the user will typically be facing the camera, so that the subject's image on the preview image is typically in the center region.
In other embodiments, the subject is an object whose image on the preview image is near the center region and whose area is greater than a preset threshold 1. The subject of interest to the user will typically be facing and closer to the camera such that the image of the subject on the preview image is closer to the center area and has an area greater than the preset threshold 1.
In other embodiments, the subject is a pre-set type of object with the image on the preview image near the center area.
In other embodiments, the subject is a preset type of object having an area greater than a preset threshold 1 with the image on the preview image near the center area.
In other embodiments, the subject is a predetermined type of object with the smallest depth and the image on the preview image is near the center area.
In other embodiments, the body is an object specified by a user through a preset touch operation, a preset gesture operation, or a preset voice command.
Wherein the body may be an object. Alternatively, the body may be a unitary body formed of a plurality of objects that remain relatively stationary.
In the following embodiments of the present application, a main body is mainly taken as a face for illustration.
When a plurality of faces appear in the shooting range, the main body can be the face nearest to the camera, the face with the largest area, the face closest to the middle position on the image picture, the face designated by the user, and the like.
For example, when the subject is a human face, the first condition may include the following sub-condition 1 to sub-condition 3, or any one of the sub-conditions 1 to 3 or a combination of any two of the sub-conditions.
Wherein, sub-condition 1: the motion amplitude of the mobile phone is larger than or equal to a preset value 1. The mobile phone carries out the main body face anti-shake processing only when the mobile phone determines that the motion amplitude of the mobile phone is large through the motion sensors such as an accelerometer or a gyroscope; thereby avoiding the problem that the background moves along with the face of the main body due to the anti-shake processing of the face of the main body when the mobile phone is stationary.
Sub-condition 2: the area ratio of the main body face on the original image is larger than or equal to a preset value 2. In the case where the main body face is smaller than the preset value 2, the main body face area is small, and the user may want a better background effect, so that the first condition is not satisfied, and thus the main body face anti-shake is not required. In the case where the main body face ratio is greater than or equal to the preset value 2, the user may want to mainly photograph the main body face, and the main body face anti-shake process may be performed.
Sub-condition 3: the number of frames of the original image in which the face of the subject continuously appears is greater than or equal to a preset value 3, the preset value 3 being an integer greater than 1. For example, the preset value 3 is L, the main face continuously appears on the original images of the continuous L frames, and the duration of the continuous appearance of the main face on the original images is greater than or equal to the preset duration. That is, the mobile phone performs the main face anti-shake processing only after the main face appears in a continuous period of time, so as to prevent the related problems caused by the frequent start or exit of the main face anti-shake when the main face frequently enters or exits the shooting picture.
If the sub-conditions 1-3 are met, the mobile phone determines that the first condition is met, so that the mobile phone can indicate that the mobile phone moves, the area occupied by the main body face is relatively large, the main body face continuously appears on the original image, and the anti-shake processing of the main body face is needed. If any one of the sub-condition 1, the sub-condition 2 or the sub-condition 3 is not satisfied, the mobile phone determines that the first condition is not satisfied, so that whether other sub-conditions in the first condition are satisfied is not continuously determined, and the mobile phone determines that the anti-shake processing of the main face is not required.
In some embodiments, the mobile phone may determine whether the sub-condition 1 is satisfied according to the S frame original image, and if the sub-condition 1 is satisfied, determine whether the sub-condition 2 and the sub-condition 3 are satisfied, so as to further determine whether the first condition is satisfied; if the sub-condition 1 is not satisfied, the mobile phone determines that the first condition is not satisfied.
In the preview state, if the number of frames of the original image acquired by the mobile phone is greater than S frames, the mobile phone determines whether the first condition is met according to the recently acquired S frames of the original image. That is, after the S-frame image is collected, each time the mobile phone collects one frame of original image, it is determined whether the first condition is satisfied according to the latest S-frame historical original image. If the first condition is met and the main body anti-shake mode is entered currently, the mobile phone continues to perform anti-shake processing in the main body anti-shake mode. If the first condition is met and the mobile phone does not enter the main body anti-shake mode currently, the mobile phone enters the main body anti-shake mode. If the first condition is not met and the anti-shake mode is entered currently, the mobile phone exits the main anti-shake mode.
In some embodiments, after the mobile phone enters/exits the main body anti-shake mode, the user may be prompted by means of display information, voice broadcast or vibration, etc.
203. If the first condition is met, the mobile phone acquires the motion information of the face of the main body according to M frames of original images, wherein M is an integer larger than 1.
If the first condition is met, it indicates that the anti-shake processing of the face of the main body needs to be performed, so that the mobile phone can start the face anti-shake mode, and then execute steps 203-204, so as to perform the anti-shake processing by combining the motion information of the face of the main body and the gyroscope data.
In other embodiments, if the mobile phone detects that the user starts the face anti-shake mode, it indicates that the anti-shake processing of the main face is required, so that the mobile phone may start the face anti-shake mode, and then execute steps 203 to 204, so as to combine the motion information of the main face and the gyroscope data to perform the anti-shake processing. For example, the preview interface includes a face anti-shake control, and the mobile phone opens the face anti-shake mode after detecting the operation of clicking the control by the user.
The motion information of the main face is used for representing the motion condition of the main face. For example, the motion information of the main body face may be a target motion curve of the main body face, and after the mobile phone meets the first condition, the mobile phone obtains the target motion curve of the main body face according to the M frames of original images, so as to combine the target motion curve of the main body face with the gyroscope data to perform anti-shake processing.
The following describes an example of a mobile phone acquiring a target motion curve of a human face of a main body according to M frames of original images.
And the mobile phone detects face feature points of the M frames of original images according to a face feature detection algorithm. For example, referring to fig. 4, a face includes 71 feature points in total. The mobile phone takes the coordinate mean value of each characteristic point as the face center. The face center of the M frames of original images forms an original motion curve of the main face, and may also be referred to as an original motion track of the main face or an original motion path of the main face.
The mobile phone performs smoothing processing on the original motion curve of the main body face, so as to obtain a smooth target motion curve of the main body face. The target motion curve of the main human face meets the following second condition:
(1) The target motion curve of the subject face is continuous everywhere. That is, the movement track of the center of the face is smoother, and the change cannot be too severe.
(2) The target motion curve of the subject's face is first and second order conductive, and the curvature is as small as possible (e.g., less than or equal to a preset threshold 2). That is, the change of the speed and acceleration of the target motion curve of the face of the subject is smoother and not too severe.
(3) The distance between the center of the face on the target motion curve of the subject face and the center of the face on the original motion curve of the subject face is less than or equal to a preset value 4. That is, the center of the face on the target motion curve of the subject face cannot deviate too far from the original motion curve of the subject face, and the center of the face on the target motion curve of the subject face should be in the vicinity of the original motion curve of the subject face.
The mobile phone can adopt various methods to carry out smoothing treatment on the original motion curve of the main body face so as to obtain the target motion curve of the main body face. For example, the mobile phone can obtain a smooth target motion curve of the main face through a quadratic programming method.
The motion of the face of the subject represented on the multi-frame original image can be represented by using 2 groups of face coordinate sequences with equal time intervals represented by formula 1:
assuming that the face center coordinate sequences are all on a curve, the mobile phone can determine a curve meeting the second condition according to the objective function 1 expressed by the formula 2 by a quadratic programming methodA smooth target motion profile of the subject face. The point on the target motion curve of the human face of the main body passes through C' k Sequence to represent.
Wherein r= [ r ] u ,r v ] T ∈R 2×1 ,c k ＝[u k ,v k ] T ,c' k ＝[u' k ,v' k ] T 。
In formula 2, s.t. means "so that" is satisfied, C' k Representing the coordinates of the center of the face on the original image of the kth frame in the target motion curve of the expected smoothed main face. C (C) k Representing the coordinates of the center of the face on the kth frame of the original image on the original motion curve of the face of the subject before smoothing. Omega 1 ，ω 2 Is a preset super parameter. r represents a difference threshold value of the centers of the face before and after smoothing, ru and rv represent a row coordinate threshold value and an ordinate threshold value of the difference, respectively.
Formula 2 may also be represented by formula 3:
minω 1 J 1 +ω 2 J 2 3
Wherein J is 1 The first derivative of the target motion curve representing the subject's face,
J 2 second derivative of target motion curve representing subject face,/->
/>
In step 203, the mobile phone acquires a target motion curve of the main face according to the M frames of original images after the first condition is satisfied, so as to combine the target motion curve of the main face with the gyroscope data to perform anti-shake processing. For example, after the mobile phone obtains the target motion curve of the main face according to the M frames of original images for the first time, each frame of original images collected in the following process can be combined with the M frames of original images collected recently, so that the mobile phone can continuously obtain the following track points on the target motion curve of the main face.
204. And the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N frames of original images and the motion information of the main body face, wherein N is an integer larger than 1.
Wherein the motion information of the camera is used for representing the motion condition of the camera. Since the camera module includes a camera, the movement of the camera is that of the camera, and the camera module is usually fixed on the mobile phone to move along with the mobile phone, the movement of the camera is that of the mobile phone. For example, the motion information of the camera is a target motion curve of the camera, which may also be referred to as a target motion track or path of the camera, that is, a target motion curve of the mobile phone or a target motion track or path of the mobile phone. The motion information of the camera is used for calculating an image stabilizing transformation matrix of the original image subsequently, and the image stabilizing transformation matrix is used for carrying out motion compensation on the original image through deformation (warp) transformation, so that the effects of image anti-shake and image stabilizing are achieved.
The following description will take the motion information of the camera as the target motion curve of the camera, and the mobile phone obtains the target motion curve of the camera as an example.
And after the mobile phone meets the first condition, acquiring an original motion curve of the camera according to the gyroscope data corresponding to the N frames of original images. And the gyroscope data corresponding to the N frames of original images are gyroscope data corresponding to the acquisition time period of the N frames of original images. And the mobile phone processes the original motion curve of the camera according to the first constraint condition, so as to obtain the target motion curve of the camera. Wherein the first constraint is associated with motion information of a face of the subject. That is, the mobile phone obtains a target motion curve of the camera according to the gyroscope data and the motion information of the face of the main body. The target motion profile of the camera satisfies the following third condition:
(1) The target motion profile of the camera is continuous throughout. That is, the movement track of the camera is smooth, and the track change cannot be too severe.
(2) The target motion profile of the camera is first, second, third order conductive and has as small a curvature as possible (e.g., less than or equal to the preset threshold 3). That is, the speed, acceleration, and rate of change of acceleration (i.e., jerk) of the target motion profile of the camera are smooth and cannot be too severe.
(3) The image stabilizing transformation matrix is calculated according to the target motion curve of the camera, so that when the original image is subjected to motion compensation according to the image stabilizing transformation matrix, the compensated original image cannot exceed the boundary of clipping (after crop) to generate black edges.
(4) When the original image is subjected to motion compensation according to the target motion curve of the camera, the distance between the center of the face after warp transformation on the original image and the center of the face on the target motion curve of the main face is smaller than or equal to a preset value 5. That is, the center of the face on the target motion curve of the subject face cannot deviate too far from the original motion curve of the subject face, and the center of the face on the target motion curve of the subject face should be in the vicinity of the original motion curve of the subject face.
The mobile phone can adopt various methods to carry out smoothing processing on the original motion curve of the camera so as to obtain the target motion curve of the camera. For example, the mobile phone can obtain a smooth target motion curve of the camera through a quadratic programming method.
Wherein points on the original motion curve of the camera represent the original pose of the camera, which can be represented using a sequence of rotation angles at 3 sets of equal time intervals represented by equation 4:
wherein, the liquid crystal display device comprises a liquid crystal display device,
representing the rotation angles in the three directions integrated from the gyroscope, respectively. The mobile phone can determine a smooth target motion curve of the camera meeting the third condition according to the target function 2 represented by the formula 5 and the first constraint condition represented by the formula 6 by using a quadratic programming method.
min w 1 J 1 +w 2 J 2 +w 3 J 3 5. The method is to
Similar to formula 3 above, J i The ith derivative representing the target motion profile of the camera, as shown in equation 6, the first constraint may include:
after the first condition is met, the mobile phone adopts the algorithm provided by the embodiment of the application, and processes according to the acquired original pose of the camera corresponding to the N frames of original images for the first time, so that the optimized target pose of the camera corresponding to the I frames of original images (I is a positive integer) is obtained, and the transition between the target poses of the camera corresponding to the I frames of original images is smoother. The optimized target pose of the camera forms a target motion curve of the camera, and the optimized target pose of the camera is a point on the target motion curve of the camera. And then, after the mobile phone acquires the subsequent I-frame original image, processing according to the original pose of the camera corresponding to the recently acquired historical N-frame original image to obtain the optimized target pose of the camera corresponding to the I-frame original image.
Wherein n=n1+i+n2, N1 and I are positive integers and N2 is a non-negative integer. According to the method, the target pose of the optimized camera corresponding to the I frame original image after the N1 frame original image is output according to the original pose of the camera corresponding to the N frame original image. For example, N is 45, N1 is 15, N2 is 27, I is 3, the algorithm processes the original pose of the camera corresponding to the 1 st-45 th frame of original image for the first time, and outputs the optimized target pose of the camera corresponding to the 16 th, 17 th and 18 th frame of original image. That is, the mobile phone obtains the target pose of the optimized camera corresponding to the I-frame original image according to the I-frame original image and the original poses of the camera corresponding to the previous N1-frame original image and the next N2-frame original image.
The sub constraint condition (1) in the formula 6 is used for taking the target pose of the camera corresponding to the optimized partial original image as input to optimize the target pose of the camera corresponding to the adjacent subsequent frames of original images, so that the overall transition of the target pose of the optimized camera is smoother, and not only the local transition among the target poses of the cameras corresponding to the I-frame original images is smoother. For example, N is 45, N1 is 15, N2 is 27, i is 3, the algorithm processes the target pose of the optimized camera corresponding to the 4 th-48 th frame of original image for the second time, and outputs the target pose of the optimized camera corresponding to the 19 th, 20 th and 21 th frames of original images. In order to ensure that the target pose of the optimized camera corresponding to the 18 th and 19 th frame of original images after optimization does not jump, the transition is smooth, and the mobile phone can replace the original positions of the cameras corresponding to the 17 th and 18 th frame of original images used in the second optimization with the target pose of the optimized camera corresponding to the 17 th and 18 th frame of original images after the first optimization based on the sub constraint condition (1).
The sub constraint (2) in equation 6 is used to make the original image not exceed the preset clipping boundary after warp transformation based on the target motion curve of the camera. For example, referring to fig. 5, a block 501 represents the range of the original image, a block 502 represents the clipping retention range during the anti-shake processing, and the preset clipping boundary may be P shown in fig. 5 w And P h A defined boundary. For example, in FIG. 5, P i Representing pixel points before warp transformation on an original image, P i ' represents pixel points subjected to warp transformation according to motion information of a camera. Four corner points of the edge of the original image, namely four vertex points of the block 501, cannot exceed P after warp transformation w And P h The boundary being defined, i.e. not being located at P w And P h The clipping represented by block 502 outside the defined boundary remains within the clipping retention range so that the image obtained by clipping does not leave a black border. For another example, the face center P on the original image c P after warp transformation c ' cannot exceed boundary P w And P h I.e. the clipping retention range indicated by block 503 cannot be exceeded so that the image obtained by clipping does not leave a black border.
In the sub-constraint (3) in equation 6, the function pi (x f ，w，w f ) Representing the original imageCoordinates X of face center f Calculating according to the pose of the camera obtained by the objective function 2 and the sub constraint conditions (1) - (2) and a transformation matrix w to obtain the projection position of the center of the face after warp transformation; x is x h Representing the ideal position of the face center on the target motion curve of the smoothed main face; epsilon represents the coordinate offset threshold between the center projection position and the ideal position of the face. That is, the sub constraint condition (3) in the formula 6 is used to make the coordinate offset between the central projection position and the ideal position of the face be smaller than epsilon, so that the position of the face on the adjacent original image is stable, the face shake is small, and the image stabilizing effect is good. Epsilon determines the degree of face stabilization in the image frame after image stabilization.
For example, the center point P of block 504 0 Is the face center on the smoothed subject face target curve. For the face center P on the original image c Face center P obtained after warp transformation c ' falls within the range of the epsilon-bounded box 504 in child constraint (3).
As can be seen from the sub constraint (3), since the target motion profile of the camera is obtained based on the target motion profile of the subject face, the original image corresponding to the point on the target motion profile of the subject face should include the point on the target motion profile of the camera. Thus, in some embodiments, M may be greater than or equal to N.
After the mobile phone obtains the target motion curve of the camera according to the N frames of original images for the first time, each time the I frames of original images are collected subsequently, the mobile phone can be combined with the N frames of recently collected original images to obtain subsequent I points on the target motion curve of the camera.
After the mobile phone determines the motion information of the camera, the rotation compensation amount is determined according to the motion information of the camera, so that the image stabilizing transformation matrix of the original image is calculated according to the rotation compensation amount. The image stabilizing transformation matrix is a homography matrix of transforming (warp) the original image. The image stabilizing transformation matrix is used for carrying out motion compensation on an original image through warp transformation, so that the effects of image anti-shake and image stabilizing are achieved. For example, the mobile phone may perform warp transformation on the original image according to the image stabilizing transformation matrix to obtain a preview image, and display the preview image on the preview interface. Therefore, when the mobile phone performs warp transformation according to the motion information of the camera, the motion information of the main face is combined with the motion information of the camera, so that the main face can be subjected to anti-shake when the background is subjected to anti-shake, the image stabilizing effect of the image is improved, and the shooting experience of a user is improved.
It will be appreciated that the background and subject stability on the original image are often conflicting upon anti-shake processing. If the anti-shake processing intensity of the background is high, the anti-shake effect of the main body is poor; if the anti-shake processing intensity of the main body is high, the anti-shake effect of the background is poor.
Therefore, in some embodiments, after the mobile phone starts the face anti-shake mode, the strength of face constraint can be dynamically adjusted according to the motion amplitude of the main face, so as to balance the anti-shake effects of the background and the main face, and the image stabilizing effects of the background and the main face are good. And when the intensity of the face constraint is dynamically regulated, the mobile phone can smoothly strengthen and weaken the face constraint intensity so as to avoid mutation of the anti-shake intensity of the background or the main face and mutation of the image effect of the background and the main face, so that the anti-shake effect of the background and the main face is smoothly transited, and the shooting experience of a user is improved.
For example, in one possible implementation, referring to fig. 6A, after the face anti-shake mode is turned on, the process of adjusting the face constraint intensity according to the face motion amplitude by the mobile phone may include:
204a, the mobile phone determines corresponding face constraint gear 1 according to the motion amplitude of the face of the main body, wherein each gear corresponds to an upper limit value of face constraint intensity.
The motion amplitude of the human face of the main body is measured by the variance of the center of the human face of the main body on the M frames of original images. The larger the variance, the larger the motion amplitude of the subject face. The larger the motion amplitude of the main body face is, the higher the corresponding face constraint gear is, and the stronger the face constraint is; the smaller the motion amplitude of the main body face is, the lower the corresponding face constraint gear is, and the weaker the face constraint is.
Wherein the face constraint intensity is controlled by epsilon in the sub constraint condition (3). The larger the epsilon value is, the weaker the face constraint strength is; the smaller the epsilon value, the stronger the face constraint intensity.
In some embodiments, the upper bound values of the face constraint intensity corresponding to different gear positions may correspond to different epsilon values.
204b, the mobile phone gradually increases the human face constraint intensity from the initial value to the upper limit value 1 of the gear 1. Then, the mobile phone performs step 204c or step 204d.
The initial value can be a preset numerical value, and the face constraint intensity corresponding to the initial value is smaller, so that after the mobile phone enters the face anti-shake mode, the face constraint intensity when the face anti-shake processing is just started is smaller, and the image change before and after the anti-shake is smoother.
204c, if the mobile phone determines that the motion amplitude of the face of the main body is increased and corresponds to the face constraint gear 2, gradually increasing the face constraint intensity from the upper limit value 1 corresponding to the gear 1 to the upper limit value 2 corresponding to the gear 2.
The face constraint intensity corresponding to the gear 2 is larger than the face constraint intensity corresponding to the gear 1, and the face constraint intensity corresponding to the upper limit value 2 is larger than the face constraint intensity corresponding to the upper limit value 1. Correspondingly, the epsilon value corresponding to the upper limit value 2 is smaller than the epsilon value corresponding to the upper limit value 1.
That is, when the movement amplitude of the main face is increased (for example, the main object is changed from walking to running), the mobile phone can correspondingly enhance the constraint intensity of the face, so that the main face with larger movement amplitude is subjected to the anti-shake treatment with larger intensity, and the image stabilizing effect of the main face is improved. In addition, the mobile phone needs to gradually, slowly and smoothly enhance the strength of the face constraint so as to enable the image effect of the main face and the background to change slowly after the anti-shake treatment, avoid the mutation of the anti-shake strength of the background or the main face and avoid the mutation of the image effect of the background and the main face.
204d, if the mobile phone determines that the motion amplitude of the face of the main body is reduced and corresponds to the face constraint gear 3, the mobile phone gradually reduces the face constraint intensity from the upper limit value 1 corresponding to the gear 1 to the upper limit value 3 corresponding to the gear 2.
The face constraint intensity corresponding to the gear 3 is smaller than the face constraint intensity corresponding to the gear 2, and the face constraint intensity corresponding to the upper limit value 3 is smaller than the face constraint intensity corresponding to the upper limit value 2. Correspondingly, the epsilon value corresponding to the upper limit value 3 is larger than the epsilon value corresponding to the upper limit value 2.
That is, when the motion amplitude of the main face is reduced, the mobile phone can correspondingly weaken the constraint intensity of the face, so that the main face with smaller motion amplitude is subjected to weaker-intensity anti-shake treatment, and the image stabilizing effect of the background is ensured while the image stabilizing effect of the main face is improved. In addition, the mobile phone needs to gradually, slowly and smoothly weaken the strength of the face constraint so as to enable the image effect of the main face and the background to change slowly after the anti-shake treatment, avoid the mutation of the anti-shake strength of the background or the main face and avoid the mutation of the image effect of the background and the main face.
205. If the first condition is not met, the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N frames of original images.
In step 205, if the first condition is not satisfied, the motion information of the camera is obtained according to the gyroscope data corresponding to the N frames of original images, without combining the motion information of the face of the subject. For example, when the motion information of the camera is a target motion curve of the camera, the target motion curve of the camera is calculated from the target function 2 represented by equation 5 and the second constraint condition. Wherein, unlike the first constraint, the second constraint does not include the above-described sub constraint (3) associated with the motion information of the subject face. For example, the second constraint may include the above-described sub constraint (1) and sub constraint (2). That is, the target motion profile of the camera is not obtained from the motion information of the subject's face. The motion information of the camera is used for calculating an image stabilizing transformation matrix of the original image subsequently, and the image stabilizing transformation matrix is used for carrying out motion compensation on the original image through warp transformation, so that the effects of image anti-shake and image stabilizing are achieved.
If the mobile phone determines that the first condition is not satisfied, it may indicate that the anti-shake processing of the face of the main body is not required, so steps 211 to 214 may be executed to perform the anti-shake processing by using the motion information of the camera obtained according to the gyroscope data corresponding to the N frames of original images, without performing the anti-shake processing in combination with the motion information of the face of the main body.
It should be noted that, if the mobile phone has previously entered the face anti-shake mode, the mobile phone exits the face anti-shake mode after determining that the first condition is not satisfied, so as to execute steps 211-214, and delete the face constraint item corresponding to the sub constraint condition (3) to perform anti-shake processing by using the motion information of the camera obtained according to the gyroscope data.
That is, if the first condition is not satisfied, the subsequent mobile phone calculates an image stabilizing transformation matrix of the original image according to the motion information of the camera, and the mobile phone performs warp transformation processing on the original image according to the image stabilizing transformation matrix to obtain a preview image, and displays the preview image on the preview interface.
In the solution described in the above embodiments of the present application, a flowchart of face anti-shake adjustment performed by a mobile phone may be referred to fig. 6B. As shown in fig. 6B, when the mobile phone determines that the face anti-shake condition is met according to the multi-frame original image, the mobile phone calculates the face movement amplitude; gradually increasing the face constraint to the corresponding constraint intensity upper limit according to the face motion amplitude; if the face movement amplitude changes, gradually increasing the face constraint to a corresponding new constraint intensity upper bound according to the face movement amplitude; if the face anti-shake condition is not met, deleting the face constraint, and exiting the face anti-shake mode.
It can be understood that the rolling shutter of the camera acquires an original image through progressive exposure, and when the object to be shot moves at a high speed or the scanning speed of the camera is not fast enough, the object on the image can tilt, swing or partially expose, and the like, which is called Rolling Shutter (RS) effect or jelly effect. In some embodiments, the mobile phone may also calculate the RS compensation amount to perform RS correction to effectively correct the jelly effect of the background on the original image. Since the background and the main body on the original image are mutually conflicting, when the mobile phone performs RS correction on the background, the distortion of the main body face is reduced or avoided as much as possible by adjusting the size of the RS compensation quantity. Based on this, after the above step 204 and step 205, the method may further include:
206. and the mobile phone calculates RS compensation quantity corresponding to the original image according to the data of the gyroscope.
Wherein the RS compensation amount is used for motion compensation of the original image at the time of warp transformation, thereby performing RS correction. Also, the rolling shutter is exposed row by row, and thus the RS compensation amount includes a compensation amount corresponding to each exposure row, respectively.
The mobile phone can calculate RS compensation quantity corresponding to the original image according to the data of the gyroscope. For example, the center line of the original image exposure line does not need RS compensation, i.e., the RS compensation amount corresponding to the center line is 0. The mobile phone can calculate the RS compensation amounts respectively corresponding to other exposure lines according to the difference value between the time stamp of the other exposure lines and the time stamp of the center line and the difference value between the gyroscope data corresponding to the other exposure lines and the gyroscope data corresponding to the center line.
For another example, the mobile phone may calculate RS compensation amounts of first and last lines of the original image according to the data of the gyroscope, and calculate RS compensation amounts respectively corresponding to other exposure lines according to differences between time stamps of other exposure lines and time stamps of the first and last lines and differences between gyroscope data corresponding to other exposure lines and gyroscope data corresponding to the first and last lines.
207. The handset determines whether the fourth condition is met. If the fourth condition is satisfied, the mobile phone executes step 208; if the fourth condition is not satisfied, the mobile phone executes step 211.
If the fourth condition is met, the fact that the face of the main body and the background have larger relative motion can be indicated, the mobile phone can enter a panning mode, so that RS compensation amount is reduced, and the reduced RS compensation amount and the motion information of the camera are combined to calculate an image stabilizing transformation matrix of the original image, so that warp transformation is carried out.
208. If the fourth condition is met, the mobile phone determines a reduction factor according to the channel amplitude, wherein the reduction factor is used for reducing the RS compensation amount.
Wherein the reduction factor is used to reduce the RS compensation amount per row. For example, the reduction factor may be a reduction ratio k, the reduced RS compensation amount=the original RS compensation amount (1-k). The original RS compensation amount is the unreduced RS compensation amount obtained from the data of the gyroscope in step 206.
If the fourth condition is satisfied, it may indicate that there is a large relative movement between the face and the background of the subject. At this time, if the original image is subjected to warp transformation by using the RS compensation amount obtained according to the data of the gyroscope, the distortion of the face of the main body is easily caused, so that the mobile phone can enter a channel mode, and the RS compensation amount is reduced according to the channel amplitude, so as to reduce or avoid the distortion of the face of the main body due to RS correction as much as possible.
In some embodiments, the panning mode refers to the camera remaining relatively stationary or moving at the same speed as the subject, while shooting the user's body rotation causes the cell phone and camera to rotate, causing the background to change, thereby causing greater relative movement of the subject's face and background.
For example, referring to fig. 7A, the mobile phone determining that the fourth condition is satisfied includes:
208a, the mobile phone determines a main shaft of the rotation motion of the camera according to the gyroscope data corresponding to the multi-frame original image.
The mobile phone calculates the angular velocity of the camera according to the gyroscope data corresponding to the period of collecting continuous multi-frame original images, integrates the angular velocity to obtain the rotation angles of three coordinate axes of x, y and z, and further obtains the main shaft of the rotation motion of the camera according to the rotation angles of the three coordinate axes of x, y and z.
208b, calculating the included angle between the main shaft and the target coordinate axis by the mobile phone.
The target coordinate axis is a preset coordinate axis and is related to the current horizontal/vertical screen state of the mobile phone. For example, if the mobile phone is in a vertical screen state, the target coordinate axis is the y axis with the same gravity direction; if the mobile phone is in the horizontal screen mode, the target coordinate axis is the x axis in the horizontal direction perpendicular to the gravity direction. After the mobile phone determines the main shaft of the camera rotating motion, the included angle between the main shaft and the target coordinate axis can be calculated.
208c, if the included angle is smaller than or equal to the preset value 6, the mobile phone determines that the fourth condition is met.
If the included angle between the main shaft and the target coordinate axis is smaller than or equal to the preset value 6, the included angle between the main shaft and the target coordinate axis can be indicated to be smaller, the camera basically rotates around the target coordinate axis, the main body face and the background have larger relative motion, and the fourth condition is met.
For example, the mobile phone can calculate a cosine value of the included angle between the main axis and the target coordinate axis, if the cosine value is greater than or equal to the preset value 7, it can be indicated that the included angle between the main axis and the target coordinate axis is smaller, the camera basically rotates around the target coordinate axis, and the main body face and the background have larger relative motion, so as to satisfy the fourth condition.
For example, when the mobile phone is in the portrait state, a schematic diagram of the mobile phone about the y-axis panning may be seen in fig. 8A.
And the mobile phone enters a panning mode after the fourth condition is met, and a reduction factor is determined according to the panning amplitude. Wherein the panning amplitude is used to represent the amplitude of rotation of the camera about the target coordinate axis. For example, if the smaller the angle between the main axis and the target coordinate axis or the larger the rotation speed of the camera, the larger the panning amplitude may be indicated. The larger the pass amplitude, the larger the reduction factor. Conversely, if the larger the angle between the main axis and the target coordinate axis or the smaller the rotation speed of the camera, the smaller the panning amplitude, the smaller the reduction factor can be indicated.
In the embodiment of the present application, when performing face distortion correction, the processing flow of the mobile phone to obtain the RS compensation amount after the reduction may also refer to fig. 7B. As shown in fig. 7B, the mobile phone may buffer gyroscope data corresponding to a plurality of frames of original images, calculate a motion principal axis when a face appears in the original images, calculate an included angle between the principal axis and a target coordinate axis, calculate a reduction factor when the included angle is smaller than or equal to a preset value 8, and reduce the RS compensation amount according to the reduction factor.
209. And the mobile phone calculates an image stabilizing transformation matrix of the original image according to the reduced RS compensation quantity and the motion information of the camera.
The mobile phone can calculate a rotation compensation amount according to the motion information of the camera, wherein the rotation compensation amount also comprises compensation amounts corresponding to each exposure row respectively. The mobile phone can add the reduced RS compensation amount to the rotation compensation amount, thereby calculating the image stabilizing transformation matrix of the original image.
For example, the motion information of the camera is a target motion curve of the camera, and the mobile phone can respectively calculate rotation compensation amounts respectively corresponding to the I-frame original images according to target poses of the camera corresponding to the I-frame original images on the target motion curve of the camera. And the mobile phone adds the rotation compensation quantity corresponding to each frame of original image in the I frame of original image with the reduced RS compensation quantity, so as to obtain the stable image transformation matrix corresponding to each frame of original image in the I frame of original image.
For example, N is 45, N1 is 15, i is 3, N2 is 27, the mobile phone processes the original pose of the camera corresponding to the 1 st to 45 th frames of original images, outputs the optimized target pose of the camera corresponding to the 16 th, 17 th and 18 th frames of original images, and calculates the rotation compensation amount corresponding to the 16 th, 17 th and 18 th frames of original images according to the optimized target pose of the camera corresponding to the 16 th, 17 th and 18 th frames of original images. The mobile phone adds the rotation compensation quantity corresponding to each frame of original image in the 16 th, 17 th and 18 th frames of original images and the reduced RS compensation quantity, so as to obtain the stable image transformation matrix corresponding to each frame of original image in the 16 th, 17 th and 18 th frames of original images.
And then, the mobile phone processes the original pose of the camera corresponding to the 4 th to 48 th frames of original images (wherein the original pose corresponding to the 17 th and 18 th frames of original images is replaced by the target pose of the optimized camera), outputs the target pose of the optimized camera corresponding to the 19 th, 20 th and 21 th frames of original images, and calculates the rotation compensation quantity corresponding to the 19 th, 20 th and 21 th frames of original images according to the target pose of the optimized camera corresponding to the 19 th, 20 th and 21 th frames of original images. The mobile phone adds the rotation compensation quantity corresponding to each frame of original image in the 19 th, 20 th and 21 st frames of original images with the reduced RS compensation quantity, so as to obtain the stable image transformation matrix corresponding to each frame of original image in the 19 th, 20 th and 21 st frames of original images.
210. And the mobile phone performs image transformation on the original image according to the image stabilizing transformation matrix to obtain a preview image, and displays the preview image on a preview interface.
And the mobile phone performs warp transformation on the corresponding original images according to the image stabilizing transformation matrixes corresponding to each frame of original images in the I-frame original images so as to obtain preview images, and the preview images are displayed on a preview interface.
For example, N is 45, N1 is 15, N2 is 27, i is 3, the mobile phone performs warp transformation on the 16 th, 17 th and 18 th original images according to the image stabilizing transformation matrix corresponding to each of the 16 th, 17 th and 18 th original images, so as to obtain 1 st, 2 nd and 3 rd preview images, and sequentially displays the 1 st, 2 nd and 3 rd preview images on a preview interface. And the mobile phone performs warp transformation on the 19 th, 20 th and 21 st original images according to the image stabilizing transformation matrixes corresponding to each of the 19 th, 20 th and 21 st original images so as to obtain 4 th, 5 th and 6 th preview images, and sequentially displays the 4 th, 5 th and 6 th preview images on a preview interface. For example, when N is 45, N1 is 15, N2 is 27, and i is 3, the time sequence correspondence relationship between the original image frame acquired by the mobile phone in the preview state and the displayed preview image may be referred to fig. 8B.
The difference between the serial number of the preview image frame displayed by the mobile phone and the serial number of the original image frame is N1. That is, the mobile phone does not display the preview image after the warp conversion of the 1 st to N1 st frame of original image on the preview interface, but starts to display the preview image after the warp conversion of the N1 st+1 st frame of original image.
It can be understood that the acquisition frame rate of the original image is larger in the preview state of the mobile phone, and a certain time is required for switching the mobile phone from other modes to the video mode, the mobile phone does not display the preview image after the warp conversion of 1 to N1 frames of the original image after switching to the video mode, so that poor visual experience such as blocking or black screen can not be generated by a user.
And the mobile phone can obtain the target pose of the camera corresponding to the N1+1st frame original image according to the initial pose of the camera corresponding to the N1+I+N2 frame original image, so as to obtain the image stabilizing transformation matrix corresponding to the N1+1st frame original image according to the target pose of the camera corresponding to the N1+1st frame original image, and further obtain the corresponding 1 st frame preview image after the warp transformation of the N1+1st frame original image. That is, the initial frame of the I-frame original image corresponds to the N1+1-th frame original image of the N-frame original images, and the preview image displayed by the mobile phone is delayed by at least N2 frames compared with the original image acquired by the mobile phone.
In other embodiments of the present application, immediately after entering the preview state, in the process of acquiring the first N frames of original images by the mobile phone, each frame of original image in the first N1 frames of original images is displayed as a preview image on the preview interface. Subsequently, after the mobile phone acquires a new original image, a preview image corresponding to the I frame original image is generated according to the N frame original image by adopting the method described in the embodiment, and the generated preview image is displayed on a preview interface.
In other embodiments of the present application, in the preview state, the frame number of the preview image displayed on the preview interface by the mobile phone corresponds to the frame number of the original image, i.e. N2 is 0 and I is 1. That is, after the mobile phone collects a frame of original image, a preview image corresponding to the frame of original image is displayed. Further, N1 may be small, for example, 5, 8, 10, or the like. In the preview state, before the mobile phone collects N1 frames of original images, the mobile phone combines the collected original images in the preview state to generate and display a preview image corresponding to the current original image.
For example, immediately after entering the preview state, the mobile phone acquires the 1 st frame of original image and then displays the 1 st frame of original image as the 1 st frame of preview image on the preview interface. After the mobile phone collects the 2 nd frame original image, generating a 2 nd frame preview image corresponding to the 2 nd frame original image according to the 1 st to 2 nd frame original image, and displaying the 2 nd frame preview image on a preview interface. After the mobile phone collects the 3 rd frame original image, a 3 rd frame preview image corresponding to the 3 rd frame original image is generated according to the 1 st to 3 rd frame original images, and the 3 rd frame preview image is displayed on a preview interface. After the mobile phone collects the N1+1 frame original image, generating an N1+1 frame preview image according to the 1 st to N1+1 frame original image, and displaying the N1+1 frame preview image on a preview interface. Subsequently, after the mobile phone collects a new original image, combining the N1 frame original image before the new original image and the new original image to generate a preview image corresponding to the new original image, and displaying the generated preview image on a preview interface.
In the case where the preview image is obtained by performing image transformation based on the image stabilizing matrix calculated from the target pose on the smoothed target motion curve of the camera, since the smoothed target motion curve of the camera satisfies the sub constraint condition (1), the entire transition between the warp transformed preview images obtained based on the target motion curve of the camera is smoothed. The smoothed target motion curve of the camera satisfies the sub constraint condition (2), so that the warp transformed preview image obtained based on the target motion curve of the camera does not exceed the clipping boundary. The smooth target motion curve of the camera meets the sub constraint condition (3), so that the image stabilizing effect of the main face on the preview image after warp transformation obtained based on the target motion curve of the camera is good.
211. If the fourth condition is not met, the mobile phone calculates an image stabilizing transformation matrix of the original image according to the RS compensation quantity and the motion information of the camera. The handset then performs step 210 described above.
If the fourth condition is not met, the fact that the face and the background of the main body do not have larger relative motion is indicated, and the RS compensation amount does not need to be reduced, so that the mobile phone can calculate an image stabilizing transformation matrix of an original image according to the original RS compensation amount and the motion information of the camera, and warp transformation is carried out according to the image stabilizing transformation matrix, a preview image is obtained, and the preview image is displayed on a preview interface.
That is, in the preview state, the mobile phone can combine the motion information of the main body and the gyroscope data to perform anti-shake processing, so that anti-shake is realized on the motion main body and the background at the same time, the image stabilizing effect of the preview image is improved, and the shooting experience of the user is improved. And the mobile phone can dynamically adjust the strength of face constraint in the preview state according to the motion amplitude of the main face so as to balance the anti-shake effect of the background and the main face on the preview image presented to the user, so that the image stabilizing effect of the background and the main face on the preview image is good. In addition, the mobile phone can also reduce or avoid the distortion of the main face on the preview image as much as possible by adjusting the size of the RS compensation quantity when carrying out RS correction on the background.
For example, in the preview state, when the mobile phone is in the panning mode, the effect of the preview image of the mobile phone not subjected to the anti-shake process may be shown in (a) - (d) of fig. 9A. As shown in fig. 9A, both the background and the subject face are dithered during the preview process. For example, the leaf position in the background is dithered up and down and rotated, and the face position is dithered up and down and rotated. The effect of the preview image of the anti-shake process in the case where the mobile phone does not turn on the face anti-shake mode can be seen from (a) - (d) in fig. 9B. The schematic effect of the preview image that is subjected to the anti-shake processing after the mobile phone starts the face anti-shake mode and is not subjected to the face distortion correction can be seen in (a) - (d) in fig. 9C. The schematic effect of the preview image after the mobile phone starts the face anti-shake mode and performs the anti-shake processing and the face distortion correction can be seen in (a) - (D) of fig. 9D.
Then, the mobile phone enters a shooting process after detecting shooting operation of a user. Referring to fig. 2, the method may further include step 212:
212. after the mobile phone detects shooting operation of a user, an original image is acquired according to a preset frame rate in the shooting process.
For example, after detecting an operation of clicking the photographing control 900 shown in (D) of fig. 9D by the user, the mobile phone determines that the photographing operation of the user is detected, thereby entering the video photographing process. For another example, after detecting the operation of starting shooting by the voice instruction of the user, the mobile phone determines that the shooting operation of the user is detected, so that the video shooting process is entered.
It can be appreciated that there may be a variety of other ways for triggering the mobile phone to enter the video capturing process, and embodiments of the present application are not limited.
It should be noted that, the image data stream in the shooting process includes a preview stream and a video stream in the video. The preview stream in the video recording is used for displaying recorded images on a shooting interface by a user in the video recording process. The preview stream used in the preview state to present the preview image to the user as opposed to the in-recording preview stream may be referred to as a pre-recording preview stream. The video stream is used to generate video images in a video file.
In some embodiments, the in-video preview stream and the pre-video preview stream are the same data stream. After entering the shooting process, the mobile phone continues to process the state in the preview state, the anti-shake mode and the processing process to generate a recorded and shot image. For example, if the mobile phone is in the face anti-shake mode before entering the shooting process, and the mobile phone generates a 60 th frame preview image; after entering the shooting process, the mobile phone is still in the face anti-shake mode, and a 61 th frame preview image is generated by combining the original image before entering the shooting process, wherein the 61 st frame preview image is the 1 st frame recorded image on the shooting interface. If the mobile phone does not enter the face anti-shake mode when entering the shooting process, the mobile phone determines whether to enter the face anti-shake mode after entering the shooting process.
In other embodiments, the in-recording preview stream is not the same data stream as the pre-recording preview stream. After entering the shooting process, the mobile phone stops the preview flow before video recording and starts the preview flow in video recording. Similarly to the preview state, the mobile phone performs anti-shake processing according to the original image in the preview stream before video recording by combining the motion information of the human face and the gyroscope data so as to generate and display a preview image; the mobile phone can perform anti-shake processing according to the original image in the preview stream in the video recording and combining the motion information of the human face and the gyroscope data in the shooting process, so as to generate a video recording image and display the video recording image. And the mobile phone can judge whether to enter the face anti-shake mode again according to the original image acquired in the shooting process, and execute the face anti-shake flow after entering the face anti-shake mode. For example, before entering a shooting process, the mobile phone is in a face anti-shake mode, and the mobile phone generates a 60 th frame preview image; after entering the shooting process, the mobile phone determines whether to enter a face anti-shake mode again, generates a 1 st frame of recorded image according to the original image acquired in the shooting process after entering the face anti-shake mode, and displays the 1 st frame of recorded image on a recorded interface. For example, referring to FIG. 2, after step 212, the method may further include the following steps 213-222 for a preview stream in a video recording:
213. And the mobile phone determines whether the first condition is met according to the S 'frame original image in the shooting process, wherein S' is an integer greater than 1. If the first condition is satisfied, the mobile phone executes steps 214-221; if the first condition is not satisfied, the handset performs steps 222-225.
Wherein S' and S may be the same or different.
214. If the first condition is met, the mobile phone acquires the motion information of the face of the main body according to the M 'frame original image, wherein M' is an integer larger than 1.
Wherein M' and M may be the same or different.
215. And the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N 'frame original image and the motion information of the main body face, wherein N' is an integer larger than 1.
Wherein N 'and N may be the same or different, N1' and N1 may be the same or different, I 'and I may be the same or different, and N2' and N2 may be the same or different.
216. If the first condition is not met, the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N' frame original image.
217. And the mobile phone calculates RS compensation quantity corresponding to the original image according to the data of the gyroscope.
218. The handset determines whether the fourth condition is met. If the fourth condition is satisfied, the handset performs step 219; if the fourth condition is not satisfied, the mobile phone performs step 222.
219. If the fourth condition is met, the mobile phone determines a reduction factor according to the channel amplitude, wherein the reduction factor is used for reducing the RS compensation amount.
220. And the mobile phone calculates an image stabilizing transformation matrix of the original image according to the reduced RS compensation quantity and the motion information of the camera.
221. And the mobile phone performs image transformation on the original image according to the image stabilizing transformation matrix to obtain a recorded image, and displays the recorded image on a preview interface.
222. If the fourth condition is not met, the mobile phone calculates an image stabilizing transformation matrix of the original image according to the RS compensation quantity and the motion information of the camera. Then, the handset performs step 221 described above.
It should be noted that, the descriptions of steps 213-222 may refer to the descriptions of steps 202-211, and are not repeated here.
In the shooting process, the mobile phone can obtain the target pose of the camera corresponding to the N1+1st frame original image according to the initial pose of the camera corresponding to the N1+I+N2 frame original image, so that the image stabilizing transformation matrix corresponding to the N1+1st frame original image can be obtained according to the target pose of the camera corresponding to the N1+1st frame original image, and further the 1 st frame recorded image corresponding to the N1+1st frame original image after warp transformation is performed. That is, the initial frame of the I-frame original image corresponds to the N1+1-th frame original image in the N-frame original images, and the recorded image displayed by the mobile phone is delayed by at least N2 frames compared with the original image acquired by the mobile phone.
In other embodiments, immediately after entering the shooting process, in the process of acquiring the first N frames of original images by the mobile phone, each frame of original image in the first N1 frames of original images is displayed as a recorded image on the shooting interface. Subsequently, after the mobile phone acquires a new original image, generating a recorded image corresponding to the I frame original image according to the N frame original image by adopting the method described in the embodiment, and displaying the generated recorded image on a shooting interface.
In other embodiments, during the shooting process, the frame number of the recorded image displayed on the shooting interface by the mobile phone corresponds to the frame number of the original image, i.e. N2 is 0 and I is 1. That is, after the mobile phone collects a frame of original image, the recorded image corresponding to the frame of original image is displayed. Further, N1 may be small, for example, 5, 8, 10, or the like. Under the shooting process, before the mobile phone acquires the N1 frame of original image, the mobile phone combines the acquired original image under the shooting process to generate a recorded image corresponding to the current original image and display the recorded image.
For example, after the mobile phone acquires the 1 st frame of original image immediately after entering the shooting process, the 1 st frame of original image is displayed on a shooting interface as the 1 st frame of recorded image. After the mobile phone collects the 2 nd frame original image, generating a 2 nd frame recorded image corresponding to the 2 nd frame original image according to the 1 st to 2 nd frame original image, and displaying the 2 nd frame recorded image on a shooting interface. After the mobile phone collects the 3 rd frame original image, generating a 3 rd frame recorded image corresponding to the 3 rd frame original image according to the 1 st to 3 rd frame original images, and displaying the 3 rd frame recorded image on a shooting interface. After the mobile phone collects the N1+1st frame original image, generating an N1+1st frame recorded image according to the 1 st to N1+1st frame original image, and displaying the N1+1st frame recorded image on a shooting interface. Subsequently, after the mobile phone acquires a new original image, combining the N1 frame original image before the new original image and the new original image to generate a recorded image corresponding to the new original image, and displaying the generated recorded image on a shooting interface.
In the case where the recorded image is obtained by performing image transformation based on the image stabilizing matrix calculated from the target pose on the smoothed target motion curve of the camera, since the smoothed target motion curve of the camera satisfies the sub constraint condition (1), the entire transition between the warp-transformed recorded images obtained based on the target motion curve of the camera is smoothed. The smoothed target motion curve of the camera satisfies the sub constraint condition (2), so that the warp-transformed recorded image obtained based on the target motion curve of the camera does not exceed the clipping boundary. The smooth target motion curve of the camera meets the sub constraint condition (3), so that the image stabilizing effect of the main face on the recorded image after warp transformation, which is obtained based on the target motion curve of the camera, is better.
Therefore, in the shooting process, the mobile phone can combine the motion information of the main body and the gyroscope data to perform anti-shake processing, so that the anti-shake processing is realized on the motion main body and the background, the image stabilizing effect of recorded and shot images is improved, and the shooting experience of a user is improved. In addition, the mobile phone can dynamically adjust the strength of face constraint in the shooting process according to the motion amplitude of the main face so as to balance the background on the recorded image presented to the user and the anti-shake effect of the main face, so that the image stabilizing effect of the background on the recorded image and the main face is good. In addition, the mobile phone can also reduce or avoid the distortion of the main face on the recorded image as much as possible by adjusting the size of the RS compensation quantity when carrying out RS correction on the background.
For example, during the shooting process, when the mobile phone is in the panning mode, the effect of the recorded images of the mobile phone that are not subjected to the anti-shake process may be shown in (a) - (d) in fig. 10A. As shown in fig. 10A, both the background and the subject face are dithered during the preview process. For example, the leaf position in the background is dithered up and down, and the face position is dithered up and down. The schematic image recording effect of the anti-shake process when the mobile phone does not turn on the face anti-shake mode can be seen in fig. 10B (a) - (d). The schematic diagrams of the effects of recording and capturing images after the mobile phone starts the face anti-shake mode and performs anti-shake processing without performing face distortion correction can be seen in fig. 10C (a) - (d). The schematic diagrams of the effects of performing anti-shake processing and recording and capturing images with face distortion correction after the mobile phone starts the face anti-shake mode can be seen in fig. 10D (a) - (D).
During shooting, the processing of the video stream and the preview stream in the video recording are parallel and independent. For the video stream, regardless of the state and the processing procedure of the mobile phone before the shooting process, the mobile phone can judge whether to enter a face anti-shake mode according to the original image acquired in the shooting process, and execute the face anti-shake flow after entering the face anti-shake mode. For example, after entering a shooting process, the mobile phone determines whether to enter a face anti-shake mode, and generates and stores a 1 st frame video image according to an original image acquired in the shooting process in the face anti-shake mode, so as to generate a video file according to the stored video image after stopping shooting. For example, referring to fig. 2, after the above step 212, the method may further include the following steps 213'-222':
213', the mobile phone determines whether the first condition is satisfied according to an S ' frame original image in the shooting process, and S ' is an integer greater than 1. If the first condition is satisfied, the mobile phone executes steps 214'-221'; if the first condition is not satisfied, the handset performs steps 222'-225'.
Wherein S 'and S' may be the same or different.
214', if the first condition is satisfied, the mobile phone acquires the motion information of the face of the main body according to M ' frames of original images, wherein M ' is an integer greater than 1.
Wherein M' and M may be the same or different.
215', the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N ' frames of original images and the motion information of the human face of the main body, wherein N ' is an integer larger than 1.
Wherein N 'and N may be the same or different, N1' and N1 may be the same or different, I 'and I may be the same or different, and N2' and N2 may be the same or different.
216', if the first condition is not met, the mobile phone obtains the motion information of the camera according to the gyroscope data corresponding to the N' frame original image.
217', the mobile phone calculates RS compensation quantity corresponding to the original image according to the data of the gyroscope.
218', the handset determines whether the fourth condition is met. If the fourth condition is satisfied, the mobile phone executes step 219'; if the fourth condition is not satisfied, the mobile phone performs step 222'.
219', if the fourth condition is satisfied, the handset determines a reduction factor according to the channel amplitude, the reduction factor being used to reduce the RS compensation amount.
220', the mobile phone calculates the image stabilizing transformation matrix of the original image according to the reduced RS compensation quantity and the motion information of the camera.
221', the mobile phone performs image transformation on the original image according to the image stabilizing transformation matrix to obtain a video image and stores the video image.
222', if the fourth condition is not satisfied, the mobile phone calculates an image stabilizing transformation matrix of the original image according to the RS compensation amount and the motion information of the camera. Then, the handset performs step 221' described above.
It should be noted that, the descriptions of steps 213-222 "may refer to the descriptions related to steps 202-211, and are not repeated herein.
Similar to the preview image and the recorded image, in the shooting process, the mobile phone can obtain the target pose of the camera corresponding to the N1+1st frame original image according to the initial pose of the camera corresponding to the N1+I+N2 frame original image, so that the image stabilizing transformation matrix corresponding to the N1+1st frame original image can be obtained according to the target pose of the camera corresponding to the N1+1st frame original image, and the corresponding 1 st recorded image after the warp transformation of the N1+1st frame original image is obtained. That is, the initial frame of the I-frame original image corresponds to the N1+1-th frame original image in the N-frame original images, and the recorded image displayed by the mobile phone is delayed by at least N2 frames compared with the original image acquired by the mobile phone.
The video image does not need to be presented to the user in real time in the shooting process, so that the anti-shake processing time length and the processing delay can be longer, and the number of frames of the original image can be more. To obtain better quality video images, S "may be greater than S, M" may be greater than M, N "may be greater than N, N1" may be greater than N1, I "may be greater than I, and/or N2" may be greater than N2. For example, for preview streams, N2 or N2' may be an integer of 0 or less (e.g., 2 or 3, etc.) to reduce latency; for video streams, N2 "may be a larger integer (e.g., 27 or 15, etc.) to improve anti-shake processing.
In other embodiments, immediately after entering the shooting process, in the process of acquiring the first N frames of original images by the mobile phone, each frame of original image in the first N1 frames of original images is taken as a video image and stored. Subsequently, after the mobile phone acquires a new original image, the method described in the above embodiment is adopted to generate a video image corresponding to the I-frame original image according to the N-frame original image.
In other embodiments, during the shooting process, the frame number of the video image generated by the mobile phone corresponds to the frame number of the original image, i.e. N2 is 0 and I is 1. That is, after the mobile phone acquires a frame of original image, the video image corresponding to the frame of original image is displayed. Further, N1 may be small, for example, 5, 8, 10, or the like. Under the shooting process, before the mobile phone acquires the N1 frame of original image, the mobile phone combines the acquired original image under the shooting process to generate and store a video image corresponding to the current original image.
For example, immediately after entering a shooting process, the mobile phone acquires the 1 st frame of original image, and then takes the 1 st frame of original image as the 1 st frame of video image. And after the mobile phone acquires the 2 nd frame of original image, generating a 2 nd frame of video image corresponding to the 2 nd frame of original image according to the 1 st to 2 nd frames of original images. And after the mobile phone acquires the 3 rd frame of original image, generating a 3 rd frame of video image corresponding to the 3 rd frame of original image according to the 1 st to 3 rd frame of original image. And after the mobile phone acquires the N1+1 frame of original image, generating the N1+1 frame of video image according to the 1 st to N1+1 frame of original image. And after the mobile phone acquires a new original image, generating a video image corresponding to the new original image according to the N1 frame original image before the new original image and the new original image.
In the case where the video image is obtained by performing image transformation based on the image stabilizing matrix calculated from the target pose on the smoothed target motion curve of the camera, since the smoothed target motion curve of the camera satisfies the sub constraint condition (1), the entire transition between the warp-transformed video images obtained based on the target motion curve of the camera is smoothed. The smoothed target motion curve of the camera satisfies the sub constraint condition (2), so that the warp transformed video image obtained based on the target motion curve of the camera does not exceed the clipping boundary. The smooth target motion curve of the camera meets the sub constraint condition (3), so that the image stabilizing effect of the main face on the video image after warp transformation, which is obtained based on the target motion curve of the camera, is good.
Therefore, in the shooting process, the mobile phone can combine the motion information of the main body and the gyroscope data to perform anti-shake processing, so that the anti-shake processing is realized on the motion main body and the background at the same time, the image stabilizing effect of the video image is improved, and the shooting experience of a user is improved. In addition, the mobile phone can dynamically adjust the strength of face constraint in the shooting process according to the motion amplitude of the main face so as to balance the anti-shake effect of the background and the main face on the video image, and the image stabilizing effect of the background and the main face on the video image is good. In addition, the mobile phone can also reduce or avoid the distortion of the main face on the video image as much as possible by adjusting the RS compensation amount when carrying out RS correction on the background.
After the above steps 222 and 222', referring to fig. 2, the method may further include:
223. and after shooting, the mobile phone generates a video file according to the video image.
Illustratively, after detecting that the user clicks the stop shooting control 1000 as shown in (D) of fig. 10D, the mobile phone determines that shooting is ended. It can be understood that the shooting stopping operation may also be other gesture operations or user voice indication operations, and the operations for triggering the mobile phone to end the shooting process are not limited in the embodiments of the present application.
The video images in the video file generated after the shooting of the mobile phone is finished are obtained by combining the motion information of the main body and the gyroscope data to perform anti-shake processing, so that the image stabilizing effects of the background on the video images and the face of the main body are good, and the shooting experience of the user is good.
For example, the effect of the video image of the mobile phone without the anti-shake process can be shown in (a) - (d) of fig. 11A. As shown in fig. 11A, both the background and the subject face shake during shooting. For example, the leaf position in the background is dithered up and down, and the face position is dithered up and down. The effect of the video image subjected to the anti-shake processing without the mobile phone turning on the face anti-shake mode can be seen from (a) to (d) in fig. 11B. The effect of the video image that is subjected to the anti-shake processing after the mobile phone starts the face anti-shake mode and is not subjected to the face distortion correction can be seen in (a) - (d) of fig. 11C. The schematic effect of the video image after the mobile phone starts the face anti-shake mode and performs the anti-shake processing and the face distortion correction can be seen in (a) - (D) of fig. 11D.
In addition, as described above, after entering/exiting the main anti-shake mode, the mobile phone may prompt the user by displaying information, voice broadcasting or vibration, etc. For example, referring to fig. 12, in the shooting process, after the mobile phone enters the face anti-shake mode, the user may be prompted on the shooting interface through text information: the started face anti-shake-!
In the shooting process described in the above steps 201 to 223, the flow of the mobile phone performing the face anti-shake process may also refer to fig. 13. The process comprises the following steps: the mobile phone detects face feature points according to the multi-frame original images, obtains a smooth target motion curve of the main face according to the face feature points, and generates a first constraint condition according to the target motion curve of the main face. The process also comprises the following steps: the mobile phone calculates a rotation angle according to gyroscope data corresponding to a plurality of frames of original images to obtain an original motion curve of the camera; and the mobile phone solves the target motion curve of the camera according to the original motion curve of the camera and the first constraint condition. The process also comprises the following steps: after entering a channel mode, the mobile phone determines the reduced RS compensation amount to correct the face distortion at the RS timing; the mobile phone calculates a motion compensation homography matrix according to the target motion curve and the reduced RS compensation quantity; and the mobile phone performs motion compensation and RS correction on the original image according to the motion compensation homography matrix, so that a stable video image frame after the anti-shake processing is obtained.
In the scheme described in the above steps 201 to 223, the mobile phone can perform anti-shake processing in combination with the motion information of the main body and the gyroscope data in the preview state and in the shooting process, so as to realize anti-shake on the motion main body and the background at the same time, improve the image stabilizing effect of the preview image, the recorded image and the video image, and improve the shooting experience of the user.
Especially, in the video scene that the motion main body such as leading self-timer is nearer to the camera of cell-phone, and the relative motion of motion main body is also great, and the shake of main body image is also more obvious, adopt this application to implement the video recording method that provides and carry out anti-shake processing, the anti-shake of image and steady image effect are better, and user's shooting experience is also better.
The above description is given by taking the method provided by the embodiment of the application as an example of performing anti-shake processing by combining the motion information of the face of the main body and the gyroscope data in the preview state and the shooting process of the mobile phone in the video mode. In other embodiments, the mobile phone does not adopt the method provided by the embodiment of the present application to perform anti-shake processing in the preview state of the video mode, but starts the method provided by the embodiment of the present application to perform anti-shake processing during the shooting process. Referring to fig. 14, steps 201-211 shown in fig. 2 above may be replaced with step 201A:
201A, after a shooting function is started, the mobile phone enters a video recording mode, an original image is collected according to a preset frame rate in a preview state, a preview image is generated according to the original image, and the preview image is displayed on a preview interface.
In the scheme, the mobile phone can combine the motion information of the main body and the gyroscope data to perform anti-shake processing in the shooting process, so that the anti-shake is realized on the motion main body and the background at the same time, the image stabilizing effect of recorded and shot images and video images is improved, and the shooting experience of a user is improved.
In other embodiments, the mobile phone does not perform anti-shake processing on the pre-video preview stream and the in-video preview stream in the video recording mode by adopting the method provided by the embodiment of the present application, but starts the anti-shake processing on the video stream in the shooting process by adopting the method provided by the embodiment of the present application. Referring to fig. 15, the method may include the above-described step 201A, the following step 212A, and steps 213' -223.
212A, after the mobile phone detects shooting operation of a user, acquiring an original image according to a preset frame rate in the shooting process, generating a recorded image according to the original image, and displaying the recorded image on a shooting interface.
In the scheme, the mobile phone can combine the motion information of the main body and the gyroscope data to perform anti-shake processing in the shooting process, so that the anti-shake is realized on the motion main body and the background at the same time, the image stabilizing effect of the video image is improved, and the shooting experience of a user is improved.
In some embodiments of the present application, the video file subjected to the subject anti-shake processing stored in the mobile phone may be specifically identified differently from other video files, so as to facilitate a user to intuitively learn the video file subjected to the subject anti-shake processing. For example, referring to fig. 16, a text identifier 1600 of "rlfd" is displayed on a video file generated through the face anti-shake processing.
In combination with the foregoing embodiments and the corresponding drawings, another embodiment of the present application provides a photographing method, which may be implemented in an electronic device having a hardware structure shown in fig. 1, where the electronic device includes a camera, and the camera includes a camera. As shown in fig. 17, the method may include:
1701. the electronic device collects the original image after the video recording function is started.
After the video recording function is started, the electronic device can continuously acquire the original image according to a preset acquisition frame rate.
1702. And the electronic equipment acquires the motion information of the main body according to the acquired multi-frame original image.
For example, the subject may be the subject face described above. The motion information of the subject may be used to represent the motion condition of the subject, for example, the motion information of the subject may be the target motion curve of the face of the subject.
1703. And the electronic equipment obtains the motion information of the camera according to the gesture sensor data corresponding to the multi-frame original image and the motion information of the main body.
For example, the gesture sensor may be a gyroscope, the motion information of the camera may be a target motion curve of the camera, and the target motion curve of the camera may be obtained according to the face constraint in the sub constraint condition (3).
1704. The electronic equipment calculates an image stabilizing transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in a plurality of frames of original images.
The image stabilizing transformation matrix is used for performing motion compensation and warp transformation on the first original image.
1705. And the electronic equipment performs image transformation on the first original image according to the image stabilizing transformation matrix to obtain a target image.
For example, the target image may be a preview image, a recorded image, or a video image in a video recording scene.
In the scheme, the electronic equipment can obtain the motion information of the camera according to the motion information of the shot main body and the gesture sensor data under the video recording scene, so that the image stabilizing transformation matrix is obtained according to the motion information of the camera, and further, the motion compensation and the anti-shake are realized on the motion main body and the background through the image stabilizing transformation matrix, the image stabilizing effect of a video image is improved, and the shooting experience of a user is improved.
In the above description, the video recording scene is taken as a front-end self-timer scene as an example, and the video recording scene according to the embodiment of the present application is not limited to this kind of scene.
The above description is given by taking the electronic device as the mobile phone, and the method is not limited to the mobile phone, the smart watch or the tablet personal computer, and other electronic devices can also be used for anti-shake processing, which is not described herein.
It will be appreciated that in order to achieve the above-described functionality, the electronic device comprises corresponding hardware and/or software modules that perform the respective functionality. The steps of an algorithm for each example described in connection with the embodiments disclosed herein may be embodied in hardware or a combination of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may implement the described functionality using different approaches for each particular application in conjunction with the embodiments, but such implementation is not to be considered as outside the scope of this application.
The present embodiment may divide the functional modules of the electronic device according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated modules described above may be implemented in hardware. It should be noted that, in this embodiment, the division of the modules is schematic, only one logic function is divided, and another division manner may be implemented in actual implementation.
Embodiments of the present application also provide an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, the one or more memories being configured to store computer program code comprising computer instructions that, when executed by the one or more processors, cause the electronic device to perform the relevant method steps described above to implement the video recording method of the above embodiments.
Embodiments of the present application also provide a computer-readable storage medium having stored therein computer instructions that, when executed on an electronic device, cause the electronic device to perform the above-described related method steps to implement the video recording method in the above-described embodiments.
Embodiments of the present application also provide a computer program product, which when run on a computer, causes the computer to perform the above-mentioned related steps to implement the video recording method performed by the electronic device in the above-mentioned embodiments.
In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component, or a module, and may include a processor and a memory connected to each other; the memory is used for storing computer-executed instructions, and when the device is operated, the processor can execute the computer-executed instructions stored in the memory, so that the chip executes the video recording method executed by the electronic device in the above method embodiments.
The electronic device, the computer readable storage medium, the computer program product or the chip provided in this embodiment are used to execute the corresponding method provided above, so that the beneficial effects thereof can be referred to the beneficial effects in the corresponding method provided above, and will not be described herein.
It will be appreciated by those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional modules is illustrated, and in practical application, the above-described functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to perform all or part of the functions described above.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another apparatus, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and the parts displayed as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions for causing a device (may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (17)
1. A video recording method applied to an electronic device, the electronic device including a camera, the method comprising:
collecting an original image after a video recording function is started;
acquiring motion information of a main body according to the acquired multi-frame original image;
acquiring motion information of a camera according to the posture sensor data corresponding to the multi-frame original image and the motion information of the main body;
calculating an image stabilizing transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in the multi-frame original image;
performing image transformation on the first original image according to the image stabilizing transformation matrix to obtain a target image;
the motion information of the main body is a target motion curve of the main body, the target motion curve of the main body is obtained according to an original motion curve of the main body, and the original motion curve of the main body is obtained according to the position of the center of the main body on the multi-frame original image;
The motion information of the camera is a target motion curve of the camera, and the obtaining the motion information of the camera according to the gesture sensor data corresponding to the multi-frame original image and the motion information of the main body includes:
acquiring an original motion curve of a camera according to posture sensor data corresponding to the multi-frame original image;
and obtaining a target motion curve of the camera according to the original motion curve of the camera and a subject constraint, wherein the subject constraint is related to the target motion curve of the subject, and the subject constraint is used for enabling the distance between the center of the subject on the target image and the center of the corresponding subject on the target motion curve of the subject to be smaller than or equal to a second preset value.
2. The method of claim 1, wherein the capturing the original image after the video recording function is turned on comprises:
after a video recording function is started and shooting operation of a user is detected, an original image is acquired;
the method further comprises the steps of:
and after the shooting stopping operation of the user is detected, generating a video file according to a video image, wherein the video image is the target image.
3. The method of claim 1, wherein the capturing the original image after the video recording function is turned on comprises:
After a video recording function is started and shooting operation of a user is detected, an original image is acquired;
the method further comprises the steps of:
and displaying a recorded image on a shooting interface, wherein the recorded image is the target image.
4. The method of claim 1, wherein the target image is a preview image, the method further comprising:
and displaying the preview image on a preview interface.
5. The method of claim 1, wherein the subject's target motion profile is continuous; the target motion curve of the main body is first-order and second-order conductive, and the curvature is smaller than or equal to a first preset threshold value; the distance between the center of the subject on the subject's target motion profile and the center of the subject on the subject's original motion profile is less than or equal to a first preset value.
6. The method of claim 5, wherein the obtaining a target motion profile of the camera from the original motion profile of the camera and the subject constraints comprises:
determining a corresponding main body constraint gear according to the motion amplitude of the main body;
gradually increasing the main body constraint intensity from an initial value to an upper limit value corresponding to the main body constraint gear;
And obtaining a target motion curve of the camera according to the original motion curve of the camera and the subject constraint intensity.
7. The method according to any one of claims 1 to 6, wherein the acquiring motion information of the subject from the acquired multi-frame raw image includes:
and after the preset condition is met, acquiring the motion information of the main body according to the acquired multi-frame original image.
8. The method of claim 7, wherein the preset conditions include:
the motion amplitude of the electronic equipment is larger than or equal to a third preset value;
the area occupation ratio of the main body on the original image is larger than or equal to a fourth preset value;
the number of frames of the original image in which the subject continuously appears is greater than or equal to a fifth preset value.
9. The method according to any one of claims 1 to 6 or 8, wherein the obtaining the motion information of the camera from the posture sensor data corresponding to the multi-frame original image and the motion information of the subject includes:
acquiring motion information of the camera according to gesture sensor data corresponding to N frames of original images and the motion information of the main body, wherein N is an integer greater than 1, N=N1+I+N2, N1 and I are positive integers, and N2 is a non-negative integer;
Calculating an image stabilizing transformation matrix of the first original image according to the motion information of the camera, wherein the image stabilizing transformation matrix comprises the following steps:
according to the target pose of the camera corresponding to the N frame original images on the motion information of the camera, calculating an image stabilizing transformation matrix of the I frame original image, wherein the I frame original image is the first original image, the image stabilizing transformation matrix of the I frame original image is used for obtaining the target image of the I frame, and a starting frame of the I frame original image corresponds to an N1+1st frame original image in the N frame original images.
10. The method of claim 9, wherein when the target image is a preview image or a recorded image, the N2 is 0.
11. The method according to claim 8 or 10, characterized in that the method further comprises:
if the preset condition is not met, obtaining the motion information of the camera according to the gesture sensor data corresponding to the multi-frame original image;
calculating an image stabilizing transformation matrix of a first original image according to the motion information of the camera, wherein the first original image is an image in the multi-frame original image;
and carrying out image transformation on the first original image according to the image stabilizing transformation matrix to obtain a target image.
12. The method of claim 11, wherein the method further comprises:
if the preset condition is met, prompting the user to start a target anti-shake mode;
if the preset condition is not met, prompting the user that the target anti-shake mode is not started.
13. The method according to any one of claims 1-6 or 8 or 10 or 12, wherein before said calculating an image stabilization transformation matrix of the first original image from the camera's motion information, the method further comprises:
calculating a rolling shutter RS compensation amount of the first original image according to data of the attitude sensor;
the calculating the image stabilizing transformation matrix of the first original image according to the motion information of the camera comprises the following steps:
calculating a rotation compensation amount of the first original image according to the motion information of the camera;
and calculating an image stabilizing transformation matrix of the first original image according to the rotation compensation quantity and the RS compensation quantity.
14. The method of claim 13, wherein the calculating the image stabilization transformation matrix of the first original image from the rotation compensation amount and the RS compensation amount comprises:
if the panning mode is determined to be entered, reducing the RS compensation amount according to the panning amplitude, wherein the panning amplitude is inversely related to the size of an included angle between a movement main shaft of the camera and a target coordinate axis;
And calculating an image stabilizing transformation matrix of the first original image according to the rotation compensation quantity and the reduced RS compensation quantity.
15. The method of any one of claims 1-6 or 8 or 10 or 12 or 14, wherein the subject is a human face.
16. An electronic device, comprising:
a camera for capturing images;
the screen is used for displaying an interface;
one or more processors;
a memory;
and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions that, when executed by the electronic device, cause the electronic device to perform the video recording method of any of claims 1-15.
17. A computer readable storage medium comprising computer instructions which, when run on a computer, cause the computer to perform the video recording method as claimed in any one of claims 1 to 15.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011057023.8A CN114339101B (en) | 2020-09-29 | 2020-09-29 | Video recording method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011057023.8A CN114339101B (en) | 2020-09-29 | 2020-09-29 | Video recording method and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114339101A CN114339101A (en) | 2022-04-12 |
| CN114339101B true CN114339101B (en) | 2023-06-20 |
Family
ID=81010807
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011057023.8A Active CN114339101B (en) | 2020-09-29 | 2020-09-29 | Video recording method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114339101B (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106331480B (en) * | 2016-08-22 | 2020-01-10 | 北京交通大学 | Video image stabilization method based on image splicing |
| CN110198412B (en) * | 2019-05-31 | 2020-09-18 | 维沃移动通信有限公司 | Video recording method and electronic equipment |
| CN111314604B (en) * | 2020-02-19 | 2021-08-31 | Oppo广东移动通信有限公司 | Video anti-shake method and apparatus, electronic device, computer-readable storage medium |
| CN111371987B (en) * | 2020-02-25 | 2021-06-25 | Oppo广东移动通信有限公司 | Image processing method and device, electronic equipment and computer readable storage medium |
-
2020
- 2020-09-29 CN CN202011057023.8A patent/CN114339101B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114339101A (en) | 2022-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022062318A1 (en) | Photographing method and device | |
| US10609273B2 (en) | Image pickup device and method of tracking subject thereof | |
| CN114339102B (en) | Video recording method and equipment | |
| KR20210130773A (en) | Image processing method and head mounted display device | |
| CN113630545B (en) | Shooting method and equipment | |
| US8400532B2 (en) | Digital image capturing device providing photographing composition and method thereof | |
| CN108093158B (en) | Image blurring processing method and device, mobile device and computer readable medium | |
| CN114390213B (en) | Shooting method and equipment | |
| CN115209057B (en) | Shooting focusing method and related electronic equipment | |
| CN110581957B (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| WO2022089341A1 (en) | Image processing method and related apparatus | |
| CN114390186A (en) | Video shooting method and electronic equipment | |
| CN114339101B (en) | Video recording method and equipment | |
| WO2023011302A1 (en) | Photographing method and related apparatus | |
| CN115546043B (en) | Video processing method and related equipment thereof | |
| CN113747044B (en) | Panoramic shooting method and equipment | |
| CN114390191B (en) | Video recording method, electronic device and storage medium | |
| CN117278839A (en) | Shooting method, electronic equipment and storage medium | |
| CN114125298A (en) | Video generation method and device, electronic equipment and computer readable storage medium | |
| CN114757866A (en) | Definition detection method, device and computer storage medium | |
| EP4280154A1 (en) | Image blurriness determination method and device related thereto | |
| CN114979458B (en) | Image shooting method and electronic equipment | |
| CN115150542B (en) | Video anti-shake method and related equipment | |
| CN116055895B (en) | Image processing method and device, chip system and storage medium | |
| CN117714840A (en) | Image processing method, device, chip, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |