WO2023284591A1

WO2023284591A1 - Video capture method and apparatus, electronic device, and storage medium

Info

Publication number: WO2023284591A1
Application number: PCT/CN2022/103943
Authority: WO
Inventors: 张羽翕
Original assignee: 华为技术有限公司
Priority date: 2021-07-12
Filing date: 2022-07-05
Publication date: 2023-01-19
Also published as: CN115623145A

Abstract

The present application relates to the technical field of smart terminals, and specifically relates to a video capture method and apparatus, an electronic device, and a storage medium. The method comprises: an electronic device obtains first data, the first data comprising first audio data and/or first image data; the electronic device determines jitter parameters on the basis of the first data; the electronic device generates second data on the basis of the jitter parameters, wherein the second data is video data, and the second data generated by the electronic device on the basis of the jitter parameters is specifically: a second video captured on the basis of the jitter parameters, or a third video obtained by the electronic device capturing the second video and processing the second video according to the jitter parameters. The present application collects video data during a video capture process, and determines corresponding jitter parameters, which are used to capture or process a video which has jitter effects. The jitter effects presented by the video can closely match video content and the feelings of a photographer, and at the same time, saves a user time for post-processing the video, which helps to improve the user experience.

Description

Video shooting method and device, electronic device and storage medium

This application claims the priority of the Chinese patent application with the application number 202110784717.X and the application name "video shooting method and device, electronic equipment and storage medium" submitted to the China Patent Office on July 12, 2021. The entire content of the application is passed References are incorporated in this application.

technical field

The present invention relates to the technical field of intelligent terminals, in particular to a video shooting method and device, electronic equipment and a storage medium.

Background technique

With the vigorous development of the short video multimedia industry, the various difficult video shooting technologies and processing technologies presented by the short video are amazing, and at the same time, people's pursuit of these technologies is constantly improving. For example, ordinary users are publishing When making short videos, dynamic special effects such as shaking may also be added to the short videos to express the resonant content in the videos.

1A to 1D show the current user's operation process of making a short video with shaking effects. As shown in Figure 1A, usually the user needs to use the camera function of the mobile phone to shoot a video first; after the video shooting is completed, the user opens the application program for making a short video installed on the mobile phone, opens the interface shown in Figure 1B, and clicks the Add Video button 101 'add the recorded video; afterward, the user clicks the special effect button 102' in the opened video processing interface shown in Figure 1C to enter the interface shown in Figure 1D, and the user presses the shake special effect button 103 on the interface shown in Figure 1D 'A jitter effect can be added to the video, and the visual effect of the jitter effect is shown in Figure 1D.

In the prior art, although the above-mentioned process can produce a short video with shaking effects, it is obvious that the above-mentioned process is the post-production of the video, that is, the process of editing the video after shooting the video. This post-production method To a large extent, it needs to rely on the personal ability of the creator (that is, the user who made the short video mentioned above). First, it is difficult to achieve a high degree of real-time matching between the shaking effect and the picture and sound in the video; secondly, post-production can add The jitter effect is single, only zooming in and out, or left and right, up and down, and it is impossible to obtain a realistic picture jitter effect in the three-dimensional (3 Dimensions, 3D) direction; in addition, post-production generally requires separate time for production, and it is time-consuming. The long-term efficiency is low, and it takes a lot of time and energy for the creator.

Contents of the invention

Embodiments of the present application provide a video shooting method and device, electronic equipment, and a storage medium, by determining the shaking parameters used to control the shaking effect of the video based on the collected video data during the shooting of the video, and then based on the determined shaking Parameter shooting or processing while shooting to obtain a video with shaking effects, wherein the video data collected in the process of shooting video includes audio data and/or image data, and the present application can extract the audio feature value and/or Extract the change feature value of the focus object corresponding to the image data to determine the matching shake parameters, so as to obtain a video with shake special effects. The shaking special effects presented in the video obtained by the video shooting method provided in the present application can highly match the video content and the emotion of the shooter, and the user does not need to spend a lot of time on the post-processing of the video, which is conducive to improving user experience.

In a first aspect, an embodiment of the present application provides a video shooting method, which is applied to an electronic device, and the method includes: the electronic device acquires first data, and the first data includes first audio data and/or first image data; the electronic device The shake parameter is determined based on the first data; the electronic device generates second data based on the shake parameter, and the second data is video data; the second data generated by the electronic device based on the shake parameter is specifically: a second video shot based on the shake parameter, or, electronic The device shoots the second video, and processes the second video to obtain a third video according to the dithering parameter.

That is, the electronic device can collect the first audio data and/or the first image data as the first data when shooting a video, and then determine the shaking parameters used to control the degree or effect of the shaking special effect according to the collected first data, and the electronic device can then Shooting according to the determined shake parameters to obtain a second video with a shake special effect, or performing shake processing on the captured second video according to the shake parameters to obtain a third video with a shake special effect. For example, when the electronic device is a mobile phone, when the mobile phone runs the camera application to shoot video, it can collect audio data through a microphone, that is, the above-mentioned first audio data that constitutes the first data, or collect image data through a camera, that is, the above-mentioned first data that constitutes When the mobile phone responds to the user's operation or detects that the collected audio data and/or image data meet the conditions for shooting a shaking video, the mobile phone can determine the shaking parameters according to the collected audio data and/or image data, and A shaking video is shot based on the shaking parameter, or a video with a shaking special effect is obtained by performing shaking processing on the shot video.

In a possible implementation of the above first aspect, the above method further includes: when the first audio data satisfies the first preset condition; and/or the first image data satisfies the second preset condition, the electronic device based on The first data determine dither parameters.

That is, the electronic device presets the first preset condition for judging whether the first audio data satisfies the conditions for shooting a jittery video, such as the jitter trigger condition that the audio characteristic value described in the embodiment below needs to satisfy, wherein, The audio feature value can be obtained based on the first audio data extraction; the electronic device can also preset a second preset condition for judging whether the first image data meets the conditions for shooting a shaking video, the second preset condition is, for example, the following implementation The jitter trigger conditions that need to be satisfied by the feature value of the focus object change described in the example, wherein the feature value of the focus object change can be obtained based on the first image data extraction, and the feature value of the focus object change can be the focus data described in the following embodiments . That is to say, when the audio characteristic value corresponding to the first audio data collected by the electronic device meets the corresponding shaking trigger condition, and/or the focus object change characteristic value corresponding to the first image data collected by the electronic device meets the corresponding shaking trigger condition, The electronics can determine corresponding dither parameters.

In a possible implementation of the above first aspect, the first audio data satisfies the first preset condition, including at least one of the following: the sampling rate of the first audio data is greater than the preset sampling rate threshold; the first audio data The frequency of the first audio data is greater than or equal to the preset frequency threshold; the loudness of the first audio data is greater than or equal to the loudness threshold.

That is, the audio feature value corresponding to the first audio data may be feature values such as audio sampling rate, frequency, loudness, etc., when the sampling rate is greater than the preset sampling rate threshold, and/or the frequency is greater than or equal to the preset frequency threshold, and/or When the loudness is greater than or equal to the loudness threshold, it can be determined that the first audio data collected by the electronic device satisfies the first preset condition, for example, in the following embodiments, the audio sampling rate corresponding to the audio data collected when the mobile phone shoots a video When it is greater than the preset sampling rate threshold in the mobile phone, and/or the frequency is greater than or equal to the preset frequency threshold in the mobile phone, and/or the loudness is greater than or equal to the loudness threshold in the mobile phone, it can be determined that the audio data collected by the mobile phone meets the corresponding requirements. The jitter trigger condition.

In a possible implementation of the first aspect above, the image data satisfies the second preset condition, including at least one of the following: the displacement of the focus object in the first image data is greater than or equal to the preset displacement threshold; the first image The movement frequency of the focus object in the data is greater than or equal to a preset frequency threshold; the change value of the contour size of the focus object in the first image data is greater than or equal to a preset change value threshold.

That is, the characteristic value of the focus object change corresponding to the first image data may be the characteristic values such as the displacement size, movement frequency, and contour size change value of the focus object in the first image data. When the displacement of the focus object in the first image data is greater than or equal to When the preset displacement threshold, and/or the movement frequency is greater than or equal to the preset frequency threshold, and/or the contour size change value is greater than or equal to the preset change value threshold, it can be determined that the first image data collected by the electronic device satisfies the first preset conditions. For example, in the following embodiments, the displacement in the focus object change characteristic value (focus data) corresponding to the image data collected when the mobile phone shoots a video is greater than or equal to the preset displacement threshold in the mobile phone, and/or the movement frequency is greater than or equal to When the preset frequency threshold and/or the change value of the outline size in the mobile phone is greater than or equal to the preset change value threshold in the mobile phone, it can be determined that the image data collected by the mobile phone satisfies the corresponding shaking trigger condition.

In a possible implementation of the first aspect above, the electronic device includes a preset correspondence between audio data and/or image data and shaking parameters, and the electronic device determines the shaking parameter based on the first data, including: A dithering parameter matching the first audio data and/or the first image data is selected from the preset corresponding relationship.

That is, the correspondence between various audio data and/or image data and shaking parameters is preset in the electronic device, and the electronic device can first find the matching preset Assuming corresponding audio data and/or image data in the corresponding relationship, and then determining the shaking parameter corresponding to the collected first audio data and/or first image data according to the above corresponding relationship. For example, the correspondence between the preset audio data and/or image data and the shaking parameters in the electronic device may be the correspondence between the audio feature value corresponding to the audio data and the shaking parameters, and/or the corresponding relationship between the image data It can be understood that the correspondence between the audio characteristic value and the shaking parameter can be the corresponding relationship between a certain audio characteristic value and the shaking parameter, or it can be multiple The corresponding relationship between the combination parameter of the audio feature value and the shaking parameter, the corresponding relationship between the changing characteristic value of the focus object and the shaking parameter can be the corresponding relationship between the changing characteristic value of a certain focus object and the shaking parameter, or can be It is the corresponding relationship between the combination parameter of the variation characteristic value of multiple focus objects and the dithering parameter, which is not limited here.

In a possible implementation of the first aspect above, the electronic device includes multiple motors, and the multiple motors include motors in the camera module of the electronic device; and the shaking parameter includes a vibration parameter of at least one of the multiple motors . The vibration parameters of the motor include at least one of vibration direction, vibration amplitude and vibration frequency. The second video shot based on the shaking parameters includes: the electronic device controls the motor to vibrate based on the vibration parameters of the motor and shoots to obtain the second video.

That is, multiple motors can be installed in the electronic device. For example, a motor for generating vibration prompt information can be installed in the mobile phone, and a motor installed in the camera module for controlling lens displacement or shaking can also be installed. There is no limitation here. The vibration parameters determined by the above-mentioned electronic device based on the first data may include vibration parameters of any one or more motors in the electronic device, including vibration direction, vibration amplitude, and vibration frequency used to control the vibration speed of the motor, etc., and the electronic device may be based on The vibration parameter controls the vibration of the corresponding motor, so that the electronic device shakes or the lens of the electronic device shakes to shoot a second video with a shaking special effect.

In a possible implementation of the foregoing first aspect, the electronic device includes an optical anti-shake module, and the shake parameter includes a lens shake parameter in the optical anti-shake module. The lens shake parameter includes at least one item of shake displacement, shake direction, and shake frequency during the lens shake process. The second video shot based on the shaking parameters includes: the electronic device turns off the anti-shaking function of the optical anti-shake module, and based on the lens shaking parameters, the optical anti-shake module controls the lens shaking to obtain the second video.

That is, the shake parameter determined by the electronic device based on the first data may be related parameters corresponding to the optical anti-shake module of the electronic device, for example, the lens shake parameter in the optical anti-shake module, including the shake displacement used to control the degree of lens shake The size, the shaking frequency used to control the shaking speed of the lens, and the shaking direction, etc., so that the electronic device can control the lens through the optical anti-shake module to generate corresponding shaking according to the determined lens shaking parameters, and obtain a second image with shaking special effects. video.

In a possible implementation of the above-mentioned first aspect, the shake parameter further includes: a change parameter of the second image data collected when the electronic device shoots the second video; or a change parameter in the second image data collected when the electronic device shoots the second video The change parameter of the focus object. The change parameter of the second image data or the change parameter of the focus object in the second image data includes at least one item of scaling, displacement, moving direction, moving speed, and moving frequency. The electronic device shoots the second video, and processes the second video according to the shaking parameters to obtain a third video, including any of the following: the electronic device shoots the second video, and processes the second video according to the change parameters of the second image data to obtain The third video: the electronic device shoots the second video, and processes the second video according to the change parameters of the focus object in the second image data to obtain the third video.

That is, the shake parameter determined by the electronic device based on the first data may be a change parameter of the second image data in the second video to be shot, or a change parameter of the focus object extracted based on the second image data, for example, including The zoom ratio, displacement, moving direction, moving speed, moving frequency, etc. of the second image or the focus object in the second image, so that the electronic device can adjust the second image in the captured second video according to the determined change parameters. The second image data is dithered to obtain a second video with a dithering effect. For example, in the following embodiments, the mobile phone can use the image sensor to perform processing such as zooming and moving on the image collected during the shooting process, or perform processing such as zooming and moving on the focus object in the image collected during the shooting process, so as to obtain a shaking effect video.

In a possible implementation of the above first aspect, the video shooting interface of the electronic device includes a shaking mode control, and the method further includes: in response to a user's operation on the shaking mode control, the electronic device generates second data based on shaking parameters.

That is, the video shooting interface of the electronic device can be provided with a shaking mode control to control whether to shoot a video with shaking effects, and the user can click the control to turn on or off the shaking mode, and the control can be set at any position on the video shooting interface of the electronic device , without limitation here. It can be understood that when the shaking mode is turned on, the electronic device can determine the shaking parameters according to the audio data and/or image data collected when shooting a video and shoot a video with shaking effects; when the shaking mode is turned off, even if the electronic device The collected audio data and/or image data can be matched to the corresponding shaking parameters, or meet the corresponding preset conditions. At this time, the electronic device will not shoot or generate the second video with shaking effects. For example, in the following embodiments, on the video shooting interface of the camera application running on the mobile phone (as shown in FIG. 6A ), a shake mode check button is set, and the user clicks the button to complete the check, and the mobile phone can enter the shake shooting mode , in this mode, when the audio characteristic value corresponding to the audio data collected by the mobile phone and/or the focus object change characteristic value corresponding to the collected image data meet the shaking trigger condition, the mobile phone can control the motor according to the determined shaking parameters Vibration, or control camera shaking etc. to shoot to obtain a video with shaking special effects; on another video shooting interface described in the following embodiments (shown in reference to Figure 6B), the above-mentioned shaking mode control can also be in the mode menu bar of the video shooting interface The shaking shooting option, the user slides left and right in the mode menu bar displayed on the mobile phone to select the shaking shooting option, and then enters the shaking shooting mode, which will not be repeated here.

In a possible implementation of the foregoing first aspect, the method further includes: when detecting that the current shooting scene is a preset scene, the electronic device generates the second data based on the shake parameter.

That is, the electronic device can preset some scenes suitable for collecting jitter effects to express the video content or the emotion of the photographer, such as the more enthusiastic concert scene, sports competition scene, hip-hop competition scene, etc. When the electronic device detects that the current location is When the scene is a preset scene, it can automatically enter the shaking shooting mode. For example, the mobile phone can judge according to whether the audio data collected by the microphone and/or the image data collected by the camera match the audio data characteristics and/or image data characteristics corresponding to the preset scenes such as concerts, sports competitions, hip-hop competitions, etc. Whether the current scene is a preset scene, if a preset scene can be matched, the camera application running on the mobile phone will automatically enter the shaking shooting mode.

In a possible implementation of the foregoing first aspect, the electronic device generating the second data based on the shaking parameter further includes: the electronic device adding a sound effect beat or a music segment that matches the content of the second data.

That is, in the process of generating a video with a shaking effect based on shaking parameters, the electronic device can simultaneously add a sound effect beat or a music clip that matches the shaking effect, for example, the audio beat is "咚~咚~咚" or "Zi~ Nourish ~ Nourish" and other special effects.

In a second aspect, an embodiment of the present application provides a video shooting method, the method comprising: the second electronic device acquires first data, the first data being first audio data and/or first image data; the second electronic device based on The first data determines the shake parameter; the second electronic device sends the shake parameter to the first electronic device; the first electronic device shoots the second video based on the shake parameter, or the first electronic device shoots the second video and processes the second video according to the shake parameter Get the third video.

That is, the video shooting method of the present application can be jointly executed by distributed multiple devices, wherein one electronic device (the second electronic device) collects the first data including the first audio data and/or the first image data, and based on the first The data determines the shaking parameters, and the electronic device sends the determined shaking parameters to another electronic device (the first electronic device) to shoot a second video with shaking effects, or process the captured second video to obtain a first shaking effect. Three videos. For example, the user's mobile phone is interconnected with the bracelet, and the bracelet is used to collect first data including the first audio data and/or first image data, and send the shaking parameters determined based on the first data to the mobile phone, and the mobile phone then receives Shoot a second video with a shaking effect based on the obtained shaking parameters, or obtain a third video with a shaking effect from the captured second video.

In a possible implementation of the second aspect above, the second electronic device includes a preset correspondence between audio data and/or image data and shaking parameters, and the second electronic device determines the shaking parameters based on the first data, including : The first electronic device selects a shaking parameter matching the first audio data and/or the first image data from preset correspondences.

That is, the second electronic device used to determine the shaking parameters is preset with a preset correspondence between the audio data and/or image data and the shaking parameters, and the second electronic device can find a corresponding relationship with the collected data from the preset correspondence. The jitter parameters corresponding to the first data are sent to the first electronic device. For example, in the aforementioned example, the wristband is preset with a preset correspondence between audio data and/or image data and shaking parameters. When the user uses a mobile phone to shoot a video, the wristband can collect the first data and determine the shaking based on the first data. Parameters are sent to the phone.

It can be understood that in some other embodiments, the wristband can also send the collected first data to the mobile phone for processing, and the mobile phone can preset audio data and/or image data and shaking parameters from the mobile phone based on the received first data. The jitter parameters that match the first data are determined in the corresponding relationship between them, which is not limited here.

In a possible implementation of the second aspect above, the shaking parameters include at least one of the following: vibration parameters of at least one of the multiple motors in the first electronic device, and the multiple motors include the vibration parameters of the first electronic device. A motor in the camera module; a lens shake parameter in the optical anti-shake module of the first electronic device. The vibration parameter of the motor includes at least one item of vibration direction, vibration amplitude, and vibration frequency; the lens shake parameter includes at least one item of shake displacement, shake direction, and shake speed during the lens shake process.

In a possible implementation of the second aspect above, the first electronic device shoots the second video based on the shaking parameters, including at least one of the following: the first electronic device controls the motor to vibrate and shoots the second video based on the vibration parameters of the motor. Two videos: the first electronic device controls the lens shake through the optical image stabilization module based on disabling the anti-shake function of the optical anti-shake module, and based on the lens shake parameters, and shoots the second video.

In a possible implementation of the above-mentioned second aspect, the shaking parameters further include: a change parameter of the second image data collected when the first electronic device shoots the second video; or a change parameter of the second image data collected when the first electronic device shoots the second video. 2. Variation parameters of the focus object in the image data. The change parameter of the second image data or the change parameter of the focus object in the second image data includes at least one item of scaling, displacement, moving direction, moving speed, and moving frequency.

In a possible implementation of the second aspect above, the first electronic device shoots the second video, and processes the second video according to the shaking parameters to obtain the third video, including any of the following: the first electronic device shoots the second video video, and process the second video according to the change parameters of the second image data to obtain the third video; the first electronic device shoots the second video, and process the second video according to the change parameters of the focus object in the second image data to obtain the third video .

In a third aspect, an embodiment of the present application provides a video shooting method, the method includes: the second electronic device acquires first data, the first data includes audio data and/or image data focus object change feature values; the second electronic device The shake parameter is determined based on the first data; the second electronic device obtains the second data shot by the first electronic device, and processes the second data according to the shake parameter to obtain a third video.

That is, when the video shooting method of the present application is jointly executed by multiple distributed devices, the first electronic device can shoot the video, but the second electronic device collects the first data, determines the shaking parameters, and controls the first electronic device according to the shaking parameters. The video captured by the device is subjected to shaking processing to obtain a third video with a shaking special effect. For example, when the first electronic device is a mobile phone and the second electronic device is a wristband, the wristband determines the shaking parameters according to the collected first data, and the second video data captured by the mobile phone is also sent to the wristband, and the mobile phone is based on the determined shaking parameters. The second video is processed with parameters to obtain a third video with a shaking effect. In this case, the wristband can be configured with chips or processors with higher computing performance and audio and image processing capabilities, without limitation here.

In a fourth aspect, the embodiment of the present application provides an electronic device, including: one or more processors; one or more memories; one or more memories storing one or more programs, when the one or more programs are When executed by one or more processors, the electronic device is made to execute the above video shooting method.

In a fifth aspect, the embodiment of the present application provides a computer storage medium, where instructions are stored on the storage medium, and when the instructions are executed on the computer, the computer executes the above video shooting method.

In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program/instruction, and when the computer program/instruction is executed by a processor, the above video shooting method is realized.

In the seventh aspect, the embodiment of the present application provides a video shooting device, the device includes: a data acquisition module, configured to acquire first data, the first data includes first audio data and/or first image data; shake parameter generation A module for obtaining the first data collected by the data acquisition module, and determining a shaking parameter based on the first data; a shaking video generation module for obtaining the shaking parameters determined by the shaking parameter generation module, and shooting a second video based on the shaking parameters, Or, after shooting the second video, process the second video according to the shaking parameters to obtain the third video.

Description of drawings

1A to 1D are schematic diagrams of UI interfaces in the operation process of adding shaking effects to videos in the prior art.

FIG. 2 is a schematic diagram of various application scenarios of the video shooting method provided by the embodiment of the present application.

FIG. 3 is a schematic structural diagram of an electronic device 100 provided by an embodiment of the present application.

FIG. 4 shows a schematic process of converting collected audio data into corresponding shaking parameters to implement adding shaking special effects provided by the embodiment of the present application.

FIG. 5 is a schematic diagram of the implementation flow of the video shooting method provided by the embodiment of the present application.

6A to 6C are schematic diagrams of UI interfaces for entering the shake shooting mode provided by the embodiment of the present application.

FIG. 7 is a schematic diagram showing the composition principle of an anti-shake system based on OIS technology.

FIG. 8 is a schematic diagram of the UI interface of the mobile phone 100 provided by the embodiment of the present application for displaying the completed shake special effect video and performing subsequent processing.

9A to 9C are schematic diagrams of a conversion process of converting real-time audio data and/or real-time image data into corresponding dithering parameters provided by the embodiment of the present application.

10A to 10D are schematic diagrams of UI interfaces corresponding to setting shaking parameters on the mobile phone 100 provided by the embodiment of the present application.

FIG. 11 is a schematic diagram of a display interface of a shake effect preview window provided by the embodiment of the present application.

12A to 12C are schematic diagrams of shaking special effects corresponding to shooting modes with different degrees of shaking provided by the embodiment of the present application.

FIG. 13 is a schematic diagram of a system architecture of a mobile phone 100 involved in a video shooting process through motor vibration provided by an embodiment of the present application.

FIG. 14 is a schematic structural diagram of a video capture device provided by an embodiment of the present application.

detailed description

Various aspects of the illustrative embodiments are described below using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art. However, it will be apparent to those skilled in the art that some alternative embodiments may be practiced using some of the described features. For purposes of explanation, specific numbers and configurations are set forth to provide a more thorough understanding of the illustrative embodiments. It will be apparent, however, to one skilled in the art that alternative embodiments may be practiced without the specific details. In other instances, well-known features have been omitted or simplified herein in order to avoid obscuring the illustrative embodiments of the application.

Illustrative embodiments of the present application include, but are not limited to, video shooting methods and devices, electronic equipment, and storage media.

FIG. 2 shows a schematic diagram of various application scenarios of the video shooting method of the present application.

As shown in Figure 2, the scene includes an electronic device 100 for shooting video, and a variety of shooting scenes suitable for shooting special effects videos using the video shooting method of this application, including but not limited to scenes that need to capture character movements/expressions, etc. , such as sports arenas, hip-hop competitions, children's performances, etc., and scenes that need to capture sound/light effects, such as concerts or singing and dancing performances, rap performances, etc. In these scenes of the above example, the audience records the actions, expressions, voices, etc. of the competitors or performers by shooting videos. For example, in hip-hop competitions, most of the competitors will make some difficult movements after certain preparatory movements. The audience is usually excited when watching these difficult movements, and at the same time, they would like to add special effects such as shaking to this moment in the shot video to express their excitement.

As mentioned above, when users publish short videos, they sometimes need to add dynamic special effects such as shaking to express the resonant content in the video. In fact, in the process of recording many exciting moments with video, for example, in the various scenes shown in Figure 2, the user can highlight the theme by adding shaking effects that match the captured pictures and the sound in the scene , Express emotion. If the video processing process described in the above-mentioned background technology is adopted, it is difficult to highly match the jitter special effects that may exist in the post-production process mentioned in the above-mentioned background technology with the picture in the video and the real-time sound, and the jitter effect that can be added is single , Unable to obtain realistic picture shaking effect in 3D direction, and the post-production takes a long time, is inefficient, and requires a lot of time and effort of creators, etc., which will bring users a very bad special effects video production experience. It is understandable that if the video production process takes too long, the user's enthusiasm will fade, and the produced special effects video will be more difficult to express the emotions at that time.

Aiming at the above-mentioned problems currently existing in the post-production special effects video, the present application provides a video shooting method, which is based on the video data collected during the video shooting process, including but not limited to audio data and/or focus data, and extracting The audio feature value and/or the change feature value of the focus object, when the audio feature value and/or the change feature value of the focus object meet the shaking trigger condition, trigger the addition of a shaking effect at a time node with a high degree of fit, and the video provided by this application The shooting method can also convert the corresponding shaking parameters based on the audio feature value and/or the change feature value of the focus object, so as to control parameters such as shaking amplitude and shaking frequency, so that the shaking special effect presented in the captured video highly matches the emotion of the video content . Wherein, the focus data may be image data about the target focus in any frame of image collected by the camera module of the electronic device, and data such as the relative position of the target focus in the frame of image.

It can be understood that the shaking effect presented in the video shot when the preset shaking trigger conditions are met can be realized through hardware shaking in the electronic device, such as through the vibration of the motor, or through the optical anti-shake module to control the lens Vibration and other methods can be used to present jitter special effects in the captured video, wherein the above-mentioned motor can be a motor in the electronic device for providing vibration prompt information, etc., or it can be a motor in the camera module of the electronic device for controlling lens displacement or shaking. The motor is not limited here. The detailed process of presenting the shaking special effect at the corresponding time node in the captured video through the vibration of the motor in the electronic device or the movement of the lens through the optical anti-shake module will be described in detail below. This will not be repeated here. In addition, the above-mentioned jitter special effect presented in the video shot under the condition of meeting the preset jitter trigger conditions can also be realized by the processor in the electronic device controlling the operation of related algorithms, for example, the image data collected by the camera module is processed After performing image processing such as noise reduction and de-distortion, the focus data is obtained based on the processed image data. It can be understood that the image processor can run a shaking special effect processing algorithm to perform operations such as displacement and/or scaling in the 3D direction on the relevant focus data of the extracted focus object, so that the corresponding video content is presented in the video obtained after shooting , Emotionally matched jitter effects.

It can be understood that the electronic device 100 applicable to the video shooting method of the present application includes, but is not limited to, mobile phones, cameras, tablet computers, desktop computers, notebook computers, drones and other electronic devices with shooting functions, and the electronic device 100 can also be It is a combined device such as a gimbal and a mobile phone, or a gimbal and a camera, and there is no limitation here.

FIG. 3 shows a schematic diagram of a hardware structure of an electronic device 100 .

As shown in FIG. 3 , the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, and a battery 142 , antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, optical image stabilization module 192 , a camera 193, a display screen 194, and a subscriber identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an acceleration sensor 180C, a distance sensor 180D, a touch sensor 180E, an ambient light sensor 180F and the like.

It can be understood that, the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction. In some embodiments of the present application, the processor 110 may perform feature analysis on the collected audio data and/or image data through the control of the controller, and determine whether the vibration trigger condition is met, and convert the Get the shaking parameters to control the presentation of shaking effects that highly match the video content and emotion at certain time nodes in the captured video. The shaking parameters are the shaking direction, shaking amplitude, and shaking frequency corresponding to the shaking effects. parameter.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system. In some embodiments of the present application, relevant preset conditions, preset parameters, and various instruction data for implementing the video shooting method provided in the present application may be stored in the memory, which is not limited here.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 may be respectively coupled to the touch sensor 180E, the charger, the flashlight, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180E through the I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the electronic device 100 .

The I2S interface can be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.

The PCM interface can also be used for audio communication, sampling, quantizing and encoding the analog signal. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the wireless communication module 160 . For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to realize the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193 , the display screen 194 , the wireless communication module 160 , the audio module 170 , the sensor module 180 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface conforming to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100 , and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 is charging the battery 142 , it can also provide power for electronic devices through the power management module 141 . The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 .

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (blue tooth, BT), global navigation Satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.

The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 193 .

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1. In the embodiment of the present application, the camera 193 may collect image data during the process of shooting video, including collecting continuous image data including shooting focus and the like. In other embodiments, the image data collected by the camera 193 can also be used to determine the scene where the electronic device 100 is located at this time. It can be understood that several scenes can be preset on the electronic device 100, for example, the scene shown in FIG. 2 above can be preset. In the various scenes shown, the feature values corresponding to the image data in different scenes are different.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or shoot videos in various encoding formats, such as moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, voice recognition, text understanding, and the like. In this embodiment of the application, the NPU can be used to identify the focus of video shooting, such as the focus person, and then obtain the image change data related to the focus person in real time, and judge whether the shaking trigger condition is satisfied based on the above real-time image change data.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card. In the embodiment of the present application, the captured video with shaking effect can be saved in the external memory card connected to the external memory interface.

The internal memory 121 may be used to store computer-executable program codes including instructions. The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor. In the embodiment of the present application, the internal memory 121 may store executable program codes, instructions, etc. for the electronic device 100 to execute the video shooting method of the present application.

The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .

Speaker 170A, also referred to as a "horn", is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music through speaker 170A, or listen to hands-free calls.

The receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the receiver 170B can be placed close to the human ear to listen to the voice.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can put his mouth close to the microphone 170C to make a sound, and input the sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 may be provided with two microphones 170C, which may also implement a noise reduction function in addition to collecting sound signals. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions, etc. In the embodiment of the present application, the microphone 170C can be used to collect real-time audio data, and the electronic device 100 can judge whether the real-time audio data meets the jitter trigger condition based on the feature analysis of the collected audio data, and then determine whether to trigger the jitter to achieve A shaking effect appears in the captured video. In some other embodiments, the audio data collected by the microphone 170C can also be used to determine the scene where the electronic device 100 is located at this time. It can be understood that several scenes can be preset on the electronic device 100, for example, the scene shown in FIG. 2 above can be preset. In the various scenarios shown, the feature values corresponding to the audio data in different scenarios are different.

The earphone interface 170D is used for connecting wired earphones. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, pressure sensor 180A may be disposed on display screen 194 . There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may be comprised of at least two parallel plates with conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. In this embodiment of the present application, during the process of shooting a video, the electronic device 100 can use the pressure sensor 180A to detect the touch position to focus, so as to select the target focus at the touch position, for example, a certain focal person, that is, the context described focus object.

The gyro sensor 180B can be used to determine the motion posture of the electronic device 100 . In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y and z axes) may be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, and the processor 110 calculates the distance to be compensated by the optical anti-shake module 192 according to the angle, and the optical anti-shake module 192 controls the lens to reverse The movement counteracts the shake of the electronic device 100 to achieve anti-shake. In the embodiment of the present application, when the shaking shooting mode is started when shooting a video, the electronic device 100 can be set to turn off the work of the anti-shaking system in the camera module in the anti-shaking shooting mode, for example, to suppress the execution of the anti-shaking algorithm, so as to facilitate In shooting video, the purpose of presenting shaking special effects in the captured video is achieved by controlling the vibration of the motor 191 or the motor in the camera module, or controlling the lens shake through the optical anti-shake module 192 . In some other embodiments, it is also possible to add a shaking algorithm based on the anti-shaking algorithm in the anti-shaking system of the electronic device 100, so that in the shaking shooting mode, the processor 110 of the electronic device 100 detects the vibration based on the gyroscope sensor 180B The distance to be compensated calculated by the jitter angle becomes multiplied, so that the purpose of presenting a jitter special effect in the captured video is achieved by over-compensating the distance, which is not limited here. In some other embodiments, when the electronic device 100 enters the shaking shooting mode, the processor 110 can also control to send a false anti-shake command including a false compensation distance to the anti-shake system, so as to control the optical anti-shake module 192 to actively control the lens shake In this way, a shake effect is generated to achieve the purpose of presenting a shake special effect video in the captured video, which is not limited here. Wherein, the relevant working principles of the anti-shake system in the camera module of the electronic device 100 and the optical anti-shake module 192 will be described in detail below, and will not be repeated here.

The acceleration sensor 180C can detect the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to recognize the posture of the electronic device 100, and be applied to applications such as horizontal and vertical screen switching, pedometer, etc. In the embodiment of the present application, the acceleration sensor 180C may be used to identify whether the electronic device 100 is in a portrait posture or a landscape posture when shooting a video, so as to adaptively adjust the display form of the camera shooting interface.

The distance sensor 180D is used to measure the distance. The electronic device 100 may measure the distance by infrared or laser. In the embodiment of the present application, in a video shooting scene, the electronic device 100 may use the distance sensor 180D for distance measurement to achieve automatic and fast focusing.

The touch sensor 180E is also called "touch device". The touch sensor 180E may be disposed on the display screen 194, and the touch sensor 180E and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180E is used to detect a touch operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .

The ambient light sensor 180F is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180F can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180F can also cooperate with the proximity light sensor to detect whether the electronic device 100 is in the pocket, so as to prevent accidental touch.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .

The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization. The motor 191 commonly used in the electronic device 100 includes a linear motor and a rotor motor. The linear motor can support precise control of vibration intensity, vibration frequency, etc., and has a strong start-stop delay performance. In the embodiment of the present application, it is possible to present shaking special effects in the captured video by setting the vibration parameters of the motor 191 , including the above-mentioned vibration direction, vibration amplitude, and vibration frequency. In some other embodiments, in the jitter shooting mode, the purpose of presenting jitter special effects in the captured video can also be achieved by controlling the vibration of a motor (such as a voice coil motor) in the camera module of the electronic device 100 , which is not limited here.

For the convenience of description, the following takes the mobile phone 100 as the electronic device 100 as an example to introduce the specific implementation process of the solution of the present application in detail. In addition, for the convenience of description, the motor 191 shown in FIG. 3 is taken as an example in the following description, and the purpose of presenting a shaking special effect in the captured video through vibration of the motor is described.

As an example, FIG. 4 shows a schematic process of converting the collected audio data into corresponding shaking parameters, so as to present shaking special effects in the captured video.

As shown in FIG. 4 , when the mobile phone 100 opens the camera application to shoot video, the microphone 170C can collect real-time audio data, the camera 193 can collect real-time image data, and the processor 110 or image processor of the mobile phone 100 can collect real-time audio data based on the collected images. Data Extraction Focus data as described above.

As an example, the processor 110 may obtain the audio data collected by the microphone 170C, and run the audio data at any moment to extract audio feature values, such as audio sampling rate, frequency, loudness, etc. In some embodiments, based on the extracted audio feature value, the processor 110 can convert the extracted audio feature value into the vibration parameters of the motor 191 by running related algorithms, such as the vibration direction, vibration frequency, Parameters such as vibration amplitude and vibration duration make the linear motor vibrate to produce jitter effects.

In some other embodiments, the processor 110 may also convert the extracted audio feature values into lens shake parameters in the optical anti-shake module 192 by running related algorithms, for example, converting the optical anti-shake module 192 to control the lens shake process The parameters such as the shaking displacement, shaking direction, shaking speed, or shaking frequency in the lens enable the optical anti-shaking module 192 to control the lens to generate a corresponding displacement to generate a special shaking effect.

In some other embodiments, the processor 110 may also convert the extracted audio feature value into a change parameter of a real-time image, or into a change parameter of a focus object in a real-time image by running a related algorithm, wherein the real-time image is the mobile phone 100 The camera 193 real-time acquisition lens can capture the overall image data of the picture, the focus object in the real-time image is the target focus of the lens when the camera 193 collects the image data, and the focus object can be extracted from the real-time image, for example, by a matting algorithm to achieve, without limitation. The above-mentioned change data includes the real-time image or the displacement of the focus object in the real-time image in the three-dimensional space direction (XYZ axis direction), the moving direction (including the left, right, up and down displacement changes in the XZ axis direction, and the far and near displacement changes in the Y axis direction) ), the moving speed or moving frequency, and the change of outline size and other parameters, and then control the real-time image or the corresponding displacement of the focus object in the real-time image to present a shaking effect.

It can be understood that, in some other embodiments, the processor 110 can also perform feature value extraction based on other data of the video shooting scene collected by other sensing elements on the mobile phone 100, and determine whether the shaking trigger condition is met, and then initiate other functions that can generate The electronic components of the shaking special effect initiate corresponding shaking to realize, and there is no limitation here.

For example, in some other embodiments, the processor 110 may also acquire the image data collected by the camera 193 and extract the focus data about the captured target focus, and then analyze the change characteristic value of the focus object in a continuous period of time based on the focus data, The change characteristic value represents the change characteristics such as the displacement, jumping or body shaking of the focus person, and the change of the focus person in the picture captured by the camera 193. The processor 110 can, for example, run a related algorithm for The above-mentioned focus data calculates change characteristic values, such as calculating the displacement of the focus object, the change value of the image outline size of the focus object, etc., and the processor 110 converts the calculated characteristic values into the vibration parameters of the motor 191, Either convert the lens shake parameters in the optical anti-shake module 192, or convert them into the change parameters of the focus object in the real-time image, etc., to realize the shake special effect, which is not limited here. The process of extracting feature values and data conversion described above will be described in detail below, and will not be repeated here.

It can be understood that, in some other embodiments, the mobile phone 100 can also extract the change characteristic value of the focus object in the focus image based on the focus data collected by the camera 193, such as the displacement, movement direction, and movement frequency of the focus object, etc. , and then convert the change characteristic value of the focus object into the vibration parameters of the above-mentioned linear motor, the lens shake parameters in the optical image stabilization module, or the change parameters of the focus object, etc., to achieve the purpose of presenting special shaking effects in the captured video. This is not limited. The process of converting the characteristic value based on the change of the focus object into the corresponding dithering parameter will be described in detail below and will not be repeated here.

Specifically, FIG. 5 shows a schematic flow chart of the implementation of the video shooting method of the present application. It can be understood that each step of the process shown in FIG. 5 is executed by the mobile phone 100. Specifically, the processor 110 of the mobile phone 100 controls and executes the implementation process of the video shooting method shown in FIG. 5 through the controller.

As shown in Figure 5, the process includes the following steps:

501: In response to a user operation, start a camera to shoot a video.

Specifically, the user can operate the mobile phone 100, find the camera application icon in the desktop application icons of the mobile phone 100 and click the icon, start the camera, and select the recording function to shoot a video.

After the mobile phone 100 starts the camera, the user interface (User Interface, UI) for the user to select the video recording function can be referred to as shown in FIG. Refer to operation ① shown in Fig. 6A.

502: The running camera application enters a shake shooting mode.

Specifically, the mobile phone 100 can enter the shake shooting mode in response to the user's selection operation, and can also determine whether it is suitable for using shake shooting based on the audio data collected by the microphone 170C and/or the change data of the focus object in the focus image collected by the camera 193. The preset conditions of the mode to enter the shaking shooting mode. It can be understood that the jittering shooting mode refers to a shooting mode that can present jittering special effects at certain time nodes during video shooting.

As an example, the manner of making the mobile phone 100 enter the shake shooting mode may refer to FIGS. 6A to 6C .

As shown in Figure 6A, the user can slide left and right in the mode menu bar 602 displayed in the mode menu bar 602 displayed on the UI interface 601 opened by running the camera application on the mobile phone 100 to select the recording option 603, refer to the operation ① shown in Figure 6A, and click on the UI corresponding to the recording mode Click the shake mode check button 605 on the interface 604, refer to the operation ② shown in FIG. 6A, check the shake mode, and enter the shake shooting mode.

As shown in FIG. 6B , the mode menu bar 602 displayed on the UI interface 601 opened by running the camera application on the mobile phone 100 includes a shaking shooting option 606, and the user can slide left and right on the mode menu bar 602 displayed on the UI interface 606 to select shaking shooting Option 606, refer to operation ① shown in FIG. 6B , to enter the shaking shooting mode.

It can be understood that the operation ① shown in FIG. 6A is the same as the operation ① shown in FIG. 6B .

As shown in Figure 6C, as an example, some video shooting scenes suitable for enhancing emotional expression by presenting shaking special effects are preset in the mobile phone 100, such as the various scenes shown in Figure 2 above, the camera interface opened by the user on the mobile phone 100 After sliding left and right in the displayed mode menu bar 602 to select the recording option 603, during the process of shooting video, the mobile phone 100 judges based on the audio data collected by the microphone 170C and/or the change data of the focus object in the focus image collected by the camera 193. The current video shooting scene is a concert scene, and it is suitable to use the shaking shooting mode to shoot video. Then, a pop-up window prompt can be displayed on the display screen 194 of the mobile phone 100, and the prompt content is, for example, "A concert scene is detected, whether to enable the shaking shooting mode?" ", referring to the pop-up window 607 shown in Figure 6C, the user can click "Open" to enter the shake shooting mode, refer to the operation ③ shown in Figure 6C. It can be understood that in the pop-up window 607 shown in FIG. 6C , the user can also click "not open" or the close button 608 in the upper right corner of the pop-up window 607 to close the pop-up window and continue to shoot video in the current shooting mode.

In addition, it can be understood that referring to the interface shown in FIG. 6C , in other embodiments, there may be no preset scene in the mobile phone 100 or the preset scene does not include scenes such as concerts, but the memo application installed on the mobile phone 100 A memorandum for attending a concert is added, or a schedule reminder about a certain concert is set in the calendar application installed on the mobile phone 100, or a concert ticket order is automatically generated in a certain ticketing application installed on the mobile phone 100. Added a schedule reminder, etc., the mobile phone 100 can obtain the concert time recorded in applications such as memos and calendars, and detect whether the user opens the camera application to shoot videos within this time. If the mobile phone 100 detects that the user clicks on the When the camera app shoots a video, it can display a pop-up notification as shown in Figure 6C. The content of the notification can be different from that shown in Figure 6C. For example, you can refer to "The system detected that the concert time recorded in your schedule reminder has arrived , do you need to enable the shaking shooting mode now?" In other embodiments, the schedule reminders that the mobile phone 100 can obtain may also be other shooting scenes that are not limited to concerts, such as hip-hop competitions, etc. When the mobile phone 100 runs the camera application The displayed notice may also be in other forms and other content, which is not limited here.

In some other embodiments, as shown above, the mobile phone 100 can also preset feature value judgment conditions, so that in the process of shooting video, the mobile phone 100 can be based on the audio data collected by the microphone 170C and/or the audio data collected by the camera 193. Extract the corresponding eigenvalues from the image change data of the focus image, and judge whether the extracted eigenvalues meet the preset conditions suitable for adopting the jitter shooting mode, thereby prompting the user to adopt the jitter shooting mode. Window 607 may also use other prompting methods, which are not limited here.

503: Obtain video data such as real-time audio data and/or image data.

Specifically, during the video shooting process, the camera application running on the mobile phone 100 collects real-time video data, which includes but is not limited to real-time audio data collected by the microphone 170C on the mobile phone 100, real-time image data collected by the camera 193; the processing of the mobile phone 100 The controller 110 can acquire the real-time audio data collected by the microphone 170C and the image data collected by the camera 193 in real time, and extract focus data corresponding to the corresponding image data. It can be understood that, in the focus images acquired at different times, the position, outline size, and degree of clarity or blur of the focus object may change.

504: Extract corresponding audio feature values based on audio data, and/or extract corresponding focus object change feature values based on image data.

Specifically, the processor 110 of the mobile phone 100 can extract audio feature values based on the acquired real-time audio data, for example, the sampling rate, frequency, loudness, etc. of the audio can be extracted based on the real-time audio data; The focus data extracted from the real-time image data is used to calculate the relevant feature values that characterize the change characteristics of the focus object, such as the displacement of the focus object, the moving speed or frequency of the focus object, and the change value of the outline size of the focus object.

Among them, the sampling rate is the sampling frequency, also known as sampling speed or sampling rate, which defines the number of samples extracted from continuous audio signals and composed of discrete signals per second, which is expressed in Hertz (Hz).

Frequency, that is, the vibration frequency of sound, represents the number of periodic audio vibrations per unit time.

Loudness, also known as sound intensity or volume, indicates the strength of sound energy, which mainly depends on the amplitude of sound waves. The loudness of sound is generally measured by sound pressure (dyne/cm2) or sound intensity (watt/cm2), and the unit of boost pressure is Pa (Pa), and the logarithmic value of its ratio to the reference sound pressure is called the sound pressure level , the unit is decibel (dB).

The displacement of the focus object can be determined based on, for example, the change value of the position of the focus object in focus images collected at adjacent moments.

The moving speed of the focus object can be determined by, for example, the moving distance of the focus image per unit time.

The frequency of movement of the focus object can be determined by, for example, the number of times the position of the focus image moves per unit time.

The change value of the image outline size of the focus object may be determined based on, for example, the change difference value of the outline size of the focus object in focus images collected at adjacent moments.

505: Based on the audio feature value and/or the focus object change feature value, determine whether a shaking trigger condition is met. If yes, execute step 506; if no, return to step 503.

Specifically, the mobile phone 100 determines whether the shaking trigger condition for triggering shaking is met based on the audio feature value extracted in step 504 and/or the change feature value of the focus object in the focus image. It can be understood that judging whether the shaking trigger condition is met can be done in a variety of ways, as an example, it can be judged in any one of the following two ways.

Firstly, the mobile phone 100 presets an audio feature value threshold as a judgment condition. For example, the preset audio loudness threshold in the mobile phone 100 is set to 90dB, the sampling rate threshold is set to 50Hz, and the frequency threshold is set to 100Hz, then when the audio feature values extracted in the above step 504, if the audio loudness is higher than 90dB, and/ Or when the audio sampling rate is higher than 50 Hz, and/or the audio frequency is higher than 100 Hz, it may be determined that the audio feature value corresponding to the currently collected audio data satisfies the jitter trigger condition.

Second, the mobile phone 100 presets a focus object change characteristic value threshold as a judgment condition. For example, the mobile phone 100 presets the displacement threshold of the focus object in the focus image as 20 mm, the movement frequency threshold of the focus object as 50 Hz, and the outline size change threshold of the focus object as 10%, then when the focus image extracted in the above step 504 In the change characteristic value of the focus object, if the displacement of the focus object is greater than 20 mm, and/or the movement frequency of the focus object is higher than 50 Hz, and/or the change value of the outline size of the focus object exceeds 10%, the currently acquired image can be determined The change characteristic value of the focus object corresponding to the data satisfies the jitter trigger condition.

Thirdly, the mobile phone 100 presets the audio feature value threshold and the focus object change feature value threshold as common judgment conditions. For example, one or more of the above-mentioned audio feature values and one or more of the above-mentioned focus object change feature values are preset in the mobile phone 100, and are comprehensively used as a judgment condition for judging whether the shaking trigger condition is met, and each feature value threshold For an example, refer to the relevant description above, and details are not repeated here.

506: Based on the audio feature value and/or the change feature value of the focus object, convert to obtain a dither parameter.

Specifically, when in the above step 505, based on the feature value of the audio data and/or the change feature value of the focus object in the focus image, it is judged that the shaking trigger condition needs to be triggered, the processor 110 of the mobile phone 100 can run a related algorithm to convert the above step 540 Convert the audio feature value extracted from and/or the change feature value of the focus object in the focus image into the vibration parameter of the motor 191, or into the lens shake parameter in the optical anti-shake module 192, or into a real-time image or focus object The displacement parameters are used to control the shaking effect in subsequent steps, and the above vibration parameters or displacement parameters can be collectively referred to as shaking parameters.

It can be understood that the above shaking parameters include but not limited to shaking direction, shaking amplitude, shaking frequency and other parameters. As an example, the vibration parameters may correspond to vibration parameters such as the vibration direction, vibration amplitude, and vibration frequency of the motor 191, for example, based on the audio feature values extracted from the real-time audio data in step 505 above, converted into the above vibration parameters of the motor 191. The processor 110 of the mobile phone 100 can control the motor 191 to initiate a corresponding vibration corresponding to the vibration parameter based on the vibration parameter of the motor 191. At this time, the mobile phone 100 drives the camera module to vibrate with the vibration of the motor 191. At this time, the camera module Because the vibration cannot be focused, the captured video will appear as a jittering special effect of the focus object with effects such as displacement and ghosting. In some other embodiments, the focus data may also be extracted based on the real-time image data in step 505 and the calculated change characteristic value of the focus object may be converted into the above vibration parameters of the motor 191 , which is not limited here.

It can be understood that in the shaking shooting mode, the processor 110 of the mobile phone 100 can control to turn off the anti-shake function of the anti-shake system corresponding to the camera module, so that the vibration of the above-mentioned motor 191 can present an obvious shaking effect in the captured video. . In some other embodiments, the above shaking parameters may also correspond to the vibration parameters of the motor in the camera module, which will not be repeated here.

As an example, the above shake parameters may also correspond to compensation parameters such as the displacement, direction, speed, or frequency of the optical image stabilization module 192 to control the lens to perform compensation, for example, based on the real-time audio data extracted in step 505 above. The audio characteristic value is converted into the above-mentioned displacement parameter in the optical anti-shake module 192 . As mentioned above, when the shaking shooting mode is started when shooting a video, the mobile phone 100 can be set to turn off the anti-shaking system in the camera module in the anti-shaking shooting mode, and the processor 110 of the mobile phone 100 is based on the calculated The lens shake parameter in 192 controls the lens to vibrate accordingly. At this time, the camera module cannot focus due to the vibration, so in the captured video screen, the focus object has a shaking effect such as displacement and ghosting. In some other embodiments, the focus data can also be extracted based on the real-time image data in step 505 above, and the calculated change characteristic value of the focus object can be converted into the above displacement parameters in the optical anti-shake module 192, which is not limited here. .

It can be understood that the anti-shake system of the mobile phone 100 may be, for example, an anti-shake system based on Optical Image Stabilization (OIS) technology. Anti-shake, wherein, the principle of anti-shake based on the OIS system is shown in FIG. 7 , when the mobile phone 100 moves during lens exposure, the image sensor is used to sense light and convert the light signal into an analog image signal, and convert the analog image After being converted into a data image, it is sent to the image processor for processing. The OIS controller reads the data of the gyroscope sensor 180B to obtain the motion data of the mobile phone. The OIS controller drives the X-axis OIS motor to move the lens in the X-axis direction according to the motion data. And push the Y-axis OIS motor to move the lens in the Y-axis direction. The X-axis Hall sensor detects the movement of the lens in the X-axis direction, the Y-axis Hall sensor detects the movement of the lens in the Y-axis direction, and the Y-axis Hall sensor transmits the real-time position of the lens moving in the Y-axis direction to the OIS control The OIS controller continues to move the lens according to the new lens position and the new motion data acquired by the gyroscope sensor 180B, so that the closed-loop anti-shake control can be realized continuously. I won't repeat them here.

As an example, the above shaking parameters may also correspond to parameters such as the displacement, moving direction, moving speed or moving frequency of the focus object, screen brightness, screen filter, etc., for example, based on the audio feature value extracted from the real-time audio data in step 505 above. Converted to the change parameter of the real-time image, or converted to the change parameter of the focus object in the real-time image, the processor 110 or the image signal processor (Image Signal Processor, ISP) of the mobile phone 100 processes the real-time image or the change parameter in the real-time image based on the above-mentioned change parameter The focus object produces a shaking effect such as zooming or moving. In some other embodiments, the focus data can also be extracted based on the real-time image data in step 505 above, and the calculated characteristic value of the focus object can be converted into a real-time image or a change parameter of the focus object in the real-time image, which is not done here. limit. For picture brightness and picture filter, different brightness values or filter values can be corresponding to different brightness values or filter values according to the obtained audio feature value and/or the change feature value of the focus object, so that the picture presents a shaking effect.

It can be understood that, in some other embodiments, based on the audio feature value and/or the change feature value of the focus object, while converting the dithering parameters, image brightness change parameters, image color transformation parameters, etc. associated with the dithering parameters can also be generated. , or add dynamic pictures at the same time, such as colorful heart-shaped, star-shaped pictures, etc., to achieve the effect of embellishment, and there is no limitation here.

The above data conversion process and the description related to the determination of the jitter mode and the jitter parameter will be described in detail below in conjunction with the accompanying drawings, and will not be repeated here.

507: Presenting the shaking special effect in the video, completing the shooting of the video with the shaking special effect.

Specifically, at certain time nodes or within a certain time period when the audio feature value and/or the focus object change feature value meet the shaking trigger condition, the processor 110 controls the motor based on the shaking parameter of the motor 191 obtained in step 506 above. 191 generates corresponding vibrations; or the processor 110 drives the optical anti-shake module 192 to control the lens to generate corresponding displacement based on the corresponding shake parameters of the optical anti-shake module 192 obtained in the above step 506; or the processor 110 in the mobile phone 100 or the ISP based on The real-time image or the shaking parameters of the focus object in the real-time image obtained in the above step 506 runs the corresponding image processing algorithm to control the real-time image or focus object to generate real-time scaling, displacement or screen brightness value changes to present shaking effects.

It can be understood that while the processor 110 of the mobile phone 100 controls the presentation of shaking effects at certain time nodes in the video, it can determine whether to add a sound effect beat or music to the shaking special effects presented at a certain time node based on the above-mentioned audio characteristic value. Fragments, in order to present better jitter effects in conjunction with the corresponding screen jitter. For example, when shooting a sports video, you can add "咚~咚~咚" or "Wow~Wow~" at a wonderful moment showing skills special effects; when shooting videos of hip-hop competitions, you can add special effects such as "bang~bang~" or "crack~crack~" at the wonderful moment of showing hip-hop skills; or when shooting robot dance videos, you can present shaking special effects Add special effects such as "Zi~Zi~Zi" to the wonderful moments. In other embodiments, if the live music of the video or the collected audio has a strong sense of rhythm, it is also more suitable for presenting shaking special effects in conjunction with the shaking of the picture, for example, when shooting a video of a rap performance or shooting some singing When jumping to the video of the performance, the rhythm of the background music is already very strong. At this time, it is not necessary to add corresponding sound effect beats to the shaking of the screen to present the shaking special effect, and there is no limitation here.

It can be understood that the processor 110 of the mobile phone 100 controls the duration of presenting the shaking special effect at certain time nodes in the video, which can be determined based on the change of the corresponding audio feature value of the real-time audio data, that is, the audio feature value corresponding to the real-time audio data satisfies the shaking trigger. The vibration effect is triggered at the moment of the condition, and the audio characteristic value corresponding to the real-time audio data does not meet the vibration trigger condition to stop the vibration effect; in other embodiments, it is also possible to set a fixed duration for the vibration effect triggered each time the vibration trigger condition is met. , for example, set to 3 seconds or 5 seconds, etc.; in other embodiments, a control for controlling the duration of the shaking effect can also be set on the video shooting interface of the camera application, for example, when the user shoots a video in the shaking shooting mode, when the When the shaking trigger condition, the user can press and hold the control to present the shaking special effect within a certain time node or time period, and the duration of pressing the control is the presentation duration of the corresponding shaking special effect; in other embodiments, the triggering of the shaking special effect and Suspension or termination can also adopt other forms of rules, which are not limited here.

It can be understood that after the above step 507, the video shooting interface of the mobile phone 100 displays the completed shake special effect video, and the user can further share, save or edit the completed shake special effect video on the interface of the mobile phone 100 displaying the shake special effect video For example, as shown in FIG. 8 , the user can click the share button 802 on the interface 801 where the mobile phone 100 displays the completed shaking special effect video to share the shaking special effect video to the circle of friends or share it with friends; Click the save button 803 on the interface 801 to save the shaking special effect video to the local gallery; the user can also click the edit button 804 to further edit the completed shaking special effect video, such as adding filter effects, clipping video length, deleting the shaking special effect Part or all of the shaking special effects and other processing in the video, the user can also click the delete button 805 to delete the video, and there is no limitation here.

In addition, it can be understood that the audio feature value and/or the change feature value of the focus object in step 503 above can be collected by other electronic devices, such as a speaker in a watch. After the watch collects audio data and processes the audio feature value, it can The collected audio data or the processed audio feature values are sent to the mobile phone 100, and the mobile phone 100 generates the shaking parameters accordingly. In other embodiments, the watch can also process the audio data by itself according to the collected audio data or the processed audio feature values And generate a shaking parameter, and then send the shaking parameter to the mobile phone 100, and the mobile phone 100 continues to execute subsequent steps 504 to 507 based on the corresponding shaking parameter. In some other embodiments, the video captured by the mobile phone 100 can also be sent to the watch for shaking processing, that is, the watch generates corresponding shaking parameters according to the audio data collected by the speaker, and performs shaking processing on the video sent by the mobile phone 100 to obtain The video with special shaking effects is then sent to the mobile phone 100, and the mobile phone displays it to the user, or the watch processes the video with special shaking effects and then directly displays it to the user. There is no limitation here.

Regarding the related description of the data conversion process involved in the above step 506 and the determination of the jitter parameters, as an example, the above data conversion process will be described in detail below with reference to FIGS. 9A to 9C .

Referring to FIG. 9A , after the mobile phone 100 starts the camera application to shoot video and enters the shaking shooting mode, the microphone 170C collects live audio to generate audio data. Referring to the related process described in step 504 above, the processor 110 of the mobile phone 100 analyzes the audio data and extracts the obtained Audio feature value, when the audio feature value meets the shaking trigger condition, the processor 110 converts the obtained audio feature value into a vibration parameter (ie, a shaking parameter) of the motor 191, for example, the converted shaking parameter of the motor 191 includes a shaking amplitude of 2mm , Jitter frequency 30Hz. It can be understood that the vibration of the motor 191 can drive the lens of the mobile phone 100 and the camera 193 to vibrate so as to realize the above-mentioned jitter effect. During the vibration of the motor 191, the lens of the camera 193 cannot focus, which will cause the imaging picture of the camera application to vibrate and lose focus. For the shaking effect, refer to the following figure 12A.

As shown in FIG. 9B , after the mobile phone 100 starts the camera application to shoot video and enters the shaking shooting mode, the microphone 170C collects the audio of the scene to generate audio data. Referring to the related process described in step 504 above, the processor 110 of the mobile phone 100 analyzes the above audio data to extract The audio feature value is obtained, and when the audio feature value satisfies the shaking trigger condition, the processor 110 converts the obtained audio feature value into lens shake parameters (ie, shake parameters) in the optical anti-shake module 192, such as the converted optical anti-shake The lens shaking parameters in the shaking module group 192 include a shaking amplitude of 5 mm and a shaking frequency of 50 Hz. It can be understood that the optical anti-shake module 192 can control the lens shake to achieve the above-mentioned shaking effects. During the process of controlling the lens shake by the optical anti-shake module 192, the lens of the camera 193 cannot focus, which will cause the imaging picture of the camera application to shake and lose focus. For the shaking effect, refer to the following figure 10B.

As shown in FIG. 9C , after the mobile phone 100 starts the camera application to shoot video and enters the shaking shooting mode, the user clicks on the camera interface displayed on the mobile phone 100 to specify the focus object, and the camera 193 acquires the real-time image of the focus object to generate focus data. Refer to the description in step 504 above. In the related process, the processor 110 of the mobile phone 100 analyzes the above focus data and extracts the feature value of the focus object change. When the audio feature value meets the shaking trigger condition, the processor 110 converts the obtained feature value of the focus object change into a real-time image or a real-time image The shaking parameters of the focus object, that is, the above-mentioned change parameters, for example, the converted real-time image or the shaking parameters of the focus object in the real-time image include a shake amplitude of 10mm (for example, the upper limit of the displacement of the focus object in the XYZ axis direction is 10mm or the focus object The upper limit of the change value of the outline size is 10 mm), and the shaking frequency is 50 Hz (for example, the frequency of the displacement change of the focus object or the change frequency of the outline size). It can be understood that the image processor or ISP on the mobile phone 100 can execute an image processing algorithm to control the focus object in the real-time image to perform real-time scaling or displacement according to the above-mentioned change parameters to generate a shaking effect. The presentation form of the shaking effect can refer to the above-mentioned FIG. 1D The presentation form of the jitter special effect is shown, and will not be repeated here. It can be understood that the image processing algorithm for presenting the shaking effect can be performed after the image processor or ISP on the mobile phone 100 performs noise reduction, distortion correction or automatic filter processing on the image data collected by the camera 193, so as not to affect the original image The data must be saved. Therefore, in some embodiments, when the image processor or ISP executes the image processing algorithm to control the real-time image or extract the focus object in the real-time image for zooming or moving to present the shaking special effect, the user completes the video shooting and then generates Videos with jittery effects may experience long processing delays.

It can be understood that the above-mentioned process of controlling the real-time image or the focus object in the real-time image to produce zoom or displacement changes through the image processing algorithm to generate the shaking special effect does not involve the shaking process of any electronic components in the mobile phone 100 that affect the imaging of the camera application. The jitter effect generated in the camera application will only cause jitter in the imaging screen of the camera application, such as real-time image zooming or displacement to produce jitter special effects, or focus object zooming or displacement in real-time images to produce jitter special effects, etc. This process will not affect the focus of the lens. Therefore, there will be no out-of-focus effects such as ghost images.

It can be understood that the special effects video shooting solution provided by the present application is based on the conversion of the audio feature value and/or the change feature value of the focus object in the focus image to obtain the shaking parameters, including but not limited to the conversion forms shown in FIGS. 9A to 9C above, for example, It may include a conversion form based on the change characteristic value of the focus object into the shake parameter of the motor 191, etc., which is not limited here. In addition, it can be understood that the transformation form adopted in the special effect video shooting solution provided by the present application may be any one or a combination of transformation forms shown in FIGS. 9A to 9C above, which is not limited here.

It can be understood that during the implementation of the video shooting method shown in FIG. 5 above, the shaking parameters can be based on the real-time audio data and/or the focus data extracted from the real-time image data, and then extract the corresponding audio feature value and/or the change feature of the focus object The value is obtained through conversion, and there will be a time delay between the time node showing the shaking effect in the captured video and the time node when the audio feature value and/or the change feature value of the focus object meets the shaking trigger condition. In addition, it can be understood that the corresponding relationship between the audio feature value and/or the change feature value of the focus object and the shaking parameter can be preset in the mobile phone 100, and refer to the data conversion process in FIGS. 9A to 9C to realize real-time audio data and/or For the process of converting real-time image data into shaking parameters, in some other embodiments, corresponding data conversion formulas can also be preset in the mobile phone 100 to realize the above data conversion, which will not be repeated here.

In addition, in some other embodiments, the mobile phone 100 may also provide the user with a shaking parameter setting interface, as shown in FIGS. 10A to 10D below.

As an example, FIGS. 10A to 10D show UI interfaces corresponding to the user operating the mobile phone 100 to set shaking parameters.

As shown in Figure 10A, the user can run the camera application on the mobile phone 100 to enter the shake shooting mode in the UI interface 1001 displayed, click the setting button 1002 in the upper right corner, refer to the operation ④ shown in Figure 10A, and enter the setting shown in Figure 10B Interface 1003.

As shown in FIG. 10B , the user can click on the shaking shooting mode option 1004 on the setting interface 1003 , refer to the operation ⑤ shown in FIG. 10B , and call out the shaking parameter setting window 1005 shown in FIG. 10C . In some other embodiments, the user clicks the shaking shooting mode option 1004 on the setting interface 1003 shown in FIG. 10B to call out the shaking parameter setting window 1006 shown in FIG. Do limit.

It can be understood that the user can also set the conditions for enabling the shake shooting mode on the setting interface shown in FIG. 10B , such as smart activation, or activation by user authorization, etc., which will not be repeated here.

As shown in FIG. 10C, the user can set the shake amplitude in the shake parameter setting window 1005. For example, the check box 1051 after selecting the close option can turn off the shake shooting mode. Referring to the operation ⑥ shown in FIG. The selection box 1052 can set the shaking range to 2mm, the check box 1053 after selecting the 5mm option can set the shaking range to 5mm, and the check box 1054 after selecting the 10mm option can set the shaking range to 10mm. In other embodiments, the shaking The parameter setting window 1005 may also provide options for other numerical jitter amplitudes, which are not limited here.

In addition, as shown in Figure 10C, the user can also set the shaking frequency in the shaking parameter setting window 1005, for example, select the check box 1055 after the close option to turn off the shaking shooting mode, refer to the operation ⑦ shown in Figure 10C, and select the 30Hz option The last check box 1056 can set the jitter frequency to 30 Hz, and the check box 1057 after selecting the 50 Hz option can set the jitter amplitude to 50 Hz. In other embodiments, the jitter parameter setting window 1005 can also provide other numerical jitter frequency options , without limitation here.

As shown in Figure 10D, the user can set the shaking mode in the shaking parameter setting window 1006, for example, select the check box 1061 after the close option to turn off the shaking shooting mode, refer to the operation ⑧ shown in Figure 10C, and select the small amplitude shaking mode option The last check box 1062 can turn on the small shake shooting mode, the check box 1063 after selecting the shake mode option can turn on the medium shake shooting mode, and the check box 1064 after choosing the big shake mode option can turn on the big shake As for the shooting mode, it can be understood that the shaking amplitude corresponding to the shooting mode with small shaking, the shooting mode with medium shaking and the shooting mode with large shaking increases from small to large, and/or the shaking frequency increases from low to high. The user can also select the check box 1065 next to the automatic option to start the automatic shooting mode. After the automatic shooting mode is turned on, the mobile phone 100 can automatically match the above-mentioned small shaking shooting mode and medium shaking shooting mode based on the collected real-time audio data and/or focus data. For video shooting in shooting mode or large shaking shooting mode, the process of obtaining and processing real-time audio data and/or focus data by mobile phone 100 refers to the relevant descriptions in steps 503 to 505 above, which will not be repeated here.

In the shake parameter setting window 1006 shown in Figure 10D, you can also click the preview button 1066 corresponding to each shake mode option to preview the shake special effect of the corresponding shake mode, refer to the operation ⑨ shown in Figure 10D, you can open the The shaking effect preview window 1101 of the selected shaking effect preview window 1101, the user clicks the play button 1102 on the window 1101 to watch the shaking special effect video preview effect of the corresponding shaking mode.

The shake special effects corresponding to the small shake shooting mode, the medium shake shooting mode, or the large shake shooting mode shown in FIG. 10D can be referred to in FIGS. 12A to 12C .

As shown in FIG. 12A , it is a schematic diagram showing a small-scale shake special effect in a video shot in the small-scale shake shooting mode.

As shown in FIG. 12B , it is a schematic diagram showing a medium-range shake special effect in a video captured in the medium-range shake shooting mode.

As shown in FIG. 12C , it is a schematic diagram showing a large-scale shake special effect in a video shot in the large-scale shake shooting mode.

As an example, FIG. 13 shows a schematic diagram of the system architecture of the mobile phone 100 involved in the process of shooting video with special effects through motor vibration in the camera module in some embodiments of the present application.

As shown in FIG. 13 , the mobile phone 100 includes an APP layer 1340 , a Hal layer 1330 , a Kernel layer 1320 and a hardware layer 1310 . Among them, the hardware layer 1310 includes physical devices such as the microphone 170C, the register 11, the encoding module 12, the focusing coil 13, the Hall sensor 14, and the motor; the Kernel layer 1320 includes the shake control module 21; the Hal layer 1330 includes the selection module 31 and The image processing module 32 ; the APP layer 1340 includes a camera application 41 .

Specifically, in the embodiment shown in FIG. 13 , the camera application 41 in the APP layer 1340 starts shooting video, the focusing coil 13 can drive the lens to move to achieve focusing, and the Hall sensor 14 can obtain the actual position of the motor during focusing to control the position of the motor. Move to complete the focusing process. The camera application 41 calls the parameter selection module 31 and the image processing module 32 in the Hal layer 1330 respectively after receiving the instruction to enter the shaking shooting mode. Then, the parameter selection module 31 obtains the collected audio data from the microphone 170C in the hardware layer 1310 and extracts corresponding audio feature values, and obtains specification parameters from the motor in the hardware layer 1310 . Then, the parameter selection module 31 determines the corresponding jitter parameters according to the above-mentioned audio feature value and specification parameters, and sends the jitter parameters to the jitter control module 21 in the Kernel layer 1320 . The shake control module 21 controls the linear motor to generate vibrations that match the above shake parameters. It can be understood that the vibration of the motor causes the image image of the camera application 41 to shake. At this time, the focus coil 13 cannot focus, resulting in out-of-focus. When the motor stops vibrating, the Hall sensor 14 obtains the actual position of the motor again to control the movement of the motor, and the auxiliary focusing coil 13 completes focusing.

In this application, the hall sensor 14, the register 11, the encoding module 12, and the focus coil 13 are used to form a closed-loop dynamic adjustment of the imaging screen shake of the camera application caused by the motor and the focus process before and after out of focus.

As an example, FIG. 14 shows a schematic structural diagram of a video shooting device according to an embodiment of the present application.

As shown in FIG. 14 , the video shooting device 1400 includes a data collection module 1410 , a shaking parameter generating module 1420 and a shaking video generating module 1430 .

Among them, the data acquisition module 1410 is used to acquire video data such as real-time audio data and/or image data. The specific process for the data acquisition module 1410 to acquire video data such as real-time audio data and/or image data can refer to the relevant description in the above-mentioned step 503, which will not be repeated here.

The shaking parameter generating module 1420 is configured to acquire video data such as audio data and/or image data collected by the data collecting module 1410, and determine shaking parameters based on the audio data and/or image data. For the specific process of determining the shaking parameters by the shaking parameter generating module 1420 based on the audio data and/or image data collected by the data collecting module 1410 , reference may be made to the relevant descriptions in the above steps 504 to 506 , which will not be repeated here.

Shaking video generating module 1430, configured to obtain the shaking parameters determined by shaking parameter generating module 1420, and shoot a video with shaking special effects based on the shaking parameters, or perform shaking processing on the captured video based on the shaking parameters to obtain a video with shaking special effects . The shaking video generation module 1430 shoots a video with shaking effects based on the shaking parameters, or performs shaking processing on the captured video based on the shaking parameters to obtain a video with shaking effects based on the shaking parameters. Let me repeat.

Reference to "one embodiment" or "an embodiment" in the specification means that a specific feature, structure or characteristic described in connection with the embodiment is included in at least one exemplary implementation or technology disclosed according to the present application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

The present disclosure also relates to means for performing operations in text. This apparatus may be specially constructed for the required purposes or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer readable medium such as, but not limited to, any type of disk including floppy disk, compact disk, CD-ROM, magneto-optical disk, read-only memory (ROM), random-access memory (RAM) , EPROM, EEPROM, magnetic or optical card, application specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions, and each may be coupled to a computer system bus. Furthermore, computers referred to in the specification may comprise a single processor or may be architectures involving multiple processors for increased computing power.

The processes and displays presented herein are not inherently related to any specific computer or other device. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform one or more method steps. The architecture for a variety of these systems is discussed in the description below. In addition, any specific programming language sufficient to implement the techniques and embodiments disclosed herein may be used. Various programming languages can be used to implement the present disclosure, as discussed herein.

Additionally, the language used in this specification has been chosen primarily for readability and instructional purposes and may not have been chosen to delineate or limit the subject matter disclosed. Accordingly, the present disclosure is intended to illustrate, not to limit, the scope of the concepts discussed herein.

Claims

A video shooting method applied to electronic equipment, characterized in that it comprises:

The electronic device acquires first data, where the first data includes first audio data and/or first image data;

determining, by the electronic device, a dithering parameter based on the first data;

The electronic device generates second data based on the jitter parameter, and the second data is video data;

The second data generated by the electronic device based on the jitter parameters is specifically: a second video shot based on the jitter parameters, or the electronic device shoots a second video and processes the second video according to the jitter parameters to obtain the third video of .
The method according to claim 1, characterized in that the method comprises:

When the first audio data satisfies a first preset condition; and/or the first image data satisfies a second preset condition,

The electronic device determines a dithering parameter based on the first data.
The method according to claim 2, wherein the first audio data satisfies a first preset condition, including at least one of the following:

The sampling rate of the first audio data is greater than a preset sampling rate threshold;

The frequency of the first audio data is greater than or equal to a preset frequency threshold;

The loudness of the first audio data is greater than or equal to a loudness threshold.
The method according to claim 2, wherein the image data satisfies a second preset condition, including at least one of the following:

The displacement of the focus object in the first image data is greater than or equal to a preset displacement threshold;

The movement frequency of the focus object in the first image data is greater than or equal to a preset frequency threshold;

The change value of the contour size of the focus object in the first image data is greater than or equal to a preset change value threshold.
The method according to any one of claims 1 to 4, wherein the electronic device includes a preset correspondence between audio data and/or image data and the shaking parameters, and the electronic device is based on The first data determines a jitter parameter, including:

The electronic device selects a dithering parameter matching the first audio data and/or the first image data from the preset correspondence.
The method according to claim 5, wherein the electronic device includes a plurality of motors, and the plurality of motors include motors in a camera module of the electronic device; and

The shaking parameters include vibration parameters of at least one motor of the plurality of motors.
The method according to claim 6, wherein the vibration parameters of the motor include at least one of vibration direction, vibration amplitude, and vibration frequency.
The method according to claim 6 or 7, wherein the second video taken based on the shaking parameters comprises:

Based on the vibration parameter of the motor, the electronic device controls the motor to generate vibration and captures the second video.
The method according to claim 5, wherein the electronic device includes an optical anti-shake module, and

The shake parameters include lens shake parameters in the optical anti-shake module.
The method according to claim 9, wherein the lens shaking parameters include at least one item of shaking displacement, shaking direction, and shaking frequency during the lens shaking process.
The method according to claim 9 or 10, wherein the second video taken based on the shaking parameters comprises:

The electronic device turns off the anti-shake function of the optical anti-shake module, and based on the lens shake parameters, controls the lens shake through the optical anti-shake module, and shoots to obtain the second video.
The method according to claim 5, wherein the jitter parameters further comprise:

Change parameters of the second image data collected when the electronic device shoots the second video; or

The change parameters of the focus object in the second image data collected when the electronic device shoots the second video.
The method according to claim 12, wherein the change parameters of the second image data or the change parameters of the focus object in the second image data include scaling, displacement, moving direction, moving speed, moving at least one of the frequencies.
The method according to claim 12 or 13, wherein the electronic device shoots the second video, and processes the second video according to the shaking parameters to obtain a third video, including any of the following:

The electronic device shoots a second video, and processes the second video according to a change parameter of the second image data to obtain the third video;

The electronic device shoots a second video, and processes the second video according to a change parameter of the focus object in the second image data to obtain the third video.
The method according to any one of claims 1 to 14, wherein the video shooting interface of the electronic device includes a shaking mode control, and the method further includes:

In response to a user's operation on the shaking mode control, the electronic device generates second data based on the shaking parameters.
The method according to any one of claims 1 to 14, further comprising:

When detecting that the current shooting scene is a preset scene, the electronic device generates second data based on the shake parameter.
The method according to any one of claims 1 to 16, wherein the electronic device generates second data based on the jitter parameters, further comprising:

The electronic device adds a sound effect beat or a music segment matching the content of the second data.
An electronic device, characterized in that it includes: one or more processors; one or more memories; the one or more memories store one or more programs, when the one or more programs are described When executed by one or more processors, the electronic device executes the video shooting method according to any one of claims 1 to 17.
A computer storage medium, characterized in that instructions are stored on the storage medium, and when the instructions are executed on a computer, the computer executes the video shooting method according to any one of claims 1 to 17.