WO2023231622A1

WO2023231622A1 - Video editing method and electronic device

Info

Publication number: WO2023231622A1
Application number: PCT/CN2023/089100
Authority: WO
Inventors: 韩钰卓; 朱世宇; 张志超; 代秋平; 张农; 杜远超
Original assignee: 荣耀终端有限公司
Priority date: 2022-05-30
Filing date: 2023-04-19
Publication date: 2023-12-07
Also published as: WO2023231622A9; CN116055861A; CN116055861B

Abstract

Provided in the present application is a video editing method. The method can be applied to terminal devices such as a mobile phone and a tablet computer. By implementing the method, a terminal device can crop a video while recording the video, generate a close-up video centred on a protagonist in the video, and store the close-up video. In addition, the terminal device can also crop a locally stored video, and also generate and store a close-up video centred on a protagonist in the video. In this way, during a video recording process or for a recorded video, a user can set a protagonist for photographing and obtain a close-up video centred on a protagonist in the video, so as to meet personalized usage requirements of the user.

Description

A video editing method and electronic device

This application claims priority to the Chinese patent application filed with the China Patent Office on May 30, 2022, with the application number 202210603653.3 and the application name "A video editing method and electronic device", the entire content of which is incorporated by reference into this application middle.

Technical field

The present application relates to the field of terminals, and in particular, to a video editing method and electronic device.

Background technique

Now, mobile phones and other terminal devices that support video shooting can implement automatic tracking shooting mode. When recording a video, the terminal device can receive the protagonist selected by the user. Then, the terminal device can always follow the protagonist during the subsequent video recording process, and obtain a close-up video in which the center of the video is always the selected protagonist.

Contents of the invention

This application provides a video editing method and electronic equipment. For the image currently collected by the camera or the image in the local video, the user can select an object in the image as the protagonist, and the electronic device can automatically track and record the protagonist in the above image. And save the close-up video of the protagonist.

In a first aspect, the present application provides a video editing method applied to electronic devices, characterized in that the method includes: displaying a first image and one or more markers associated with the first image in a first interface; An image includes one or more objects, and one or more markers associated with the first image respectively correspond to one or more objects in the first image; the first image is an image currently collected by the camera of the electronic device, or stored in the electronic device A frame of image in the first video; detecting the first operation acting on the first mark; in response to the first operation, determining the first object as the protagonist, obtaining a close-up image centered on the protagonist; a first image associated or the plurality of marks include a first mark, one or more objects in the first image include the first object, and the first mark corresponds to the first object; based on the close-up image centered on the protagonist, a second video centered on the protagonist is generated .

Implementing the embodiments of this application, the user can select an object in the image collected by the camera as the protagonist; when recording the original video collected by the camera, the electronic device can automatically track the protagonist in the image sequence collected by the camera and record a close-up of the protagonist video. The electronic device can display the local video selected by the user, and the user can select an object in one frame of the local video as the protagonist; the electronic device can automatically track and record the protagonist in the above frame of the image and subsequent images in the local video. And save the close-up video of the protagonist.

In one implementation, after determining that the first object is the protagonist, the method further includes: displaying a second image and one or more markers associated with the second image in the first interface, where the second image includes one or more objects, One or more markers associated with the second image respectively correspond to one or more objects in the second image; the second image is an image after the first image collected by the camera of the electronic device, or after the first image in the first video A frame of image; detecting the fifth operation acting on the second mark; in response to the fifth operation, switching the protagonist to the second object, one or more marks associated with the second image include the second mark, and the second image The one or more objects include a second object, and the second mark corresponds to the second object; the obtaining a close-up image centered on the protagonist includes: generating a A close-up image centered on the first object, based on the second image and subsequent images, generates a second pair of a close-up image centered on the object; the second video includes a first sub-video and a second sub-video, the first sub-video is a video generated based on a close-up image centered on the first object, and the second sub-video is based on a close-up image centered on the second object Video generated for a close-up image of the center. Implementing the embodiments of this application, during the process of recording the original video, when recording, pausing the recording, or stopping recording the close-up video of the first object, the electronic device can also determine a new protagonist based on the image collected by the camera, for example, switch the protagonist from the first object as the second object, and then record a close-up video of the second object. When the electronic device displays the local video, the user can also select another object in another frame of the local video as the new protagonist, for example, switching the protagonist from the first object to the second object; the electronic device can select the other object in the local video. The main character in one frame of image and subsequent images is automatically tracked, and a close-up video of the second object is recorded. In this application, the electronic device can save the close-up video of the first object and the close-up video of the second object respectively, or it can combine the close-up video of the first object and the close-up video of the second object into one video and save them.

In one implementation, the step of obtaining a close-up image centered on the protagonist is specifically: based on the image including the first object from the first image to the last frame of the image in the first video, generating a close-up image centered on the first object. close-up image. By implementing the embodiments of this application, for local videos, a close-up video with only one object as the protagonist can be recorded.

In one implementation, when the second image is a frame of image after the first image in the first video, before the first interface displays the first image and one or more markers associated with the first image, the method It also includes: displaying a thumbnail of the first video; detecting a second operation acting on the thumbnail of the first video; displaying the first image and one or more markers associated with the first image on the first interface, including: In response to the second operation, the first frame image of the first video and one or more marks corresponding to one or more objects in the first frame image are displayed on the first interface, and the first image is the first frame image. Implementing the embodiments of this application, the user can trigger the electronic device to display the first interface for playing the local video through the thumbnail of the local video displayed in a specific application of the electronic device (such as the gallery); display the first interface of the local video on the first interface. When an image is framed, the markers corresponding to each object in the image can be automatically displayed without user operation, allowing the user to select the protagonist.

In one implementation, when the second image is a frame of image after the first image in the first video, before the first interface displays the first image and one or more markers associated with the first image, the method It also includes: displaying the first frame image of the first video and the first control on the first interface; detecting a third operation acting on the first control; responding to the third operation, playing the first video; displaying the first video on the first interface. An image and one or more tags associated with the first image include: when the first video is played to the M-th frame image, displaying the above-mentioned M-th frame image on the first interface, and one or more tags associated with the above-mentioned M-th frame image. mark. When implementing the embodiments of this application, when the first interface displays an image in a local video, the user can trigger the electronic device to display marks corresponding to each object in the image through a specified operation. In this way, there is no need to perform object recognition on each frame of the local video, saving the power consumption of object recognition.

In one implementation, when the first video is played to the Mth frame image, the Mth frame image and one or more markers associated with the Mth frame image are displayed on the first interface, including: when the Mth frame image is played, the Mth frame image is displayed on the first interface. When a video is played to the Mth frame image, a fourth operation acting on the first control is detected; in response to the fourth operation, the playback of the first video is paused, and the Mth frame image currently played is displayed; in response to the operation of pausing playback , displaying one or more markers associated with the M-th frame image on the M-th frame image. When implementing the embodiments of the present application, when the local video is paused, the electronic device only displays the marks corresponding to each object in the image for the image currently displayed in the local video. In this way, there is no need to perform object recognition on each frame of the local video, saving the power consumption of object recognition.

In one implementation, the first interface also includes a second control, which is based on a close-up image centered on the protagonist, Generating a second video centered on the protagonist includes: detecting a sixth operation acting on the second control; and generating a second video centered on the protagonist based on a close-up image centered on the protagonist in response to the sixth operation. When implementing the embodiments of this application, during the process of recording a close-up video of the protagonist, the user can control the electronic device to stop recording the close-up video through a preset operation.

In one implementation, when the first image is an image currently collected by a camera of the electronic device, the second control is a control used to stop recording. Implementing the embodiment of the present application, the first interface for recording video includes a control for stopping the recording. During the process of recording the close-up video of the protagonist, the user can control the electronic device to stop recording the close-up video through the above control.

In one implementation, the method further includes: in response to the sixth operation, the camera stops collecting images, and the original video is generated and saved based on the images collected by the camera. When implementing the embodiments of this application, when the user controls the electronic device to stop recording the original video through a preset operation, the electronic device also automatically stops recording the close-up video of the protagonist.

In one implementation, after determining that the first object is the protagonist, the method further includes: displaying a first window, and displaying a close-up image centered on the protagonist in the first window. When implementing the embodiment of the present application, when recording a close-up video of the protagonist, the user can preview the recording process of the close-up video of the protagonist in real time through the first window.

In one implementation, when the first image is an image currently collected by the camera of the electronic device, the method further includes: detecting a first trigger condition, and the first trigger condition is the consecutive Y frame images after the first image. The protagonist is not included in; generating a second video centered on the protagonist based on the close-up image centered on the protagonist is specifically: in response to the first trigger condition, based on the close-up image centered on the protagonist, generating a second video centered on the protagonist second video. When implementing the embodiments of this application, it is detected that the Y-frame images continuously collected by the camera do not include the protagonist selected by the user, it is determined that the protagonist is out of the shooting range of the camera, and the electronic device stops recording the close-up video of the protagonist.

In one implementation, generating a close-up image centered on the first object based on an image including the first object between the first image and the second image includes: obtaining the first object as the center from the first image. A first close-up image of the center; a third close-up image centered on the first object is obtained from the third image; the third image is an image after the first image and before the second image; the second video includes the first close-up image and Second close-up image. By implementing the embodiments of the present application, you can locate the protagonist in the image collected by the camera or the local video image, obtain a close-up image of the protagonist in the above image, and then record the close-up image to generate a close-up video.

In one implementation, before obtaining the third close-up image centered on the first object from the third image, the method further includes: determining whether the first object is included in the third image; Obtaining a third close-up image centered on the first object, specifically: when the third image includes the first object, acquiring a third close-up image centered on the first object from the third image. By implementing the embodiments of the present application, the protagonist in the image collected by the camera or the local video can be located, and the above image can be cropped to obtain a close-up image centered on the protagonist.

In one implementation, determining that the first object is included in the third image includes: using a human body detection algorithm to identify the human body image area in the third image; when the human body image areas in the third image do not overlap, calculating the third The intersection ratio IoU distance of each human body image area in the three images and the human body image area of the protagonist in the first image; determine the first human body image area with the smallest IoU distance and satisfy the IoU distance threshold; the object corresponding to the first human body image area is Protagonist; when the human body image area in the third image overlaps, calculate the IoU distance and relocation ReID distance between each human body image area in the third image and the human body image area of the protagonist in the first image; determine the IoU distance and ReID distance and the first human body image area that is the smallest and satisfies the IoU+ReID distance threshold; the object corresponding to the first human body image area is the protagonist. By implementing the embodiments of this application, the human body image area of the protagonist can be accurately identified through the IoU distance and ReID distance of each object in the image.

In one implementation, obtaining the third close-up image centered on the protagonist from the third image specifically includes: determining the third close-up image including the first human body image area based on the first human body image area. By implementing the embodiments of this application, after identifying the human body image area of the protagonist in the image, the image can be cropped based on the human body image area to obtain the cropped close-up image of the protagonist.

In one implementation, determining a third close-up image including the first human body image area based on the first human body image area specifically includes: determining a first scaling ratio based on the first human body image area; determining a third close-up image based on the first human body image area. Three close-up image sizes. When implementing the embodiments of this application, the scaling ratio is used to reflect the size of the protagonist in the original image. The size of the close-up image of the protagonist determined based on the scaling ratio can be adapted to the small window used to display the close-up image, thereby preventing the small window from displaying the close-up image. Image deformation problem occurs.

In one implementation, determining the first scaling ratio based on the first human body image area specifically includes: determining the first scaling ratio based on the size of the largest human body image area in the third image and the size of the first human body image area. Compare. By implementing the embodiments of the present application, the zoom ratio can be adjusted in real time based on the size of the human body image area of the protagonist in each image collected by the camera or each image displayed in the local video.

In one implementation, determining the size of the third close-up image based on the first scaling ratio specifically includes: determining the size of the third close-up image based on the first scaling ratio and the preset size of the second video. When implementing the embodiments of this application, the size of the close-up image cropped from each of the above images can be adjusted in real time based on the scaling ratio corresponding to each image collected by the camera or the local video, as well as the size of the close-up video, to ensure that the close-up image can be Adapts to a small window used to display close-up images.

In one implementation, the aspect ratio of the third close-up image is the same as the preset aspect ratio of the second video. Implementing the embodiments of the present application ensures that the close-up image can adapt to the small window used to display the close-up image, thereby avoiding the problem of image deformation when the small window displays the close-up image.

In a second aspect, the present application provides an electronic device, including one or more processors and one or more memories. The one or more memories are coupled to one or more processors. The one or more memories are used to store computer program codes. The computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device causes the electronic device to execute The video editing method in any possible implementation of the first aspect above.

In a third aspect, embodiments of the present application provide a computer storage medium that includes computer instructions. When the computer instructions are run on an electronic device, the electronic device causes the electronic device to execute the video editing method in any of the possible implementations of the first aspect. .

In a fourth aspect, embodiments of the present application provide a computer program product. When the computer program product is run on a computer, it causes the computer to execute the video editing method in any of the possible implementations of the first aspect.

Description of the drawings

1A to 1M and 1O to 1P are schematic user interface diagrams of a set of protagonist mode shooting methods provided by embodiments of the present application;

FIG. 1N is a schematic diagram of a terminal 100 saving a captured close-up video in a shooting scene provided by an embodiment of the present application;

Figures 2A-2B and 2D-2E are schematic user interface diagrams of a set of protagonist mode shooting methods provided by embodiments of the present application;

Figure 2C is a schematic diagram of a terminal 100 saving a captured close-up video in a shooting scene provided by an embodiment of the present application. picture;

3A-3C are schematic user interface diagrams of a set of protagonist mode shooting methods provided by embodiments of the present application;

4A-4H are schematic user interface diagrams of a set of protagonist mode shooting methods provided by embodiments of the present application;

Figures 5A, 5B-1 to 5B-4, and 5C to 5E are schematic user interface diagrams of a set of protagonist mode shooting methods provided by embodiments of the present application;

Figure 6 is a flow chart of the terminal 100 editing and generating a close-up video in a shooting scene provided by the embodiment of the present application;

Figure 7A is a flow chart of the terminal 100 performing object recognition and marking provided by the embodiment of the present application;

Figure 7B is a schematic diagram of the terminal 100 determining the face image and human body image in the image provided by the embodiment of the present application;

Figure 7C is a schematic diagram of the terminal 100 determining the display position of the selection box provided by the embodiment of the present application;

Figure 8A is a flow chart for the terminal 100 to determine the close-up image centered on the protagonist provided by the embodiment of the present application;

Figure 8B is a flow chart for the terminal 100 to determine the size of the close-up image provided by the embodiment of the present application;

Figures 8C-8D are schematic diagrams of terminal 100 adaptively adjusting close-up images to adapt to window display provided by embodiments of the present application;

Figure 9 is a flow chart for positioning the protagonist of the terminal 100 in the rear image frame provided by the embodiment of the present application;

Figure 10A is a frame of images in which objects do not overlap in a multi-object scene provided by an embodiment of the present application;

Figure 10B is a frame image of overlapping objects in a multi-object scene provided by an embodiment of the present application;

Figures 10C to 10D are schematic diagrams of the terminal 100 using IoU position to locate the protagonist provided by the embodiment of the present application;

Figure 11 is a schematic diagram of the terminal 100 determining the ReID distance of the protagonist in the image provided by the embodiment of the present application;

Figure 12A is a flow chart of another terminal 100 locating the protagonist in the rear image frame provided by the embodiment of the present application;

Figure 12B is a flow chart of another terminal 100 editing and generating close-up videos in a shooting scene provided by an embodiment of the present application;

Figure 13 is a flow chart for the terminal 100 to generate a close-up video in the scenario of editing local videos provided by the embodiment of the present application;

Figure 14 is a schematic system structure diagram of the terminal 100 provided by the embodiment of the present application;

Figure 15 is a schematic diagram of the hardware structure of the terminal 100 provided by the embodiment of the present application.

Detailed ways

The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

In an embodiment provided by this application, terminal devices (denoted as terminal 100 , and terminal 100 will be used collectively to refer to the above-mentioned terminal devices) such as mobile phones and tablet computers with functions of photographing and image processing can identify objects in a multi-object scene. Multiple objects in the image, and automatically track the user-specified object, generate and save a close-up video of the object. At the same time, the terminal 100 can also save the original video.

Among them, the original video is composed of original images collected by the camera. The close-up video is based on the original image and cropped with the protagonist in the original image as the center. A close-up video is one in which the main character is always the center of the shot. In this way, after selecting the protagonist, the user can not only shoot a close-up video centered on the protagonist, but also obtain the original video composed of the original images collected by the original camera.

Further, the terminal 100 can also identify the objects included in the local video, and then determine the objects included in the local video according to the user's selection operation. Determine the protagonist of the video. After determining the protagonist, the terminal 100 may also perform an editing operation on the above-mentioned local video to extract the close-up video of the protagonist, thereby obtaining a close-up video with the protagonist always as the center of the shooting.

Not limited to mobile phones and tablet computers, the terminal 100 can also be a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, or a personal digital assistant. (personal digital assistant, PDA), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (artificial intelligence, AI) devices, wearable devices, vehicle-mounted devices, smart home devices and / or smart city equipment, the embodiment of this application does not place special restrictions on the specific type of the terminal.

The following is a detailed introduction to the user diagram of the terminal 100 implementing the video editing method provided by the embodiment of the present application.

First, FIG. 1A exemplarily shows the user interface 101 of the terminal 100 enabling the camera to perform a shooting action.

As shown in FIG. 1A , the user interface 101 may include a mode bar 111 , a shooting control 112 , a preview window 113 , a review control 114 , and a conversion control 115 .

The mode bar 111 may display multiple shooting mode options, such as night scene, video, photo, portrait and other shooting modes. Night scene mode can be used to take photos in dark scenes, such as taking photos at night. Video mode can be used to record videos. Photo mode can be used to take photos in daylight scenes. Portrait mode can be used to take close-up photos of people. In this embodiment of the present application, the mode bar 111 also includes a protagonist mode. The protagonist mode corresponds to the shooting method provided by the embodiment of the present application: during the process of shooting a video, the protagonist in the video is determined and automatically tracked, and the original video and the close-up video of the protagonist with the protagonist as the shooting center are saved.

The shooting control 112 may be used to receive a user's shooting operation. In a photographing scene (including photographing mode, portrait mode, and night scene mode), the above-mentioned photographing operation is an operation of controlling photographing that acts on the photographing control 112 . In the scene of recording video (video recording mode), the above-mentioned shooting operation includes an operation of starting recording on the shooting control 112 .

The preview window 113 can be used to display the sequence of image frames collected by the camera in real time. The image displayed in the preview window 113 may be called an original image. In some embodiments, what is displayed in the preview window 113 is a down-sampled sequence of image frames. At this time, the image frame sequence without downsampling processing corresponding to the image displayed in the preview window 113 may be called an original image.

The review control 114 can be used to view photos or videos taken previously. Generally, the review control 114 can display a thumbnail of a photo taken previously or a thumbnail of the first frame of a video taken previously.

User interface 101 may also include a settings bar 116 . Multiple setting controls may be displayed in the setting bar 116 . A setting control is used to set a type of parameters of the camera, thereby changing the images collected by the camera. For example, the setting bar 116 may display setting controls such as aperture 1161, flash 1162, filter 1164, etc. Aperture 1161 can be used to adjust the camera aperture size, thereby changing the brightness of the image captured by the camera; flash 1162 can be used to turn on or off the flash, thereby changing the brightness of the image captured by the camera; filter 1164 can be used to select a filter style, Then adjust the image color. The settings bar 116 may also include further settings controls 1165 . More setting controls 1165 can be used to provide more controls for adjusting camera shooting parameters or image optimization parameters, such as white balance controls, ISO controls, beauty controls, body beauty controls, etc., thereby providing users with richer shooting services. .

By default, when camera shooting is enabled, the terminal 100 can first select the camera mode, refer to the user interface 101 . During this process, the terminal 100 may detect the user operation on the mode bar 111 to select the protagonist mode, such as the operation of clicking the protagonist shooting mode option as shown in FIG. 1A , or the operation of sliding the mode bar 111 to select the protagonist shooting mode option, etc. . In response to the above operation, the terminal 100 may determine to turn on the protagonist mode for shooting.

FIG. 1B exemplarily shows the user interface 102 of the terminal 100 for shooting in the protagonist mode.

After selecting the protagonist mode, the terminal 100 can perform image content recognition (object recognition) on the image collected by the camera, and identify the objects included in the image. The above-mentioned objects include but are not limited to humans, animals, and plants. The following description of the embodiments of this application will mainly take characters as examples. While the terminal 100 displays the image collected by the camera in the preview window 113, the terminal 100 may also display a selection box on each recognized object.

Referring to the user interface 102, the images collected by the camera at a certain moment include Person 1, Person 2, and Person 3. After receiving the above-mentioned image collected and generated by the camera and before displaying the above-mentioned image, the terminal 100 may use a preset object recognition algorithm to identify objects included in the image. Here, the object recognition algorithm may include a face recognition algorithm and a human body recognition algorithm. At this time, using the above object recognition algorithm, the terminal 100 can recognize that the above image includes three objects: Person 1, Person 2, and Person 3.

Of course, in some examples, the terminal 100 is not limited to the characters 1, 2, and 3 introduced in the above user interface 102, and the terminal 100 also supports the recognition of animals and plant-type objects. Correspondingly, the above-mentioned object recognition algorithm also includes a recognition algorithm for one or more animals, and a recognition algorithm for one or more plants, which are not limited in the embodiments of the present application.

On the one hand, the terminal 100 can display the above-mentioned images including Person 1, Person 2, and Person 3 in the preview window 113. On the other hand, before displaying the above-described image, the terminal 100 may determine a selection box corresponding to each of the above-described objects. When displaying the above image, the terminal 100 may display selection boxes corresponding to each object, such as the selection box 121 corresponding to Person 1, the selection box 122 corresponding to Person 2, and the selection box 123 corresponding to Person 3. At this time, the user can confirm the video protagonist through the above selection box.

At the same time, the user interface 102 can also display prompts 125, such as "Please click on the protagonist to start automatic focus recording." Prompt 125 prompts the user to determine the protagonist of the video. According to the prompt 125, the user can click any one of the above selection boxes. The object corresponding to the selection box that the user clicks on is the video protagonist determined by the user.

The user interface 102 (protagonist mode shooting interface) may also include a focus control 126 and a beauty control 127 . The focal length control 126 can be used to set the focal length of the camera to adjust the viewing range of the camera. When the viewing range of the camera changes, the image displayed in the preview window will change accordingly. The beauty control 127 can be used to adjust the face image of the person in the image. After detecting the user operation on the beautification control 127, the terminal 100 can perform beautification processing on the characters in the image, and display the beautified image in the preview window. The user interface 102 may also display other shooting controls, which are not listed here.

When displaying the user interface 102 shown in FIG. 1B , the terminal 100 can detect a user operation on any selection box. In response to the above operation, the terminal 100 may determine that the object corresponding to the above selection box is the protagonist. For example, referring to the user interface 103 shown in FIG. 1C , the terminal 100 may detect a user operation acting on the selection box 123 . In response to the above operation, the terminal 100 may determine that the character 3 corresponding to the selection box 123 is the protagonist of the shooting.

Subsequently, the terminal 100 may display a small window in the preview window 113 in a picture-in-picture format, and display a close-up image of the character 3 in the small window. The above close-up image refers to the image obtained by cropping the selected protagonist as the center based on the original image collected by the camera (the image displayed in the preview window).

FIG. 1D exemplarily shows the user interface 104 in which the terminal 100 displays a small window and displays a close-up image of the character 3 in the small window.

As shown in FIG. 1D , the preview window 113 of the user interface 104 may include a small window 141 . At this time, a close-up image of the character 3 can be displayed in the small window 141 . As the image displayed in the preview window 113 changes, the image displayed in the small window 141 will also change accordingly. Furthermore, the image displayed in the small window 141 is always centered on the person 3. In this way, the video composed of the image displayed in the small window 141 is a close-up video of the character 3 .

Optionally, the close-up image displayed in the small window 141 and the original image displayed in the preview window 113 may also come from different cameras. For example, the close-up image displayed in the small window 141 may be from an image collected by an ordinary camera, and the original image displayed in the preview window 113 may be from an image collected by a wide-angle camera. Regular cameras and wide-angle cameras can capture images at the same time. The images collected by the ordinary camera and the wide-angle camera correspond at the same time. In this way, the user can browse a larger range of landscape in the preview window 113, and at the same time, a more detailed protagonist image is displayed in the small window 141.

After character 3 is determined to be the protagonist of the shooting, the selection box 123 corresponding to character 3 can become as shown in the selection box 142 in Figure 1D. The user can determine the selected shooting protagonist by checking the box 142 . Not limited to the check box 142 shown in the user interface 104, the terminal 100 may also display other styles of icons to indicate that the character 3 is selected as the protagonist to show distinction.

Optionally, the small window 141 for displaying close-up images may also include a close control 143 and a transpose control 144. The close control 143 can be used to close the small window 141. The transpose control can be used to resize the small window 141.

In some examples, after closing the small window 141 according to the user operation acting on the closing control 143, the terminal 100 may cancel the previously determined protagonist (Character 3). Then, the terminal 100 may instruct the user to re-select the shooting protagonist among the recognized objects. At this time, the terminal 100 can display the small window 141 in the preview window 113 again based on the redetermined protagonist. At this time, a close-up image obtained by processing the original image with the center of the new protagonist is displayed in the small window 141 .

In some examples, the close control 143 may also be used to pause the recording of the close-up video after starting to record the video. At this time, the terminal 100 will not cancel the previously determined protagonist. After the recording is paused, the off control 143 can be replaced with an on control. After detecting the user operation for opening the control, the terminal 100 can continue to record a close-up video centered on the above-mentioned protagonist.

In other examples, after closing the small window 141, the terminal 100 only does not display the small window, that is, it does not display the close-up image of the previously determined protagonist (Character 3), but the terminal 100 still maintains the previously determined protagonist. At this time, the preview window 113 will not be partially blocked by the small window 141 that displays the close-up image of the protagonist. Users can better monitor the image content of the original video, resulting in higher quality original video. At this time, the user can cancel the selected protagonist character 3 by clicking on the check box 142, thereby re-selecting a new protagonist among the recognized objects.

Optionally, after determining the protagonist, the terminal 100 may first generate a small window (vertical window) with an aspect ratio of 9:16 for displaying a close-up image, refer to the small window 141 in FIG. 1D. The above aspect ratio is an exemplary example, and the aspect ratio of vertical windows includes but is not limited to 9:16. After detecting a user operation on the transpose control 144, the terminal 100 can change the original vertical window into a horizontal window (horizontal window) with an aspect ratio of 16:9. Of course, the terminal 100 can also generate a horizontal window by default, and then adjust the horizontal window to a vertical window according to user operations. This embodiment of the present application does not limit this. In this way, the user can adjust the video content of the close-up video through the transpose control 144 to meet his or her own personalized needs.

Optionally, the terminal 100 may permanently display a small window showing a close-up image at the lower left (or lower right, upper left, or upper right) of the screen. In some examples, the above-mentioned small window can also adjust its display position according to the position of the protagonist in the preview window to avoid blocking the protagonist in the preview window.

Furthermore, the terminal 100 can also adjust the position and size of the small window according to user operations. In some examples, the terminal 100 may also detect long press operations and drag operations on the small window 141, and in response to the above operations, the terminal 100 may The small window moves to the position where the user's drag operation last stopped. In other examples, the terminal 100 may also detect a double-click operation on the small window 141, and in response to the above operation, the terminal 100 may enlarge or reduce the small window 141. Not limited to the long press operation, drag operation and double-click operation introduced above, the terminal 100 can also control and adjust the position and size of the small window through gesture recognition and voice recognition. For example, the terminal 100 can recognize that the user has made a fist gesture through the image collected by the camera, and in response to the fist gesture, the terminal 100 can reduce the small window 141 . The terminal 100 can recognize that the user has made a hand-opening gesture through the image collected by the camera. In response to the hand-opening gesture, the terminal 100 can enlarge the small window 141 .

After determining the protagonist, the terminal 100 may detect a user operation to start shooting. After starting shooting, the terminal 100 may also detect a user operation to end shooting. In response to the above-mentioned operations of starting and ending shooting, the terminal 100 may save the sequence of image frames collected by the camera during the above-mentioned operations as a video.

Referring to the user interface 105 shown in FIG. 1E , the terminal 100 may detect a user operation on the shooting control 112 . The above-mentioned user operation on the shooting control 112 may be called a user operation for starting shooting. In response to the above user operation of starting shooting, the terminal 100 may write the original image corresponding to the preview window 113 and the close-up image corresponding to the small window 141 into a specific storage space.

On the one hand, the terminal 100 can write the original image collected by the camera (the uncropped image displayed in the preview window 113) into a specific storage space to generate an original video; on the other hand, the terminal 100 can also use the protagonist to The close-up image (the image displayed in the small window 141) in the center is written into a specific storage space, thereby generating a close-up video.

Referring to the user interface 106 shown in FIG. 1F , after detecting a user operation to start shooting, the terminal 100 may change the shooting control 112 to the shooting control 161 in the user interface 106 . Capture control 161 may be used to indicate that recording is currently in progress.

At some point after the shooting starts, the protagonist initially selected by the user may leave the viewing range of the camera of the terminal 100 (that is, the protagonist is not included in the preview window 113). Referring to the user interface 107 shown in FIG. 1G , the identifiable objects in the preview window 113 include character 1 and character 2, but do not include the protagonist selected by the user: character 3.

At this time, the terminal 100 may display and close the small window 141 displaying the close-up image of the protagonist. Referring to FIG. 1G , the preview window 113 does not include the small window 141 at this time. At the same time, the terminal 100 may display a prompt 162, such as "The protagonist is missing, please aim and shoot at the protagonist" to remind the user that the protagonist is missing and the close-up image of the protagonist cannot be determined.

In response to the above prompt 162, the user can adjust the camera position so that the protagonist is within the viewing range of the camera, so that the camera can re-capture an image including the protagonist. Referring to the user interface 108 shown in Figure 1H, at this time, character 3 (the protagonist) is re-detected in the preview window 113. Therefore, the terminal 100 can regenerate the small window 141 and display the current protagonist-centered image in the small window 141. close-up image.

In some embodiments, the terminal 100 may decide whether to close the small window 141 after several frames. For example, after the moment shown in the user interface 107 shown in FIG. 1G (no protagonist is detected), the terminal 100 can continue to detect N frames of images after this frame image. If none of the N frames of images include the protagonist, the terminal 100 Close the small window 141. After the protagonist disappears and before confirming to close the small window 141, the terminal 100 may determine the image content displayed in the small window 141 during the above period based on the cropping area of the last frame before the protagonist disappears.

After recording the video for a period of time, the user can end recording the video through the shooting control 161 . Referring to Figure 1I In the user interface 109, the terminal 100 can detect a user operation acting on the shooting control 161. The above user operation acting on the shooting control 161 can be called a user operation to end shooting.

In response to a user operation to end shooting, the terminal 100 may encapsulate the original image frame sequence written into a specific storage space into a video, that is, an original video. At the same time, the terminal 100 can also encapsulate the close-up image frame sequence written into a specific storage space into a video, that is, a close-up video.

After ending the shooting, the terminal 100 may display the user interface 110 shown in FIG. 1J.

As shown in the user interface 110, after ending the shooting, the terminal 100 can change the shooting control 161 to the shooting control 112 to indicate to the user that the video recording has ended. At the same time, the terminal 100 may display logos representing the original video and the close-up video in the review control 114 . Generally, the above-mentioned identification may be a thumbnail of the first frame of the above-mentioned original video, or a thumbnail of the first frame of the above-mentioned close-up video.

The user can browse the captured video through the review control 114 . Here, after completing one video shooting in protagonist mode, the terminal 100 can obtain two videos. One of the above two videos is the above original video and the other is the above close-up video. Referring to the user interface 110 shown in FIG. 1J , the terminal 100 may detect user operations on the lookback control 114 . In response to the above operation, the terminal 100 can display the above two videos for the user to browse.

FIG. 1K exemplarily shows the user interface 111 of the terminal 100 displaying the captured video.

User interface 111 may include window 191 . Window 191 can be used to play captured video. Optionally, the terminal 100 may first play the original video captured based on the preview window 113 in the protagonist mode in the window 191 . At the same time, the terminal 100 may display the prompt 192. Prompt 192 such as "Swipe left to browse the close-up video of the protagonist." Through the above prompts, users can perform a left swipe operation to obtain a close-up video.

As shown in user interface 111, the terminal can detect a left sliding operation. In response to the above left swipe operation, the terminal 100 may play a close-up video centered on the protagonist shot in the protagonist mode. Referring to the user interface 112 shown in FIG. 1L and the user interface 113 shown in FIG. 1M, at this time, the close-up video shot based on the small window 141 can be played in the window 191.

In some examples, the terminal 100 may encapsulate the close-up video centered on the protagonist as one close-up video. For example, referring to the user interface 107 shown in FIG. 1G , during the video shooting process, the initially selected protagonist may disappear within the viewing range of the terminal 100 , and after a period of time, the initially selected protagonist may reappear in the terminal 100 within the viewing range. At this point, there is an interruption in the close-up video centered on the protagonist. Preferably, the terminal 100 can also ignore the above interruption and encapsulate all the close-up images into one close-up video.

Specifically, FIG. 1N exemplarily shows a schematic diagram in which the terminal 100 encapsulates all close-up images into one close-up video.

As shown in Figure 1N, T1 can represent the moment when the video recording starts, T2 can represent the time when the video recording ends, T3 can represent the detection of the lost video of the protagonist (the user interface 107 shown in Figure 1G), and T4 can represent the re-detection of the protagonist. It is time for the protagonist (user interface 108 shown in Figure 1H). During T1-T2, the original images collected by the camera constitute the original video. During T1-T3, the close-up image centered on person 3 extracted based on the original image collected by the camera constitutes close-up video 1. During T3-T4, the close-up image centered on person 3 extracted based on the original image collected by the camera constitutes close-up video 2. After the shooting is completed, the terminal 100 can package the above close-up video 1 and close-up video 2 into one close-up video.

In other examples, the terminal 100 may also save multiple close-up videos. For example, the terminal 100 may encapsulate the close-up image of the protagonist before the interruption into a close-up video 1, and encapsulate the close-up image of the protagonist after the interruption into a close-up video 2. Then, the terminal 100 can save the above close-up videos respectively.

In some embodiments, the terminal 100 can also enable the protagonist mode through the method shown in FIGS. 1O-1P. As shown in FIG. 1O , in the video recording mode, the terminal 100 may display the protagonist mode control 1166 in the setting bar 116 . When a user operation on the protagonist mode control 1166 is detected, the terminal 100 may turn on the protagonist mode, see FIG. 1P.

In some embodiments, after detecting that the protagonist is missing, the terminal 100 may also determine a new protagonist and shoot a close-up video centered on the new protagonist.

In conjunction with the user interface 107 shown in FIG. 1G , when it is detected that all objects in the preview window 113 do not include the initially selected protagonist (Character 3), the terminal 100 may confirm that the loss of the protagonist is detected. At this time, the terminal 100 can close the small window 141 that displays the close-up image of the protagonist, and display a prompt 162 to instruct the user to adjust the camera position, thereby reacquiring a new image containing the protagonist.

In this embodiment of the present application, referring to the user interface 201 shown in FIG. 2A , the user can also select character 2 as the protagonist. For example, the terminal 100 may detect a user operation on the selection box 122 . In response to the above operation, the terminal 100 may determine a new protagonist: character 2.

Among them, during the process of switching the protagonist, the small window 141 can directly display the close-up image of the switched character 2, presenting a jumping display effect. Optionally, the small window 141 can also achieve a non-jumping protagonist switching display effect through a smoothing strategy. For example, after switching the protagonist to character 2, the terminal 100 can determine a set of smoothly moving image frames based on the path from character 3 to character 2 in the preview window 113, and then display the above image frames in the small window 141 to achieve non-stop operation. Jumping protagonist switching display. For example, the terminal 100 can also use a fixed transition effect to connect the close-up images of the protagonist before and after switching. The above-mentioned fixed transition effects include overlay, vortex, translation, etc. commonly used in video editing. The embodiments of the present application do not limit this.

Likewise, in conjunction with the focus control 126 introduced in Figure 1B. When the focus control 126 is used to switch the current focus (or camera), the switching effect of the close-up image displayed in the small window 141 may also refer to the switching effect when switching characters mentioned above. For example, when it is detected that the current focal length is changed from 1 times the focal length (1×) to 2 times the focal length (2×), optionally, the terminal 100 can directly display a close-up based on the 2× original image in the small window 141 image; optionally, the terminal 100 can also determine a set of image frames with a gradient transition effect based on the 1× and 2× original images, thereby achieving a non-jumping focus switching display effect in the small window 141; optionally, the terminal 100 can also use fixed transition effects such as superposition, vortex, and translation to achieve a non-jumping focus switching display effect in the small window 141, which will not be described again here.

Referring to the user interface 202 shown in FIG. 2B , after determining the new protagonist character 2 , the terminal 100 can regenerate the small window 141 and display a close-up image centered on the new protagonist in the small window 141 . Then, the terminal 100 can continuously track the person 2 and display a close-up image of the person 2 in the small window 141 in real time.

At this time, after the shooting is completed, the close-up video generated by the terminal 100 is a close-up video including a plurality of protagonists.

Referring to the schematic diagram shown in Figure 2C, time T3 is the time when the loss of the initially selected protagonist (Character 3) is detected. Time T5 is the time when it is detected that the user selects a new protagonist (Character 3). At this time, within T1-T3, close-up video 1 is a close-up video centered on the initially selected protagonist (Character 3); within T5-T2, close-up video 2 is centered on the re-selected protagonist (Character 2) Close-up video.

In this embodiment of the present application, the terminal 100 can combine the above close-up video 1 and the close-up video 2 and package them into one video. frequency. Then, the terminal 100 can play the combined close-up video for the user to browse. Referring to the user interfaces 204 and 205 shown in Figures 2D to 2E, the above merged close-up video can be played in the window 191. In the user interface 204, the protagonist of the previous close-up video (that is, close-up video 1) is character 3. In the user interface 205, the protagonist of the latter close-up video (that is, close-up video 2) is character 2.

In some embodiments, the terminal 100 may also first detect a user operation to start shooting on the shooting control 112 and start recording the video. During the process of recording a video, the terminal 100 can detect objects contained in the image in real time and display a selection box corresponding to each object. After detecting the user operation of clicking a certain selection box, the terminal 100 can determine that the object corresponding to the selection box is the protagonist, and display a small window showing a close-up image of the protagonist. At the same time, the terminal 100 can also record the close-up in the small window. image. In the above method, the video length of the close-up video must be smaller than the original video.

After finishing shooting, users can also browse local videos at any time through the gallery application. The above-mentioned local videos include original videos and close-up videos captured and saved during the aforementioned process.

FIG. 3A exemplarily shows a user interface 301 of the terminal 100 displaying locally saved videos and/or pictures.

As shown in Figure 3A, user interface 301 may display multiple thumbnail icons. A thumbnail icon corresponds to a video or picture obtained by a shooting operation. For example, the plurality of thumbnail icons may include icon 213 . The icon 213 may correspond to the video generated by the aforementioned shooting operation shown in FIGS. 1E-1I.

The terminal 100 may detect user operations on the icon 213 . In response to the above operations, the terminal 100 may display the videos captured by the aforementioned shooting operations shown in FIGS. 1E-1I: original videos and close-up videos, refer to FIG. 3B.

As shown in FIG. 3B , user interface 302 shown in FIG. 3B may include window 221 . Window 221 can be used to display captured video: original video and close-up video. At this time, the window 221 can display the video 222 and the video 223. Among them, video 222 is the original video shot in the protagonist mode. Video 223 is a close-up video shot in protagonist mode.

In some examples, when the terminal 100 displays the user interface 302, the video 222 and the video 223 can be played simultaneously. This way, users can browse the original video and the close-up video at the same time. In some examples, the terminal 100 may also play the video 222 first and then the video 223 to facilitate user browsing.

Based on the user interface 302, the terminal 100 may detect a user operation, such as a click operation, acting on the video 222 or the video 223. Taking video 222 as an example, after detecting a click operation on video 222, the terminal 100 may display the user interface 111 shown in FIG. 1K to further display the original video. Correspondingly, after detecting a click operation on video 223, the terminal 100 may display the user interface 112 shown in FIG. 1L to further display the close-up video.

Optionally, based on the user interface 301 shown in Figure 3A, after detecting the user operation on the icon 213, the terminal 100 can also directly display the user interface 111 shown in Figure 1K to display the original video. Then, after detecting the left swipe operation, the terminal 100 may display the user interface 112 shown in FIG. 1L to display the close-up video.

FIG. 3C exemplarily shows another user interface 303 of the terminal 100 displaying locally saved videos and/or pictures.

In the user interface 303, the terminal 100 may display two thumbnail icons, such as icon 231 and icon 232. These two thumbnail icons correspond to the original video and close-up video captured in protagonist mode. For example, icon 231 may correspond to the above-mentioned original video, and icon 232 may correspond to the above-mentioned close-up video.

After detecting the user operation on the icon 231, the terminal 100 may display the user interface 111 shown in FIG. 1K to display the original video. After detecting the user operation on the icon 232, the terminal 100 may display the user operation shown in FIG. 1L. User interface 112 displays a close-up video.

Similarly, after the original video is displayed, users can browse the close-up video by swiping left or right. After displaying the close-up video, users can browse the original video by swiping right or left.

Implementing the above video editing method, in a multi-object video shooting scene, the terminal 100 can automatically track the movement trajectory of the protagonist selected by the user in the image, and generate a close-up video that is always centered on the protagonist. Then, the terminal 100 can also save the close-up video and the original video at the same time for the user to browse and use, so as to meet the user's more diverse needs. The original video can retain all the image content captured by the camera during the recording process. Close-up videos can focus on displaying the video content of the protagonist selected by the user. Among them, during the process of recording the video, the terminal 100 can also change the protagonist in real time according to the user's operation, so as to meet the user's need to change the shooting protagonist and further enhance the user experience.

Not limited to generating and saving close-up videos centered on the protagonist during the video shooting process. The terminal 100 can also perform object recognition and protagonist tracking on the local video that has been captured. Based on the tracked protagonist in each frame, the terminal 100 can perform editing operations such as cropping, combining, and packaging on the above-mentioned local video, thereby obtaining a close-up video centered on the protagonist.

4A to 4F show a set of user interfaces for the terminal 100 to edit a local video to obtain a close-up video centered on the protagonist. First, FIG. 4A exemplarily shows a user interface 401 of the terminal 100 displaying locally saved videos and/or pictures.

The user interface 401 may display a plurality of thumbnail icons corresponding to locally saved videos and/or pictures, such as icon 411. The icon 411 corresponds to a local video stored on the terminal 100.

The terminal 100 may detect a user operation on the icon 411. In response to the above operation, the terminal 100 may display the above local video. Referring to user interface 402 shown in FIG. 4B , user interface 402 may include window 412 . Window 412 may be used to display locally stored videos and/or pictures. At this time, the terminal 100 can play the local video corresponding to the above icon 411 in the window 412.

User interface 402 also includes menu bar 413. The menu bar 413 includes one or more controls for setting pictures or videos, such as sharing controls, collection controls, editing controls, deleting controls, and so on. The menu bar 413 also includes controls 414 for displaying more setting items. When a user operation on the control 414 is detected, the terminal 100 may display more setting items.

Referring to FIG. 4C , after detecting a user operation on control 414 , the terminal 100 may display the menu bar 413 . The menu bar 413 may include more setting items, such as "detailed information", "category tags" and other setting items. "Details" can be used to display the shooting information of the currently displayed picture or video, such as shooting time, shooting location, camera parameters, etc. "Category Tag" can be used to set the tag of the currently displayed image or video, so that users can quickly obtain the image or video through the tag.

In this embodiment of the present application, the menu bar 413 may also include a setting item “Extract Protagonist”. Extract Protagonist can be used to generate a close-up video centered on a selected protagonist. As shown in FIG. 4C , the terminal 100 may detect a user operation on the “extract protagonist” setting item. In response to the above operation, the terminal 100 may display the user interface 404 shown in FIG. 4D. User interface 404 may be used to determine the protagonist, generate and save a close-up video centered on the protagonist.

As shown in Figure 4D, user interface 404 may include window 420. The window 420 can play the local video, that is, display the image frame sequence of the local video in sequence. User interface 404 also includes progress bar 424. The progress bar 424 can be used to indicate the playback progress; when the video is paused or played, the progress bar 424 can also be used to switch the currently displayed image frame, that is, the user You can change the playback progress by manually dragging the progress bar to switch the currently displayed image frame.

Optionally, when the video is played or paused, the window 420 may also display selection boxes corresponding to each object in the currently displayed image frame, such as selection boxes 421, 422, and 423. Among them, the selection box 421 corresponds to the person 1 in the current image frame, the selection box 422 corresponds to the person 2 in the current image frame, and the selection box 423 corresponds to the person 3 in the current image frame. Optionally, during video playback, the window 420 only displays selection boxes corresponding to each object in the currently displayed image frame when a preset operation on the window 420 is detected. For example, the above-mentioned preset operation is an operation for pausing video playback; for example, the above-mentioned preset operation is an operation for dragging the progress bar to switch image frames; for example, the above-mentioned preset operation is a touch operation for the currently playing image frame , double-click operation or long-press operation, etc.

For example, referring to FIG. 4D , when displaying the first frame image of the above-mentioned local video, the terminal 100 may detect the user's operation of selecting character 3 as the protagonist, such as the operation of clicking the selection box 423 corresponding to character 3. In response to the above operation, the terminal 100 may determine that the character 3 is the protagonist, and then the terminal 100 may sequentially determine the position of the character 3 in subsequent image frames, and determine the size of the close-up image centered on the character 3. By combining the close-up images centered on the character 3, the terminal 100 can obtain the close-up video of the character 3.

The terminal 100 may also detect the user operation of determining the protagonist when the window 420 displays any image frame after the first frame image of the local video. For example, referring to the user interface 405 shown in FIG. 4E, when the i-th frame is played, the terminal 100 may detect the user's operation of selecting character 3 (or other objects) as the protagonist. Optionally, after the user selects the protagonist, the window 420 may also automatically display selection boxes corresponding to each object in each currently displayed image frame. In this way, it is convenient for the user to switch the protagonist at any time through the selection box corresponding to the subsequent image frame. Optionally, after the user selects the protagonist and detects a preset operation on the window 420, the window 420 displays a selection box corresponding to each object in the currently displayed image frame. In this way, the selection box is displayed based on the preset operation only when the user intends to switch the protagonist, which can save the energy consumption of object recognition.

User interface 404 may also include controls 425 . Control 425 may be used to save the currently generated close-up video. For example, after detecting the user operation on the control 425, the terminal 100 may save the close-up video of the above-mentioned character 3 to the local storage space. After completing the saving operation, the terminal 100 may display the user interface 407 shown in FIG. 4G to display the above-mentioned saved close-up video. At this time, users can browse the above close-up video at any time.

Of course, when editing a close-up video of a local video, the terminal 100 can also support the user to switch the protagonist to obtain a close-up video including multiple objects.

For example, in the user interface 404 (first frame image) shown in FIG. 4D , in response to the user's operation of clicking the selection box 423 , the terminal 100 may determine that the current protagonist is character 3. Then, referring to the user interface 408 shown in FIG. 4H , when the terminal 100 displays an image including at least one object in any frame after the first frame of the image (for example, the Nth frame), the terminal 100 may also display the selection corresponding to the at least one object. frame, for example, the selection box 422 corresponding to character 2; the terminal 100 can detect the user's operation of clicking the selection box 422 (corresponding to character 2), and in response to the above operation, the terminal 100 can switch the protagonist to character 2. At this time, the protagonist from the M-th frame to the N-1th frame is character 3, and the protagonist from the N-th frame to the end of the video is character 2.

It should be noted that if the user starts to select the protagonist as character 3 when playing the M-th frame image of the local video (for example, the first frame image shown in Figure 4D), and when playing the N-th frame image of the local video (for example, the first frame image shown in Figure 4D (image shown in 4H), switch the protagonist from character 3 to character 2, and then save the close-up video, then the Mth frame image in the local video will be the N-1th The protagonist of the frame image is character 3, and the protagonist from the Nth frame image to the end of the video is character 2. Optionally, the first half of the close-up video saved by the terminal 100 is a close-up video centered on character 3, which is generated based on the image of character 3 in the M-th frame image to the N-1-th frame image of the local video; the above The second half of the close-up video is a close-up video centered on character 2, which is generated based on images including character 2 from the Mth frame to the last frame of the local video. Optionally, the terminal 100 can also save two close-up videos respectively, that is, the close-up video centered on person 3 and the close-up video centered on person 2.

In some embodiments, the user interface for editing protagonist and close-up videos shown in Figure 4D may also be as shown in Figure 5A.

As shown in FIG. 5A , the terminal 100 may first traverse the currently displayed local video and determine all objects included in the video. At this time, the terminal 100 can display all the above objects, such as Person 1, Person 2, and Person 3 in the user interface 501. Then, the terminal 100 can detect the user operation on any of the above characters, determine that the selected character is the protagonist, and then obtain a close-up image of the protagonist based on the image of the protagonist in the local video; and then combine the close-up images of the above protagonists , for close-up videos centered on the aforementioned protagonists.

Of course, in the user interface 501 shown in FIG. 5A , the user can also set multiple protagonists to obtain a close-up video including multiple protagonists, or a close-up video corresponding to multiple protagonists.

Optionally, the user interface 501 may also include a split control 511. The split control 511 can split the local video displayed in the window 420 into multiple video segments. For example, referring to the user interface 502-1 shown in FIG. 5B-1, the terminal 100 may detect a user operation acting on the split control 511. In response to the above user operation, the terminal 100 may display the user interface 502-2 shown in FIG. 5B-2. At this time, the user can divide the local video into one or more video segments by performing a dividing operation on the progress bar 424 .

For example, as shown in the user interface 502-2, the terminal 100 may detect the user's operation of clicking the progress bar 424. In response to the above user operation, the terminal 100 may display the user interface 502-3 shown in FIG. 5B-3. At this time, the terminal 100 may display the dividing frame 512 on the progress bar 424. Then, the user can divide the local video into two video segments through the above-mentioned dividing box 512.

Referring to the user interface 502-4 shown in Figure 5B-4, the terminal 100 can divide the original local video into two segments. At this time, 0:00-2:30 is a video (video segment 1); 2:30-4:00 is a video (video segment 2). The currently selected video segment can be represented in black. Furthermore, when a video segment is selected, the user can further divide the video segment into two video segments through the split control 511. In this way, the terminal 100 can divide the original video into multiple video segments.

Taking user interface 502-4 as an example, in the scenario where video segment 1 is selected, terminal 100 can determine all objects included in video segment 1, and then display them, such as characters 1, 2, and 3. The user can select any object from all displayed objects as the protagonist. For example, the terminal 100 may determine that the character 3 is the protagonist of the video segment 1 based on the detected user operation acting on the character 3 .

Then, the user can change the selected video segment and determine the protagonist in the changed video segment. For example, referring to FIG. 5C , the terminal 100 may detect the user's operation of clicking on video segment 2. In response to the above operation, the terminal 100 may display the user interface 504 shown in FIG. 5D. At this time, in the user interface 504, the terminal 100 can display all objects included in the video segment 2, such as character 1 and character 2 (the video segment 2 does not include character 3). At this time, the user can select character 2 as the protagonist of video segment 2.

Referring to FIG. 5E , terminal 100 may detect operations on control 425 . In response to the above operations, the terminal 100 may Save the close-up video based on the above local video. At this time, the protagonist in the local video from 0:00 to 2:30 is character 3, and the protagonist from 2:30 to 4:00 is character 2. The first half of the close-up video is a close-up video centered on character 3. It is generated based on the image of character 3 from 0:00-2:30 of the local video; the second half of the above close-up video is a close-up video centered on character 2 and is based on 2:30-4 of the local video: 00 includes the image of character 2. Similarly, the terminal 100 can also store two close-up videos respectively, that is, the close-up video centered on character 3 and the close-up video centered on character 2.

By implementing the video editing method introduced in the above embodiment, the terminal 100 can perform object recognition and protagonist tracking on the local video that has been captured, and then the terminal 100 can generate and save a close-up video centered on the protagonist. In this way, for any video stored on the terminal 100, the user can use the above method anytime and anywhere to obtain a close-up video of any object in the video, thereby meeting the user's personalized editing needs.

FIG. 6 exemplarily shows a flow chart for the terminal 100 to generate a close-up video of the protagonist during the shooting process.

S601: A user operation acting on the first control is detected.

Referring to the user interface 102 shown in FIG. 1B , implementing the video editing method corresponding to the protagonist mode requires real-time identification and marking of objects (such as people, animals, plants, etc.) in images collected by the camera. This requires occupying a large amount of computing resources of the terminal 100. Therefore, in the embodiment of the present application, the protagonist mode is turned off by default when the camera is turned on.

The terminal 100 can provide the user with a control for turning on or off the protagonist mode, which is recorded as the first control. After detecting a user operation on the first control, the terminal 100 can turn on the protagonist mode and execute the shooting algorithm corresponding to the protagonist mode, such as identifying objects in the image, protagonist tracking, and so on. For example, in the user interface 102 shown in FIG. 1B , the protagonist mode option in the mode bar 111 may be called a first control. After detecting a user operation on the protagonist mode option, the terminal 100 may provide the user with the shooting service shown in FIGS. 1B-1I.

In this way, users can determine whether to turn on the protagonist mode according to their own needs, thereby avoiding occupying the computing resources of the terminal 100, reducing the computing efficiency of the terminal 100, and affecting the user experience.

S602: Perform object detection on the i-th frame image collected by the camera, and determine the objects included in the i-th frame image.

Referring to the user interfaces shown in FIGS. 1B to 1D , in the protagonist mode, the terminal 100 needs to determine the protagonist according to the user's selection operation. At this time, the terminal 100 needs to first recognize the objects included in the images collected by the camera, and then mark the recognized objects. In this way, the user can select any object as the protagonist among the above-mentioned recognized objects. Correspondingly, the terminal 100 can also determine the protagonist based on user operations.

FIG. 7A exemplarily shows a flowchart for the terminal 100 to recognize objects in the image after turning on the protagonist mode.

S701: Perform face recognition and human body recognition on the i-th frame image collected by the camera, and determine the face image and human body image in the i-th frame.

The terminal 100 may include a face recognition algorithm and a human body recognition algorithm as thresholds. Face recognition algorithms can be used to identify faces in images. Human body recognition algorithms can be used to identify human body images in images, including faces, bodies, and limbs.

Taking the i-th frame image collected by the camera as an example, the terminal 100 can execute the face recognition algorithm and the human body recognition algorithm respectively, and then determine the face image and human body image in the i-th frame image. Among them, the i-th frame image is any frame image collected by the camera after turning on the protagonist mode.

As shown in the i-th frame image shown in Figure 7B, through the face recognition algorithm, the terminal 100 can determine that the frame image includes faces face1, face2, and face3; through the human body recognition algorithm, the terminal 100 can determine that the frame image includes human bodies body1, body2 , body3.

S702: Match the recognized face image and human body image to determine the object included in the i-th frame image.

After determining the face image and body image in the i-th frame image, the terminal 100 can calculate the intersection over union (IoU) of each face image and body image, which is recorded as IoU _face&body . Then, the terminal 100 can use the above-mentioned IoU _face&body to match the recognized face image and the human body image, and determine the object included in the i-th frame image.

According to experience, the intersection between the face of any one of the two non-overlapping people in the image and the other person's human body is 0, while the intersection with the own human body is basically close to the own face. Therefore, the smaller the IoU _face&body is, the closer it is to 0, and the face and human _{body corresponding to the IoU face&} body do not match, that is, they cannot be regarded as the face and human body of the same person.

Therefore, the first threshold M1 may be preset in the terminal 100 . When IoU _face&body ≥M1, the face corresponding to the IoU _face&body matches the human body, otherwise, it does not match. A matching set of face images and body images identifies an object. In this way, the terminal 100 can determine the M objects included in the i-th frame image based on the recognized face image and human body image.

Specifically, taking the human faces face1, face2, and face3 and the human bodies body1, body2, and body3 shown in FIG. 7B as an example, the terminal 100 can calculate the IoU of face1, face2, face3, and body1, body2, and body3 respectively. Taking face1 as an example, the IoU values of face1, body2, and body3 are all 0, and the IoU values of face1 and body1 are not 0 and satisfy M1. At this time, the terminal 100 can determine that face1 and body1 can constitute an object (i.e. Character 1). Similarly, the terminal 100 can determine that face2 and body2 can constitute an object (ie, character 2), and face3 and body3 can constitute an object (ie, character 3).

In order to improve calculation efficiency, after determining an object, when subsequently calculating the IoU of each face image and human body image, the terminal 100 may no longer calculate the IoU of the face image and human body image that have constituted an object. For example, the terminal 100 may first calculate the IoU between face1 and all bodies (body1, body2, and body3). At this time, the terminal 100 can determine body1 that matches face1. Therefore, the terminal 100 can determine that face1 and body1 constitute an object. Then, the terminal 100 can calculate the IoU of face2 and all remaining bodies (body2, body3). At this time, the terminal 100 can no longer calculate the IoU of face2 and body1 to reduce redundant calculations and improve calculation efficiency.

Optionally, in S701, the terminal 100 may also directly use the human body detection algorithm to identify the object in the i-th frame image. At this time, the terminal 100 does not need to match the face image and the human body image.

The above method can better identify the objects included in the i-th frame image in single-object scenes and in scenes with multiple objects and non-overlapping between the objects. However, when there are many people photographed and there are overlapping people, the accuracy of the above method in identifying the objects included in the i-th frame image is low, and it is easy to cause recognition misalignment or not recognize the overlapping people at all.

Therefore, in a multi-object scene, especially in a multi-object scene with overlapping people, the object recognition method shown in S701-S702 can more stably and correctly identify multiple objects included in the image frame.

It can be understood that in a scenario where the terminal 100 supports the recognition of objects such as animals and plants, the above-mentioned object recognition algorithm also includes a recognition algorithm for specific animals and a recognition algorithm for specific plants. In this way, the terminal 100 can identify whether the i-th frame image includes objects of types such as animals, plants, etc. Furthermore, the terminal 100 can set the above-mentioned animals, plants and other objects as protagonists. The object recognition algorithm's identification of objects that can support recognition depends on the developer's presets.

S603: Display the i-th frame image and markers corresponding to each object in the i-th frame image.

After determining that M objects are included in the i-th frame image, the terminal 100 may create tags respectively corresponding to the above-mentioned M objects. When displaying the i-th frame image, the terminal 100 may display the above mark at the same time. This mark can be used to prompt the user terminal 100 to recognize an object that can be determined as a protagonist. Further, the mark can be used by the user to instruct the terminal 100 to determine which object is the protagonist.

Combined with the i-th frame image shown in FIG. 7B, after determining the three objects (person 1, person 2, and person 3) included in the i-th frame image, the terminal 100 can determine three markers corresponding to the above three objects. Referring to the user interface 102 shown in FIG. 1B , the above mark may be a selection box in the preview window 113 . When the terminal 100 displays the i-th frame image, the terminal 100 may display the selection boxes 121, 122, and 123. Among them, the selection boxes 121, 122, and 123 are respectively used to mark Person 1, Person 2, and Person 3 in the image.

In this way, the user can not only browse the images collected by the camera in the preview window 113, but also obtain the objects recognized by the terminal 100, that is, the protagonists that support setting. Furthermore, the user can also click on any selection box (for example, selection box 123) to determine that the object (character 3) corresponding to the selection box is the protagonist. After detecting the user operation of clicking any selection box, the terminal 100 may set the object corresponding to the clicked selection box as the protagonist. Subsequently, the terminal 100 can locate the above-mentioned protagonist in the image sequence collected by the camera, thereby realizing protagonist tracking and generating a close-up video of the protagonist.

It can be understood that in a scenario where the terminal 100 supports the recognition of objects such as animals and plants, the terminal 100 can display a selection box on the image of the above-mentioned animal or plant accordingly. Users can also choose the above-mentioned animals and plants as protagonists.

Specifically, the display position of the selection box can be determined based on the face image and the human body image. FIG. 7C exemplarily shows a schematic diagram in which the terminal 100 determines the display position of the selection box. As shown in FIG. 7C , after recognizing the face image and the human body image, the terminal 100 can determine the midpoints of the face image and the human body image: the midpoint P1 of the face image and the midpoint P2 of the human body image. Based on the above P1 and P2, the terminal 100 can determine the midpoint P3 of the object (ie, the person 3) corresponding to the above face image and the human body image. The midpoint of the selection frame 123 is the above-mentioned P3.

S604: Determine whether a user operation of selecting the first object in the i-th frame image is detected. If yes, determine that the first object is the protagonist, and set the frame index number FrameID of the i-th frame image to 1; if not, obtain the next frame image of the i-th frame image (i=i+1), and repeat S602.

After the terminal 100 completes the operation shown in S603, the user can see the i-th frame image of the camera on the screen of the terminal 100, as well as the marks (selection boxes) corresponding to each object in the i-th frame image. Refer to Figure 1B The user interface 102 shown.

After displaying the i-th image frame carrying the mark, the terminal 100 may detect a user operation acting on any mark. In response to the above operation, the terminal 100 may determine that the object corresponding to the above mark is the protagonist, and set the frame index number FrameID of the i-th frame image to 1.

For example, referring to the user interface 103 shown in FIG. 1C , the terminal 100 may detect a user operation acting on the selection box 123 . In response to the above operation, the terminal 100 may determine that the object corresponding to the selection box 123 is the protagonist, that is, the character 3 is the protagonist, and set the frame index number FrameID of the frame image to 1, that is, FrameID=1. The above-mentioned character 3 is the first object, and the user operation performed on the selection box 123 is the operation of selecting the first object. FrameID can be used to reflect which frame image the frame image is to determine the protagonist.

The time for the terminal 100 to display the i-th frame image is short. During the time it takes to display the i-th frame image, the terminal 100 varies It will be able to detect the user's operation of selecting an object as the protagonist. At the same time, after displaying the i-th frame image, the terminal 100 needs to continue to display the image frames after the i-th frame collected by the camera. Therefore, if the terminal 100 does not detect a user operation on any selection box during the above display period, the terminal 100 can perform the operation of i=i+1, obtain the next frame image of the i-th frame image, and Repeat S602. In this way, the terminal 100 can identify the objects captured by the camera in real time and display the corresponding marks so that the user can select a protagonist at any time.

S604: Determine the close-up image centered on the protagonist.

The above close-up image refers to an image in which the main image is cropped with the selected protagonist as the center based on the original image collected by the camera (the image displayed in the preview window), and the resulting image content is the protagonist.

After determining that the first object in the i-th frame image is the protagonist, the terminal 100 may determine a close-up image centered on the protagonist corresponding to the i-th frame image based on the i-th frame image. For example, in the user interface 403, after determining that the character 3 is the protagonist, the terminal 100 can crop the image currently displayed in the preview window 113 with the character 3 as the center to obtain a close-up image of the character 3 as the image content.

FIG. 8A exemplarily shows a flowchart in which the terminal 100 determines a close-up image centered on a protagonist.

S801: Determine the zoom ratio ZoomRatio of the i-th image frame based on the human body image of the protagonist in the i-th frame image.

If the selected protagonist is farther away from the camera, the protagonist image occupies a smaller image area in the entire original image. At this time, the size of the close-up image centered on the protagonist becomes smaller. On the contrary, the larger the image area occupied by the protagonist image in the entire original image, the larger the size of the close-up image with the protagonist as the center.

Specifically, referring to the i-th frame image shown in FIG. 8B, if the protagonist is character 1, the close-up image of character 1 expected to be displayed in the small window should be an image surrounded by a dotted frame 61. At this time, the size of the dotted frame 61 is the size of the close-up image of the protagonist in the i-th frame image. If the protagonist is character 3, the close-up image of character 3 expected to be displayed in the small window should be an image surrounded by a dotted frame 62. At this time, the size of the dotted frame 62 is the size of the close-up image of the protagonist in the i-th frame image. It can be seen from this that in order to ensure the integrity of the protagonist, the terminal 100 needs to determine the size of the close-up image according to the size of the protagonist in the original image.

The zoom ratio ZoomRatio can be used to reflect the size of the main character in the original image. After determining the ZoomRatio, the terminal 100 may determine the size of the close-up image of the protagonist in the current frame.

Specifically, the calculation process for terminal 100 to determine ZoomRatio is as follows:

First, in the object recognition step shown in S602, the terminal 100 can use a preset human body recognition algorithm to recognize human body images in the image, such as body1, body2, body3, etc. After determining that character 3 is the protagonist, the terminal 100 may determine the ZoomRatio using the size of the human body image (body3) of character 3.

Among them, the calculation formula (Q1) for determining ZoomRatio using human body images is as follows:

Among them, maxBboxSize refers to the size of the largest recognized human body image; detectBboxSize refers to the size of the protagonist's human body image; minZoomRatio is the minimum value of the preset ZoomRatio; maxZoomRatio is the maximum value of the preset ZoomRatio.

By inputting maxBboxSize[i] and detectBboxSize[i] in the i-th frame image into Q1, the terminal 100 can determine the ZoomRatio[i] of the i-th frame image.

S802: Determine the size of the close-up image of the protagonist corresponding to the i-th frame image according to ZoomRatio[i]: CropRagionWidth, CropRagionHeight.

CropRagionWidth is used to represent the width of the close-up image, and CropRagionHeight is used to represent the height of the close-up image. CropRagionWidth and CropRagionHeight can be determined based on the ZoomRatio introduced above. Specifically, the calculation formulas (Q2, Q3) of CropRagionWidth and CropRagionHeight are as follows:

Among them, WinWidth is used to represent the width of the small window; WinHeight is used to represent the height of the small window. The CropRagionWidth and CropRagionHeight obtained based on WinWidth, WinHeight and ZoomRatio can just correspond to the width and height of the small window, thus avoiding the problem of image deformation when displaying close-up images in the small window. Preferably, when the small window is a vertical window, the value of WinWidth may be 1080p (pixel), and the value of WinHeight may be 1920p. When the small window is a horizontal window, the value of WinWidth can be 1920p and the value of WinHeight can be 1080p.

S803: Crop the i-th frame image according to CropRagionWidth, CropRagionHeight and the object midpoint, and determine the protagonist's close-up image corresponding to the i-th frame image.

After determining CropRagionWidth and CropRagionHeight, combined with the known character midpoint (P3) of the protagonist, the terminal 100 can crop the original image to obtain a close-up image centered on the protagonist. Referring to FIG. 8B , the image in the area formed by P3 as the center, the width and height being CropRagionWidth and CropRagionHeight respectively, is a close-up image of the protagonist (Character 3).

S606: Display the above close-up image in a small window to generate a frame of the close-up video.

After detecting a user operation that determines the first object in the i-th frame image as the protagonist, the terminal 100 may generate a small window for displaying a close-up image. Preferably, the small window can be embedded in the preview window in a picture-in-picture form.

Referring to the user interface 104 shown in FIG. 1D , a small window 141 is embedded in the preview window 113 in the form of a picture-in-picture. Preferably, the above-mentioned small window may be a rectangle with an aspect ratio of 9:16 (vertical window) or 16:9 (horizontal window). Of course, in other embodiments, the preview window and the small window can also be arranged in other ways, and the small window can also present other sizes and shapes. For example, after detecting that the protagonist is determined, the terminal 100 may divide the preview window 113 into two windows arranged side by side on the left and right. One window is used to display the original image collected by the camera in real time; the other window is used to display the close-up image centered on the protagonist. The embodiments of this application do not limit the specific form used to display close-up images.

After the processing of S605, the terminal 100 may determine the close-up image centered on the protagonist corresponding to the i-th frame. At this time, the terminal 100 may display a close-up image centered on the protagonist in the above-mentioned small window.

In some examples, the width CropRagionWidth and height CropRagionHeight of the close-up image are respectively equal to the width WinWidth and height WinHeight of the small window used to display the close-up image, see Figure 8C. For example, CropRagionWidth=1080p, CropRagionHeight=1920p, and at the same time, WinWidth=1080p, WinHeight=1920p. At this time, the close-up images cropped according to 1080p and 1920p just match the small window, and the terminal 100 The above close-up image can be displayed directly in the small window.

However, in other examples, the CropRagionWidth, CropRagionHeight and WinWidth, WinHeight of the close-up image are not equal. At this time, the terminal 100 can adaptively adjust the close-up image to obtain a close-up image that matches the size of the small window, and then display the close-up image in the small window. The above-mentioned adaptive adjustment includes equal proportional expansion and equal proportional reduction. For example, referring to Figure 8D, CropRagionWidth=540p, CropRagionHeight=960p, and the small window used to display a close-up image has WinWidth=1080p, WinHeight=1920p. At this time, the terminal 100 can enlarge the close-up images of 540p and 960p in equal proportions to obtain close-up images of 1080p and 1920p. In this way, the terminal 100 can also display the above close-up image in a small window.

During the video recording process, the above-mentioned close-up image that has undergone adaptive adjustment processing and is sent to the small window for display is one frame of the close-up video.

S607: Obtain the jth frame image after FrameID=1, FrameID=1+j, and determine the protagonist in the jth frame image.

Referring to the introduction in S603, after detecting the user's operation on the i-th frame image to determine the first object as the protagonist, the terminal 100 can set the FrameID of the i-th frame image to 1 to indicate that the frame is the protagonist. The first frame of the image.

When displaying the close-up image of the protagonist in FrameID=1, the terminal 100 can also simultaneously obtain the image frames collected by the camera after FrameID=1. After receiving the image frame after FrameID=1, the terminal 100 can identify the object in the image frame after FrameID=1 and determine whether the image frame includes the protagonist.

Taking the j-th image frame after FrameID=1 (FrameID=1+j) as an example, the method for the terminal 100 to locate the protagonist in the image frame after FrameID=1 is specifically introduced below.

method one:

In some examples, after acquiring the j-th frame image, the terminal 100 may use a human body recognition algorithm to first identify objects included in the j-th frame image, refer to S602. Then, the terminal 100 can use the similarity algorithm to calculate the similarity between the above-mentioned objects and the protagonist in the j-1th frame image, and then determine the similarity distance between each of the above-mentioned objects and the protagonist in the j-1th frame image (similarity distance = 1- similarity). The smaller the similarity distance, the smaller the difference between the object and the protagonist, that is, the more similar the object is. Therefore, the object in the jth frame image with the smallest similarity distance and lower than the similarity distance threshold can be determined as the protagonist.

Of course, the terminal 100 can also directly use the similarity to determine the protagonist in the j-th frame image. At this time, the object with the highest similarity and higher than the similarity threshold in the j-th frame image can be determined as the protagonist.

However, it is very computationally expensive to calculate the similarity between each object in the j-th frame image and the protagonist in the j-1th frame image every time. Moreover, if the image contents of two or more objects overlap in the j-th frame image, the similarity between the above-mentioned objects and the protagonist in the j-1th frame image will be affected, thus affecting the accuracy of the protagonist recognition results. .

Method Two:

In view of the shortcomings of method one, the terminal 100 can also use method two to locate the protagonist in the j-th frame image. Figure 9 exemplarily shows a second flow chart for locating the protagonist in the j-th frame image.

First, the terminal 100 still needs to perform object recognition on the j-th frame image to determine the objects included in the j-th frame image. Then, the terminal 100 may determine whether the objects in the j-th image frame overlap, and then determine to use different protagonist positioning methods to determine the protagonist in the j-th image frame based on whether the objects in the j-th image frame overlap.

When the objects in the j-th image frame do not overlap, the terminal 100 can pass all objects in the j-th image frame The intersection-union ratio distance (IoU distance, recorded as [IoU]) with the protagonist in j-1 is used to determine the protagonist of the j-th frame image. On the contrary, when overlapping, the terminal 100 can determine the protagonist of the j-th frame image through the IoU distance and re-identification distance (ReID distance, denoted as [ReID]) of all objects in the j-th frame image from the protagonist in j-1 .

Referring to the introduction of S602, when determining the objects included in the j-th frame, the terminal 100 may use the human body ranges (ie, human body frames) of the multiple objects recognized by the human body detection algorithm. At this time, the terminal 100 can use whether the human body frames intersect to determine whether the objects in the j-th frame image overlap. As shown in FIG. 10A , if any two objects in the j-th frame image do not overlap (the human body frames of any two objects do not intersect), such as person 3 and person 4, then the terminal 100 determines that the objects in the j-th frame image do not overlap. As shown in FIG. 10B , if there are at least two overlapping objects in the j-th frame image, such as person 3 and person 4, then the objects in the j-th frame image overlap.

In the case of no overlap, the terminal 100 can determine the protagonist of the j-th frame image through the IoU distance of all objects in the j-th frame image to the protagonist in the j-1 th frame.

This is because during the video recording process, the time interval between the two frames before and after is very short. Taking the frame rate of 30fps as an example, the time interval between the two frames before and after is 1/30s. At this time, within the time period of two adjacent frames, it is more difficult for an object to have a large IoU distance in the two frames before and after the image. Therefore, in the case of non-overlapping, the terminal 100 may first determine the IoU distances of all objects in the j-th frame image from the protagonist in j-1, and determine the minimum IoU distance [IoU] _min in the j-th frame image.

Specifically, refer to Figure 10C, in which the dotted box 3 can represent the human body frame of the protagonist in the j-1th frame; the dotted box 1' can represent the human body frame of the character 1 in the jth frame; the dotted box 2' can represent the jth frame The human body frame of character 2 in the j-th frame; the dotted-line frame 3' can represent the human body frame of character 3 in the j-th frame; the dotted-line frame 4' can represent the human body frame of character 4 in the j-th frame.

Taking the dotted box 3 and the dotted box 1' as an example, using the area formed by the two dotted boxes, the terminal 100 can determine the intersection ratio of the dotted box 3 and the dotted box 1', which is recorded as IoU ₃₁ . Therefore, the terminal 100 can determine the IoU distance [IoU ₃₁ ] between character 1 in the j-th frame and the protagonist in the j-1th frame:

[IoU ₃₁ ]=1-IoU ₃₁ ;

Similarly, the terminal 100 can obtain the IoU distances between characters 2, 3, and 4 in the j-th frame and the protagonist in the j-1th frame: [IoU ₃₂ ], [IoU ₃₃ ], [IoU ₃₄ ]. Referring to FIG. 10C, at this time, the terminal 100 can determine that [IoU] _min in the j-th frame image is [IoU ₃₃ ]. The smaller the IoU distance, the smaller the position change between the two objects. Combined with the short time between two adjacent frames, it is difficult for the protagonist to have a large displacement in the two frames before and after. Therefore, the smaller the IoU distance, the more likely the object is to be the protagonist.

However, the object in the j-th frame image that has the smallest IoU distance from the protagonist in j-1 is not necessarily the protagonist. For example, referring to Figure 10D, the IoU distance between characters 1, 2, and 4 in the j-th frame image and the protagonist in j-1 is 1, and the IoU distance between character 3 in the j-th frame image and the protagonist in j-1 is 0.9. . At this time, the IoU distance between character 3 and the protagonist is the smallest. However, in fact, the IoU distance between character 3 and the protagonist is very large. Therefore, if the object with the smallest IoU distance (Character 3) is directly determined as the protagonist, misidentification of the protagonist will easily occur, which will lead to the failure of automatic tracking of the protagonist and affect the user experience.

Therefore, after determining [IoU] _min in the j-th frame image, the terminal 100 also needs to determine whether the above-mentioned [IoU] _min is less than the preset IoU distance threshold (denoted as D1).

If [IoU] _min <D1, the terminal 100 may determine that the object corresponding to the above [IoU] _min is the protagonist. For example, D1=0.2. Combined with the minimum [IoU] distance determined in Figure 10C: [IoU ₃₃ ]. When [IoU ₃₃ ]<0.2, the terminal 100 can determine that the character 3 in the j-th frame image and the protagonist in the j-1-th frame image are the same object, that is, determine that the character 3 in the j-th frame image is the protagonist.

If [IoU] _min <D1 does not hold, the terminal 100 can mark that the protagonist in the frame image is missing (the protagonist is not matched). Ran The rear terminal 100 may determine whether to terminate the protagonist tracking based on the currently accumulated number of image frames lost by the protagonist, which will not be discussed here.

IoU distance is an optional indicator to determine the similarity between each object in the next frame of video and the protagonist in the previous frame. Of course, the terminal 100 may also choose other indicators. For example, the terminal 100 can also directly use IoU to determine the degree of similarity between all objects in the j-th frame image and the protagonist in the j-1th frame. At this time, the object in the j-th frame image that has the largest IoU with the protagonist in j-1 and is greater than the IoU threshold can be confirmed as the protagonist.

In the case of overlap, the terminal 100 may determine the protagonist of the j-th frame image through the IoU distance and ReID distance of all objects in the j-th frame image from the protagonist in the j-1th frame.

If the objects in the j-th frame image overlap, then the distance between the overlapping objects is relatively close to each other. At this time, within two adjacent frames, it is easy for an object to move from its position in the previous frame to the position of another object. Therefore, in overlapping scenes, the non-protagonist object is very likely to move to the original protagonist's position in the next frame. At this time, the terminal 100 cannot determine the protagonist of the j-th frame image only through the IoU distance of all objects in the j-th frame image to the protagonist in the j-1th frame.

For example, an object that overlaps with the protagonist in the j-1th frame image may appear in the j-th frame image at the position of the original protagonist in the j-1th frame image. At this time, the IoU distance between the above-mentioned object and the protagonist is the closest, but the object is not the protagonist. This can easily lead to misidentification.

Therefore, in the case of overlap, in addition to using the IoU distance of each object in the two frames of images to determine the protagonist, the terminal 100 also needs to determine whether the object in each position is the protagonist originally determined by the user. At this time, the terminal 100 also needs to calculate the ReID distance between all objects in the j-th image and the protagonist in the j-1-th image. ReID distance is based on the use of neural networks and is a parameter that reflects the degree of similarity between image contents.

FIG. 11 exemplarily shows a schematic diagram of the terminal 100 determining the ReID distance between each object in the j-th image and the protagonist in the j-1-th image. As shown in Figure 11, using a convolutional neural network (CNN), the terminal 100 can determine the feature vector F0 of the protagonist in the j-1th frame image. Similarly, using CNN, the terminal 100 can determine the feature vectors F1 to F4 of each object (people 1 to 4) in the j-th frame image. Then, the terminal 100 can calculate the inner product of the feature vectors (F1˜F4) of each object in the j-th frame image and the feature vector F0 of the protagonist in the j-1th frame image: <F0, F1>, <F0, F2>, <F0,F3>, <F0,F4>.

Taking character 1 as an example, after determining the inner product <F0, F1> of the feature vector F1 and the feature vector F0 of the protagonist in the j-1th frame image, the terminal 100 can determine the ReID distance between the character 1 and the protagonist (denoted as [ReID ] ₃₁ )：

[ReID] ₃₁ =1-<F0,F1>;

Similarly, the terminal 100 can obtain the ReID distances between characters 2, 3, and 4 in the j-th frame and the protagonist in the j-1th frame: [ReID] ₃₂ , [ReID] ₃₃ , [ReID] ₃₄ . The smaller the ReID distance, the higher the similarity between the object and the protagonist. After determining the ReID distance between each object in the jth frame and the protagonist in the j-1th frame, the terminal 100 may determine the minimum ReID distance [ReID] _min in the jth frame image. Referring to FIG. 11 , at this time, the terminal 100 can determine that [ReID] _min in the j-th frame image is [ReID] ₃₃ .

Then, the terminal 100 can determine the IoU+ReID distance between each of the above objects and the protagonist, that is, the sum of the IoU distance and the ReID distance, which is recorded as [IoU+ReID]. The smaller [IoU+ReID] means the smaller the IoU between the object and the protagonist, and the smaller the ReID distance. Image-wise, the object is located close to the original protagonist and has similar image content. Therefore, the terminal 100 can determine the protagonist of the j-th frame image using [IoU+ReID]. Moreover, the smaller the [IoU+ReID], the more likely the object is to be the protagonist.

Similarly, in the case of overlap, the object with the smallest [IoU+ReID] in the j-th frame image and the protagonist in j-1 will not Must be the protagonist. Therefore, after determining [IoU+ReID] _min in the j-th frame image, the terminal 100 also needs to determine whether the above-mentioned [IoU+ReID] _min is less than the preset IoU+ReID distance threshold (denoted as D2). If [IoU+ReID] _min <D2, the terminal 100 may determine that the object corresponding to the above [IoU+ReID] _min is the protagonist. On the contrary, if [IoU+ReID] _min <D2 does not hold, the terminal 100 can mark that the main character in the frame image is missing.

In some embodiments, in order to improve calculation efficiency, the terminal 100 may also periodically execute the method of locating the protagonist in the j-th frame image shown in FIG. 9 , see FIG. 12A .

As shown in Figure 12A, after acquiring the jth frame image after FrameID=1, the terminal 100 can first determine whether the frame index number FrameID of the image frame is divisible by N (the embodiment of this application uses N=4 as an example for explanation). ).

When FrameID%4=0 is established, that is, FrameID is divisible by 4, the terminal 100 can determine the protagonist in the j-th frame image through the method introduced in Figure 9. On the contrary, when FrameID%4=0 does not hold, that is, FrameID is not divisible by 4, the terminal 100 can use the Kernel Correlation Filter algorithm (Kernel Correlation Filter, KCF) to determine the protagonist in the j-th frame image. The KCF algorithm is existing and will not be described in detail here.

In this way, the terminal 100 can avoid calculating the IoU distance and ReID distance every time, thereby saving computing resources and improving computing efficiency.

S608: Confirm whether the protagonist is matched. If so, perform the steps shown in S605; if not, determine whether the lost frames are less than the lost frame number threshold Y.

After determining the protagonist in the j-th frame image, the terminal 100 can perform two operations: First, perform the steps shown in S605: determine a close-up image centered on the protagonist based on the protagonist in the j-th frame image, and then The above close-up image is displayed in the window to generate a frame of the close-up video; the second is to obtain the next frame image of the j-th frame image (j=j+1), repeat the steps shown in S607, and then determine the protagonist in the next frame image , display a close-up image of the protagonist in the next frame of the image, and generate another frame of the close-up video.

For the method of determining the close-up image of the protagonist in the j-th frame image, please refer to the introduction of S605, which will not be described again here. In particular, in some examples, the terminal 100 may calculate the ZoomRatio every few frames, for example, calculate the ZoomRatio every 4 frames.

This is because within a period of 4 consecutive frames (taking 4 frames as an example), it is difficult for a certain object in the image to change significantly. At this time, the ZoomRatio corresponding to these 4 frames is almost the same. Therefore, after a ZoomRatio is determined in the kth frame, the k+1th frame, the k+2th frame, and the k+3th frame can continue to use the above ZoomRatio, thereby saving the calculation frequency of the ZoomRatio and saving computing resources.

In some examples, when the ZoomRatio changes significantly between two times, the terminal 100 can perform smoothing processing in the process of determining the close-up image to avoid image jumps.

If the protagonist is not matched in the j-th frame image (for example, [IoU] _min ≥ D1 of the j-th frame image, or [IoU+ReID] _min ≥ D2 of the j-th frame image), at this time, the terminal 100 can modify the missing frame Count: Increases the count of lost frames by 1. The number of lost frames refers to the number of image frames of the protagonist that the terminal 100 has not recognized continuously. Then, the terminal 100 may determine whether to end the protagonist tracking based on the number of lost frames.

Specifically, the terminal 100 may be set with a lost frame number threshold Y. If the number of currently recorded lost frames ≥ Y, the terminal 100 may determine that the objects captured by the camera no longer include the protagonist initially selected by the user. At this time, the terminal 100 can confirm that the protagonist tracking is completed. If the number of currently recorded lost frames is <Y, the terminal 100 can obtain the next frame image (the j+1th frame image), Determine whether the next frame includes the main character. It can be understood that when determining whether the next frame image includes the protagonist, the terminal 100 can perform the protagonist tracking calculation shown in Figure 9 on the next frame image and the last frame image previously matched to the protagonist, and determine that the protagonist Whether there is a protagonist in the next frame of the image. Here, the next frame image is the j-th frame image, and the last frame image matched to the protagonist is the j-1th frame image. When the initially selected protagonist cannot be recognized in subsequent image frames, the number of lost frames will continue to increase until the number of lost frames ≥ Y, and the protagonist tracking ends.

In some embodiments, after confirming the end of protagonist tracking, the terminal 100 may maintain the initially set protagonist and continue to locate the protagonist in subsequent images collected by the camera. After re-detecting the above-mentioned protagonist, the terminal 100 can continue to record the close-up video of the protagonist.

Combined with the user interface shown in FIGS. 1F to 1H , when it is detected that the initial set protagonist character 3 is not included in the consecutive multi-frame images, the terminal 100 can close the small window 141 and stop recording the close-up video of the character 3 . When the person 3 is detected again, the terminal 100 can re-display the small window 141 and restart recording the close-up video of the person 3.

In some embodiments, after confirming to end protagonist tracking, the terminal 100 may also instruct the user to select a new protagonist. After detecting the user operation of selecting the protagonist again, the terminal 100 can determine the new protagonist, locate the new protagonist in subsequent images, and simultaneously display and save a close-up video of the new protagonist.

Combined with the user interface shown in FIGS. 2A and 2B , after determining to end tracking character 3 and re-detecting the user operation of selecting character 2 as the protagonist, the terminal 100 can re-determine character 2 as the protagonist, generate a small window 141 and A close-up image of the character 2 is displayed in the small window 141 .

Combined with the user interface shown in FIGS. 2A and 2B , in some embodiments, the terminal 100 supports switching protagonists during shooting. At this time, the terminal 100 may also determine whether the j-th frame image corresponds to a user operation of switching the protagonist after acquiring the j-th frame image, so as to change the protagonist and display the close-up image in the small window 141.

Referring to Figure 12B, after the step shown in S607, the terminal 100 can determine whether a user operation of switching the protagonist is detected, for example, clicking the selection box 122 corresponding to the character 2 shown in Figures 2A-2B switches the protagonist character 3 to the character. 2 user operations. When detecting the above-mentioned user operation of switching the protagonist, the terminal 100 may determine that character 2 in the j-th frame image is the protagonist. Then, the terminal 100 can reset the FrameID of the jth frame image to 1 (S604), obtain the image frame after the image frame, and locate the person 2 in the subsequent image frame, thereby displaying the new image in the small window 141. A close-up image of the main character 2.

Next, FIG. 13 illustrates a flow chart for the terminal 100 to edit the captured local video, generate and save the close-up video.

Referring to the user interface 403 shown in FIG. 4C , at this time, the first control may be the "Extract Protagonist" option control in the menu bar 413 . The user operation of the user clicking on the above-mentioned "Extract Protagonist" can be called a user operation acting on the first control.

Then, the terminal 100 can obtain the image frame sequence of the local video and determine the objects included in each image frame. Referring to the user interface 404 shown in FIG. 4D, the terminal 100 can display each image frame in the local video. When displaying the above image frame, the terminal 100 will also display marks (selection boxes) corresponding to each object. Furthermore, the terminal 100 can determine the protagonist among the plurality of objects based on user operations. Then, the terminal 100 can traverse subsequent image frames at a time, determine the protagonist in the subsequent image frame, thereby obtain a subsequent close-up image centered on the protagonist, and generate a close-up video. This process is the same as the method of determining the protagonist in the real-time shooting process shown in Figures 1A-1I, and will not be described again here.

Differently, in the method of editing local video and generating close-up video shown in Figure 13, when it is determined that the j-th frame image does not match the protagonist, the terminal 100 may not record the number of missing frames. Here, the terminal 100 only needs to determine whether the local video has been traversed, that is, whether the j-th frame image is the last frame of the local video. If the video is not over, that is, the j-th frame image is not the last frame of the local video, the terminal 100 can continue to obtain the next frame image and locate the protagonist in the next frame image.

In addition, in the method of editing a local video and generating a close-up video shown in FIG. 13 , after determining the protagonist in a certain frame image and the close-up image of the protagonist, the terminal 100 may not display the close-up image. This is because, during the editing process of the local video, the terminal 100 does not need to play the local video, so the terminal 100 does not need to play the close-up video during the editing process. After the editing is completed, the terminal 100 can save the above close-up video. Users can then browse the above close-up video at any time.

Figure 14 is a schematic system structure diagram of the terminal 100 provided by the embodiment of the present application.

The layered architecture divides the system into several layers, and each layer has clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer, driver layer and hardware layer.

The application layer can include a series of application packages. In this embodiment of the present application, the application package may include a camera, a gallery, etc.

The application framework layer provides application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. In this embodiment of the present application, the application framework layer may include a camera access interface and a video editing interface. Among them, the camera access interface may include camera management and camera equipment. The camera access interface is used to provide an application programming interface and programming framework for camera applications. The video editing interface is used to provide an application programming interface and programming framework for editing pictures and/or videos for the gallery application. In the embodiments of this application, the application programming interface and programming framework for editing videos provided by the video editing interface are mainly used.

The hardware abstraction layer is the interface layer between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In this embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

Among them, the camera hardware abstraction layer can provide virtual hardware of camera device 1, camera device 2 or more camera devices. The camera algorithm library may include running code and data to implement the video editing method provided by the embodiment of the present application.

The driver layer is the layer between hardware and software. The driver layer includes drivers for various hardware. The driver layer may include camera device drivers, digital signal processor drivers, image processor drivers, etc.

Among them, the camera device driver is used to drive the sensor of the camera to collect images and drive the image signal processor to preprocess the images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

The following is a detailed description of the video editing method in the embodiment of the present application in conjunction with the above system structure:

1. While recording the video, crop the original video to generate a close-up video of the protagonist:

In response to the user's operation of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface of the application framework layer, starts the camera application, and then calls the camera device (camera device 1 and/or Other camera devices) send instructions to start the camera. The camera hardware abstraction layer sends this instruction to the camera device driver at the kernel layer. The camera device driver can start the corresponding camera sensor and collect image light signals through the sensor. A camera device in the camera hardware abstraction layer corresponds to a camera sensor in the hardware layer.

Then, the camera sensor can transmit the collected image optical signal to the image signal processor for pre-processing to obtain the image electrical signal (original image), and transmit the above-mentioned original image to the camera hardware abstraction layer through the camera device driver.

On the one hand, the camera hardware abstraction layer can send the above raw image to the display for display.

The camera hardware abstraction layer, on the other hand, sends raw images to the camera algorithm library. The camera algorithm library stores program codes that implement the video editing methods (processing processes such as object recognition, protagonist tracking, and cropping close-up images) provided by the embodiments of this application. Based on the digital signal processor and image processor, executing the above code, the camera algorithm library can also output the recognized objects in the image frame and determine the close-up image centered on the protagonist, thereby locating the protagonist in the original image and cropping the protagonist. Features centered close-up image.

The camera algorithm library can send determined close-up images to the camera hardware abstraction layer. The camera hardware abstraction layer can then send it to the display. This allows the camera app to display a close-up image centered on the selected protagonist alongside the original image.

While sending to display, the camera hardware abstraction layer can also write the original image sequence and the close-up image sequence to a specific storage space. In this way, the terminal 100 can realize the function of recording video, and save the original image stream collected by the camera in real time and the close-up image stream obtained based on the original image as local videos (original video and close-up video).

2. Crop the local video to generate a close-up video of the protagonist:

In response to the user's operation of extracting the protagonist from the local video, such as the click of "Extract Protagonist" as shown in Figure 4C, the camera application calls the image editing interface of the application framework layer, and then calls the camera algorithm library stored in the camera algorithm library to implement the present application. The embodiment provides the program code of the video editing method. Based on the digital signal processor and image processor, the camera algorithm library executes the above code to realize the function of locating the protagonist in the original image, cropping the close-up image centered on the protagonist, and then realizing the function of editing the original video to obtain the close-up video.

In this embodiment of the present application, the mark corresponding to each object in the image may also be called a selection box; the second video may also be called a close-up video.

In some embodiments, the first interface may be the user interface 102 shown in Figure 1B; the first image may be an image collected by the camera displayed in the preview window 113 in the user interface 102. For example, the first image is the image in Figure 1B The image displayed in the preview window 113, or the i-th frame image collected by the aforementioned camera; referring to FIG. 1C, the first mark may be the selection box 123 corresponding to the character 3, and the first operation may be an input operation acting on the selection box 123. One object may be character 3; the second image may be the image displayed in the preview window of the user interface 201 shown in FIG. 2A; the second mark may be the selection box 122 corresponding to character 2 in the above image; the second object may be the aforementioned character 2. The fifth operation may be an input operation acting on the selection box 122; the first sub-video may be the aforementioned close-up video centered on character 3, and the second sub-video may be the aforementioned close-up video centered on character 2. The second control may be the aforementioned control 161; the first window may be the aforementioned small window 141;

In some embodiments, the first video may also be called a local video. Referring to Figures 4A and 4B, the first video may be the local video corresponding to the aforementioned icon 411, and the thumbnail of the first video may be the icon 411 of the local video. The second operation may be an operation of clicking the icon 411; the first interface may be the user interface 404 shown in Figure 4D; the first image may be a frame of image in the local video, for example, the first image may be as shown in Figure 4D The image displayed in the window 420 in the user interface 404, or the i-th frame image of the aforementioned local video; referring to Figure 4D, the first mark may be the selection box 423 corresponding to the character 3, and the first operation may be to click the selection box 423 In operation, the first object may be character 3; the second image may be the image displayed in the window of user interface 404 shown in FIG. 4H, and the second mark may be character 2 in the above image. Corresponding to the selection box 422, the second object can be the aforementioned character 2, and the fifth operation can be an input operation of clicking the selection box 422; the first sub-video can be the aforementioned close-up video centered on the character 3, and the second sub-video can be It is the aforementioned close-up video centered on character 2.

The terminal 100 may include a processor 119, an external memory interface 120, an internal memory 129, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 149, an antenna 1, an antenna 2, Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone interface 170D, sensor module 180, button 190, motor 199, indicator 198, camera 193, display screen 194, and user Identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

It can be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown in the figures, or some components may be combined, or some components may be separated, or may be arranged differently. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

The processor 119 may include one or more processing units. For example, the processor 119 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors.

The controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.

The processor 119 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 119 is cache memory. This memory may hold instructions or data that have been recently used or recycled by the processor 119 . If the processor 119 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 119 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 119 may include one or more interfaces. Interfaces may include integrated circuit (inter-integrated circuit, I2C) interface, integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, pulse code modulation (pulse code modulation, PCM) interface, universal asynchronous receiver and transmitter (universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and /or universal serial bus (USB) interface, etc.

It can be understood that the interface connection relationships between the modules illustrated in the embodiment of the present invention are only schematic illustrations and do not constitute a structural limitation on the terminal 100 . In other embodiments of this application, the terminal 100 may also adopt the above implementation. Different interface connection methods in the example, or a combination of multiple interface connection methods.

The charging management module 140 is used to receive charging input from the charger. While the charging management module 140 charges the battery 149, it can also provide power to the electronic device through the power management module 141. The power management module 141 is used to connect the battery 149, the charging management module 140 and the processor 119. The power management module 141 receives input from the battery 149 and/or the charging management module 140 and supplies power to the processor 119, internal memory 129, display screen 194, camera 193, wireless communication module 160, etc.

The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied to the terminal 100. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves through the antenna 1 for radiation.

The wireless communication module 160 can provide applications on the terminal 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (bluetooth, BT), and global navigation satellite system. (global navigation satellite system, GNSS), frequency modulation (FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 119 . The wireless communication module 160 can also receive the signal to be sent from the processor 119, frequency modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the terminal 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal 100 can communicate with the network and other devices through wireless communication technology.

The terminal 100 implements the display function through the GPU, the display screen 194, and the application processor. The GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 119 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, videos, etc. Display 194 includes a display panel. Display 194 includes a display panel. The display panel can use a liquid crystal display (LCD). The display panel can also use organic light-emitting diode (OLED), active matrix organic light-emitting diode or active matrix organic light-emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode ( Manufacturing of flex light-emitting diodes (FLED), miniled, microled, micro-oled, quantum dot light emitting diodes (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

In the embodiment of the present application, the terminal 100 tracks the protagonist and determines the close-up image of the protagonist, and displays the figures shown in Figures 1A-1M, 2A-2D, 3A-3C, 4A-4H, and 5A-5E. The ability to display the user interface depends on the display functions provided by the above-mentioned GPU, display screen 194, and application processor.

The terminal 100 can implement the shooting function through the ISP, camera 193, video codec, GPU, display screen 194, application processor, etc.

The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, you open the shutter and light is transmitted through the lens. Passed to the camera photosensitive element, the optical signal is converted into an electrical signal. The camera photosensitive element passes the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

Camera 193 is used to capture still images or video. The object passes through the lens to produce an optical image that is projected onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other format image signals. In some embodiments, the terminal 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the terminal 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

Video codecs are used to compress or decompress digital video. Terminal 100 may support one or more video codecs. In this way, the terminal 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can continuously learn by itself. The NPU can realize intelligent cognitive applications of the terminal 100, such as image recognition, face recognition, speech recognition, text understanding, etc.

In the embodiment of the present application, the terminal 100 collects original images through the ISP and the shooting capabilities provided by the camera 193, and uses the video codec and the image computing and processing capabilities provided by the GPU to perform computing processing such as tracking the protagonist and determining the close-up image of the protagonist. Among them, the terminal 100 can implement neural network algorithms such as face recognition, human body recognition, and re-identification (ReID) through the computing processing capabilities provided by the NPU.

Internal memory 129 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

Random access memory can include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), double data rate synchronous Dynamic random access memory (double data rate synchronous dynamic random access memory, DDR SDRAM, such as the fifth generation DDR SDRAM is generally called DDR5SDRAM), etc.

Non-volatile memory can include disk storage devices and flash memory. Flash memory can be divided according to the operating principle to include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. According to the storage unit potential level, it can include single-level storage cells (single-level cell, SLC), multi-level storage cells (multi-level cell, MLC), third-level storage unit (triple-level cell, TLC), fourth-level storage unit (quad-level cell, QLC), etc., which can include universal flash storage (English: universal flash storage, UFS) according to storage specifications. , embedded multi media card (embedded multi media Card, eMMC), etc.

The random access memory can be directly read and written by the processor 119 and can be used to store the operating system or other ongoing The executable program (such as machine instructions) of a running program can also be used to store user and application data, etc. The non-volatile memory can also store executable programs and user and application data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 119.

In the embodiment of the present application, the code for implementing the video editing method described in the embodiment of the present application may be stored in a non-volatile memory. When running the camera application, the terminal 100 may load the executable code stored in the non-volatile memory into the random access memory.

The external memory interface 120 can be used to connect an external non-volatile memory to expand the storage capability of the terminal 100 . The external non-volatile memory communicates with the processor 119 through the external memory interface 120 to implement the data storage function.

The terminal 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. Speaker 170A, also called "speaker", is used to convert audio electrical signals into sound signals. The terminal 100 can listen to music through the speaker 170A, or listen to a hands-free call. Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the terminal 100 answers a call or a voice message, the voice can be heard by bringing the receiver 170B close to the human ear. Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. The headphone interface 170D is used to connect wired headphones.

In this embodiment of the present application, during the process of enabling the camera to collect images, the terminal 100 can simultaneously enable the microphone 170C to collect sound signals, and convert the sound signals into electrical signals to store them. In this way, users can get videos with sound.

The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 180A may be disposed on display screen 194 .

The gyro sensor 180B may be used to determine the movement posture of the terminal 100 . In some embodiments, the angular velocity of terminal 100 about three axes (ie, x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyro sensor 180B detects the angle at which the terminal 100 shakes, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shake of the terminal 100 through reverse movement to achieve anti-shake.

Air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation. Magnetic sensor 180D includes a Hall sensor. The terminal 100 may use the magnetic sensor 180D to detect the opening and closing of the flip cover. The acceleration sensor 180E can detect the acceleration of the terminal 100 in various directions (generally three axes). When the terminal 100 is stationary, the magnitude and direction of gravity can be detected. Distance sensor 180F for measuring distance. The terminal 100 can measure distance by infrared or laser. In some embodiments, when shooting a scene, the terminal 100 can use the distance sensor 180F to measure distance to achieve fast focusing. Proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal 100 emits infrared light through a light emitting diode. The terminal 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal 100 . When insufficient reflected light is detected, the terminal 100 may determine that there is no object near the terminal 100 . The ambient light sensor 180L is used to sense ambient light brightness. The terminal 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. ambient light Sensor 180L can also be used to automatically adjust white balance when taking pictures. Fingerprint sensor 180H is used to collect fingerprints. The terminal 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application lock, fingerprint photography, fingerprint answering incoming calls, etc. Temperature sensor 180J is used to detect temperature. In some embodiments, the terminal 100 uses the temperature detected by the temperature sensor 180J to execute the temperature processing policy.

Touch sensor 180K, also known as "touch device". The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is used to detect a touch operation on or near the touch sensor 180K. The touch sensor can pass the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the terminal 100 in a position different from that of the display screen 194 .

In the embodiment of the present application, the terminal 100 can use the touch sensor 180K to detect the user's click, slide and other operations on the display screen 194 to implement FIGS. 1A-1M, 2A-2D, 4A-4H, and FIG. 5A - The video editing method shown in Figure 5E.

Bone conduction sensor 180M can acquire vibration signals. The buttons 190 include a power button, a volume button, etc. The terminal 100 may receive key input and generate key signal input related to user settings and function control of the terminal 100. The motor 199 can generate vibration prompts. The motor 199 can be used for vibration prompts for incoming calls and can also be used for touch vibration feedback. The indicator 198 may be an indicator light, which may be used to indicate charging status, power changes, or may be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card. The terminal 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

The term "user interface (UI)" in the description, claims and drawings of this application is a media interface for interaction and information exchange between an application or operating system and a user, which implements the internal form of information. Conversion to and from a user-acceptable form. The user interface of an application is source code written in specific computer languages such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize. Such as pictures, text, buttons and other controls. Control, also called widget, is the basic element of user interface. Typical controls include toolbar, menu bar, text box, button, and scroll bar. (scrollbar), images and text. The properties and contents of controls in the interface are defined through tags or nodes. For example, XML specifies the controls contained in the interface through nodes such as <Textview>, <ImgView>, and <VideoView>. A node corresponds to a control or property in the interface. After parsing and rendering, the node is rendered into user-visible content. In addition, many applications, such as hybrid applications, often include web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in an application interface. A web page is source code written in a specific computer language, such as hypertext markup language (GTML), cascading styles Tables (cascading style sheets, CSS), java scripts (JavaScript, JS), etc., web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with functions similar to the browser. The specific content contained in the web page is also defined through tags or nodes in the web page source code. For example, GTML defines the elements and attributes of the web page through <p>, <img>, <video>, and <canvas>.

The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed graphically. It can be an image displayed on the display of an electronic device Interface elements such as icons, windows, and controls, where controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "above," "the" and "the" are intended to also include Plural expressions unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used in this application refers to and includes any and all possible combinations of one or more of the listed items. As used in the above embodiments, the term "when" may be interpreted to mean "if..." or "after" or "in response to determining..." or "in response to detecting..." depending on the context. Similarly, depending on the context, the phrase "when determining..." or "if (stated condition or event) is detected" may be interpreted to mean "if it is determined..." or "in response to determining..." or "on detecting (stated condition or event)” or “in response to detecting (stated condition or event)”.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, e.g., the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available media may be magnetic media (eg, floppy disk, hard disk, tape), optical media (eg, DVD), or semiconductor media (eg, solid state drive), etc.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments are implemented. This process can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, , may include the processes of the above method embodiments. The aforementioned storage media include: ROM, random access memory (RAM), magnetic disks, optical disks and other media that can store program codes.

Claims

A video editing method, applied to electronic devices, characterized in that the method includes:

A first image and one or more markers associated with the first image are displayed in the first interface; the first image includes one or more objects, and the one or more markers associated with the first image are respectively related to the Corresponding to one or more objects in the first image; the first image is an image currently collected by the camera of the electronic device, or a frame of image in the first video stored by the electronic device;

detecting a first operation acting on the first marker;

In response to the first operation, the first object is determined to be the protagonist, and a close-up image centered on the protagonist is obtained; one or more markers associated with the first image include the first marker, and the first image One or more objects in include the first object, and the first mark corresponds to the first object;

Based on the close-up image centered on the protagonist, a second video centered on the protagonist is generated.
The method according to claim 1, characterized in that, after determining the first object as the protagonist, further comprising:

A second image and one or more markers associated with the second image are displayed in the first interface, the second image includes one or more objects, and the one or more markers associated with the second image are respectively Corresponding to one or more objects in the second image; the second image is an image after the first image collected by the camera of the electronic device, or the first image in the first video The next frame of image;

detecting a fifth operation acting on the second marker;

In response to the fifth operation, the protagonist is switched to a second object, one or more markers associated with the second image include the second marker, and one or more objects in the second image include The second object, the second mark corresponds to the second object;

Obtaining a close-up image centered on the protagonist includes: generating a close-up image centered on the first object based on an image including the first object between the first image and the second image. , generate a close-up image centered on the second object based on the second image and subsequent images;

The second video includes a first sub-video and a second sub-video. The first sub-video is a video generated based on a close-up image centered on the first object. The second sub-video is based on a close-up image centered on the first object. Video generated from a close-up image centered on the second object.
The method according to claim 1, characterized in that the obtaining a close-up image centered on the protagonist is specifically:

A close-up image centered on the first object is generated according to an image including the first object from the first image to the last frame image in the first video.
The method according to claim 1, characterized in that when the second image is a frame of image after the first image in the first video, the first image and the first image are displayed on the first interface Before associating the one or more markers, the method further includes:

Display the thumbnail of the first video;

detecting a second operation acting on the thumbnail of the first video;

Displaying the first image and one or more marks associated with the first image on the first interface includes:

In response to the second operation, the first frame of the first video is displayed on the first interface, and one or more markers corresponding to one or more objects in the first frame of the image, the first The image is the first frame image.
The method according to claim 1, characterized in that when the second image is a frame of image after the first image in the first video, the first image and the first image are displayed on the first interface Before associating the one or more markers, the method further includes:

Display the first frame image of the first video and the first control on the first interface;

detecting a third operation acting on the first control;

In response to the third operation, playing the first video;

Displaying the first image and one or more marks associated with the first image on the first interface includes:

When the first video is played to the M-th frame image, the M-th frame image and one or more markers associated with the M-th frame image are displayed on the first interface.
The method according to claim 5, characterized in that when the first video is played to the Mth frame image, the Mth frame image is displayed on the first interface, and the Mth frame image One or more associated tags, including:

When the first video is played to the M-th frame image, a fourth operation acting on the first control is detected;

In response to the fourth operation, pause the playback of the first video and display the Mth frame image currently played;

In response to the operation of pausing playback, one or more markers associated with the M-th frame image are displayed on the M-th frame image.
The method according to claim 1 or 2, characterized in that the first interface further includes a second control that generates a second video centered on the protagonist based on the close-up image centered on the protagonist. ,include:

detecting a sixth operation acting on the second control;

In response to the sixth operation, a second video centered on the protagonist is generated based on the close-up image centered on the protagonist.
The method of claim 7, wherein when the first image is an image currently collected by a camera of the electronic device, the second control is a control for stopping recording.
The method of claim 8, further comprising: in response to the sixth operation, the camera stops collecting images, and the original video is generated and saved based on the images collected by the camera.
The method according to claim 8, characterized in that, after determining that the first object is the protagonist, the method further includes: displaying a first window, and displaying the first object centered on the protagonist in the first window. Close-up image.
The method according to claim 1 or 2, characterized in that when the first image is an image currently collected by the camera of the electronic device, the method further includes: detecting a first trigger condition, and the third A trigger condition is that the consecutive Y frame images after the first image do not include the protagonist;

The second video centered on the protagonist is generated based on the close-up image centered on the protagonist, specifically:

In response to the first trigger condition, a second video centered on the protagonist is generated based on the close-up image centered on the protagonist.
The method of claim 2, wherein the close-up image centered on the first object is generated based on the image including the first object between the first image and the second image. ,include:

Obtain a first close-up image centered on the first object from the first image;

A third close-up image centered on the first object is obtained from the third image; the third image is an image after the first image and before the second image; the second video includes The first close-up image and the second close-up image.
The method of claim 12, wherein before obtaining a third close-up image centered on the first object from the third image, the method further includes:

Determine whether the first object is included in the third image;

The third close-up image centered on the first object is obtained from the third image, specifically:

When the first object is included in the third image, a third close-up image centered on the first object is obtained from the third image.
The method of claim 13, wherein determining that the first object is included in the third image includes:

Using a human body detection algorithm to identify the human body image area in the third image;

When the human body image areas in the third image do not overlap, calculate the intersection-over-union ratio IoU distance between each human body image area in the third image and the human body image area of the protagonist in the first image; determine the The first human body image area with the smallest IoU distance and satisfying the IoU distance threshold; the object corresponding to the first human body image area is the protagonist;

When the human body image areas in the third image overlap, calculate the IoU distance and relocation ReID distance between each human body image area in the third image and the human body image area of the protagonist in the first image; determine The first human body image area where the sum of the IoU distance and the ReID distance is the smallest and satisfies the IoU+ReID distance threshold; the object corresponding to the first human body image area is the protagonist.
The method according to claim 14, wherein said obtaining a third close-up image centered on said protagonist from said third image specifically includes: determining based on said first human body image area including the The third close-up image of the first human body image area.
The method according to claim 15, wherein determining the third close-up image including the first human body image area based on the first human body image area specifically includes:

Determine a first scaling ratio according to the first human body image area;

The size of the third close-up image is determined based on the first scaling ratio.
The method of claim 16, wherein determining the first scaling ratio based on the first human body image area specifically includes: based on the size of the largest human body image area in the third image and the The size of the first human body image area determines the first scaling ratio.
The method of claim 17, wherein determining the size of the third close-up image based on the first scaling ratio specifically includes: based on the first scaling ratio, the preset second The size of the video determines the size of the third close-up image.
The method of claim 18, wherein the aspect ratio of the third close-up image is the same as the preset aspect ratio of the second video.
An electronic device, characterized by comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories For storing computer program code, the computer program code includes computer instructions that, when executed by the one or more processors, cause the method of any one of claims 1-19 to be performed.
A computer-readable storage medium comprising instructions, characterized in that when the instructions are run on an electronic device, the method according to any one of claims 1-19 is executed.