WO2024040865A1

WO2024040865A1 - Video editing method and electronic device

Info

Publication number: WO2024040865A1
Application number: PCT/CN2023/073141
Authority: WO
Inventors: 王龙
Original assignee: 荣耀终端有限公司
Priority date: 2022-08-25
Filing date: 2023-01-19
Publication date: 2024-02-29
Also published as: CN115134646B; CN115134646A

Abstract

The present application relates to the field of video processing, and provides a video editing method and an electronic device. The video editing method is applied to an electronic device, and comprises: displaying a first interface, the first interface comprising video icons, and videos indicated by the video icons being videos stored in the electronic device; detecting a first operation on N video icons among the video icons; in response to the first operation, acquiring information of N videos, N being an integer greater than 1; obtaining a video theme of the N videos on the basis of the information of the N videos; selecting M video clips in the N videos on the basis of the similarities between images in the N videos and the video theme; on the basis of the video theme, obtaining music matching the video theme; obtaining a first video on the basis of the M video clips and the music; and displaying the first video. On the basis of the solution of the present application, the problem that an edited video has image content unrelated to the overall video theme of the N videos can be avoided, and the video quality of the edited video is improved.

Description

Video editing methods and electronic equipment

This application claims priority to the Chinese patent application submitted to the State Intellectual Property Office on August 25, 2022, with application number 202211024258.6 and the application name "Video Editing Method and Electronic Device", the entire content of which is incorporated into this application by reference. middle.

Technical field

The present application relates to the field of video, and specifically to a video editing method and electronic device.

Background technique

With the development of short video technology in electronic devices, users have higher and higher demands for video editing functions. Video mixing and editing refers to a video editing technology that divides multiple videos to select target segments, then reorganizes the video segments and adds background music to generate a new video.

Currently, users can automatically edit multiple videos through existing applications to achieve video mixing and cutting; however, existing applications are less professional when editing multiple videos, resulting in the processed video being There are issues; for example, the edited video may have graphic content that is not relevant to the overall video theme of multiple videos.

Therefore, how to improve the professionalism of automatic video editing in electronic devices and improve the video quality of edited videos has become an urgent problem that needs to be solved.

Contents of the invention

This application provides a video editing method and electronic device, which can avoid the problem of image content irrelevant to the overall video theme of N videos in the edited video, and improve the video quality of the edited video.

The first aspect provides a video editing method applied to electronic devices, including:

Display a first interface, which includes a video icon, and the video indicated by the video icon is a video stored in the electronic device;

The first operation on N video icons among the video icons is detected;

In response to the first operation, obtain information of N videos, where N is an integer greater than 1;

Based on the information of N videos, the video topics of N videos are obtained;

Based on the similarity between the images in the N videos and the video topics, select M video clips from the N videos;

Based on the video theme, get music that matches the video theme;

Based on M video clips and music, the first video is obtained;

Show the first video.

In the embodiment of the present application, M video clips can be selected from the N videos based on the similarity between the images in the N videos and the video themes; the processed video, that is, the first video, can be obtained based on the M video clips and music. ; In the solution of this application, based on the similarity between the images included in the N videos and the video themes, M video clips in the N videos that are highly relevant to the video themes can be determined; Based on the solution of this application, it is possible to determine Effectively delete video clips from N videos that are irrelevant to the overall video topic information, ensure that the filtered video clips are related to the video topic, and improve the video quality of the edited first video.

Combined with the first aspect, in some implementations of the first aspect, based on images in N videos and video themes Similarity, select M video clips from N videos, including:

Input N videos and video topics into the pre-trained similarity matching model to obtain the similarity confidence values of the images and video topics in the N videos. The pre-trained similarity matching model includes image encoders and text encoding. and the first similarity measurement module. The image encoder is used to extract image features from N videos. The text encoder is used to extract text features from video topics. The first similarity measurement module is used to measure the images in N videos. The similarity between the feature and the text feature of the video topic, the similarity confidence value is used to represent the probability that the image in N videos is similar to the video topic;

Based on the similarity confidence values between the images in the N videos and the video topics, M video clips from the N videos are selected.

In the embodiments of this application, the similarity between the image features in the video and the text features of the video subject can be identified through a pre-trained similarity matching model; the pre-trained similarity matching model can be a multi-modal model, Supports two different types of input data, image and text, at the same time; text features and image features can be mapped into a unified space through a pre-trained similarity matching model, thereby improving visual and text understanding capabilities; in the solution of this application, Based on the pre-trained similarity matching model, it can intelligently identify the similarity between the image features in the video and the text features of the video subject.

Combined with the first aspect, in some implementations of the first aspect, the first video is obtained based on M video clips and music, including:

Sort the M video clips to obtain the sorted M video clips;

The sorted M video clips and music are synthesized into the first video.

In the embodiment of the present application, the image content in the M video clips can be made more consistent with the music rhythm in the music; for example, if the video image content is scenery, it can correspond to the prelude of the music or the soothing music part; the video image If the content is the user's sports scene, it can correspond to the climax of the background music; by sorting the M video clips, the M video clips can better match the rhythm stuck points of the music; thereby solving the problem in the first video The problem of the video clips not matching the background music can be solved, that is, the problem of the video content of the first video not fully matching the rhythm of the music can be solved; and the video quality of the first video can be improved.

Combined with the first aspect, in some implementations of the first aspect, the M video clips are sorted to obtain the sorted M video clips, including:

Sort the M video clips based on the rhythm of the music to obtain the sorted M video clips.

In the solution of this application, background music can be selected based on the overall video theme information of N videos; and M videos can be sorted based on the rhythm of the background music, so as to achieve video sorting of M video clips according to the rhythm of the background music. , so that the picture content of the video clip matches the rhythm of the music; compared with the video matching the music directly according to the input order, the solution of this application can improve the consistency of the image content in the video and the rhythm of the background music, and improve the video quality of the edited video .

The M video clips are sorted based on the video contents in the M video clips to obtain the sorted M video clips.

In the solution of this application, for N videos with strong story lines, the N videos can be sorted based on the text description information of the N videos to obtain the sorted N videos; from the sorted N videos, select M video clips with high relevance to the video topic information are used to obtain sorted M video clips; based on the sorted M video clips and video topic information, music matching the sorted M video clips is determined as the background Music; when the picture content of N videos with strong storylines matches the rhythm of the music, and the playback sequence of the video picture content conforms to the causal connection, it improves Improve the video quality of the edited video.

It should be understood that a video with a strong story line can refer to a causal connection between N videos. Based on the video editing method, the cause and effect between the N videos can be identified and the N videos can be sorted based on the order of the cause and effect; for example, a strong story line The videos can include travel-themed videos or travel-themed videos.

Combined with the first aspect, in some implementations of the first aspect, the M video clips are sorted based on the rhythm of the music to obtain the sorted M video clips, including:

Input music and M video clips into the pre-trained audio-visual rhythm matching model to obtain sorted M video clips. The pre-trained audio-visual rhythm matching model includes an audio encoder, a video encoder and a first similarity measurement module. The audio encoder is used to extract features from music to obtain audio features, the video decoder is used to extract features from M video clips to obtain video features, and the first similarity measurement module is used to measure the similarity between the audio features and the M video clips.

In the embodiment of the present application, the similarity between the video features of the M video clips and the audio features of the music can be identified through a pre-trained audio-visual rhythm matching model; the pre-trained audio-visual rhythm matching model can be a multi-modal model. , simultaneously supporting two different types of input data, video and audio; through the pre-trained audio-visual rhythm matching model, video features and audio features can be mapped into a unified space, thereby improving visual and audio understanding capabilities; in the solution of this application , based on the pre-trained audio-visual rhythm matching model, it can intelligently identify the similarity between the video features of M video clips and the audio features of music.

Combined with the first aspect, in some implementations of the first aspect, based on the information of the N videos, video themes of the N videos are obtained, including:

Convert the video content of N videos into N text description information. The N text description information corresponds to the N videos one-to-one. One text description information among the N text description information is used to describe the image of one video among the N videos. content information;

Based on the N pieces of text description information, the topic information of the N videos is obtained, and the text description information is used to convert the video content in the N videos into text information.

In the embodiment of the present application, when identifying the video themes of N videos, the video theme information corresponding to the N videos is obtained through the text description information of the N videos; that is, the N videos can be obtained based on the text description information of the N videos The overall video topic information; compared with the video topic information based on the image semantics of N videos, text information has more abstract semantic information than image information, and there is language correlation between multiple text information, which helps to infer multiple The theme information hidden behind the text can improve the accuracy of the overall video theme corresponding to N videos; for example, N videos include videos of users packing their luggage, videos of users going out and taking a car to the airport, and videos of users taking a plane. A video with a user walking on the beach; based on image semantics, only some image tags may be obtained, including clothing, suitcases, users, beach, etc. Based on these image tags, it is impossible to abstract the video theme of N videos as travel; however, based on N When identifying the video theme from the text description information of N videos, the video theme information of N videos can be accurately obtained based on the linguistic logical correlation between N video text description information and N video text description information; for example, based on N The text description information included in the video is "a user is packing luggage", "a user is taking a plane", "a user is walking on the beach". Based on this text description information, the video theme information of N videos can be abstracted as travel; therefore, The video theme information of N videos is obtained through the text description information of N videos, which can improve the accuracy of the theme information.

Combined with the first aspect, in some implementations of the first aspect, based on the N pieces of text description information, the topic information of the N videos is obtained, including:

Input N pieces of text description information into a pre-trained topic classification model to obtain topic information of N videos. The pre-trained topic classification model is a deep neural network used for text classification.

In the embodiment of the present application, the video topic information corresponding to the text description information of N videos can be obtained based on the pre-trained topic classification model; the video topic information corresponding to the text description information of the N videos is identified through the pre-trained topic classification model. frequency topic information; compared with video topic information based on the image semantics of N videos, text information has more abstract semantic information than image information, and there is language correlation between multiple text information, which helps to speculate on the background of multiple texts The implicit topic information can improve the accuracy of the overall video topic corresponding to N videos; in addition, the pre-trained topic classification model can more intelligently identify the video topic information corresponding to N text description information.

Combined with the first aspect, in some implementations of the first aspect, when the pre-trained topic classification model outputs at least two video topics, the at least two video topics correspond to N pieces of text description information, which also includes:

Display a second interface, the second interface includes a prompt box, and the prompt box includes information on at least two video topics;

Input N text description information into the pre-trained topic classification model to obtain the topic information of N videos, including:

A second action on at least two video subjects is detected;

In response to the second operation, the theme information of N videos is obtained.

In embodiments of the present application, when the electronic device outputs at least two video themes, the electronic device can display a prompt box; based on detecting the user's operation on the candidate video theme in the prompt box, the video theme information of N videos can be determined; To a certain extent, it can be avoided that the electronic device cannot recognize the video themes of the N videos when the video contents of the N videos do not completely comply with the pre-set video themes.

Combined with the first aspect, in some implementations of the first aspect, based on the video theme, music matching the video theme is obtained, including:

Based on the duration and video theme of the M video clips, music that matches the video theme is obtained, and the duration of the music is greater than or equal to the duration of the M video clips.

In the embodiment of the present application, the total duration of the background music can be determined based on the duration of the M video clips. The background music usually selected when performing music matching needs to be greater than or equal to the total duration of the M video clips; based on the video theme information, the total duration of the background music can be determined. Determine the music style of the music; in the solution of this application, the music matching the M video clips can be more accurately selected as background music based on the duration and video theme of the M video clips, thereby improving the video quality of the edited video; that is, improving the The video quality of a video.

Combined with the first aspect, in some implementations of the first aspect, the pre-trained similarity matching model is a Transformer model.

Combined with the first aspect, in some implementations of the first aspect, the pre-trained similarity matching model is obtained through the following training method:

Based on the first training data set, the contrastive learning training method is used to train the similarity matching model to be trained, and a pre-trained similarity matching model is obtained; wherein, the first training data set includes positive example data pairs and negative example data pairs. The example data pair includes the first sample text description information and the first sample video theme information. The first sample description information matches the first sample video theme information. The positive example data pair includes the first sample text description information and the first sample video theme information. The second sample video theme information and the first sample description information do not match the second sample video theme information.

Combined with the first aspect, in some implementations of the first aspect, the pre-trained audio-visual rhythm matching model is a Transformer model.

Combined with the first aspect, in some implementations of the first aspect, the pre-trained audio-visual rhythm matching model is obtained through the following training method:

Based on the second training data set, the contrastive learning training method is used to train the similarity matching model to be trained, and a pre-trained similarity matching model is obtained; wherein, the second training data set includes positive example data pairs and negative example data pairs. The example data pair includes the first sample music and the first sample video. The rhythm of the first sample music matches the content of the first sample video. The negative example data pair includes the first sample music and the second sample video. The tempo of the first sample music does not match the content of the second sample video.

In a second aspect, an electronic device is provided. The electronic device includes one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code. The computer program code includes computer instructions, one or more Multiple processors invoke computer instructions to cause the electronic device to:

The first operation on N video icons among the video icons is detected;

Based on the video theme, get music that matches the video theme;

Based on M video clips and music, the first video is obtained;

Show the first video.

In conjunction with the second aspect, in some implementations of the second aspect, one or more processors invoke computer instructions to cause the electronic device to execute:

Sort the M video clips to obtain the sorted M video clips;

The sorted M video clips and music are synthesized into the first video.

Convert the video content of N videos into N text description information, and the N text description information is one-to-one with the N videos. Accordingly, one text description information among the N pieces of text description information is used to describe the image content information of one video among the N videos;

Combined with the second aspect, in some implementations of the second aspect, when the pre-trained topic classification model outputs at least two video topics, and the at least two video topics correspond to N pieces of text description information, one or more processes The processor calls computer instructions to cause the electronic device to perform:

A second action on at least two video subjects is detected;

Based on the duration and video theme of the M video clips, music matching the video theme is obtained, and the duration of the music is greater than or equal to the duration of the M video clips.

Combined with the second aspect, in some implementations of the second aspect, the pre-trained similarity matching model is a Transformer model.

Combined with the second aspect, in some implementations of the second aspect, the pre-trained similarity matching model is obtained through the following training method:

Combined with the second aspect, in some implementations of the second aspect, the pre-trained audio-visual rhythm matching model is a Transformer model.

Combined with the second aspect, in some implementations of the second aspect, the pre-trained audio-visual rhythm matching model is obtained through the following training method:

In a third aspect, an electronic device is provided, including a module/unit for executing the video editing method in the first aspect or any implementation of the first aspect.

A fourth aspect provides an electronic device, the electronic device comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, The computer program code includes computer instructions that are invoked by the one or more processors to cause the electronic device to perform Perform the video editing method in the first aspect or any one of the implementations of the first aspect.

In a fifth aspect, a chip system is provided. The chip system is applied to an electronic device. The chip system includes one or more processors. The processor is used to call computer instructions to cause the electronic device to execute the first aspect. Or any of the video editing methods in the first aspect.

In a sixth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program code. When the computer program code is run by an electronic device, the electronic device causes the electronic device to execute the first aspect or the first aspect. The video editing method in any implementation of the aspects.

In a seventh aspect, a computer program product is provided. The computer program product includes: computer program code. When the computer program code is run by an electronic device, the electronic device causes the electronic device to execute the first aspect or any of the first aspects. A video editing method in an implementation manner.

In addition, in the embodiments of the present application, the problem of mismatch between the video clips and the music existing in the edited video can be solved, that is, the problem of the edited video content not fully matching the rhythm stuck point of the background music can be solved; in this application In the embodiment of the application, the image content in the M video clips can be made more consistent with the music rhythm in the music; for example, if the video image content is scenery, it can correspond to the prelude of the music or the soothing music part; the video image content is The user's sports scene can correspond to the climax of the background music; by sorting the M video clips, the M video clips can better match the rhythm of the music; and the video quality of the edited video can be improved.

Description of drawings

Figure 1 is a schematic diagram of a hardware system suitable for electronic equipment of the present application;

Figure 2 is a schematic diagram of the structure of a transformer model suitable for this application;

Figure 3 is a schematic diagram of the structure of the encoder and decoder in the Transformer model;

Figure 4 is a schematic diagram of a software system suitable for the electronic device of the present application;

Figure 5 is a schematic diagram of a graphical user interface suitable for embodiments of the present application;

Figure 6 is a schematic diagram of a graphical user interface suitable for embodiments of the present application;

Figure 7 is a schematic diagram of a graphical user interface suitable for embodiments of the present application;

Figure 8 is a schematic diagram of a graphical user interface suitable for embodiments of the present application;

Figure 9 is a schematic diagram of a graphical user interface suitable for embodiments of the present application;

Figure 10 is a schematic diagram of a graphical user interface suitable for embodiments of the present application;

Figure 11 is a schematic diagram of a graphical user interface suitable for embodiments of the present application;

Figure 12 is a schematic flow chart of a video editing method provided by an embodiment of the present application;

Figure 13 is a schematic flow chart of another video editing method provided by an embodiment of the present application;

Figure 14 is a schematic flow chart of a method for determining M video clips related to video theme information in N videos provided by an embodiment of the present application;

Figure 15 is a schematic diagram of the processing flow of a similarity evaluation model implemented in this application;

Figure 16 is a flow chart of a method for rhythm matching processing of M video clips and background music provided by an embodiment of the present application;

Figure 17 is a schematic diagram of the processing flow of an audio-visual rhythm matching model provided by an embodiment of the present application;

Figure 18 is a schematic flow chart of another video editing method provided by an embodiment of the present application;

Figure 19 is a schematic flow chart of another video editing method provided by an embodiment of the present application;

Figure 20 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

Figure 21 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

In the embodiments of the present application, the following terms “first”, “second”, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

In order to facilitate understanding of the embodiments of the present application, the relevant concepts involved in the embodiments of the present application are briefly described first.

1. Image features

Image features refer to a set of attributes that characterize the characteristics or content of an image; for example, image features can include color features, texture features, shape features, and spatial relationship features of the image, etc., or they can be obtained through some kind of mapping. Formula attribute expression.

2. Video characteristics

Video features refer to a set of attributes that can characterize the characteristics of the video obtained from the image sequence in the video through some mapping.

3. Text features

Text features refer to the set of attributes that characterize a word or sentence's specific semantics obtained through vectorization and subsequent mapping.

4. Image and text multimodal (contrastive language–image pre-training, CLIP) model

The CLIP model is a cross-modal pre-training model based on contrastive picture-text learning.

5. Neural network

Neural network refers to a network formed by connecting multiple single neural units together, that is, the output of one neural unit can be the input of another neural unit; the input of each neural unit can be connected to the local receptive field of the previous layer, To extract the features of the local receptive field, the local receptive field can be an area composed of several neural units.

6. Comparative learning

Contrastive learning is one of the self-supervised learning methods; contrastive learning refers to a training method that learns knowledge from unlabeled images without relying on labeled data.

It should be understood that the goal of contrastive learning is to learn an encoder that encodes similar data similarly and makes the encoding results of different types of data as different as possible.

7. Transformer model

The Transformer model can be composed of an encoder and a decoder. As shown in Figure 2, the encoder and decoder can include multiple sub-modules; for example, an encoder can include 6 encoding modules; a decoder can include 6 decoding modules.

For example, as shown in Figure 3, a coding module can include: embedding layer, position coding, multi-head attention mechanism module, residual connection and linear normalization and forward network modules; where the embedding layer is used to Each input data Words are represented by a vector; position encoding is used to construct a matrix with the same vector dimension as the input data, so that the data input to the multi-head attention mechanism module contains position information; the multi-head attention mechanism module is used to utilize multiple queries of the same query. Different versions implement the work of multiple attention modules in parallel; the idea is to use different weight matrices to linearly transform the query to obtain multiple queries. Each newly formed query essentially requires a different type of relevant information, thus allowing attention The model introduces more information into the context vector calculation; residual connections are used to prevent network degradation; linear normalization is used to normalize the activation values of each layer; the forward network module is used to do the obtained word representation Further transformations.

For example, as shown in Figure 3, a decoding module can include: embedding layer, position coding, masked multi-head attention mechanism module, residual connection and linear normalization, forward network module and multi-head attention mechanism module ; Among them, the embedding layer is used to represent each word in the input data with a vector; the position encoding is used to construct a matrix with the same vector dimension as the input data, so that the data input to the multi-head attention mechanism module contains position information ; The mask multi-head attention mechanism module is used to ensure that the previous words do not have information about the following words by using masks, thereby ensuring that the output data predicted by the Transformer model will not change based on the number of input words; the multi-head attention mechanism module is used to implement the work of multiple attention modules in parallel by leveraging multiple different versions of the same query; the idea is to linearly transform the query using different weight matrices to obtain multiple queries, each newly formed query essentially requires Different types of relevant information, thus allowing the attention model to introduce more information in the context vector calculation; residual connections are used to prevent network degradation; linear normalization is used to normalize the activation values of each layer; forward network The module is used to further transform the obtained word representation.

8. Deep neural network (DNN)

Deep neural network can also be called multi-layer neural network, which can be understood as a neural network with multiple hidden layers. DNN is divided according to the position of different layers. The neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected; that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

9. Backpropagation algorithm

The neural network can use the error back propagation (BP) algorithm to modify the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forward propagation of the input signal until the output will produce an error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss converges. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

10. Transition effect

Transition effects can also be called transition effects. Transition effects refer to the use of certain techniques between two scenes, such as wipes, stacking changes, page curls, etc., to achieve smooth transitions between scenes or plots, or to achieve rich The effect of the picture.

The video editing method and electronic device provided in the embodiments of the present application will be described below with reference to the accompanying drawings.

Figure 1 shows a hardware system suitable for the electronic device of the present application.

The electronic device 100 may be a mobile phone, a smart screen, a tablet, a wearable electronic device, a vehicle-mounted electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a notebook computer, or a super mobile personal computer ( Ultra-mobile personal computer (UMPC), netbook, personal digital assistant (personal digital assistant, PDA), projector, etc. The embodiment of the present application does not place any restrictions on the specific type of the electronic device 100.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2. , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone interface 170D, sensor module 180, button 190, motor 191, indicator 192, Camera 193, display screen 194, and subscriber identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

It should be noted that the structure shown in FIG. 1 does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or less components than those shown in FIG. 1 , or the electronic device 100 may include a combination of some of the components shown in FIG. 1 , or , the electronic device 100 may include sub-components of some of the components shown in FIG. 1 . The components shown in Figure 1 may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), an image signal processor (image signal processor) , ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, neural network processing unit (NPU). Among them, different processing units can be independent devices or integrated devices. The controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. For example, the processor 110 may include at least one of the following interfaces: an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, pulse code modulation, PCM) interface, universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, SIM interface, USB interface.

Illustratively, in the embodiment of the present application, the processor 110 may be configured to execute the video editing method provided by the embodiment of the present application; for example, display a first interface, the first interface includes a video icon, and the video indicated by the video icon is Videos stored in electronic devices; detect the first operation on N video icons in the video icons; in response to the first operation, obtain information of N videos, where N is an integer greater than 1; based on the information of N videos, obtain Video themes of N videos; based on the similarity between the images in the N videos and the video themes, select M video clips from the N videos; based on the video themes, obtain music that matches the video theme; based on the M video clips With music, get the first video; show the first video.

The connection relationship between the modules shown in FIG. 1 is only a schematic illustration and does not constitute a limitation on the connection relationship between the modules of the electronic device 100 . Optionally, each module of the electronic device 100 may also adopt a combination of various connection methods in the above embodiments.

The wireless communication function of the electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, baseband processor and other components.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be reused as a diversity antenna for a wireless LAN. In other embodiments, antennas may be used in conjunction with tuning switches.

The electronic device 100 may implement display functions through a GPU, a display screen 194, and an application processor. GPU is the graphics processing unit The processing microprocessor is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

Display 194 may be used to display images or videos.

Optionally, display screen 194 may be used to display images or videos. Display 194 includes a display panel. The display panel can use liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode (AMOLED), flexible Light-emitting diode (flex light-emitting diode, FLED), mini light-emitting diode (Mini LED), micro light-emitting diode (micro light-emitting diode, Micro LED), micro OLED (Micro OLED) or quantum dot light emitting Diodes (quantum dot light emitting diodes, QLED). In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

For example, in the embodiment of the present application, the display screen 194 can display a video or photo selected by the user; and display the processed video.

For example, the electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

Illustratively, the ISP is used to process data fed back by the camera 193 . For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the camera, the light signal is converted into an electrical signal, and the camera sensor passes the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can algorithmically optimize the noise, brightness and color of the image. ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

Illustratively, a camera 193 (which may also be referred to as a lens) is used to capture still images or videos. It can be triggered by application instructions to realize the camera function, such as capturing images of any scene. The camera can include imaging lenses, filters, image sensors and other components. The light emitted or reflected by the object enters the imaging lens, passes through the optical filter, and finally converges on the image sensor. The imaging lens is mainly used to collect and image the light emitted or reflected by all objects in the camera angle (which can also be called the scene to be shot, the target scene, or the scene image that the user expects to shoot); the filter is mainly used to To filter out excess light waves in the light (such as light waves other than visible light, such as infrared); the image sensor can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) ) phototransistor. The image sensor is mainly used to photoelectrically convert the received optical signal into an electrical signal, and then transfer the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other format image signals.

For example, the digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

Illustratively, video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3 and MPEG4.

For example, the gyro sensor 180B may be used to determine the motion posture of the electronic device 100 . In some embodiments, the angular velocity of electronic device 100 about three axes (ie, x-axis, y-axis, and z-axis) may be determined by gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyro sensor 180B detects the angle at which the electronic device 100 shakes, and calculates the distance that the lens module needs to compensate based on the angle, so that the lens can offset the shake of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used in scenarios such as navigation and somatosensory games.

For example, the acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally the x-axis, y-axis, and z-axis). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E can also be used to identify the posture of the electronic device 100 as an input parameter for applications such as horizontal and vertical screen switching and pedometer.

Illustratively, distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, such as in a shooting scene, the electronic device 100 may utilize the distance sensor 180F to measure distance to achieve fast focusing.

Illustratively, the ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touching.

Illustratively, fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering incoming calls.

For example, the touch sensor 180K is also called a touch device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen. The touch screen is also called a touch screen. The touch sensor 180K is used to detect a touch operation acted on or near the touch sensor 180K. The touch sensor 180K may pass the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 and at a different position from the display screen 194 .

The hardware system of the electronic device 100 is described in detail above, and the software system of the electronic device 100 is introduced below.

FIG. 4 is a schematic diagram of a software system of an electronic device provided by an embodiment of the present application.

As shown in FIG. 4 , the system architecture may include an application layer 210 , an application framework layer 220 , a hardware abstraction layer 230 , a driver layer 240 and a hardware layer 250 .

Application layer 210 may include a gallery application.

Optionally, the application layer 210 may also include application programs such as camera application, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

The application framework layer 220 provides an application programming interface (API) and programming framework for applications in the application layer; the application framework layer may include some predefined functions.

For example, the application framework layer 220 may include a gallery access interface; the gallery access interface may be used to obtain relevant data of the gallery.

Hardware abstraction layer 230 is used to abstract hardware.

For example, the hardware abstraction module may include a video editing algorithm; based on the video editing algorithm, the video editing related methods of the embodiments of the present application may be executed.

The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include display driver.

Hardware layer 250 may include a display screen and other hardware devices.

In view of this, embodiments of the present application provide a video editing method and electronic device; in the embodiment of the present application, the image content information of N videos can be converted into text description information; based on the text description of N videos Information to obtain the video theme information of N videos; based on the correlation between the images in the N videos and the video theme information, select M video clips from the N videos; obtain the processed video based on the M video clips and background music ; In the solution of this application, the video theme information of N videos is obtained through the text description information of N videos; compared with obtaining the video theme information of N videos based on the image information of N videos, the text information has more advantages than the image information. Rich information; in addition, there is language correlation between multiple text information, and the video subject information of the video is obtained based on the text description information of N videos, which can improve the accuracy of the video subject information; in addition, in the embodiment of the present application , based on the correlation between the images in the N videos and the video theme information, M video clips in the N videos that are highly relevant to the video theme can be determined; based on the solution of this application, the N videos can be effectively deleted that are related to the overall video Video clips with irrelevant topic information ensure that the filtered video clips are related to the video topic information and improve the video quality of the edited video.

Furthermore, in the embodiments of the present application, the problem of the video clips in the edited video not matching the background music can be solved, that is, the problem of the edited video content not completely matching the rhythm stuck points of the background music can be solved. .

Optionally, in the solution of this application, background music can be selected based on the overall video theme information of the N videos; and the M videos can be sorted based on the rhythm of the background music, so that the M videos can be sorted according to the rhythm of the background music. Sort the video clips so that the picture content of the video clips matches the rhythm of the music; compared with matching the video directly with the music in the input order, the solution of this application can improve the consistency of the image content in the video and the rhythm of the background music, and improve the quality of the video clips after editing. The video quality of the video.

Optionally, in the solution of this application, for N videos with strong story lines, the N videos can be sorted based on the text description information of the N videos to obtain the sorted N videos; from the sorted N videos Select M video clips from the video that are highly relevant to the video topic information to obtain the sorted M video clips; based on the sorted M video clips and the video topic information, determine the match with the sorted M video clips background music; when the picture content of the N videos with a strong storyline matches the music rhythm, and the playback sequence of the video picture content conforms to the causal relationship, the video quality of the edited video is improved.

The video editing method provided by the embodiment of the present application can effectively filter out the video clips in the N videos that are not relevant to the overall video theme based on the correlation between the video content in the N videos and the overall video theme; based on the video content (for example, The emotions expressed in the video and the picture of the video) match the background music; based on the rhythm of the background music or the logical correlation of the video clips, multiple video clips are reasonably connected in series; so that the edited video does not include content that is irrelevant to the overall video theme. Moreover, the video content matches the rhythm of the background music, thereby improving the professionalism of electronic device video editing and improving the video quality of the edited video.

Illustratively, the video editing method provided by the embodiment of the present application is suitable for automatically generating mixed-cut videos in electronic devices; for example, the electronic device detects the user's selection operations on multiple videos; identifies the video themes of multiple videos, and matches Background music related to multiple video themes; automatically synthesize multiple videos and background music to generate a mixed-cut video.

Optionally, the method provided by the embodiment of the present application is not only applicable to videos saved in electronic devices; it is also applicable to photos saved in electronic devices; for example, a mixed-cut video is generated based on photos saved in electronic devices; where the photos include but Not limited to: gif animations, JPEG format images, PNG format images, etc.

The following is a detailed description of the relevant interface diagrams of the video editing method provided by the embodiment of the present application with reference to Figures 5 to 11.

Exemplarily, as shown in Figure 5, the graphical user interface (GUI) shown in (a) of Figure 5 is the desktop 301 of the electronic device; the electronic device detects that the user clicks on the gallery application on the desktop. The operation of the control 302 is as shown in (b) of Figure 5; after the electronic device detects that the user clicks on the control 302 of the gallery application on the desktop, the gallery display interface 303 as shown in (c) of Figure 5 is displayed. ; The gallery display interface 303 includes all photo icons, video icons and controls 304 for more options, and the electronic device detects the user's operation of clicking on the controls 304 for more options, As shown in (d) in Figure 5; after the electronic device detects that the user clicks on the control 304 for more options, the display interface 305 shown in (a) in Figure 6 is displayed; in the display interface 305, including The one-click blockbuster control 306; the electronic device detects the user's operation of clicking the one-click blockbuster control 306, as shown in (b) in Figure 6; after the electronic device detects the user's click on the one-click blockbuster control 306, The display interface 307 shown in (c) of Figure 6 is displayed; the display interface 307 includes the icon of the video saved in the electronic device and the multi-select control 308; the electronic device detects the user's operation of clicking the multi-select control 308, as shown in Figure As shown in (d) in Figure 6; after the electronic device detects that the user clicks on the multi-select control 308, the display interface 309 as shown in (a) in Figure 7 is displayed, and the display interface 309 includes a video icon 310; the electronic device The user's operation of clicking the icon 310 is detected, as shown in (b) of Figure 7; after the electronic device detects the user's operation of clicking the icon 310, the display interface 311 is displayed as shown in (c) of Figure 7; in The display interface 311 includes a video icon 312, and the electronic device detects the user's click on the icon 312, as shown in (d) in Figure 7; after the electronic device detects the user's click on the icon 312, the display is shown in Figure 8 The display interface 313 shown in (a); the display interface 313 includes a video icon 314, and the electronic device detects the user's click on the icon 314, as shown in (b) of Figure 8; when the electronic device detects the user's click After the icon 314 is operated, the display interface 315 shown in (c) in Figure 8 is displayed; the display interface 315 includes a one-click blockbuster control 316; the electronic device detects the user's operation of clicking the one-click blockbuster control 316, such as As shown in (d) in Figure 8.

In one example, after the electronic device detects the user's operation of clicking on the one-click blockbuster control 316, the electronic device can execute the video editing method provided by the embodiment of the present application to perform video editing processing on multiple videos selected by the user, displaying the following: The display interface 317 shown in Figure 9.

It should be understood that for different video theme information, the electronic device can pre-configure a template corresponding to the video theme information; therefore, the electronic device can display the display interface 317 as shown in FIG. 9 .

In one example, after the electronic device detects the user's operation of clicking on the one-click blockbuster control 316, the electronic device can execute the video editing method provided by the embodiment of the present application to perform video editing processing on multiple videos selected by the user, displaying the following: The display interface 318 shown in Figure 10; the display interface 318 includes the video theme information obtained based on the solution of the present application as "travel"; in addition, the template 1, template 2, and template corresponding to "travel" can be displayed in the display interface 318 3 with template 4.

It should be understood that for different video theme information, the electronic device can pre-configure multiple templates corresponding to the video theme information; therefore, the electronic device can display the display interface 318 as shown in FIG. 10 .

In one example, after the electronic device detects the user's operation of clicking on the one-click blockbuster control 316, the electronic device can execute the video editing method provided by the embodiment of the present application to perform video editing processing on multiple videos selected by the user, displaying the following: Display interface 319 shown in Figure 11; Based on the solution of this application, if the electronic device obtains two or more video theme information, the prompt box 320 can be displayed on the electronic device; as shown in Figure 11, the prompt box 320 includes two video themes, namely scenery and travel; based on the user's operation on the video theme in the prompt box 320, the electronic device can determine one video theme information from two or more video themes.

It should be understood that the above is an example of selecting a video for video editing processing in an electronic device; the video editing method provided in the embodiment of the present application is also applicable to performing video editing processing on photos saved in an electronic device to generate a mixed-cut video; wherein , photos include but are not limited to: gif animations, JPEG format images, PNG format images, etc.; this application does not make any restrictions on this.

The video editing method provided by the embodiment of the present application will be described in detail below with reference to Figures 12 to 19.

Figure 12 is a schematic flow chart of a video editing method provided by an embodiment of the present application. The video editing method 400 can be executed by the electronic device shown in Figure 1; the video editing method includes steps S410 to S480, and steps S410 to 480 are described in detail below respectively.

Step S410: Display the first interface.

The first interface includes a video icon, and the video indicated by the video icon is a video stored in the electronic device.

For example, the first interface may refer to the display interface of the gallery application in the electronic device, such as the display interface 307 shown in (c) of Figure 6; the display interface 307 includes 6 video icons, and the 6 video images The corresponding video is a video stored in the electronic device.

Step S420: Detect the first operation on N video icons among the video icons.

For example, the first operation may be a click operation on N video images in the video icon, or the first operation may be another selection operation on N video images.

For example, as shown in (b) of Figure 7 , the electronic device detects a click operation on the icon 310 in the video icon; for another example, as shown in (d) of Figure 7 , the electronic device detects a click operation on the icon 310 in the video icon. Click operation of icon 312.

Optionally, the first operation on the N video images in the video icon may be operations performed one after another, or may be performed simultaneously.

It should be understood that the above example takes the first operation as a click operation. The first operation may also be an operation of selecting N video icons among the video icons indicated by voice, or the first operation may also be other instructions for selecting video icons. This application does not impose any restrictions on the operation of N video icons.

Step S430: In response to the first operation, obtain information of N videos.

Among them, N is an integer greater than 1.

For example, as shown in (b) of FIG. 8 , based on the first operation, the electronic device can obtain information of three videos.

Step S440: Based on the information of the N videos, obtain the video themes of the N videos.

It should be understood that the video theme can refer to the theme idea associated with the overall image content in the video; for different video themes, the corresponding video processing methods can be different; for example, different video themes can use different music and different transitions. Special effects, different image processing filters, or different video editing methods can be used.

Optionally, in a possible implementation, based on the information of N videos, the video topics of N videos are obtained, including:

Convert the video content of N videos into N text description information. The N text description information corresponds to the N videos one-to-one. One text description information among the N text description information is used to describe the image of one video among the N videos. Content information; based on N pieces of text description information, the subject information of N videos is obtained, and the text description information is used to convert the video content in the N videos into text information.

In the embodiment of the present application, when identifying the video themes of N videos, the video theme information corresponding to the N videos is obtained through the text description information of the N videos; that is, the N videos can be obtained based on the text description information of the N videos The overall video topic information; compared with the video topic information based on the image semantics of N videos, text information has more abstract semantic information than image information, and there is language correlation between multiple text information, which helps to infer multiple The theme information hidden behind the text can improve the accuracy of the overall video theme corresponding to the N video.

For example, the N videos include videos of the user packing luggage, videos of the user taking a car to the airport, videos of the user taking a plane, and videos of the user walking on the beach; only some image tags may be obtained based on image semantics, including clothing, Suitcases, users, seaside, etc. Based on these image tags, it is impossible to abstract that the video theme of N videos is travel; however, when identifying the video theme based on the text description information of N videos, the video theme can be identified based on the N video text description information and N The linguistic logic correlation between video text description information can accurately obtain the video theme information of N videos; for example, based on the text description information included in N videos, "a user is packing luggage", "a user is taking a plane", "A user is walking on the beach." Based on this text description information, the video theme information of N videos can be abstracted as travel; therefore, the video theme information of N videos can be obtained through the text description information of N videos, which can improve the theme information. accuracy.

Optionally, in a possible implementation, based on N pieces of text description information, the topic information of N videos is obtained, including:

Optionally, N videos can be input to the image-text conversion model to obtain text description information of N videos; for example, N text description information; the text description information of N videos can be input to the pre-trained topic classification model, Get the topic information of N videos. Optionally, the implementation may refer to step S530 in FIG. 13 or the related description of step S620 and step S630 in FIG. 18 .

Optionally, in a possible implementation, when the pre-trained topic classification model outputs at least two video topics, the at least two video topics correspond to N pieces of text description information, which also includes:

A second action on at least two video subjects is detected;

Optionally, if the theme information output in step S440 is one theme information, no user operation is required; if the theme information output in step S440 is two or more theme information, a prompt box can be displayed in the electronic device; the prompt box can Including candidate video topic information, the video topic information of N videos is determined based on the user's operation on the candidate video topic information in the prompt box.

For example, if two theme information are output in step S440, the second interface can be displayed on the electronic device, such as the display interface 319 shown in Figure 11; the display interface 319 includes a prompt box 320, and the prompt box 320 includes two The candidate video topic information is scenery and travel respectively. If the electronic device detects that the user clicks "Landscape", then the video topic information of the N videos is scenery; if the electronic device detects that the user clicks "Travel", then the video topic information of the N videos is scenery. The theme information is travel.

Step S450: Based on the similarity between the images in the N videos and the video themes, select M video clips from the N videos.

For example, the similarity between the images in N videos and the video topic can be represented by a similarity confidence value or a distance value; for example, if the similarity between an image feature in a video and the text feature of the video topic is greater than is high, the greater the similarity confidence value and the smaller the distance measurement value; if the similarity between an image feature in a video and the text feature of the video subject is lower, the smaller the similarity confidence value and the smaller the distance measurement value. big.

In the embodiment of the present application, based on the correlation between the images in the N videos and the video theme information, M video clips among the N videos that are highly relevant to the video theme can be determined; based on the solution of the present application, it can effectively Delete the video clips in the N videos that are not related to the video topic information to ensure that the filtered video clips are related to the video topic information; on the other hand, the similarity confidence value of some or all image features in the N videos and the video topic information can be calculated , by using video clips obtained by selecting consecutive multiple frames of images in a video, so the continuity of the video clips is better.

In one example, for each of N videos, all image features in a video can be traversed, and the similarity between each image feature in a video and the text information of the video topic information can be determined.

In one example, for each of N videos, part of the image features in a video can be extracted; for example, for one of the N videos, image frames can be selected at equal intervals, and the selected image frames can be characterized. extract Get image features.

Optionally, in the embodiment of the present application, M may be greater than N, or M may be equal to N, or M may be less than N; the numerical size of M is based on each video segment and video theme information in N videos. The similarity confidence value is determined.

It should be understood that in the solution of this application, if the similarity confidence values of all images in a video and the video topic information are less than or equal to the preset threshold, it means that the video has nothing to do with the video topic information, and the video does not need to be retained. Any video clip; if the similarity confidence value of some or all images in a video and the video topic information is greater than the preset threshold, some or all video clips in the video can be retained.

Optionally, in a possible implementation, M video clips from the N videos are selected based on the similarity between the images in the N videos and the video themes, including:

Optionally, in a possible implementation, the pre-trained similarity matching model is a Transformer model.

Optionally, in a possible implementation, the pre-trained similarity matching model is obtained through the following training method:

Optionally, the implementation of step S450 can refer to the following steps S540 and S550 in Figure 13, or Figure 14, or Figure 15, or Step S640 and Step S650 in Figure 18, or Step S750 and Step S750 in Figure 19 Related description of S760.

Step S460: Based on the video theme, obtain music that matches the video theme.

Optionally, in a possible implementation, based on the video theme, music matching the video theme is obtained, including:

For example, the total duration of the background music can be determined based on the duration of the M video clips. The background music usually selected when performing music matching needs to be greater than or equal to the total duration of the M video clips; based on the video theme information, the total duration of the background music can be determined. music style.

Optionally, the implementation of step S460 may refer to the subsequent step S560 in FIG. 13 , or step S660 in FIG. 18 , or the related description of step S770 in FIG. 19 .

Step S470: Obtain the first video based on the M video clips and music.

Optionally, in a possible implementation, the first video is obtained based on M video clips and music, including:

Sort the M video clips to obtain the sorted M video clips;

The sorted M video clips and music are synthesized into the first video.

In the embodiment of the present application, the image content in the M video clips can be made more consistent with the music rhythm in the music; for example, if the video image content is scenery, it can correspond to the prelude of the music or the soothing music part; the video image If the content is the user's sports scene, it can correspond to the climax of the background music; by sorting the M video clips, the M video clips can better match the rhythm stuck points of the music; thereby solving the problem in the edited first video The existing problem of video clips and background music not matching can solve the problem of the edited first video content not fully matching the rhythm of the music; improve the video quality of the edited first video.

Optionally, in a possible implementation, the M video clips are sorted to obtain the sorted M video clips, including:

For example, for a video without a strong story line, the best positions of M video clips can be matched based on the rhythm of the music; a processed video can be generated. Optionally, for the implementation method, please refer to the relevant description in the subsequent Figure 18.

It should be understood that a video without a strong storyline may refer to a video in which N videos are in equal order; there is no strong causal correlation between the N videos; for example, a video without a strong storyline may include a video with a sports theme.

For example, in the solution of this application, background music can be selected based on the overall video theme information of N videos; and M videos can be sorted based on the rhythm of the background music, so that M video clips can be sorted according to the rhythm of the background music. The video is sorted so that the picture content of the video clips matches the rhythm of the music; compared with matching the video directly with the music according to the input order, the solution of this application can improve the consistency of the image content in the video and the rhythm of the background music, and improve the quality of the edited video. Video quality.

Optionally, in a possible implementation, the M video clips are sorted based on the rhythm of the music to obtain the sorted M video clips, including:

It should be noted that the above implementation method can be referred to the subsequent relevant descriptions of FIG. 16 or FIG. 17 .

In the embodiment of the present application, music and M video clips are input to a pre-trained audio-visual rhythm matching model to obtain sorted M video clips; through the pre-trained audio-visual rhythm matching model, the relationship between audio features and video features can be realized match.

Optionally, in a possible implementation, M video clips are sorted to obtain sorted M video clips, including:

For example, for a video with a strong story line, the sorted M video segments can be determined based on the similarity confidence values between the video segments included in the sorted N videos and the video theme; based on the sorted M video segments and the video theme The information determines background music matching the sorted M video clips; generates a processed video.

For example, in the solution of this application, for N videos with strong story lines, the N videos can be sorted based on the text description information of the N videos to obtain the sorted N videos; from the sorted N videos Select M video clips that are highly relevant to the video topic information to obtain the sorted M video clips; based on the sorted M video clips and video theme information, determine the background music that matches the sorted M video clips; make the picture content of the N videos with strong storylines match the music rhythm, and the picture content of the videos is played in order Comply with the causal relationship and improve the video quality of the edited video.

Step S480: Display the first video.

For example, the first video may be a mixed-cut video obtained based on M video clips and music; the mixed-cut video may be displayed on the electronic device.

Optionally, in a possible implementation, after generating the first video based on M video clips and music, the electronic device can save the first video; after the electronic device detects an operation indicating to display the first video, display First video.

It should be understood that the above is exemplified by editing N videos; the solution of this application can also be applied to photos saved in electronic devices; for example, photos can include but are not limited to: gif animations, JPEG format images, PNG format images, etc. .

In the embodiment of the present application, the image content information of N videos can be converted into text description information; the video theme information of N videos can be obtained based on the text description information of N videos; based on the images and video themes in N videos According to the relevance of the information, M video clips are selected from N videos; the processed video is obtained based on the M video clips and background music; in the solution of this application, N videos are obtained through the text description information of the N videos video topic information; compared with the video topic information of N videos based on the image information of N videos, text information has richer information than image information; in addition, there is language correlation between multiple text information, based on N The video theme information of the video can be obtained from the text description information of the videos, which can improve the accuracy of the video theme information; in addition, in the embodiment of the present application, N can be determined based on the correlation between the images in the N videos and the video theme information. M video clips in the videos that are highly relevant to the video topic; based on the solution of this application, the video clips in the N videos that are irrelevant to the overall video topic information can be effectively deleted, ensuring that the filtered video clips are related to the video topic information. Improve the video quality of edited videos.

In addition, in the embodiments of the present application, the problem of mismatch between the video clips and the background music existing in the edited first video can be solved, that is, the rhythm lag between the image content of the edited first video and the background music can be solved. The problem of incomplete matching; based on the rhythm of background music or the logical correlation of video clips, multiple video clips can be reasonably connected in series; thereby improving the video quality of the edited video.

Figure 13 is a schematic flow chart of a video editing method provided by an embodiment of the present application. The video editing method 500 can be executed by the electronic device shown in Figure 1; the video editing method includes steps S510 to S570, and steps S510 to S570 are described in detail below respectively.

Step S510: Obtain N videos.

For example, the N videos may be videos stored in the electronic device; wherein the N videos may be videos collected by the electronic device; or some or all of the N videos may be downloaded videos; this application applies to the N videos The source of the video is not limited in any way.

For example, the electronic device detects the user's click operation on N videos in the gallery application; the N videos are obtained.

Step S520: Obtain text description information of N videos.

It should be understood that a video can correspond to a text description information, and the text description information is used to describe the content information in a video; the image content in the video can be converted into text description information through the text description information.

It should be noted that the text description information is used to describe the image content in a video, and the text description information and the subtitle content in the video may be different.

For example, if Video 1 is a video of a user packing his luggage, the text description information of Video 1 may be "A person is packing his luggage"; Video 2 is a video of the user taking a plane at the airport, then the text description information of Video 2 may be "A person is taking a plane"; Video 3 is a video of the user walking on the beach, then the text description information of Video 3 can be "A person is taking a plane." People walking on the beach."

Step S530: Obtain the video theme information of the N videos based on the text description information of the N videos.

It should be noted that in the embodiment of the present application, the video theme information of N videos is one theme information, that is, the video theme information is the video theme information corresponding to the N videos as a whole.

For example, video themes may include but are not limited to: travel, parties, pets, sports, scenery, parent-child, work, etc.

Optionally, the text description information of N videos can be input to a pre-trained video topic classification model to obtain the video topic information of N videos; wherein, the pre-trained video topic classification model can output video topic tags.

For example, the pre-trained video topic classification model may refer to a text classification model. The pre-trained video topic classification model may be used to classify input text description information and obtain classification labels corresponding to the text description information.

For example, the pre-trained video topic classification model can be a neural network; for example, the pre-trained video topic classification model can be a deep neural network.

Optionally, the pre-trained video topic classification model can be trained through the back propagation algorithm based on the following training data set; the training data set includes sample text description information and video topic text information, and the sample text description information is related to the video topic information. Corresponding; among them, the sample text description information can be one or more sentence texts; the video topic text information can be a phrase text; the video topic classification model to be trained can obtain the trained video topic by learning a large number of training data sets Classification model.

For example, the sample text description information may include: "Multiple people are eating", "Multiple people are playing games", and "Multiple people are talking"; the video theme text information corresponding to the sample text description information may be "Party"; and For example, the sample text description information may include "an adult and a child are taking pictures", "an adult and a child are playing games"; the video theme information corresponding to the sample text description information is "parent-child".

It should be understood that the above is an example; the embodiment of the present application does not place any limitations on the sample text description information and the sample video theme information.

Optionally, if the theme information output in step S530 is one video theme information, no user operation is required; if the output in step S530 is two or more video theme information, a prompt box can be displayed on the electronic device; prompt box Candidate video theme information may be included. Based on the user's operation on the candidate video theme information in the prompt box, the electronic device determines the video theme information of N videos.

In one possible implementation, if the confidence thresholds of multiple text description information corresponding to each video theme information are all small, it may be that the input text description information does not fully conform to a certain video theme; at this time, the electronic device can Candidate video topic information is displayed in the video topic information, and the video topic information corresponding to multiple text description information is determined based on the user's operation.

For example, as shown in Figure 11, if two video theme information are output in step S530, the display interface 319 can be displayed on the electronic device; the display interface 319 includes a prompt box 320, and the prompt box 320 includes two candidate video themes. The information is scenery and travel respectively. If the electronic device detects that the user clicks "Landscape", then the video theme information of the N videos is scenery; if the electronic device detects that the user clicks "Travel", then the video theme information of the N videos is travel. .

Step S540: Based on the similarity between the images in the N videos and the video topic information, obtain the similarity confidence values of the images in the N videos and the video topic information.

Optionally, the similarity between the image features in the N videos and the text features of the video topic information can be obtained based on the similarity evaluation model, and the similarity confidence values between the image features in the N videos and the video topic information can be obtained. Optionally, for the implementation method, refer to the subsequent descriptions of FIG. 14 and FIG. 15 .

In one example, for each of N videos, part of the image features in a video can be extracted; for example, for one of the N videos, image frames can be selected at equal intervals, and the selected image frames can be characterized. Extract image features.

For example, you can extract the image features of one frame every 4 frames, and you can select the 1st frame, 5th frame, 10th frame, 15th frame, etc. in a video.

Optionally, the image features in the N videos and the text features of the video topic information can be extracted based on the similarity evaluation model, and the similarity between the image features in the N videos and the text features of the video topic information can be evaluated, and N The similarity confidence value between the image features in the video and the video topic information; the specific implementation method is as described in the subsequent figures 14 and 15.

Step S550: Obtain M video clips from the N videos based on the similarity confidence values between the images in the N videos and the video topic information.

For example, as shown in Figure 15, assume that N videos include video 310, video 312 and video 314; curve 561 is the similarity curve between the image features included in video 310 and the text features of the video topic information; curve 562 is the similarity curve between the image features included in the video 312 and the text features of the video topic information; the curve 563 is the similarity curve between the image features included in the video 314 and the text features of the video topic information; based on the curve 561, It is determined to select image 3101 and image 3102 in video 310 to form video segment 1; based on curve 562, it can be determined to select image 3121, image 3122 and image 3123 in video 312 to form video segment 2; based on curve 563, it can be determined to select the image in video 314 3141, image 3142, image 3143 and image 3144 constitute video clip 3.

It should be understood that Figure 15 is an example, and two or more video clips can also be selected from one video. The two video clips can be two consecutive video clips, or they can also be two discontinuous video clips. Video clips (for example, frames 1 to 5 make up a video clip; frames 10 to 13 make up a video clip); however, for a video clip, the multiple frames included in the video clip are continuous Multi-frame images; alternatively, you can not select any video clip from a video; whether to select a video clip from a video depends on the similarity confidence value between the image features included in the video and the video topic information; if a video If there are no image features related to the video theme, the video clips in the video do not need to be selected.

In the embodiment of the present application, based on the correlation between the images in the N videos and the video theme information, M video clips among the N videos that are highly relevant to the video theme can be determined; based on the solution of the present application, it can effectively Delete the video clips in the N videos that are not related to the video topic information to ensure that the filtered video clips are related to the video topic information; on the other hand, the similarity confidence value of some or all images in the N videos and the video topic information can be calculated , by using video clips obtained by selecting consecutive multiple frames of images in a video, so the continuity of the video clips is better.

Optionally, the original sounds in some or all of the M video clips can be retained.

Step S560: Perform music matching processing based on M video clips and video theme information to obtain background music.

It should be understood that the background music obtained in step S560 may refer to the music in step S460 of Figure 12 .

For example, if the video theme is a party, the background music can be a cheerful music style; if the video theme is landscape, the background music can be a soothing music style.

It should be understood that the above is an example, and this application does not impose any restrictions on the music style of the video theme and background music.

Optionally, music matching processing can be performed in the candidate music library based on the M video clips and video theme information to obtain background music information; the candidate music library can include music of different music styles and music durations.

For example, the total duration of the background music can be determined based on the duration of M video clips; the music style of the background music can be determined based on the video theme information; based on the total duration and the music style, the candidate music of the music style in the candidate music library can be determined Choose randomly and get background music.

For example, the total duration of the background music can be determined based on the duration of M video clips; the music style of the background music can be determined based on the video theme information; based on the total duration and music style, selection can be made according to the music popularity in the candidate music library, and we get Background music.

For example, the total duration of the background music can be determined based on the duration of M video clips; the music style of the background music can be determined based on the video theme information; based on the total duration and music style, selection can be made based on the user's preferences in the candidate music library, Get background music.

For example, background music that satisfies the total duration and music style is selected from the candidate music library based on the frequency with which the user plays music.

For example, the total duration of the background music can be determined based on the duration of M video clips; the music style of the background music can be determined based on the video theme information; the music with the highest matching degree to the video theme can be selected from the candidate music library as the background music.

For example, the total duration of the background music can be determined based on the duration of M video clips; the music style of the background music can be determined based on the video theme information; multiple music can be selected from the candidate music library for editing to obtain the background music; where, multiple The weight or duration of each piece of music can be based on the user's preferences or preset fixed parameters.

It should be understood that the above description is an example, and this application does not impose any limitations on the specific implementation of the music matching process.

Step S570: Match M video clips and background music to obtain a processed video (an example of the first video).

For example, the order of the M video clips can be determined based on the music rhythm of the background music, so that the picture content and the music rhythm between the M video clips and the background music are consistent.

It should be understood that the rhythm matching process is to achieve a better integration of the M video clips and the background music, so that the image content in the M video clips is more consistent with the music rhythm in the background music; for example, if the video image content is scenery, then It can correspond to the prelude or soothing music part of the background music; the video image content is the user's sports scene, so it can correspond to the climax part of the background music; through rhythm matching processing, the rhythm of the M video clips and the background music is stuck The points are more closely matched and the quality of the processed video is improved.

Optionally, M video clips and background music can be input to a pre-trained audio-visual rhythm matching model to obtain position information of all or part of the M video clips; wherein, the audio-visual rhythm matching model can include an audio encoder. , video encoder and similarity measurement module; among them, the audio encoder is used to extract audio features of background music; the video encoder can be used to extract video features; the similarity measurement module is used to measure the similarity between audio features and video features ; Optionally, for implementation, please refer to the subsequent descriptions of Figure 16 and Figure 17 .

It should be noted that in the implementation of this application, the network of the audio-visual rhythm matching model can be a neural deep network; for example, the audio-visual rhythm matching model can adopt the structure of the Transformer model as shown in Figure 2; when training the audio-visual rhythm matching model, Comparative learning training methods can be used.

For example, the audio-visual rhythm matching model can be a neural network, and the audio-visual rhythm matching model to be trained can be trained by obtaining sample music clips to obtain a trained audio-visual rhythm matching model. For example, the overall training architecture of the audio-visual rhythm matching model can use a contrastive learning model; when constructing training data pairs, data pairs that match background music and video content can be used as positive examples, and data pairs that do not match background music and video content can be used as negative examples. For example, train a video encoder and an audio encoder so that the similarity of positive data pairs is greater than the similarity of negative data pairs.

It should be understood that the audio-visual rhythm matching model can have a multi-modal pre-training architecture and can simultaneously support two different types of input data: images and text; text and images are mapped into a unified space through a cross-modal contrastive learning method, thereby improving Visual and text comprehension skills.

In the embodiment of the present application, by performing rhythm matching processing on the M video clips and the background music, the M video clips can be sorted according to the rhythm of the background music, so that the picture content of the video clips matches the music rhythm. That is to say, the image content and background music of M video clips are stuck. Compared with matching the video with the music directly according to the input order, the solution of this application can improve the consistency of the image content in the video and the rhythm of the background music, and improve the user experience.

Optionally, since the duration of the background music is greater than or equal to the total duration of the M video clips; when the duration of the background music is greater than the duration of the M video clips, the last video clip among the M video clips can be played in slow motion. ; Or, add transition effects so that M video clips can be played repeatedly after the M video contents are played.

Optionally, in step S570, the processed video can also be obtained based on the uploading order of the M video clips, or the order of the timestamp information of the M video clips and the background music.

It should be noted that the above is an example of editing N videos included in the gallery application; the solution of this application can also be applied to editing photos in the gallery application; for example, the photos can include but are not limited to: gif animations. Pictures, JPEG format images, PNG format images, etc.

In the embodiment of the present application, the image content information of N videos can be converted into text description information; the video theme information of N videos can be obtained based on the text description information of N videos; based on the images and video themes in N videos According to the relevance of the information, M video clips are selected from N videos; the processed video is obtained based on the M video clips and background music; in the solution of this application, N videos are obtained through the text description information of the N videos video topic information; compared with the video topic information of N videos based on the image information of N videos, text information has richer information than image information; in addition, there is language correlation between multiple text information, based on N The text description information of each video is obtained The video theme information of the video can improve the accuracy of the video theme information; in addition, in the embodiment of the present application, based on the correlation between the images in the N videos and the video theme information, it can be determined that the N videos are related to the video theme. M video clips with high degree of accuracy; based on the solution of this application, on the one hand, it can effectively delete the video clips that are irrelevant to the video topic information in N videos, ensuring that the filtered video clips are related to the video topic information; on the other hand, when calculating When calculating the similarity confidence value between each video clip in N videos and the video topic information, the video clip is obtained by selecting consecutive multi-frame images in a video, so the continuity of the video clip is better; thereby improving the quality of the edited video. Video quality.

Further, in the embodiment of the present application, the background music of M videos is selected based on the video theme information of N videos; and the M videos can be sorted based on the rhythm of the background music, so that the M videos can be sorted according to the rhythm of the background music. The video clips are sorted so that the picture content of the video clips matches the rhythm of the music; compared with matching the video directly with the music according to the input order, the solution of this application can improve the consistency of the image content in the video and the rhythm of the background music, and improve the user experience. experience.

For example, the implementation of step S540 and step S550 in FIG. 13 will be described in detail below in conjunction with FIG. 14 and FIG. 15 .

FIG. 14 is a schematic flowchart of a method for determining M video segments related to video theme information among N videos provided by an embodiment of the present application. The method can be executed by the electronic device shown in Figure 1; the method includes steps S551 to S555, and steps S551 to S555 will be described in detail below.

Step S551: Extract features from N videos based on the image encoder in the similarity evaluation model to obtain image features in N videos.

Optionally, in the embodiment of the present application, a training data set can be obtained to train the similarity evaluation model to be trained, and a trained similarity evaluation model can be obtained; for example, the overall training architecture of the similarity evaluation model can adopt a contrastive learning model. ; When constructing training data pairs, you can use data pairs where the text description information matches the video topic information as positive examples, and use data pairs where the text description information does not match the video topic information as negative examples to train the image encoder and text encoder. Make the similarity of positive data pairs greater than the similarity of negative data pairs.

For example, the training data set includes sample videos, video theme information that matches the sample videos, and video theme information that does not match the sample videos; for example, the sample videos can include travel videos, and the video theme information of the sample videos is "travel". Text information; the video topic information that does not match the video topic of the sample video is the text information of "motion"; through a large number of training data sets, the similarity evaluation model can identify matching text features and image features, for example, making the input consistent When text features and image features are matched, the distance measurement value output by the similarity measurement module in the similarity evaluation model to be trained is smaller; so that when text features and image features are input that do not match, the similarity in the similarity evaluation model to be trained is smaller. The greater the distance measurement value output by the degree measurement module; or, the greater the similarity confidence value output by the similarity measurement module in the similarity evaluation model to be trained when inputting matching text features and image features; making the input mismatch When the text features and image features are different, the similarity confidence value output by the similarity measurement module in the similarity evaluation model to be trained is smaller.

It should be understood that the trained similarity evaluation model can identify matching text features and image features.

Optionally, image features can be extracted from each frame of the N videos through the image encoder in the similarity evaluation model to obtain all image features included in the N videos.

Optionally, the image features in the N videos can be extracted through the image encoder in the similarity evaluation model based on the same number of interval frames to obtain partial image features in the N videos.

For example, you can extract the image features of one frame at an interval of 4 frames, then you can extract the first frame of one of N videos. Like, the 5th frame image, the 10th frame image, the 15th frame image, etc.

It should be understood that the above is an example. In the embodiment of the present application, for one video among N videos, all image features in a video can be extracted by traversing each frame of the image; or, frames can be equally spaced. Number, extract some image features in a video; this application does not make any restrictions on this.

Optionally, the similarity evaluation model can be as shown in Figure 15. The similarity evaluation model can include a text encoder, an image encoder and a similarity measurement module (an example of the first similarity measurement module); where, the text encoding The encoder is used to extract text features; the image encoder can be used to extract image features; the similarity measurement module is used to measure the similarity between text features and image features.

For example, the similarity evaluation model may be a contrastive learning model.

Step S552: Extract features from the video topic information based on the text encoder in the similarity evaluation model to obtain text features of the video topic information.

It should be understood that text features refer to a set of attributes obtained by vectorization and subsequent mapping of words or sentences that can characterize their specific semantics.

Step S553: Obtain the similarity confidence value between the image feature and the text feature based on the similarity measurement module in the similarity evaluation model.

For example, based on the similarity evaluation model, the image features in N videos and the text features of the video topic information can be extracted; the image features and text features are compared to obtain the similarity between the image features and the text features. Among them, the similarity evaluation model can output a distance measurement value, or the similarity evaluation model can output a similarity confidence value; if the similarity evaluation model outputs a distance measurement value, the smaller the distance measurement value indicates the difference between the image feature and the text feature. The higher the similarity; the similarity confidence value between the image feature and the text feature can be obtained based on the distance measurement value; if the similarity evaluation model outputs the similarity confidence value, the greater the similarity confidence value, the greater the similarity between the image feature and the text feature. The higher the similarity between them.

For example, the distance measurement value can be the cos value between the image feature and the text feature.

Step S554: Select consecutive multi-frame image features in the video based on the similarity confidence value of the image feature and the text feature to obtain a video clip.

For example, as shown in Figure 15, for a video, the similarity curve of the image features in the video and the text features of the video topic information can be obtained; one or more video clips can be selected from a video based on the similarity curve, and a A video clip consists of multiple consecutive frames of images.

In the embodiment of the present application, multiple consecutive frames of images related to the video theme are selected to obtain a video clip; based on the solution in the present application, it can be ensured that the selected video clip is related to the overall video theme.

Step S555: Based on the similarity confidence value of the image feature and the text feature, select M video clips from the N videos.

For example, for a video, the text feature similarity curve between the image features in the video and the video topic information can be obtained; based on the similarity curve, the images in the video related to the video topic can be determined from the entire video; then from a video Extract consecutive multiple frames of images to obtain a video clip.

For example, as shown in Figure 15, assume that N videos are video 310, video 312 and video 314; curve 561 is the similarity curve between the image features included in video 310 and the text features of the video topic information; curve 562 is the similarity curve between the image features included in the video 312 and the text features of the video topic information; the curve 563 is the similarity curve between the image features included in the video 314 and the text features of the video topic information; based on the curve 561, It is determined to select image 3101 and image 3102 in video 310 to form video segment 1; based on curve 562, it can be determined to select image 3121, image 3122 and image 3123 in video 312 to form video segment 2; based on curve 563, it can be determined to select the image in video 314 3141, image 3142, image 3143 and image 3144 constitute video clip 3.

It should be understood that Figure 15 is an example, and two or more video clips can also be selected from one video, where the two video clips can be two consecutive video clips or they can also be two discontinuous videos. Segments (for example, frames 1 to 5 constitute a video segment; frames 10 to 13 constitute a video segment); however, for a video segment, the multiple frames of images included in the video segment are consecutive multiple frame image; alternatively, you can not select any video clip from a video; whether to select a video clip from a video depends on the similarity confidence value of the image features in the video and the video topic information; if there is no similarity in a video with If the image features related to the video theme are not selected, the video clips in the video may not be selected.

In the embodiment of the present application, the image features related to the overall video theme in N videos can be identified through the pre-trained similarity evaluation model; based on the image features related to the video theme, N videos are filtered out that are related to the video theme. M video clips, and the video clips in the N videos that are not related to the video theme are eliminated; based on the solution of this application, on the one hand, the video clips in the N videos that are not related to the video topic information can be effectively deleted to ensure that the filtered video clips Related to the video topic information; the edited video is obtained based on the filtered video clips and background music, thereby improving the video quality of the edited video.

Exemplarily, the implementation of step S570 in Figure 13 will be described in detail below with reference to Figures 16 and 17 .

Figure 16 is a flow chart of a method for matching M video clips and background music provided by an embodiment of the present application. The method can be executed by the electronic device shown in Figure 1; the method includes steps S571 to S574, and steps S571 to S574 are described in detail below respectively.

Step S571: Extract features of the background music based on the audio encoder in the audio-visual rhythm matching model to obtain audio features.

For example, the audio-visual rhythm matching model can be as shown in Figure 17; the audio-visual rhythm matching model can include an audio encoder, a video encoder and a similarity measurement module; wherein the audio encoder is used to extract the audio features of the background music; video encoding The module can be used to extract video features; the similarity measurement module is used to measure the similarity between audio features and video features.

Step S572: Extract features from the M video clips based on the video encoder in the audio-visual rhythm matching model to obtain video features.

It should be understood that one video feature includes multi-frame image features; M video segments can correspond to M video features.

Step S573: Obtain the similarity confidence value between the audio feature and the video feature based on the similarity measurement module in the audio-visual rhythm matching model.

For example, the background music can be divided into multiple audio features; by traversing the correlation between each video feature in the M video features and each audio feature in the multiple audio features, the correlation between each video feature in the M video features and multiple audio features is obtained. The audio feature with the highest similarity among the audio features; based on the positions of multiple audio features in the overall background music, the ranking of the video clips corresponding to the audio features among the M video clips can be determined.

For example, based on the similarity measurement module in the audio-visual rhythm matching model, the distance measurement value between the audio feature and the video feature can be output; the larger the distance measurement value, the smaller the similarity between the audio feature and the video feature, then the audio The lower the correlation between features and video features, the smaller the similarity confidence value; the smaller the distance measurement value, the higher the similarity between audio features and video features, the greater the correlation between audio features and video features. The higher the similarity, the greater the confidence value. For example, The distance measurement value can be the cos value between the audio feature and the video feature.

Step S574: Based on the similarity confidence value, obtain the best matching positions corresponding to the background music of the M video clips.

For example, based on the similarity confidence value, the best positions for matching the M video clips and the background music can be obtained, so that the image content of the M video clips matches the music rhythm of the background music.

For example, M video clips include video clip 1, video clip 2 and video clip 3; the background music can be divided into 3 audio features, namely audio feature 1, audio feature 2 and audio feature 3; respectively determine the audio features 1 and 3. The correlation between video clip 1, video clip 2 and video clip 3 is used to obtain the audio feature that has the highest matching degree with audio feature 1 among the three video clips; judge whether audio feature 2 is related to video clip 1, video clip 2 and video clip 3, get the audio feature with the highest matching degree with audio feature 2 among the 3 video clips; judge the correlation between audio feature 3 and video clip 1, video clip 2 and video clip 3, get 3 videos The audio feature in the clip that has the highest matching degree with audio feature 3; finally, the video clip corresponding to each audio feature can be output.

For example, as shown in Figure 17, assuming that M video clips are 3 video clips, the audio-visual rhythm matching model can output audio feature 1 corresponding to video clip 3, audio feature 2 corresponding to video clip 2, and audio feature 3 corresponding to video clip 1; thus Obtain a sequence of M video clips that match the rhythm of the background audio.

Optionally, if the video image content is scenery, it may correspond to the prelude or soothing music part of the background music; if the video image content is a user movement scene, it may correspond to the climax part of the background music.

Optionally, in the embodiment of the present application, a training data set can be obtained to train the audio-visual rhythm matching model to be trained, and a trained audio-visual rhythm matching model can be obtained; wherein, the training data set includes sample matching music shorts and sample non-matching music Short film; sample-matching music short refers to a music short whose music matches the image content; sample-unmatched music short refers to a music short whose music and image content do not match; for example, the background music of music short 1 and the image of music short 2 The videos are mixed to obtain sample-unmatched music clips; by learning a large number of training data sets, the audio-visual rhythm matching model can sort the input M video clips based on the rhythm of the input background music.

In the embodiment of the present application, by sorting the M video clips through the audio-visual rhythm matching model, the M video clips can be sorted according to the rhythm of the background music, so that the picture content of the video clips matches the music rhythm; and Compared with matching videos directly with music according to the input sequence, the solution of this application can improve the consistency between the image content in the video and the rhythm of the background music, and improve the user experience.

Optionally, in the embodiment of the present application, the electronic device detects N videos selected by the user. The N videos may refer to videos with strong story lines; or, the N video contents may be videos without strong story lines. ; The non-storytelling video editing method and the strong story-telling video editing method will be described in detail below with reference to Figures 18 and 19.

It should be understood that a video with a strong story line can refer to a causal connection between N videos. Based on the video editing method, the cause and effect between the N videos can be identified and the N videos can be sorted based on the order of the cause and effect; for example, a strong story line Videos can include travel-themed videos or travel-themed videos; videos with non-strong story lines can refer to videos in which N videos are in equal order; there is no strong causal correlation between N videos; for example, non-strong story lines The videos can include sports themed videos.

For example, videos with strong story lines can include videos with a video theme of travel; for example, the N videos include videos of packing luggage at home; videos of taking a taxi to the airport; videos of taking a plane; arriving at the destination and walking on the beach. videos; then these 4 videos are causally connected. You need to pack your luggage first and then take a man-machine to reach the destination and travel at the destination.

For example, videos with non-strong story lines may include videos with a video theme of sports; for example, the N videos include videos of running on the basketball court; videos of layups and shots; videos of passing the ball on the basketball court; then this The three videos do not have a strong causal relationship. For a ball game, there can be multiple layups, passes on the basketball court, and shots on the basketball court. During the running process, there is no unique ordering requirement for the ordering of the three videos.

Implementation method 1: For videos without strong story lines, obtain N videos; obtain the video themes of N videos based on the text description information of the N videos; determine N based on the similarity confidence values between the images in the N videos and the video themes M video clips in the videos; determine background music based on the M video clips and video themes; match the best positions of the M video clips based on the rhythm of the background music; generate a processed video.

Figure 18 is a schematic flow chart of a video editing method provided by an embodiment of the present application. The video editing method 600 can be executed by the electronic device shown in FIG. 1; the video editing method 600 includes steps S610 to S680, and steps S610 to S680 are described in detail below respectively.

Step S610: Obtain N videos.

For example, the electronic device detects the user's click operation on N videos in the gallery application; the N videos can be obtained.

Optionally, the sorting of the N videos that do not have a strong story line can be based on the order in which the N videos are uploaded; or, it can be based on the timestamp information of the video (for example, recording a video or downloading a video time information), sort N videos.

Step S620: Obtain the text description information of N videos through the image-text conversion model.

For example, one video can correspond to one piece of text description information; N pieces of text description information can be obtained from N videos through the image-to-text conversion model.

Optionally, the image-to-text conversion model can be used to convert the video into text information; that is, the image information included in the video can be converted into text description information; and the image content included in the image can be described based on the text description information.

For example, the image-to-text conversion model may include a CLIP model.

Step S630: Input the text description information of N videos into the pre-trained video topic classification model to obtain video topic information.

For example, the pre-trained video topic classification model may be a pre-trained text classification model, and the text classification model may be a deep neural network.

Optionally, the video topic classification model can be obtained through training based on the following training data set; the training data set includes sample text description information and video topic text information, and the sample text description information corresponds to the video topic information; wherein, the sample text description The information can be one or more sentence texts; the video topic text information can be phrase text.

For example, the sample text description information may include: "Multiple people are eating", "Multiple people are playing games", and "Multiple people are talking"; the video theme text information corresponding to the sample description text may be "Party"; and for example , the sample text description information may include "an adult and a child are taking pictures", "an adult and a child are playing games"; the video theme information corresponding to the sample description text is "parent-child".

For example, inputting a video to the image-to-text conversion model can obtain a text description information; N videos can Obtain N pieces of text description information; input the N pieces of text description information into the pre-trained video topic classification model, and obtain the video topic information corresponding to the N pieces of text description information; where the video topic information can include but is not limited to: travel, party , pets, sports, scenery, parent-child, work, etc. In the embodiment of the present application, when identifying the video theme information of N videos, the video theme information of N videos is obtained through the text description information of N videos; and the video theme information of N videos is obtained based on the image information of N videos. Compared with topic information, text information has richer information than image information; in addition, there is language correlation between multiple text information. The video topic information of the video can be obtained based on the text description information of N videos, which can improve the accuracy of the topic information. sex; for example, the N videos include videos of the user packing luggage, videos of the user going out and taking a car to the airport, videos of the user taking an airplane, and videos of the user's behavior at the beach; only some labels may be obtained based on image information, including clothing , suitcases, users, seaside, etc. Based on these image tags, it is impossible to abstract that the theme of N videos is travel; however, when identifying the theme of N videos based on the text description information of N videos, it can be based on the text description information of N videos. The video theme information of N videos can be accurately obtained based on the language logic correlation with the text description information of N videos; for example, based on the text description information included in N videos, "A user is packing luggage", "A user is taking a ride""Airplane","A user walks on the beach", based on these text description information, the video theme information of N videos can be abstracted as travel; therefore, the video theme information of N videos can be obtained through the text description information of N videos, which can improve Accuracy of subject information.

Optionally, if the theme information output in step S630 is one theme information, no user operation is required; if the theme information output in step S630 is two or more theme information, a prompt box can be displayed in the electronic device; the prompt box can Including candidate video topic information, the video topic information of N videos is determined based on the user's operation on the candidate video topic information in the prompt box.

For example, as shown in Figure 11, if two theme information are output in step S630, the display interface 319 can be displayed on the electronic device; the display interface 319 includes a prompt box 320, and the prompt box 320 includes two candidate video theme information. They are scenery and travel respectively. If the electronic device detects that the user clicks "Landscape", then the video theme information of the N videos is scenery; if the electronic device detects that the user clicks "Travel", then the video theme information of the N videos is travel.

Optionally, for the implementation of step S630, please refer to the relevant description in step S530 in FIG. 13 .

Step S640: Obtain similarity confidence values of image features and video topic information in N videos based on the similarity evaluation model.

It should be understood that the similarity evaluation model may be a pre-trained neural network model; the similarity evaluation model is used to output the correlation between the image features included in each of the N videos and the video topic information. As shown in Figure 15, the similarity evaluation model can include an image encoder, a text encoder and a similarity measurement module; among them, the image encoder is used to extract features from images in the video to obtain image features; the text encoder is used to Feature extraction is performed on video topic information to obtain text information; the similarity measurement module is used to evaluate the similarity between image features and text features.

In the embodiment of the present application, all image features in the N videos can be extracted; or some image features in the N videos can be extracted; this application does not impose any limitation on this.

For example, the image features of one frame can be extracted at intervals of 4 frames, and the 1st frame image, 5th frame image, 10th frame image, 15th frame image, etc. in one video of N videos can be extracted.

Optionally, for the specific description of step S640, please refer to the related description of step S540 in Figure 13; or the related description of step S551 to step S553 in Figure 14; or the related description of Figure 15.

Step S650: Obtain M video clips from the N videos based on the similarity confidence values between the images in the N videos and the video topic information.

For example, based on the similarity confidence value between the image features included in the N videos and the text features of the video topic information, continuous multi-frame image features can be selected from the N videos to obtain a video clip.

For example, as shown in Figure 15, for a video, the similarity curve of the image features in the video and the video theme information can be obtained; based on the similarity curve, multiple consecutive frames of images can be selected from the video to obtain a video clip.

In the embodiments of the present application, multiple consecutive frames of images related to the video theme can be selected from a video to obtain a video clip; based on the solution of the present application, it can be ensured that the selected video clip is related to the overall video theme.

Step S660: Based on the duration and video theme information of the M video clips, perform music matching processing in the candidate music library to obtain background music.

For example, the total duration of the background music can be determined based on the duration of the M video clips. The background music usually selected when performing music matching needs to be greater than or equal to the total duration of the M video clips; the background music can be determined based on the video theme information. style.

For example, if the video theme is a party, the background music will be a cheerful music style; if the video theme is landscape, the background music will be a soothing music style.

For example, the total duration of background music can be determined based on the duration of M video clips; the music style of the background music can be determined based on the video theme information; and the background music can be randomly selected from the candidate music library based on the total duration and music style.

For example, the total duration of the background music can be determined based on the duration of M video clips; the music style of the background music can be determined based on the video theme information; and the background music can be selected from the candidate music library according to music popularity based on the total duration and music style.

For example, the total duration of the background music can be determined based on the duration of M video clips; the music style of the background music can be determined based on the video theme information; and the background music can be selected based on the user's preferences in the candidate music library based on the total duration and music style. .

Step S670: Input M video clips and background music to the pre-trained audio-visual rhythm matching model to obtain sorted M video clips.

For example, the audio-visual rhythm matching model can be a neural network, and the audio-visual rhythm matching model to be trained can be trained by obtaining sample music clips to obtain a trained audio-visual rhythm matching model. For example, as shown in Figure 17, the audio-visual rhythm matching model can include an audio encoder, a video encoder and a similarity measurement module; the audio encoder is used to extract the audio features of background music; the video encoder can be used to extract video features ;The similarity measurement module is used to measure the similarity between audio features and video features.

In one example, the audio-visual rhythm matching model can output a distance metric value, which is used to represent the distance between audio features and video features; the larger the distance metric value, the smaller the similarity between the audio features and video features; The smaller the distance metric value, the greater the similarity between audio features and video features; based on the distance metric value, the confidence value of the similarity between audio features and video features can be obtained.

In one example, the audio-visual rhythm matching model can output a similarity confidence value. The similarity confidence value is used to represent the probability value that the audio feature is similar to the video feature. The larger the similarity confidence value is, the greater the similarity between the audio feature and the video feature. The higher the degree; the larger and smaller the similarity confidence value is, indicating that the similarity between audio features and video features is smaller.

For example, background music and M video clips can be input to a pre-trained audio-visual rhythm matching model, and the audio-visual rhythm matching model can output the sorting of the M video clips; assuming that the M video clips include video clip 1, video clip 2 and video Fragment 3; the background music can be divided into three audio features, namely audio feature 1, audio feature 2 and audio feature 3; determine the correlation between audio feature 1 and video clip 1, video clip 2 and video clip 3 respectively. , get the audio feature that has the highest matching degree with audio feature 1 among the 3 video clips; judge the correlation between audio feature 2 and video clip 1, video clip 2 and video clip 3, and get the audio feature among the 3 video clips. 2 audio feature with the highest matching degree; determine the correlation between audio feature 3 and video clip 1, video clip 2 and video clip 3, and obtain the audio feature with the highest matching degree with audio feature 3 among the three video clips; finally it can be output The video clip corresponding to each audio feature.

In the embodiment of the present application, by sorting the M video clips through the audio-visual rhythm matching model, the M video clips can be sorted according to the rhythm of the background music, so that the picture content of the video clips matches the music rhythm; and Compared with matching videos directly with music according to the input order, the solution of this application can improve the consistency between the image content in the video and the rhythm of the background music, and improve the user experience.

Step S680: Obtain the processed video based on the sorted M video clips and background audio.

For example, a processed video can be obtained based on the video content of the sorted M video clips and the audio information of the background music.

Optionally, after adding background music to the sorted M video clips, you can also perform other editing processes on the videos to obtain the processed videos; where the other editing processes can include: adding image special effects to the video, adding text to the video , or add transition animation effects to the video, etc.

It should be understood that the transition effect of a video refers to the use of certain techniques between two scenes (for example, two pieces of material); for example, wipes, stacking changes, page curls, etc., to achieve a smooth transition between scenes or plots. Or achieve the effect of enriching the picture to attract the audience fruit.

It should be noted that, in addition to the above description, the same parts in Fig. 18 as in Figs. 12 to 17 can be referred to the relevant descriptions of Figs. 12 to 17 and will not be described again here.

Implementation method two: For videos with strong story lines, obtain N videos; obtain the video themes of N videos based on the text description information of the N videos; sort the N videos based on the text description information of the N videos, and obtain the sorted of N videos; determine the sorted M video clips based on the similarity confidence value between the video clips included in the sorted N videos and the video topic; determine the sorted M video clips based on the sorted M video clips and video topic information. Matching background music to M video clips; generating processed video.

Figure 19 is a schematic flow chart of a video editing method provided by an embodiment of the present application. The video editing method 700 can be executed by the electronic device shown in FIG. 1; the video editing method 700 includes steps S710 to S780, and steps S710 to S780 are described in detail below respectively.

Step S710: Obtain N videos.

Step S720: Obtain the text description information of N videos through the image-text conversion model.

For example, the image-to-text conversion model may include a CLIP model.

Step S730: Input the text description information of N videos into the pre-trained video topic classification model to obtain video topic information.

Optionally, if the theme information output in step S730 is one theme information, no user operation is required; if the theme information output in step S730 is two or more theme information, a prompt box can be displayed in the electronic device; the prompt box can Including candidate video topic information, the video topic information of N videos is determined based on the user's operation on the candidate video topic information in the prompt box.

For example, as shown in Figure 11, if two theme information are output in step S730, the display interface 319 can be displayed on the electronic device; the display interface 319 includes a prompt box 320, and the prompt box 320 includes two candidate video theme information. They are scenery and travel respectively. If the electronic device detects that the user clicks "Landscape", then the video theme information of the N videos is scenery; if the electronic device detects that the user clicks "Travel", then the video theme information of the N videos is travel.

Step S740: Sort the N videos based on the text description information of the N videos to obtain the sorted N videos.

It should be noted that in the embodiment of the present application, there is no restriction on the execution order of step S730 and step S740; step S730 can be executed first and then step S740; or step S740 can be executed first and then step S730; or, step S730 can be executed first and then step S730 can be executed. Step S730 and step S740 are executed simultaneously.

It should be understood that for N videos with strong story lines, based on the order in which the user uploads the N videos; or based on the timestamp information of the N videos, the order of the N videos may be the wrong order; for example, the order of the N videos There are 3 downloaded videos. Based on the chronological order of downloading, the 3 videos are: Video 1: A person goes home from the amusement park; Video 2: A person plays with entertainment facilities in the amusement park; Video 3: A person goes to the amusement park by car; If sorted based on the timestamp information of the video, the order is: video 1, video 2 and video 3; however, the normal order of a person's day trip should be to go out to the destination, arrive at the destination, and go home from the destination; the above The sorting of videos based on timestamps obviously does not conform to the reasonable logical sequence of travel videos; therefore, for videos with strong story lines, the sorting of N videos based on video timestamps may be wrong, causing users to The viewing experience of the videos is poor; in the solution of this application, for videos with strong story lines, N videos can be sorted based on the text description information of the videos, and it is determined that the sorted N videos conform to normal causes and consequences, improving User’s viewing experience.

For example, in the embodiment of the present application, the ranking of the N videos can be obtained based on the text description information of the N videos and the correlation between the natural languages between the text description information;

For example, the text information of N videos can be input into a pre-trained ranking model. The ranking model can be a neural Network; the pre-trained ranking model can be trained through the back propagation algorithm based on the training data set. The training data set can include the ranking of sample topic text and multiple sample description texts, and the sample topic text corresponds to multiple sample description texts. ;For example, the sample topic text is "appear"; the order of multiple sample description texts is: sample description text 1 "A person goes out", sample description text 2 is "A person is on the way to the destination", sample description text 3 is "A person arrives at the destination", sample description text 4 is "A person is active at the destination", sample description text 5 is "A person leaves from the destination on the way to the departure place", and sample description text 6 is "A person leaves from the destination on the way to the departure place". "People arrive at the starting point"; by learning from a large number of sample training data sets, when multiple description texts are input, the pre-trained ranking model can output the ranking of multiple description texts.

Step S750: Obtain similarity confidence values between images and video topic information in N videos based on the similarity evaluation model.

Step S760: Based on the similarity confidence values between the images in the N videos and the video topic information, obtain the sorted M video clips.

For example, based on the similarity confidence value between the image features included in the N videos and the text features of the video topic information, continuous multi-frame image features can be selected from the N videos to obtain a video clip. For example, when the similarity confidence value of consecutive multi-frame images is greater than a preset threshold, a video segment composed of multi-frame images is obtained.

It should be understood that since the N videos are sorted videos, the M video clips selected from the sorted N videos are sequential video clips, that is, the sorted M video clips are obtained.

For example, N videos include 3 videos, and the order of the 3 videos is video 2, video 1 and video 3; if video clip 2-1 and video clip 2-2 are selected from video 2, the time of video clip 2-1 Before the time of video clip 2-2; select video clip 1-1 in video 1; select video clip 3-1, video clip 3-2, and the time of video clip 3-1 in video 3 before the time of video clip 3-2; then the order of the five video clips is video clip 2-1, video clip 2-2, video clip 1-1, video clip 3-1, and video clip 3-2.

Step S770: Based on the duration and video theme information of the sorted M video clips, perform music matching processing in the candidate music library to obtain background music that matches the sorted M video clips.

It should be understood that since the M video clips are sorted video clips, it is necessary to use the sorted M video clips as a benchmark to select appropriate background music to match the sorted M videos; the selected background music The rhythm should be matched based on the style of the images of different video clips in the sorted M video clips.

For example, if the styles of the sorted M video clips are soothing and cheerful image content, the selected background music should be music with a soothing intro and a cheerful middle tempo.

Step S780: Obtain the processed video based on the sorted M video clips and background music.

It should be understood that the transition effect of a video refers to the use of certain techniques between two scenes (for example, two pieces of material); for example, wipes, stacking changes, page curls, etc., to achieve a smooth transition between scenes or plots. Or achieve the effect of enriching the picture to attract the audience.

It should be noted that, in addition to the above description, the same parts in Fig. 19 as in Figs. 12 to 17 can be referred to the relevant descriptions of Figs. 12 to 17 and will not be described again here.

It should be understood that in the first implementation method, for the M video clips without strong story lines, the sorted M video clips are obtained based on the rhythm of the background music; in the second implementation method, for the M video clips with strong story lines, The sorted M video clips are obtained based on the forward and backward causal connections of the M video clips; the music of the sorted M video clips that matches the rhythm is selected as the background music.

It should be understood that the above examples are to help those skilled in the art understand the embodiments of the present application, but are not intended to limit the embodiments of the present application to the specific numerical values or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.

The image processing method provided by the embodiment of the present application is described in detail above with reference to Figures 1 to 19; below, the device embodiment of the present application will be described in detail with reference to Figures 20 to 21. It should be understood that the devices in the embodiments of the present application can perform various methods of the foregoing embodiments of the present application, that is, for the specific working processes of the following various products, reference can be made to the corresponding processes in the foregoing method embodiments.

Figure 20 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 800 includes a display module 810 and a processing module 820 .

Wherein, the display module 810 is used to display a first interface, the first interface includes a video icon, and the video indicated by the video icon is a video stored in the electronic device; the processing module 820 is used to detect that the video is The first operation of the N video icons in the icon; in response to the first operation, obtain the information of the N videos, where N is an integer greater than 1; based on the information of the N videos, obtain the videos of the N videos Theme; based on the similarity between the images in the N videos and the video theme, select M video clips from the N videos; based on the video theme, obtain music that matches the video theme; Based on the M video clips and the music, a first video is obtained; the display module 810 is also used to display the first video.

Optionally, as an embodiment, the processing module 820 is specifically used to:

Input the N videos and the video topic into a pre-trained similarity matching model to obtain the similarity confidence value of the images in the N videos and the video topic, where the pre-trained similarity The matching model includes an image encoder, a text encoder and a first similarity measurement module. The image encoder is used to extract image features from the N videos. The text encoder is used to extract text from the video topic. Feature processing, the first similarity measurement module is used to measure the similarity between the image features in the N videos and the text features of the video subject, and the similarity confidence value is used to represent the N The probability that the images in the video are similar to the subject of the video;

Based on the similarity confidence values between the images in the N videos and the video themes, M video clips from the N videos are selected.

Sort the M video clips to obtain sorted M video clips;

The sorted M video clips and the music are synthesized into the first video.

The M video clips are sorted based on the rhythm of the music to obtain the sorted M video clips.

The music and the M video clips are input into a pre-trained audio-visual rhythm matching model to obtain the sorted M video clips. The pre-trained audio-visual rhythm matching model includes an audio encoder and a video encoder. With the first similarity measurement module, the audio encoder is used to extract features of the music to obtain audio features, the video decoder is used to extract features of the M video segments to obtain video features, and the first similarity The degree measurement module is used to measure the similarity between the audio features and the M video clips.

Convert the video content of the N videos into N pieces of text description information, the N pieces of text description information correspond to the N videos one-to-one, and one of the N pieces of text description information is used to describe the Image content information of one video among N videos;

Based on the N pieces of text description information, the theme information of the N videos is obtained, and the text description information is used to convert the video content in the N videos into text information.

The N pieces of text description information are input into a pre-trained topic classification model to obtain the topic information of the N videos. The pre-trained topic classification model is a deep neural network for text classification.

Optionally, as an embodiment, when the pre-trained topic classification model outputs at least two video topics, and the at least two video topics correspond to the N pieces of text description information, the display module 810 is also configured to :

Display a second interface, the second interface includes a prompt box, the prompt box includes information on the at least two video themes;

The processing module 820 is specifically used for:

detecting a second operation on the at least two video subjects;

In response to the second operation, the theme information of the N videos is obtained.

Optionally, as an embodiment, obtaining music that matches the video theme based on the video theme includes:

Based on the duration of the M video clips and the video theme, music matching the video theme is obtained, so The duration of the music is greater than or equal to the duration of the M video clips.

Optionally, as an embodiment, the pre-trained similarity matching model is a Transformer model.

Optionally, as an embodiment, the pre-trained similarity matching model is obtained through the following training method:

The similarity matching model to be trained is trained using the contrastive learning training method based on the first training data set to obtain the pre-trained similarity matching model; wherein, the first training data set includes positive example data pairs and negative examples Data pair, the positive example data pair includes a first sample text description information and a first sample video theme information, the first sample description information matches the first sample video theme information, and the positive example data pair An example data pair includes the first sample text description information and the second sample video theme information, and the first sample description information does not match the second sample video theme information.

Optionally, as an embodiment, the pre-trained audio-visual rhythm matching model is a Transformer model.

Optionally, as an embodiment, the pre-trained audio-visual rhythm matching model is obtained through the following training method:

The similarity matching model to be trained is trained using the contrastive learning training method based on the second training data set to obtain the pre-trained similarity matching model; wherein, the second training data set includes positive example data pairs and negative examples Data pair, the positive example data pair includes a first sample music and a first sample video, the rhythm of the first sample music matches the content of the first sample video, and the negative example data pair It includes the first sample music and the second sample video, and the rhythm of the first sample music does not match the content of the second sample video.

It should be noted that the above-mentioned electronic device 800 is embodied in the form of a functional module. The term "module" here can be implemented in the form of software and/or hardware, and is not specifically limited.

For example, a "module" may be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor) for executing one or more software or firmware programs. etc.) and memory, merged logic circuitry, and/or other suitable components to support the described functionality.

Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

Figure 21 shows a schematic structural diagram of an electronic device provided by this application. The dotted line in Figure 21 indicates that this unit or module is optional; the electronic device 900 can be used to implement the video editing method described in the above method embodiment.

The electronic device 900 includes one or more processors 901, and the one or more processors 901 can support the electronic device 900 to implement the video editing method in the method embodiment. Processor 901 may be a general-purpose processor or a special-purpose processor. For example, the processor 901 may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (field programmable). gate array, FPGA) or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

Optionally, the processor 901 can be used to control the electronic device 900, execute software programs, and process data of the software programs. The electronic device 900 may also include a communication unit 905 to implement input (reception) and output (transmission) of signals.

For example, the electronic device 900 may be a chip, and the communication unit 905 may be an input and/or output circuit of the chip, or the communication unit 905 may be a communication interface of the chip, and the chip may be used as a component of a terminal device or other electronic device. .

For another example, the electronic device 900 may be a terminal device, and the communication unit 905 may be a transceiver of the terminal device, or Alternatively, the communication unit 905 may include one or more memories 902 in which a program 904 is stored. The program 904 may be run by the processor 901 to generate an instruction 903, so that the processor 901 executes the above method embodiment according to the instruction 903. Describes video editing methods.

Optionally, data may also be stored in the memory 902 .

Optionally, the processor 901 can also read data stored in the memory 902. The data can be stored at the same storage address as the program 904, or the data can be stored at a different storage address than the program 904.

Optionally, the processor 901 and the memory 902 can be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

For example, the memory 902 can be used to store the related program 904 of the video editing method provided in the embodiment of the present application, and the processor 901 can be used to call the related program 904 of the video editing method stored in the memory 902 when executing the video editing method. , execute the video editing method of the embodiment of the present application; for example, display a first interface, the first interface includes a video icon, and the video indicated by the video icon is a video stored in the electronic device; detect that N video icons in the video icon are The first operation; in response to the first operation, obtain the information of N videos, where N is an integer greater than 1; based on the information of the N videos, obtain the video themes of the N videos; based on the images and video themes in the N videos Similarity, select M video clips from N videos; based on the video theme, get the music that matches the video theme; based on the M video clips and music, get the first video; display the first video.

Optionally, this application also provides a computer program product, which when executed by the processor 901 implements the video editing method in any method embodiment of this application.

For example, the computer program product may be stored in the memory 902, such as the program 904. The program 904 is finally converted into an executable object file that can be executed by the processor 901 through processes such as preprocessing, compilation, assembly, and linking.

Optionally, this application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a computer, the video editing method of any method embodiment in this application is implemented. The computer program may be a high-level language program or an executable object program.

For example, the computer-readable storage medium is memory 902. Memory 902 may be volatile memory or non-volatile memory, or memory 902 may include both volatile memory and non-volatile memory. Among them, non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically removable memory. Erase electrically programmable read-only memory (EPROM, EEPROM) or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of illustration, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) ) and direct memory bus random access memory (direct rambus RAM, DR RAM).

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the systems and devices described above For the specific working process of the unit, please refer to the corresponding process in the foregoing method embodiment, which will not be described again here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the embodiments of the electronic equipment described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, for example, multiple units or components may be The combination can either be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit.

It should be understood that in the various embodiments of the present application, the size of the sequence numbers of each process does not mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not be used in the embodiments of the present application. The implementation process constitutes any limitation.

In addition, the term "and/or" in this article is only an association relationship describing related objects, indicating that there can be three relationships. For example, A and/or B can mean: A alone exists, and A and B exist simultaneously. , there are three situations of B alone. In addition, the character "/" in this article generally indicates that the related objects are an "or" relationship.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code. .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application. should be covered by the protection scope of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims. In short, the above descriptions are only preferred embodiments of the technical solution of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the protection scope of this application.

Claims

A video editing method, characterized in that it is applied to electronic devices, including:

Display a first interface, the first interface includes a video icon, and the video indicated by the video icon is a video stored in the electronic device;

Detecting a first operation on N video icons among the video icons;

In response to the first operation, obtain information of N videos, where N is an integer greater than 1;

Based on the information of the N videos, obtain the video themes of the N videos;

Based on the correlation between the N videos and the video theme, select M video clips from the N videos;

Based on the duration of the M video clips and the video theme, obtain music that matches the video theme, and the duration of the music is greater than or equal to the duration of the M video clips;

Based on the M video clips and the music, a first video is obtained;

Display the first video.
The video editing method of claim 1, wherein selecting M video clips from the N videos based on the correlation between the N videos and the video theme includes:

Input the N videos and the video topic into a pre-trained similarity matching model to obtain the similarity confidence value of the images in the N videos and the video topic, where the pre-trained similarity The matching model includes an image encoder, a text encoder and a first similarity measurement module. The image encoder is used to extract image features from the N videos. The text encoder is used to extract text from the video topic. Feature processing, the first similarity measurement module is used to measure the similarity between the image features in the N videos and the text features of the video subject, and the similarity confidence value is used to represent the N The probability that the images in the video are similar to the subject of the video;

Based on the similarity confidence values between the images in the N videos and the video themes, M video clips from the N videos are selected.
The video editing method of claim 1 or 2, wherein obtaining the first video based on the M video clips and the music includes:

Sort the M video clips to obtain sorted M video clips;

The sorted M video clips and the music are synthesized into the first video.
The video editing method of claim 3, wherein said sorting the M video clips to obtain the sorted M video clips includes:

The M video clips are sorted based on the rhythm of the music to obtain the sorted M video clips.
The video editing method of claim 3, wherein said sorting the M video clips to obtain the sorted M video clips includes:

The M video clips are sorted based on the video contents in the M video clips to obtain the sorted M video clips.
The video editing method of claim 4, wherein the M video clips are sorted based on the rhythm of the music to obtain the sorted M video clips, including:

The music and the M video clips are input into a pre-trained audio-visual rhythm matching model to obtain the sorted M video clips. The pre-trained audio-visual rhythm matching model includes an audio encoder and a video encoder. With the second similarity measurement module, the audio encoder is used to extract features of the music to obtain audio features, the video encoder is used to extract features of the M video clips to obtain video features, and the second similarity measure module is used to extract features of the music to obtain audio features. The degree measurement module is used to measure the similarity between the audio features and the M video clips.
The video editing method according to any one of claims 1 to 6, characterized in that, based on the N views The video information of the N videos is obtained, including:

Convert the video content of the N videos into N pieces of text description information, the N pieces of text description information correspond to the N videos one-to-one, and one of the N pieces of text description information is used to describe the Image content information of one video among N videos;

Based on the N pieces of text description information, the video themes of the N videos are obtained, and the text description information is used to convert the video content in the N videos into text information.
The video editing method according to claim 7, wherein said obtaining video themes of said N videos based on said N pieces of text description information includes:

The N pieces of text description information are input into a pre-trained topic classification model to obtain the video topics of the N videos. The pre-trained topic classification model is a deep neural network for text classification.
The video editing method of claim 8, wherein when the pre-trained topic classification model outputs at least two video topics, the at least two video topics correspond to the N pieces of text description information, Also includes:

Display a second interface, the second interface includes a prompt box, the prompt box includes information on the at least two video themes;

The step of inputting the N pieces of text description information into a pre-trained topic classification model to obtain the video topics of the N videos includes:

detecting a second operation on the at least two video topics;

In response to the second operation, video themes of the N videos are obtained.
The video editing method according to any one of claims 2 to 9, wherein the pre-trained similarity matching model is a Transformer model.
The video editing method according to claim 10, characterized in that the pre-trained similarity matching model is obtained through the following training method:

The similarity matching model to be trained is trained using the contrastive learning training method based on the first training data set to obtain the pre-trained similarity matching model; wherein, the first training data set includes positive example data pairs and negative examples Data pair, the positive example data pair includes a first sample text description information and a first sample video theme information, the first sample description information matches the first sample video theme information, and the negative sample An example data pair includes the first sample text description information and the second sample video theme information, and the first sample description information does not match the second sample video theme information.
The video editing method according to any one of claims 6 to 11, wherein the pre-trained audio-visual rhythm matching model is a Transformer model.
The video editing method according to claim 12, wherein the pre-trained audio-visual rhythm matching model is obtained through the following training method:

The similarity matching model to be trained is trained using the contrastive learning training method based on the second training data set to obtain the pre-trained similarity matching model; wherein, the second training data set includes positive example data pairs and negative examples Data pair, the positive example data pair includes a first sample music and a first sample video, the rhythm of the first sample music matches the content of the first sample video, and the negative example data pair It includes the first sample music and the second sample video, and the rhythm of the first sample music does not match the content of the second sample video.
An electronic device, characterized by including:

one or more processors and memories;

The memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors invoke the computer instructions to cause the The electronic device executes the video editing method according to any one of claims 1 to 13.
A chip system, characterized in that the chip system is applied to electronic equipment, and the chip system includes one or more processors, and the processor is used to call computer instructions to cause the electronic equipment to execute claims 1 to 1 The video editing method described in any one of 13.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor causes the processor to execute any one of claims 1 to 13. video editing methods.
A computer program product, characterized in that the computer program product includes computer program code. When the computer program code is run by an electronic device, the electronic device causes the electronic device to execute the video described in any one of claims 1 to 13. Edit method.