WO2023011094A1 - Video editing method and apparatus, electronic device, and storage medium - Google Patents

Abstract

The present disclosure provides a video editing method and apparatus, an electronic device, and a storage medium, and relates to the technical field of artificial intelligence, in particular to the field of deep learning and video analysis. A specific implementation solution is as follows: classifying each event scene according to first frame information corresponding to respective first partial frames of at least one event scene comprised in a video main section, so as to obtain a scene classification result; when the scene classification result indicates that a target event scene corresponding to the scene classification result is a segment segmentation point, splitting the video main section according to start time information of the target event scene into at least one video segment, wherein each video segment comprises at least one event scene; and performing a video editing operation on the basis of the at least one video segment.

Description

Video editing method, device, electronic device and storage medium

This application claims the priority of Chinese Patent Application No. 202110883507.6 submitted on August 2, 2021, the contents of which are hereby incorporated by reference.

technical field

The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning and video analysis, and in particular to a video editing method, device, electronic equipment, and storage medium.

Background technique

Video (Video) technology generally refers to the technology of capturing, recording, processing, storing, transmitting and reproducing a series of static images in the form of electrical signals. In related technologies, editing operations such as segmentation, classification, description, and indexing can be performed on video data according to certain standards and rules.

Contents of the invention

The disclosure provides a video editing method, device, electronic equipment and storage medium.

According to an aspect of the present disclosure, a video editing method is provided, including: performing classification processing on each of the event scenes according to the first frame information corresponding to the first partial frames of at least one event scene included in the main video , to obtain the scene classification result; when the scene classification result indicates that the target event scene corresponding to the scene classification result is a segment segmentation point, according to the start time information of the target event scene, the video The feature film is split into at least one video clip, wherein each of the video clips includes at least one event scene; and a video editing operation is performed based on the at least one video clip.

According to another aspect of the present disclosure, there is provided a video editing device, including: a first processing module, configured to process each performing classification processing on each of the event scenes to obtain a scene classification result; the first splitting module is used to represent that the target event scene corresponding to the scene classification result is a segment segmentation point when the scene classification result indicates that the scene classification result is a segment segmentation point, According to the start time information of the target event scene, the video feature film is split into at least one video clip, wherein each of the video clips includes at least one of the event scene; and a video editing module, configured to A video editing operation is performed on the at least one video segment.

According to another aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores Instructions executed by the at least one processor to enable the at least one processor to perform the method as described above.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above method.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method as described above.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The accompanying drawings are used to better understand the present solution, and do not constitute a limitation to the present disclosure. in:

FIG. 1 schematically shows an exemplary system architecture to which a video editing method and device can be applied according to an embodiment of the present disclosure;

Fig. 2 schematically shows a flow chart of a video editing method according to an embodiment of the present disclosure;

Fig. 3 schematically shows a schematic diagram of video classification according to an embodiment of the present disclosure;

Fig. 4 schematically shows an example diagram of dividing video clips according to event scenes according to an embodiment of the present disclosure;

FIG. 5 schematically shows a schematic flowchart of a video editing method according to an embodiment of the present disclosure;

Fig. 6 schematically shows a block diagram of a video editing device according to an embodiment of the present disclosure; and

FIG. 7 shows a schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved are all in compliance with relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good customs.

In the technical solution of the present disclosure, before acquiring or collecting the user's personal information, the user's authorization or consent is obtained.

Video editing technology can use image, audio, subtitle and other information in the video to analyze, summarize and record video materials based on content and form characteristics, and organize and create various retrieval catalogs or retrieval methods, including video tagging, video disassembly, etc. requirements etc. For example, during editing, it is possible to perform timeline processing on the program layer, segment layer, scene layer, and shot layer, determine video keyword tags, describe video content, and explain video titles.

In the process of realizing the concept of the present disclosure, the inventors found that the video tag, video stripping, and video description need to combine the context structure information of the video to extract high-level semantic information. Traditional processing mainly relies on manual work. At present, the amount of multimedia data is increasing rapidly. In order to balance the speed of manual processing and the growth of media data, video editors need to spend more manpower to complete video editing and storage.

Some other solutions for implementing video editing include, for example: (1) Shot detection technology assists manual video splitting. Video editing tools use video frame difference information to achieve video lens layer segmentation; computer vision technology uses face detection to achieve lens splitting. (2) Video description technology based on machine learning. For example, video-caption technology uses image and audio information of video to realize simple scene description, such as "someone is doing something in a certain space". (3) Video intelligent labeling technology based on machine learning. Such as image classification, image detection, video classification, etc.

In the process of realizing the concept of the present disclosure, the inventors found that the scheme (1) can only realize the granularity splitting of shots, but cannot realize higher-level semantic aggregation by using single-frame image information or short-sequence image information. For the division of higher semantic levels (such as scene layer and fragment layer), it still needs to be completed with human assistance. The AI model of solution (2) requires a lot of manual annotation to realize model training. Moreover, the model's description of the scene is too simple and blunt, which cannot meet the needs of actual application. Due to the large amount of redundant information in video timing, especially in long video dramas, adopting scheme (3) to process all key frames indiscriminately will lead to low processing efficiency.

Based on this, the present disclosure introduces an automatic editing technology to realize automatic editing requirements through a multi-mode system based on machine learning. It mainly solves the current video editing part that is highly dependent on artificial understanding: intelligent labeling, automatic stripping, video description generation and other functions.

Fig. 1 schematically shows an exemplary system architecture to which a video editing method and device can be applied according to an embodiment of the present disclosure.

It should be noted that, what is shown in FIG. 1 is only an example of the system architecture to which the embodiments of the present disclosure can be applied, so as to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other device, system, environment or scenario. For example, in another embodiment, the exemplary system architecture to which the video editing method and device can be applied may include a terminal device, but the terminal device may implement the video editing method and device provided by the embodiments of the present disclosure without interacting with the server. .

As shown in FIG. 1 , a system architecture 100 according to this embodiment may include terminal devices 101 , 102 , 103 , a network 104 and a server 105 . The network 104 is used as a medium for providing communication links between the terminal devices 101 , 102 , 103 and the server 105 . Network 104 may include various connection types, such as wired and/or wireless communication links, among others.

Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages and the like. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and/or social platform software, etc. (only example).

The terminal devices 101, 102, 103 may be various electronic devices with display screens and supporting video browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers and the like.

The server 105 may be a server that provides various services, such as a background management server that supports content browsed by users using the terminal devices 101 , 102 , 103 (just an example). The background management server can analyze and process received data such as user requests, and feed back processing results (such as webpages, information, or data obtained or generated according to user requests) to the terminal device. The server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the problem of traditional physical host and VPS service ("Virtual Private Server", or "VPS") Among them, there are defects such as difficult management and weak business scalability. The server can also be a server of a distributed system, or a server combined with a blockchain.

It should be noted that, generally, the video editing method provided by the embodiment of the present disclosure may be executed by the terminal device 101 , 102 , or 103 . Correspondingly, the video editing apparatus provided by the embodiment of the present disclosure may also be set in the terminal device 101 , 102 , or 103 .

Or, generally, the video editing method provided by the embodiment of the present disclosure may also be executed by the server 105 . Correspondingly, the video editing apparatus provided by the embodiments of the present disclosure may generally be set in the server 105 . The video editing method provided by the embodiments of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101 , 102 , 103 and/or the server 105 . Correspondingly, the video editing apparatus provided by the embodiments of the present disclosure may also be set in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101 , 102 , 103 and/or the server 105 .

For example, when the video needs to be edited, the terminal devices 101, 102, and 103 can classify each event scene according to the first frame information corresponding to the first partial frame of at least one event scene included in the main video, Get the scene classification result. Then, when the scene classification result indicates that the target event scene corresponding to the scene classification result is a segment segmentation point, the main video is split into at least one video segment according to the start time information of the target event scene. Wherein, each video segment includes at least one event scene. Afterwards, a video editing operation is performed based on at least one video segment. Alternatively, a server or server cluster capable of communicating with the terminal devices 101, 102, 103 and/or server 105 can process the main video according to the scene classification results of the video scenes in the main video, and implement video editing operations.

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

Fig. 2 schematically shows a flowchart of a video editing method according to an embodiment of the present disclosure.

As shown in FIG. 2, the method includes operations S210-S230.

In operation S210, according to the first frame information corresponding to the first partial frame of at least one event scene included in the main video, classify each event scene to obtain a scene classification result.

In operation S220, when the scene classification result indicates that the target event scene corresponding to the scene classification result is a segment segmentation point, split the main video into at least one video segment according to the start time information of the target event scene. Wherein, each video clip includes at least one event scene.

In operation S230, a video editing operation is performed based on at least one video segment.

According to the embodiment of the present disclosure, the video can be divided into four levels according to the split granularity from small to large: shot layer, scene layer, segment layer, and program layer. The shot layer can refer to the continuous pictures recorded by the same camera at one time, that is, the shot picture. The scene layer may refer to a continuous video consisting of one or more shots covered by the same time and space with the same scene. A Fragment Layer can consist of more than one associated Event Scene. The program layer is generally a complete piece of video and audio data input.

Fig. 3 schematically shows a schematic view of video classification according to an embodiment of the present disclosure.

As shown in FIG. 3 , at the program level, feature video 300 includes two segments 310 , 320 . At the fragment layer, the fragment 310 includes four scenes 330 , 340 , 350 , and 360 , and the fragment 320 corresponds to one scene 370 . At the scene layer, each scene includes multiple shots, for example, the scene 330 includes four shots 331 , 332 , 333 , and 334 .

According to the embodiment of the present disclosure, segment splitting is used to aggregate semantically continuous scenes, and the same continuous event scene is merged into one segment. The criteria for separating the segments may include at least one of the following: (1) the two segments belong to different time and space; (2) the two segments are not closely related in event semantics. The performance of fragment splitting in the video may include at least one of the following: (1) the splitting position is at the scene picture change; (2) there are usually obvious audio changes at the splitting position, such as obvious silent pauses, background music changes, Audio changes, sudden changes in background noise such as whistles, car sounds, etc.; (3) The split position is often accompanied by obvious buffered video segments, such as scenery shots without obvious characters, black screen shots, and special shots such as spinning in and out; ( 4) The theme of the storyline changed before and after the split.

According to an embodiment of the present disclosure, one or more event scenes may be included in the video. The first partial frame may be a partial frame in multiple frames corresponding to an event scene. The first frame information may include at least one of graphic feature vectors, audio feature vectors, and text feature vectors corresponding to the first partial frame, as well as inter-scene image difference vectors, inter-scene audio difference vectors, and inter-scene texts At least one of the difference vector and so on.

According to an embodiment of the present disclosure, the process of classifying event scenes to obtain scene classification results may be completed by a scene granularity-based boundary classification model. On the premise of determining the event scenes included in the current feature video, partial frames may be extracted from each event scene to obtain the first partial frame. By inputting multiple first partial frames corresponding to multiple event scenes into the boundary classification model as input features, the boundary classification result of each scene in the video can be obtained. The boundary classification result can be expressed in the form of 0 or 1, and 1 can indicate that the target event scene to which the frame corresponding to the result belongs is a boundary point under the scene granularity, which means that the target event scene to which the frame with a classification result of 1 belongs is Fragment segmentation point, 0 is a non-boundary point, that is, a non-fragment segmentation point.

It should be noted that the segment segmentation point determined at the scene granularity can be embodied as an event scene. Scene-grained video features are re-divided into fragment-grained video segments.

Fig. 4 schematically shows an example diagram of dividing video segments according to event scenes according to an embodiment of the present disclosure.

As shown in FIG. 4 , feature video 400 includes five event scenes: 410 , 420 , 430 , 440 , 450 . For each event scene, some frames can be extracted, and the number of partial frames extracted for different event scenes can be different, such as extracting m frames for event scene 410: 411, ..., 41m, for event scene 450 Extract n frames: 451, ..., 45n, etc. Then at least one of the graphic feature vectors, audio feature vectors, and text feature vectors corresponding to each extracted frame, as well as the difference vector of images between two adjacent frames, the difference vector of audio and the text At least one of the difference vector and the like is input into the boundary classification model as an input feature, for example, a result such as 00000000100000000100 can be output. Based on the output results, for example, two frames representing segment segmentation points can be determined. For each frame representing a segment segmentation point, the target event scene corresponding to the frame representing the segment segmentation point may be further determined, for example, 430 and 440 respectively. Therefore, the feature video can be split according to the starting time points of 430 and 440, and three video clips can be obtained, 410, 420 form the first video segment 460, 430 form the second video segment 470, 440, 450 form The third video segment 480.

According to an embodiment of the present disclosure, for both the feature video and the divided video segments, a corresponding commentary video can be generated, and the commentary video can include text introduction, voice description, and spliced multiple shots for the feature video or video segment.

Through the above-mentioned embodiments of the present disclosure, scene layer segmentation information can be used to selectively sample some frames, so as to achieve efficient video clip editing effect with low resource consumption.

The method shown in FIG. 2 will be further described below in conjunction with specific embodiments.

According to an embodiment of the present disclosure, performing a video editing operation based on at least one video segment includes: determining a target video segment for which a commentary video needs to be generated. Identifying information of the target video segment is determined according to the character features and text features in the target video segment. The summary information of the target video segment is determined according to the text features of the target video segment. A target shot related to summary information is determined from a target video clip. The title information of the target video segment is determined according to the summary information. Generate an explainer video based on identification information, summary information, target shot and title information.

According to an embodiment of the present disclosure, the target video segment may be a video segment determined from at least one video segment obtained by splitting the main video. The character features may include features represented by character information in the target video segment. The text features may include features represented by at least one of subtitle information, voice information, etc. in the target video segment. The identification information may include the video name corresponding to the target video segment, such as the title of a film and television drama segment. The summary information may represent textual commentary for the target video segment. The target shot may be one or more shots of the target video segment. The title information may be a title or name redefined for the target video segment.

It should be noted that the target video segment may also be replaced by a positive video. That is, for the main video, the above operations can be performed to generate a corresponding commentary video.

According to an embodiment of the present disclosure, taking a film and television drama segment as an example, according to the character features and text features in the target video segment, determining the identification information of the target video segment can be specifically expressed as: according to the human face in the segment interval corresponding to the target video segment Detect the aggregation results to obtain the star names of the characters in the segment. Extracting the subtitle keywords can obtain the movie and TV character names of the characters in the segment interval. By combining the name of the star and the name of the film and television character, searching in the film and television drama knowledge graph can obtain the name of the film and television drama corresponding to the clip.

According to an embodiment of the present disclosure, in the case where there are multiple segments for which identification information needs to be determined in the same target video segment, the name of the film and television drama of each segment may be firstly determined through the aforementioned method. Since the names of the film and television dramas identified for each segment may be different, the scoring votes of different results can be further combined, and the name of the film and television drama with the most votes can be used as the final output result of each segment in the same target video segment.

Through the above-mentioned embodiments of the present disclosure, the video commentary is generated for the main video and the video segment, which can further improve the richness of the video editing result.

According to an embodiment of the present disclosure, the text feature includes line text in the target video segment. Determining the abstract information of the target video segment according to the text feature of the target video segment includes: determining the generator identifier of each line text in the text feature. Information extraction is performed on each line text marked with the generator ID to obtain summary information.

According to an embodiment of the present disclosure, the process of determining the summary information of the target video segment according to the text features of the target video segment may be completed by a text summary model. The chapter summary model can take the line text corresponding to the input segment as input, and obtain the summary description corresponding to the segment as output for training. The segment summaries for training can be derived from the corresponding plot introductions of TV dramas and movies on the Internet.

According to an embodiment of the present disclosure, the text summarization model obtained through training may use the subtitle text of the target video segment as input to generate summary information of the target video segment. The subtitle text may include the narrator and the narration content, and the narrator may include the names of the characters in the film and television drama. The names of characters in film and television dramas can be determined by first obtaining the stars through face detection, and then obtaining the names of the characters from the corresponding relationship between stars and roles. The narrative content may include at least one of lines and subtitle content.

Through the above-mentioned embodiments of the present disclosure, an intelligent summary information generation method is provided, and the summary information is determined based on the character and line information of the target video segment, which can effectively improve the accuracy and completeness of the summary description.

According to an embodiment of the present disclosure, the text feature includes line text in the target video segment. Determining the target shot related to the summary information from the target video segment includes: determining the duration of the voice broadcast for the summary information. At least one line text associated with the summary information is determined. For each line text, determine the shot fragments that match the line text in time, and obtain multiple shot fragments. According to the voice broadcast duration, at least one target shot segment is determined from the plurality of shot segments. Wherein, the total duration of at least one target shot segment matches the duration of the voice broadcast. At least one target shot segment is determined as a target shot.

According to an embodiment of the present disclosure, in the chapter summary model, a self-attention (self-attention) operation can be introduced for text timing characteristics, and the self-attention value can represent the contribution of a certain line text to the final output summary information. For each line text, a shot-level video with the highest temporal overlap, or a shot-level video associated with the line text, may be selected as a shot segment corresponding to the line text. The voice broadcast duration of the summary information can be calculated and determined according to the AI automatic broadcast speech rate. At least one target shot segment may be selected according to the shot score from high to low until the selected shot segment can fill the entire voice broadcast duration. The shot score can be the normalized score corresponding to self-attention.

Through the above-mentioned embodiments of the present disclosure, by extracting target shots to fill in the voice broadcast process, the richness of the commentary video can be further increased.

According to an embodiment of the present disclosure, the process of determining the title information of the target video segment according to the abstract information may be completed by prediction of a chapter title generation model. A large number of movie and TV drama plot introductions, as well as titles corresponding to episodes and segments can be crawled through the Internet. Through these data, the chapter title generation model can be trained.

According to an embodiment of the present disclosure, the title information can be predicted by inputting the summary information into the above chapter title generation model.

According to an embodiment of the present disclosure, the title information of the target video segment is determined according to the summary information. According to the identification information, abstract information, target shot and title information, the generation of commentary video can be embodied as: playing the aforementioned selected target shots in time sequence, adding the video name, segment title, text summary subtitle and abstract AI human voice Broadcast to get a commentary video for the target video segment.

Through the above-mentioned embodiments of the present disclosure, the obtained commentary video can efficiently reflect the target video segment, and effectively ensure completeness, accuracy and richness.

According to an embodiment of the present disclosure, the video editing method may further include: according to the second frame information corresponding to the second partial frame of at least one shot included in the main video, classify each shot to obtain the Shot classification results. When the shot classification result indicates that the target shot corresponding to the shot classification result is a scene segmentation point, the main video is split into at least one event scene according to the start time information of the target shot. Wherein, each event scene includes at least one shot.

According to an embodiment of the present disclosure, the criteria for shot splitting may include camera switching. You can use methods such as color histogram, inter-frame difference, image feature similarity, video-splitter tools, etc. to split the video. Scene splits are usually based on criteria such as spatial or temporal transformations of shots. For example, when a video changes from reality to memory narrative, it is a temporal scene change. The video changes from indoor to airport, which belongs to the space scene change.

According to an embodiment of the present disclosure, a video is usually spliced from one or more shots. The second partial frame may be a partial frame in multiple frames corresponding to one shot. The second frame information may include at least one of image features, face features, and audio features corresponding to the second partial frame.

According to an embodiment of the present disclosure, the process of performing classification processing on shots to obtain a shot classification result may be completed by a shot granularity-based boundary classification model. The open source dataset "MovieScenes dataset" and the labeled data in the local database can be combined to encode whether each shot in the feature film is a boundary, and jointly train the boundary classification model.

According to an embodiment of the present disclosure, when predicting whether a certain shot is a boundary, partial frames may be extracted from each shot of the main video to obtain a second partial frame. By inputting the second part of the frame corresponding to the shot into the boundary classification model as an input feature, the shot classification result of each shot in the video can be obtained, and the judgment of whether the shot is a boundary can be realized. The boundary classification result can also be expressed in the form of 0 or 1, 1 can indicate that the frame corresponding to the result is a boundary point under the shot granularity, and 0 is a non-boundary point. If it is judged that a shot is a boundary, the start time of the shot can be used as the start time of scene splitting, so as to split the main video into event scenes.

Through the above-mentioned embodiments of the present disclosure, it is possible to use the shot layer segmentation information to selectively sample some frames, and achieve an efficient video scene splitting effect with low resource consumption.

According to an embodiment of the present disclosure, after multi-level splitting is performed on the feature video, based on the splitting results, different strategy models may be used to extract tags for different levels of video data.

According to an embodiment of the present disclosure, the video editing method may further include: for each shot: acquiring a fourth partial frame corresponding to the shot; performing feature extraction on each frame in the fourth partial frame to obtain a second feature extraction result. A shot label of the shot is determined according to the second feature extraction result.

According to an embodiment of the present disclosure, the shot layer may use an image granularity model to extract shot labels. For example, use the face detection model to identify different stars; use the object detection model to detect different objects, such as guns, flags, etc.; use the object attribute model to detect task attributes, such as character image type, clothing type, etc.; Aesthetics, picture style, etc. are tested. Due to the long duration of the video, if all the key frames are sampled to generate labels, the image labels between different frames still have great redundancy. Video-level analysis is slower if every frame goes into the image model for analysis. Based on the shot granularity, a small number of frames are taken to obtain the fourth part of the frame, and the second feature extraction result obtained by analyzing and averaging the fourth part of the frames can be used as the shot label.

Through the above-mentioned embodiments of the present disclosure, based on the granularity of shots, some frames are selectively sampled, and shot tags are determined with relatively low resources, which not only ensures the tag recall rate but also improves the overall analysis speed of the video.

According to an embodiment of the present disclosure, the video editing method may further include: for each event scene: acquiring a third partial frame corresponding to each target shot included in the event scene; performing feature extraction on each frame in the third partial frame, Obtain the first feature extraction result. A scene label of the event scene is determined according to the first feature extraction result.

According to an embodiment of the present disclosure, the scene layer may adopt a video granularity timing model to provide videos with tags such as scenes, activities, and actions, such as tags such as airport, living room, sleepiness, and conversation. When judging the label of the event scene, for each target shot in the scene, some frames can be sampled to obtain the third part of the frame, which can be used as the input of the scene model, so as to obtain the scene label of each event scene, and then determine the event scene. Information such as the place where the scene takes place, activities, actions, etc.

It should be noted that the number of frames captured for different shots in an event scene may be different. In an embodiment, the frame number of each shot may be positively correlated with the shot duration.

Through the above-mentioned embodiments of the present disclosure, based on scene layer segmentation information, selective sampling of partial frames and determination of scene labels with relatively low resources can further ensure label recall and improve overall video analysis speed.

According to an embodiment of the present disclosure, the video editing method may further include: performing at least one of the following processes on the video frame sequence information: when the first target video frame sequence is detected, determining that the first target video frame sequence is the initial video The sequence of video frames at the beginning or end of the credits; if the second target video frame sequence is detected, determine that the second target video frame sequence is the sequence of video frames at the beginning or end of the original video. Wherein, the video frame sequence information includes the video frame sequence in the initial video and the audio corresponding to the video frame sequence, the first target video frame sequence includes a video frame whose subtitle is located at the first position of the video frame picture, and the second target video frame The audio corresponding to the sequence is the audio of the target type. A video feature is determined according to at least one of the first target video frame sequence and the second target video frame sequence.

According to an embodiment of the present disclosure, the initial video may be a video including credits and credits. The head and tail detection model can detect the beginning and end of the video, and the model can be realized by subtitle detection and audio feature classification. In terms of subtitles, in the feature film, the subtitles usually appear at the bottom of the picture; in the opening and closing credits, the subtitles often appear in the picture. Therefore, it can be defined that the first position includes in the picture. In terms of audio, there is generally no actor narration audio at the beginning and end of the film, and most of them are pure music or music mixed with some special effects and background sounds. Therefore, it is possible to define target types including pure music, background sound with special effects, and the like.

According to an embodiment of the present disclosure, for the initial video, the start and end positions of the main video may be determined through a credits and credits detection model. Specifically, it is possible to classify whether a certain piece of audio features within a sliding window with a unit of 1 second is an opening and closing credits. Combining the results of subtitle position detection and audio feature detection, it is possible to time-mark the beginning and end of the video at the second granularity, so as to determine the main video.

According to the above-mentioned embodiments of the present disclosure, a screening method is provided before video processing, which can screen relatively main video content, and can effectively improve video editing efficiency.

According to an embodiment of the present disclosure, the video editing method may further include: determining a plurality of third target video frame sequences in the main video film. Feature extraction is performed on each third target video frame sequence to obtain a third feature extraction result. Determine the type label of the positive video according to the plurality of third feature extraction results.

According to an embodiment of the present disclosure, the type label of the main video, that is, the type label of the program layer, can be obtained based on a long video sequence model of images and audio sequences. Through this model, comprehensive labels can be provided for the entire video, such as TV drama-family ethics, movie-sci-fi, etc. The long video timing model can set the maximum number of video analysis frames during training to maintain machine memory stability. If the number of analyzed video frames is greater than the threshold, the maximum number of consecutive frames can be randomly intercepted as training input during training. When predicting, a non-overlapping sliding window can be used, and the maximum number of frames is sequentially taken to obtain the third target video frame sequence as input. By averaging all the sliding window score results obtained, it can be output as the type label of the video positive.

Through the above-mentioned embodiments of the present disclosure, based on the program layer information, selectively sampling part of the frame sequence, and determining the type label of the video with relatively low resources can effectively ensure the label recall rate and improve the overall analysis speed of the video at the same time.

According to an embodiment of the present disclosure, the first frame information may include at least one of image features, audio features, and text features corresponding to the first partial frame, and an image difference vector between two adjacent frames in the first partial frame, At least one of an audio delta vector and a text delta vector. And/or, the second frame information may include at least one of image features and audio features corresponding to the second partial frame. It is not limited here.

Through the above-mentioned embodiments of the present disclosure, processing based on the information of each dimension of the video can effectively ensure the accuracy and richness of the time editing structure.

Fig. 5 schematically shows a schematic diagram of the flow of a video editing method according to an embodiment of the present disclosure.

As shown in FIG. 5 , for an initial video, it may first be detected through a credits and credits detection model 510 to screen out credits and credits to obtain a main video. Combining the image model 521 , video model 522 , and text model 523 in the video classification module 520 , the feature video can be further processed to divide the feature video into shot granularity, scene granularity, and segment granularity. Based on the shot granularity information, scene granularity information, segment granularity information and other results obtained by hierarchical division, the shot-level image labels, scene-level spatio-temporal labels, segment-level video commentary generation, and program-level types obtained for the initial video editing can be further determined.

Through the above-mentioned embodiments of the present disclosure, a method for generating smart labels, smart splitting, and smart commentary is provided. The whole method can effectively reduce the dependence on manual processing and improve the editing and processing speed of the video. Video processing based on partial frames or key frames can mark the entire video at different levels, providing a basis for video storage indexing.

Fig. 6 schematically shows a block diagram of a video editing device according to an embodiment of the present disclosure.

As shown in FIG. 6 , the video editing device 600 includes a first processing module 610 , a first splitting module 620 and a video editing module 630 .

The first processing module 610 is configured to classify each event scene according to the first frame information corresponding to the first partial frame of at least one event scene included in the main video, to obtain a scene classification result.

The first splitting module 620 is used to split the main video into at least A video clip. Wherein, each video clip includes at least one event scene.

A video editing module 630, configured to perform video editing operations based on at least one video segment.

According to an embodiment of the present disclosure, the video editing module includes a first determining unit, a second determining unit, a third determining unit, a fourth determining unit, a fifth determining unit and a generating unit.

The first determination unit is configured to determine a target video segment that needs to generate a commentary video.

The second determination unit is configured to determine the identification information of the target video segment according to the character features and text features in the target video segment.

The third determination unit is configured to determine the summary information of the target video segment according to the text features of the target video segment.

The fourth determination unit is configured to determine a target shot related to the summary information from the target video segment.

The fifth determining unit is configured to determine the title information of the target video segment according to the summary information.

The generation unit is used to generate the commentary video according to the identification information, summary information, target shot and title information.

According to an embodiment of the present disclosure, the text feature includes line text in the target video segment. The third determining unit includes a first determining subunit and an obtaining subunit.

The first determination subunit is used to determine the generator identifier of each line text in the text feature.

The obtaining subunit is configured to extract information from each line text marked with the generator ID to obtain summary information.

According to an embodiment of the present disclosure, the text feature includes line text in the target video segment. The fourth determination unit includes a second determination subunit, a third determination subunit, a fourth determination subunit, a fifth determination subunit and a sixth determination subunit.

The second determining subunit is used to determine the duration of the voice broadcast of the summary information.

The third determining subunit is configured to determine at least one line text associated with the summary information.

The fourth determining subunit is configured to determine, for each line text, a shot segment that matches the line text in time to obtain a plurality of shot segments.

The fifth determining subunit is configured to determine at least one target shot segment from a plurality of shot segments according to the voice broadcast duration. Wherein, the total duration of at least one target shot segment matches the duration of the voice broadcast.

A sixth determining subunit, configured to determine at least one target shot segment as the target shot.

According to an embodiment of the present disclosure, the video editing device further includes a second processing module and a second splitting module.

The second processing module is configured to classify each shot according to the second frame information corresponding to the second partial frame of at least one shot included in the main video, and obtain a shot classification result of each shot.

The second splitting module is used to split the video feature into at least one event scene according to the start time information of the target shot when the shot classification result indicates that the target shot corresponding to the shot classification result is a scene segmentation point . Wherein, each event scene includes at least one shot.

According to an embodiment of the present disclosure, the video editing device further includes a first feature extraction module and a first determination module.

The first feature extraction module is used for each event scene: obtaining the third partial frame corresponding to each target lens included in the event scene; performing feature extraction on each frame in the third partial frame to obtain the first feature extraction result .

The first determining module is configured to determine the scene label of the event scene according to the first feature extraction result.

According to an embodiment of the present disclosure, the video editing device further includes a second feature extraction module and a second determination module.

The second feature extraction module is configured to, for each shot: acquire a fourth partial frame corresponding to the shot; perform feature extraction on each frame in the fourth partial frame to obtain a second feature extraction result.

The second determination module is configured to determine the lens label of the lens according to the second feature extraction result.

According to an embodiment of the present disclosure, the video editing device further includes a third determining module, a fourth determining module and a fifth determining module.

The third determination module is configured to determine that the first target video frame sequence is a sequence of video frames at the beginning or end of the initial video when the first target video frame sequence is detected. Wherein, the first target video frame sequence includes a video frame whose subtitle is located at the first position of the video frame picture.

The fourth determination module is configured to determine that the second target video frame sequence is a sequence of video frames at the beginning or end of the original video when the second target video frame sequence is detected. Wherein, the audio corresponding to the second target video frame sequence is the audio of the target type.

The fifth determination module is configured to determine the main video according to at least one of the first target video frame sequence and the second target video frame sequence.

According to an embodiment of the present disclosure, the video editing device further includes a sixth determination module, a third feature extraction module, and a seventh determination module.

The sixth determination module is configured to determine a plurality of third target video frame sequences in the main video film.

The third feature extraction module is configured to perform feature extraction for each third target video frame sequence to obtain a third feature extraction result.

The seventh determination module is used to determine the type label of the main video according to the multiple third feature extraction results.

According to an embodiment of the present disclosure, the first frame information includes at least one of image features, audio features, and text features corresponding to the first partial frame, and image difference vectors, audio At least one of a delta vector and a text delta vector. And/or, the second frame information includes at least one of image features and audio features corresponding to the second partial frame.

According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by at least one processor, and the instructions are processed by at least one The processor is executed, so that at least one processor can perform the method as described above.

According to an embodiment of the present disclosure, there is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

According to an embodiment of the present disclosure, a computer program product includes a computer program, and the computer program implements the above method when executed by a processor.

FIG. 7 shows a schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present disclosure. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 7, the device 700 includes a computing unit 701 that can execute according to a computer program stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random-access memory (RAM) 703. Various appropriate actions and treatments. In the RAM 703, various programs and data necessary for the operation of the device 700 can also be stored. The computing unit 701, ROM 702, and RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to the bus 704 .

Multiple components in the device 700 are connected to the I/O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc. ; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks.

The computing unit 701 may be various general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of computing units 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 701 executes various methods and processes described above, such as video editing methods. For example, in some embodiments, the video editing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and/or installed on the device 700 via the ROM 702 and/or the communication unit 709. When the computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the video editing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the video editing method in any other suitable manner (for example, by means of firmware).

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may include or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN) and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The specific implementation manners described above do not limit the protection scope of the present disclosure. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (20)

1. A method of video editing, comprising:

   According to the first frame information corresponding to the first partial frames of at least one event scene included in the main video, classify each event scene to obtain a scene classification result;

   When the scene classification result indicates that the target event scene corresponding to the scene classification result is a segment segmentation point, according to the start time information of the target event scene, split the video feature into at least one video clips, wherein each of said video clips includes at least one of said event scenes; and

   A video editing operation is performed based on the at least one video segment.
2. The method according to claim 1, said performing video editing operations based on said at least one video segment comprising:

   determining a target video segment from the at least one video segment;

   Determine the identification information of the target video segment according to the character features and text features in the target video segment;

   determining summary information of the target video segment according to the text features of the target video segment;

   determining a target shot related to the summary information from the target video segment;

   determining title information of the target video segment according to the summary information; and

   An explanatory video corresponding to the target video segment is generated according to the identification information, the abstract information, the target shot and the title information.
3. The method according to claim 2, wherein the text feature comprises the line text in the target video segment; the determination of the summary information of the target video segment according to the text feature of the target video segment comprises:

   Determining the generator identifier of each line text in the text feature; and

   Information extraction is performed on each line text marked with the generator identifier to obtain the summary information.
4. The method according to claim 2, wherein said text feature comprises lines text in said target video segment; said determining from said target video segment the target shot relevant to said abstract information comprises:

   Determine the voice broadcast duration for voice broadcast of the summary information;

   determining at least one line text associated with the summary information;

   For each line text, determine a shot segment that matches the line text in time to obtain a plurality of shot segments;

   Determine at least one target shot segment from the plurality of shot segments according to the voice broadcast duration, wherein the total duration of the at least one target shot segment matches the voice broadcast duration; and

   The at least one target shot segment is determined as the target shot.
5. The method according to any one of claims 1 to 4, further comprising:

   Classify each of the shots according to the second frame information corresponding to the second partial frame of at least one shot included in the main video, and obtain a shot classification result for each of the shots;

   When the shot classification result indicates that the target shot corresponding to the shot classification result is a scene segmentation point, according to the start time information of the target shot, split the video feature into the at least one Event scenes, wherein each event scene includes at least one shot.
6. The method according to claim 5, further comprising:

   For each of the described event scenarios:

   Acquiring the third partial frame corresponding to each target shot included in the event scene;

   performing feature extraction on each frame in the third partial frame to obtain a first feature extraction result; and

   Determine the scene label of the event scene according to the first feature extraction result.
7. The method according to claim 5 or 6, further comprising:

   For each said lens:

   Obtain the fourth part of the frame corresponding to the shot;

   performing feature extraction on each frame in the fourth partial frame to obtain a second feature extraction result; and

   Determine the shot label of the shot according to the second feature extraction result.
8. The method according to any one of claims 1 to 7, further comprising:

   In the case that the first target video frame sequence is detected, it is determined that the first target video frame sequence is the video frame sequence of the beginning or end of the original video, wherein the first target video frame sequence includes subtitles located in the video frame the video frame at the first position of the screen;

   When the second target video frame sequence is detected, determine that the second target video frame sequence is the video frame sequence of the beginning or end of the initial video, wherein the audio corresponding to the second target video frame sequence is the target type audio; and

   The video feature is determined according to at least one of the first target video frame sequence and the second target video frame sequence.
9. The method according to any one of claims 1 to 8, further comprising:

   determining a plurality of third target video frame sequences in the positive video;

   performing feature extraction for each of the third target video frame sequences to obtain a third feature extraction result; and

   Determine the type label of the video positive film according to a plurality of the third feature extraction results.
10. The method according to any one of claims 5 to 9, wherein the first frame information includes at least one of image features, audio features and text features corresponding to the first partial frame, and the first partial frame At least one of image difference vectors, audio difference vectors and text difference vectors between two adjacent frames in the frame; and/or

    The second frame information includes at least one of image features and audio features corresponding to the second partial frame.
11. A video editing device comprising:

    The first processing module is configured to perform classification processing on each of the event scenes according to the first frame information corresponding to the first partial frames of at least one event scene included in the main video film, and obtain a scene classification result;

    The first splitting module is configured to divide the target event scene according to the start time information of the target event scene when the scene classification result indicates that the target event scene corresponding to the scene classification result is a segment segmentation point. The feature film of the video is split into at least one video segment, wherein each of the video segments includes at least one event scene; and

    A video editing module, configured to perform video editing operations based on the at least one video segment.
12. The device according to claim 11, wherein the video editing module comprises:

    The first determination unit is used to determine the target video segment segment that needs to generate the commentary video;

    A second determining unit, configured to determine the identification information of the target video segment according to the character features and text features in the target video segment;

    A third determining unit, configured to determine summary information of the target video segment according to text features of the target video segment;

    A fourth determining unit, configured to determine a target shot related to the summary information from the target video segment;

    A fifth determining unit, the segment is configured to determine the title information of the target video segment according to the summary information; and

    A generating unit, configured to generate an explanatory video corresponding to the target video segment according to the identification information, the abstract information, the target shot and the title information.
13. The device according to claim 12, wherein the text features include line text in the target video segment; the third determination unit includes:

    The first determining subunit is used to determine the generator identifier of each line text in the text feature; and

    The obtaining subunit is configured to perform information extraction on each line text marked with the generator ID to obtain the summary information.
14. The device according to claim 12, wherein the text features include line text in the target video segment; the fourth determination unit includes:

    The second determination subunit is used to determine the voice broadcast duration for voice broadcast of the summary information;

    A third determining subunit, configured to determine at least one line text associated with the summary information;

    The fourth determining subunit is used to determine, for each line text, a shot segment that matches the line text in time to obtain a plurality of shot segments;

    The fifth determining subunit is configured to determine at least one target shot segment from the plurality of shot segments according to the voice broadcast duration, wherein the total duration of the at least one target shot segment matches the voice broadcast duration ;as well as

    A sixth determining subunit, configured to determine the at least one target shot segment as the target shot.
15. An apparatus according to any one of claims 11 to 14, further comprising:

    The second processing module is configured to perform classification processing on each of the shots according to the second frame information corresponding to the second partial frame of at least one shot included in the main video film, and obtain a shot classification result of each of the shots. ;

    The second splitting module is used to divide the video according to the start time information of the target shot when the shot classification result indicates that the target shot corresponding to the shot classification result is a scene segmentation point The feature film is divided into the at least one event scene, wherein each event scene includes at least one shot.
16. The apparatus of claim 15, further comprising:

    The first feature extraction module is used for each of the event scenarios:

    Acquiring the third partial frame corresponding to each target shot included in the event scene;

    performing feature extraction on each frame in the third partial frame to obtain a first feature extraction result; and

    A first determining module, configured to determine the scene label of the event scene according to the first feature extraction result.
17. An apparatus according to any one of claims 11 to 16, further comprising:

    The second feature extraction module is used for each of the shots:

    Obtain the fourth part of the frame corresponding to the shot;

    performing feature extraction on each frame in the fourth partial frame to obtain a second feature extraction result; and

    The second determination module is configured to determine the lens label of the lens according to the second feature extraction result.
18. An electronic device comprising:

    at least one processor; and

    a memory communicatively coupled to the at least one processor; wherein,

    The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can perform any one of claims 1-10. Methods.
19. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.
20. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-10.

Images