CN113591570A - Video processing method and device, electronic equipment and storage medium - Google Patents

Abstract

The application discloses a video processing method, a video processing device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical field of computer vision and deep learning, and is particularly used in a video analysis scene. The specific implementation scheme is as follows: acquiring a video; performing feature extraction on the video to acquire feature information of the video; calling a boundary prediction model to perform time sequence boundary prediction on the characteristic information so as to generate a time sequence boundary of the video; and segmenting the video according to the time sequence boundary to generate a video segment of the video. Therefore, the accuracy of time sequence boundary prediction can be improved, and the recall rate of time sequence nomination in the un-segmented video is increased.

Description

Video processing method, apparatus, electronic device and storage medium

technical field

The present application relates to the technical field of artificial intelligence, in particular to the technical field of computer vision and deep learning, and is specifically used in video analysis scenarios, and in particular, to a video processing method, apparatus, electronic device and storage medium.

Background technique

Time-series action positioning, that is, inputting an unsegmented video, locating the action segment according to the video content, including its start time and end time, and the produced action segment is called a time-series proposal. Temporal action localization is one of the most important and challenging problems in the field of computer vision video understanding, which is attributed to its great application potential in video highlight generation, video recommendation, retrieval, etc. ,

Among them, an important aspect for evaluating temporal action localization methods is the average recall rate. Most of the current methods focus on generating flexible and accurate temporal boundaries with reliable nomination confidence.

In related technologies, methods related to deep learning are mainly divided into two categories:

1. A method based on predefined anchor box regression generates a large number of candidate time series nominations that may contain behaviors, and then selects the correct candidate time series nominations through the classification task;

②. Model the video frame timing relationship, use the local details around the boundary to predict the boundary, and use the boundary combination to generate the timing nomination.

SUMMARY OF THE INVENTION

The present application provides a video processing method, apparatus, electronic device and storage medium.

According to an aspect of the present application, a video processing method is provided, comprising:

get video;

performing feature extraction on the video to obtain feature information of the video;

Calling a boundary prediction model to perform temporal boundary prediction on the feature information to generate a temporal boundary of the video; and

The video is segmented according to the timing boundary to generate video segments of the video.

According to another aspect of the present application, a video processing apparatus is provided, comprising:

The first acquisition module is used to acquire the video;

A second acquisition module, configured to perform feature extraction on the video to acquire feature information of the video;

a first generation module for invoking a boundary prediction model to perform time-series boundary prediction on the feature information to generate a time-series boundary of the video; and

A second generating module, configured to segment the video according to the timing boundary to generate video segments of the video.

According to another aspect of the present application, an electronic device is provided, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can execute the video processing method according to the embodiment of the above aspect .

According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, and a computer program is stored thereon, and the computer instructions are used to cause the computer to execute the above-mentioned embodiments of the one aspect. video processing method.

According to another aspect of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements the video processing method described in the embodiments of the foregoing aspect.

It should be understood that the content described in this section is not intended to identify key or critical features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the present application will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present application. in:

1 is a schematic flowchart of a video processing method provided by an embodiment of the present application;

2 is a schematic flowchart of another video processing method provided by an embodiment of the present application;

3 is a schematic flowchart of another video processing method provided by an embodiment of the present application;

4 is a schematic diagram of generating global information of a video provided by a specific embodiment of the present application;

5 is a schematic flowchart of another video processing method provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a video processing apparatus provided by an embodiment of the present application; and

FIG. 7 is a block diagram of an electronic device of a video processing method according to an embodiment of the present application.

Detailed ways

Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

The video processing method, apparatus, electronic device, and storage medium according to the embodiments of the present application are described below with reference to the accompanying drawings.

Artificial intelligence is a discipline that studies the use of computers to simulate certain thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), both in the technical field of hardware and software. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies include computer vision technology, speech recognition technology, natural language processing technology, as well as deep learning, big data and other technologies. Processing technology, knowledge graph technology and other major directions.

Deep learning is a new research direction in the field of machine learning. Deep learning is to learn the inherent laws and representation levels of sample data, and the information obtained during these learning processes is of great help to the interpretation of data such as text, images, and sounds. Its ultimate goal is to enable machines to have the ability to analyze and learn like humans, and to recognize data such as words, images, and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies.

Computer vision is a science that studies how to make machines "see". More specifically, it refers to the use of cameras and computers instead of human eyes to identify, track, and measure targets, and further graphics processing to make computers. Processed into images that are more suitable for human observation or transmitted to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, trying to build artificial intelligence systems that can obtain 'information' from images or multi-dimensional data. The information referred to here refers to the information defined by Shannon (Shannon formula) that can be used to help make a "decision". Because perception can be viewed as extracting information from sensory signals, computer vision can also be viewed as the science of how to make artificial systems "perceive" from images or multidimensional data.

The video processing method provided by the embodiments of the present application may be performed by an electronic device, and the electronic device may be a PC (Personal Computer) computer, a tablet computer, or a palmtop computer, etc., which is not limited herein.

In this embodiment of the present application, the electronic device may be provided with a processing component, a storage component, and a driving component. Optionally, the driving component and the processing component may be integrated, the storage component may store an operating system, an application program or other program modules, and the processing component implements the video provided by the embodiments of the present application by executing the application program stored in the storage component. Approach.

FIG. 1 is a schematic flowchart of a video processing method provided by an embodiment of the present application.

The video processing method in the embodiment of the present application can also be executed by the video processing apparatus provided in the embodiment of the present application, and the apparatus can be configured in an electronic device to perform feature extraction on the obtained video, so as to obtain feature information of the video, and The boundary prediction model is called to predict the time sequence boundary of the feature information to generate the time sequence boundary of the video, and the video is segmented according to the time sequence boundary to generate the video segment of the video, so that the accuracy of the time sequence boundary prediction can be improved.

As a possible situation, the video processing method in the embodiment of the present application may also be executed on the server side, the server may be a cloud server, and the video processing method may be executed in the cloud.

As shown in Figure 1, the video processing method may include:

Step 101, acquiring a video.

In the embodiment of the present application, there may be multiple ways for the electronic device to obtain the video, wherein (1), the electronic device may obtain the video from the video providing device, for example, the electronic device may use the Uniform Resource Locator (Uniform Resource Locator) corresponding to the video. URL) to download videos from a video providing device, wherein the video providing device may include a digital universal CD player, an audio-visual CD player, a server, a U disk, a smart hard disk, a mobile phone, etc.; ②, the electronic device may store video, and the electronic device may Obtain the target video from the video stored by itself; 3. The electronic device can capture the video through the built-in camera to obtain the video. No limitation is made here.

As a possible situation, the above video may also be a video downloaded by a user through a related video website.

It should be noted that the video described in this embodiment may be a target video for which the user wants to perform time-series action positioning to produce behavior segments (ie, video segments).

Step 102: Perform feature extraction on the video to obtain feature information of the video.

In the embodiment of the present application, feature extraction may be performed on the video according to a preset feature extraction algorithm to obtain feature information of the video, wherein the preset feature extraction algorithm may be calibrated according to the actual situation.

Specifically, after acquiring the video, the electronic device may perform feature extraction on the video according to a preset feature extraction algorithm to acquire feature information of the video. The feature information may be feature sequence information of the video.

As a possible situation, after acquiring the video, the electronic device may further perform feature extraction on the video by using a feature extraction tool (eg, a plug-in) to obtain feature information of the video.

Step 103 , call the boundary prediction model to perform time-series boundary prediction on the feature information, so as to generate the time-series boundary of the video.

It should be noted that the boundary prediction model described in this embodiment may be trained in advance and pre-stored in the storage space of the electronic device to facilitate application retrieval, and the storage space is not limited to entity-based storage space For example, a hard disk, the above-mentioned storage space may also be a storage space (cloud storage space) of a network hard disk connected to the electronic device.

The training and generation of the above-mentioned boundary prediction model can be performed by a related server. The server can be a cloud server or a host of a computer. The server is connected to an electronic device that can execute the video processing method provided by the application embodiment. A communication connection is established therebetween, and the communication connection may be at least one of a wireless network connection and a wired network connection. The server can send the trained boundary prediction model to the electronic device, so that the electronic device can call it when needed, thereby greatly reducing the computing pressure of the electronic device.

Specifically, after acquiring the feature information of the video, the electronic device can firstly perform a boundary prediction model from its own storage space, and then input the feature information into the boundary prediction model, so that the feature information can be sequenced through the boundary prediction model. Boundary prediction to obtain the temporal boundary of the video output (generated) by the boundary prediction model.

Step 104 , segment the video according to the timing boundary to generate video segments of the video.

In the embodiment of the present application, the above timing boundaries may be multiple, and a video segment needs two timing boundaries, the start and the end, to be determined, that is, the timing boundaries corresponding to the start time and the end time respectively. That is, the above-mentioned timing boundaries may be multiple, and may be an even number.

Specifically, after obtaining the timing boundary of the video, the electronic device may segment the video according to the timing boundary to generate video segments of the video.

For example, assuming that there are multiple timing boundaries, after obtaining the multiple timing boundaries of the video, the electronic device can first analyze the multiple timing boundaries to determine the timing boundaries of multiple sets of start times and end times, and then The video may be segmented according to the timing boundaries of each set of start time and end time, respectively, to generate multiple video segments of the video.

As a result, relevant personnel can perform video highlight generation, video recommendation, and retrieval based on the video processing method of the embodiment of the present application.

In the embodiment of the present application, the video is first obtained, and feature extraction is performed on the video to obtain the feature information of the video, and then the boundary prediction model is called to perform temporal boundary prediction on the feature information to generate the temporal boundary of the video, and finally, according to the temporal boundary, the The video is segmented to generate video segments of the video. As a result, the accuracy of timing boundary prediction can be improved, thereby increasing the recall rate of timing nominations in unsegmented videos.

In order to clearly illustrate the previous embodiment, in an embodiment of the present application, as shown in FIG. 2 , feature extraction is performed on the video to obtain feature information of the video, which may include:

Step 201, acquiring a feature extraction model.

It should be noted that, the feature extraction model described in this embodiment may be trained in advance and pre-stored in the storage space of the electronic device to facilitate retrieval of applications.

Step 202, input the video to the feature extraction model.

Step 203 , perform feature extraction on the video by using the feature extraction model to obtain feature information of the video.

Specifically, after acquiring the video, the electronic device can call up (obtain) a feature extraction model from its own storage space, and input the video into the feature extraction model, and the feature extraction model performs feature extraction on the video to output Feature information of the video. Therefore, by assisting the extraction of video feature information by the feature extraction model, the recognition accuracy can be improved.

Further, in an embodiment of the present application, the boundary prediction model may be a boundary prediction model based on the Transformer mechanism, as shown in FIG. 3 , the boundary prediction model performs time series boundary prediction on the feature information through the following steps to generate a video. Timing Boundaries:

Step 301: Generate multiple feature vectors according to the feature information.

Specifically, after the electronic device inputs the feature information of the video into the boundary prediction model, the boundary prediction model can map the feature information into different vectors (that is, multiple Feature vector)

Step 302: Generate global information of the video according to the plurality of feature vectors.

In this embodiment of the present application, multiple feature vectors can be fused by calculating the similarity of different time series, so as to obtain the global information of the video.

In order to clearly illustrate the previous embodiment, in an embodiment of the present application, multiple feature vectors may include a first feature vector, a second feature vector, and a third feature vector, and the global information of the video is generated according to the multiple feature vectors, which may be The method includes: acquiring the dimension of the first feature vector, and generating global information of the video according to the first feature vector, the second feature vector, the third feature vector and the dimension.

It should be noted that, the first feature vector, the second feature vector, and the third feature vector described in this embodiment may be a Query vector, a Key vector, and a Value vector, respectively.

Specifically, referring to the figure, it is assumed that a plurality of feature vectors may include a first feature vector (Query vector), a second feature vector (Key vector), and a third feature vector (Value vector), wherein the above-mentioned boundary prediction model is obtained after obtaining After multiple eigenvectors, first obtain the dimension of the first eigenvector (Query vector), multiply the first eigenvector (Query vector) with the second eigenvector (Key vector) to obtain the first intermediate value, and then obtain the first intermediate value. An intermediate value is divided by the above-mentioned dimension to obtain the second intermediate value, and finally the second intermediate value is multiplied by the third feature vector (Value vector) to obtain the global information of the video.

Step 303, generating a video timing boundary according to the global information.

Specifically, after calculating the above-mentioned global information through the self-attention mechanism inside the Transformer, the boundary prediction model can perform timing boundary prediction according to the global information to generate the timing boundary of the video.

Therefore, the feature information of the video can be effectively predicted by the boundary prediction model, thereby improving the accuracy of the time series boundary prediction.

Further, in an embodiment of the present application, as shown in FIG. 5 , the boundary prediction model can be generated in the following manner:

In step 501, a sample video is obtained, and tags of multiple sample video segments in the sample video are obtained.

In the embodiment of the present application, there are multiple ways to obtain sample videos. Among them, videos can be downloaded through relevant video websites to obtain sample videos, and sample videos can also be created manually. For example, video cameras can be used to capture samples to obtain samples. Video, without any limitation here.

In the embodiment of the present application, the above-mentioned tags may be marked by relevant personnel in the sample video, and are pre-stored in the storage space of the electronic device to facilitate retrieval of applications.

Step 502, perform feature extraction on the sample video to obtain sample feature information of the sample video.

In the embodiment of the present application, feature extraction may be performed on the sample video according to the above-mentioned preset feature extraction algorithm, so as to obtain sample feature information of the sample video.

Specifically, after the sample video is obtained, tags of multiple sample video segments in the sample video can be obtained from the storage space of the sample video according to the sample video. Then, feature extraction can be performed on the sample video according to the above-mentioned preset feature extraction algorithm to obtain sample feature information of the sample video.

As a possible situation, after the labels of multiple sample video clips in the sample video are obtained, feature extraction may be performed on the sample video according to the above-mentioned feature extraction model to obtain sample feature information of the sample video.

As another possible situation, after the labels of multiple sample video clips in the sample video are obtained, a feature extraction tool (for example, a plug-in) can also be used to perform feature extraction on the sample video to obtain sample feature information of the sample video .

Step 503 , input the sample feature information into the boundary prediction model to generate a predicted boundary score.

Step 504: Generate a loss value according to the predicted boundary score and label, and train the boundary prediction model according to the loss value.

Specifically, after obtaining the sample feature information of the sample video, the sample feature information can be input into the boundary prediction model to generate the predicted boundary score, and then the loss value is generated according to the predicted boundary score and the above label, and according to the loss value The boundary prediction model is trained (for example, using stochastic gradient descent (SGD) to optimize the loss value, and the network weight layers and scaling parameters in the boundary prediction model are continuously updated until the loss value converges and training stops), thereby optimizing the boundary prediction model to improve the recognition accuracy.

It should be noted that the loss value described in this embodiment can be obtained by calculating the categorical cross entropy.

According to the video processing method of the embodiment of the present application, firstly obtain the video, and perform feature extraction on the video to obtain the feature information of the video, and then call the boundary prediction model to predict the time sequence boundary of the feature information to generate the time sequence boundary of the video, and finally according to Timing boundaries slice the video to generate video segments of the video. As a result, the accuracy of timing boundary prediction can be improved, thereby increasing the recall rate of timing nominations in unsegmented videos.

FIG. 6 is a schematic structural diagram of a video processing apparatus according to an embodiment of the present application.

The video processing apparatus according to the embodiment of the present application may be configured in an electronic device to perform feature extraction on the acquired video to obtain feature information of the video, and call a boundary prediction model to perform time-series boundary prediction on the feature information to generate a video and segment the video according to the timing boundary to generate video segments of the video, thereby improving the accuracy of timing boundary prediction.

As shown in FIG. 6 , the video processing apparatus 600 may include: a first obtaining module 610 , a second obtaining module 620 , a first generating module 630 and a second generating module 640 .

Wherein, the first acquisition module 610 is used for acquiring video.

In this embodiment of the present application, the first acquisition module 610 may acquire videos in multiple ways, wherein ①, the first acquisition module 610 may acquire videos from a video providing device, for example, an electronic device may use a unified resource corresponding to the video The locator (Uniform ResourceLocator, URL) downloads the video from the video providing device, wherein the video providing device may include a digital universal CD player, an audio-visual CD player, a server, a U disk, a smart hard disk, a mobile phone, etc.; ②, the electronic device can store If there is a video, the first acquisition module 610 can acquire the target video from the video stored in the electronic device; 3. The first acquisition module 610 can capture the video through a camera built in the electronic device to acquire the video. No limitation is made here.

As a possible situation, the above video may also be a video downloaded by a user through a related video website.

It should be noted that the video described in this embodiment may be a target video for which the user wants to perform time-series action positioning to produce behavior segments (ie, video segments).

The second obtaining module 620 is configured to perform feature extraction on the video to obtain feature information of the video.

In the embodiment of the present application, feature extraction may be performed on the video according to a preset feature extraction algorithm to obtain feature information of the video, wherein the preset feature extraction algorithm may be calibrated according to the actual situation.

Specifically, after the first obtaining module 610 obtains the video, the second obtaining module 620 may perform feature extraction on the video according to a preset feature extraction algorithm to obtain feature information of the video. The feature information may be feature sequence information of the video.

As a possible situation, after acquiring the video, the second acquiring module 620 may further perform feature extraction on the video by using a feature extraction tool (eg, a plug-in) to acquire feature information of the video.

The first generation module 630 is configured to call the boundary prediction model to perform temporal boundary prediction on the feature information, so as to generate the temporal boundary of the video.

It should be noted that the boundary prediction model described in this embodiment may be trained in advance and pre-stored in the storage space of the electronic device to facilitate application retrieval, and the storage space is not limited to entity-based storage space For example, a hard disk, the above-mentioned storage space may also be a storage space (cloud storage space) of a network hard disk connected to the electronic device.

The training and generation of the above-mentioned boundary prediction model can be performed by a related server. The server can be a cloud server or a computer host. A communication connection is established therebetween, and the communication connection may be at least one of a wireless network connection and a wired network connection. The server can send the trained boundary prediction model to the electronic device, so that the electronic device can call it when needed, thereby greatly reducing the computing pressure of the electronic device.

Specifically, after the second acquisition module 620 acquires the feature information of the video, the first generation module 630 can first predict the model from the boundary in its own storage space, and then input the feature information into the boundary prediction model, so as to pass the boundary The prediction model performs temporal boundary prediction on the feature information to obtain temporal boundaries of the video output (generated) by the boundary prediction model.

The second generating module 640 is configured to segment the video according to the time sequence boundary to generate video segments of the video.

In the embodiment of the present application, the above timing boundaries may be multiple, and a video segment needs two timing boundaries, the start and the end, to be determined, that is, the timing boundaries corresponding to the start time and the end time respectively. That is, the above-mentioned timing boundaries may be multiple, and may be an even number.

Specifically, after the first generation module 630 obtains the timing boundary of the video, the second generation module 640 may segment the video according to the timing boundary to generate video segments of the video.

For example, assuming that there are multiple timing boundaries, after the first generation module 630 obtains the multiple timing boundaries of the video, the second generation module 640 can first analyze the multiple timing boundaries to determine multiple sets of start times and the timing boundaries of the end time, and then the video may be segmented according to the timing boundaries of each set of start time and end time, respectively, to generate multiple video segments of the video.

In the embodiment of the present application, a video is acquired through a first acquisition module, and feature extraction is performed on the video through a second acquisition module to acquire feature information of the video, and then a boundary prediction model is invoked by the first generation module to perform a time-series boundary on the feature information. Prediction is used to generate the temporal boundary of the video, and finally the second generation module divides the video according to the temporal boundary to generate video segments of the video. As a result, the accuracy of timing boundary prediction can be improved, thereby increasing the recall rate of timing nominations in unsegmented videos.

In an embodiment of the present application, the second obtaining module 620 is specifically configured to: obtain a feature extraction model; input the video into the feature extraction model; perform feature extraction on the video through the feature extraction model to obtain feature information of the video.

In an embodiment of the present application, the boundary prediction model may be a boundary prediction model based on the Transformer mechanism. As shown in FIG. 6 , the first generation module 630 may include: a first generation unit 631 , a second generation unit 632 and a third generation unit 632 . Generation unit 633.

The first generating unit 631 is configured to generate multiple feature vectors according to the feature information.

The second generating unit 632 is configured to generate global information of the video according to the plurality of feature vectors.

The third generating unit 633 is configured to generate the timing boundary of the video according to the global information.

In an embodiment of the present application, the plurality of feature vectors may include a first feature vector, a second feature vector, and a third feature vector, and the second generating unit 632 is specifically configured to: obtain the dimension of the first feature vector; The feature vector, the second feature vector, the third feature vector and the dimension, generate global information of the video.

In an embodiment of the present application, as shown in FIG. 6 , the video processing apparatus 600 may further include: a training module 650, wherein the training module 650 is configured to generate a boundary prediction model by: acquiring a sample video, and acquiring a sample Labels of multiple sample video clips in the video; perform feature extraction on the sample video to obtain sample feature information of the sample video; input the sample feature information into the boundary prediction model to generate the predicted boundary score; generate the loss according to the predicted boundary score and label value, and train the boundary prediction model based on the loss value.

It should be noted that, the foregoing explanation of the video processing method embodiment is also applicable to the video processing apparatus of this embodiment, and details are not repeated here.

In the video processing apparatus according to the embodiment of the present application, the video is acquired by the first acquisition module, the feature extraction is performed on the video by the second acquisition module to acquire the feature information of the video, and then the feature information is processed by the first generation module by calling the boundary prediction model. The timing boundary is predicted to generate the timing boundary of the video, and finally the second generation module divides the video according to the timing boundary to generate video segments of the video. As a result, the accuracy of timing boundary prediction can be improved, thereby increasing the recall rate of timing nominations in unsegmented videos.

In the technical solution of this application, the acquisition, storage and application of the user's personal information involved are in compliance with the relevant laws and regulations, and do not violate public order and good customs.

According to the embodiments of the present application, the present application further provides an electronic device, a readable storage medium, and a computer program product.

FIG. 7 shows a schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the application described and/or claimed herein.

As shown in FIG. 7 , the device 700 includes a computing unit 701 that can be executed according to a computer program stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage unit 708 Various appropriate actions and handling. In the RAM 703, various programs and data necessary for the operation of the device 700 can also be stored. The computing unit 701 , the ROM 702 , and the RAM 703 are connected to each other through a bus 704 . An input/output (I/O) interface 705 is also connected to bus 704 .

Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc. ; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 701 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as video processing methods. For example, in some embodiments, the video processing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 700 via ROM 702 and/or communication unit 709 . When a computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the video processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the video processing method by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of this application, a machine-readable medium may be a tangible medium that may contain or store the program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), the Internet, and blockchain networks.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present application can be performed in parallel, sequentially or in different orders, and as long as the desired results of the technical solutions disclosed in the present application can be achieved, no limitation is imposed herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (13)

1. A video processing method, comprising: get video; performing feature extraction on the video to obtain feature information of the video; Calling a boundary prediction model to perform temporal boundary prediction on the feature information to generate a temporal boundary of the video; and The video is segmented according to the timing boundary to generate video segments of the video. 2. The method according to claim 1, wherein the performing feature extraction on the video to obtain feature information of the video, comprising: Get the feature extraction model; inputting the video to the feature extraction model; Feature extraction is performed on the video through the feature extraction model to obtain feature information of the video. 3. The method according to claim 1, wherein the boundary prediction model is a boundary prediction model based on a Transformer mechanism, and the boundary prediction model performs time series boundary prediction on the feature information through the following steps to generate the video The timing boundaries of : generating a plurality of feature vectors according to the feature information; generating global information of the video according to the plurality of feature vectors; Timing boundaries of the video are generated based on the global information. 4. The method of claim 3, wherein the plurality of feature vectors comprises a first feature vector, a second feature vector and a third feature vector, and the global image of the video is generated according to the plurality of feature vectors information, including: Get the dimension of the first feature vector; Global information of the video is generated according to the first feature vector, the second feature vector, the third feature vector and the dimension. 5. The method of claim 1, wherein the boundary prediction model is generated by: Obtain a sample video, and obtain the labels of multiple sample video clips in the sample video; performing feature extraction on the sample video to obtain sample feature information of the sample video; inputting the sample feature information into the boundary prediction model to generate a predicted boundary score; A loss value is generated from the predicted boundary score and the label, and the boundary prediction model is trained according to the loss value. 6. A video processing device, comprising: The first acquisition module is used to acquire the video; A second acquisition module, configured to perform feature extraction on the video to acquire feature information of the video; a first generation module for invoking a boundary prediction model to perform time-series boundary prediction on the feature information to generate a time-series boundary of the video; and A second generating module, configured to segment the video according to the timing boundary to generate video segments of the video. 7. The apparatus according to claim 6, wherein the second obtaining module is specifically used for: Get the feature extraction model; inputting the video to the feature extraction model; Feature extraction is performed on the video through the feature extraction model to obtain feature information of the video. 8. The device according to claim 6, wherein the boundary prediction model is a boundary prediction model based on a Transformer mechanism, and the first generation module comprises: a first generating unit for generating a plurality of feature vectors according to the feature information; a second generating unit, configured to generate global information of the video according to the plurality of feature vectors; A third generating unit, configured to generate a timing boundary of the video according to the global information. 9. The apparatus according to claim 8, wherein the plurality of eigenvectors comprise a first eigenvector, a second eigenvector and a third eigenvector, and the second generating unit is specifically configured to: Get the dimension of the first feature vector; Global information of the video is generated according to the first feature vector, the second feature vector, the third feature vector and the dimension. 10. The apparatus of claim 6, further comprising: A training module for generating the boundary prediction model by: Obtain a sample video, and obtain the labels of multiple sample video clips in the sample video; performing feature extraction on the sample video to obtain sample feature information of the sample video; inputting the sample feature information into the boundary prediction model to generate a predicted boundary score; A loss value is generated from the predicted boundary score and the label, and the boundary prediction model is trained according to the loss value. 11. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-5 video processing method. 12. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the video processing method of any one of claims 1-5. 13. A computer program product comprising a computer program which, when executed by a processor, implements the video processing method of any one of claims 1-5.

Images