WO2020215722A1 - Method and device for video processing, electronic device, and computer-readable storage medium - Google Patents

Abstract

Disclosed are a method and device for video processing, an electronic device, and a computer-readable storage medium. With the embodiments of the present application, a first video file of at least one first user and a second video file of a second user are acquired, the first video file and the second video file are traversed and captured on the basis of at least one window duration to acquire a first video candidate set, acquired on the basis of a first vector and a second vector in a unit time cycle of the first video file for expressing a state of the user in the video is a fourth vector that characterizes the probability distribution values, multiple first video clips and corresponding second video clips are selected from the first video candidate set on the basis of the fourth vector, and a target video clip is thus synthesized. As such, high-quality or highlight video clips can be extracted to fully embody exchanges between the first user and the second user, thus providing enriched user experience.

Description

Video processing method and device, electronic equipment and computer readable storage medium

This application claims the priority of a Chinese patent application filed on April 26, 2019, with the application number 2019103456254, and the title of the invention "video processing method and device, electronic equipment, and computer-readable storage medium", the entire content of which is incorporated by reference Incorporated in this application.

Technical field

The present invention relates to the technical field of video processing, in particular to a video processing method and device, electronic equipment and computer-readable storage media.

Background technique

With the rapid development of Internet technology and education informatization, the application of mobile multimedia teaching platforms has become more and more extensive. In the prior art, the extraction of wonderful videos is mainly to generate a model by training samples, and then use the model to process the teaching videos. Therefore, it is impossible to ensure that the wonderful moments of the interaction between the teacher and the students are well captured. If it is extracted manually, the workload is huge and it is almost impossible to complete. As a result, it is impossible to accurately extract the wonderful video clips of the interaction between the teacher and the student, and further leads to a poor user experience.

Summary of the invention

In view of this, embodiments of the present invention provide a video processing method and device, electronic equipment, and computer-readable storage medium, which can extract high-quality or wonderful video clips based on the interaction between a first user and a second user, and further provide more Rich user experience.

According to a first aspect of the embodiments of the present invention, a video processing method is provided, including:

Acquiring at least one first video file of the first user and at least one second video file of the second user;

Traverse and intercept the first video file and the second video file according to at least one window duration to obtain a first video candidate set. The first video candidate set includes a plurality of video clip pairs, and each video clip pair Including a first video clip and a corresponding second video clip with the same window duration and the same time axis position;

Obtain a first vector corresponding to each unit time period according to the first video file, and obtain a second vector corresponding to each unit time period according to the second video file, and the first vector is used to characterize the corresponding unit A first user state in a time period, where the second vector is used to represent a second user state in a corresponding unit time period;

Obtaining a third vector for each unit time period according to the first vector and the second vector corresponding to each unit time period;

Determine the fourth vector according to the time axis position corresponding to each video segment pair and the third vector;

Selecting multiple first video segments and second video segments from the first video candidate set according to the fourth vector;

Obtain the target video segment according to the selected first video segment and the second video segment.

Preferably, obtaining the first vector corresponding to each unit time period according to the first video file includes:

Determining the video data and audio data corresponding to the target unit time period according to the first video file;

Performing face recognition on a plurality of image frames of the video data respectively, and obtaining face state parameters corresponding to each image frame;

Perform voice recognition on the audio data to obtain voice distribution parameters;

The first vector is determined according to the face state parameter and the voice distribution parameter.

Preferably, obtaining the second vector corresponding to each unit time period according to the second video file includes:

Determine the video data and audio data corresponding to the target unit time period according to the second video file;

Performing face recognition on multiple image frames of the video data, respectively, to obtain face state parameters corresponding to each image frame;

Perform voice recognition on the audio data to obtain voice distribution parameters;

Obtaining the second vector according to the face state parameter and the voice distribution parameter.

Preferably, the face state parameter includes a first value that characterizes the appearance of the face and a second value that characterizes the expression state of the face.

Preferably, the obtaining a third vector according to the first vector and the second vector corresponding to each unit time period includes:

The first vector and the second vector corresponding to the same unit time period are combined into a third vector corresponding to the unit time period.

Preferably, determining the fourth vector according to the time axis position corresponding to each video segment pair and the third vector includes:

Determine the target video clip pair;

Determining corresponding multiple target unit time periods according to the time axis position of the target video clip pair;

Calculating the sum vector of the third vector of the multiple target unit time periods;

Determine the element random distribution function corresponding to each element according to the average vector and the window duration, and the average vector is calculated by averaging the third vector in each unit time period in the multiple video files;

The fourth vector of the target video segment pair is determined according to the element random distribution function corresponding to each element and the sum vector, wherein each element of the fourth vector is the random element of the sum vector in the corresponding element. The quantile value in the distribution function.

Preferably, selecting a plurality of first video clips and second video clips from the first video candidate set according to the fourth vector includes:

Determining a filtered video segment pair according to the fourth vector;

The filtered video segment pair is removed from the first video candidate set to obtain a second video candidate set.

Preferably, the determining to filter the video clip according to the fourth vector includes:

In response to any one of the elements in the fourth vector being less than the corresponding quantile threshold, the corresponding video segment pair is determined as the filtered video segment.

Preferably, the element random distribution function is a binomial distribution function with a corresponding element in the average vector as a mean value and a length matching the window duration.

Preferably, selecting a plurality of first video segments and second video segments from the first video candidate set according to the fourth vector further includes:

Calculating the score value of each video segment pair in the second video candidate set according to the fourth vector;

Sorting and filtering multiple first video clips in the second video candidate set according to the score value until the number of remaining first video clips in the second video candidate set meets a predetermined condition.

According to a second aspect of the embodiments of the present invention, there is provided a video processing device, including:

The first obtaining unit is configured to obtain at least one first video file of the first user and at least one second video file of the second user;

The interception unit is configured to traverse and intercept the first video file and the second video file according to at least one window duration to obtain a first video candidate set, where the first video candidate set includes a plurality of video clip pairs, each The video clip pair includes a first video clip and a corresponding second video clip with the same window duration and the same time axis position;

The second obtaining unit is configured to obtain a first vector corresponding to each unit time period according to the first video file, and obtain a second vector corresponding to each unit time period according to the second video file. The vector is used to characterize the first user state in the corresponding unit time period, and the second vector is used to characterize the second user state in the corresponding unit time period;

A third obtaining unit, configured to obtain a third vector according to the first vector and the second vector corresponding to each unit time period;

A fourth acquiring unit, configured to determine a fourth vector according to the time axis position corresponding to each video segment pair and the third vector;

A selecting unit, configured to select a plurality of first video clips and second video clips from the first video candidate set according to the fourth vector; and

The synthesis unit obtains the target video segment according to the selected first video segment and the second video segment.

According to a third aspect of the embodiments of the present invention, there is provided an electronic device, including a memory and a processor, where the memory is used to store one or more computer program instructions, where one or more computer program instructions are executed by the processor to Implement the method as described in the first aspect.

According to a fourth aspect of the embodiments of the present invention, there is provided a computer-readable storage medium on which computer program instructions are stored, wherein the computer program instructions implement the method as described in the first aspect when executed by a processor.

The embodiment of the application obtains the first video file of at least one first user and the second video file of the second user, and traverses and intercepts the first video file and the second video file according to the at least one window duration to obtain the first video candidate set According to the first vector and the second vector representing the state of the user in the video in the unit time period of the first video file, the fourth vector representing the probability distribution value is obtained, and the fourth vector is selected from the first video candidate set according to the fourth vector. A first video segment and a corresponding second video segment, and then synthesize the target video segment. As a result, the target video segment (such as high-quality or wonderful video segment) can be extracted to fully reflect the interaction between the first user and the second user, thereby providing a richer user experience.

Description of the drawings

Through the following description of the embodiments of the present invention with reference to the accompanying drawings, the above and other objectives, features and advantages of the present invention will be clearer, in the accompanying drawings:

FIG. 1 is a flowchart of a video processing method according to an embodiment of the present invention;

Figure 2 is a data flow diagram of a video processing method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a video processing device according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The present invention will be described below based on embodiments, but the present invention is not limited to these embodiments. In the following detailed description of the present invention, some specific details are described in detail. Those skilled in the art can fully understand the present invention without the description of these details. In order to avoid obscuring the essence of the present invention, the well-known methods, processes, procedures, components and circuits are not described in detail.

In addition, those of ordinary skill in the art should understand that the drawings provided herein are for illustrative purposes, and the drawings are not necessarily drawn to scale.

Unless the context clearly requires, the words "including", "including" and other similar words in the entire specification and claims should be interpreted as inclusive rather than exclusive or exhaustive meanings; in other words, "including but not limited to" Meaning.

In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

Fig. 1 is a flowchart of a video processing method according to an embodiment of the present invention. As shown in Figure 1, the execution subject of this method is a server, and the video processing method of this embodiment includes:

Step S110: Obtain at least one first video file of the first user and at least one second video file of the second user.

In the embodiment of the present invention, the server obtains at least one first video file of the first user and the second video file of the second user. Here, the first user may be a student, and the number of the first user may be one, two, four or more, which is not limited in the present invention. The second user may be a teacher, and the number of second users may be one. Preferably, in this embodiment, the number of first users is four, and the number of second users is one, that is, the online teaching mode of the embodiment of the present invention is "one to four".

Further, the first video file may be a multimedia file when the first user performs online learning, and it may include real-time audio and video information of the first user. The second video file may be a multimedia file during online teaching by the second user, and it may include real-time audio and video information of the second user. In addition, the formats of the first video file and the second video file may include, but are not limited to, .AVI, .MOV, .RM, .MPEG, .ASF, etc.

Step S120: Traverse and intercept the first video file and the second video file according to at least one window duration to obtain a first video candidate set. The first video candidate set includes multiple video clips, and the multiple video clips include windows with the same duration and Multiple first video clips and multiple corresponding second video clips with the same time axis position.

In the embodiment of the present invention, the window duration is represented by t, and the first video candidate set is represented by R. The server uses t as the window duration to traverse and intercept the first video file and the second video file to obtain the first video candidate set. The first video candidate set is represented by R, R={Rt}, where Rt is multiple video clips in the first video candidate set, and the multiple video clips include multiple first video clips and multiple second video clips, Wherein, the plurality of second video clips correspond to the plurality of first video clips one-to-one. The second video segment and the first video segment corresponding to each other have the same window duration and time axis position.

Here, the window duration may be, for example, 10 seconds, 13 seconds, 16 seconds, 19 seconds, 22 seconds, 25 seconds, etc. Assuming that the value range of the window duration t is [10, 13, 16, 19, 22, 25], first use 10 seconds as the window duration to slide and intercept the video clips according to a predetermined sliding step (eg: 1 second), and then Use 13 seconds as the window duration to slide and capture video clips in accordance with the sliding step (e.g. 1 second), and so on. The finally obtained first video candidate set R can be expressed as {0-10s, 1-11s,..., 0-13s, 1-14s,...}. That is, the first video candidate set may include multiple video clips with a window duration of 10 seconds, or multiple video clips with a window duration of 13 seconds, and may also include a window duration of 16 seconds, 19 seconds, and 22 seconds. And multiple video clips of 25 seconds.

It should be noted that the window time length and the sliding step length may be the default time length of the system, or the time length preset by the administrator according to needs, and the present invention does not limit this. In addition, it should be noted that the first video candidate set is not limited to include multiple video clips with different window durations as described above, but may only include video clips with the same window duration, for example, the first video candidate set It may only include multiple video clips with a window length of 10 seconds, or only include multiple video clips with a window length of 13 seconds.

Step S130: Obtain a first vector corresponding to each unit time period according to the first video file, and obtain a second vector corresponding to each unit time period according to the second video file, where the first vector is used to characterize Corresponding to the first user state in the unit time period, the second vector is used to represent the second user state in the corresponding unit time period.

It should be noted that there is no dependency relationship between step S130 and step S120. The order of execution of the two can be performed simultaneously or in a predetermined order. For example, step S120 is executed first, and then step S130 is executed. Step S130 is performed first, and then step S120 is performed.

In the embodiment of the present invention, the server analyzes multiple video clips according to a predetermined unit time period (e.g., in a second-by-second manner) and/or a set number of frames (e.g., in a frame-by-frame manner) to obtain each The first vector corresponding to each time period, where the first vector is represented by Vs, is used to characterize the state of the first user or the second user of each video segment in the multiple video segments.

In an optional implementation of this embodiment, the state of the first user or the second user is represented based on three dimensional information of voice, facial appearance, and facial expression. Here, the status of the first user or the second user includes: in the video clip, whether the first user or the second user is talking at each set time period (for example, analyzing and judging whether the first user or the second user is talking every second in a second-by-second manner) ), whether the face of the first user or the second user appears in each frame, and whether the expression of the first user or the second user is happy, etc.

Specifically, obtaining the first vector corresponding to each unit time period according to the first video file in step S130 includes:

Step S131: Determine video data and audio data corresponding to the target unit time period according to the first video file.

Step S132: Perform face recognition on multiple image frames of the video data respectively, and obtain face state parameters corresponding to each image frame.

Specifically, the face state parameter includes a first value that characterizes the appearance of the face and a second value that characterizes the expression state of the face.

Step S133: Perform voice recognition on the audio data to obtain voice distribution parameters.

Step S134: Determine the first vector according to the face state parameter and the voice distribution parameter.

Face recognition is a kind of biometric recognition technology based on human facial feature information. It uses a camera or camera to collect images or video streams containing human faces, and automatically detects and tracks the human face in the image, and then detects Face recognition and other related technologies. Face recognition algorithms can include, but are not limited to, recognition algorithms based on facial feature points (Feature-based recognition algorithms), recognition algorithms based on the entire face image (Appearance-based recognition algorithms), and template-based recognition algorithms (Template- based on recognition algorithms), recognition algorithms using neural networks, based on light estimation model theory, etc. The face state parameter refers to whether a human face appears every second, and whether the facial expression is happy.

Speech recognition is to convert a piece of speech signal into corresponding text information. The system mainly includes four parts: feature extraction, acoustic model, language model, dictionary and decoding. In addition, in order to extract features more effectively, it is necessary to The sound signal performs audio data preprocessing such as filtering and framing to appropriately extract the audio signal that needs to be analyzed from the original signal; the feature extraction work converts the sound signal from the time domain to the frequency domain to provide appropriate features for the acoustic model Vector; the acoustic model calculates the score of each feature vector on the acoustic feature according to the acoustic characteristics; while the language model calculates the probability of the sound signal corresponding to the possible sequence of phrases according to the linguistic theory; finally, according to the existing dictionary, Decode the phrase sequence to get the final possible text representation. Speech recognition algorithms may include, but are not limited to, Gaussian Mixed Model (GMM) algorithms, Dynamic Time Warping (DTW) algorithms, Connectionist temporal classification (CTC) algorithms, etc. The speech distribution parameter refers to whether someone is speaking every second.

Take obtaining the first vector as an example. In this step, for the first video file, extract the audio information and video information from it, and analyze it according to a predetermined unit time period (for example, a period of 1 second) to obtain information from voice, human The three dimensions of face appearance and facial expressions obtain the attribute information of the video part (including audio and video) corresponding to the time period, denoted as [Ss, Fs, Es]. Among them, Ss represents the state of the voice in the corresponding unit time period, Fs represents the appearance of the face in the corresponding unit time period; Es represents the facial expression in the corresponding unit time period.

Specifically, in the voice dimension, for each unit time period (for example, every second), it is determined by voice analysis whether the first user is continuously speaking in the audio data of the video segment, and the determination result is represented by Ss. For example, for 1 second in a video clip, if voice information is continuously detected during this time period, it means that in the clip, the first user is speaking during this time period, so Ss is 1, otherwise the value is zero.

In the face appearance situation and the face expression dimension, the second-by-second analysis is performed to obtain information that characterizes the face appearance situation and the face expression dimension. First, extract all image frames or extract some image frames from the video data of each second of the video file. For each extracted image frame, it is determined by image recognition whether the face of the first user appears in each image frame and whether the facial expression is happy. Use Ff to indicate whether a human face appears in an image frame, and use Ef to indicate whether the facial expression is happy in an image frame. Among them, the value of Ff is 0 or 1, and the value of Ef is 0 or 1. Thus, for each image frame extracted in each second of the video, a corresponding [Ff, Ef] can be obtained. For example, taking 24 frames of image data per second as an example, for each second in the video clip, the images of each image frame of 24 frames can be extracted, and face recognition can be performed respectively to obtain 24 [Ff, Ef] components the sequence of. For another example, taking partial image frames (such as 8 frames) extracted from 24 frames of image data per second as an example, 8 frames of images can be extracted at intervals for each second in a video clip, and face recognition can be performed separately. Get 8 sequences composed of [Ff,Ef].

Then, the sequence of the above-mentioned [Ff, Ef] values corresponding to each second of the video data in the video slice pair is combined for a predetermined period of time (when the predetermined period of time is 1 second, the combination is performed second by second), that is, if every second Ff contains 2 or more 1 values, then the face value Fs or expression value Es in that second is 1; otherwise, it is 0. Finally, the above results are combined to obtain the first vector corresponding to each unit time period, the first vector is represented by Vs, and Vs=[Ss, Fs, Es].

For example, for the second second of a first video clip, if voice information is continuously detected, Ss=1. At the same time, if a face is detected in 24 frames of the 24 frames in this second, then Fs=1, and at the same time , If there are 12 frames detected that the facial expression is a smile, then Es=1. Therefore, the first vector Vs of the second second of the above-mentioned first video segment=[1,1,1].

In step S130, obtaining the second vector corresponding to each unit time period according to the second video file specifically includes:

Step S135: Determine the video data and audio data corresponding to the target unit time period according to the second video file.

Step S136: Perform face recognition on multiple image frames of the video data respectively, and obtain face state parameters corresponding to each image frame.

Step S137: Perform voice recognition on the audio data to obtain voice distribution parameters.

Step S138: Obtain the second vector according to the face state parameter and the voice distribution parameter.

It should be understood that the execution order of steps S131 to S134 and steps s135 to S138 is not distinguished, and they can be executed in parallel or in a set sequence.

Thus, for every second in the first video file, a corresponding first vector Vss can be obtained. For each second in the second video file, a corresponding second vector Vts can be obtained.

Step S140: Obtain a third vector for each unit time period according to the first vector and the second vector corresponding to each unit time period.

In this step, the third vector can be obtained by combining Vss and Vts. As mentioned above, the third vector is a 6-dimensional vector, which can characterize the relationship between the first user (such as a student) and the second user (such as a teacher) in the same unit time period in the first video file and the second video file. status.

In this embodiment, a third vector with a dimension of 6 can be obtained by combining the first vector of the first video file and the second vector of the second video file with the same time axis coordinate. The third vector includes voice data, face data and expression data of the first user and voice data, face data and expression data of the second user.

Step S150: Determine a fourth vector according to the time axis position corresponding to each video segment pair and the third vector.

Specifically, this step includes the following sub-steps:

Step S151: Determine the target video segment pair.

Step S152: Determine corresponding multiple target unit time periods according to the time axis position of the target video clip pair.

Step S153: Calculate the sum vector of the third vector of the multiple target unit time periods.

Step S154: Determine the element random distribution function corresponding to each element according to the average vector and the window duration, and the average vector is calculated based on the average calculation of the third vector of each unit time period in the multiple video files.

Step S155: Determine a fourth vector of the target video segment pair according to the element random distribution function corresponding to each element and the sum vector, where each element of the fourth vector is the corresponding element of the sum vector The quantile value in the random distribution function of the elements.

Specifically, in step S153, for each unit time period (that is, every second) in each video segment pair, a corresponding third vector can be obtained. Then, by summing the third vector of multiple unit time periods in the time axis covered by a video clip (that is, summing each element of the third vector), a video clip can be obtained. For the corresponding fourth vector.

Specifically, in step S154, a large number (for example, 10,000) video files similar to the first video file and the second video file may be extracted in advance to determine the average vector. For network classroom application scenarios, a large number of historical teaching videos can be obtained. The teaching videos include student video files and teacher video files. For the above video files, analyze and merge according to the unit time period to obtain each unit time period The third vector. Then average the third vector to get the mean value.

The average vector can be represented by Vs,avg, and Vs,avg=[Ss,savg,Fs,savg,Es,savg,Ss,tavg,Fs,tavg,Es,tavg], where Ss,savg means the first The average value of the user’s voice data, Fs, savg represents the average value of the first user’s face data, Es, savg represents the average value of the first user’s facial data, Ss, tavg represents the average value of the second user’s voice data , Fs, tavg represents the average value of the face data of the second user, and Es, tavg represents the average value of the facial expression data of the second user.

In this embodiment, it is assumed that the characteristics of voice and expression conform to the binomial distribution. Therefore, in step S154, the element random distribution function of each element corresponding to different time window lengths can be obtained according to the average value in the average value vector obtained above and the corresponding length of the video segment.

The binomial distribution is to repeat n independent Bernoulli tests. In each experiment, there are only two possible outcomes, and the two outcomes are opposite to each other and independent of each other. The probability of the occurrence of the event remains unchanged in each independent experiment. , Then this series of experiments is collectively called n-fold Bernoulli experiment. When the number of trials is 1, the binomial distribution obeys the 0-1 distribution. The binomial distribution is determined by the mean and the number or length of experiments.

It should be understood that those skilled in the art can also use other types of element random distribution functions that meet the characteristics of the sounds and expressions in the video.

In this embodiment, it is assumed that each element conforms to the binomial distribution B(t, avg), where t is the window duration of the video segment. The value interval of the independent variable of B is [0, t].

Thus, according to the mean value of each element in the third vector and the length of the predetermined time window, six mutually independent binomial distributions can be determined, and the element random distribution vector B=[Bsst, Bsft, Bset, Btst, Btft, Btet].

In step S155, for each element random distribution function, its quantile value in the corresponding element random distribution function can be determined according to the corresponding element value in the third vector, thereby determining the fourth vector. That is, each element of the fourth vector is the quantile value of the corresponding element of the sum vector in the corresponding element random distribution function.

Step S160: Select multiple first video clips and second video clips from the first video candidate set according to the fourth vector.

In the embodiment of the present invention, the server screens and sorts multiple first video clips in the first video candidate set according to the fourth vector, and selects multiple first video clips according to the screening and ranking results (for example, the top three The first video segment) serves as the first target video segment.

Specifically, step S160 includes the following sub-steps:

Step S161: Determine a filtered video segment pair according to the fourth vector.

In an optional implementation manner, in response to any item of each element in the fourth vector being less than a corresponding quantile threshold, the corresponding video segment pair is determined as the filtered video segment. Preferably, the filter video segment pair is determined according to the screening conditions of Bsst<0.4, Bsft<0.4, Bset<0.2, Btst<0.4, Btft<0.4, and Btet<0.2.

Step S162 removes the filtered video segment pair from the first video candidate set to obtain a second video candidate set.

Preferably, step S160 further includes:

Step S163: Calculate the score value of each video segment pair in the second video candidate set according to the fourth vector.

In an optional implementation manner, each video segment in the second video candidate set is added to each element of the corresponding fourth vector to obtain the score value.

Step S164: Sort and filter multiple first video clips in the second video candidate set according to the score value until the number of remaining first video clips in the second video candidate set meets a predetermined condition.

In an optional implementation manner, the top N first video clips and the corresponding second video clips can be directly selected as the basis for the next step, where N≥1 and an integer.

In another optional implementation manner, iteratively, in each cycle, the video clip with the highest score is selected, and all the time axes are removed from the second video candidate set, and there is overlap with the video clip with the highest score. For video segments, update the second video candidate set, and then enter the next iteration, until the number of remaining first video segments in the second video candidate set meets a predetermined condition.

Therefore, the embodiment of the present application obtains the first video file of at least one first user and the second video file of the second user, and traverses and intercepts the first video file and the second video file according to the at least one window duration to obtain the first video file. Video candidate set, according to the first vector and second vector representing the state of the user in the video in the unit time period of the first video file, obtain the fourth vector representing the probability distribution value of the first video file, and obtain the fourth vector representing the probability distribution value from the first video candidate according to the fourth vector Collectively select multiple first video clips and corresponding second video clips, and then synthesize the target video clips. In this way, high-quality or wonderful video clips can be extracted to fully embody the interaction between the first user and the second user, thereby providing a richer user experience.

In another embodiment of the present invention, the video processing method of FIG. 1 further includes:

Step 170: Obtain a target video segment according to the selected first video segment and second video segment.

Specifically, the target video segment is obtained by splicing multiple first video segments and second video segments selected from the first video candidate set. For example, if three first video clips such as 0-10s, 15-33s, and 35-57s are selected from the first video candidate set as the first target video clip, the corresponding second target video clip also includes 0- Three second video clips of 10s, 15-33s and 35-57s.

Fig. 2 is a data flow diagram of a method according to an embodiment of the present invention. In the following, with reference to FIG. 2, the data processing process of the embodiment of the present invention will be exemplified by taking a student video in a network classroom and a teacher teaching video synchronized recording as an example.

In step S110, the first video file S of the first user (the student's video file in this example) and the second user's second video file T (the teacher's video file in this example) are acquired.

Thus, a pair of video files is actually obtained.

In step S120, video clips are clipped by sliding through multiple different window durations to obtain the first video candidate set. In this example, two window durations of 10s and 13s are used for sliding interception. First, by sliding the first video file S and the second video file T through the 10s window, the first video file with the time axis {0-10s, 1-11s, 2-12s,...} can be obtained. Video clip and second video clip. Then slide the first video file S and the second video file T through the 13s window, and you can get the first video segment and the second video segment with the applied axis {0-13s, 1-14s, 2-15s,...} Video clips. The duration of the video clip is the same as the duration of the window used for sliding capture. The first video segment and the second video segment with the same time axis constitute a video segment pair.

In step S130, the data of the first video file and the second video file are analyzed second by second (that is, the unit time period is 1 second). Acquire the first vector Vss corresponding to each second in the first video file and the second vector Vts corresponding to each of the second video files.

For example, for the 5th second of the timeline, the first vector Vss of the first video file = [1,0,0], that is, the voice continues to be detected in this second, but the student's face does not appear, which is impossible A smiling face is detected, and at the same time, the second vector Vts of the second video file=[1, 1, 0], that is, the voice is continuously detected in this second, and the teacher's face is also detected, but no smiling face is detected.

In step S140, the first vector Vss and the second vector Vts of each second are merged into a third vector Vs.

For example, for the 5th second on the time axis, the third vector Vs=[1, 0, 0, 1, 1, 0]. Similarly, for the 6th second on the time axis, the third vector Vs=[1,1,1,1,1,1].

In step S150, a fourth vector is determined according to the time axis position corresponding to each video segment pair and the third vector.

First, the above-mentioned window duration t is combined with the predetermined average value of each dimension of the third vector Vs and avg of a large number of other video files to generate six binomial distributions, using B Means, B=[Bsst, Bsft, Bset, Btst, Btft, Btet]. In this example, Bsst is the random distribution function of the elements of the student’s voice, Bsft is the random distribution function of the elements of the student’s face, Bset is the random distribution function of the elements of the student’s facial expressions, and Btst is the random distribution of the elements of the teacher’s voice Function, Btft is the random distribution function of the elements of the teacher's face, and Bset is the random distribution function of the elements of the teacher's face. The above-mentioned element random distribution function is determined according to the pre-calculated mean value and the corresponding window duration. Taking Bsst as an example, Bsst obeys the binomial distribution B(t, savg), and the range of the independent variable of B is [0, t].

Next, each solution of the multiple video segments Rt (that is, the video segment pair) is respectively summed in six dimensions to obtain the sum vector corresponding to the video segment pair Rt. For example, for the first video segment and the second video segment in the 0-10s time interval. The third vector Vs1-Vs10 corresponding to each second is added to obtain a six-dimensional sum vector.

Based on the above-mentioned sum vector, the quantile vector (that is, the fourth vector) of each feasible solution in B can be calculated. Specifically, the quantile value of the element value in the sum vector in the random distribution function of the corresponding element is calculated to obtain a fourth vector composed of six quantile values. For example, if the first element in the sum vector is 4, calculate the median value of 4 in the Bsst distribution.

In step S160, a plurality of first video clips and second video clips are selected from the first video candidate set according to the fourth vector.

In this example, feasible solutions are filtered according to predetermined filtering conditions. If satisfy any of Bsst<0.4, Bs,f,t<0.4, Bs,e,t<0.2, Bt,s,t<0.4, Bt,f,t<0.4, Bt,e,t<0.2, The video segment is filtered out and the remaining videos are retained. At this time, the first video candidate set becomes the second video candidate set, denoted by R', and R'={Rt'}, where Rt' denotes remaining Multiple video clips in the video candidate set R'. It should be noted that 0.4 and 0.2 are preset quantile values, which may be system default values, or may be preset by the administrator according to needs, and the present invention does not limit this.

Further, for the second video candidate set R', the six elements in the fourth vector of each video segment pair (that is, the solution) are summed to obtain a sum. Then, sort based on the result of the summation to extract the maximum value, remove all candidate sets in R'that overlap with the time period represented by the maximum value to obtain a new R', and repeat this step until three video clips are taken out.

For example, there are 6 video segment pairs r1-r6 in the second video candidate set R', and the corresponding fourth vectors are b1-b6 respectively. In this step, for each fourth vector b1-b6, the 6 elements in the vector are summed to obtain the sum value s1-s6. For example, b1={0.5,0.5,0.3,0.5,0.5,0.4}, the corresponding s1=0.5+0.5+0.3+0.5+0.5+0.4=2.7. Then, s1-s6 are sorted, and if s2 is the largest, other video segment pairs that overlap the time axis and video segment pair r2 (s2 and its corresponding) are removed, and the second video candidate set R'is updated. Iteratively executes until the number of remaining video segment pairs in the updated video candidate set R'meets the requirement (for example, there are 3 remaining).

Further, taking the voice data of the first user as an example, assuming that the window duration is 10s, the binomial distribution function satisfied by the voice data of the first user is B(10, 0.2), and the corresponding distribution and quantile values are:

prb(0): 0.1074, 0.1074

prb(1): 0.2684, 0.3758

prb(2): 0.3020, 0.6778

prb(3): 0.2013, 0.8791

prb(4): 0.0881, 0.9672

prb(5): 0.0264, 0.9936

prb(6): 0.0055, 0.9991

prb(7): 0.0008, 0.9999

prb(8): 0.0001, 1.0000

prb(9): 0.0000, 1.0000

prb(10): 0.0000, 1.0000.

Among them, prb(i), i=1-10 refers to the probability distribution in a 10-second video segment according to the binomial distribution function B(10, 0.2). Specifically, the first column is that the duration of the voice is at i The probability between and i-1, the probability that the duration of the speech in the second column is less than or equal to i seconds. The second column above can be used as the quantile value of the corresponding parameter i. The above parameters can be calculated according to the probability calculation formula of the binomial distribution when the binomial distribution function is determined.

From the above data, it can be seen that when the first user’s speech duration is less than or equal to 0 seconds, the quantile value is 0.1074; when the speech duration is less than or equal to 1 second, the quantile value is 0.3758; when the speech duration is less than or equal to 2 seconds, the quantile value is 0.3758. The value is 0.6778, and so on. Further, the quantile value is compared with the preset quantile value (for example, 0.4), and if the quantile value is less than or equal to the preset quantile value, the video clip corresponding to the quantile value is filtered out; If the value is greater than the preset quantile value and meets the requirements of other elements at the same time, the corresponding video clip is retained.

In step 170, a target video segment is obtained according to the selected first video segment and second video segment.

Optionally, as another embodiment of the present invention, the first video segment and the second video segment pair in the last remaining video segment pair in the second video candidate set are synthesized to form a wonderful video.

Specifically, splicing can be performed, so that the selected first video segment and the second video segment can be displayed in the same screen at the same time, that is, the first video segment and the second video segment can be displayed in the same screen at the same time.

All the above-mentioned optional technical solutions can be combined in any way to form an optional embodiment of the present invention, which will not be repeated here.

The embodiment of the application obtains the first video file of at least one first user and the second video file of the second user, and traverses and intercepts the first video file and the second video file according to the at least one window duration to obtain the first video candidate set According to the first vector and the second vector representing the state of the user in the video in the unit time period of the first video file, the fourth vector representing the probability distribution value is obtained, and the fourth vector is selected from the first video candidate set according to the fourth vector. A first video segment and a corresponding second video segment, and then synthesize the target video segment. In this way, high-quality or wonderful video clips can be extracted to fully embody the interaction between the first user and the second user, thereby providing a richer user experience.

The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiment of the present invention, please refer to the method embodiment of the present invention.

Fig. 3 is a schematic diagram of a video processing device according to an embodiment of the present invention. As shown in FIG. 3, the video processing device 3 of this embodiment includes a first acquisition unit 31, an interception unit 32, a second acquisition unit 33, a third acquisition unit 32, a fourth acquisition unit 35, a selection unit 36, and a synthesis unit 37. .

The first obtaining unit 31 is configured to obtain at least one first video file of the first user and at least one second video file of the second user.

The interception unit 32 is configured to traverse and intercept the first video file and the second video file according to at least one window duration to obtain a first video candidate set. The first video candidate set includes a plurality of video clip pairs, each The video clip pair includes a first video clip and a corresponding second video clip with the same window duration and the same time axis position.

The second obtaining unit 33 is configured to obtain a first vector corresponding to each unit time period according to the first video file, and obtain a second vector corresponding to each unit time period according to the second video file. The vector is used to characterize the first user state in the corresponding unit time period, and the second vector is used to characterize the second user state in the corresponding unit time period.

The third obtaining unit 34 is configured to obtain a third vector according to the first vector and the second vector corresponding to each unit time period.

The fourth acquiring unit 35 is configured to determine a fourth vector according to the time axis position corresponding to each video segment pair and the third vector.

The selecting unit 36 is configured to select multiple first video clips and second video clips from the first video candidate set according to the fourth vector.

The synthesis unit 37 is configured to obtain a target video segment according to the selected first video segment and second video segment.

Therefore, the embodiment of the present application obtains the first video file of at least one first user and the second video file of the second user, and traverses and intercepts the first video file and the second video file according to the at least one window duration to obtain the first video file. Video candidate set, according to the first vector and second vector representing the state of the user in the video in the unit time period of the first video file, obtain the fourth vector representing the probability distribution value of the first video file, and obtain the fourth vector representing the probability distribution value from the first video candidate according to the fourth vector Collectively select multiple first video clips and corresponding second video clips, and then synthesize the target video clips. In this way, high-quality or wonderful video clips can be extracted to fully embody the interaction between the first user and the second user, thereby providing a richer user experience.

Fig. 4 is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 4 shown in FIG. 4 is a general data processing device, which includes a general computer hardware structure, and at least includes a processor 41 and a memory 42. The processor 41 and the memory 42 are connected by a bus 43. The memory 42 is suitable for storing instructions or programs executable by the processor 41. The processor 41 may be an independent microprocessor, or a collection of one or more microprocessors. In this way, the processor 41 executes the command stored in the memory 42 to execute the method procedure of the embodiment of the present invention described above to realize data processing and control of other devices. The bus 43 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 44 and the display device and the input/output (I/O) device 45. The input/output (I/O) device 45 may be a mouse, a keyboard, a modem, a network interface, a touch input device, a motion sensing input device, a printer, and other devices known in the art. Typically, an input/output (I/O) device 45 is connected to the system through an input/output (I/O) controller 46.

Among them, the memory 42 may store software components, such as an operating system, a communication module, an interaction module, and an application program. Each module and application program described above corresponds to a set of executable program instructions that complete one or more functions and methods described in the embodiments of the invention.

The above-mentioned flowcharts and/or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention describe various aspects of the present invention. It should be understood that each block of the flowchart illustrations and/or block diagrams and combinations of blocks in the flowchart illustrations and/or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine, so that the instructions (executed by the processor of the computer or other programmable data processing device) are created for implementation Flow chart and/or block diagram block or means of function/action specified in the block.

At the same time, as those skilled in the art will realize, various aspects of the embodiments of the present invention may be implemented as a system, a method, or a computer program product. Therefore, various aspects of the embodiments of the present invention can take the following forms: a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or can be generally referred to as "circuits" and "modules" in this document. "Or "system" is an implementation that combines software and hardware. In addition, aspects of the present invention may take the following form: a computer program product implemented in one or more computer-readable media, the computer-readable medium having computer-readable program codes implemented thereon.

Any combination of one or more computer readable media can be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any appropriate combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media would include the following: electrical connections with one or more wires, portable computer floppy disks, hard disks, random access memory (RAM), read-only memory ( ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any appropriate combination of the foregoing. In the context of the embodiments of the present invention, a computer-readable storage medium may be any tangible medium that can contain or store a program used by an instruction execution system, device, or device or a program used in conjunction with an instruction execution system, device, or device.

The computer-readable signal medium may include a propagated data signal having computer-readable program code implemented therein as in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including but not limited to: electromagnetic, optical, or any suitable combination. The computer-readable signal medium may be any of the following computer-readable media: it is not a computer-readable storage medium, and it can communicate and propagate the program used by the instruction execution system, device or device or used in conjunction with the instruction execution system, device or device Or transmission.

Any suitable medium including but not limited to wireless, wired, fiber optic cable, RF, etc. or any appropriate combination of the foregoing can be used to transmit the program code implemented on the computer-readable medium.

The computer program code used to perform operations directed to various aspects of the present invention can be written in any combination of one or more programming languages, and the programming languages include: object-oriented programming languages such as Java, Smalltalk, C++, PHP, Python, etc.; And conventional process programming languages such as "C" programming language or similar programming languages. The program code can be executed as an independent software package entirely on the user's computer, partly on the user's computer; partly on the user's computer and partly on a remote computer; or entirely on the remote computer or server. In the latter case, the remote computer can be connected to the user computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected with an external computer (for example, by using the Internet of an Internet service provider) .

The foregoing flowchart illustrations and/or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention describe various aspects of the present invention. It will be understood that each block of the flowchart illustrations and/or block diagrams and combinations of blocks in the flowchart illustrations and/or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine, so that the instructions (executed by the processor of the computer or other programmable data processing device) are created for implementation Flow chart and/or block diagram block or means of function/action specified in the block.

These computer program instructions can also be stored in a computer-readable medium that can direct a computer, other programmable data processing equipment, or other devices to operate in a specific manner, so that the generation of instructions stored in the computer-readable medium includes implementation in the flowcharts and / Or block diagram block or the product of the instruction of the function/action specified in the block.

Computer program instructions can also be loaded on a computer, other programmable data processing equipment or other devices, so that a series of operable steps are executed on the computer, other programmable equipment or other devices to generate a computer-implemented process, so that the computer Or instructions executed on other programmable devices provide a process for implementing the functions/actions specified in the flowchart and/or block diagrams or blocks.

The above descriptions are only preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (13)

1. A video processing method, including:

   Acquiring at least one first video file of the first user and at least one second video file of the second user;

   Traverse and intercept the first video file and the second video file according to at least one window duration to obtain a first video candidate set. The first video candidate set includes a plurality of video clip pairs, and each video clip pair Including a first video clip and a corresponding second video clip with the same window duration and the same time axis position;

   Obtain a first vector corresponding to each unit time period according to the first video file, and obtain a second vector corresponding to each unit time period according to the second video file, and the first vector is used to characterize the corresponding unit A first user state in a time period, where the second vector is used to represent a second user state in a corresponding unit time period;

   Obtaining a third vector for each unit time period according to the first vector and the second vector corresponding to each unit time period;

   Determine the fourth vector according to the time axis position corresponding to each video segment pair and the third vector;

   Selecting multiple first video segments and second video segments from the first video candidate set according to the fourth vector;

   Obtain the target video segment according to the selected first video segment and the second video segment.
2. The method according to claim 1, wherein obtaining the first vector corresponding to each unit time period according to the first video file comprises:

   Determining the video data and audio data corresponding to the target unit time period according to the first video file;

   Performing face recognition on a plurality of image frames of the video data respectively, and obtaining face state parameters corresponding to each image frame;

   Perform voice recognition on the audio data to obtain voice distribution parameters;

   The first vector is determined according to the face state parameter and the voice distribution parameter.
3. The method according to claim 1, wherein obtaining the second vector corresponding to each unit time period according to the second video file comprises:

   Determine the video data and audio data corresponding to the target unit time period according to the second video file;

   Performing face recognition on a plurality of image frames of the video data respectively, and obtaining face state parameters corresponding to each image frame;

   Perform voice recognition on the audio data to obtain voice distribution parameters;

   Obtaining the second vector according to the face state parameter and the voice distribution parameter.
4. The method according to claim 2 or 3, wherein the face state parameter includes a first value that characterizes the appearance of the face and a second value that characterizes the expression state of the face.
5. The method according to claim 1, wherein the obtaining a third vector according to the first vector and the second vector corresponding to each unit time period comprises:

   Combining the first vector and the second vector corresponding to the same unit time period into the third vector corresponding to the unit time period.
6. The method according to claim 5, wherein determining the fourth vector according to the time axis position corresponding to each video segment pair and the third vector comprises:

   Determine the target video clip pair;

   Determining corresponding multiple target unit time periods according to the time axis position of the target video clip pair;

   Calculating the sum vector of the third vector of the multiple target unit time periods;

   Determine the element random distribution function corresponding to each element according to the average vector and the window duration, and the average vector is obtained by average calculation according to the third vector of each unit time period in the multiple video files;

   The fourth vector of the target video segment pair is determined according to the element random distribution function corresponding to each element and the sum vector, wherein each element of the fourth vector is the corresponding element of the sum vector The quantile value in the element random distribution function.
7. The method according to claim 6, wherein selecting a plurality of first video clips and second video clips from the first video candidate set according to the fourth vector comprises:

   Determining a filtered video segment pair according to the fourth vector;

   The filtered video segment pair is removed from the first video candidate set to obtain a second video candidate set.
8. The method according to claim 7, wherein the determining to filter the video segment according to the fourth vector comprises:

   In response to any one of the elements in the fourth vector being less than the corresponding quantile threshold, the corresponding video segment pair is determined as the filtered video segment.
9. The method according to claim 5, wherein the element random distribution function is a binomial distribution function that takes the corresponding element in the average vector as the mean value and the length matches the window duration.
10. 8. The method according to claim 7, wherein selecting a plurality of first video clips and second video clips from the first video candidate set according to the fourth vector further comprises:

    Calculating the score value of each video segment pair in the second video candidate set according to the fourth vector;

    Sorting and filtering multiple first video clips in the second video candidate set according to the score value until the number of remaining first video clips in the second video candidate set meets a predetermined condition.
11. A video processing device includes:

    The first obtaining unit is configured to obtain at least one first video file of the first user and at least one second video file of the second user;

    The interception unit is configured to traverse and intercept the first video file and the second video file according to at least one window duration to obtain a first video candidate set, where the first video candidate set includes a plurality of video clip pairs, each The video clip pair includes a first video clip and a corresponding second video clip with the same window duration and the same time axis position;

    The second obtaining unit is configured to obtain a first vector corresponding to each unit time period according to the first video file, and obtain a second vector corresponding to each unit time period according to the second video file. The vector is used to characterize the first user state in the corresponding unit time period, and the second vector is used to characterize the second user state in the corresponding unit time period;

    A third acquiring unit, configured to acquire a third vector according to the first vector and the second vector corresponding to each unit time period;

    A fourth acquiring unit, configured to determine a fourth vector according to the time axis position corresponding to each video segment pair and the third vector;

    A selecting unit, configured to select a plurality of first video clips and second video clips from the first video candidate set according to the fourth vector; and

    The synthesis unit obtains the target video segment according to the selected first video segment and the second video segment.
12. An electronic device comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to realize The method of any one of 1 to 10 is required.
13. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the method according to any one of claims 1 to 10 when executed by a processor.

Images