CN112135188A - Video clipping method, electronic device and computer-readable storage medium - Google Patents

Abstract

The embodiments of the present invention disclose a video cropping method, an electronic device and a computer-readable storage medium, which belong to the technical field of video processing. The video cropping method includes: acquiring a target video to be processed; determining the coreness of each object in the target video respectively; and determining an object whose coreness is greater than a preset threshold according to the coreness of each object as the A target object corresponding to the target video; taking the target object as a clipping target, clipping video frames in the target video to obtain a plurality of clipping video frames; and generating a clipping video according to the plurality of clipping video frames. Therefore, according to this embodiment, the target object to be cropped can be selected based on the degree of coreness, thereby ensuring effective cropping of the core object in the target video and improving the effect of cropping the video.

Description

Video cropping method, electronic device and computer-readable storage medium

technical field

The invention belongs to the technical field of video processing, and in particular relates to a video cropping method, an electronic device and a computer-readable storage medium.

Background technique

In the prior art, for video cropping, such as cropping a horizontal screen video into a vertical screen video, a cropped video with a fixed position and a fixed video aspect ratio is usually generated by video editing software. Therefore, in this case, the cropped video may not be cropped to the core characters, etc., resulting in a poor effect of cropping the video.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a video cropping method, an electronic device and a computer-readable storage medium, so as to solve the problem that the video cropped by the existing video cropping method has poor effect.

In order to solve the above-mentioned technical problems, the present invention is achieved in this way:

In a first aspect, an embodiment of the present invention provides a video cropping method, which is applied to an electronic device, and the method includes:

Get the target video to be processed;

Determine the core degree of each object in the target video respectively;

According to the core degree of each object, the object whose core degree is greater than the preset threshold is determined as the target object corresponding to the target video;

Taking the target object as a clipping target, clipping the video frames in the target video to obtain a plurality of clipping video frames;

A cropped video is generated from the plurality of cropped video frames.

Optionally, the separately determining the coreness of each object in the target video includes:

Determine the coreness of each object in the target video based on at least one of the following:

the sequential ranking of each object in the target video;

Whether the each object is occluded in the target video;

Whether each of the objects is a preset object;

whether each object is speaking in the target video;

Whether the each object has a display of preset actions in the target video;

Emotional representation of each object in the target video.

Optionally, the separately determining the coreness of each object in the target video includes:

For each of said objects, perform the following procedures separately:

Analyzing the object to obtain a plurality of characteristic values of the object;

According to the plurality of eigenvalues and the weight of each eigenvalue, the coreness of the object is calculated.

Optionally, the plurality of eigenvalues include eigenvalue S, eigenvalue R, eigenvalue K, eigenvalue T, eigenvalue A, and eigenvalue E; Weight, calculate the core degree of the object, including:

The following formula is used to calculate the core degree I of the object:

I=a*S+b*(1/R)+c*K+d*T+e*A+f*E

Wherein, S is 0 or 1, 0 means the object is occluded in the target video, 1 means the object is not occluded in the target video; R means the object is in the target video The continuity ranking of , which is a positive integer; K is 0 or 1, 0 means that the object is not a preset object, 1 means that the object is a preset object; T is 0 or 1, 0 means all The object is not speaking in the target video, 1 means the object is speaking in the target video; A is 0 or 1, 0 indicates that the object has no preset action in the target video Display, 1 indicates that the object has a preset action display in the target video; E is 0 or 1, 0 indicates that the object has no preset emotional expression in the target video, and 1 indicates that the object There are preset emotional expressions in the target video; a is the weight of S, b is the weight of (1/R), c is the weight of K, d is the weight of T, e is the weight of A, and f is the weight of E Weights.

Optionally, when the number of the target object is one, the video frame in the target video is cropped to obtain a plurality of cropped video frames, including:

The abscissa of the center point of the target object is used as the abscissa of the center point of the cropping frame, and the video frames in the target video are cropped to obtain the plurality of cropped video frames.

Optionally, when the area of the video frame in the target video occupied by the target object is smaller than a preset area threshold, the height of the cropped video frame is determined based on the height of the target object, and the height of the target object is determined. The ratio to the height of the cropped video frame is equal to a preset ratio threshold.

Optionally, when the number of the target objects is multiple, the video frames in the target video are cropped to obtain a plurality of cropped video frames, including:

In the case where there is a first video frame including one target object in the target video, the abscissa of the center point of the one target object is taken as the abscissa of the center point of the cropping frame, and from the first video frame Crop to get cropped video frame;

and / or,

In the case that a second video frame including multiple target objects exists in the target video, a cropped video frame is obtained by cropping from the second video frame based on the position of the core target object in the plurality of target objects.

Optionally, the core target object satisfies any one of the following conditions:

The core degree is the highest among the plurality of target objects;

Among the plurality of target objects, the position of the center point of the core target object is closest to the center position of the second video frame.

Optionally, when the core target object has the highest coreness among the plurality of target objects, the cropping from the second video frame is based on the position of the core target object in the plurality of target objects. Get cropped video frames, including:

According to the position of the core target object, determine the distance between the core target object and other target objects;

According to the distance, determine a first target object that can be in the cropping frame at the same time as the core target object;

Select a target cropping frame according to a preset condition, and use the target cropping frame to crop a cropped video frame from the second video frame; wherein, the target cropping frame completely covers the core target object and at least one first video frame. target.

Optionally, when the position of the center point of the core target object is closest to the center position of the second video frame, based on the position of the core target object in the plurality of target objects, the Crop from frame to get cropped video frames, including:

Taking the abscissa of the center position point of the core target object as the abscissa of the center point of the cropping frame, sliding the cropping frame with a preset step size, and using the sliding cropping frame, from the second video frame The cropped video frame is obtained by cropping in the middle; wherein, the sliding cropping frame completely covers the core target object, and there is no incompletely covered second target object in the sliding cropping frame; the second target object is the other target objects except the core target object among the plurality of target objects.

In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor, the program or instruction being The processor, when executed, implements the steps of the method as described above.

In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, where a program or an instruction is stored on the computer-readable storage medium, and when the program or instruction is executed by a processor, the steps of the above method are implemented.

In this embodiment of the present invention, the electronic device may acquire the target video to be processed, determine the coreness of each object in the target video, and classify the objects whose coreness is greater than a preset threshold according to the coreness of each object. Determine the target object corresponding to the target video, and take the target object as the cropping target, crop the video frames in the target video to obtain a plurality of cropped video frames, and generate a plurality of cropped video frames according to the plurality of cropped video frames. Crop video. Therefore, the target object to be cropped can be selected based on the coreness, thereby ensuring effective cropping of the core object in the target video and improving the effect of cropping the video.

Description of drawings

1 is a flowchart of a video cropping method according to an embodiment of the present invention;

Fig. 2 is one of the cropping schematic diagrams in the embodiment of the present invention;

3A, 3B and 3C are the second schematic diagrams of cropping in the embodiment of the present invention;

Fig. 4 is the third schematic diagram of cropping in the embodiment of the present invention;

5 is a schematic structural diagram of a video cropping device according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The terms "first", "second" and the like in the description and claims of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that embodiments of the invention can be practiced in sequences other than those illustrated or described herein, and distinguish between "first", "second", etc. The objects are usually of one type, and the number of objects is not limited. For example, the first object may be one or more than one. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the associated objects are in an "or" relationship.

The video cropping method provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings through specific embodiments.

Please refer to FIG. 1. FIG. 1 is a flowchart of a video cropping method provided by an embodiment of the present invention. The method is applied to an electronic device. As shown in FIG. 1, the method includes the following steps:

Step 101: Acquire the target video to be processed.

In this embodiment, the target video can be selected as a horizontal screen video or a vertical screen video. For the current video content, the story is generally narrated by switching between different shots, and the core characters and the positions of the characters are generally different between different shots. Therefore, in order to accurately crop, the target video obtained above to be processed can be selected as Video from one shot.

In one embodiment, after the electronic device obtains the original video to be cropped (such as a landscape video), if the original video includes videos from multiple different shots, the original video can be disassembled by shot through a shot detection algorithm. The target video is obtained by shot as a unit, and the target video under each shot is subsequently analyzed and cropped.

In one embodiment, the aspect ratio of the target video to be processed may be 9:16. In addition, the aspect ratio of the cropped video (that is, the cropped video) can also be selected as 9:16.

Step 102: Determine the coreness of each object in the target video respectively.

In this embodiment, the object in the target video may be a person, an object (for example, a vehicle), etc., which is not limited here. Preferably, the object in the target video is a person. The core degree determined in this step can be understood as the importance degree of the corresponding object.

Taking the object in the target video as a character as an example, the core degree of the character can be determined through artificial intelligence (AI) algorithms such as character recognition, character tracking, face detection, preset action recognition, and/or character emotion recognition. .

Step 103: According to the coreness of each object, determine an object whose coreness is greater than a preset threshold as the target object corresponding to the target video.

Optionally, the preset threshold may be preset based on actual requirements, such as preset as 60, etc., which is not limited in this embodiment. The target objects determined in this step may be understood as core objects, and the corresponding number may be one or more.

Step 104: Using the target object as a cropping target, crop video frames in the target video to obtain a plurality of cropped video frames.

In this embodiment, when the video frames in the target video are cropped, each video frame in the target video may be cropped. In a special case, if a certain video frame in the target video does not include the target object or the included target object is incomplete, etc., the video frame can be skipped, that is, the video frame is not cropped. The sizes of the cropped video frames obtained by cropping are preferably the same.

Step 105: Generate a cropped video according to the plurality of cropped video frames.

Optionally, when the cropped video is generated, the plurality of cropped video frames may be synthesized according to the time sequence of the plurality of cropped video frames.

The above cropped video can be selected as a vertical screen video, for example, a horizontal screen video is cropped into a vertical screen video.

It should be noted that, in this embodiment of the present invention, after the target video under each shot is analyzed and cropped to obtain the cropped video, the cropped videos under different shots may be merged based on the switching relationship between different shots in the original video , so as to obtain the cropped video corresponding to the original video.

In the video cropping method according to the embodiment of the present invention, the electronic device can obtain the target video to be processed, determine the coreness of each object in the target video, and classify the objects whose coreness is greater than a preset threshold according to the coreness of each object. A target object corresponding to the target video is determined, and the target object is used as a cropping target, and video frames in the target video are cropped to obtain a plurality of cropped video frames, and a cropped video is generated according to the plurality of cropped video frames. Therefore, the target object to be cropped can be selected based on the coreness, thereby ensuring effective cropping of the core object in the target video and improving the effect of cropping the video.

In the embodiment of the present invention, considering the factors affecting the importance of the object, such as continuity in the target video, whether it is a preset object, whether there is a preset action display, etc., the above-mentioned determination of each object in the target video The core degree process can include:

Determine the coreness of each object in the target video separately based on at least one of the following:

1) Continuity ranking of each object in the target video

For the continuity ranking in 1), the tracking coordinates of each object in the target video can be obtained through a tracking detection algorithm (such as a person tracking detection algorithm), and based on the number of tracking coordinates of each object and the number of interruptions, for each object Objects are ranked by their continuity in the target video. Understandably, the higher the continuity rank, the higher the core.

In one embodiment, taking the above-mentioned object as a person as an example, the person tracking detection algorithm can use a human skeleton detection and tracking algorithm to track each person in the target video to obtain the tracking coordinates of the person and the coordinates of the center point of the person's head, And according to the obtained character tracking coordinates, the number of frames that each tracked character appears in is calculated and ranked. If the two characters appear in the same number of frames, the ranking can be further based on the number of interruptions in the tracking coordinates. The less the number of interruptions, the higher the ranking. The coordinates of the center point of the character's head can be used to ensure that the position of the character's head can be intercepted during subsequent cropping

For example, the target video under one shot includes 5 seconds and a total of 100 frames, and the head coordinate sequence of each character corresponding to each frame is obtained through the character tracking detection algorithm. For example, if character A appears in 90 frames, the frame sequence is [1-90], character B appears in 90 frames, the frame sequence is [1-40, 45-95], and character C appears in 50 frames, then character A, character B and The continuous ranking of character C can be shown in Table 1 below:

Table 1

Continuity ranking figure 1 A 2 B 3 C

2) Whether each object is occluded in the target video

In this 2), the video frame in the target video can be recognized by image recognition technology to determine whether the object in it is blocked. Understandably, compared to occluded objects, unoccluded objects usually have a higher core level.

3) Whether each object is a preset object

In this 3), the video frame in the target video can be recognized by image recognition technology to determine whether the object in it is a preset object, and the preset object can be preset based on actual requirements. It is understandable that, compared with non-preset objects, the core level of preset objects is usually higher.

For example, taking the above object as a person as an example, it can be determined whether the person in the target video is a specific person or a known star person through face recognition technology.

4) Whether each object is speaking in the target video

In this 4), the object can be selected as a person, and it can be recognized by lip movements to determine whether each person in the target video is speaking in the corresponding shot. Understandably, speaking subjects are usually more central to the core than non-speaking subjects.

5) Whether each object has a preset action display in the target video

In this 5), it can be determined through action recognition whether each object in the target video has a preset action displayed in the corresponding shot. The preset action may be preset based on actual requirements. It is understandable that, compared with objects displayed without preset actions, the core level of objects displayed with preset actions is usually higher.

For example, taking the above object as a character as an example, the preset action can be selected from, but not limited to, stretching an arm, kicking a ball, dunking, and/or dancing.

6) Emotional representation of each object in the target video

In this 6), the object can be selected as a character, and through facial expression recognition, it can be determined whether each character in the target video has an emotional expression in the corresponding shot, and the emotional expression can be selected as but not limited to surprise, laughter, cry etc. Understandably, characters with emotional expressions usually have a higher level of core than those without emotional expressions.

Further, in this embodiment of the present invention, in addition to determining whether there is an emotional expression, a specific emotional expression can also be determined, and the importance of the corresponding character is evaluated based on the specific emotional expression. Optionally, the specific facial expression recognition may be to mark each person's emotion by recognizing the facial expressions of different people in different frames of images in the target video.

In one embodiment, for the target video segment, the frame can be drawn according to a certain frequency first, and then a multi-task convolutional neural network (Multi-Task Convolutional Neural Network, MTCNN) model is used to extract the appearance of the corresponding person in the extracted video frame image. Finally, the Xception network model (a deep separable convolutional network model) is used to classify the facial expressions of all characters and count whether the characters have emotional expressions.

In this way, with the aid of the above-mentioned influencing factors 1) to 6), the coreness of the object in the target video can be accurately determined.

For example, taking the determination of the coreness of the characters in the target video based on the above 1), 3) to 6) as an example, if the characters in the target video include character A, character B and character C, the correspondingly recognized results can be As shown in Table 2 below:

Table 2

In the embodiment of the present invention, optionally, the above-mentioned process of respectively determining the coreness of each object in the target video may include: for each object, respectively performing the following process:

Analyzing the object to obtain a plurality of characteristic values of the object;

According to the multiple eigenvalues and the weight of each eigenvalue, the coreness of the object is calculated.

It should be pointed out that the above-mentioned methods for analyzing the object include but are not limited to the content in 1) to 6) above, such as: analyzing the continuous ranking of the object in the target video, analyzing whether the object is occluded in the target video, analyzing Whether the object is a preset object, analyze whether the object is speaking in the target video, analyze whether the object has a preset action display in the target video, and/or analyze the emotional performance of the object in the target video, etc., and The corresponding eigenvalues can be set based on actual requirements. For example, if an object is occluded in the target video, the corresponding eigenvalue can take a value of 0, and if an object is not occluded in the target video, the corresponding eigenvalue can take a value of 1; and so on.

Further, if the above-mentioned multiple eigenvalues include eigenvalue S, eigenvalue R, eigenvalue K, eigenvalue T, eigenvalue A and eigenvalue E, the above-mentioned multiple eigenvalues and the weight of each eigenvalue are calculated to obtain The process of object coreness may include:

The following formula is used to calculate the core degree I of the object:

I=a*S+b*(1/R)+c*K+d*T+e*A+f*E

Wherein, the value of S is 0 or 1, 0 means occluded, that is, the object is occluded in the target video; 1 means that it is not occluded, that is, the object is not occluded in the target video;

R represents the continuous ranking of the object in the target video, which is a positive integer, such as 1, 2, 3...;

K is 0 or 1, 0 means it is not a preset object, that is, the object is not a preset object; 1 means it is a preset object, that is, the object is a preset object;

T is 0 or 1, 0 means not speaking, that is, the object is not speaking in the target video; 1 means speaking, that is, the object is speaking in the target video;

A takes a value of 0 or 1, 0 means no preset action display, that is, the object has no preset action display in the target video; 1 means there is a preset action display, that is, the object has a preset action in the target video. Display of preset actions;

E takes the value of 0 or 1, 0 means there is no preset emotional expression, that is, the object has no preset emotional expression in the target video; 1 means there is a preset emotional expression, that is, the object has a preset emotional expression in the target video Performance;

a is the weight of S, b is the weight of (1/R), c is the weight of K, d is the weight of T, e is the weight of A, and f is the weight of E.

In one embodiment, c can take a value of 20, while a, b, d, e, and f can take a value of 10. In this case, the calculated coreness is a minimum of 10/R and a maximum of 70. At this time, the preset threshold for judging whether it is the target object can be preset to a maximum of 60, but it is not limited to this, and can also be preset to 50 or 40, etc., which can be adjusted based on needs.

It should be pointed out that if the height of the video frame in the target video is h, the width is w, the aspect ratio is 9:16, the height of the cropped video frame is h', the width is w', the aspect ratio is 9:16, and The position to be cropped of the video frame in the target video is (xmin, ymin, xmax, ymax) (at this time, the coordinate system constructed with the upper left point of the video frame in the target video as the origin), then the following relationship can exist:

ymin=h-h′

ymax=h

为裁剪视频帧的高度与原始视频帧(即目标视频中视频帧)的高度的比例值，可选为小于或等于1的正数。该

可以基于实际需求预设，也可以基于实际情况调整。cx为裁剪框的中心点横坐标，也可理解为裁剪视频帧的中心点横坐标。该裁剪视频帧可以是基于裁剪框裁剪得到。in,

It is the ratio of the height of the cropped video frame to the height of the original video frame (that is, the video frame in the target video), which can be a positive number less than or equal to 1. Should

It can be preset based on actual needs or adjusted based on actual conditions. cx is the abscissa of the center point of the cropping frame, which can also be understood as the abscissa of the center point of the cropped video frame. The cropped video frame may be obtained by cropping based on the cropping frame.

In this embodiment of the present invention, for the target video to be processed, the number of corresponding target objects may be one or more. However, based on the number of target objects being one or more, different video cropping methods can be adopted, as described below.

Scenario 1: The number of target objects is one

In this scenario, since the number of target objects is one, that is, there is one core object to be cropped, cropping can be performed directly based on the position of the target object.

Optionally, the specific cropping method may be: the abscissa of the center point of the target object is used as the abscissa of the center point of the cropping frame, and the video frames in the target video are cropped to obtain multiple cropped video frames. The cropped video frame is obtained by cropping the video frame in the target video based on the cropping frame, and the size of the cropped video frame and the cropping frame is the same.

In one embodiment, taking the target object as person A as an example, the center point of the target object can be selected as the center point of the head position of person A.

In one embodiment, taking the target object as person A as an example, when cropping the video frames in the target video, each frame can be cropped according to the coordinates of the center point of the head position of person A, so as to ensure the cropping of subsequent synthesis. Complete video. For example, as shown in Figure 2, if the video frame 1 in the target video is cropped, the abscissa (x') of the center point of the head position of character A is equal to the abscissa (cx) of the center point of the cropping frame. A schematic diagram of the cropping of frame 1 can be shown in FIG. 2 .

It should be pointed out that the size of the above-mentioned cropping frame may be preset, for example, the width is w', the height is h', the aspect ratio is 9:16, and the ratio of the height h' to the height h of the video frame in the target video value is

In addition, the size of the above-mentioned cropping frame can also be adjusted. Optionally, when the area of the video frame in the target video occupied by the target object corresponding to the target video is less than the preset area threshold, the height of the cropped video frame (that is, the cropping frame) can be determined based on the height of the target object, and the target The ratio of the height of the object to the height of the cropped video frame is equal to the preset scale threshold. The preset area threshold and the preset ratio threshold may be preset based on actual requirements.

为1，则可以人物高度ph占裁剪框高度h'，3/4(此值可调整)的比例缩小裁剪框，以缩小裁剪视频帧高度，此时

为：

For example, taking the target object as a character as an example, if the area of the video frame in the corresponding target video is less than 1/8 of the video frame area (this value can be adjusted), the preset cropping frame is the same size as the video frame which is

If it is 1, you can reduce the cropping frame by the ratio of the height of the character ph to the height of the cropping frame h', 3/4 (this value can be adjusted) to reduce the height of the cropped video frame.

for:

Scenario 2: The number of target objects is multiple

In this scenario, when the number of target objects corresponding to the target video is multiple, the video frame in the target video may include only one target object, or may include multiple target objects. At this time, based on the difference in the number of target objects, different video cropping methods can be adopted, as described below.

Case 1: When there is a first video frame including a target object in the target video, for the first video frame, the abscissa of the center point of the target object can be used as the abscissa of the center point of the cropping frame , and a cropped video frame is obtained by cropping the first video frame. The cropped video frame is obtained by cropping the first video frame based on the cropping frame. The first video frame may be any video frame in the target video.

In an implementation manner, taking the one target object as a character B as an example, the center point of the one target object can be selected as the center point of the head position of the character B.

Case 2: In the case where there is a second video frame including a plurality of target objects in the target video, for this second video frame, based on the position of the core target object in the plurality of target objects, from the second video frame Crop a video frame to get a cropped video frame. The second video frame may be any video frame in the target video.

It can be understood that the above-mentioned case 1 and case 2 may be in the relationship of and/or. That is to say, when the number of target objects corresponding to the target video is multiple, the situation where the video frame in the target video includes the target object may be any of the following: 1) each video frame includes only one target object, Different video frames may include different target objects; 2) each video frame includes multiple target objects; 3) some video frames include only one target object, while some video frames include multiple target objects. This embodiment of the present invention is not limited.

Optionally, in the above case 2, the core target object may satisfy any one of the following conditions:

(1) The core degree is the highest among multiple target objects. At this time, the core target object may be located at any position of the second video frame, such as a center position, a side position, and the like. In this way, the target object with the highest core degree can be preferentially selected to crop the video, so as to ensure that the picture of the core target object is effectively cropped.

In one embodiment, taking the multiple target objects as multiple core characters as an example, the coreness of the multiple core characters in the target video, that is, the same shot, is higher than a set threshold.

In this (1), the specific video cropping process may include: first, according to the position of the core target object, determine the distance between the core target object and other target objects; then, according to the distance, determine the distance between the core target object and other target objects; The core target object is the first target object of the cropping frame at the same time; the first target object may be one or more objects in multiple target objects; finally, the target cropping frame is selected according to preset conditions, and the target cropping frame is selected using the target A cropping frame is obtained by cropping a cropped video frame from the second video frame; wherein, the target cropping frame completely covers the core target object and at least one first target object. For the preset condition, the first target object that meets as many as possible can be in the target cropping frame, or the first target object that meets the preset requirement can be in the target cropping frame. The preset requirement can be preset based on actual requirements. , such as ranking higher than a preset threshold for continuity, showing preset actions, showing emotions, etc.

In one embodiment, in the video cropping process of (1), after the first target object is determined, under the condition that the first target object can be completely covered by the cropping frame, the core target object and the first target object can be directly For the cropping target, crop the cropped video frame from the second video frame; or, in the case that the number of the first target objects is multiple, but the multiple first target objects cannot be covered by the cropping frame, the cropping frame can be obtained from multiple first target objects. A first target object that meets the preset requirements is selected from the first target objects, and the core target object and the first target object that meets the preset requirements are used as clipping targets, and the clipped video frame is obtained by clipping from the second video frame.

For example, taking multiple target objects as multiple core characters as an example, if the multiple core characters include core character D, core character E, and core character F, and the central core character (ie, the core target object) is the core character E, Core character F has preset actions to be displayed. If core character D and core character F can be in the cropping frame at the same time as core character E, but core character D and core character F cannot be completely covered by the cropping frame, you can select the The preset action is on the displayed core character F, and is cropped with the core character E and the core character F as the cropping targets, so that the cropped video frame includes images of the core character E and the core character F.

(2) Among the multiple target objects, the position of the center point of the core target object is closest to the center position of the second video frame. At this time, the core degrees of the multiple target objects may or may not be the same. In this way, the center position of the second video frame can be preferentially selected to be effectively cropped, thereby improving the cropping effect.

In one embodiment, taking the multiple target objects as multiple core characters as an example, the core degrees of the multiple core characters are the same or different. For example, the multiple core characters are all known star characters, and none of them speak, act, or express emotions in the target video.

Optionally, when the center point position of the core target object is closest to the center position of the second video frame, the above-mentioned process based on the position of the core target object in the plurality of target objects is obtained from the second video frame. It may include: taking the abscissa of the center point of the core target object as the abscissa of the center point of the cropping frame, sliding the cropping frame with a preset step size, and using the sliding cropping frame, from the first The cropped video frame is obtained by cropping the two video frames; wherein, the sliding cropping frame completely covers the core target object, and there is no incompletely covered second target object in the sliding cropping frame; the second target The objects are other target objects in the plurality of target objects except the core target object.

In this (2), when the target object closest to the center of the video frame is used as the core target object, in addition to the core target object, it can be considered that more target objects can be captured as much as possible. The specific video cropping process can include:

1) When the abscissa of the center point of the core target object is used as the abscissa of the center point of the cropping frame, the cropping frame can cover the core target object (in this case, the cropping frame may only cover the core The target object may also cover other target objects in addition to the core target object), and when there is no incomplete second target object in the cropping frame, it can be cropped from the second video frame based on the cropping frame. Get cropped video frames.

2) When the abscissa of the center point of the core target object is used as the abscissa of the center point of the cropping frame, the cropping frame can cover the core target object (in this case, the cropping frame may only cover the core The target object may also cover other target objects in addition to the core target object), and when there is a second target object that is not fully covered in the cropping frame, the cropping frame can be slid with a preset step size, and when sliding In the case where the rear cropping frame can cover the core target object, and there is no incompletely covered second target object in the sliding cropping frame, cropping from the second video frame based on the sliding cropping frame Obtain the cropped video frame; or, in the case where there is always a second target object that is not fully covered in the cropped frame after sliding, the abscissa of the center point of the core target object is used as the abscissa of the central point of the cropping frame, A cropped video frame is obtained by cropping from the second video frame.

Wherein, the second target object is other target objects except the core target object among the plurality of target objects. The size of the cropping frame can be preset based on actual needs. The preset step size can be preset based on actual needs, and when sliding the cropping frame, it can be slid left or right.

In one embodiment, taking the core target object as character B as an example, the center point of the core target object can be selected as the center point of the head position of character B.

For example, taking multiple target objects as multiple core characters as an example, if the central core character (ie, the core target object) is closest to the center of the video frame, and considering that as many core characters as possible can be intercepted, the central core character can be Slide the head center coordinates left and right to find a better cropping position. The value of the abscissa cx of the center point of the cropping frame is initialized to the abscissa X of the head center position of the central core character, and then it is judged whether the central core character is complete and whether it covers other core characters. When it just covers the central core character or When there are multiple core characters including the central core character, the value of cx at this time is used for video cropping. If the coverage is incomplete, slide the cropping frame to the right or left according to the set step size, and judge the coverage (in this case, the setting must cover the default middle core character). When the coverage is complete, there is no incomplete interception. , the video is captured based on the position of the cropping frame at this time. If this is not the case, the video is cropped based on the initial value of the abscissa cx of the center point of the cropping frame.

For example, as shown in FIGS. 3A to 3C , if the video frame 2 includes a core character L, a core character M and a core character N, and the core character M is close to the center of the video frame 2, that is, the core character M is the central core character. When the abscissa X of the center position of the head is initially assigned as the abscissa cx of the center point of the cropping frame, and the cropping frame does not completely cover the core character L and the core character N (as shown in Figure 3A), the cropping frame can be slid to the right (as shown in Figure 3A). 3B), or slide the cropping frame to the left (as shown in Figure 3C) to find a suitable cropping position, that is, to cover the characters completely without clipping (as shown in Figure 3C). The abscissa of the center position of the entire head area of M and the core character L is assigned as cx) for video cropping. If no suitable cropping position is found, assign the abscissa X of the central position of the head of the core character M as the abscissa cx of the central point of the cropping frame to perform video cropping, as shown in FIG. 4 .

Please refer to FIG. 5. FIG. 5 is a schematic structural diagram of a video cropping apparatus provided by an embodiment of the present invention. The apparatus is applied to electronic equipment. As shown in FIG. 5, the video cropping apparatus 50 may include:

an acquisition module 51, for acquiring the target video to be processed;

The first determination module 52 is used to determine the coreness of each object in the target video respectively;

The second determination module 53 is configured to, according to the coreness of each object, determine an object whose coreness is greater than a preset threshold as the target object corresponding to the target video;

The cropping module 54 is used for cropping the video frames in the target video by taking the target object as a cropping target to obtain a plurality of cropping video frames;

The generating module 55 is configured to generate a cropped video according to the plurality of cropped video frames.

Optionally, the first determining module 52 is specifically configured to:

Determine the coreness of each object in the target video based on at least one of the following:

the sequential ranking of each object in the target video;

Whether the each object is occluded in the target video;

Whether each of the objects is a preset object;

whether each object is speaking in the target video;

Whether the each object has a display of preset actions in the target video;

Emotional representation of each object in the target video.

Optionally, the first determining module 52 is specifically configured to: for each object, perform the following processes respectively: analyze the object to obtain multiple feature values of the object; according to the multiple features value and the weight of each feature value, the coreness of the object is calculated.

Optionally, the multiple eigenvalues include eigenvalue S, eigenvalue R, eigenvalue K, eigenvalue T, eigenvalue A, and eigenvalue E; the first determining module 52 is specifically configured to:

For each of the objects, the following formula is used to calculate the core degree I of the object:

I=a*S+b*(1/R)+c*K+d*T+e*A+f*E

Wherein, S is 0 or 1, 0 means the object is occluded in the target video, 1 means the object is not occluded in the target video; R means the object is in the target video The continuity ranking of , which is a positive integer; K is 0 or 1, 0 means that the object is not a preset object, 1 means that the object is a preset object; T is 0 or 1, 0 means all The object is not speaking in the target video, 1 means the object is speaking in the target video; A is 0 or 1, 0 indicates that the object has no preset action in the target video Display, 1 indicates that the object has a preset action display in the target video; E is 0 or 1, 0 indicates that the object has no preset emotional expression in the target video, and 1 indicates that the object There are preset emotional expressions in the target video; a is the weight of S, b is the weight of (1/R), c is the weight of K, d is the weight of T, e is the weight of A, and f is the weight of E Weights.

Optionally, when the number of the target objects is one, the cropping module 54 is specifically used for:

The abscissa of the center point of the target object is used as the abscissa of the center point of the cropping frame, and the video frames in the target video are cropped to obtain the plurality of cropped video frames.

Optionally, when the area of the video frame in the target video occupied by the target object is smaller than a preset area threshold, the height of the cropped video frame is determined based on the height of the target object, and the height of the target object is determined. The ratio to the height of the cropped video frame is equal to a preset ratio threshold.

Optionally, when the number of the target objects is multiple, the cropping module 54 includes:

The first cropping unit is used for taking the abscissa of the center position point of the target object as the abscissa of the center point of the cropping frame when there is a first video frame including a target object in the target video, from The first video frame is clipped to obtain a clipped video frame;

a second cropping unit, configured to, in the case that there is a second video frame including multiple target objects in the target video, based on the position of the core target object in the multiple target objects, from the second video frame Crop in to get the cropped video frame.

Optionally, the core target object satisfies any one of the following conditions:

The core degree is the highest among the plurality of target objects;

Among the plurality of target objects, the position of the center point of the core target object is closest to the center position of the second video frame.

Optionally, when the core target object has the highest core degree among the multiple target objects, the second cropping unit includes:

a first determination subunit, configured to determine the distance between the core target object and other target objects according to the position of the core target object;

a second determination subunit, configured to determine, according to the distance, a first target object that can be in a cropping frame simultaneously with the core target object;

The first cropping subunit is used to select a target cropping frame according to a preset condition, and use the target cropping frame to crop a cropped video frame from the second video frame; wherein, the target cropping frame completely covers the A core target object and at least one first target object.

Optionally, when the position of the center point of the core target object is closest to the center position of the second video frame, the second cropping unit is specifically configured to: convert the abscissa of the center point of the core target object. As the abscissa of the center point of the cropping frame, slide the cropping frame with a preset step size, and use the sliding cropping frame to crop the cropped video frame from the second video frame; wherein, after the sliding The cropping frame completely covers the core target object, and there is no second target object that is not fully covered in the sliding cropping frame; the second target object is the core target object among the plurality of target objects. other target objects.

It is understandable that the video cropping apparatus 50 according to the embodiment of the present invention can implement each process of the method embodiment shown in FIG. 1 above, and can achieve the same technical effect. To avoid repetition, details are not repeated here.

In addition, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a program or instruction stored on the memory and executable on the processor, wherein the program or instruction is When executed by the processor, each process of the method embodiment shown in FIG. 1 can be implemented and the same technical effect can be achieved. To avoid repetition, details are not repeated here.

Referring to FIG. 6 , an embodiment of the present invention further provides an electronic device 60 , including a bus 61 , a transceiver 62 , an antenna 63 , a bus interface 66 , a processor 65 and a memory 66 .

In the embodiment of the present invention, the electronic device 60 further includes: a computer program stored on the memory 66 and executable on the processor 65 . Optionally, when the computer program is executed by the processor 65, the following steps may be implemented:

Get the target video to be processed;

Determine the core degree of each object in the target video respectively;

According to the core degree of each object, the object whose core degree is greater than the preset threshold is determined as the target object corresponding to the target video;

Taking the target object as a clipping target, clipping the video frames in the target video to obtain a plurality of clipping video frames;

A cropped video is generated from the plurality of cropped video frames.

It is understandable that when the computer program is executed by the processor 65, each process of the method embodiment shown in FIG. 1 can be implemented, and the same technical effect can be achieved. In order to avoid repetition, details are not repeated here.

In FIG. 6, a bus architecture (represented by bus 61), which may include any number of interconnected buses and bridges, bus 61 will include one or more processors represented by processor 65 and memory represented by memory 66 The various circuits are linked together. The bus 61 may also link together various other circuits such as peripherals, voltage regulators and power management circuits, etc., which are well known in the art and therefore will not be described further herein. Bus interface 66 provides an interface between bus 61 and transceiver 62 . Transceiver 62 may be a single element or multiple elements, such as multiple receivers and transmitters, providing means for communicating with various other devices over a transmission medium. The data processed by the processor 65 is transmitted on the wireless medium through the antenna 63 , and further, the antenna 63 also receives the data and transmits the data to the processor 65 .

Processor 65 is responsible for managing bus 61 and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. Instead, memory 66 may be used to store data used by processor 65 in performing operations.

Optionally, the processor 65 may be a CPU, ASIC, FPGA or CPLD.

Embodiments of the present invention further provide a computer-readable storage medium, on which programs or instructions are stored, and when the programs or instructions are executed by a processor, each process of the method embodiment shown in FIG. 1 can be implemented and can achieve the same In order to avoid repetition, the technical effect will not be repeated here.

Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a service classification device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims (10)

1. A video clipping method is applied to electronic equipment and is characterized by comprising the following steps:

acquiring a target video to be processed;

respectively determining the core degree of each object in the target video;

determining the object with the core degree larger than a preset threshold value as a target object corresponding to the target video according to the core degree of each object;

with the target object as a cutting target, cutting video frames in the target video to obtain a plurality of cutting video frames;

and generating a cutting video according to the plurality of cutting video frames.

2. The method of claim 1, wherein the separately determining the core degree of each object in the target video comprises:

respectively determining the core degree of each object in the target video based on at least one of the following items:

a continuity ranking of the each object in the target video;

whether each object is occluded in the target video;

whether each object is a preset object or not;

whether each of the objects is speaking in the target video;

whether each object has the display of a preset action in the target video or not;

and the emotional expression of each object in the target video.

3. The method of claim 1, wherein the separately determining the core degree of each object in the target video comprises:

for each of the objects, the following processes are respectively performed:

analyzing the object to obtain a plurality of characteristic values of the object;

and calculating to obtain the core degree of the object according to the plurality of characteristic values and the weight of each characteristic value.

4. The method according to claim 3, wherein the plurality of feature values include a feature value S, a feature value R, a feature value K, a feature value T, a feature value A, and a feature value E;

the calculating the core degree of the object according to the plurality of characteristic values and the weight of each characteristic value comprises:

calculating to obtain the core degree I of the object by adopting the following formula:

I＝a*S+b*(1/R)+c*K+d*T+e*A+f*E

wherein, the value of S is 0 or 1, 0 represents that the object is shielded in the target video, and 1 represents that the object is not shielded in the target video; r represents the continuity ranking of the object in the target video, and the value of R is a positive integer; k takes the value of 0 or 1, 0 represents that the object is not a preset object, and 1 represents that the object is a preset object; the value of T is 0 or 1, 0 represents that the object does not speak in the target video, and 1 represents that the object speaks in the target video; the value of A is 0 or 1, wherein 0 represents that the object has no display of a preset action in the target video, and 1 represents that the object has the display of the preset action in the target video; the value of E is 0 or 1, wherein 0 represents that the object has no preset emotional expression in the target video, and 1 represents that the object has the preset emotional expression in the target video; a is the weight of S, b is the weight of (1/R), c is the weight of K, d is the weight of T, E is the weight of A, and f is the weight of E.

5. The method according to claim 1, wherein when the number of the target objects is plural, the cropping the video frame in the target video to obtain plural cropped video frames comprises:

under the condition that a first video frame comprising a target object exists in the target video, the abscissa of the central position point of the target object is used as the abscissa of the central point of the cutting frame, and the cutting video frame is obtained by cutting from the first video frame;

and/or the presence of a gas in the gas,

and in the case that a second video frame comprising a plurality of target objects exists in the target video, cutting out a cut video frame from the second video frame based on the position of a core target object in the plurality of target objects.

6. The method of claim 5, wherein the core target object satisfies any one of the following conditions:

a core degree is highest among the plurality of target objects;

the center point position of the core target object among the plurality of target objects is closest to the center position of the second video frame.

7. The method of claim 6, wherein the cropping from the second video frame based on the position of the core target object of the plurality of target objects when the core target object has a highest core degree among the plurality of target objects comprises:

determining the distance between the core target object and other target objects according to the position of the core target object;

determining a first target object which can be in a cutting box with the core target object at the same time according to the distance;

selecting a target cutting frame according to a preset condition, and cutting the second video frame by using the target cutting frame to obtain a cut video frame; wherein the target crop box completely covers the core target object and the at least one first target object.

8. The method of claim 6, wherein the cropping from the second video frame based on the position of the core target object from the plurality of target objects when the center point position of the core target object is closest to the center position of the second video frame comprises:

taking the abscissa of the central position point of the core target object as the abscissa of the central point of the cutting frame, sliding the cutting frame by a preset step length, and cutting the second video frame by using the sliding cutting frame to obtain a cut video frame; wherein the slid crop box completely covers the core target object, and the slid crop box does not have an incomplete second target object covered therein; the second target object is the other target objects except the core target object in the plurality of target objects.

9. An electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, the program or instructions when executed by the processor implementing the steps of the video cropping method of any of claims 1 to 8.

10. A computer-readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps of the video cropping method of any one of claims 1 to 8.

Images