TWI704805B - Video editing method and device - Google Patents

Abstract

A video editing method is implemented by a video editing device. The video editing device stores a video including a speaker. The video editing method includes the following steps: (A) Obtain multiple target sub-videos based on the audio of the video Paragraphs; (B) sort the target sub-video segments according to the content of the target sub-video segments; (C) filter out multiple target sub-videos to be synthesized according to the order of the target sub-video segments Sub-video segment; and (D) synthesize the sub-video segment waiting to be synthesized to generate a composite video. In addition, the present invention also provides a video editing device.

Description

Video editing method and device

The present invention relates to a video editing method, in particular to a video editing method used for editing lecture videos or teaching videos.

With the advent of the digital age, videos can be stored, transmitted and circulated more conveniently. Therefore, many existing audio-visual platforms provide speech videos or teaching videos for the public to watch and learn.

However, the content of a complete speech or instructional video will fluctuate. Sometimes the video is too long, which will make the viewer less interested in watching. If you need to find the highlights of the speaker and the main speaking screen in the video, capture the video For the essence part, it is usually necessary for the film editor to filter out the highlights for a long time, and then make the highlight film, which is very time-consuming. Moreover, the selected highlights are often one-way for the film editor. Subjectively determined, so other key highlights may be missed, and the highlight film cannot be presented objectively.

Therefore, the purpose of the present invention is to provide a video editing method that can improve the efficiency of video editing and can be objectively presented.

Therefore, the video editing method of the present invention is implemented by a video editing device that stores a video, the video includes a speaker, and the video editing method includes one step (A), one step (B), one Step (C), and one step (D).

In this step (A), the video editing device obtains a plurality of target sub-video segments according to the audio of the video.

In the step (B), the video editing device sorts the target sub-video segments according to the content of the target sub-video segments.

In the step (C), the video editing device selects multiple sub-video segments to be synthesized from the target sub-videos according to the order of the target sub-video segments.

In the step (D), the video editing device synthesizes the sub-video segments waiting to be synthesized to generate a synthesized video.

The purpose of the present invention is to provide a video editing device that can improve the efficiency of video editing and can present objectively.

Therefore, the video editing device includes a storage unit and a processing unit.

The storage unit stores a video, and the video includes a speaker.

The processing unit is electrically connected to the storage unit. The processing unit obtains a plurality of target sub-video segments according to the audio information of the video, and then sorts the target sub-video segments according to the content of the target sub-video segments, and then according to the targets The sequence of the sub-video segments is to filter out multiple sub-video segments to be synthesized from the target sub-videos, and finally synthesize the sub-video segments to be synthesized to generate a synthesized video.

The effect of the present invention is that the video editing device sorts the target sub-video segments according to the content of the target sub-video segments, and filters the target sub-videos according to the order of the target sub-video segments The waiting to be synthesized sub-movie paragraphs are generated to improve the efficiency of movie editing and present the highlight movie objectively.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are represented by the same numbers.

Referring to FIG. 1, an embodiment of a video editing apparatus 100 of the present invention includes a storage unit 11 and a processing unit 12 electrically connected to the storage unit 11. The storage unit 11 stores a video, and the video includes a speaker.

Referring to FIGS. 1 and 2, it will be described how the video editing apparatus 100 of the present invention executes an embodiment of the method for generating a gold flow relationship graph of the present invention.

In step 201, the processing unit 12 filters out sounds outside a predetermined frequency range in the movie. It is worth noting that in this embodiment, the predetermined frequency range, for example, non-human voice frequencies of 500 Hz to 2000 Hz, to remove background sounds And noise, but not limited to this.

In step 202, the processing unit 12 divides the movie into multiple movie segments according to the audio of the movie. In this embodiment, the processing unit 12 performs Voice Activity Detection on the audio without cutting off the completeness. The segmentation is performed on the premise of the speech paragraph, and the main method is that when the time interval between sound waves is less than a predetermined time (for example, 3 seconds), the same paragraph is represented and regarded as the same movie paragraph.

In step 203, the processing unit 12 filters out the video segments of interest from the video segments according to the audio information of the video segments, wherein the video length of the filtered video segments of interest is greater than a first predetermined period of time (E.g. 6 seconds).

In step 204, for each video segment of interest, the processing unit 12 performs voice recognition on the video segment of interest to obtain a text file.

In step 205, for each movie segment of interest, the processing unit 12 divides the movie segment of interest into a plurality of complete sentences and the length of the movie is less than a second predetermined according to the text file corresponding to the movie segment of interest. The candidate sub-video segments of a time period (for example, 30 seconds), each candidate sub-video segment corresponds to a sub-text file. It is worth noting that, in this embodiment, the processing unit 12 uses Natural Language Processing to segment the text file to obtain the candidate sub-video paragraphs. In other embodiments, it can also be used The Bi-LSTM-CRF model or the deep learning model performs word segmentation on the text file, which is not limited.

With reference to FIG. 3, it can be seen from the sound wave diagram that the processing unit 12 divides the video into a plurality of video segments A, B, and C according to the audio information of the video, and the time interval between the sound waves of the video segments A, B, and C Is greater than or equal to the predetermined time, and the video lengths of the video segments A, B, and C are all greater than the first predetermined time period, so they are all interesting video segments, and the processing unit 12 then according to the text corresponding to the interesting video segment A File, divide the movie segment A of interest into multiple candidate sub-movie paragraphs A1, A2, A3, and divide the movie segment B of interest into multiple candidate sub-movie paragraphs B1 according to the text file corresponding to the movie segment B of interest. B2, B3, according to the text file corresponding to the interesting video segment C, divide the interesting video segment A into a plurality of candidate sub-movie segments C1, C2.

In step 206, the target sub-video segments are filtered out from the candidate sub-video segments corresponding to the video segments of interest. Referring to FIG. 4 in conjunction, step 206 includes sub-steps 61 and 62. Steps 61 and 62 are described below.

In step 61, for each candidate sub-video segment, the processing unit 12 corresponds to the sub-text file corresponding to the sub-text file corresponding to the candidate sub-video segment, including words that are repeated a predetermined number of times (for example, 3 times) continuously. The candidate sub-movie paragraphs are deleted.

In step 62, for each undeleted candidate sub-film segment, the processing unit 12 deletes the candidate sub-film segment corresponding to a loudness greater than a predetermined decibel (for example, 90 decibels).

It should be particularly noted that in this embodiment, step 61 precedes step 62. In other embodiments, step 62 may also precede step 61, which is not limited thereto.

In step 207, the processing unit 12 sorts the target sub-film segments according to the content of the target sub-film segments. With reference to FIG. 5, step 207 includes sub-steps 71-80, and steps 71-80 are described below.

In step 71, for each image of the target sub-video segment, the processing unit 12 obtains a plurality of first facial feature points (such as eyes, nose, mouth, left sideburn, left sideburn, etc.) related to the speaker in the image. Right sideburn, etc.) to determine the position range of the speaker’s face in the image. It is worth noting that in this embodiment, the processing unit 12 uses the open source OpenCV as a tool for capturing the first facial feature points, and uses the first facial feature points to calculate the angle and range of the face , And provide a lot of data to train its accuracy before use, but not limited to this.

In step 72, for each target sub-film segment, the processing unit 12 determines, according to all the first facial feature points of the target sub-film segment, that the face related to the speaker is in the face of the target sub-film segment. The position state of the face, where the position state of the face indicates a centered state and a non-centered state. It is worth noting that in this embodiment, for each image, when the length and width of the face of the speaker are within a predetermined range (for example, 40% to 70%), and the The ratio of the distance between the speaker’s face in the image and each edge of the image to the image is greater than or equal to a preset value (for example (100%-average aspect ratio)*k%), where 0＜k <1, the processing unit 12 regards the image as a face centered, and for each target sub-movie segment, when the number of image frames deemed to be face centered is greater than the number of image frames deemed to be face uncentered , The face position state of the target sub-movie segment indicates the center state, but not limited to this.

Refer to Figure 6 together. For example, in one of the images, the length is X and the width is Y. The distance of the speaker’s face in the image is x ₁ from the left edge of the image and x ₃ from the right edge of the image. The upper edge of the distance image is y ₁ and the lower edge of the distance image is y _3. The length of the speaker’s face in the image is x ₂ and the width is y ₂ , then when x ₂ /X and y ₂ /Y When within the predetermined range and x ₁ /X, x ₃ /X, y ₁ /Y, and y ₃ /Y are all greater than or equal to the preset value, the processing unit 12 regards the image as a face centered.

In step 73, the processing unit 12 sorts the target sub-video segments according to the face position states corresponding to the target sub-video segments. It is worth noting that in this embodiment, the processing unit 12 divides the target sub-movie segments into 2 groups, and the face position state indicates that the target sub-movie segment in the centered state is divided into a group and sorted first. The partial position status indicates that the target sub-movie segment in the non-centered state is divided into another group and sorted later, as shown in Table 1 below. Table 1 Order: first -------------------------------------- second Face position status Set in Non-centered

In step 74, for each image in the target sub-video segment, the processing unit 12 obtains a plurality of second facial feature points related to the speaker in the target sub-video segment. It is worth noting that, in this embodiment, the processing unit 12 uses, for example, the concept of Facial Action Coding System (Facial Action Coding System, FACS), and uses OpenCV to capture the second facial feature points, but not This is limited.

In step 75, for each target sub-film segment, the processing unit 12 determines an expression emotional state related to the speaker according to all the second facial feature points of the target sub-film segment, wherein the expression emotional state Indicates one of a positive state, a general state, and a negative state. It is worth noting that in this embodiment, if the processing unit 12 recognizes that the speaker’s eyes are open and the corners of his mouth are raised according to the second facial feature points, it is determined that the emotional state of the expression indicates the positive direction. State; if the processing unit 12 recognizes that the speaker’s eyes are open and the corners of his mouth are flush based on the second facial feature points, it is determined that the emotional state of the expression indicates the general state; if the processing unit 12 is based on the After the second facial feature point recognizes that the speaker has eyes closed and the corner of his mouth is downward, it is determined that the emotional state of the expression indicates the negative state, but not limited to this.

In step 76, the processing unit 12 sorts the target sub-video segments according to the emotional state of the expressions corresponding to the target sub-video segments. It is worth noting that, in this embodiment, the processing unit 12 divides the target sub-movie paragraphs into 5 groups, and respectively indicate the centered state for the face position state and the facial emotion state indicates the positive The target sub-movie segment of the state, the face position state indicates the centered state and the expression emotional state indicates the target sub-movie paragraph of the general state, the face position state indicates the non-centered state, and the expression emotional state The target sub-video segment indicating the positive state, the face position state indicating the non-centered state and the target sub-video segment indicating the general state of the expression emotional state, and the expression emotional state indicating the target of the negative state Sub-movie paragraphs are shown in Table 2 below. Table 2 Order: first -------------------------------------- second Face position status Set in Non-centered Centered/not centered Emotional state positive general positive general Negative

In step 77, for each image of the target sub-video segments, the processing unit 12 obtains a plurality of physical feature points in the image related to the speaker.

In step 78, for each target sub-film segment, according to all the physical feature points of the target sub-film segment, a physical emotional state related to the speaker is determined, wherein the physical emotional state indicates a positive state, One of a general state and a negative state. It is worth noting that, in this embodiment, the processing unit 12 first determines the position of the speaker’s limbs in each frame of the image based on all the limb feature points of each target sub-movie, and then determines the position of the limbs For the emotional state of the limbs, if the speaker raises his hands, raises one hand, and moves at a normal speed, the processing unit 12 determines that the emotional state of the limbs indicates the positive state; if the torso of the limbs is skewed or the limbs are abnormally shaking , The body’s torso moves too fast, the processing unit 12 determines that the emotional state of the limb indicates the negative state; in other cases, the processing unit 12 determines that the emotional state of the limb indicates the general state, in which hands are raised, Raising one hand, skewed limbs, etc. can be judged by identifying the position of main limb feature points (such as shoulders, elbows, etc.), while normal movement speed, excessive movement speed, abnormal shaking, etc. can be determined by specific limb feature points (such as The movement speed of the limbs and trunk) is used to determine, but not limited to this.

In step 79, the processing unit 12 sorts the target sub-video segments according to the emotional state of the limbs corresponding to the target sub-video segments. It is worth noting that in this embodiment, the processing unit 12 divides the target sub-movie paragraphs into 9 groups, and respectively indicate the centered state for the face position state and the emotional state of the facial expression indicates the front face. State and the limb emotional state indicates the target sub-video segment of the positive state, the face position state indicates the centered state, the expression emotional state indicates the positive state, and the limb emotional state indicates the target of the general state The sub-video segment, the face position state indicates the centered state, the expression emotional state indicates the target sub-video segment of the general state, and the body emotional state indicates the target sub-video segment of the positive state, and the face position The state indicates the centered state and the emotional state of expression indicates the target sub-movie section of the general state and the body emotional state indicates the target sub-movie section of the general state, the face position state indicates the non-centered state, and The emotional state of the expression indicates the positive state and the emotional state of the limb indicates the target sub-video segment of the positive state, the position state of the face indicates the non-centered state, and the emotional state of the expression indicates the positive state and the physical emotion The state indicates the target sub-movie section of the general state, the face position state indicates the non-centered state, the expression emotional state indicates the general state, and the body emotional state indicates the target sub-movie section of the positive state, the The face position state indicates the non-centered state, the expression emotional state indicates the general state, and the body emotional state indicates the target sub-video segment of the general state, and the expression emotional state indicates the negative state or the body emotion The status indicates the target sub-movie segment of the negative status, as shown in Table 3 below. table 3 Order: first-------------------------------------------- second Face position status Set in Non-centered Centered/not centered Emotional state positive general positive general Positive/normal/negative Physical emotional state positive general positive general positive general positive general Positive/normal/negative

In step 80, the processing unit 12 sorts the target sub-videos according to the number of occurrences of at least one predetermined word (such as keyword words, synonyms) in the sub-text file corresponding to each target sub-video paragraph paragraph. It is worth noting that, in this embodiment, the processing unit 12 sorts the 9 groups respectively, and the more the number of occurrences of the at least one predetermined word in the corresponding sub-text file, the higher the sequence of the target sub-video paragraph ( If the number of occurrences is the same, the longer the length of the movie, the earlier), as shown in Table 4. Table 4 Sequence: first---------------------------------------------last Face position status Set in Non-centered Centered/not centered Emotional state positive general positive general Positive/normal/negative Physical emotional state positive general positive general positive general positive general Positive/normal/negative Predetermined term More at least More at least More at least More at least More at least More at least More at least More at least More at least

It should be particularly noted that in other embodiments, steps 71 to 73 can be after steps 74 to 76 or 77 to 79, and steps 74 to 76 can be after steps 77 to 79. It is not limited to this. The order is different, the order obtained is also different.

In step 208, the processing unit 12 filters out a plurality of sub-video segments to be synthesized from the target sub-videos according to the ordering of the target sub-video segments, wherein the total length of the sub-video segments waiting to be synthesized is less than A third predetermined time period (for example, 60 seconds). It should be noted that, in this embodiment, the sub-movie segments waiting to be synthesized belong to different video segments of interest, but it is not limited to this.

In step 209, the processing unit 12 synthesizes the sub-movie segment waiting to be synthesized to generate a synthesized movie.

To sum up, in the video editing method and device of the present invention, the processing unit 12 of the video editing device 100 sorts the target sub-video segments according to the content of the target sub-video segments, and according to the target sub-video segments The sequence of the video segments is to filter out the sub-video segments waiting to be synthesized from the target sub-videos, so as to improve the editing efficiency of the video and present the highlights of the video objectively, so it can indeed achieve the purpose of the invention.

However, the above are only examples of the present invention. When the scope of implementation of the present invention cannot be limited by this, all simple equivalent changes and modifications made in accordance with the scope of the patent application of the present invention and the content of the patent specification still belong to Within the scope of the patent for the present invention.

100: Video editing device

11: storage unit

12: Processing unit

201~209: Steps

61, 62: steps

71~80: steps

Other features and effects of the present invention will be clearly presented in the embodiments with reference to the drawings, in which: Figure 1 is a block diagram illustrating an embodiment of the video editing device of the present invention; Figure 2 is a flowchart illustrating an embodiment of the film editing method of the present invention; Figure 3 is a schematic diagram illustrating that a movie is divided into multiple movie segments; Fig. 4 is a flowchart to assist in explaining sub-steps 61 and 62 of step 206 in Fig. 2; FIG. 5 is a flowchart to assist in explaining sub-steps 71 to 80 of step 207 in FIG. 2; and FIG. 6 is a schematic diagram illustrating the state of determining a face position of an image of a target sub-movie segment.

201~209: Steps

Claims (13)

A video editing method is implemented by a video editing device. The video editing device stores a video including a speaker. The video editing method includes the following steps: (A) Obtain multiple target sub-videos based on the audio of the video Paragraphs; (B) sort the target sub-video segments according to the content of the target sub-video segments, wherein step (B) includes the following sub-steps (B-1) for each image of the target sub-video segments , To obtain multiple facial feature points related to the speaker in the image, (B-2) For each target sub-video segment, determine that it is related to the speaker based on all the facial feature points in the target sub-video segment The face of is in a face position state of the target sub-video segment, where the face position state indicates a centered state and a non-centered state, and (B-3) corresponding to the target sub-video segments Sort the target sub-video segments according to the position state of the face; (C) filter out multiple sub-video segments to be synthesized from the target sub-videos according to the sorting of the target sub-video segments; and (D) the waiting to be synthesized The sub-video paragraphs are synthesized to produce a synthesized video. The video editing method according to claim 1, wherein step (A) includes the following sub-steps: (A-1) divide the video into multiple video segments according to the audio of the video; (A-2) according to these The audio of video segments is filtered from those video segments Out the video segments of interest; (A-3) For each video segment of interest, according to the audio of the video segment of interest, divide the video segment of interest into multiple candidate sub video segments; and (A-4) ) Filter out the target sub-video segments from the candidate sub-video segments corresponding to the interesting video segments. The video editing method according to claim 2, wherein in step (A-2), the length of the selected video segments of interest is greater than a first predetermined time period. The video editing method according to claim 2, wherein step (A-3) includes the following sub-steps: (A-3-1) For each video segment of interest, perform voice recognition on the video segment of interest to Obtain a text file; and (A-3-2) For each video segment of interest, according to the corresponding text file, segment the video segment of interest into the segment including complete sentences and the video length less than a second predetermined period of time Wait for candidate sub-video paragraphs, and each candidate sub-video paragraph corresponds to a sub-text file. The video editing method according to claim 4, wherein, in step (A-4), for each candidate sub-video paragraph, corresponding to the sub-text file corresponding to the sub-text file includes repeating a predetermined number of times continuously The candidate sub-video segments of the word "Zhi" are deleted to filter out the target sub-video segments. The video editing method according to claim 4, wherein, in step (B), the target sub-video segments are sorted according to the sub-text files corresponding to the target sub-video segments. The video editing method according to claim 6, wherein, in step (B), the video editing device is based on the number of occurrences of at least one predetermined word in the sub-text file corresponding to each target sub-video paragraph Sort the target sub-video segments. The film editing method according to claim 4, wherein, in step (A-4), for each candidate sub-movie section, delete candidate sub-movie sections corresponding to a loudness greater than a predetermined decibel to filter out the The target sub-movie paragraph. The film editing method as described in claim 1, before step (A), further includes the following steps: (G) filtering out sounds outside a predetermined frequency interval in the film. The film editing method according to claim 1, wherein, in step (C), the total film length of the sub-segments waiting to be synthesized is lower than a third predetermined time period. The video editing method according to claim 1, wherein step (B) further includes the following sub-steps: (B-4) For each image in the target sub-video paragraphs, obtain information related to the speaker in the image Multiple physical feature points; (B-5) For each target sub-film segment, according to all the physical feature points in the target sub-film segment, determine a physical emotional state related to the speaker, where the physical emotional state indicates One of a positive state, a general state, and a negative state; and (B-6) sort the target sub-video segments according to the emotional state of the body corresponding to the target sub-video segments. The film editing method according to claim 1, wherein step (B) further includes: The next sub-step: (B-4) For each image in the target sub-video segment, obtain multiple facial feature points related to the speaker in the target sub-video segment; (B-5) For the target sub-video In the video segment, according to all the facial feature points of the target sub-video segment, an expression emotional state related to the speaker is determined, wherein the expression emotional state indicates a positive state, a general state, and a negative state. One of them; and (B-6) sort the target sub-video segments according to the emotional state of the expressions corresponding to the target sub-video segments. A video editing device includes: a storage unit storing a video including a speaker; a processing unit electrically connected to the storage unit to obtain a plurality of target sub-video segments according to the audio of the video, and then according to the targets The content of the sub-video paragraphs sorts the target sub-video paragraphs, wherein for each image of the target sub-video paragraphs, multiple facial feature points related to the speaker in the image are obtained, and for each target sub-video A video segment, based on all facial feature points of the target sub-video segment, it is determined that the face related to the speaker is in a face position state of the target sub-video segment, wherein the face position state indicates a centered state And a non-centered state, sort the target sub-video segments according to the face position states corresponding to the target sub-video segments, and then filter out multiple target sub-videos according to the order of the target sub-video segments The sub-video segment to be synthesized is finally synthesized to generate a synthesized video.

Images