TW202023285A - Video extracting method and electronic device using the same - Google Patents

Abstract

A video extracting method is provided. The method is used for extracting at least one clip from a video by an electronic device, where the video corresponds to a viewer chat room. The video extracting method includes: obtaining feature information of the video at a plurality of time points of the video, where the feature information includes a refresh frequency of the viewer chat room; and extracting the at least one clip from the video according to the feature information at each time point. In addition, an electronic device using the method is also provided.

Description

Video capturing method and electronic device using the method

The present invention relates to a capturing technology, and more particularly to a video capturing method and an electronic device using the method.

With the advancement of science and technology, the popularization of online live broadcast platforms and the rise of various online audio and video platforms, entertainment audio and video are readily available, which has changed people's audiovisual viewing behavior. From sitting in front of the TV screen at a fixed time, to on-demand online audio and video, online audio and video brings new interaction and viewing experience.

However, due to the lowering of the live broadcast threshold, everyone can become a live broadcast host anytime and anywhere as long as they download a live application (APP) or log in to the live broadcast platform. Live broadcast has become a new entertainment mode. However, often a live broadcast often exceeds one hour and the time is uncertain, and the audience will not always pay attention to the complete content of a live broadcaster anytime every day. Therefore, in order to attract viewers, live broadcasters usually spend extra time filtering lengthy live broadcast content and editing essential clips to attract fans to watch. In addition, live broadcasters can also share the streamlined version of the best videos to other social platforms to increase visibility and increase subscription audiences. Based on this, how to design an automatic method to edit the essence clips is one of the topics studied by the technicians in the field.

The present invention provides a video capturing method and an electronic device using the method, which can automatically analyze the video to capture the essence of the video, thereby saving the user's time to edit the video.

The video capturing method of the present invention is used to capture at least one clip of a video with an electronic device, wherein the video corresponds to an audience chat room, and the video capturing method includes: obtaining characteristics of the video at multiple time points Information, wherein the characteristic information includes the refresh rate of the audience chat room; and the at least one clip is extracted from the video according to the characteristic information at each time point.

In an embodiment of the present invention, the step of extracting the at least one clip from the video based on the feature information at each time point includes: using a neural network model based on the feature information at each time point The images of the film at each point in time are classified into either a highlight category or a non-essential category, wherein the neural network model is constructed based on a plurality of historical highlights; and a generated image is generated based on the images classified as the highlight category Describe at least one clip.

In an embodiment of the present invention, the step of obtaining the feature information of the movie at the time point includes: recording the latest update message of the audience chat room at the first time point; and at the second time point Find out the location of the most recently updated message in the audience chat room; and calculate the movie at the first time point according to the first time point, the second time point, and the location The refresh frequency.

In an embodiment of the present invention, the above-mentioned feature information includes expression score and volume, and the step of obtaining the feature information of the movie at the time point includes: performing a human image of the movie at each time point. Face detection and obtaining one of multiple expression categories; and setting the expression score according to the expression category.

In an embodiment of the present invention, the step of extracting the at least one clip from the movie according to the feature information at each time point includes: according to the refresh frequency, the expression score, and Calculating a feature score by the volume; and extracting the at least one clip from the movie according to the feature score, wherein when calculating the feature score, the weight value of the refresh frequency is higher than the weight value of the expression score And the weight value of the volume.

The electronic device of the present invention is used to capture at least one clip of a movie, wherein the movie corresponds to the audience chat room, and the electronic device includes: a storage device that records a plurality of modules; and a processor coupled to the A storage device to access and execute the module recorded in the storage device, the module includes: a data collection module to obtain feature information of the film at multiple time points, wherein the feature information includes The refresh rate of the audience chat room; and a video capture module, which captures the at least one clip from the video according to the characteristic information at each time point.

In an embodiment of the present invention, the above-mentioned video capture module includes: according to the feature information at each time point, a neural network model is used to classify the image of the video at each time point into a highlight category or a non-essential category Categories, wherein the neural network model is constructed based on a plurality of historical highlights videos; and the at least one clip is generated according to the images classified into the highlights category.

In an embodiment of the present invention, the above-mentioned data collection module includes: recording the most recent update message of the audience chat room at the first time point; and finding out that the most recent update message is in the audience at the second time point. The location in the chat room; and calculating the refresh frequency of the movie at the first time point according to the first time point, the second time point, and the location.

In an embodiment of the present invention, the aforementioned feature information includes expression scores and volume, and the data collection module includes: performing face detection on images of the video at various points in time and obtaining one of multiple expression categories One; and setting the expression score according to the expression category.

In an embodiment of the present invention, the above-mentioned video capture module includes: calculating a feature score according to the refresh frequency, the expression score, and the volume at each time point; and according to the feature score, from the The video captures the at least one clip, wherein when calculating the feature score, the weight value of the refresh frequency is higher than the weight value of the expression score and the weight value of the volume.

Based on the above, the video capturing method and the electronic device using the method provided by the embodiments of the present invention can input the feature information of the video at multiple time points into the neural network model to classify the images at each time point into the essence category Or non-essential category, and perform the capture operation on the images classified as the essential category, which can automatically analyze the video to capture the essential fragments in the movie. In this way, the time for users to edit the video can be saved.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, the embodiments are specifically described below and described in detail in conjunction with the accompanying drawings.

FIG. 1 shows a block diagram of an electronic device in an embodiment of the invention.

Please refer to FIG. 1, the electronic device 100 includes a storage device 110 and a processor 120. The electronic device 100 is, for example, a desktop computer (Desktop), a notebook computer (Notebook), a tablet computer (Tablet PC), a smart mobile phone (Smart Phone), or a portable game console (PSP), etc., with computing capabilities to provide editing The functional device is not limited by the present invention.

The storage device 110 is used to record the data collection module 112 and the video capture module 114, and these modules are, for example, programs stored in the storage device 110. In some embodiments, the storage device 100 is, for example, any type of fixed or removable random access memory (random access memory, RAM), read-only memory (read-only memory, ROM), or flash memory. Flash memory or similar components or combinations of the above components are not limited by the present invention.

The processor 120 is coupled to the storage device 110, and loads the data collection module 112 and the video capture module 114 code from the storage device 110 to execute the video capture method according to the embodiment of the present invention. In some embodiments, the processor 120 is, for example, a central processing unit (Central Processing Unit, CPU), or other programmable general-purpose or special-purpose microprocessors, or digital signal processors (Digital Signal Processors). DSP), programmable controllers, application specific integrated circuits (Application Specific Integrated Circuits, ASIC), programmable logic devices (Programmable Logic Device, PLD) or other similar devices or a combination of these devices, the present invention does not Be restricted.

In order to be able to automatically capture the essence of the video, the electronic device 100 of the embodiment of the present invention analyzes the feature information of the video at each time point to perform a video capture operation, wherein the analyzed feature information includes the corresponding video at each time point The refresh rate of the audience chat room. In this way, the electronic device 100 can immediately provide the essence of the video, so as to reduce the time for the user to edit the video.

FIG. 2 shows a flowchart of a video capturing method according to an embodiment of the invention.

The video capturing method of the embodiment of FIG. 2 is applicable to the electronic device 100 of the embodiment of FIG. 1. Hereinafter, the video capturing method of the embodiment of FIG. 2 will be described in detail with reference to various elements of the embodiment of FIG. 1. It should be particularly noted that the film in this embodiment corresponds to an audience chat room. During the video playback, the audience of the video can leave a message in the audience chat room. In some embodiments, the image in the film includes the audience chat room, which is not limited by the present invention.

In step S220, the processor 120 executes the data collection module 112, and the processor 120 obtains feature information of the movie at multiple time points, and the feature information includes the refresh rate of the audience chat room. Specifically, the way for the processor 120 to obtain the refresh frequency of the audience chat room is as follows: First, the processor 120 records the latest update message of the audience chat room at the first time point. Then, the processor 120 finds the location of the most recently updated message in the audience chat room at the second time point. Finally, the processor 120 counts how many new messages appear between the first time point and the second time point according to the first time point, the second time point, and the location, and then the processor 120 calculates the number of new messages The number is divided by the difference between the first time point and the second time point to calculate the refresh frequency of the movie at the first time point.

The following examples are given to illustrate the specific manner in which the processor 120 obtains the refresh frequency of the audience chat room.

3A and 3B are schematic diagrams of images in a movie according to an embodiment of the invention. FIG. 3A shows an image 300a at the time point t seconds in the movie, and the latest update message of the audience chat room 320 is displayed as the message 322 in the image 300a. FIG. 3B shows the image 300b at the time point t+3 seconds in the movie, and the position of the message 322 in the audience chat room 320 is displayed in the image 300b at the time point t+3 seconds. Based on the time point t seconds, time point t+3 seconds, and the location of the message 322 at time point t+3 seconds, the processor 120 counts that 9 new messages appear between time point t seconds and time point t+3 seconds Leave a message, so the video refresh frequency at time t is 3 times/sec.

In some embodiments, the characteristic information further includes expression score and volume. The way for the processor 120 to obtain the volume is as follows: the processor 120 counts the decibels at each time point according to the audio track of the movie, and the decibels are the volume. The way for the processor 120 to obtain the expression score is as follows: First, the processor 120 uses an emotion detection model (for example, Python’s DLIB library, but not limited to this) to perform face detection on images of the movie at each time point. Measure and obtain one of a variety of expression categories. The expression categories include plain expressions, scared expressions, or happy expressions, which are not limited by the present invention. Then, the processor 120 sets an expression score according to the expression category. In detail, the processor 120 takes out facial expressions that are scared or happy, and quantizes the scared or happy expressions to a value of 1 (the maximum value), that is, the processor 120 quantizes the expressions to a value of 0 to 1. The value 0 is a plain expression, and the value 1 is a scared or happy expression (the value is the expression score). It is worth noting that, in some embodiments, the images in the movie also include the live main window, and the processor 120 only performs face detection for the live main window to reduce the computational burden of the system, but the present invention does not limit.

For example, referring to FIGS. 3A and 3B, the image 300a in FIG. 3A and the image 300b in FIG. 3B include the live main window 340. In the example of FIG. 3A, at the time point t seconds in the movie, the face in the live main window 340 is displayed as a bland expression. Therefore, the processor 120 obtains the face as a bland expression and sets The expression score is 0. In the example of FIG. 3B, at the time t+1 second in the video, the face of the live main window 340 is displayed as a scared expression. Therefore, the processor 120 sets the expression score to be after obtaining the scared expression on the face. 1.

Returning to the flowchart of FIG. 2, in step S240, the processor 120 executes the video capture module, and the processor 120 captures at least one clip from the video according to the characteristic information of each time point. In some embodiments, the processor 120 uses a neural network model to classify the images of the movie at each time point into a highlight category or a non-essential category based on the feature information at each time point, and then generates at least one image based on the images classified as the highlight category. Clip.

In some embodiments, the processor 120 calculates a feature score based on the refresh frequency, expression score, and volume at each time point, and extracts at least one clip from the movie based on the feature score. In particular, when the processor 120 calculates the feature score, the weight value of the refresh frequency is higher than the weight value of the expression score and the weight value of the volume. For example, the processor 120 inputs the feature score into a neural network model to output a probability value, and determines whether the probability value is greater than a preset threshold value, and if the probability value is greater than the preset threshold value, the processor 120 120 classifies the image at the time point into a highlight category; if the probability value is not greater than the preset threshold, the processor 120 classifies the image at the time point as a non-essential category. Then, the processor 120 captures the images classified as the essence category to generate at least one clip. In some embodiments, the feature score is equal to the refresh rate plus the product of the expression score and the volume, which is not limited by the present invention.

In other embodiments, the processor 120 may also directly input the refresh rate, expression score, and volume into the neural network model, so as to classify the images at each time point into the essence category or the non-essential category. However, the present invention is not limited.

It is worth mentioning that in some embodiments, the neural network model is constructed based on multiple historical highlights. Please refer to the following description for the establishment of neural network model. First, the processor 120 obtains at least one historical highlight video, the historical highlight video includes a plurality of highlight images, and each highlight image includes feature information. For example, the historical highlight video is, for example, a highlight video completed by a live broadcast master editing. In detail, after the processor 120 obtains the historical highlights, it uses video object detection to classify the highlights of the historical videos into the live main window and the audience chat room, and the processor 120 obtains the essence of the images. The refresh rate of the audience chat room, the expression score and voice of the live broadcast main window. Then, the processor 120 trains the neural network model by using the essence image and the characteristic information of each essence image (the refresh rate of the audience chat room, the expression score and the voice of the live broadcast main window).

In addition, in some embodiments, the processor 120 may use a data feedback method to perform a re-training process (Re-Training Process) based on whether the user uses the at least one clip as an essential clip. The retraining process performs machine learning in the neural network model for the different refresh frequencies, different expression scores, and different sounds of the images in the newly acquired essence clips, so as to optimize the neural network model and make the future experience The output of the neural network model is more accurate and more in line with personal needs.

In summary, the video capture method and the electronic device using the method provided by the embodiments of the present invention can input the feature information of the video at multiple time points into the neural network model to classify the images at each time point into Highlight category or non-essential category, and perform the capture operation on the images classified as the highlight category, so as to automatically analyze the video to capture the highlights of the video. In this way, the time for users to edit the video can be saved.

Although the present invention has been disclosed as above by the embodiments, it is not intended to limit the present invention. Any person with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be subject to the scope defined in the appended patent application.

100: Electronic device 110: Storage device 112: Data collection module 114: Video capture module 120: Processor S220, S240: Steps 300a, 300b of the video capture method: Image 320: Audience chat room 322: Message 340: Live main window

FIG. 1 shows a block diagram of an electronic device in an embodiment of the invention. FIG. 2 shows a flowchart of a video capturing method according to an embodiment of the invention. 3A and 3B are schematic diagrams of images in a movie according to an embodiment of the invention.

S220, S240: Steps of video capture method

Claims (10)

A video capturing method for capturing at least one clip of a video with an electronic device, wherein the video corresponds to an audience chat room, and the video capturing method includes: obtaining feature information of the video at multiple time points; The feature information includes the refresh rate of the audience chat room; and the at least one clip is extracted from the video according to the feature information at each time point. The method for capturing a video according to claim 1, wherein the step of capturing the at least one clip from the video according to the feature information at each time point includes: according to the feature information at each time point , Using a neural network model to classify the images of the video at each point in time into a highlight category or a non-essential category, wherein the neural network model is constructed based on a plurality of historical highlights; and classified as the highlight category The image of generates the at least one clip. The method for capturing a video as described in claim 1, wherein the step of obtaining the feature information of the video at the time point includes: recording the latest update message of the audience chat room at the first time point ; Find out the location of the most recently updated message in the audience chat room at the second time point; and calculate the location of the movie based on the first time point, the second time point and the location The refresh frequency at the first time point. The video capture method described in any one of items 1 to 3 of the scope of patent application, wherein the feature information includes expression score and volume, and the feature information of the video at the time point is obtained The steps include: performing face detection on the image of the movie at each time point and obtaining one of multiple expression categories; and setting the expression score according to the expression category. The method for capturing a video according to claim 4, wherein the step of capturing the at least one clip from the video according to the characteristic information at each time point includes: according to the refresh frequency at each time point , The expression score and the volume to calculate a feature score; and according to the feature score, extract the at least one clip from the movie, wherein when calculating the feature score, the refresh frequency weight value is higher than the The weight value of the expression score and the weight value of the volume. An electronic device for capturing at least one clip of a movie, wherein the movie corresponds to an audience chat room, the electronic device includes: a storage device that records a plurality of modules; and a processor coupled to the storage device , To access and execute the module recorded in the storage device, the module includes: a data collection module to obtain feature information of the movie at multiple points in time, wherein the feature information includes the The refresh frequency of the audience chat room; and the video capture module, which captures the at least one clip from the video according to the characteristic information at each time point. According to the electronic device described in item 6 of the scope of patent application, the video capture module includes: according to the feature information at each time point, a neural network model is used to classify the image of the video at each time point into A highlight category or a non-essential category, wherein the neural network model is constructed based on a plurality of historical highlights; and the at least one clip is generated based on the images classified into the highlight category. The electronic device according to item 6 of the scope of patent application, wherein the data collection module includes: recording the latest update message of the audience chat room at the first time point; and finding the latest update at the second time point The location of the message in the audience chat room; and calculating the refresh frequency of the movie at the first time point according to the first time point, the second time point, and the location. The electronic device according to any one of items 6 to 8 of the scope of patent application, wherein the characteristic information includes expression scores and volume, and the data collection module includes: The image performs face detection and obtains one of multiple expression categories; and sets the expression score according to the expression category. The electronic device according to item 9 of the scope of patent application, wherein the video capture module includes: calculating a feature score according to the refresh frequency, the expression score, and the volume at each time point; and according to the feature Score, extract the at least one clip from the movie, wherein when calculating the feature score, the weight value of the refresh frequency is higher than the weight value of the expression score and the weight value of the volume.

Images