TWI712316B - Method and device for generating video summary - Google Patents

Abstract

The embodiment of the application discloses a method and device for generating a video summary, wherein the video has text description information, and the method includes: extracting a plurality of scene switching frames from the video, and setting the scene switching frame Scene label, wherein the similarity between two adjacent scene switching frames meets a specified condition; extract the topic tag corresponding to the video from the text description information; switch the scene tag of the frame according to the scene tag and the topic Based on the correlation between the tags, a target frame is filtered from the multiple scene switching frames, and a video summary of the video is generated based on the target frame. The technical solution provided by this application can accurately characterize the subject of the video while improving efficiency.

Description

Method and device for generating video summary

This application relates to the field of Internet technology, and in particular to a method and device for generating video abstracts.

Currently, in order to let users know the content of the video in a short time, the video playback platform usually makes a corresponding video summary for the uploaded video. The video summary may be a short video, and the video summary may include a part of scenes in the original video. In this way, the user can quickly understand the approximate content of the original video when viewing the video summary.

At present, when making a video summary, on the one hand, manual editing can be used to first watch the entire video by the staff of the video playback platform, and then edit the more critical segments to form the video summary of the video. The video summary produced in this way can more accurately characterize the information contained in the video, but with the rapid growth of the number of videos, this method of producing video summary will consume a lot of manpower, and the speed of preparing the video summary is also considerable. slow.

In view of this, in order to save manpower and improve the production efficiency of video summaries, video summaries are usually produced through image recognition technology. Specifically, the uploaded video can be sampled at a fixed time interval to extract multiple frames of images in the video. Then the similarity between two adjacent frames of images can be calculated in turn, and the two images with lower similarity can be retained, so as to ensure that the retained image frames can display the content of multiple scenes. In this way, the finally retained image frames can be formed into a video summary of the video.

In the prior art, the method of making a video summary through image recognition can improve the efficiency of production, but it is easy to miss the key points in the video by selecting the image frames in the video summary by means of fixed sampling and comparing similarity. The scene, resulting in the generated video summary cannot accurately reflect the subject of the video.

The purpose of the embodiments of the present application is to provide a method and device for generating a video summary, which can accurately characterize the subject of the video while improving efficiency.

In order to achieve the above objective, the embodiments of the present application provide a method for generating a video summary, the video having text description information, and the method includes: extracting a plurality of scene switching frames from the video, and setting the scene switching frame Scene label, wherein the similarity between two adjacent scene switching frames meets a specified condition; extract the topic tag corresponding to the video from the text description information; switch the scene tag of the frame according to the scene tag and the topic Based on the correlation between the tags, a target frame is filtered from the multiple scene switching frames, and a video summary of the video is generated based on the target frame.

In order to achieve the above objective, the embodiments of the present application also provide a device for generating a video summary, the video having text description information, and the device includes: a scene switching frame extraction unit for extracting multiple scene switching frames from the video , And set a scene tag for the scene switching frame, where the similarity between two adjacent scene switching frames meets a specified condition; the topic tag extraction unit is used to extract the video corresponding to the text description information Topic tag; a video summary generating unit for filtering out target frames from the plurality of scene switching frames according to the correlation between the scene tags of the scene switching frames and the topic tags, and based on the target frame Generate a video summary of the video.

It can be seen from the above that this application can first extract scene switching frames whose similarity meets the specified conditions from the video, and set corresponding scene tags for the scene switching frames. Then, the subject tag of the video can be determined by combining the text description information of the video. The subject tag can accurately represent the subject of the video. Then, by determining the relevance between the scene tag and the topic tag, the target frame that is more closely related to the topic can be retained from the scene switching frame. In this way, the video summary generated based on the target frame can accurately characterize the subject matter of the video.

100‧‧‧Scene switching frame extraction unit

200‧‧‧Subject tag extraction unit

300‧‧‧Video Summary Generation Unit

S1, S3, S5‧‧‧Step

In order to more clearly explain the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are merely the present For some of the implementations described in the application, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

Fig. 1 is a flowchart of a method for generating a video summary in an embodiment of the application; Fig. 2 is a schematic diagram of a target frame and a scene switching frame in an embodiment of the application; Fig. 3 is a schematic diagram of extracting a scene switching frame in an embodiment of the application; This is a schematic diagram of extracting scene tags in an embodiment of this application; FIG. 5 is a functional block diagram of an apparatus for generating a video summary in an embodiment of this application.

In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the drawings in the embodiments of this application. Obviously, the described The embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the implementation in this application, all other implementations obtained by a person of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The present application provides a method for generating a video digest, which can be applied to electronic equipment with data processing functions. The electronic device may be, for example, a desktop computer, a tablet computer, a notebook computer, a smart phone, a digital assistant, a smart wearable device, a shopping guide terminal, a TV with Internet access function, etc. The method can also be applied to software running in the above electronic equipment. The software may be a software with a video production function or a video playback function. In addition, the method can also be applied to the server of a video broadcasting website. The video playback website may be, for example, iQiyi, Sohu Video, Acfun, etc. In this embodiment, the number of the servers is not specifically limited. The server may be one server, or several servers, or a server cluster formed by several servers.

In this embodiment, the video summary may be generated based on video. The video can be a local video of the user, or a video uploaded by the user to a video playback website. Wherein, the video can usually have text description information. The text description information may be the title of the video or the introduction of the video. The title and the introduction may be pre-edited by the video producer or the video uploader, or added by the staff reviewing the video, and the comparison of this application is not limited. Of course, in practical applications, in addition to the title and introduction of the video, the text description information may also include the text tag of the video or a descriptive phrase extracted from the barrage information of the video.

Please refer to FIG. 1 and FIG. 2. The method for generating a video digest provided by this application may include the following steps.

S1: Extract multiple scene switching frames from the video, and set a scene label for the scene switching frame, wherein the similarity between two adjacent scene switching frames meets a specified condition.

In this embodiment, the video may be a video stored locally or a video stored in other devices. In this way, the method for obtaining the video may include loading the video locally according to a designated path or downloading the video according to a Uniform Resource Locator (URL) provided by other devices.

In this embodiment, after the video is acquired, each frame of the video can be analyzed to extract multiple scene switching frames. In order to be able to obtain the scene switching frames corresponding to each scene of the video, in this embodiment, the extraction can be carried out through a frame-by-frame comparison. Specifically, first, a reference frame may be determined in the video, and the similarity between each frame after the reference frame and the reference frame may be sequentially calculated.

In this embodiment, the reference frame may be a randomly designated frame within a certain range. For example, the reference frame may be a frame selected randomly within the first 2 minutes of the video. Of course, in order not to omit the scene in the video, the first frame of the video can be used as the reference frame.

In this embodiment, after the reference frame is determined, starting from the reference frame, each frame after the reference frame may be sequentially compared with the reference frame to calculate the difference between the subsequent frames The similarity between the reference frames. Specifically, when calculating the similarity between each frame and the reference frame, the first feature vector and the second feature vector of the reference frame and the current frame may be extracted respectively.

In this embodiment, the first feature vector and the second feature vector may have various forms. Among them, the feature vector of the frame can be constructed based on the pixel value of the pixel in each frame. Each frame of picture is usually composed of a number of pixels arranged in a certain order, and the pixels correspond to their respective pixel values, which can form a colorful picture. The pixel value may be a value within a specified interval. For example, the pixel value may be any value from 0 to 255. The size of the value can indicate the depth of the color. In this embodiment, the pixel value of each pixel in each frame can be obtained, and the obtained pixel value can be used to form the feature vector of the frame. For example, for the current frame with 9*9=81 pixels, the pixel values of the pixels can be obtained in sequence, and then the obtained pixel values can be arranged in order from left to right and top to bottom, thus It constitutes an 81-dimensional vector. The 81-dimensional vector can be used as the feature vector of the current frame.

In this embodiment, the feature vector may also be a CNN (Convolutional Neural Network, convolutional neural network) feature of each frame. Specifically, the reference frame and the frames after the reference frame can be input into the convolutional neural network, and then the convolutional neural network can output the reference frame and the feature vectors corresponding to the other frames .

In this embodiment, in order to accurately characterize the content displayed in the reference frame and the current frame, the first feature vector and the second feature vector may also represent the reference frame and the current frame, respectively The scale invariant feature. In this way, even if the rotation angle, image brightness, or shooting angle of view of the image is changed, the extracted first feature vector and the second feature vector can still well reflect the content in the reference frame and the current frame. Specifically, the first feature vector and the second feature vector may be Sift (Scale-invariant feature transform) features, surf features (Speed Up Robust Feature, fast robust features), or color Histogram features, etc.

In this embodiment, after the first feature vector and the second feature vector are determined, the similarity between the first feature vector and the second feature vector can be calculated. Specifically, the similarity can be expressed as the distance between two vectors in the vector space. The closer the distance, the more similar the two vectors, so the higher the similarity. The farther the distance, the greater the difference between the two vectors, and therefore the lower the similarity. Therefore, when calculating the similarity between the reference frame and the current frame, the spatial distance between the first feature vector and the second feature vector can be calculated, and the reciprocal of the spatial distance can be used as The similarity between the reference frame and the current frame. In this way, the smaller the spatial distance, the greater the corresponding similarity, which indicates that the reference frame and the current frame are more similar. Conversely, the larger the spatial distance, the smaller the corresponding similarity, which indicates that the reference frame and the current frame are less similar.

In this embodiment, the similarity between each frame after the reference frame and the reference frame can be calculated sequentially in the above-mentioned manner. The content displayed in the two frames with higher similarity is usually relatively similar, and the main purpose of the video summary is to show the content of different scenes in the video to the user. Therefore, in this embodiment, when the reference frame When the similarity with the current frame is less than or equal to the specified threshold, the current frame may be determined as a scene switching frame. Wherein, the specified threshold may be a preset numerical value, which can be flexibly adjusted according to actual conditions. For example, when the number of scene switching frames filtered out according to the specified threshold is too large, the size of the specified threshold can be appropriately reduced. For another example, when the number of scene switching frames filtered out according to the specified threshold is too small, the size of the specified threshold may be appropriately increased. In this embodiment, the similarity is less than or equal to the specified threshold, which can indicate that the content in the two frames is already significantly different. Therefore, it can be considered that the scene shown in the current frame has changed from the scene shown in the reference frame. . At this time, the current frame can be retained as a frame of scene switching.

In this embodiment, when the current frame is determined as a scene switching frame, other subsequent scene switching frames can be determined continuously. Specifically, from the reference frame to the current frame, it can be considered that the scene has changed once, so the current scene is the content displayed in the current frame. Based on this, the current frame can be used as a new reference frame, and the similarity between each frame after the new reference frame and the new reference frame can be calculated in sequence, so as to be based on the calculated similarity Determine the next scene switching frame. Similarly, when determining the next scene switching frame, the similarity between the two frames can still be determined by extracting the feature vector and calculating the spatial distance, and the determined similarity can still be compared with the specified threshold. By comparison, the next scene switching frame in which the scene changes again after the new reference frame is determined.

Referring to FIG. 3, in this embodiment, after the next scene switching frame is determined, the scene switching frame can be used as a new reference frame to continue the extraction process of subsequent scene switching frames. In this way, by sequentially changing the reference frame, each frame in the video whose scene has changed can be extracted, so that the scene displayed in the video will not be omitted, so as to ensure the completeness of the video summary. In FIG. 3, a rectangular bar filled with diagonal lines can be used as a scene switching frame, and the similarity between two adjacent scene switching frames can be less than or equal to the specified threshold.

In this embodiment, among the scene switching frames extracted by the above method, the similarity between any two adjacent scene switching frames will be less than or equal to the specified threshold. Therefore, between two adjacent scene switching frames If the similarity of satisfies the specified condition, it may mean that the similarity between two adjacent scene switching frames is less than or equal to the specified threshold.

In this embodiment, after the multiple scene switching frames are extracted, a scene label may be set for the scene switching frame. The scene label may be a text label used to characterize the content displayed in the scene switching frame. For example, if two people are fighting in a scene switching frame, the scene label corresponding to the scene switching frame can be "martial arts", "fighting" or "kung fu".

In this embodiment, the content in the scene switching frame can be identified to determine the scene tag corresponding to the scene switching frame. Specifically, the feature of the scene switching frame can be extracted, where the feature can include at least one of a color feature, a texture feature, and a shape feature. Wherein, the color features may be features extracted based on different color spaces. The color space may include, for example, RGB (Red, Green, Blue, red, green, blue) space, HSV (Hue, Saturation, Value, hue, saturation, lightness) space, HIS (Hue, Saturation, Intensity, hue, Saturation, brightness) space, etc. In the color space, each can have multiple color components. For example, an R component, a G component, and a B component may be provided in the RGB space. For different pictures, the color components will also be different. Therefore, the color component can be used to characterize the characteristics of the scene switching frame.

In addition, the texture feature may be used to describe the material corresponding to the scene switching frame. The texture feature can usually be embodied by the distribution of gray levels. The texture feature may correspond to low-frequency components and high-frequency components in the image spectrum. In this way, the low-frequency component and the high-frequency component of the image contained in the scene switching frame can be used as features of the scene switching frame.

In this embodiment, the shape features may include edge-based shape features and region-based shape features. Specifically, a Fourier transform boundary can be used as the edge-based shape feature, and a moment invariant descriptor can also be used as the region-based shape feature.

Referring to FIG. 4, in this embodiment, after the features in each scene switching frame are extracted, the extracted features can be compared with each feature sample in the feature sample library. The feature sample library may be a sample set summarized based on historical data of image recognition. In the feature sample library, there may be feature samples that characterize different content. The characteristic sample may also be at least one of the aforementioned color characteristic, texture characteristic and shape characteristic. For example, in the feature sample library, there are feature samples that characterize playing football, there are feature samples that characterize dancing, and there are feature samples that characterize fighting. Specifically, the feature samples in the feature sample library can all be associated with text labels, and the text labels can be used to describe the display content corresponding to the feature samples. For example, the text label associated with the feature sample that characterizes playing football may be "playing football", and the text label of the feature sample that characterizes dancing may be "square dance".

In this embodiment, both the extracted features and the feature samples in the feature sample library can be represented in the form of vectors. In this way, comparing the extracted feature with each feature sample in the feature sample library may refer to calculating the distance between the feature and each feature sample. The closer the distance, the more similar the extracted feature and the feature sample. In this way, the target feature sample that is most similar to the extracted feature in the feature sample library can be determined. Wherein, the calculated distance between the most similar target feature sample and the extracted feature sample may be the smallest. The extracted feature is the most similar to the target feature sample, indicating that the content displayed by the two is also the most similar. Therefore, the text label associated with the target feature sample can be used as the scene label corresponding to the scene switching frame, which can be Set the corresponding scene label for each scene switching frame.

As shown in Figure 4, the distance between the feature extracted from the scene switching frame and each feature sample in the feature sample library can be 0.8, 0.5, 0.95, and 0.6 respectively, so that the text label corresponding to the feature sample with a distance of 0.5 It can be used as the scene label corresponding to the scene switching frame.

S3: Extract the topic tag corresponding to the video from the text description information.

In this embodiment, the text description information can more accurately indicate the subject of the video. Therefore, the topic tag corresponding to the video can be extracted from the text description information. Specifically, a video broadcasting website can summarize and summarize the text description information of a large number of videos, filter out various text tags that may be the subject of the video, and form a text tag library with the selected text tags. The content in the text label library can be continuously updated. In this way, when the topic tags are extracted from the text description information, the text description information can be matched with each text tag in the text tag library, and the matched text tags can be used as the topic tags of the video. For example, the text description information of the video is "Foreign guys and Chinese aunts dance square dance, shocking everyone!" Then when the text description information is matched with each text tag in the text tag library, you can get "square "Dance" this matching result. Therefore, "square dance" can be used as the theme tag of the video.

It should be noted that since the text description information of the video is usually relatively long, when matching with the text tags in the text tag library, at least two results may be obtained by matching. For example, the text description information of the video is "Foreign guys and Chinese aunts dance square dance, shocking everyone!", then when the text description information is matched with each text tag in the text tag library, you can get " The three matching results of "Foreign Guy", "Chinese Aunt" and "Square Dance". On the one hand, these three matching results can be used as the topic tags of the video at the same time. On the other hand, when the number of topic tags of the video is limited, suitable topic tags can be filtered from multiple matching results. Specifically, in this embodiment, each text tag in the text tag library may be associated with a counted number of times, where the counted number of times may be used to characterize the total number of times the text tag is used as a topic tag. The greater the number of statistics, the more the total number of times the corresponding text label is used as the topic label of the video, and the higher the credibility of the text label as the topic label. Therefore, when the number of matching text labels is at least two, the matching text labels can be sorted in the order of statistical times, and the top specified number of text labels in the sorting result can be used as all the text labels. Describe the topic tag of the video. Wherein, the specified number may be a predetermined number of topic tags of the video. For example, if the number of topic tags of the video is limited to a maximum of 2, then the three matching results of "foreign guy", "Chinese aunt" and "square dance" can be sorted according to the number of statistics, and finally the top 2 "Chinese Auntie" and "Square Dance" are the theme tags of this video.

S5: According to the association between the scene tag of the scene switching frame and the topic tag, a target frame is filtered from the multiple scene switching frames, and a video summary of the video is generated based on the target frame.

In this embodiment, considering that there will be more scenes in the video, not all scene switching frames corresponding to the scenes are closely related to the theme of the video. In order to enable the generated video summary to accurately reflect the theme of the video, the target frame may be filtered from the multiple scene switching frames according to the correlation between the scene tag of each scene switching frame and the topic tag.

In this embodiment, the association between the scene tag and the topic tag may refer to the degree of similarity between the scene tag and the topic tag. The more similar the scene tag and the topic tag are, the more relevant the content displayed in the scene switching frame is to the topic of the video. Specifically, the manner of determining the association between the scene tag and the topic tag may include calculating the similarity between the scene tag of each scene switching frame and the topic tag. In practical applications, both the scene label and the topic label can be composed of vocabulary. When calculating the similarity between the two, the scene label and the topic label can be respectively represented by a word vector. Describe the topic tags. In this way, the similarity between the scene label and the topic label can be expressed by the spatial distance between two word vectors. The closer the spatial distance between two word vectors, the higher the similarity between the scene label and the topic label; on the contrary, the farther the spatial distance between the two word vectors, the higher the similarity between the scene label The lower the similarity with the topic tag. In this way, in an actual application scenario, the reciprocal of the spatial distance between two word vectors can be used as the similarity between the scene label and the topic label.

In this embodiment, after calculating the similarity between the scene tag and the topic tag, a scene switching frame whose calculated similarity is greater than a specified similarity threshold may be determined as the target frame. Wherein, the specified similarity threshold can be used as a threshold to measure whether the scene switching frame and the subject are sufficiently related. When the similarity is greater than the specified similarity threshold, it can indicate that the current scene switching frame and the subject of the video have been With sufficient association, the content displayed in the scene switching frame can accurately reflect the theme of the video, so the scene switching frame can be determined as the target frame.

In this embodiment, the target frames selected from the scene switching frames are closely related to the main body of the video. Therefore, a video summary of the video can be generated based on the target frame. Specifically, the way of generating the video summary of the video may be that each target frame is arranged in sequence according to the sequence of the video, so as to form the video summary of the video. In addition, considering that the content shown in the video summary does not need to maintain the normal logic of the content between the previous and subsequent frames, each target frame can be randomly arranged, and the sequence of the target frames after the arrangement can be used as the video summary of the video .

In one embodiment of the present application, considering that the scene label of each scene switching frame is usually set for the overall content of the scene switching frame, the scene label cannot accurately reflect the local details in the scene switching frame. In order to further improve the relevance between the target frame and the video theme, in this embodiment, the target objects included in the scene switching frame can be identified, and the target frame can be filtered on the basis of the identified target objects. Specifically, after calculating the similarity between the scene label of each scene switching frame and the topic label, a weight coefficient may be set for the corresponding scene switching frame according to the calculated similarity. Among them, the higher the similarity between the scene label and the topic label, the larger the weight coefficient set for the corresponding scene switching frame. The weight coefficient may be a value between 0 and 1. For example, if the theme tag of the current video is "Square Dance", for the two scene switching frames with the scene tags "Dance" and "Kung Fu", the weight coefficient set for the scene switching frame with the scene tag "Dance" can be 0.8, and the weight coefficient set for the scene switching frame with the scene label "Kung Fu" can be 0.4.

In this embodiment, after setting the weight coefficient for each scene switching frame, the target object contained in the scene switching frame can be identified. Specifically, when recognizing the target object contained in the scene switching frame, adaboost algorithm, R-CNN (Region-based Convolutional Neural Network, region-based convolutional neural network) algorithm, or SSD (Single Shot Detector, single target Detection) algorithm to detect the target object contained in the scene switching frame. For example, for a scene switching frame with a scene label of "dance", it can be recognized through the R-CNN algorithm that the scene switching frame includes two target objects: "woman" and "sound". In this way, after identifying the target object contained in each scene switching frame, an association value can be set for the scene switching frame according to the identified association between the target object and the topic tag. Specifically, the topic tag may be associated with at least one object. The object may be an object closely related to the topic tag. At least one object associated with the topic tag may be obtained by analyzing historical data. For example, when the topic tag is "beach", at least one object associated with it may include "sea water", "beach", "seagull", "swimsuit", "sunshade" and so on. In this way, the target object recognized from the scene switching frame can be compared with the at least one object, and the number of target objects appearing in the at least one object can be counted. Specifically, for the theme label of "beach", assuming that the target objects identified from the scene switching frame are "parasols", "cars", "beach", "trees" and "sea water", then the target objects are When the at least one object is compared, it can be determined that the target objects appearing in the at least one object are "parasol," "beach," and "sea water." That is, the number of target objects appearing in the at least one object is three. In this embodiment, the product of the counted number and a specified value may be used as the associated value of the scene switching frame. The specified value may be a preset value, for example, the specified value may be 10, then the associated value of the scene switching frame in the above example may be 30. In this way, the greater the number of target objects appearing in the at least one object, the closer the correlation between the local details in the scene switching frame and the video theme, and the higher the corresponding correlation value.

In this embodiment, when determining the target frame, the judgment can be made based on the overall feature and the local feature of the scene switching frame. Specifically, the product of the weight coefficient and the associated value of each of the scene switching frames may be calculated, and the scene switching frame whose product is greater than a specified product threshold may be determined as the target frame. The product is used as the basis for judgment, which can synthesize the overall and local features of the scene switching frame. The specified product threshold may be a threshold for measuring whether the scene switching frame is the target frame. The specified product threshold can be flexibly adjusted in actual application scenarios.

In an embodiment of the present application, considering that in some scenarios, the total number of picture frames (or total duration) in the video summary may be limited in advance. In this case, when determining the target frame, it is also necessary to comprehensively consider the pre-restricted total number of frames. Specifically, when the total number of each of the scene switching frames is greater than or equal to the specified total number of frames, it indicates that a sufficient number of frames can be extracted from the scene switching frames to form a video summary. In this case, based on the product of the weight coefficient corresponding to each scene switching frame and the associated value calculated in the foregoing embodiment, the scene switching frames may be sorted in descending order of the product. Then, the specified total number of scene switching frames at the top of the sorting result may be determined as the target frame. For example, the total number of frames in the video summary is currently limited to 1440 frames, while the current number of scene switching frames extracted from the video is 2000 frames. In this way, the product of the weight coefficient and the associated value corresponding to each scene switching frame can be calculated in sequence, and after the product is sorted from largest to smallest, the top 1440 scene switching frames are used as the target frame, which can be determined by 1440 The frame target frame constitutes a video digest that meets the requirements.

In this embodiment, when the total number of each of the scene switching frames is less than the specified total number of frames, it indicates that all the currently extracted scene switching frames are not enough to form a video digest that meets the requirements. In this case, it is necessary to insert a certain number of picture frames in the original video between the extracted scene switching frames, so as to meet the requirement of the total number of frames limited by the video summary. Specifically, when inserting a frame in the original video, it can be performed between two scene switching frames with a larger scene jump, so that the continuity of the content can be maintained. In this embodiment, at least one video frame of the video can be inserted between two adjacent scene switching frames whose similarity is less than the determination threshold. Among them, two adjacent scene switching frames whose similarity is less than the determination threshold may be regarded as two scene switching frames with weaker content relevance. In this embodiment, the picture frames in the original video can be inserted frame by frame between two scene switching frames with weaker correlation, until the total number of scene switching frames after inserting the at least one video frame is equal to the specified The total number of frames. In this way, the whole of the original scene switching frame and the inserted picture frame can be used as the target frame, thereby constituting the video summary of the video.

In one embodiment of the present application, the number of topic tags extracted from the text description information of the video may be at least two. In this case, the scene tags of the scene switching frame may be calculated for the scene switching frame Similarity with each of the topic tags. For example, if the current topic tags are tag 1 and tag 2, then the similarity between the current scene switching frame and tag 1 and tag 2 can be calculated respectively, so that the first similarity and the first similarity corresponding to the current scene switching frame can be obtained. Two similarity. After the respective similarities corresponding to the scene switching frames are calculated, the respective similarities calculated for the scene switching frames may be accumulated to obtain the accumulated similarities corresponding to the scene switching frames. For example, the sum of the aforementioned first similarity and the second similarity may be used as the accumulated similarity corresponding to the current scene switching frame. In this embodiment, after the accumulated similarity corresponding to each scene switching frame is calculated, the accumulated similarity can also be compared with the specified similarity threshold, and the scene switching frame with the accumulated similarity greater than the specified similarity threshold can be determined Is the target frame.

Referring to FIG. 5, the present application also provides a video summary generating device, the video having text description information, and the device includes: a scene switching frame extraction unit 100 for extracting multiple scene switching frames from the video; And set a scene tag for the scene switching frame, where the similarity between two adjacent scene switching frames meets a specified condition; the topic tag extraction unit 200 is configured to extract the video corresponding to the text description information Topic tag; video summary generating unit 300, used to filter out the target frame from the multiple scene switching frames according to the correlation between the scene tag of the scene switching frame and the topic tag, and based on the target The frame generates a video summary of the video.

In this embodiment, the scene switching frame extraction unit 100 includes: a similarity calculation module for determining a reference frame in the video, and sequentially calculating the difference between the frame after the reference frame and the reference frame Similarity; a scene switching frame determination module, used to determine the current frame as a scene switching frame when the similarity between the reference frame and the current frame is less than or equal to a specified threshold; a loop execution module, used to The current frame is used as a new reference frame, and the similarity between the frames after the new reference frame and the new reference frame is sequentially calculated to determine the next scene switching frame according to the calculated similarity .

In this embodiment, the scene switching frame extraction unit 100 includes: a feature extraction module for extracting features of the scene switching frame, the features including at least one of a color feature, a texture feature, and a shape feature; A module for comparing the extracted features with feature samples in a feature sample library, wherein the feature samples in the feature sample library are all associated with text labels; the target feature sample determination module is used for Determine the target feature sample most similar to the extracted feature in the feature sample library, and use the text label associated with the target feature sample as the scene label corresponding to the scene switching frame.

In this embodiment, the video summary generating unit 300 includes: a similarity calculation module for calculating the similarity between the scene tag of the scene switching frame and the topic tag; and a weight coefficient setting module for calculating The calculated similarity is used to set the weight coefficient for the corresponding scene switching frame; the associated value setting module is used to identify the target object contained in the scene switching frame, and according to the identified target object and the theme The correlation between the tags sets the correlation value for the scene switching frame; the target frame determination module is used to calculate the product of the weight coefficient of the scene switching frame and the correlation value, and the scene where the product is greater than the specified product threshold The switching frame is determined as the target frame.

This application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments. In these distributed computing environments, remote processing devices connected through a communication network perform tasks. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

Those skilled in the art also know that in addition to implementing the device in a purely computer-readable program code manner, it is completely possible to program the method steps to make the device use logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers. To achieve the same function in the form of a device, etc. Therefore, such a device can be regarded as a hardware component, and the devices included in it for realizing various functions can also be regarded as a structure within the hardware component. Or even, the device for realizing various functions can be regarded as both a software module for realizing the method and a structure in a hardware component.

It can be seen from the above that this application can first extract scene switching frames whose similarity meets the specified conditions from the video, and set corresponding scene tags for the scene switching frames. Then, the subject tag of the video can be determined by combining the text description information of the video. The subject tag can accurately represent the subject of the video. Then, by determining the relevance between the scene tag and the topic tag, the target frame that is more closely related to the topic can be retained from the scene switching frame. In this way, the video summary generated based on the target frame can accurately characterize the subject matter of the video.

In the 1990s, the improvement of a technology can be clearly distinguished between hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (for method flow Improve). However, with the development of technology, the improvement of many methods and processes of today can be regarded as a direct improvement of the hardware circuit structure. Designers almost always get the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by hardware entity modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user's programming of the device. It is programmed by the designer to "integrate" a digital system on a PLD without requiring the chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly realized by using "logic compiler" software, which is similar to the software compiler used in program development and writing, but before compilation The original code must also be written in a specific programming language, which is called Hardware Description Language (HDL), and HDL is not only one, but there are many, such as ABEL (Advanced Boolean Expression Language) ), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., currently the most common VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are used. It should also be clear to those skilled in the art that only a little logic programming of the method flow using the above hardware description languages and programming into an integrated circuit can easily obtain the hardware circuit that implements the logic method flow.

From the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM/RAM, magnetic disk, An optical disc, etc., includes a number of instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments of this application.

The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the implementation of the device, all can be explained with reference to the introduction of the foregoing method implementation.

Although the application has been described through the embodiments, those of ordinary skill in the art know that there are many variations and changes in this application without departing from the spirit of the application, and it is hoped that the scope of the appended patent application includes these variations and changes without departing from the spirit of the application .

S1, S3, S5‧‧‧Step

Claims (14)

A method for generating a video summary, the video having text description information, the method comprising: extracting a plurality of scene switching frames from the video, and setting a scene label for the scene switching frame, wherein two adjacent scenes The similarity between the switching frames satisfies a specified condition; extracting the topic tag corresponding to the video from the text description information; according to the correlation between the scene tag of the scene switching frame and the topic tag, from the Selecting the target frame from multiple scene switching frames includes: calculating the similarity between the scene tag of the scene switching frame and the topic tag; and setting a weight for the corresponding scene switching frame according to the calculated similarity Coefficient; identifying the target object contained in the scene switching frame, and setting an associated value for the scene switching frame according to the identified association between the target object and the topic tag; calculating the scene switching frame The product of the weight coefficient of and the associated value, and the scene switching frame whose product is greater than a specified product threshold is determined as the target frame; and a video summary of the video is generated based on the target frame. The method according to claim 1, wherein extracting a plurality of scene switching frames from the video includes: determining a reference frame in the video, and sequentially calculating the distance between the frame after the reference frame and the reference frame When the similarity between the reference frame and the current frame is less than or equal to the specified threshold, the current frame is determined as a scene switching frame; the current frame is taken as the new reference frame and calculated in sequence The new benchmark The similarity between the frame after the frame and the new reference frame is used to determine the next scene switching frame according to the calculated similarity. The method according to claim 2, wherein the similarity between two adjacent scene switching frames satisfying a specified condition includes: the similarity between two adjacent scene switching frames is less than or equal to the specified threshold. The method according to claim 2, wherein calculating the similarity between the frame after the reference frame and the reference frame comprises: extracting a first feature vector and a second feature vector of the reference frame and the current frame, respectively , Wherein the first feature vector and the second feature vector respectively represent the scale-invariant features of the reference frame and the current frame; calculating the difference between the first feature vector and the second feature vector The spatial distance, and the reciprocal of the spatial distance is used as the similarity between the reference frame and the current frame. The method according to claim 1, wherein setting a scene label for the scene switching frame includes: extracting features of the scene switching frame, the features including at least one of a color feature, a texture feature, and a shape feature; The features in the feature sample library are compared with feature samples in the feature sample library, wherein the feature samples in the feature sample library are associated with text labels; it is determined that the feature sample library is most similar to the extracted feature And use the text label associated with the target feature sample as the scene label corresponding to the scene switching frame. The method according to claim 1, wherein the text description information includes the title and/or brief introduction of the video; accordingly, extracting the topic tag corresponding to the video from the text description information includes: The text description information is matched with the text tags in the text tag library, and the matched text tags are used as the topic tags of the video. The method according to claim 6, wherein the text tags in the text tag library are associated with statistical times, and the statistical times are used to characterize the total number of times the text labels are used as topic tags; accordingly, when matching is obtained When the number of text labels in is at least two, the method further includes: sorting the matched text labels in the order of statistical times, and taking the top specified number of text labels in the sorting result as all text labels. Describe the topic tag of the video. The method according to claim 1, wherein the topic tag is associated with at least one object; correspondingly, setting an associated value for the scene switching frame includes: the target object identified from the scene switching frame is associated with The at least one object is compared, and the number of target objects appearing in the at least one object is counted; the product of the counted number and a specified value is used as the associated value of the scene switching frame. The method according to claim 1, wherein the video summary of the video has a specified total number of frames; accordingly, after calculating the product of the weight coefficient of the scene switching frame and the associated value, the method further includes: When the total number of scene switching frames is greater than or equal to the specified total number of frames In the case of the quantity, the scene switching frames are sorted in the descending order of the product, and the specified total number of scene switching frames are determined as the target frame. The method according to claim 9, wherein the method further comprises: when the total number of scene switching frames is less than the specified total number of frames, switching between two adjacent scenes whose similarity is less than a determination threshold Between frames, at least one video frame in the video is inserted, so that the total number of scene switching frames after the insertion of the at least one video frame is equal to the specified total number of frames. The method according to claim 1, wherein when the number of the subject tags is at least two, filtering out the target frame from the multiple scene switching frames includes: calculating the scene for the scene switching frame The similarity between the scene label of the switching frame and the topic label; the similarity calculated for the scene switching frame is accumulated to obtain the cumulative similarity corresponding to the scene switching frame; the cumulative similarity is greater than The scene switching frame with the specified similarity threshold is determined as the target frame. A device for generating a video summary, the video is provided with text description information, the device includes: a scene switching frame extraction unit for extracting a plurality of scene switching frames from the video, and setting a scene label for the scene switching frame , Wherein the similarity between two adjacent scene switching frames meets a specified condition; the topic tag extraction unit is configured to extract the topic tag corresponding to the video from the text description information; The video summary generating unit is configured to filter out target frames from the multiple scene switching frames according to the correlation between the scene tags of the scene switching frames and the topic tags, and generate the target frames based on the target frames The video summary of the video; wherein, the video summary generating unit includes: a similarity calculation module for calculating the similarity between the scene tag of the scene switching frame and the topic tag; and a weight coefficient setting module for The calculated similarity is used to set the weight coefficient for the corresponding scene switching frame; the associated value setting module is used to identify the target object contained in the scene switching frame, and according to the identified target object and the theme The correlation between the tags sets the correlation value for the scene switching frame; the target frame determination module is used to calculate the product of the weight coefficient of the scene switching frame and the correlation value, and the scene where the product is greater than the specified product threshold The switching frame is determined as the target frame. The device according to claim 12, wherein the scene switching frame extraction unit includes: a similarity calculation module, configured to determine a reference frame in the video, and sequentially calculate the frame after the reference frame and the reference frame Similarity between frames; a scene switching frame determination module, used to determine the current frame as a scene switching frame when the similarity between the reference frame and the current frame is less than or equal to a specified threshold; loop execution module , Used to use the current frame as a new reference frame, and sequentially calculate the similarity between the frames after the new reference frame and the new reference frame, so as to determine the next reference frame according to the calculated similarity A scene switch frame. The apparatus according to claim 12, wherein the scene switching frame extraction unit includes: a feature extraction module configured to extract features of the scene switching frame, the features including at least one of a color feature, a texture feature, and a shape feature One; a comparison module for comparing the extracted features with feature samples in a feature sample library, wherein the feature samples in the feature sample library are all associated with text labels; target feature samples are determined The module is used to determine the target feature sample most similar to the extracted feature in the feature sample library, and use the text label associated with the target feature sample as the scene label corresponding to the scene switching frame.

Images