JP5600040B2 - Video summarization apparatus, video summarization method, and video summarization program - Google Patents

Description

  The present invention relates to a video summarization apparatus, a video summarization method, and a video summarization for generating one short summary video by evaluating the semantic comprehensiveness and visual diversity of a video from a video group including one or more videos. Regarding the program.

  Distribution and sharing of video using communication has become active, and the number of videos distributed online has become enormous. A video sharing site is currently reporting that 24 hours of video per minute are newly released. On video sharing sites, it is said that the average video per video is about 2 minutes and 30 seconds. Assuming that 24 hours of video per minute are released, this is a calculation in which more than 500 videos / minute continue to be released.

Viewers need to find a video of interest that they want to watch from a large number of videos. The process of searching for a video is roughly as follows.
(1) Search for video by some method. A keyword search is usually performed.
(2) Play / listen to the search results that are likely to match your interests.
(3) Continue watching if it suits your interest, otherwise return to (1) or (2).

  This process is very time consuming, and particularly takes a long time for (2) and (3). Since the video is a media with a time axis, the contents cannot be understood unless it is actually played back and watched and auditioned over time. For this reason, the time required to judge whether it was really interesting or what you wanted to watch would increase.

  In order to reduce the time for confirming / listening to the content of the video, a technology related to video summarization has been invented. For example, the technique described in Patent Document 1 discloses a technique for determining a voice emphasis state in a video and including a video section including a more emphasized speech in the summary. With this technology, it is possible to efficiently audition the lively video section in the video, so that efficient audition is possible.

  Further, Patent Document 2 detects characteristic changes seen in images and sounds in video on the assumption that they are important events, and speeds up (skips) a section in which only a small number of events are detected. A video summarization technique for skipping insignificant sections by playing is disclosed.

JP 2003-259111 A JP 2010-39877 A

  The conventional video summarization technology described above was aimed at generating one summary video from one video. That is, if 10 videos are summarized, 10 summary videos are generated. These can be efficient and effective solutions because the content can be confirmed in a short time when there are only a few videos to be summarized. However, when there are a very large number of target videos, such a technique alone cannot provide a practically effective summary video.

  As described above, 500 videos per minute are newly released on the current video sharing site. Suppose you want to configure a system that summarizes all these new videos and allows users to check new videos in a summary. In the above-described technique, one summary video is generated for each video, so the number of summary videos to be generated is 500 lines / minute.

  If a summary video of 10 seconds per video is generated, the total time length of the generated summary video is 500 lines / minute × 10 seconds = 5000 lines / second / minute, that is, the total time is about 1 A summary video for 20 minutes will be generated. A summary video of a time that far exceeds one minute per minute is generated. It is clear that the technology that generates the summary video one by one does not give an essential solution in principle.

  In order to solve this problem, an object of the present invention is to provide a video summarization technique for generating a single summary video from a plurality of videos instead of generating a single summary video from a single video. It is.

  The focus of the present invention will be described. It is said that a large amount of video contains many videos that are the same or very similar in meaning and appearance even if they are different videos. For example, in a certain video sharing site, when a search is performed using a specific keyword, an average of about 14% of videos listed in the search result, and more than 70% are complete, or on average, , Is said to be a video with partially overlapping sections.

  One of the basic points of view of the present invention is to summarize a plurality of videos together by utilizing the fact that “a plurality of videos have similar semantic contents / visual video sections”. That is. That is, the summary video is generated so that similar sections of different video files are not overlapped as much as possible.

  Give an example. Suppose there are two images. For simplicity, each time length is 60 seconds. Suppose that these two video files include a similar video segment for 10 seconds, and that this video segment is an important video segment that should be included in the summary video. In the conventional “method of summarizing one by one”, if 10 seconds of summary video is generated per line, 2 × 10 seconds, a total of 20 seconds of summary video is generated. The 20-second summary video becomes a summary video composed of 10 seconds from two videos and 20 seconds of similar video sections.

  On the other hand, in the “method for summarizing a plurality of videos together” according to the present invention, since a summary video is generated so as not to include similar sections of two videos as much as possible, only one of the similar sections is included. The summary video for 10 seconds is output as a summary video combining two videos.

Based on the above points, in order to solve the above problems, the first invention provides a video group composed of a plurality of videos and metadata attached to each video. A video summarization device for generating a summary video, a video segment dividing unit that divides each video included in the video group into video segments, a video segment feature amount extracting unit that extracts a feature amount for each video segment, Storage means for storing metadata incidental to the video, division information of the video section, and feature values for each video section, accepts a summary generation request, refers to the storage means, and responds to the received request Video segment group selection means for selecting one or more video segments, and metadata of the video to which the video segments included in the selected video segment group belong or the extracted feature quantity, or information of both Using one or more of the video segments, and an evaluation function optimum for obtaining an ordered video segment set that maximizes an evaluation value calculated by a predetermined evaluation function for the ordered video segment set in which the time order is determined and means, coupled to video section set with the determined order and a video summary generation means for generating a video summary, the evaluation function is a term for obtaining a viewing ease of ordered video section , Including a term for evaluating similarity of metadata assigned to adjacent video sections, similarity of feature quantities of adjacent video sections, or similarity of both of the ordered set of video sections .

  According to the first aspect of the present invention, it is possible to automatically generate one summary video in which video groups are grouped for video sections included in a plurality of videos.

  According to a second invention, in the first invention, the feature amount is a brightness feature, a color feature, a motion feature, a texture feature, a concept feature, a landscape from an image frame at a predetermined position included in the video section. It is a vector constituted by extracting at least one of the features.

  According to a third invention, in the first or second invention, the feature value is a pitch feature, a sound pressure feature, a spectrum feature, a rhythm feature, from a sound signal at a predetermined position included in the video section. It is a vector constituted by extracting at least one of speech features, music features, and sound event features.

  According to the second and third inventions, the visual similarity can be accurately measured based on the image and sound characteristics of the video section, so that a more effective summary video can be generated.

  According to a fourth invention, in the first, second, or third invention, the video section group selecting means accepts a summary generation request by a keyword from a user, and the video and the video stored in the storage means. The relevance between the keyword and the video is determined by determining the matching with the metadata, and the video section divided from the related video is selected.

  According to the fourth invention, when a user wants to obtain a summary video, it is possible to generate a summary video conforming to the request only in a simple format called a keyword.

The fifth invention is the invention according to any one of the first to fourth inventions, wherein the evaluation function is further included in a term for obtaining conformity to the requirement of the ordered video segment set, the ordered video segment set. term for obtaining a coverage of the semantic content of the video section is, term for obtaining the coverage characteristics of a video section included in the ordered video section set, among the term for obtaining the time length of the order in with video section, at least one In addition, it is characterized by being configured.

  According to the fifth aspect, since the evaluation function appropriately evaluates the characteristics to be satisfied as the summary video such as the semantic completeness of the video and the variety of appearance, the summary video that is easy to use and highly effective can be obtained. Can be generated.

  The video summarization method executed by the video summarization apparatus is also a feature of the present invention.

  The video summarization method according to the present invention can also be realized by a computer program. The computer program may be provided by being recorded on an appropriate computer-readable recording medium or provided via a network. In many cases, the present invention is realized by being installed and operating on a control means such as a CPU when the present invention is implemented.

  As described above, according to the present invention, a video summarization apparatus, a video summarization method, and a video summarization program that combine a plurality of videos into one summary video that retains semantic completeness and appearance diversity are provided. can do.

It is a figure which shows the structural example of the image | video summary apparatus which concerns on this invention. It is a flowchart of an image | video summary process. It is a figure which shows an example of the video area information stored in the memory | storage part. It is a figure which shows an example of an evaluation function optimization process.

  Hereinafter, the present invention will be described in detail according to embodiments.

  FIG. 1 illustrates an example of a device configuration of a video summarization device according to the present invention. As shown in this figure, the video summarizing apparatus 1 includes a video input unit 100, a storage unit 101, a video segment dividing unit 102, a video segment feature value extracting unit 103, a video segment group selecting unit 104, an evaluation function. An optimization unit 105 and a summary video generation unit 106 are provided.

  As an example of this embodiment, video input, video segmentation, and video segment feature value extraction are performed in advance as pre-processing, and later only summary processing for generating a summary video in response to a request from the user. An example of performing is described.

  The video input unit 100 inputs a video group 2 including a plurality of videos and stores it in the storage unit 101. At this time, metadata such as a video title is also stored.

  The video segment dividing unit 102 divides each video stored in the storage unit 101 into a plurality of video segments. In an example of this embodiment, time information (start point, end point) of the divided video section is stored in the storage unit 101.

  The video segment feature value extraction unit 103 analyzes the image information and sound information of the video segment, and extracts and outputs the feature value of each video segment. In an example of the present embodiment, the extracted feature amount for each video section is output to the storage unit 101 and stored.

  Through the processing so far, the storage unit 101 stores the metadata of the video, the time information (start point and end point) of the divided video sections, and the characteristics of each video section with respect to the video input in advance. The quantity will be associated and stored. If no misunderstanding occurs, this stored information may be referred to as video section information.

  The video segment group selection unit 104 selects a video segment of the video that is the subject of the summary generation process, and reads accompanying video segment information from the storage unit 101. The evaluation function optimizing unit 105 selects and outputs a video segment group that maximizes a pre-designed evaluation function for the read video segment information group. Finally, the summary video generation unit 106 combines the video segment groups output by the evaluation function optimization unit 105 to generate and output the summary video 3.

  In this way, the video summarization apparatus of the present invention summarizes and outputs a plurality of videos into one summary video 3 without human intervention.

  FIG. 2 shows a flowchart of video summarization processing executed by the video summarization apparatus 1 configured as described above. The video summarization process executed in an example of the present embodiment will be described in detail using this flowchart.

  The video summarization processing is roughly divided into preprocessing shown in FIG. 2A and summarization processing shown in FIG. In the preprocessing, video segment division processing is performed on the input video group 2 to extract necessary video segment information and register it in the storage unit 101. In the summary process, in response to a request from the user, the section information that is the subject of the summary process is read from the storage unit 101 and the summary generation process is performed. The details of preprocessing and summary processing are described below.

〔Preprocessing〕
As shown in the flowchart of FIG. 2 (A), in the pre-processing, three steps of steps S201 to S203 are performed.

  First, in step S201, the video input unit 100 inputs the video group 2 to be processed. These video groups 2 may all be input at the same time, or may be input at different timings. Each input video is stored in the storage unit 101 together with video metadata. Metadata is data describing information related to a video, and includes, for example, a video title, a summary sentence, a time length, a format, a keyword expressing video content, and the like. In an example of the present embodiment, a case where at least one of a keyword, a title, and a summary sentence expressing video content is included will be described. When the storage unit 101 is a relational database system, it is preferable because such metadata can be stored in association with video.

Subsequently, in step S202, video segment division processing is performed by the video segment division unit 102 on each of the videos stored in the storage unit 101. This video segmentation may be performed at predetermined intervals, for example, by the cut points that are points at which the video is discontinuously cut, such as the method described in Reference 1 below. It is good to do.
[Reference 1]: Y. Tonomura, A. Akutsu, Y. Taniguchi, and G. Suzuki, “Structured Video Computing”, IEEE Multimedia, pp. 34-43, 1994.

  Preferably, the latter method is adopted. As a result of the video segmentation process, the start point (start time) and end point (end time) of the segment are obtained. This time information is also stored in the storage unit 101 as video segment information.

  Next, in step S203, the video segment feature value extraction unit 103 extracts video segment feature values from the image / sound information in the video segment. Video segment feature values are extracted from images and extracted from sounds. Both are extracted from a minute section (frame) such as 50 ms.

  Features extracted from an image include, for example, brightness features, color features, motion features, texture features, concept features, and landscape features.

  The brightness feature can be extracted as a histogram by counting the V values in the HSV color space.

  The color feature can be extracted as a histogram by counting the values of the respective axes (L *, a *, b *) in the L * a * b * color space. The number of histogram bins on each axis may be, for example, 4 for L *, 14 for a *, 14 for b *, etc. In this case, the total number of bins for 3 axes is 4 X14x14 = 784.

  Motion features can be extracted as a histogram, for example, by calculating a motion vector between two image frames at a fixed interval and counting the magnitude and angle values of the vector. The number of bins in the histogram of size and angle may be, for example, 25 for the size and 8 for the angle. In this case, the total number of bins for the two axes is 25 × 4 = 100.

As a texture feature, a statistic (contrast) of a density histogram, a power spectrum, or the like may be obtained. Alternatively, it is preferable to use a local feature amount because it can be extracted in the form of a histogram as in the case of color and movement. As the local feature, for example, SIFT (Scale Invariant Feature Transform) described in the following Reference 2 or SURF (Speeded Up Robust Features) described in the following Reference 3 can be used.
[Reference 2]: DG Lowe, “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, pp. 91-110, 2004.
[Reference 3]: H. Bay, T. Tuytelaars, and LV Gool, “SURF: Speeded Up Robust Features”, Lecture Notes in Computer Science, vol. 3951, pp. 404-417, 2006.

  The local features extracted by these are, for example, 128-dimensional real value vectors. This vector is converted into a code with reference to a code length that has been learned and generated in advance, and a histogram can be generated by counting the number of codes. In this case, the number of bins in the histogram matches the code number of the code length. Any number of codes may be used. For example, 512 or 1024 may be used.

  Next, the concept features will be described. A concept is an object included in an image or an event captured by the image. Anything may be used, but examples include "sea", "mountain", "ball", and the like. If “sea” appears in an image, the image belongs to the “sea” concept. Whether or not the image belongs to each concept can be determined using a concept classifier. Usually, one concept discriminator is prepared for each concept, and the feature amount of the image is input, and whether or not the image belongs to a certain concept is output as an attribution level. The concept classifier is learned and acquired in advance, and it is a correct answer that represents a specific image feature, for example, the local feature described above and the concept that the image belongs to in advance by hand. Earn by learning the relationship with the label. For example, a support vector machine may be used as the learning device. Concept features can be obtained by expressing the attribution levels for each concept together as a vector.

A landscape feature is a feature amount representing a landscape or a scene of an image. For example, the GIST descriptor described in Reference 4 can be used.
[Reference 4]: A. Oliva and A. Torralba, “Building the gist of a scene: the role of global image features in recognition”, Progress in Brain Research, 155, pp. 23-36, 2006.

  On the other hand, features extracted from sound information include pitch features, sound pressure features, spectrum features, rhythm features, speech features, music features, sound event features, and the like.

The pitch feature may be a pitch, for example, and can be extracted using the method described in Reference 5 below.
[Reference 5]: Sadaaki Furui, “Digital Audio Processing, 4.9 Pitch Extraction”, pp. 57-59, 1985.

  As the sound pressure feature, an amplitude value of speech waveform data may be used, or a short-time power spectrum may be obtained, and an average power in an arbitrary band may be calculated and used.

  As the spectrum feature, for example, Mel-Frequency Cepstral Coefficients (MFCC) can be used.

For example, the tempo may be extracted as the rhythm feature. To extract the tempo, for example, the method described in Reference Document 6 below can be used.
[Reference 6]: ED Scheirer, “Tempo and Beat Analysis of Acoustic Musical Signals”, Journal of Acoustic Society America, Vol. 103, Issue 1, pp. 588-601, 1998.

The utterance feature and the music feature represent the presence or absence of utterance and the presence or absence of music, respectively. In order to find a section where speech / music exists, for example, a method described in Reference Document 7 below may be used.
[Reference 7]: K. Minami, A. Akutsu, H. Hamada, and Y. Tonomura, “Video Handling with Music and Speech Detection”, IEEE Multimedia, vol.5, no.3, pp.17-25, 1998.

As sound event information, for example, emotional sounds such as laughter and loud voice, or occurrence of environmental sounds such as gunshots and explosion sounds may be used. In order to detect such a sound event, for example, a method described in Reference Document 8 (Patent Document) below may be used.
[Reference 8]: International Publication WO / 2008/032787
Any feature amount other than those listed here may be used. The extracted feature values are stored together in the storage unit 101 as described above.

  An example of the storage format of the video section information stored in the storage unit 101 by the above preprocessing is shown in FIG. In this example, a video id that can be uniquely identified is assigned to each video, and a title and a keyword are stored as corresponding metadata. In addition, a uniquely identifiable video section id is assigned to the video section obtained by the video section division processing in step S202, and the corresponding start time and end time are described. Further, for each video section, the path to the file storing the feature quantity extracted by the video section feature quantity extraction process of step S203 is described in the feature quantity file column.

[Summary processing]
Next, summary processing is described. The summarization process is a process for generating the summary video 3 based on the metadata and the feature amount of the video obtained by the preprocessing when a summary request is generated from the user.

  The summarization process is a process of steps S204 to S206 shown in FIG. Each process will be described.

  First, in step S204, the video segment group selection unit 104 selects a video segment group that is a candidate video segment included in the summary in accordance with a request from the user. There are several methods for this, but in the example in the present embodiment, an example will be described in which a user designates a keyword and requests a summary. In many cases, a user makes an inquiry using a keyword query to a search engine in order to search for a video to be viewed. Similarly to this, this example can be used when a query by a keyword query is received and a summary for a related video group is generated and provided.

  In this case, the video segment group selection is performed by matching with metadata included in the video segment information. When a keyword is designated by the user, the metadata of the video section information is referred to, and only the video section of the video having metadata matching the keyword is selected as a video section group. As metadata to be used, anything stored in the storage unit 101 may be used, but a title, a summary sentence, a keyword expressing video content, or the like may be used as in a normal keyword search.

  Subsequently, in step S205, the evaluation function optimizing unit 105 selects a video section to be used for the summary video from the video section group selected in step S204 based on optimization of the evaluation function, and converts it into one summary video. When grouping together, the order in which the video segment groups are connected is determined.

  Since the summary video is a combination of a plurality of video sections in the time direction, it can be regarded as an ordered set of video sections. The evaluation function is a function that takes an ordered set of video segments as input and calculates the quality of the ordered video segment set as a summary video.

Describe the formal definition of the evaluation function. In step S204, the set of the selected video _{section, S = {s 1, s} 2, ..., s n} represents the. At this time, an ordered set in which an order is further added to the subset of S is represented as S ′ = {s ₁ → s ₂ →... → s _m } (m <n). At this time, the evaluation function f is a mapping f: Ξ → R from the space Ｓ of the ordered set S ′ to the scalar space R.

  If the evaluation function f expresses the goodness of the ordered set S ′, it is optimal as a summary by selecting the ordered set S ′ that maximizes the value of the function of f. Video segments and their order can be determined. Hereinafter, for the sake of simplicity, the value of the evaluation function f is expressed as goodness of the summary video.

  What is important here is that the evaluation function f is designed to express the goodness of the ordered set S ′. Hereinafter, the design of the evaluation function f in an example of the present embodiment will be described. The evaluation function f evaluates the goodness of the summary video in consideration of the following five points.

  (1) Conformity to user request: In an example of this embodiment, a request from a user is given by a keyword query. It can be considered that the summary video including many video sections of the video that match this query is closer to the summary video desired by the user and is a better summary video.

  (2) Semantic comprehensiveness: Summarization requires a comprehensive and short summary of various contents. In (1), the user's requirements are evaluated for suitability. However, when considering from the viewpoint of summarization, if only video segments that deal with similar contents appear, the summarization is good. That's not true. Therefore, the more diverse the semantic content of the video that appears in the summary, the better the summary.

  (3) Appearance completeness: Similar to (2), if video segments with the same “look” (image or sound) or overlapping video segments appear, this is a good summary from the viewpoint of completeness. It can not be said. Therefore, it can be considered that a summary video that includes various video segments with as little “look” as possible is a better summary.

  (4) Ease of viewing: Since this technology generates one summary video from multiple videos, the generated summary video includes videos with different contents. From the viewpoint of (2), there is a request to generate a summary video that has as much variety of semantic content as possible. However, when videos with dramatically different content appear one after another, From the viewpoint of the user who views this, it cannot be said that the video is easy to understand. Similarly, when video sections with dramatically different “looks” are played back continuously, for example, suddenly switching from a dark video section to a bright video section will generate a strong visual stimulus. The video is not easy to watch. Therefore, for users who play video and watch the video sections in sequence, those that are close to the semantic content should be placed as close as possible, and the “look” will be as smooth as possible. The generated summary video can be considered a good summary.

  (5) Time length: As a basic viewpoint of the summary video, a shorter one is better. Therefore, the shorter the summary time length of the generated video, the better the summary.

  The evaluation function is designed in consideration of the above viewpoints. In an example of the present embodiment, the following evaluation function f is designed.

This evaluation function is composed of five terms, and each term corresponds to the above-described five evaluation scales (1) to (5). λ _{1 to} λ ₅ are scaling parameters. Each section is explained in detail.

First, the first term rel (s _i ) corresponding to the evaluation scale (1). Evaluate the compatibility of keyword queries with video metadata. For this purpose, it is necessary to measure the semantic closeness between the keyword query and the metadata (corresponding video) of each video section. For this, for example, Normalized Google Distance (NGD) described in Reference 9 can be used.
[Reference 9]: RL Cilibrasi and PMB Vitanyi, “The Google Similarity Distance”, IEEE Transactions on Knowledge and Data Engineering, Vol. 19, No. 3, pp. 370-383, 2007.

NGD calculates the semantic proximity of the two keywords t ₁ and t ₂ as numerical values. Using this, rel (s _i ) can be defined as follows, for example.

Here, T is a set of keywords appearing in the metadata of the video section (video), t _k is the element, and q is a keyword query given by the user. Thereby, the first term of the formula [1] can be defined.

Subsequently, sem (s _i , s _j ) of the second term of the formula [1] will be described. The second term evaluates the semantic diversity of the video included in the summary, and evaluates the semantic similarity between any two video sections (including the original video) included in the summary. Should be designed as follows.

Based on the same idea as in item 1, the semantic content of the video section can be read from the summary text of the metadata. At this time, the similarity between the semantic contents of the two video sections can be evaluated using NGD. It is assumed that there are two video sections s _i and s _j , and the keyword sets in the metadata assigned to these are set as T _i and T _j , respectively. At this time, the semantic similarity between the two video sections s _i and s _j is defined as follows, for example.

The sem (s _i , s _j ) defined in this way becomes a larger value if the keyword sets of the two video sections s _i and s _j are similar, and conversely if they are not similar. The smaller the value.

Next, app (s _i , s _j ) in the third term of the formula [1] will be described. This third term evaluates whether or not various visual segments are included as “look”, and the similarity of “look” between any two video segments (including the original video) included in the summary. Should be designed to evaluate.

As described above, “look” includes not only visual (image) proximity but also auditory (sound) proximity. The “appearance” of the video section can be evaluated using the feature quantity of the video section, and the similarity of the “appearance” of the two video sections can be defined as the similarity of the feature quantity of the video section. As for the feature amount of the video section, various types of feature amounts are extracted for both the image feature and the sound feature in step S203 of the preprocessing. The feature amount group of the video section s _i is expressed as V = {v ₁ , v ₂ ,..., V _P }. v _k is each feature quantity (vector), for example, v ₁ is a color, v ₂ is a texture, v ₃ is a pitch, and so on.

Each feature quantity defines a distance dv _k (k = 1, 2,..., P) of the same feature quantity. For example, for a color (L * a * b * histogram), a chi-square distance for measuring the distance between the histograms, Earth Mover's distance (EMD) described in Reference 10, or the like can be used.
[Reference 10]: S. Peleg, M. Werman, and H. Rom. A unified approach to the change of resolution: Space and gray-level. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, pp. 739 -742, 1989.

  Further, for example, Euclidean distance can be used for a vector type feature such as GIST.

As described above, it is assumed that there are two video sections s _i and s _j under the given conditions, and these feature amounts are expressed as V _i = {v _k ⁱ , v _k ⁱ ,..., V _k ⁱ }, V _j = { Let v _k ^j , v _k ^j ,..., v _k ^j }. At this time, the “look” similarity between the two video sections s _i and s _j is defined as follows, for example.

Here, θ _k is a scaling parameter.

  Next, the fourth term sm (S ′) in the formula [1] will be described. Section 4 evaluates the ease of viewing the entire summary. More specifically, for the ordered set of video sections, the semantic content of adjacent video sections, the similarity of “look”, or the similarity of both is evaluated.

The connection of semantic contents can be calculated in the same way as in item 2. The connection between the semantic contents of two adjacent video sections s _i and s _{i + 1} is defined as follows using, for example, the formula [3].

The “look” connection can also be calculated in the same way as in Section 3. The connection of “look” of two adjacent video sections s _i and s _{i + 1} is defined as follows using, for example, the formula [4].

  Based on the above definitions, the semantic content of the entire summary video and the goodness of “look” connection can be defined as follows:

Finally, the fifth term dur (s _i ) of the formula [1] will be described. The fifth term evaluates the time length of the video included in the summary, and dur (s _i ) may be a negative time length of s _i .

Alternatively, instead of using dur (s _i ), another evaluation term may be introduced. For example, when there is a desired length of the summary video, the summary video closer to the time length may be designed to obtain a better evaluation value. The video summary length of time desired tau, the digest video time length of s _i and dur (s _i), defined for example by the following equation.

− | Τ−dur (s _i ) |
The above is an example of the evaluation function design. The definition of each item listed here is an example, and any other form may be adopted as long as the above-mentioned requirements are satisfied.

  In step S205, a summary video that maximizes the designed evaluation function is found. This is a kind of combinatorial optimization problem for obtaining ordered sets, and there is generally no efficient solution. Therefore, in an example of this embodiment, this is approximately obtained. As an approximate solution to the combinatorial optimization problem, there are Greedy-Search and genetic algorithms, and the optimum S ′ may be obtained by applying these.

As an example, FIG. 4 shows a flow of optimization processing when Grayy-Search is used.
(A) List all permutations with 2 video segments.
(B) The value of the evaluation function f is calculated for all permutations.
(C) The permutation having the maximum value of the evaluation function f is merged as one video section.
(D) The processes from (a) onward are repeated until there is one video section.
The above is an example of the processing details of step S205.

  Finally, in step S206, the summary video output in step S205 is connected as video and output as summary video 3.

  The above is the description of the video summarization apparatus and the video summarization method performed in the example of the present embodiment. It goes without saying that the processing process carried out by this video summarization method can be described as a computer-readable program.

  The video summarization apparatus according to the example of the present embodiment has been described in detail above. The present invention is not limited to the example of the embodiment described, and various modifications can be made within the technical scope described in the claims.

DESCRIPTION OF SYMBOLS 1 Video summarization apparatus 2 Video group 3 Summary video 100 Video input part 101 Storage part 102 Video segment division part 103 Video segment feature-value extraction part 104 Video segment group selection part 105 Evaluation function optimization part 106 Summary video generation part

Claims (10)

A video summarization device that generates a summary video for a group of videos and given video groups composed of metadata attached to each video,
Video segment dividing means for dividing each video included in the video group into video segments;
Video segment feature quantity extraction means for extracting feature quantities for each video segment;
Storage means for storing metadata incidental to the video, division information of the video section, and a feature amount for each video section;
Video segment group selection means for receiving a summary generation request, referring to the storage means, and selecting one or more video segments corresponding to the received request;
Using the metadata of the video to which the video section included in the selected video section group belongs, the extracted feature amount, or both information, one or more video sections are selected, and the time order is determined. Evaluation function optimizing means for obtaining an ordered video segment set that maximizes an evaluation value calculated by a predetermined evaluation function for the ordered video segment set;
A summary video generation means for generating a summary video by combining the obtained ordered video segment sets;
The evaluation function is a term for obtaining ease of viewing of an ordered video section, and is a similarity of metadata assigned to adjacent video sections or an amount of feature of adjacent video sections for an ordered set of video sections. A video summarization apparatus including a term for evaluating similarity or similarity of both . The video summarization apparatus according to claim 1,
The feature amount is configured by extracting at least one of a brightness feature, a color feature, a motion feature, a texture feature, a concept feature, and a landscape feature from an image frame at a predetermined position included in the video section. A video summarization apparatus characterized by being a vector to be generated. The video summarization apparatus according to claim 1 or 2,
At least one of a pitch feature, a sound pressure feature, a spectrum feature, a rhythm feature, an utterance feature, a music feature, and a sound event feature is obtained from the sound signal at the predetermined position included in the video section. A video summarization apparatus characterized by being a vector constructed by extraction. In the video summarization apparatus according to claim 1, claim 2 or claim 3,
The video section group selecting means accepts a summary generation request by a keyword from a user, and determines the relevance between the keyword and the video by determining a match between the keyword and the metadata of the video stored in the storage means. The video summarization apparatus is characterized in that the video section divided from the related video is selected. The video summarization device according to any one of claims 1 to 4,
The evaluation function further includes:
A term to determine suitability for the requirements of the ordered video segment set, a term to determine the completeness of the semantic content of the video segment included in the ordered video segment set, and the completeness of the features of the video segment included in the ordered video segment set A video summarization apparatus comprising at least one of a term to be obtained and a term to obtain a time length of an ordered video section. A video summarization method executed by a video summarization apparatus, comprising storage means for storing a video group composed of a plurality of videos and metadata attached to each video, and generating a summary video for these video groups. ,
A video segmentation process in which each video included in the video group is divided into video segments and stored in a storage means;
A video segment feature value extraction process for extracting a feature value for each video segment and storing it in the storage means;
A video section group selection process for receiving a summary generation request, referring to the storage means, and selecting one or more video sections corresponding to the received request;
Using the metadata of the video to which the video section included in the selected video section group belongs, the extracted feature amount, or both information, one or more video sections are selected, and the time order is determined. This is a term for determining the ease of viewing of a predetermined ordered video section with respect to the ordered set of video sections, and the similarity of metadata assigned to adjacent video sections or the adjacent videos for the ordered set of video sections. An evaluation function optimization process for obtaining an ordered set of video segments that maximizes an evaluation value calculated by an evaluation function including a term that evaluates the similarity of the feature values of the segments or the similarity of both ;
A video summarization method comprising: executing a summary video generation process for generating a summary video by combining the obtained ordered video segment sets. The video summarization method according to claim 6,
The feature amount is configured by extracting at least one of a brightness feature, a color feature, a motion feature, a texture feature, a concept feature, and a landscape feature from an image frame at a predetermined position included in the video section. And extracting at least one of pitch feature, sound pressure feature, spectrum feature, rhythm feature, speech feature, music feature, sound event feature from the sound signal at a predetermined position. A video summarization method characterized by being a vector constituted by: The video summarization method according to claim 6 or 7,
In the video section group selection process, a summary generation request based on a keyword from a user is received, and the relevance between the keyword and the video is determined by determining the matching between the keyword and video metadata stored in the storage means. The video summarization method is characterized in that the video section divided from the related video is selected. The video summarization method according to claim 6, claim 7 or claim 8,
The evaluation function further includes:
A term to determine suitability for the requirements of the ordered video segment set, a term to determine the completeness of the semantic content of the video segment included in the ordered video segment set, and the completeness of the features of the video segment included in the ordered video segment set A video summarization method comprising at least one of a term to be obtained and a term to obtain a time length of an ordered video section.   A video summarization program for causing a computer to execute the video summarization method according to any one of claims 6 to 9.

Images