JP6321945B2 - Digest video generation device, digest video generation method, and digest video generation program - Google Patents

Description

  The present invention relates to a digest video generation device, a digest video generation method, and a digest video generation program that edit video and generate a digest video.

  Since it may take a very long time to view all the videos in order for the user to determine whether or not he / she is interested in the contents of the videos, it may be difficult to determine after viewing all the videos. In order to grasp the contents of a video in a short time, a digest video of the video is generated and the digest video is viewed.

  As a technique for generating a digest video, for example, there is a technique described in Patent Document 1. In this technology, the operation status of other users is analyzed, the attention level of the scene is calculated based on the interest information of the other users, the scene with the other user attention level is selected from the video, and the digest is selected in the selected scene. A video is generated and presented.

JP 2012-248987 A

  However, simply viewing a scene with a high degree of attention may generate a digest video that does not consider the context of the content or the relationship between the scenes, which may make it difficult to grasp the content. For example, in a soccer game video, it is considered that the degree of attention increases in the scene at the moment when a score is entered. However, just looking at the scene at the moment when the score was entered, it was impossible to grasp the context of the scene with the score, such as what kind of play had led to the score. It may not be possible to grasp. For example, in the case of video that is broadcast live for a long time, if the number of viewers varies depending on the time zone and the number of viewers is small, even if there are scenes similar to other scenes with high attention, There is also a problem that it cannot be added to the video. For this reason, sufficient information regarding the content of the video cannot be obtained only with the digest video generated as described above, and the video has to be viewed in order to grasp the content of the video.

  The present invention is intended to solve the above-described problems, and provides a digest video generation device, a digest video generation method, and a digest video generation program that generate a digest video that can increase the degree of user understanding of video content. The purpose is to do.

One aspect of the present invention includes a scene dividing unit that divides a designated video into a plurality of scenes, a user interest degree calculating unit that calculates an interest level for each of the plurality of scenes, and two scenes in the plurality of scenes. Inter-scene similarity calculation unit that calculates the similarity between scenes for each combination, the interest degree of each scene calculated by the user interest calculation unit, and the similarity between scenes calculated by the inter-scene similarity calculation unit A scene score calculation unit that calculates a score for each scene based on the above, a scene is selected from the plurality of scenes based on the score calculated for each of the plurality of scenes, and the video including the selected scene is selected. It includes a digest video generation unit for generating a digest video, the said scene score calculation unit, each of the plurality of scenes to a node, similar between scenes Each node after moving randomly between nodes using the node's degree of interest as the existence probability for each node in the initial state and the similarity assigned to the edge between nodes as the transition probability a digest video generation device characterized that you calculate the existence probability of the node as the score.

  According to another aspect of the present invention, in the digest video generation device described above, the inter-scene similarity calculation unit includes a start time of each of the plurality of scenes in the video, a keyword related to the scene, and a video feature of the scene. The similarity between scenes is calculated using at least one of the above.

Another aspect of the present invention is a digest video generation method performed by a digest video generation apparatus that generates a digest video of a specified video, a scene division step for dividing the video into a plurality of scenes, A user interest level calculating step for calculating an interest level for each scene, an interscene similarity level calculating step for calculating a similarity level between scenes for each combination of two scenes in the plurality of scenes, and the user interest level calculating step. A scene score calculating step for calculating a score for each scene based on the calculated interest degree of each scene and the similarity between scenes calculated in the inter-scene similarity calculating step, and calculating for each of the plurality of scenes Selecting a scene from the plurality of scenes based on the score obtained, and selecting the selected scene Anda digest video generation step of generating a digest video of the video, including, in the scene score calculating step to generate a graph edge between the similarity nodes between scene each of the plurality of scenes and nodes, The node's degree of interest is the probability of existence for each node in the initial state, the similarity assigned to the edge between nodes is used as the transition probability, and the existence probability of each node after moving randomly between nodes is calculated as a score A digest video generation method.

  According to another aspect of the present invention, in the digest video generation method described above, in the inter-scene similarity calculation step, a start time of each of the plurality of scenes in the video, a keyword related to the scene, and a video feature of the scene The similarity between scenes is calculated using at least one of the above.

  One embodiment of the present invention is a digest video generation program for causing a computer to function as the digest video generation apparatus described above.

  According to the present invention, a score is calculated based on the degree of interest calculated for each scene and the similarity between scenes, and each scene included in the video is selected by selecting a scene to be included in the digest video based on the calculated score. It is possible to generate a digest video including a scene that is easily interesting among the scenes and a scene related to the scene, and it is possible to increase a user's level of understanding of the video content.

It is a block diagram which shows the structural example of the digest video generation apparatus 1 in this embodiment. It is a figure which shows an example of the video information management table which the video information storage part 11 in this embodiment has memorize | stored. It is a figure which shows an example of the video-scene relationship management table which the video information storage part 11 in this embodiment has memorize | stored. It is a figure which shows an example of the keyword information management table which the video information storage part 11 in this embodiment has memorize | stored. It is a figure which shows an example of the video-keyword relationship management table which the video information storage part 11 in this embodiment has memorize | stored. It is a figure which shows an example of the scene interest degree management table which the video information storage part 11 in this embodiment has memorize | stored. It is a figure which shows an example of the scene by which the scene score calculation part 16 in this embodiment produces | generates a graph. It is a figure which shows an example of the graph which the scene score calculation part 16 in this embodiment produces | generates. It is a figure which shows an example of the vector v (score of each scene) which the scene score calculation part 16 in this embodiment calculated. It is a flowchart which shows the digest video generation process which the digest video generation apparatus 1 in this embodiment performs.

  Hereinafter, a digest video generation device, a digest video generation method, and a digest video generation program according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of a digest video generation apparatus 1 according to the present embodiment. As shown in FIG. 1, the digest video generation device 1 includes a video information storage unit 11, a video designation unit 12, a scene division unit 13, a user interest level calculation unit 14, an interscene similarity calculation unit 15, and a scene score calculation unit 16. A digest video generation unit 17 and a digest video output unit 18.

  The video information storage unit 11 stores video and information accompanying the video. The information accompanying the video includes, for example, a video information management table, a video-scene relationship management table, a keyword information management table, a video-keyword relationship management table, and a scene interest level management table. An example of each table will be described below.

  FIG. 2 is a diagram illustrating an example of a video information management table stored in the video information storage unit 11 according to the present embodiment. As shown in FIG. 2, the video information management table has columns of video ID, video title, and description of contents. Each row in the video information management table exists for each video. The video ID is an identifier for identifying a video stored in the video information storage unit 11. The video title is the title of the stored video. The description of the content is text information indicating the content of the stored video. The video stored in the video information storage unit 11 can be read out by designating a video ID, a video title, and a keyword included in the content description.

  FIG. 3 is a diagram illustrating an example of a video-scene relationship management table stored in the video information storage unit 11 according to the present embodiment. As shown in FIG. 3, the video-scene relationship management table has columns of items of video ID, scene ID, and time stamp. Each row in the video-scene relationship management table exists for each scene included in the video stored in the video information storage unit 11. The scene ID is an identifier for identifying each scene included in the video identified by the video ID. The time stamp is an elapsed time (start time) from the start point of the video identified by the video ID to the start of the scene. Each scene included in the video stored in the video information storage unit 11 can be read by designating the video ID and the scene ID.

  FIG. 4 is a diagram illustrating an example of the keyword information management table stored in the video information storage unit 11 according to the present embodiment. As shown in FIG. 4, the keyword information management table has columns of items of keyword ID and keyword name. Each row in the keyword information management table exists for each keyword. The keyword ID is an identifier for identifying a keyword. The keyword name is text information indicating the keyword.

  FIG. 5 is a diagram illustrating an example of a video-keyword relationship management table stored in the video information storage unit 11 according to the present embodiment. As shown in FIG. 5, the video-keyword relationship management table has columns of items of video ID, scene ID, and keyword ID. Each row in the video-keyword relationship management table exists for each keyword associated with a scene included in the video. By specifying the video ID and the scene ID, the keyword ID of the keyword associated with the scene (part of the video) specified by the video ID and the scene ID can be obtained.

  FIG. 6 is a diagram illustrating an example of a scene interest level management table stored in the video information storage unit 11 according to the present embodiment. As shown in FIG. 6, the scene interest level management table has columns of items of video ID, scene ID, and interest level. The degree of interest is a value obtained by quantifying the user's interest in the scene specified by the video ID and the scene ID. By specifying the video ID and the scene ID, it is possible to obtain the degree of interest associated with the scene (part of the video) specified by the video ID and the scene ID.

Returning to FIG. 1, the description of the configuration of the digest video generation device 1 will be continued.
The video designating unit 12 receives information specified by a user's operation as a target video for generating a digest video. The video designation unit 12 outputs the input information to the scene division unit 13. The information input by the video designating unit 12 includes, for example, a video title of the video, a word indicating the content of the video, a keyword related to the video, a video ID assigned to the video, and the like.

The scene dividing unit 13 specifies a target video for generating a digest video based on information output from the video designating unit 12 and each table stored in the video information storage unit 11. The scene dividing unit 13 reads the specified video and divides the read video into a plurality of scenes. When the scene dividing unit 13 divides an image into a plurality of scenes, a known method, for example, a cut point detection method described in Reference Document 1 is used. The scene division unit 13 assigns a scene ID corresponding to each of the plurality of divided scenes, and uses the elapsed time in the video where the scene starts as a time stamp. The scene dividing unit 13 stores the video ID, the scene ID, and the time stamp in association with each scene.
[Reference 1]: Japanese Patent No. 2869398

  The user interest level calculation unit 14 calculates the interest level for each scene obtained by the scene division unit 13, and stores the calculated interest level, the scene ID, and the video ID in association with each other in the scene interest level management table. As the interest level calculated by the user interest level calculation unit 14, for example, an audience rating for each scene of the video can be used. The degree of interest of the user can be calculated by any method. For example, when the video is provided by a system capable of obtaining viewer feedback, the number of comments input, the push of a button indicating empathy, etc. The number may be used so that the degree of interest increases as the frequency of feedback increases.

  The inter-scene similarity calculation unit 15 calculates the similarity between scenes of a target video for generating a digest video. The inter-scene similarity calculation unit 15 calculates the similarity based on the co-occurrence of keywords associated with the scene, the proximity of the scene time stamp, or the video characteristics of the scene.

  When calculating the similarity using the co-occurrence of keywords associated with the scene, the inter-scene similarity calculation unit 15 calculates the similarity R (A, B) between the scene A and the scene B by the following equation (1). calculate.

In equation (1), | K _A | is the number of keywords associated with scene A, | K _B | is the number of keywords associated with scene B, and | K _A ∩K _B | B is the number of keywords that accompany B. As keywords associated with the scene, keywords extracted from text attached to the video as subtitles or keywords included in comments input by the viewer to the scene can be used. The inter-scene similarity calculation unit 15 stores the extracted keyword and the keyword ID for identifying the keyword in association with each other in the keyword information management table. The inter-scene similarity calculation unit 15 stores the keyword ID, the video ID, and the scene ID in association with each other in the video-keyword relationship management table.

When calculating the similarity using the closeness of the time stamp of the scene, the scene similarity calculation unit 15 calculates the similarity R (A, B) between the scene A and the scene B by the following equation (2). To do. T _A is an elapsed time from the start point of the video until the scene A starts, and T _B is an elapsed time from the start point of the video to the start of the scene B.

  When calculating the similarity using the video feature of the scene, the scene similarity calculation unit 15 quantifies the similarity of the color and sound of the video in the scene and the motion in the video, and calculates the similarity R (A, B). calculate.

The scene similarity calculation unit 15 may calculate the similarity R (A, B) between scenes by combining similarities based on a plurality of indices including the above-described indices. In this case, the similarity R (A, B) is calculated by the following equation (3). Note that R _i (A, B) is the similarity between the scene A and the scene B calculated based on an arbitrary index, and w _i is a weighting factor for the similarity R _i (A, B).

  The scene score calculation unit 16 calculates the score of each scene using the similarity between scenes calculated by the inter-scene similarity calculation unit 15. The scene score calculation unit 16 calculates the score of each scene as follows. The scene score calculation unit 16 generates an undirected graph (hereinafter referred to as a graph) in which a scene is a node and a similarity between scenes is an edge between nodes of the corresponding scene.

  FIG. 7 is a diagram illustrating an example of a scene for which the scene score calculation unit 16 according to the present embodiment generates a graph. FIG. 8 is a diagram illustrating an example of a graph generated by the scene score calculation unit 16 in the present embodiment. Here, an example is shown in which a graph is created by calculating a similarity based on a time stamp for a soccer game video. FIG. 7 shows information (time stamp and degree of interest) related to scenes 1 to 5 which are nodes when creating a graph. FIG. 8 shows a graph generated by the scene score calculation unit 16.

The scene score calculation unit 16 has a row and a column corresponding to each scene, and calculates a value calculated from the degree of interest of the scene corresponding to the row and the similarity between the scenes corresponding to the row and the column. A probability matrix P as a value is calculated. The element _pij of i row j column of the probability matrix P is calculated by following Formula (4), for example.

In Equation (4), α is a weighting factor indicating which of the interest level and the similarity level is important, I (S _i ) is the interest level of the i-th scene Si in the video, and R (S _i , S _j ) is the similarity between the scenes of the i-th scene S _i and the j-th scene S _j in the video. In addition, the scene score calculation unit 16 includes a sum of the interest levels of each scene in the video (Σ _i I (S _i )) and a similarity level between the scenes (Σ _i R (S _i , S _j )). Normalize so that becomes 1.

  The scene score calculation unit 16 calculates a vector v satisfying the following expression (5) for the calculated probability matrix P.

The calculated vector v has a probability p _ij (= αI (S _i ) + (1−α) R (S _i , S _j )) when an arbitrary node in the graph is the starting point and the user is in the scene S _j. This is the existence probability in each scene (node in the graph) when the movement to the scene S _i is repeated infinitely. The vector v corresponds to the eigenvector when the eigenvalue of the probability matrix P is maximized. The value of each dimension when the vector v is normalized to length 1 is set as the score of the scene corresponding to each dimension of the vector v.

  For example, when the score of each scene when α = 0.5 is calculated for the five scenes 1 to 5 shown in FIGS. 7 and 8, the values shown in FIG. 9 are obtained. FIG. 9 is a diagram illustrating an example of a vector v (score of each scene) calculated by the scene score calculation unit 16 in the present embodiment. In FIG. 9, for example, the value of the element in 5 rows and 3 columns is obtained by multiplying the degree of interest of the scene 5 by α (= 0.5) and the similarity between the scene 3 and the scene 5 by (1−α). The value is based on the sum of the multiplied value. The elements of the probability matrix P are normalized so that the sum of the elements in each column is 1. However, in the illustrated example, since the second decimal place is shown, an error occurs in the second column and the fourth column.

  The digest video generation unit 17 generates a digest video from the video based on the score of each scene calculated by the scene score calculation unit 16. Scenes to be added to the digest video are selected in order from the scene with the highest scene score so that the number of scenes or the total time of the scenes is within a predetermined threshold. For example, when the digest video generation unit 17 selects two scenes from the scenes 1 to 5 illustrated in FIGS. 7 to 9 to generate a digest video, the digest 3 having the highest score and the next scene are scored next. Scene 2 (attack scene) with a high score is selected. The digest video generation unit 17 arranges the two selected scenes in ascending order of time stamps and outputs them as a digest video.

  The digest video output unit 18 displays the digest video generated by the digest video generation unit 17 and presents it to the user. In addition, when the place where the digest video production | generation apparatus 1 and a user are located apart, the digest video output part 18 will transmit a digest video to the apparatus which the user is operating. When the digest video is distributed via a server device provided on the network, the digest video output unit 18 transmits the digest video to the server device for storage.

  FIG. 10 is a flowchart showing digest video generation processing performed by the digest video generation device 1 according to this embodiment. When the digest video generation process is started, the video designating unit 12 receives information from the user and inputs information for designating a video to be presented as a digest video (step S11).

  The scene dividing unit 13 specifies a video based on the information input by the video designating unit 12 and each table stored in the video information storage unit 11, and divides the specified video into a plurality of scenes (step S12). .

  The user interest level calculation unit 14 repeatedly performs the process of calculating the user interest level for the scene for each of the plurality of scenes obtained by the scene division unit 13 (step S13).

  The inter-scene similarity calculation unit 15 repeatedly performs the process of calculating the similarity between scenes for every combination of two scenes in the plurality of scenes obtained by the scene division unit 13 (step S14).

  The scene score calculation unit 16 calculates the score of each scene based on the degree of interest of the user of each scene calculated in step S13 and the similarity between the two scenes calculated in step S14 (step S15). ).

  The digest video generation unit 17 selects a scene to be included in the digest video based on the score of each scene calculated by the scene score calculation unit 16, and extracts the selected scene from the video stored in the video information storage unit 11. The extracted scene images are connected in the order of time stamps to generate a digest image (step S16).

  The digest video output unit 18 presents the digest video generated in step S16 to the user (step S17), and ends the digest video generation process.

  The digest video generation apparatus 1 of the present embodiment calculates and assigns an interest level based on the behavior of another user (viewer) for each scene obtained by dividing a video specified by the user. Subsequently, the degree of similarity between scenes is calculated based on information such as time, keywords, and video features. Then, select not only high-interest scenes but also scenes related to high-interest scenes based on the similarity between scenes, that is, scenes that may be of high interest to the user. By generating and presenting a digest video including a scene similar to the scene, it is possible to increase the user's level of understanding of the video content.

  In addition, the digest video generation apparatus 1 according to the present embodiment analyzes a comment that is obtained as a reaction of the viewer of the video and a similarity of the feature of the video, so that a scene similar to a scene that is likely to be interesting is digest video. Can be included. Thereby, even if it is a scene with few viewers, the scene which is easy to be interesting can be included in a digest image | video, and can be shown and produced.

  In addition, the digest video generation apparatus 1 according to the present embodiment generates a graph in which each scene is a node in the graph and the similarity between the scenes is an edge, and the interest value of each scene is set to the initial value of each node. As the existence probability, the similarity between scenes is used as a transition probability, and the existence probability at each node after moving randomly between nodes is used as the score of the scene. Since the value of each element of the probability matrix P used for calculating the score can be adjusted by the parameter α indicating which of the high degree of interest and the similarity between scenes is important, a flexible digest Generation is possible.

  Since the digest video generation apparatus 1 of the present embodiment has the above-described features, for example, in addition to a scene at the moment when a score is entered for a soccer game video, an attacking scene before the score is entered, The scene of another play by the player who has scored the goal can be included in the digest video and presented to the user.

  In the present embodiment, the configuration in which the digest video device 1 includes the video information storage unit 11 has been described. However, the video information storage unit 11 is a server device that can be accessed by the digest video generation device 1 via a network or the like. Also good. In this case, each functional unit included in the digest video device 1 accesses the server device via the network and performs the above-described processing.

  You may make it implement | achieve the digest image | video production | generation apparatus 1 in embodiment mentioned above with a computer. In that case, a program for realizing each functional unit included in the digest video generation apparatus 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. It may be realized. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” is a program that dynamically holds a program for a short time, like a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  It can also be applied to applications where it is indispensable to increase the degree of grasp of the user with respect to the video content by digest video combining partial videos included in the video.

DESCRIPTION OF SYMBOLS 1 ... Digest image | video production | generation apparatus 11 ... Image | video information storage part 12 ... Image | video designation | designated part 13 ... Scene division | segmentation part 14 ... User interest degree calculation part 15 ... Interscene similarity calculation part 16 ... Scene score calculation part 17 ... Digest image generation part 18 ... Digest video output section

Claims (5)

A scene dividing unit that divides the specified video into a plurality of scenes;
A user interest degree calculation unit for calculating an interest degree for each of the plurality of scenes;
An inter-scene similarity calculation unit that calculates the similarity between scenes for each combination of two scenes in the plurality of scenes;
A scene score calculator that calculates a score for each scene based on the interest of each scene calculated by the user interest calculator and the similarity between scenes calculated by the inter-scene similarity calculator;
A digest video generation unit that selects a scene from the plurality of scenes based on the score calculated for each of the plurality of scenes, and generates a digest video of the video including the selected scene;
With
The scene score calculation unit
A graph in which each of the plurality of scenes is a node and the similarity between scenes is an edge between the nodes is generated, and the similarity that is assigned to the edge between the nodes is defined as the existence probability for each node in the initial state. A digest video generation device characterized by calculating, as a score, the existence probability of each node after randomly moving between nodes using as a transition probability. The digest video generation device according to claim 1 ,
The inter-scene similarity calculation unit
A digest video generation apparatus, wherein the similarity between scenes is calculated using at least one of a start time of each of the plurality of scenes in the video, a keyword related to the scene, and a video feature of the scene. A digest video generation method performed by a digest video generation device that generates a digest video of a specified video,
A scene dividing step of dividing the video into a plurality of scenes;
A user interest degree calculating step for calculating an interest degree for each of the plurality of scenes;
An inter-scene similarity calculation step of calculating a similarity between scenes for each combination of two scenes in the plurality of scenes;
A scene score calculation step for calculating a score for each scene based on the interest level of each scene calculated in the user interest level calculation step and the similarity level between scenes calculated in the inter-scene similarity calculation step;
Selecting a scene from the plurality of scenes based on the score calculated for each of the plurality of scenes, and generating a digest video of the video including the selected scene; and
Including
In the scene score calculation step,
Generate a graph in which each of the plurality of scenes is a node and the similarity between the scenes is an edge between the nodes, and the interest degree of the node is the existence probability for each node in the initial state, and the similarity assigned to the edge between the nodes is A digest video generation method characterized by calculating, as a score, the existence probability of each node after moving randomly between nodes as a transition probability. The digest video generation method according to claim 3 ,
In the inter-scene similarity calculation step,
A digest video generation method, wherein the similarity between scenes is calculated using at least one of a start time of each of the plurality of scenes in the video, a keyword related to the scene, and video characteristics of the scene. A digest video generation program for causing a computer to function as the digest video generation device according to claim 1 .

Images