JP2019003585A - Summary video creation device and program of the same - Google Patents

Abstract

To provide a summary video creation device that alleviates semantic discontinuity to create summary videos from original videos.SOLUTION: A summary video creation device 1 comprises: video analysis means 10 that detects shots from original videos, and classifies the shots into scenes; importance calculation means 30 that calculates the importance score of each shot; consistency score evaluation means 50 that acquires a consistency score corresponding to a distance of a scene from consistency score storage means 40 as an evaluation value of consistency for each shot detected by the video analysis means 10; and shot selection/coupling means 60 that selects a preset number of shots high in importance from the original video, and selects the shots great in consistency level with respect to the shot high in importance until a summary time stands a preset time length and couples the selected shots.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a summary video generation apparatus that generates a summary video from a video and a program thereof.

In recent years, video summarization technology for generating summary video summarizing content from video such as broadcast programs has evolved, and a method for automatically generating summary video has been disclosed (see Patent Document 1).
The method disclosed in Patent Document 1 (hereinafter referred to as a conventional method) divides a given video (original video) and obtains a feature amount of the divided video (shot). Further, the conventional method calculates a score for determining that the feature amount of the shot is characteristic in the original video.
In the conventional method, shots are selected in descending order of the calculated score until the desired summary video time length is reached, and the selected shots are combined in the order of time of the original video to generate a summary video.

This conventional method uses a score obtained by clustering feature amounts generated from video features such as motion vectors, or a score calculated from the score and feature amounts related to words in character data obtained by recognizing the voice of the original video. Based on the score obtained by adding together, each divided video constituting the summary video is determined.
Thus, the conventional method can automatically generate a summary video from the original video without human intervention.

JP 2012-10265 A

The conventional method described above generates a summary video by independently extracting and combining shots with high scores. That is, the conventional method does not consider the context in the original video of each shot, the context between shots, or the semantic role in the entire content of the original video.
Therefore, the summary video generated by the conventional method has abruptness caused by the lack of video in the introduction part at the start of the video, unnaturalness due to the division of a series of shots explaining a specific event, etc. In some cases, the viewer may feel the summary video.
As described above, when the summary video generated by the conventional method is used as a new content, the summary video obtained by mechanically combining shots that do not necessarily match contents is a qualitative problem. Is likely to occur.

  SUMMARY OF THE INVENTION The present invention has been made in view of such a problem, and a summary video generation apparatus and a program thereof that can generate a summary video by relaxing semantic discontinuity between shots in an original video It is an issue to provide.

  In order to solve the above problems, a summary video generation device according to the present invention is a summary video generation device that generates a summary video having a video duration shorter than the original video from the original video, the video analysis means, , Importance calculation means, consistency score storage means, consistency evaluation means, and shot selection / combination means.

In such a configuration, the summary video generation apparatus detects shots that are changing points of the original video based on motion vectors, color distributions, and the like, and classifies a series of shots having similar feature quantities as scenes. Thereby, the video analysis means classifies the original video into a shot and a scene to which the shot belongs.
Then, the summary video generation device calculates an importance score, which is an index indicating that the shot is characteristic in the original video, based on the feature amount of the shot by the importance calculation means. For example, the importance calculation means gives a higher score to a characteristic shot such as a color distribution different from other shots in the original video.

The summary video generation apparatus stores in advance a consistency score indicating a degree of consistency between shots corresponding to the distance in the time direction of the scene in the consistency score storage unit. This consistency score was previously learned from learning data, for each distance (scene difference) of the scene to which the shot belongs, the appearance probability indicating how far the adjacent shot in the summary video appears in the scene in the original video. Is. Here, the learning data is an original video whose shot and scene are known, and a summary video whose shot selected from the original video is known.
Then, the summary video generation device acquires a consistency score corresponding to the scene distance for each shot detected by the video analysis unit from the consistency score storage unit by the consistency evaluation unit, and matches the shot combination. Associate a sex score.

Then, the summary video generation device selects a preset number of high importance shots based on the importance score from the original video by the shot selection / combination means, and matches the high importance shots with consistency. Shots having a high degree of consistency according to the score are selected and combined until the time length of the summary video set in advance is reached.
As a result, the shot selection / combination means makes sense by adding a shot with a high degree of consistency to a shot with a high importance even if the shot with a high importance only has a large change between shots. Discontinuity can be mitigated.

  The summary video generation apparatus can be operated by a summary video generation program for causing a computer to function as each unit of the summary video generation apparatus.

The present invention has the following excellent effects.
According to the present invention, instead of extracting only shots with high importance that are characteristic in the original video, a summary video is added by adding shots with a high degree of consistency to shots with high importance. Can be generated.
Accordingly, the present invention can alleviate the semantic discontinuity between shots in the original video, and can generate a summary video in which the semantic flow of the video is more natural.

It is a block block diagram which shows the structure of the summary image | video production | generation apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the example of the division | segmentation video list which a video analysis means produces | generates. It is a figure which shows the example of the importance score list | wrist which an importance calculation means produces | generates. It is a figure which shows the example of the appearance probability for every scene difference (scene distance) which is a consistency score memorize | stored in a consistency score memory | storage means. It is a figure which shows the example of the consistency score list | wrist which a consistency evaluation means produces | generates. It is a figure which shows the example of the candidate shot list which the candidate shot selection means of a shot selection / combination means produces | generates. It is a figure which shows the example of the interpolation shot list which the interpolation shot search means of a shot selection / combination means produces | generates. It is a figure which shows the example of the candidate shot list updated by the candidate shot update means of a shot selection / combination means. It is a flowchart (1/2) which shows operation | movement of the summary image | video production | generation apparatus which concerns on 1st Embodiment of this invention. It is a flowchart (2/2) which shows operation | movement of the summary image | video production | generation apparatus which concerns on 1st Embodiment of this invention. It is a block block diagram which shows the structure of the summary image | video production | generation apparatus which concerns on 2nd Embodiment of this invention.

Embodiments of the present invention will be described below with reference to the drawings.
<< First Embodiment >>
[Configuration of summary video generator]
First, the configuration of the summary video generation device 1 according to the first embodiment of the present invention will be described with reference to FIG.

The summary video generation device 1 generates summary video (video content) having a video duration shorter than that of the original video from the original video (video content) such as a MPEG2 stream.
As shown in FIG. 1, the summary video generation apparatus 1 includes a video analysis unit 10, a parameter setting unit 20, an importance calculation unit 30, a consistency score storage unit 40, a consistency evaluation unit 50, and a shot selection. A coupling means 60 is provided.

  The video analysis means 10 detects a shot that becomes a change point of the original video and classifies a series of shots having similar feature quantities as a scene. The video analysis unit 10 includes a feature amount extraction unit 11, a shot detection unit 12, and a scene classification unit 13.

  The feature quantity extraction unit 11 extracts feature quantities from the original video in time units (for example, frame units). For example, the feature amount extraction unit 11 includes a motion vector, color information (color distribution, etc.), SIFT (Scale-Invariant Feature Transform) feature amount, etc. for each block area having a predetermined size with respect to the previous frame as image features. Is calculated. Further, the feature amount extraction unit 11 may calculate the presence / absence of a person authentication result in the frame or a person identifier.

  Note that the feature amount extracted by the feature amount extraction unit 11 is not limited to an image feature, and may be anything as long as it shows a time unit feature. For example, if sound is attached to the original video, its acoustic features (for example, logarithmic mel filter bank output, mel frequency cepstrum coefficient, etc. generally used in speech recognition) are extracted, and the above features are extracted. You may use together with quantity.

The shot detection means 12 detects a video section where the original video changes as a shot (divided video). Here, the shot detection unit 12 detects a change point in the time direction as a segment of the video section based on the motion vector extracted by the feature amount extraction unit 11.
The shot detection unit 12 includes an index, time information (for example, hour, minute, second, frame number) indicating a start point and an end point of the shot, a feature amount, in the order of the detected shots from the beginning of the original video. Are associated with each other and output to the scene classification means 13.

Note that a general method may be used as a shot detection method in the shot detection unit 12. For example, as described in Reference Document 1 below, a local image feature and a global image feature are used in combination. It is also possible to detect a shot.
(Reference 1) Evlampios Apostolidis, Vasileios. Mezaris, "Fast Shot Segmentation Combining Global and Local Visual Descriptors", Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014.

The scene classification unit 13 classifies the shots detected by the shot detection unit 12 as scenes for each partial series of continuous shots having similar feature amounts. The scene classification unit 13 calculates a representative value (average value, median value, etc.) of the feature amount for each shot, and classifies adjacent shots whose representative values approximate within a predetermined range as a scene.
The scene classification unit 13 generates a divided video list in which the index of the shot order input from the shot detection unit 12, the time information, and the feature amount are further associated with the scene index.

Note that, here, an example is shown in which a scene is divided by one type of feature quantity (for example, only an image feature, only an acoustic feature) as a feature quantity. It is good also as dividing. Note that a scene dividing method using both image features and acoustic features is a known technique as described in Reference Document 2 below, and thus description thereof is omitted.
(Reference 2) Panagoiotis Sidiropoulos, Vasileios. Mezaris, Hugo Meinedo, Miguel Bugalho, Isbel Trancoso, "Temporal Video Segmentation to Scenes Using High-Level Audiovisual Features", IEEE Transactions on Circuits and Systems for Video Technology, vol. 21, no 8, pp. 1163-1177, 2011.

Here, an example of a divided video list generated by the video analysis unit 10 will be described with reference to FIG. As shown in FIG. 2, the divided video list L1 includes a shot index N _shot , a scene index N _scene , a shot start point T _start , a shot end point T _end , and a feature amount F for each shot.

The shot index N _shot is an index assigned to the shots detected by the shot detection unit 12 in order from the head of the original video.
The scene index N _scene is an index assigned to the scene classified by the scene classification unit 13 in order from the head of the original video.

The shot start point T _start and the shot end point T _end indicate the start point (hour, minute, second, frame number) and end point (hour, minute, second, frame number) detected by the shot detection unit 12. It is time information.

The feature amount F is a feature amount for each frame in the video section of the shot start point T _start and the shot end point T _end among the feature amounts extracted by the feature amount extraction unit 11.
In FIG. 2, data for one shot is shown in two lines via a line feed “\”.
For example, a shot having # 00002 in the divided video list L1 as a shot index belongs to the scene of * 0001, has a start point “00: 00: 04.18” and an end point “00: 00: 06.09”. This indicates that the feature amount is “0.2416” or the like.

Returning to FIG. 1, the description of the configuration of the summary video generation apparatus 1 will be continued.
The video analysis unit 10 outputs the generated divided video list to the parameter setting unit 20, the importance level calculation unit 30, the consistency evaluation unit 50, and the shot selection / combination unit 60.

The parameter setting means 20 sets various parameters input via the external input device 2. The parameter setting means 20 displays a parameter setting screen on the display device 3 and receives parameter settings.
The parameter setting means 20 generates a thumbnail image of the shot indicated by the divided video list from the original video and displays it on the display device 3. Further, the parameter setting unit 20 may reproduce the original video according to an instruction from the user.
The user sets various parameters with reference to the thumbnail image and the reproduced image. The parameter setting means 20 may store the initial parameter values in advance and display them on the display device 3.

  Further, the parameter setting means 20 generates a thumbnail image in the original video of the candidate shot generated by the shot selection / combination means 60 described later when the summary video is generated with the initial value or the parameter set by the user, It is good also as displaying on the display apparatus 3. FIG. As a result, the user can roughly check the contents of the summary video.

Here, the parameters set by the parameter setting means 20 are the length of the summary video and the number of candidate shots.
The “summary video length” is the length of the summary video generated by the summary video generation device 1 and is, for example, a time length such as “05:00 (5 minutes)”.

  “Number of candidate shots” is an initial value of the number of shots to be selected as the summary video. The number of candidate shots indicates the number of shots selected by the importance score in the shot selecting / combining means 60 described later. Note that the number of candidate shots may be set by the user with reference to the thumbnail image, or may be set in advance according to the length of the summary video.

Returning to FIG. 1, the description of the configuration of the summary video generation apparatus 1 will be continued.
The parameter setting unit 20 outputs the set parameters to the shot selection / combination unit 60.

The importance level calculation means 30 calculates an importance level score, which is an index indicating that the shot is characteristic in the original video, based on the feature amount of the shot.
Here, the importance calculation means 30 calculates a score (importance score) indicating the degree of importance of the shot in the original video from the divided video list generated by the video analysis means 10. For the calculation of the importance score in the importance calculation means 30, a general method as described in JP2012-10265A may be used.
Specifically, the importance calculation means 30 uses a block region of a key frame arbitrarily extracted from a shot as a visual word, and uses a TF-IDF (term frequency-inverse document) from the visual words of all shots. frequency) is used as the importance score.

  The importance level calculation means 30 generates an importance level score list by associating the calculated importance level score with a shot. Specifically, the importance calculation unit 30 generates the importance score list by replacing the feature amount for each shot in the divided video list generated by the video analysis unit 10 with the importance score.

Here, an example of the importance score list generated by the importance calculation means 30 will be described with reference to FIG. As shown in FIG. 3, the importance score list L _{<b > 2} includes a shot index N _shot , a scene index N _scene , a shot start point T _start , a shot end point T _end , and an importance score P for each shot. .
Note that the shot index N _shot , the scene index N _scene , the shot start point T _start, and the shot end point T _end are the same as those in the divided video list L1 described in FIG. Therefore, the scene index N _scene , the shot start point T _start and the shot end point T _end may be omitted from the importance score list L2 by referring to the divided video list L1.

Returning to FIG. 1, the description of the configuration of the summary video generation apparatus 1 will be continued.
The importance calculation means 30 outputs the generated importance score list to the shot selection / combination means 60.

The consistency score storage means 40 stores a consistency score indicating the degree of consistency in advance according to the distance in the time direction of the scene. The consistency score storage means 40 can be configured by a general storage device such as a semiconductor memory.
Normally, the degree of continuity of meaning content becomes smaller (weaker) as the distance between scenes increases. Therefore, here, as an index of consistency between shots in the original video, a consistency score is stored in advance according to the difference of the scene to which each shot belongs.
Specifically, the consistency score is a probability that shots co-occur, and an appearance probability indicating how far adjacent shots of the summary video appear in the original video is represented by the scene to which the shot belongs. For each distance (scene difference). The calculation of the appearance probability will be described in the second embodiment.

FIG. 4 shows an example of the appearance probability of a scene difference that is a consistency score. The example of FIG. 4 shows the appearance probability of a scene difference, with the distance of a scene separated in the forward direction in the time direction as positive and the distance of the scene separated in the reverse direction as negative, based on one shot. .
For example, the appearance probability that the scene difference in the original video corresponding to the adjacent shot of the summary video is “2” is “0.2175”. A scene difference of “0” means that adjacent shots are included in the same scene in the original video. The higher the appearance probability is, the higher (stronger) the degree of simultaneous selection as a summary video for a certain shot.
Note that this consistency score may be simply set in advance as a value that increases as the distance of the scene decreases and decreases as the distance of the scene increases.

  Based on the divided video list and the consistency score stored in the consistency score storage unit 40, the consistency evaluation unit 50 matches the shots that are adjacent shots of the summary video. Is to evaluate. Here, to evaluate means to associate a consistency score indicating a degree of consistency with a combination of shots.

  Specifically, the consistency evaluation unit 50 obtains the difference of the scene to which the shot belongs for each combination of shots included in the divided video list generated by the video analysis unit 10. Then, the consistency evaluation unit 50 generates a consistency score list by associating the consistency score (appearance probability) corresponding to the scene difference stored in the consistency score storage unit 40 for each combination of shots. This consistency score list is a result of evaluating the degree of consistency in which the combination of shots is an adjacent shot of the summary video.

Here, an example of the consistency score list generated by the consistency evaluation unit 50 will be described with reference to FIG. As shown in FIG. 5, the consistency score list L _<b> 3 includes a corresponding shot index N _shot1 , N _shot2 and a consistency score C for each combination of shots.
Shot index _{N Shot1, N Shot2} is one in which the combination of the shots of the original image, described as pairs of shot index _{N shot} that is described in the segmented image list L1 described in Fig.
The consistency score C is a consistency score (appearance probability) stored in the consistency score storage unit 40 corresponding to the distance (scene difference) of the scene to which each shot of the shot indexes _Nshot1 and _Nshot2 belongs.

  For example, the third line of the consistency score list L3 indicates that the consistency score of two shots whose shot indexes are # 00001 and # 00004 is “0.3561”. Since each shot index of # 00001 and # 00004 belongs to each scene of * 0001 and * 0002 in the divided video list L1 shown in FIG. 2, the scene difference is “1”. The consistency score for the scene difference “1” is “0.3561” depending on the contents of the consistency score storage means 40 shown in FIG.

Returning to FIG. 1, the description of the configuration of the summary video generation apparatus 1 will be continued.
The consistency evaluation unit 50 outputs the generated consistency score list to the shot selection / combination unit 60.

  The shot selection / combination unit 60 generates a summary video by interpolating shots having a high consistency score by the consistency evaluation unit 50 with respect to shots having a high importance score calculated by the importance calculation unit 30. is there. The shot selection / combination means 60 includes candidate shot selection means 61, interpolation shot search means 62, candidate shot update means 63, and candidate shot combination means 64.

Candidate shot selection means 61 selects a preset number of shots as candidate shots of the summary video in order from the shot with the highest importance score.
Candidate shot selection means 61 determines the number of candidate shots that are parameters set by parameter setting means 20 from the divided video list based on the importance score for each shot in the importance score list generated by importance calculation means 30. Only select shots (candidate shots).

The selection of candidate shots based on the importance score is not particularly limited. For example, the candidate shot selection means 61 uses a numerical value obtained by normalizing the importance score for each shot in the importance score list L2 (see FIG. 3) (a numerical value scaled so that the sum is “1”). A probability distribution as a probability value to be selected is obtained, and a sample according to the probability distribution is extracted by non-restoration extraction to be a candidate shot.
Further, for example, the candidate shot selection unit 61 may simply select candidate shots in order from the shot having the highest importance score for each shot in the importance score list L2 (see FIG. 3).

Candidate shot selection means 61 lists the selected candidate shots in descending order of importance score. For example, the candidate shot selection means 61 may select a part (shot index N _shot , scene index N _scene , shot start point T _start , shot end point T _end) corresponding to the selected candidate shot from the divided video list L1 shown in FIG. , The importance score P) is extracted to generate the candidate shot list L4 shown in FIG.
The candidate shot selection means 61 outputs the generated candidate shot list to the interpolation shot search means 62 and the candidate shot update means 63.

The interpolation shot search unit 62 searches for a shot having a high consistency score with respect to the candidate shot selected by the candidate shot selection unit 61 as a shot (interpolation shot) for interpolating the candidate shot.
The interpolated shot search means 62 refers to the consistency score list generated by the consistency evaluation means 50 and interpolates the shot with the highest consistency score for each candidate shot in descending order of importance score of the candidate shot list. Select as a shot and list. For example, as illustrated in FIG. 7, the interpolation shot search unit 62 includes the candidate shot and the shot index N _shot and time information (shot start point T _start , shot end point) of the interpolation shot having a high consistency score for the candidate shot. _Tend ) is arranged to generate an interpolation shot list L5.

Here, when the total time length of the candidate shot and the interpolation shot exceeds the length of the summary video set by the parameter setting unit 20, the interpolation shot search unit 62 ends the search for the interpolation shot. The interpolation shot search means 62 does not register in the interpolation shot list in order to avoid duplication of shots when the same shot as the searched interpolation shot is included in the candidate shot list.
Further, when searching for an interpolated shot corresponding to a certain candidate shot, the interpolated shot searching means 62 exceeds the time indicated by the time information of other candidate shots preceding and following the original video on the time of the original video. Therefore, the interpolation shot is not searched.
The interpolation shot search means 62 outputs the generated interpolation shot list to the candidate shot update means 63.

  The interpolation shot search means 62 has a higher consistency score for a new candidate shot when there is an instruction to search for an interpolation shot for the candidate shot updated by the candidate shot update means 63. The shot is searched as an interpolation shot, an interpolation shot list is generated, and is output to the candidate shot update means 63.

The candidate shot update unit 63 updates the candidate shot by adding the interpolation shot searched by the interpolation shot search unit 62 to the candidate shot selected by the candidate shot selection unit 61.
The candidate shot update unit 63 adds the interpolation shots searched by the interpolation shot search unit 62 to the candidate shot list generated by the candidate shot selection unit 61 in order of importance score.

  For example, as shown in FIG. 8, the candidate shot update means 63 is generated by the interpolation shot search means 62 shown in FIG. 7 in the candidate shot list L4 generated by the candidate shot selection means 61 shown in FIG. Information (shot index, time information, importance score) corresponding to the interpolation shot described in the interpolation shot list is added. In this case, for example, the interpolation score of # 00005 corresponding to the candidate shot of # 00004 shown in FIG. 7 has an importance score smaller than the importance score of # 00006. It is arranged behind 00006 candidate shots.

If the total time length of the shots registered in the candidate shot list is less than the length of the summary video specified by the parameter, the candidate shot update unit 63 searches the candidate shot list after the update for the interpolation shot search. Output to means 62 to instruct the search for further interpolation shots. In this manner, the candidate shot update unit 63 updates the candidate shot list until the total time length of the shot becomes the length of the summary video.
The candidate shot update unit 63 outputs a candidate shot list in which the total time length of the shot has reached the length of the summary video to the candidate shot combination unit 64.

The candidate shot combining means 64 cuts out and combines the video (shot video) in the time interval corresponding to the candidate shot updated by the candidate shot update means 63 from the original video in the order of the time of the original video.
It should be noted that the process of cutting out a video of a predetermined time interval from the original video and the process of combining the cut shot video can be performed by a known method according to the video format of the original video. For example, the candidate shot combining unit 64 can cut and combine videos using FFmpeg shown in Reference Document 3 below.
(Reference 3) Frantisek Korbel, “FFmpeg Basics: Multimedia handling with a fast audio and video encoder,” CeateSpace Independent Publishing Platform, ISBN: 978-1479327836, 2012.

  Candidate shot combining means 64 outputs the combined video as a summary video for the original video. Note that the candidate shot combining unit 64 may output the summary video to the display device 3 and update the parameters by allowing the user to visually recognize the summary video.

By configuring the summary video generation device 1 as described above, the summary video generation device 1 adds a shot to the shot with high importance in the original video in consideration of consistency, and adds the summary video. Can be generated.
As a result, the summary video generation apparatus 1 can generate a summary video in which semantic discontinuity between shots in the original video is reduced.

[Operation of summary video generator]
Next, the operation of the summary video generation apparatus 1 according to the first embodiment of the present invention will be described with reference to FIGS. 9 and 10 (refer to FIG. 1 as appropriate for the configuration). Here, it is assumed that the consistency score (appearance probability) corresponding to the difference of the scene to which the shot belongs is stored in advance in the consistency score storage means 40.

As shown in FIG. 9, first, the summary video generation device 1 uses the video analysis means 10 to generate a divided video list in the following steps S1 to S4.
In step S1, the feature quantity extraction unit 11 of the video analysis unit 10 extracts a feature quantity in units of frames from the original video.
In step S2, the shot detection means 12 detects a video section in which the original video changes as a shot based on the feature amount extracted in step S1.
In step S3, the scene classification unit 13 classifies the shot detected in step S2 as a scene for each partial series of consecutive shots having similar feature values.
In step S4, the video analysis means 10 generates a divided video list L1 (see FIG. 2) composed of a shot index, a scene index, a shot start point, a shot end point, and a feature amount in the order of shots detected in step S2. .

Then, the summary video generating apparatus 1 sets parameters in the following steps 5 to S7 by the parameter setting unit 20.
In step S5, the parameter setting means 20 generates a thumbnail image for each shot from the original video based on the divided video list generated in step S4.
In step S6, the parameter setting means 20 displays the initial value of the parameter and the thumbnail image generated in step S5 on the display device 3.
In step S <b> 7, the parameter setting unit 20 receives a parameter change via the external input device 2.

Next, the summary video generation device 1 generates the importance score list by the importance calculation means 30 according to the following steps S8 to S9.
In step S9, the importance calculation means 30 calculates an importance score indicating the degree of importance in the original video for each shot of the divided video list generated in step S4.
In step S10, the importance calculation means 30 associates the importance score of the shot with the shot index and generates the importance score list L2 (see FIG. 3).

In addition, the summary video generation apparatus 1 generates a consistency score list by the consistency evaluation unit 50 in the following steps S10 to S11.
In step S10, the consistency evaluation unit 50 refers to the consistency score corresponding to the scene difference stored in the consistency score storage unit 40, and for the combination of shots, the difference of the scene to which each shot belongs. And the consistency score corresponding to the scene difference is set as the evaluation value of the combination of the shots.
In step 11, the consistency evaluation unit 50 generates a consistency score list L <b> 3 (see FIG. 5) that includes shot index pairs indicating combinations of shots and corresponding consistency scores.

  The procedure of the importance evaluation unit 30 in steps S8 to S9 and the procedure of the consistency evaluation unit 50 in steps S10 to S11 may be performed in advance. Alternatively, the procedure of the importance calculation unit 30 and the procedure of the consistency evaluation unit 50 may be performed in parallel.

Then, as shown in FIG. 10, the summary video generation device 1 uses the shot selection / combination means 60 to generate a summary video in the following steps S12 to S19.
In step S12, the candidate shot selection unit 61 of the shot selection / combination unit 60 extracts the number of shots set as parameters in order from the shot with the highest importance score in the consistency score list generated in step S11. The candidate shot list L4 (see FIG. 6) is generated.

  In step S13, the interpolated shot search means 62 refers to the consistency score list generated in step S11, and sets consistency for each candidate shot in descending order of importance score of the candidate shot list generated in step S12. An interpolation shot list L5 (see FIG. 7) is generated by selecting a shot with the highest score as an interpolation shot.

In step S14, the candidate shot update unit 63 scans the interpolation shot list generated in step S13 from the top (in order of importance score).
In step S15, when the candidate shot update unit 63 adds the interpolated shot scanned in step S14 to the candidate shot, the total time of the candidate shot reaches the length (time length) of the summary video specified by the parameter. Determine whether or not.
If the time length of the summary video is not reached even after the interpolation shot is added (Yes in step S15), the shot selecting / combining means 60 advances the operation to step S16. On the other hand, when the interpolation shot is added and the time length of the summary video is reached (No in step S15), the shot selecting / combining means 60 advances the operation to step S18.

In step S16, the candidate shot update unit 63 adds the interpolated shot scanned in step S14 to the candidate shot list, and generates an updated candidate shot list L6 (see FIG. 8).
In step S <b> 17, the candidate shot update unit 63 determines whether the end of the interpolation shot list has been scanned.
When scanning to the end of the interpolation shot list (Yes in step S17), the shot selection / combination means 60 returns to step S13 to generate an interpolation shot list for the new candidate shot list. On the other hand, if the end of the interpolation shot list has not been scanned (No in step S17), the shot selection / combination means 60 returns to step S14, continues scanning the interpolation shot list, and changes to the candidate shot of the interpolation shot. Add.

In step S18, the candidate shot combining unit 64 cuts out the video of the time section corresponding to the candidate shot from the original video based on the candidate shot list that has reached the time length of the summary video.
In step S19, the candidate shot combining means 64 generates a summary video by combining the shot video cut out in step S18 in the time order of the original video.

When the summary video generated in step S19 is displayed on the display device 3 and the user does not change the parameter (Yes in step S20), the summary video generation device 1 ends the operation. On the other hand, when the user changes the parameter (No in step S20), the summary video generating apparatus 1 returns to step S7 (FIG. 9) and performs the parameter changing operation.
With the above operation, the summary video generation apparatus 1 adds a shot having a high degree of continuity to the shot with high importance in the original video to reduce the semantic discontinuity. Can be generated.

<< Second Embodiment >>
Next, with reference to FIG. 11, a summary video generation apparatus 1B according to a second embodiment of the present invention will be described. The summary video generation device 1 described with reference to FIG. 1 has a configuration in which the consistency score is stored in the consistency score storage unit 40 in advance.

The summary video generation device 1B is configured to learn the consistency score from the learning data.
The summary video generation apparatus 1B includes a video analysis unit 10, a parameter setting unit 20, an importance calculation unit 30, a consistency score storage unit 40, a consistency evaluation unit 50, a shot selection / combination unit 60, And a sex score learning means 70. Since the configuration other than the consistency score learning unit 70 is the same as that of the summary video generation device 1 described with reference to FIG. 1, the same reference numerals are given and description thereof is omitted.

  The consistency score learning means 70 learns, from known learning data, an appearance probability that an adjacent shot of the summary video appears in the original video shot as a consistency score for each distance of the scene to which the shot belongs. This learning data is data in which the shot index and scene index in the original video and the shot index in the original video of the summary video generated from the original video are known.

For example, the shot index N _shot and the scene index N _scene in the divided video list L1 (see FIG. 2) generated by operating only the video analysis unit 10 are used for the shot index and the scene index in the original video. As the shot index in the original video of the summary video, the shot index in the original video when the summary video is manually generated from the original video may be used.

This consistency score learning means 70 learns as a probability model how far a shot that is continuous with the shot in the summary video of the learning data appears as a shot of the original video. This probability model, as shown in the first term of the formula (1), in summary video is contiguous set of shots s _i, s _i-d is, preceding shot s _i-d is a known condition in the original, the column _s i-d of shot of the original video, _..., the probability of occurrence from _{s i-1, s i,} shot column prior to the shot _{s i s i-d, ...} , s i-1 is Generally, it can be expressed as a conditional probability (N-gram probability) at a given element. However, in the probability model in the present embodiment, as shown in the third term of the following equation (1), the conditional probability is calculated between the shot s _i in the summary video and the shot s _i-d preceding it. This model is assumed to depend only on the distance d in the original video. Here, the distance d is an integer greater than or equal to “0” and less than or equal to the maximum value of the scene index of the learning data.

The consistency score learning means 70 gives a probability to the distance d by a general smoothing method when there is no shot of a scene separated by a distance d with respect to a certain shot as learning data. . As a smoothing method, for example, a kernel density estimation method that estimates a distribution of true probability values by fitting a uniform kernel function such as a normal distribution to an observation value on which a noise component is superimposed can be applied. .
Thereby, the consistency score learning means 70 can obtain | require the appearance probability (refer FIG. 4) for every difference (distance d) of a scene by learning as a consistency score.
The consistency score learning unit 70 writes and stores the learned consistency score in the consistency score storage unit 40 in association with the scene difference.

By configuring the summary video generation device 1B as described above, the summary video generation device 1B can increase the accuracy of the consistency score and increase the degree of consistency by learning from past learning data. A summary video can be generated.
The operation of the summary video generation apparatus 1B is the same as the operation of the summary video generation apparatus 1 described with reference to FIGS. 9 and 10 after learning of the consistency score in the consistency score learning means 70, and therefore the description will be given. Omitted.

  The summary video generation apparatuses 1 and 1B according to the embodiment of the present invention have been described above. However, the summary video generation apparatuses 1 and 1B operate the summary video generation program for causing the computer to function as each unit described above. Can do.

DESCRIPTION OF SYMBOLS 1,1B Summary image | video production | generation apparatus 10 Image | video analysis means 11 Feature-value extraction means 12 Shot detection means 13 Scene classification means 20 Parameter setting means 30 Importance calculation means 40 Consistency score storage means 50 Consistency evaluation means 60 Shot selection / combination means 61 Candidate shot selection means 62 Interpolated shot search means 63 Candidate shot update means 64 Candidate shot combination means 70 Consistency score learning means

Claims (5)

A summary video generation device that generates a summary video having a video duration shorter than the original video from the original video,
Video analysis means for detecting a shot as a change point of the original video and classifying the series of shots having similar feature quantities as a scene;
Importance calculating means for calculating an importance score that is an index indicating that the shot is characteristic in the original video based on the feature amount of the shot;
Consistency score storage means for storing in advance a consistency score indicating the degree of consistency between shots corresponding to the distance in the time direction of the scene;
Consistency evaluation means for acquiring the consistency score for each shot detected by the video analysis means from the consistency score storage means, and associating the consistency score with the combination of shots;
From the original video, select a preset number of shots with high importance according to the importance score, and with respect to the shot with high importance, shots with a high degree of consistency with the consistency score, Shot selection / combination means for selecting and combining until the time length of the summary video set in advance is reached,
A summary video generation apparatus comprising: From the learning data in which the shot and scene in the original video and the position of the shot in the original video of the summary video generated from the original video are known,
It further comprises a consistency score learning means for learning, for each distance of a scene in the original video, the probability that adjacent shots in the summary video of the learning data co-occur in the original video as the consistency score. The summary video generation apparatus according to claim 1, wherein: The shot selecting / combining means includes:
Candidate shot selection means for selecting a number of shots having a high importance score as a candidate shot of the summary video in a preset number;
Interpolation shot search means for searching for shots having the highest consistency score from all the shots as the interpolation shot within a range not exceeding the time length of the summary video for each candidate shot in descending order of importance score When,
Candidate shot updating means for updating the candidate shot by adding the interpolation shot to the candidate shot;
Shot combining means for cutting out and combining the candidate shots from the original video in time order of the original video;
The summary video generation device according to claim 1, further comprising:   The candidate shot update means updates the candidate shot after the update to the interpolation shot search means when the total time length of the updated candidate shot is less than the preset time length of the summary video. The summary video generation apparatus according to claim 3, wherein a simple interpolation shot is searched.   A summary video generation program for causing a computer to function as the summary video generation device according to any one of claims 1 to 4.

Images