JP5070179B2 - Scene similarity determination device, program thereof, and summary video generation system - Google Patents

Description

  The present invention relates to a technique for determining a similar scene from a material video including overlapping scenes such as dramas and movies.

  In recent years, the amount of material video (material image) produced in a program or movie such as a drama may be 100 times that of a final program. At this time, in order to know what scenes are included in each material video, for example, the creator may play all of the material videos quickly, and this work takes time and effort. Take it. In addition, each material video includes a re-shoot of the same scene over and over, and it is not possible to quickly play back each material video to know if those scenes are the same or different. , It takes a lot of time and effort for the creator. In view of this, several methods have been proposed for determining whether the captured scene is the same or different in the material video.

For example, a technique for determining whether scenes are similar by comparing certain time-series feature data in each shot has been proposed (see, for example, Patent Document 1). In addition, for example, a technique has been proposed in which a target material video is divided into scenes and then simply compared using the first frame (still image) of each scene (see, for example, Non-Patent Document 1). Furthermore, for example, using a plurality of still images extracted at regular intervals from each scene of the target material video, clustering is performed on all the extracted still images using certain image characteristics, and the same cluster There has been proposed a technique for determining that the scene belongs to the same scene (see, for example, Non-Patent Document 2).
JP 2000-339474 A Z. Liu, E .; Zavesky, D.C. Gibbon, B.M. Shahraray, and P.M. Haffner, “AT & T Research at TRECVID 2007” A. Haptmann, M.M. Christel, W.M. Lin, B .; Maher, J .; Yang, R.A. Baron, and G.G. Xiang, “Summarizing BBC Rushes the Information Way”

  However, in the techniques proposed in Patent Document 1 and Non-Patent Document 1, when a comparison is made using a still image such as the beginning of each scene or the middle of each scene, a preparation part for shooting is provided at the beginning or middle of each scene. The accuracy of similarity determination is very low because it is included and there is a time shift even in the same scene. In addition, the technique proposed in Non-Patent Document 2 extracts a plurality of images and uses clustering to supplement the accuracy of the similarity determination. However, this clustering process generally takes a long processing time, and the similarity determination is slow. At this time, the feature of the image to be compared may be time-series data. However, the combination of images to be compared becomes enormous and the time axis expands and contracts, so that the processing time becomes very long. At the same time, the accuracy of similarity determination may not be improved, which is not realistic.

  Therefore, the present invention provides a scene similarity determination device and program thereof that is simple and high-speed and has high accuracy of scene similarity determination, and a summary video generation system that can easily and quickly generate a summary video with extremely few overlapping scenes. The purpose is to provide.

  In order to solve the above-described problem, a scene similarity determination device according to claim 1 determines whether or not two scenes extracted from a material video are similar for a material video including overlapping scenes. In the configuration, the input unit, the scene dividing unit, the still image extracting unit, the still image similarity determining unit, and the scene similarity determining unit are provided.

  According to such a configuration, the scene similarity determination apparatus inputs two scenes by the input unit. In addition, the scene similarity determination device calculates a motion amount between frames constituting the scene by the scene dividing unit, and the scene includes a large motion section in which the motion amount is equal to or greater than a preset threshold value, and the motion amount is less than the threshold value. It divides into the motion subsection which is. Here, in the shooting of dramas, movies, etc., the same scene is often re-shooted many times. In this case, the camera and the actor repeat the same behavior, and thus the movement is the same. That is, since the same motion is included in the large motion section of the overlapping scene, the scene similarity determination device divides the scene into the large motion section and the small motion section in order to determine the similarity of the scene.

  In addition, the scene similarity determination device uses a still image extraction unit for each small motion section of the scene, a predetermined time before the start time of the large motion section located immediately after the small motion section, or an intermediate time of the small motion section, The frame of the small motion section at one time close to the start time of the large motion section is extracted as a still image. Here, since the large motion sections extracted from the two scenes have a slight expansion and contraction in the time direction, the scene similarity determination device uses the still image extraction unit to move less and have the same composition (still image). A still image is extracted from the small motion section immediately before the large motion section.

  In addition, the scene similarity determination device calculates the feature amount of the still image extracted from one of the two scenes by the still image similarity determination means, and extracts from the other of the two scenes. The feature amount of the still image is calculated, and for all combinations of the still image of one scene and the still image of the other scene, the feature amount of the still image of one scene and the feature amount of the still image of the other scene Based on the above, it is determined whether or not the still image of one scene is similar to the still image of the other scene. When the scene similarity determination unit determines that the still image similarity determination unit is similar for one or more combinations of the still image of one scene and the still image of the other scene by the scene similarity determination unit, It is determined that the other scene is similar to the other scene. As described above, the scene similarity determination device only performs similarity determination of one or more sets of still images for scene similarity determination, and does not perform complicated calculations such as clustering or time-series comparison of still images. .

  In the scene similarity determination apparatus according to claim 2, when the still image similarity determination unit determines that any one combination of the still image of one scene and the still image of the other scene is similar, The determination whether or not another combination of the image and the still image of the other scene is similar is completed.

  According to such a configuration, when the scene similarity determination device determines that any one combination of the still image of one scene and the still image of the other scene is similar, the scene similarity determination ends.

  In the scene similarity determination apparatus according to claim 3, the still image similarity determination unit uses the still image feature of one scene and the still image feature of the other scene as the feature amount of the still image of the other scene. The image is divided so that the corresponding still images correspond to each other, and the Euclidean distance of the average value of a predetermined color space between the still image of one scene and the still image of the other scene is calculated, and the Euclidean distance is preset. When the distance is less than the distance, it is determined that the still image of one scene is similar to the still image of the other scene, and when the Euclidean distance exceeds a preset distance, the still image of one scene and the still image of the other scene It is determined that the image is not similar.

  According to this configuration, the scene similarity determination device can perform still image similarity determination with a simple calculation.

  In order to solve the above-described problem, a scene similarity determination program according to claim 4 is provided for determining whether two scenes extracted from a material video are similar for a material video including overlapping scenes. The computer is configured to function as an input unit, a scene division unit, a still image extraction unit, a still image similarity determination unit, and a scene similarity determination unit.

  According to such a configuration, the scene similarity determination program inputs two scenes by the input unit. In addition, the scene similarity determination program calculates the amount of motion between frames constituting the scene by the scene dividing unit, and the scene is divided into a large motion section in which the motion amount is equal to or greater than a preset threshold value, and the motion amount is less than the threshold value. It divides into the motion subsection which is. Here, in the production of programs such as dramas, the same scene is often re-photographed many times. In this case, the camera and the actor repeat the same behavior, and thus the movement is the same. That is, since the same motion is included in the large motion section of the overlapping scene, the scene similarity determination program divides the scene into the large motion section and the small motion section in order to determine the similarity of the scene.

  In addition, the scene similarity determination program uses a still image extraction unit for each small motion section of the scene, a certain time before the start time of the large motion section located immediately after the small motion section, or an intermediate time of the small motion section, The frame of the small motion section at one time close to the start time of the large motion section is extracted as a still image. Here, since the large motion sections extracted from the two scenes have a slight expansion / contraction (deviation) in the time direction, the scene similarity determination program uses the still image extraction means to move little and the same composition (still A still image is extracted from the small motion section immediately before the large motion section.

  In addition, the scene similarity determination program calculates the feature amount of the still image extracted from one of the two scenes by the still image similarity determination means, and extracts from the other of the two scenes. The feature amount of the still image is calculated, and for all combinations of the still image of one scene and the still image of the other scene, the feature amount of the still image of one scene and the feature amount of the still image of the other scene Based on the above, it is determined whether or not the still image of one scene is similar to the still image of the other scene. When the scene similarity determination unit determines that the still image similarity determination unit is similar for one or more combinations of the still image of one scene and the still image of the other scene by the scene similarity determination unit, It is determined that the other scene is similar to the other scene. As described above, the scene similarity determination program only performs similarity determination of one or more sets of still images for scene similarity determination, and does not perform complicated operations such as clustering and comparison of still images in time series. .

  In order to solve the above-described problem, the summary video generation system according to claim 5 generates a summary video that generates a summary video composed of different scenes excluding a duplicate scene from a material video including a duplicate scene. The system includes a scene extraction unit, a scene similarity determination device according to claim 1, a representative scene selection unit, and a summary video generation unit.

  According to this configuration, the summary video generation system detects a scene change point by the scene extraction unit and extracts a plurality of scenes included in the material video. In the summary video generation system, the scene similarity determination device determines whether the combination of scenes extracted by the scene extraction unit is similar. Here, the summary video generation system can perform a simple, high-speed and high-accuracy scene similarity determination using a scene similarity determination device.

  The summary video generation system also includes a combination of a scene determined by the representative scene selection unit that the scene similarity determination device determines that the scene similarity determination device is similar to a scene that the scene similarity determination device determines that the scene similarity determination device is similar. Is selected as the representative scene. Here, the summary video generation system includes, in the representative scene, the longest scene among the combinations of scenes determined by the scene similarity determination device to be similar by the representative scene selection unit, for example, re-shooting a drama, a movie, or the like. The possibility that the scene is not included in the summary video can be minimized. Further, the summary video generation system generates a summary video by connecting a part or all of the representative scene selected by the representative scene selection unit by the summary video generation unit.

According to the present invention, the following excellent effects can be obtained.
According to the first and fourth aspects of the invention, since the still image extracted from the small motion section immediately before the large motion section having a low motion and high probability of the same composition is used, the accuracy of the scene similarity determination is increased. can do. In addition, according to the inventions according to claims 1 and 4, since only the similarity determination of one or more sets of still images is performed, and complicated calculations such as clustering and comparison of still images in time series are not performed. Scene similarity determination can be performed at high speed.

According to the second aspect of the present invention, when it is determined that any one combination of the still image of one scene and the still image of the other scene is similar, the similarity determination of the still image is terminated to end the similarity determination of the still image. The determination can be performed at a higher speed.
According to the third aspect of the present invention, since the still image similarity determination can be performed with a simple calculation, the scene similarity determination can be performed at higher speed.

  According to the fifth aspect of the invention, it is possible to perform a simple, high-speed and high-accuracy scene similarity determination, and to minimize the possibility that a re-captured scene or the like is not included in the summary video. It is possible to generate a summary video that is simple and high speed with very few overlapping scenes.

  Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function and the same member are denoted by the same reference numerals, and description thereof is omitted.

(First embodiment)
[Contents of material video]
First, before explaining the scene similarity determination apparatus according to the first embodiment of the present invention, the contents of the material video in the present invention will be described with reference to FIG. FIG. 1 is a diagram for explaining the contents of a material video in the present invention.

  The material video is, for example, a video including duplicate scenes because the same scene is shot twice or more in shooting a drama, a movie, or the like. In the example of FIG. 1, the material image is composed of four shots St1 to St4. Here, the shot St1 includes the color bar C. The shot St2 includes the junk video J at the head of the shot St2, the scene Sc1-T1 obtained by shooting the scene 1 for the first time, the junk video J between the scene Sc1-T1 and the scene Sc1-T2, It includes a scene Sc1-T2 in which the scene 1 is re-photographed for the second time, and a junk video J at the end of the shot St2. This junk image J is not used as a drama or movie image, and is, for example, an image taken by an actor or staff who is preparing. Further, the junk video J includes a video in which a clapperboard is photographed (not shown).

  The shot St3 includes a blank video B that does not include any video signal. Further, the shot St4 includes a junk video J at the beginning of the shot St4, a scene Sc2-T1 obtained by shooting the scene 2 for the first time, and a junk video J at the end of the shot St4. That is, in the example of FIG. 1, the material video includes the scenes Sc1-T1 and the scenes Sc1-T2 as overlapping scenes. The material video is not limited to a drama or movie, but may be any video that is shot according to the shooting rules described later.

[Configuration of scene similarity determination device]
In the scene similarity determination apparatus according to the first embodiment of the present invention, two scenes extracted in advance from a material video (for example, the scene Sc1-T1 and the scene Sc1-T2 extracted from the shot St2 in FIG. 1) are similar to each other. Whether or not. Hereinafter, the configuration of the scene similarity determination apparatus according to the first embodiment of the present invention will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the scene similarity determination apparatus according to the first embodiment of the present invention. As shown in FIG. 2, the scene similarity determination apparatus 1 includes an input unit 10, a scene division unit 11, a still image extraction unit 12, a still image similarity determination unit 13, a scene similarity determination unit 14, and a parameter storage unit. 15.

  The input means 10 is an interface such as a port or a terminal for inputting two scenes previously extracted from the material video. The first scene corresponds to one of the two scenes, and the second scene corresponds to the other of the two scenes.

  The scene dividing unit 11 calculates a motion amount between frames constituting the first scene input to the input unit 10, and the first scene is divided into a motion large section in which the motion amount is equal to or greater than a preset threshold, The amount is divided into small motion sections whose amount is less than the threshold value. Here, the scene dividing unit 11 calculates, for example, the sum of motion vectors between frames constituting the first scene, and divides a section where the sum is equal to or greater than a preset threshold as a large motion section. In addition, the scene dividing unit 11 divides, for example, a section in which the sum is less than the threshold value as a small motion section. Note that the above-described threshold value may be set in advance in parameter information described later, and the scene dividing unit 11 may acquire this threshold value with reference to the parameter information.

  Further, the scene dividing unit 11 may divide a frame before and after the first scene into predetermined blocks and calculate a pixel value difference for each block of the preceding and following frames. Then, the scene dividing unit 11 divides a section where the pixel value difference is equal to or larger than a difference set in advance in the parameter information as a large motion section, and divides a section where the difference is less than the above difference as a small motion section. Also good. The scene dividing means 11 divides the second scene into the large motion section and the small motion section by the same method as that for the first scene.

  The still image extracting means 12 is, for each small motion section of the first scene divided by the scene dividing means 11, for a certain time before the start time of the large motion section located immediately after the small motion section or at an intermediate time of the small motion section. Among them, the frame of the small motion section at one time close to the start time of the large motion section is extracted as a still image. Further, the still image extracting means 12 extracts a still image from the second scene as well as the first scene. Note that the predetermined time (for example, 5 seconds) described above may be set in advance in the parameter information, and the still image extracting unit 12 may acquire the predetermined time with reference to the parameter information.

  The still image similarity determination unit 13 calculates the feature amount of the still image extracted from the first scene by the still image extraction unit 12, and calculates the feature amount of the still image extracted from the second scene by the still image extraction unit 12. Is. Then, the still image similarity determination unit 13 determines the feature amount of the still image of the first scene and the feature amount of the still image of the second scene for all combinations of the still image of the first scene and the still image of the second scene. Based on the above, it is determined whether or not the still image of the first scene is similar to the still image of the second scene. Here, the still image similarity determination unit 13 outputs a signal indicating that the number of still images in the first scene still image and the second scene still image are similar to the scene similarity determination unit 14. Also good. Details of still image similarity determination by the still image similarity determination means 13 will be described later.

  When the scene similarity determination unit 14 determines that the still image similarity determination unit 13 is similar for one or more combinations (number of sets) of the still image of the first scene and the still image of the second scene, the first scene And the second scene are determined to be similar. Note that, by setting the number of sets in the parameter information in advance, the scene similarity determination unit 14 can obtain the number of sets by referring to the parameter information. For example, when the number of sets is set to “1” in the parameter information, when the signal from the still image similarity determination unit 13 indicates that one or more sets of still images are similar, the scene similarity determination unit 14 selects the first scene. And the second scene are determined to be similar. For example, when the number of sets is set to “2” in the parameter information, when the signal from the still image similarity determination unit 13 indicates that two or more sets of still images are similar, the scene similarity determination unit 14 It is determined that the first scene is similar to the second scene. In other words, the scene similarity determination unit 14 increases the accuracy of the scene similarity determination as the number of sets increases, and the scene similarity determination increases as the number of sets decreases. become.

  In addition, the scene similarity determination unit 14 may output a signal indicating a determination result that the scene is similar or the scene is not similar in accordance with the above determination. Further, the scene similarity determination unit 14 may output the determination result to a display unit such as a liquid crystal screen (not shown).

  The parameter storage unit 15 is a storage unit such as a ROM (Read Only Memory) or a HDD (Hard Disk Drive) that stores the parameter information. This parameter information sets various parameters necessary for the operation of the scene similarity determination apparatus 1.

<Still image extraction and still image similarity determination>
The details of still image extraction by the still image extraction unit 12 and still image similarity determination by the still image similarity determination unit 13 will be described with reference to FIG. 3 (see FIG. 2 as appropriate). FIG. 3A is a diagram for explaining still image extraction by the still image extraction unit 12 in FIG. 2, and FIG. 3B is a diagram for explaining still image similarity determination by the still image similarity determination unit 13. It is. In FIG. 3A, the large motion section of the first scene and the second scene is illustrated by hatching, and the intermediate time of the small motion section is illustrated by a broken line. In FIG. 3A, t indicates the passage of time.

  In the example of FIG. 3A, the scene dividing unit 11 divides the first scene into three small motion sections a, c, e and two large motion sections b, d. The scene dividing means 11 divides the second scene into two small motion sections f and h and two large motion sections g and i.

  First, the still image extraction means 12 extracts a still image from the motion subsection a of the first scene. Specifically, the still image extracting unit 12 determines which one of the intermediate time of the small motion section a and the time a certain time before the start time of the large motion section b located immediately after the small motion section a. It is determined whether it is close to the start time of the section b. Here, since the intermediate time of the small motion section a is close to the start time of the large motion section b, the still image extraction unit 12 extracts the frame at the intermediate time of the small motion section a as the still image 1-1.

  Next, the still image extracting means 12 extracts a still image from the motion subsection c of the first scene. Specifically, the still image extraction unit 12 determines which of the intermediate time of the small motion section c and the time before a certain time from the start time of the large motion section d located immediately after the small motion section c. It is determined whether it is close to the start time of the section d. Here, since the time before a certain time before the start time of the large motion section d located immediately after the small motion section c is close to the start time of the large motion section d, the still image extracting means 12 A frame at a certain time before the start time of the large motion section d located immediately after is extracted as a still image 1-2.

  Further, the still image extracting means 12 extracts still images 2-1 and 2-2 from the small motion sections f and h of the second scene, respectively, as in the first scene. As shown in FIG. 3A, the motion sub-section e of the first scene is the last motion of the first scene and there is no large motion section immediately after the first scene. It is not necessary to extract a still image from e.

  In general, material images such as dramas and movies often re-shoot the same scene many times. In this case, the actors behave the same, and before the start of shooting, all the actors stand still. The filming is performed according to a specific shooting rule such that the performance is started with a voice of “action” or “start”. In other words, due to the nature of this shooting rule, it is considered that the same motion is included in a large motion section of the same scene, and the probability of the same composition (still image) immediately before the large motion section (predetermined time) increases. It is done. Therefore, the scene similarity determination device 1 uses the characteristics of the shooting rule to divide the scene into a large motion section and a small motion section by the scene dividing means 11 and the still image extracting means 12 as described above. The method extracts two still images with a high probability of having the same composition (still image).

  When the scene input to the input unit is in the MPEG (Moving Picture Experts Group) 2 format, the still image extraction unit 12 uses the time code corresponding to the first frame of the large motion section as the start time of the large motion section. good. Still image extraction means 12 may calculate a time that is intermediate between the time codes corresponding to the first and last frames of the small motion section, and may use this as the intermediate time.

  As shown in FIG. 3B, the still image similarity determination unit 13 performs all of the still images 1-1 and 1-2 of the first scene and the still images 2-1 and 2-2 of the second scene. It is determined whether the combinations are similar. That is, the still image similarity determination means 13 is a combination of the still image 1-1 and the still image 2-1, a combination of the still image 1-1 and the still image 2-2, a still image 1-2 and a still image 2- 1 and the combination of the still image 1-2 and the still image 2-2 are determined to be similar to each other.

  Specifically, the still image similarity determination unit 13 may correspond to, for example, each of the still images 1-1 and 1-2 of the first scene and the still images 2-1 and 2-2 of the second scene. Divide into 4 by 2 × 2 Then, the still image similarity determination unit 13 uses the still images 1-1 and 1-2 of the first scene divided into four as the feature amount of the still image of the first scene and the feature amount of the still image of the second scene and the second image. The Euclidean distance of the average value of the HSB (Hue Saturation Brightness) color space with the still images 2-1 and 2-2 of the scene is calculated. The still image similarity determination unit 13 has been described as dividing the still image of the first scene and the still image of the second scene into four, but the number of division is not limited as long as it is two or more. Further, the still image similarity determination unit 13 has been described as using the HSB color space. However, the present invention is not limited to this. For example, an arbitrary color space such as an RGB (Red Green Blue) color space or an L * a * b color space. May be used.

  Then, the still image similarity determination means 13 determines that the still image of the first scene 1-1 and the still image of the second scene 2-1 are similar when the Euclidean distance is equal to or greater than a preset distance. On the other hand, when the Euclidean distance is less than the above-described distance, the still image similarity determination unit 13 determines that the still image of the first scene 1-1 and the still image of the second scene 2-1 are not similar. As described above, the still image similarity determination unit 13 determines that the Euclidean distance is the distance described above for the remaining combinations of the still image 1-2 of the first scene and the still images 2-1 and 2-2 of the second scene. It is determined whether or not:

  Here, when the still image similarity determination unit 13 determines that any one combination of the still image of the first scene and the still image of the second scene is similar, the still image of the first scene and the still image of the second scene The determination as to whether another combination with the image is similar may be terminated. Accordingly, the still image similarity determination unit 13 can perform still image similarity determination at higher speed. The parameter information includes information indicating the number of still images to be divided, information indicating which of the HSB color space, the RGB color space, and the L * a * b color space is used, information indicating the distance, and still image Information indicating whether to end the similarity determination when one set of images is similar may be set in advance. In this case, the still image similarity determination unit 13 may control which color space to use or whether to end the similarity determination with reference to the parameter information.

  In FIG. 3, the still image similarity determination unit 13 has been described using an example in which a color space is used as the feature amount of the still image of the first scene and the feature amount of the still image of the second scene. The feature amount of the image and the feature amount of the still image of the second scene are not limited to this. For example, the still image similarity determination unit 13 uses a spatial frequency such as an edge histogram or a DCT (discrete cosine transform) coefficient as the feature amount of the still image of the first scene and the feature amount of the still image of the second scene. it can. Specifically, the still image similarity determination unit 13 calculates an edge histogram or a DCT coefficient for each of the still image feature amount of the first scene and the still image of the second scene. Then, the still image similarity determination means 13 compares the calculated edge histogram or DCT coefficient for each of the still image feature amount of the first scene and the still image of the second scene, and these are within a preset range. The still image similarity determination can be performed depending on whether or not there is. As described above, even if the edge histogram or the DCT coefficient is used as the feature value of the still image of the first scene and the feature value of the still image of the second scene, the still image similarity determination unit 13 can easily and quickly Similarity determination can be performed.

  In FIG. 3, an example in which the same number of still images (for example, two) are extracted in the first scene and the second scene has been described. However, different numbers of still images are extracted in the first scene and the second scene. May be extracted. Also in this case, as in FIG. 3, the still image similarity determination unit 13 determines whether the combination of all the still images of the first scene and the still images of the second scene is similar. For example, when three still images are extracted from the first scene and two still images are extracted from the second scene, the still image similarity determination unit 13 determines that the still image of the first scene and the still image of the second scene A total of 6 combinations of 3 sheets × 2 sheets are determined, and it is determined whether or not each combination is similar.

[Operation of scene similarity determination device]
Hereinafter, the operation of the scene similarity determination apparatus in FIG. 2 will be described with reference to FIG. 4 (see FIG. 2 as appropriate). FIG. 4 is a flowchart showing the operation of the scene similarity determination apparatus of FIG.

  First, the scene similarity determination apparatus 1 inputs two scenes (first scene and second scene) extracted in advance from a material video by the input means 10 (step S1).

  Following the processing of step S1, the scene similarity determination apparatus 1 calculates the amount of motion between frames constituting the first scene input to the input unit 10 by the scene dividing unit 11, and moves the first scene to the motion. The movement is divided into a large motion section whose amount is equal to or greater than a preset threshold value and a small motion section whose motion amount is less than the threshold value (step S2).

  Subsequent to the processing of step S2, the scene similarity determination apparatus 1 uses the still image extraction unit 12 for each small motion section of the first scene divided by the scene dividing unit 11, and then the large motion section located immediately after the small motion section. The frame of the small motion section at one time close to the start time of the large motion section among the intermediate time of the small motion section or a certain time before the start time is extracted as a still image (step S3).

  Subsequent to the process of step S3, the scene similarity determination device 1 calculates the amount of motion between frames constituting the second scene input to the input unit 10 by the scene dividing unit 11, and moves the second scene to the motion. The movement is divided into a large motion section whose amount is equal to or larger than a preset threshold value and a small motion section whose motion amount is less than the threshold value (step S4).

  Following the processing in step S4, the scene similarity determination apparatus 1 uses the still image extraction unit 12 to move the large motion section located immediately after the small motion section for each small motion section of the second scene divided by the scene dividing unit 11. The frame of the small motion section at one time close to the start time of the large motion section among the intermediate time of the small motion section or a certain time before the start time is extracted as a still image (step S5).

  Following the process of step S5, the scene similarity determination apparatus 1 calculates the feature quantity of the still image of the first scene extracted by the process of step S3 by the still image similarity determination unit 13, and extracts it by the process of step S5. The feature amount of the still image of the second scene is calculated (step S6).

  Following the processing of step S6, the scene similarity determination apparatus 1 uses the still image similarity determination unit 13 to determine the first scene still image for all combinations of the first scene still image and the second scene still image. Based on the feature amount and the feature amount of the still image of the second scene, it is determined whether or not the still image of the first scene and the still image of the second scene are similar (step S7).

  When the still image similarity determination unit 13 determines that any one of the still image of the first scene and the still image of the second scene is similar (step S7: present), the scene similarity determination apparatus 1 The determination unit 14 determines that the first scene and the second scene are similar (step S8). Then, the scene similarity determination device 1 outputs a determination result that the scenes are similar by the scene similarity determination unit 14 according to this determination.

  On the other hand, when the still image similarity determination unit 13 determines that all the combinations of the still image of the first scene and the still image of the second scene are not similar (step S7: None), the scene similarity determination apparatus 1 The similarity determination unit 14 determines that the first scene and the second scene are not similar (step S9). Then, the scene similarity determination device 1 outputs a determination result that the scenes are not similar by the scene similarity determination unit 14 in accordance with this determination.

  As described above, the scene similarity determination apparatus 1 can increase the accuracy of scene similarity determination and can perform scene similarity determination easily and at high speed. In each embodiment, the scene similarity determination apparatus according to the present invention has been described as an independent apparatus. However, in the present invention, hardware resources such as general computer calculation means and storage means are coordinated as the above-described means. It may be a program to be operated. This program may be distributed via a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

(Second Embodiment)
[Configuration of Summary Video Generation System]
Hereinafter, the configuration of a summary video generation system according to the second embodiment of the present invention will be described with reference to FIG. FIG. 5 is a block diagram showing a configuration of a summary video generation system according to the second embodiment of the present invention. As shown in FIG. 5, the summary video generation system 2 generates a summary video composed of different scenes excluding the duplicate scene from the material video including the duplicate scene. 2, the scene similarity determination device 1, the representative scene selection unit 22, the unnecessary video deletion unit 23, and the summary video generation unit 24.

  The scene extraction means 21 inputs a material video and detects a scene change point of the material video by a known method to extract a plurality of scenes included in the material video. For example, the scene extraction unit 21 divides each frame of the material video into a predetermined number of blocks (for example, 4 × 4 16 blocks), calculates a color change amount for each block, and the color change amount is constant. A frame that is equal to or greater than the value is detected as a scene change point (see, for example, JP-A-2006-270301).

  The scene extraction means 21 may detect a skin color area included in the material video as a person and detect a scene change point based on a change in the motion vector of the person (skin color area). Further, when audio information is added to the material video, the scene extraction unit 21 may detect a part where there is no voice part for a certain period of time or a part where the audio has changed suddenly as a scene change point.

  Here, the scene extraction means 21 combines the two scenes in order from the first scene of the material video and outputs the combined scenes to the scene similarity determination apparatus 1. For example, when five scenes are extracted from the material video, the scene extraction unit 21 combines the scene 1 and the scene 2, the combination of the scene 2 and the scene 3, the combination of the scene 3 and the scene 4, and the scene. The combination of 4 and scene 5 is output to the scene similarity determination device 1. The scene extraction unit 21 combines all five scenes and outputs the combined scenes to the scene similarity determination apparatus 1. The scene similarity determination apparatus 1 determines whether all the combinations of the five scenes are similar. Also good.

  As described above, the scene similarity determination apparatus 1 determines whether two scenes input from the scene extraction unit 21 are similar. The scene similarity determination device 1 is the same as that shown in FIG.

  The representative scene selection means 22 is the most of the combinations of the scenes determined by the scene similarity determination device 1 not to be similar to the scenes determined by the scene similarity determination device to be similar among all the scenes extracted by the scene extraction means 21. A long scene is selected as a representative scene.

  Hereinafter, selection of a representative scene by the representative scene selection unit 22 of FIG. 5 will be described with reference to FIG. 6 (see FIG. 5 as appropriate). FIG. 6 is a diagram for explaining selection of a representative scene by the representative scene selection means 22 of FIG. In FIG. 6, a longer scene is illustrated as a rectangle that is longer in the left-right direction.

  In the example of FIG. 6, the scene similarity determination device 1 determines that the combination of the scene 1 and the scene 2 is similar, determines that the combination of the scene 2 and the scene 3 is not similar, Are determined not to be similar, and the combination of scene 4 and scene 5 is determined to be similar.

  First, since the combination of scene 1 and scene 2 is similar, representative scene selection means 22 compares the lengths of scene 1 and scene 2 and selects scene 1 longer than scene 2 as the first representative scene. To do. The representative scene selection unit 22 selects the scene 3 as the second representative scene because the combination of the scene 2 and the scene 3 is not similar and the combination of the scene 3 and the scene 4 is not similar. Furthermore, since the combination of the scene 4 and the scene 5 is similar, the representative scene selection unit 22 compares the lengths of the scene 4 and the scene 5 and selects the scene 5 as the third representative scene. Note that the representative scene selection unit 22 can obtain the length of each scene by, for example, calculating the difference in time codes from the beginning to the end of each scene.

  Note that, when it is determined that three or more scenes are similar, the representative scene selection unit 22 may select the longest scene among the combinations of these scenes as the representative scene. For example, in FIG. 6, a case will be described as an example where the combination of scene 1 and scene 2 is determined to be similar and the combination of scene 2 and scene 3 is determined to be similar. At this time, since the combination of scene 1 and scene 3 is also considered to be similar, the representative scene selection means 22 has the longest scene from scene 1 to scene 3, for example, the length of scene 1 is 10 seconds, When the length of 2 is 10 seconds and the length of the scene 3 is 17 seconds, the longest scene 3 is selected as a representative scene.

  Hereinafter, returning to FIG. The unnecessary video deletion unit 23 detects and deletes an unnecessary video included in the representative scene selected by the representative scene selection unit 22 by a known method. This unnecessary video is a video that is not subject to viewing by the viewer or a video that is not subject to editing by the producer among the videos included in each scene. For example, the color bar C, the blank video B, and the like shown in FIG. Junk video J. For example, the unnecessary video deletion unit 23 calculates the ratio of the primary color pixels for each frame constituting the representative scene, and detects that the frame is a color bar if the ratio is equal to or greater than a preset threshold. , Delete that frame. For example, the unnecessary video deletion unit 23 detects a frame that does not include a video signal as a blank video, and deletes the frame.

  In production of dramas and the like, each scene starting with “action” and ending with “cut” is often longer than a certain length (for example, 10 seconds). If the length is less than a certain length, the representative scene itself may be deleted. Further, the unnecessary video deleting means 23 registers in advance a voice that can identify the start and end of the scene such as “action” and “cut” of the director who produces the drama and the like, and “action” included in the representative scene. And “cut” is recognized by voice. Then, the unnecessary video deleting unit 23 may delete the video before detecting “action” and the video after detecting “cut” as unnecessary video in each representative scene. As described above, since the summary video generation system 2 deletes the unnecessary video included in the representative scene selected by the representative scene selection unit 22 by the unnecessary video deletion unit 23, the summary video can be easily grasped. Can be generated.

  The summary video generation unit 24 generates a summary video by connecting a part of the representative scene from which the unnecessary video deletion unit 23 deleted the unnecessary video. Here, the summary video generation means 24 may display the generated summary video on a display device (not shown) or record it on a recording means (not shown).

  For example, the summary video generation unit 24 calculates a motion vector between frames of each representative scene for all the representative scenes from which the unnecessary video deletion unit 23 deleted the unnecessary video, and the motion vector amount is equal to or greater than a preset threshold value. A summary video is generated by connecting the frames. Further, the summary video generation unit 24 may generate a summary video by connecting videos of a predetermined time (for example, 3 seconds from the top) from the top of each representative scene. Further, the summary video generation means 24 may generate a summary video by linking videos of portions where the volume change of each representative scene is sharp (video of a climax portion). Needless to say, the summary video generation unit 24 may generate a summary video by connecting all the representative scenes from which the unnecessary video deletion unit 23 deleted the unnecessary video.

[Operation of summary video generation system]
Hereinafter, the operation of the summary video generation system of FIG. 5 will be described with reference to FIG. 7 (see FIG. 5 as appropriate). FIG. 7 is a flowchart showing the operation of the summary video generation system of FIG.

  First, the summary video generation system 2 inputs the material video by the scene extraction means 21 and detects scene change points of the material video by a known method to extract all scenes included in the material video ( Step S11). Following the processing of step S11, the summary video generation system 2 determines whether the two scenes input from the scene extraction unit 21 are similar by the scene similarity determination device 1 (step S12).

  Following the processing of step S12, the summary video generation system 2 uses the representative scene selection unit 22 to determine a scene that the scene similarity determination device 1 has determined to be dissimilar among the plurality of scenes extracted by the scene extraction unit 21. The longest scene among the combinations of scenes determined to be similar by the similarity determination device is selected as a representative scene (step S13).

  Following the processing of step S13, the summary video generation system 2 deletes the unnecessary video from the representative scene selected by the representative scene selection unit 22 by the unnecessary video deletion unit 23 by a known method (step S14). Following the processing of step S13, the summary video generation system 2 generates a summary video by connecting a part of the representative scene from which the unnecessary video deletion unit 23 deleted the unnecessary video by the summary video generation unit 24 (step S15). ).

  As described above, the summary video generation system can easily generate a summary video from which duplicate scenes are deleted from material videos that are frequently re-taken. By referring to this summary video, the creator can quickly understand the key points of the material video, leading to a significant reduction in time and labor.

  The summary video generation system according to the present invention is not limited to the production of dramas and movies. Currently, home camera devices such as digital cameras and digital videos, and home video recorders that record a large amount of material footage taken with these home camera devices have become widespread in ordinary households, and anyone can easily record images. Can generate content. For this reason, it is considered that there is a demand for easy video editing even in ordinary households. At this time, it is considered essential to select a video required by the user from a large amount of material video, and if the summary video generation system according to the present invention is applied, the user does not need to see all the material video, Users can work efficiently. That is, the summary video generation system according to the present invention can have a function of providing a summary video to a user by being incorporated in a home photographing device or a home recording device.

  In this case, since it is highly possible that the video shot in a general home does not include audio such as “action” or “clapperboard” video, the summary video generation system is incorporated into a home video recording device or home video recording device. The summary video generation system according to the present invention may be configured not to include the above-described unnecessary video deletion means.

It is a figure explaining the content of the material image | video in this invention. It is a block diagram which shows the structure of the scene similarity determination apparatus which concerns on 1st Embodiment of this invention. (A) is a figure explaining the extraction of the still picture by the still picture extraction means 12 of FIG. 2, (b) is a figure explaining the similarity determination of the still picture by the still picture similarity judgment means 13. It is a flowchart which shows operation | movement of the scene similarity determination apparatus of FIG. It is a block diagram which shows the structure of the summary video generation system which concerns on 2nd Embodiment of this invention. It is a figure explaining selection of the representative scene by the representative scene selection means 22 of FIG. It is a flowchart which shows operation | movement of the summary video generation system of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Scene similarity determination apparatus 10 Input means 11 Scene division means 12 Still image extraction means 13 Still image similarity determination means 14 Scene similarity determination means 15 Parameter storage means 2 Summary video generation system 21 Scene extraction means 22 Representative scene selection means 23 Unnecessary video deletion 23 Means 24 Summary video generation means

Claims (5)

In a scene similarity determination device that determines whether or not two scenes extracted from the material video are similar for a material video including an overlapping scene,
Input means for inputting the two scenes;
The amount of motion between frames constituting the scene is calculated, and the scene is divided into a large motion section in which the motion amount is greater than or equal to a preset threshold and a small motion section in which the motion amount is less than the threshold. Scene dividing means to perform,
For each small motion section of the scene, a certain time before the start time of the large motion section located immediately after the small motion section or an intermediate time of the small motion section close to the start time of the large motion section Still image extraction means for extracting the frame of the motion subsection at the time of as a still image;
Calculating the feature amount of the still image extracted from one of the two scenes, calculating the feature amount of the still image extracted from the other of the two scenes, and For all combinations of still images of one scene and still images of the other scene, the one of the scenes based on the feature of the still image of the one scene and the feature of the still image of the other scene A still image similarity determination means for determining whether a still image of a scene is similar to a still image of the other scene;
When the still image similarity determination unit determines that the one or more combinations of the still image of the one scene and the still image of the other scene are similar, the one scene and the other scene are similar Then, a scene similarity determination means for determining,
A scene similarity determination apparatus comprising:   When the still image similarity determination unit determines that any one combination of the still image of the one scene and the still image of the other scene is similar, the still image of the one scene and the other scene The scene similarity determination device according to claim 1, wherein the determination as to whether another combination with the still image is similar is terminated.   The still image similarity determining means corresponds to the still image of the one scene and the still image of the other scene as the feature amount of the still image of the one scene and the feature amount of the still image of the other scene. And calculating an Euclidean distance of an average value of a predetermined color space between the still image of the one scene and the still image of the other scene, and the Euclidean distance is equal to or less than a preset distance. When the still image of the one scene and the still image of the other scene are determined to be similar, and the Euclidean distance exceeds the distance, the still image of the one scene and the still image of the other scene The scene similarity determination apparatus according to claim 1, wherein the scene similarity determination device is determined not to be similar to each other. In order to determine whether or not two scenes extracted from the material video are similar for a material video including overlapping scenes,
Input means for inputting the two scenes;
The amount of motion between frames constituting the scene is calculated, and the scene is divided into a large motion section in which the motion amount is greater than or equal to a preset threshold and a small motion section in which the motion amount is less than the threshold. Scene dividing means to perform,
For each small motion section of the scene, a certain time before the start time of the large motion section located immediately after the small motion section or an intermediate time of the small motion section close to the start time of the large motion section A still image extracting means for extracting the frame of the motion subsection at the time of
Calculating the feature amount of the still image extracted from one of the two scenes, calculating the feature amount of the still image extracted from the other of the two scenes, and For all combinations of still images of one scene and still images of the other scene, the one of the scenes based on the feature of the still image of the one scene and the feature of the still image of the other scene A still image similarity determining means for determining whether a still image of a scene is similar to a still image of the other scene;
When the still image similarity determination unit determines that the one or more combinations of the still image of the one scene and the still image of the other scene are similar, the one scene and the other scene are similar Then, a scene similarity determination means for determining,
A scene similarity determination program characterized by functioning as In a summary video generation system for generating a summary video composed of different scenes excluding the duplicate scene from a material video including the duplicate scene,
A scene extracting means for detecting a scene change point and extracting a plurality of the scenes included in the material video;
The scene similarity determination device according to claim 1, wherein the scene similarity determination device determines whether the combinations of the scenes extracted by the scene extraction unit are similar.
Of the scenes extracted by the scene extraction unit, the scene determined by the scene similarity determination device and the longest scene among the combinations of scenes determined by the scene similarity determination device to be similar are selected as representative scenes. Representative scene selection means;
Summary video generation means for generating a summary video by connecting a part or all of the representative scene selected by the representative scene selection means;
A summary video generation system comprising:

Images