JP2004336556A - Method, device, and program for editing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image editor which automatically divides inputted images on the basis of the feature quantities of scenes of the images. <P>SOLUTION: Scene changes being timings when scenes of images of inputted image data change are detected to divide the image data into a plurality of scenes. The feature quantities of divided scenes are extracted, and a plurality of scene groups are generated by classifying respective scenes into groups in accordance with the extracted feature quantities, and corresponding information for specifying each scene group is made to correspond to the scene group. An array of corresponding information appearing in the same order a plurality of times on the time base is specified as a scene group. Scene groups are specified by matching between arrays of corresponding information, which are not specified, and scene groups. Information resulting from classifying the image data into scene groups is stored. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a video editing device, a video editing method, and a video editing program for dividing video data into video scenes.
[0002]
[Prior art]
In recent years, moving images have been handled by personal computers and the like, and it has become important how to efficiently perform operations such as searching and editing of moving images. When searching or editing a moving image, it is necessary to decompose the moving image into scenes that constitute the moving image, and to grasp the arrangement and configuration of each scene in the moving image. Here, a scene means a unit at which a camera switches in a moving image, a unit at which a sound (for example, a speaker) switches, and the like.
[0003]
As an example of a video editing system that edits a video by using video data that is a signal obtained by encoding a video when editing a video that is a moving image, an example of a conventional video editing system described in Patent Document 1 is described. A configuration example is shown in the block diagram of FIG. In the conventional video editing system having the configuration shown in FIG. 12, a coding parameter extracting unit 121 extracts a coding parameter from coded video data, and a scene change frame detecting unit 122 generates a video based on the coding parameter. A scene change frame, which is a frame at which the scene is switched, is detected. Then, the scene group determining unit 123 divides the scene based on the position of the scene change frame, specifies a scene group that is a group of scenes, and generates scene group information that is information on the position of the first frame in the scene group. The scene information hierarchization unit 124 hierarchizes the information of the position of the scene change frame and the scene group information over the entire video and accumulates them in the accumulation unit 125.
[0004]
Here, the scene group determination unit 123 calculates a time difference between scene change frames, which is a difference between positions of scene change frames of adjacent scenes. Then, comparing the calculated time difference between the frames and a predetermined threshold value, if the time difference between the scene change frames is equal to or more than the predetermined threshold value, it is determined that the adjacent scenes belong to different scene groups, A temporally later scene among the adjacent scenes is set as a head scene of a new scene group. If the time difference between the scene change frames is equal to or less than a predetermined threshold value, it is determined that adjacent scenes belong to the same scene group. As described above, the scene group determination unit 123 compares the time difference between the scene change frames of all adjacent scenes with the predetermined threshold, and classifies the entire video into the scene groups.
[0005]
FIG. 13 is a block diagram showing an example of a configuration of a conventional video structuring apparatus described in Patent Literature 2, for example. In the video structuring apparatus having the configuration shown in FIG. 13, the feature amount extraction unit 131 extracts a feature amount vector of a time-divided section of an input video, and the quantization unit 132 assigns a number to the feature amount vector. And express the video as a number sequence. Then, the counting unit 133 counts the number of appearances of the number sequence, and extracts a subsequence having a high appearance frequency. In this way, among the time-divided sections of the input video, a sequence of sections that frequently appear in a specific pattern is extracted.
[0006]
[Patent Document 1]
JP 2001-326901 A (Page 4-7, FIG. 1)
[Patent Document 2]
Japanese Patent Application Laid-Open No. H11-242885 (Page 4-10, FIG. 1)
[0007]
[Problems to be solved by the invention]
The video editing system described in Patent Literature 1 determines a scene group based on a time difference between scene change frames. Therefore, when a repetitive structure in which similar scenes repeatedly appear in the same order is included within a predetermined time, the scene group determination unit 123 does not specify each scene forming the repetitive structure as a scene group.
[0008]
Further, the video structuring apparatus described in Patent Document 2 extracts a sequence of sections that appear with a high frequency in a specific pattern among time-divided sections of an input video. Processing is not performed on sections that are not extracted as a sequence of appearing sections. Therefore, an editing operation on a section that is not extracted as a sequence of sections that appear with high frequency in the input video must be performed using a unit other than the video structuring apparatus described in Patent Document 2. .
[0009]
Therefore, the present invention provides a video editing apparatus, a video editing method, and a video editing method that automatically classify the entire input video into scene groups by using the appearance order and the number of appearances of scenes included in the input video. It aims to provide an editing program.
[0010]
[Means for Solving the Problems]
A video editing apparatus according to the present invention detects a scene change, which is a timing at which a scene of a video changes according to input video data, and extracts a feature amount of each scene by dividing the video data into a plurality of scenes. A scene grouping unit that generates a scene group in which each scene in the video data is classified into a plurality of groups in accordance with the extracted feature amount, and associates correspondence information specifying each scene group with each scene; A scene group extracting unit that identifies and extracts a group of scenes corresponding to a sequence of corresponding information appearing in the same arrangement a plurality of times as a scene group, an arrangement of scene groups that are not subject to the extraction by the scene group extracting unit, and a scene. Matching is performed with the scene group extracted by the group extraction means, and the scene group arrangement is included in the scene group according to the result of the matching. And a scene group determination means that, characterized by comprising a means for storing the information of the result of classifying the image data into the scene group.
[0011]
The scene group determining means determines whether the scene group of the first scene in the scene group matches the scene group of the first scene in the sequence of the scene groups, and determines the scene group of the last scene in the scene group. It may be determined whether or not the scene group of the last scene in the arrangement of the scene groups matches, and the arrangement of the scene groups determined to coincide with each other may be included in the scene group. According to such a configuration, an arrangement of scene groups similar to the configuration of the scene group can be specified as the scene group.
[0012]
The scene group determination means may select a scene group in descending order of the number of appearances in the video data and perform matching, and if there are a plurality of scene groups having the same number of appearances in the video data, The matching may be performed by selecting a scene group in the order of appearance. According to such a configuration, each scene forming a repetitive structure in which similar scenes repeatedly appear in the same order can be specified as a scene group.
[0013]
The scene group determination means may specify a sequence of scenes not specified as a scene group as a result of the matching as a scene group. According to such a configuration, the arrangement of all scenes can be specified as a scene group.
[0014]
The video editing device according to the present invention detects a scene change that is a timing at which a video scene is switched by input video data or a timing at which a sound is switched, generates position information for specifying a position of the scene change, and generates a position information on the time axis. A scene detecting unit that divides video data into a plurality of sections sandwiched by scene changes, creates a scene by assigning a scene number to each of the divided sections in chronological order, and outputs the plurality of created scenes;
The feature amount of the plurality of scenes created by the scene detection means is extracted, the similarity between the scenes is calculated, and the plurality of scenes are classified into a plurality of groups based on the similarity between the scenes. A scene classifying unit that creates a plurality of scene groups by assigning a scene group ID to a group, and assigns a scene group ID that specifies a scene group to each of the plurality of scenes, and a scene in which the scene group ID repeatedly appears in the same order A group of group IDs is extracted, a scene group ID is assigned to the extracted group of scene group IDs, a plurality of scene group groups are created, and the number of times that the plurality of scene group groups appear in the video data is counted. Are arranged in descending order of the number of appearances, and the scene group group ID and the appearance order of the scene group ID are input for each scene group. The scene group information including the number of appearances in the image data is output, and a set of a plurality of scenes along a time axis in which the order of appearance of the scene group ID matches the order of appearance of the scene group IDs of the scene group A scene group ID that specifies a scene group is assigned to a plurality of extracted scene groups as a group, and the appearance order of the scene group ID and the scene group ID matches the appearance order of the scene group ID in the extracted scene group. Group extraction means for outputting scene group information composed of a scene group ID of a scene group to be performed and position information of the first scene change of the extracted scene group, and the number of appearances of the scene group in the video data Are selected one by one in the descending order of the number of scenes. A scene group having the same scene group ID as the scene group ID of the first scene group in the selected scene group exists in the sequence of the selected scenes. If there is a scene having the same scene group ID as the scene group ID of the last scene of the selected scene group group, it has the same scene group ID as the scene group ID of the first scene of the selected scene group group. A group of a plurality of scenes starting with a scene and ending with a scene having the same scene group ID as the scene group ID of the last scene of the selected scene group is extracted as a scene group, and the scene group is specified in the extracted plurality of scene groups. The scene group ID of the extracted scene group and the selected scene group Group information composed of the scene group IDs of the scenes and the position information of the first scene change of the extracted scene group, and a list of all the scenes selected to extract the scene group and all the selected scenes This is repeated for the group group, and there is no scene group group in which the arrangement order of one or more scenes along the time axis, which is not extracted as a scene group even when the scene group information is used, matches the appearance order of the scene groups. A scene group is determined, a scene group ID is assigned to the determined plurality of scene groups, and a value indicating that there is no scene group group in which the appearance order of the scene group and the scene group match does not exist. A scene determining means for generating scene group information composed of position information of the first scene change; and a scene group information database for storing the scene group information. Characterized by comprising a means for storing the scan.
[0015]
The video editing method according to the present invention detects a scene change, which is a timing at which a video scene changes according to input video data, divides the video data into scenes, extracts a feature amount of the scene, and A scene group in which scenes are classified into groups according to the scene groups is generated, and a sequence of scene groups that appear in the same sequence multiple times on the time axis of the video data is identified and extracted as a scene group, which is a group of scenes, and is extracted. It performs matching between the scene group sequence that did not match the scene group and the scene group according to the matching result, identifies the scene group sequence as the scene group, and stores the information of the result of classifying the video data into the scene group. Features.
[0016]
The video editing method according to the present invention detects a scene change, which is a timing at which a video scene is switched by input video data or a timing at which a sound is switched, generates position information for specifying a position of the scene change, and generates a position information on the time axis. The video data is divided into a plurality of sections sandwiched by scene changes, scene numbers are assigned to the plurality of divided sections in chronological order, scenes are created, a plurality of created scenes are output, and the created plurality of scenes are output. Calculating the similarity between the scenes by extracting the feature amounts of the scenes, classifying a plurality of scenes into a plurality of groups based on the similarity between the scenes, and assigning a scene group to a frame constituting the classified plurality of groups. A plurality of scene groups are created by assigning IDs, and a scene group ID for specifying a scene group is assigned to each of the plurality of scenes. A group of scene group IDs in which scene group IDs repeatedly appear in the same order is extracted, a scene group group ID is assigned to the extracted group of scene group IDs, and a plurality of scene group groups are created. The number of times the group group appears in the video data is counted and arranged in descending order of the number of appearances. Each scene group is composed of a scene group group ID, the appearance order of the scene group ID, and the number of appearances in the input video data. It outputs scene group information and extracts a set of a plurality of scenes along a time axis in which the order of appearance of the scene group IDs matches the order of appearance of the scene group IDs of the scene group groups as a scene group. A scene group ID for specifying the scene group is assigned to the group, and the scene group ID and the appearance order of the scene group ID are extracted. And outputs the scene group information including the scene group ID of the scene group that matches the order of appearance of the scene group ID in the scene group and the position information of the first scene change of the extracted scene group. The scenes are selected one by one in descending order of the number of appearances in the video data, and successive scenes on the time axis are selected one by one from the remaining scenes not extracted as a scene group. There is a scene having the same scene group ID as the scene group ID of the first scene of the selected scene group, and the scene group ID of the last scene of the selected scene group after the scene on the time axis. If there is a scene having the same scene group ID, the scene group ID of the first scene of the selected scene group A plurality of scene sets are extracted as a scene group, starting with a scene having the same scene group ID and ending with a scene having the same scene group ID as the scene group ID of the last scene of the selected scene group group. A scene group ID for specifying a scene group is assigned to the group, and the scene group ID of the extracted scene group, the scene group group ID of the selected scene group, and the position information of the first scene change of the extracted scene group are assigned. Generated scene group information and repeat the scene group extraction for all selected scenes and all selected scene group groups. Even if the scene group information is used, they are not extracted as scene groups. The sequence of one or more scenes along the time axis Is determined as a scene group that does not exist, a scene group ID is assigned to the determined plurality of scene groups, and a value indicating that there is no scene group group in which the appearance order of the scene group and the scene group match is determined. It is characterized in that scene group information composed of position information of the first scene change of a scene group is generated, and the scene group information is stored in a scene group information database.
[0017]
A video editing program according to the present invention is a video editing program installed in a video editing device that edits video data to generate a scene group that is a group of scenes, and includes a computer that edits video based on video data input. Detecting a scene change when a scene changes, and dividing video data into scenes, extracting scene features, and generating a scene group in which scenes are classified into groups according to the extracted features The process of identifying and extracting the sequence of scene groups that appear in the same sequence multiple times on the time axis of video data as a scene group that is a collection of scenes, and the sequence and extraction of scene groups that were not targeted for extraction Matching with the set of scenes performed, and including the arrangement of the scene groups in the scene group according to the result of the matching. Characterized in that it comprises a process of storing the information of the result of classifying the image data into the scene groups in the storage device.
[0018]
BEST MODE FOR CARRYING OUT THE INVENTION
An embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing one configuration example of an embodiment of the present invention. The video editing device according to the present invention divides the video data 101 into scenes by detecting a scene included in the input video data 101, which is a timing at which a camera is switched, or a timing at which a sound (for example, a speaker) is switched. A scene detecting unit 102, a scene classifying unit 103 that calculates a feature amount of the scene, and generates a scene group in which the scenes are classified into groups according to the calculated feature amount, and a scene including scene groups that repeatedly appear in the same order. A scene group extracting unit 104 for extracting a scene group that is a set of groups; a scene group determining unit 105 for classifying a scene group that has not been a target of the scene group generation to generate a scene group; Storage means 106 for storing the information of the result of the classification into the scene group information database 107 Including the. The video editing apparatus according to the present invention is realized by a computer or the like, and each means is realized by a program or the like.
[0019]
When the video data 101, which is data of a moving image, is input, the scene detection unit 102 detects, in the video of the video data 101, a scene change that is a timing at which a camera is switched or a timing at which an audio is switched. The method of detecting a scene change is, for example, calculating a difference in layout of color information of pixels between consecutive frames of an image based on the image data 101, and, when the calculated difference is equal to or greater than a predetermined threshold, detecting a scene change. This is performed by determining that there is. The scene detecting means 102 divides the video data 101 at a timing when a scene change is detected. Then, a scene information file which is a file of information corresponding to each scene which is each section of the divided video data 101 is generated.
[0020]
The scene detecting means 102 assigns a number to each of the divided scenes from the top in time order, and records the assigned number in the scene information file in association with each scene. Then, the scene detection unit 102 outputs the video data 101 divided into scenes and the scene information file to the scene classification unit 103. Further, the scene detecting means 102 generates scene change position information which is information for specifying a scene change frame which is a scene changed frame, and outputs the information to the scene group determining means 105. Here, the video data 101 may be in a signal format in which the scene detecting means 102 can detect a scene change. For example, data recorded on a recording medium such as an analog VTR or DV (Digital Video), MPEG, or the like Data.
[0021]
When the video data 101 divided into scenes is input, the scene classification unit 103 extracts the feature amount of each scene and compares the extracted feature amounts of each scene with each other to calculate a similarity. Here, the feature amount of a scene is a layout of color information of each pixel in a video frame based on the video data 101. Further, the type of audio signal such as whether the audio signal included in the video data is stereo audio, monaural audio, or multiplex audio, or the waveform of the audio signal may be used as the feature amount. Further, information such as the presence or absence of a subtitle, the display position of the subtitle, and the display language may be used as the feature amount. The similarity is, for example, a sum of absolute differences of feature amounts between scenes. Then, the scenes are classified into scene groups in which the degree of similarity of the scene is smaller than a predetermined threshold, and correspondence information (for example, a scene group ID) which is a symbol for specifying the scene group is assigned to each scene. I do. That is, the scene classification unit 103 associates a scene group ID that specifies each scene group with each scene.
[0022]
The scene classification unit 103 classifies scenes into groups based on the similarity of the feature amounts using an existing method such as a singular value decomposition (SVD) method. Specifically, by extracting the feature amount of each scene and mapping it to the feature amount space, a cluster of scenes having similar feature amounts in the feature amount space is created. If the distance between the feature values is smaller than a predetermined threshold in the feature value space, they are assumed to belong to one block. Each group is a group. The scene classification means 103 records the scene group ID assigned to each scene in the scene information file in association with each scene. The scene classification unit 103 outputs the video data 101 and the scene information file to the scene group extraction unit 104.
[0023]
When the video data 101 is input, the scene group extraction unit 104 is a set of scene group IDs in which the same scene group ID repeatedly appears in the same order based on the scene group ID recorded in the scene information file. Extract group groups. The extraction of the scene group may use an existing method such as a dynamic programming method which is a method of text data mining. Then, a scene group ID, which is a symbol for specifying the scene group, is assigned to each scene. The scene group extracting means 104 records the scene group ID assigned to each scene in the scene information file in association with each scene. Then, the scene group extracting unit 104 specifies the number of the scene group groups existing in the video data 101 and the order on the time axis in the video data 101 of each scene group group.
[0024]
The scene group extraction unit 104 includes a scene group composed of a scene group group ID, information on the number of scene group groups present in the video data 101, and information on the order of the scene group groups in the video data 101 on the time axis. Generate group information. At this time, the order of the scene group IDs in the scene group information may be the order in which the number of scene group present in the video data 101 is large.
[0025]
The scene group extracting unit 104 extracts a scene group, and sets each extracted scene group as a scene group. Then, a scene group ID, which is a symbol for specifying the scene group, is assigned to each scene group. The scene group extracting means 104 records the scene group ID in the scene information file in association with each scene. The scene group extracting means 104 is configured by a scene group ID of a scene group, information of a scene group group ID of each scene constituting the scene group, and information for specifying a first scene change frame of each scene group. Generated scene group information. The scene group extracting unit 104 outputs the scene group, the video data 101 of the portion not extracted as the scene group, the scene group information, and the scene group information to the scene group determining unit 105. The scene group extraction unit 104 may output the scene group information to the storage unit 106 and store the scene group information in the scene group information database.
[0026]
The scene group determining unit 105 divides the video data 101 of a portion not extracted as a scene group into scenes based on the scene change position information output by the scene detecting unit 102. Then, the scene group determining means 105 selects one scene group. The scene group ID of the first scene of the selected scene group and the scene group ID of the last scene of the selected scene group are extracted. In addition, of the remaining scenes not extracted as a scene group by the scene group extraction unit 104, a sequence of continuous scenes on the time axis is specified. Then, a scene having the same scene group ID as the scene group ID of the first scene of the selected scene group is identified from the remaining scenes not extracted by the scene group extraction unit 104 as the scene group.
[0027]
Then, if there is a scene that is continuous with the specified scene and has the same scene group ID as the scene group ID of the last scene of the selected scene group group behind on the time axis, the selected scene is selected. A scene between a scene having the same scene group ID as the scene group ID of the first scene of the scene group group and a scene having the same scene group ID as the scene group ID of the last scene of the selected scene group group A scene group is extracted as a group, and a scene group ID for specifying the scene group is assigned to each scene constituting the scene group. The scene group determining means 105 records the scene group ID in the scene information file in association with each scene.
[0028]
The scene group determination unit 105 selects the scene groups in the order of the number of appearances for all the input scene groups and performs the above-described processing. If there are a plurality of scene groups with the same number of appearances, The scene groups are selected in the order in which they appear, and the above processing is performed to extract the scene groups. Then, the above processing is performed for all the input scene groups. Scenes that have not been extracted even after performing the above processing are extracted as scene groups, and a scene group ID for specifying the scene group is assigned to each scene constituting the scene group. The scene group determining means 105 records the scene group ID in the scene information file in association with each scene.
[0029]
The scene group determining means 105 includes: a scene group ID of the scene group extracted by the scene group determining means 105; a scene group group ID of each scene constituting the scene group; and information for specifying the first scene change frame of each scene group. Is generated, and scene group information is newly generated. Then, the scene group determining unit 105 outputs the scene information file, the scene group group information, and the scene group information to the storage unit 106.
[0030]
The storage unit 106 includes a scene group information database 107, and stores the input scene information file, scene group information, and scene group information in the scene group information database 107.
[0031]
A video editing program according to the present invention is a video editing program that edits video data and is mounted on a video editing apparatus that generates a scene group that is a group of scenes and realizes each unit, and is input to a computer. The process of detecting a scene change, which is the timing at which a video scene changes with video data, dividing the video data into scenes, extracting the features of the scene, and classifying the scenes into groups according to the extracted features. A process of generating a group, a process of extracting a scene group sequence that appears in the same sequence on the time axis of video data a plurality of times as a scene group that is a collection of scenes, and a scene that is not an extraction target Performs matching between the group arrangement and the scene group, and identifies the scene group arrangement as a scene group according to the result of the matching. Comprising a processing, and processing to store the information of the result of classifying the image data into the scene groups in the storage device.
[0032]
Next, the operation of the embodiment of the present invention will be described with reference to the drawings. FIG. 2 is a flowchart showing the operation of the video editing device according to the embodiment of the present invention. 2, step S201 represents the operation of the scene detection means 102, step S202 represents the operation of the scene classification means 103, step S203 represents the operation of the scene group extraction means 104, and step S204 is the scene group determination means 105 Step S205 represents the operation of the storage means 106.
[0033]
The operation of each means will be described. First, the operation of the scene detecting means 102 will be described. FIG. 3 is a flowchart showing the operation of the scene detecting means 102 according to the present invention. FIG. 4 is a diagram schematically showing video data to be video-edited by the video editing device according to the present invention. When the video data 101 to be edited is input (step S301), the scene detection unit 102 detects a scene change of the video based on the input video data 101 (step S302). That is, as shown in the example of FIG. 4, the frames 701 to 724 of the scene change are detected. Then, scene change position information for specifying a scene change frame is generated and output to the scene group determining means 105.
[0034]
The scene detecting means 102 divides the video data 101 at the timing of the detected scene change (step S303), and creates a scene by assigning a scene number from the beginning to the frame of each scene, which is each of the divided sections, in order of time. (Step S304), the scene number is recorded in the scene information file. An example of the scene number assigned to each scene is shown in the scene number column of FIG. Here, the scene constituted by the frame to which the scene number “S1” is assigned includes the scene change frame 701 and the subsequent frames up to the frame before the scene change frame 702. Hereinafter, similarly, the scene constituted by the frame to which the scene number “S2” is assigned includes the frame after the scene change frame 702 and the frame before the scene change frame 703. Then, each created scene is output to the scene classification means 103 (step S305).
[0035]
Next, the operation of the scene classification means 103 will be described. FIG. 6 is a flowchart showing the operation of the scene detecting means 103 in the present invention. When a scene obtained by dividing the video data 101 by the scene detection unit 102 is input (step S401), the scene classification unit 103 extracts a feature amount of each scene (step S402), and sets a similarity between feature amounts of a plurality of scenes. The plurality of scenes are classified into a plurality of groups based on (Step S403). Then, a scene group is created by assigning a scene group ID, which is the ID of each group, to each group (step S404), and each scene group ID is assigned to each scene (step S405). The scene classification means 103 records the scene group ID assigned to each scene in the scene information file in association with each scene. An example of a scene group ID assigned to each scene is shown in the scene group ID column of FIG. The scene classification unit 103 assigns a scene group ID “a” to each frame to which the scene number “S1” is assigned, and thereafter, as shown in FIG. 5, the scene group IDs “b” to “f” Shall be given. The scene classification unit 103 outputs the video data 101 and the scene information file to the scene group extraction unit 104.
[0036]
Next, the operation of the scene classification unit 104 will be described. FIG. 7 is a flowchart showing the operation of the scene detecting means 104 according to the present invention. When the video data 101 and the scene information file are input (step S501), the scene group extraction unit 104 repeats the scene group ID in the video data 101 based on the scene group ID recorded in the scene information file. A set of scene group IDs appearing in the same order is extracted (step 502). Referring to FIG. 5, it can be seen that there are consecutive sets of scene group IDs in the order of “abc” and that there are consecutive sets in the order of “dbe”. Then, a scene group ID is assigned to the extracted set of scene group IDs to create a plurality of scene group groups (step 503).
[0037]
The scene group extracting means 104 records the scene group ID assigned to each scene in the scene information file in association with each scene. As shown in FIG. 8, a scene group group is created by assigning a scene group ID “sg1” to each frame forming a set of consecutive scene group IDs in the order of “abc”, and the order of “dbe” It is assumed that a scene group group is created by adding a scene group group ID “sg2” to each frame constituting a set of consecutive scene group IDs.
[0038]
The scene classification unit 104 specifies the number of scene groups present in the video data 101 with reference to the scene information file (step S504). Referring to FIG. 5, as shown in the column of the number of appearances in FIG. 8, the scene group group “sg1” appears three times, and the scene group group “sg1” appears twice. Then, the scene group groups are arranged in the order of appearance frequency, the scene group group ID, information on the number of scene group groups existing in the video data 101, and information on the order of the scene group groups on the time axis in the video data 101. And generates scene group information. That is, the scene group information is configured as shown in FIG. The scene classifying means 104 outputs the scene group information to the scene group determining means 105 (step S506).
[0039]
The scene group extracting means 104 extracts a scene group, and sets each extracted scene group as a scene group (step S507). Then, a scene group ID is assigned to each extracted scene group (step S508). The scene group extracting means 104 records the scene group ID assigned to each scene in the scene information file in association with each scene. It is assumed that a scene group ID “SG1” is given to the scene numbers “S1, S2, S3”, and thereafter, as shown in FIG. 9, scene group IDs “SG2” to “SG5” are given. Then, scene group information composed of a scene group ID of the scene group, information of the scene group group ID of each scene constituting the scene group, and information for specifying the first scene change frame of each scene group is generated. Is output to the scene group determining means 105 (step S509). Further, the scene group extracting unit 104 outputs the scene information file, the scene group, and each scene not extracted as the scene group to the scene group determining unit 105.
[0040]
Next, the operation of the scene group determining means 105 will be described. FIG. 7 is a flowchart showing the operation of the scene group determining means 105 according to the present invention. When the scene group determining unit 105 receives the scene group group information and the remaining scenes not extracted as the scene group by the scene group extracting unit 104 (step S601), the scene group determining unit 105 divides the scene based on the scene change position information. . Then, one scene group is selected in descending order of appearance frequency (step S602). Referring to FIG. 8, the scene group “sg01” is selected because the number of appearances of the scene group “sg01” is the largest. Then, of the remaining scenes that have not been extracted as a scene group by the scene group extracting unit 104, one continuous scene sequence along the time axis is specified (step S603). For example, scene numbers “S10” to “S12” in FIG. 9 are specified.
[0041]
Then, the specified arrangement of scenes is matched with the scene group. That is, a scene having the same scene group ID as the scene group ID of the first scene of the selected scene group is specified from the remaining scenes not extracted by the scene group extracting unit 104 as the scene group. Then, if there is a scene that is continuous with the specified scene and has the same scene group ID as the scene group ID of the last scene of the selected scene group group behind on the time axis, the selected scene is selected. Scenes from the scene having the same scene group ID as the scene group ID of the first scene group of the scene group to the scene having the same scene group ID as the scene group ID of the last scene of the selected scene group are extracted as the scene group. Then, a scene group ID is assigned (steps S604 and S605). The scene group determining means 105 records the scene group ID in the scene information file in association with each scene. The scene group determining unit 105 may generate scene group information that newly includes information on the scene group extracted at this time, and output the generated scene group information to the storage unit 106.
[0042]
Referring to FIG. 8, the scene group ID of the first scene of the scene group group “sg01” is “a”. Therefore, when a scene with the scene group ID “a” is searched from the scenes “S10” to “S12” in FIG. 9 using the scene information file, the scene group ID of the scene “S10” is “a”. You can see that. The scene group ID of the last scene of the scene group group “sg01” is “c”. Therefore, among the scenes “S10” to “S12”, a scene whose scene group ID is “c” is searched backward on the time axis from the scene “S10”. Then, it is found that the scene group ID of the scene “S12” is “c”. Therefore, scenes “S10” to “S12” are extracted as a scene group, and a scene group ID “SG06” is assigned.
[0043]
The scene group determining unit 105 determines whether or not there is a scene that has not been matched with the scene group in the specified sequence of scenes (step S604). If the scene exists, the operation of step S605 is performed. Do. Since there is no scene sequence out of the specified scene sequence “S10” to “S12” that has not been matched with the scene group, the operation in step S605 is not performed. Then, it is determined whether or not the operations of Steps S605 and S606 have been performed for a sequence of continuous scenes along the time axis in other scenes not extracted as a scene group (Step S607), and no operation has been performed. An arrangement is selected (step S603), and the same operation is performed.
[0044]
Referring to FIG. 9, the same operation is performed for scenes “S16” to “S20”. That is, a search for a scene whose scene group ID is “a” from the scenes “S16” to “S20” reveals that the scene group ID of the scene “S16” is “a”. The scene group ID of the last scene of the scene group group “sg01” is “c”. Therefore, among the scenes “S16” to “S20”, a scene whose scene group ID is “c” is searched backward on the time axis from the scene “S16”. Then, it is found that the scene group ID of the scene “S17” is “c”. Therefore, scenes “S16” to “S17” are extracted as a scene group, and a scene group ID “SG07” is assigned.
[0045]
Next, when a scene with a scene group ID of “a” is searched from scenes “S18” to “S20”, no scene with a scene group ID of “a” exists. When the matching with the scene group group “sg01” is completed for all the arrangements of scenes, the scene group determining unit 105 determines whether the matching by all the scene group groups is completed (step S607). Referring to FIG. 8, since the matching has not been performed on the scene group group “sg02”, the scene group determining unit 105 selects the scene group group “sg02” (step S602) and performs the matching. Then, a sequence of continuous scenes along the time axis is specified from the sequence of remaining scenes not extracted as a scene group (step S603). Then, scene numbers “S18” to “S20” in FIG. 9 are specified.
[0046]
Referring to FIG. 8, the scene group ID of the first scene of the scene group group “sg02” is “d”. Therefore, when the scene with the scene group ID “d” is searched from the scenes “S18” to “S20” in FIG. 9, it is found that the scene group ID of the scene “S18” is “d”. The scene group ID of the last scene of the scene group group “sg02” is “e”. Therefore, among the scenes “S18” to “S20”, a scene whose scene group ID is “e” is searched backward on the time axis from the scene “S18”. Then, it is found that the scene group ID of the scene “S19” is “e”. Therefore, scenes “S18” to “S19” are extracted as a scene group, and a scene group ID “SG08” is assigned.
[0047]
The only remaining scene is scene “S20”. When there is only one scene, the scene group determination unit 105 does not extract the scene group. The scene group determining unit 105 determines that the arrangement of all scenes and the matching by all scene group groups have been completed.
[0048]
The scene group determination unit 105 determines the arrangement of all scenes and the arrangement of one or more scenes along the time axis that have not been extracted as a scene group even after performing matching by all scene group groups. A scene group that does not have the same order of scene group is determined (step S608), and a scene group ID is assigned to the determined plurality of scene groups (step S609). A scene group ID “SG09” is assigned to the scene “S20”. FIG. 11 shows the result of assigning a scene group ID to all the scenes of the video data 101.
[0049]
The scene group determining unit 105 determines the scene group ID of the scene group extracted by the scene group determining unit 105, the information of the scene group group ID of each scene constituting the scene group, and the first scene change frame of each scene group. Scene group information including information to be newly specified is generated. Then, the scene group determining unit 105 outputs the scene group group information and the scene group information to the storage unit 106 (Step S609).
[0050]
The storage unit 106 stores the scene group information and the scene group information output by the scene group determining unit 105 in the scene group information database 107 (Step S205).
[0051]
【The invention's effect】
As described above, according to the present invention, it is possible to automatically perform editing for classifying an input video based on a feature amount of a scene and dividing the video into a scene group which is a group of scenes. Further, when a repetitive structure in which scenes repeatedly appear in the same order is included in the input video, the repetitive structure can be extracted as a scene group. Then, after editing the video by the video editing apparatus according to the present invention, when searching for one scene included in one repetitive structure, the effect that a video of a desired scene can be found by searching for a scene group. There is.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration example of an embodiment of the present invention.
FIG. 2 is a flowchart showing the operation of the embodiment of the present invention.
FIG. 3 is a flowchart showing the operation of the scene detecting means according to the present invention.
FIG. 4 is a diagram illustrating a frame configuration example of video data to be edited by the embodiment of the present invention;
FIG. 5 is a diagram showing a scene number of each scene of video data and a scene group ID assigned to each scene.
FIG. 6 is a flowchart showing the operation of the scene classification means according to the present invention.
FIG. 7 is a flowchart showing the operation of the scene extracting means according to the present invention.
FIG. 8 is a diagram showing a scene group ID, the order of appearance of the scene group ID, and the number of appearances of the scene group.
FIG. 9 is a diagram showing a scene number, a scene group ID, and a scene group ID of a scene group extracted by a scene extracting unit.
FIG. 10 is a flowchart showing the operation of the scene determining means according to the present invention.
FIG. 11 is a diagram showing a scene number, a scene group ID, a scene group ID of a scene group extracted by a scene extracting unit, and a scene group ID of a scene group determined by a scene determining unit.
FIG. 12 is a block diagram showing a configuration of an embodiment of a conventional video editing system described in Patent Document 1.
FIG. 13 is a block diagram showing a configuration of an embodiment of a conventional video editing system described in Patent Document 2.
[Explanation of symbols]
101 Video data
102 Scene detecting means
103 Scene Classification Means
104 scene group extraction means
105 Scene group determination means
106 storage means
107 Scene group information database
121 code parameter extraction unit
122 Scene Change Frame Detector
123 scene group determination unit
124 Scene Information Hierarchy Unit
125 storage unit
131 Feature Extraction Unit
132 Quantizer
133 counting section
701 First frame of the first scene change
702 First frame of second scene change
703 First frame of the third scene change
704 First frame of the fourth scene change
705 First frame of the fifth scene change
706 First frame of the sixth scene change
707 First frame of the seventh scene change
708 First frame of the eighth scene change
709 First frame of ninth scene change
710 First frame of the tenth scene change
711 First frame of the eleventh scene change
712 First frame of twelfth scene change
713 First Frame of 13th Scene Change
714 First Frame of Fourteenth Scene Change
715 First Frame of 15th Scene Change
716 First frame of 16th scene change
717 First Frame of 17th Scene Change
718 First Frame of 18th Scene Change
719 First Frame of 19th Scene Change
720 First frame of 20th scene change
721 First Frame of 21st Scene Change
722 First frame of the 22nd scene change
723 First Frame of 23rd Scene Change
724 First Frame of 24th Scene Change

Claims (8)

Scene detection means for detecting a scene change, which is a timing at which a video scene changes according to input video data, and dividing the video data into a plurality of scenes;
A scene classification that extracts a feature amount of each scene, generates a scene group in which each scene in the video data is classified into a plurality of groups according to the extracted feature amount, and associates corresponding information specifying each scene group with each scene. Means,
A scene group extracting means for identifying and extracting a group of scenes corresponding to a sequence of corresponding information appearing in the same arrangement a plurality of times on the time axis as a scene group;
A sequence of scene groups not extracted by the scene group extraction unit is matched with a scene group extracted by the scene group extraction unit, and the sequence of scene groups is included in the scene group according to the result of the matching. Scene group determining means;
A video editing device comprising: a storage unit configured to store information as a result of classifying video data into scene groups. The scene group determining means is
It is determined whether the scene group of the first scene in the scene group matches the scene group of the first scene in the arrangement of the scene groups,
It is determined whether the scene group of the last scene of the scene group matches the scene group of the last scene in the arrangement of the scene group,
The video editing apparatus according to claim 1, wherein the sequence of scene groups determined to match both is included in the scene group. The scene group determination means selects a scene group in descending order of the number of appearances in the video data and performs matching. If a plurality of scene groups having the same number of appearances in the video data exist, they appear on the time axis. 3. The video editing apparatus according to claim 2, wherein a scene group is selected in order and matching is performed. 4. The video editing apparatus according to claim 2, wherein the scene group determining unit specifies a sequence of scenes that are not specified as a scene group as a result of the matching as a scene group. 5. Detects a scene change, which is a timing at which a video scene is switched or a timing at which an audio is switched, based on input video data, generates position information specifying a position of the scene change, and generates a plurality of positions sandwiched by the scene changes on the time axis. Scene detection means for dividing video data into sections, creating scenes by assigning scene numbers to the divided sections in chronological order, and outputting the created plurality of scenes;
Extracting a feature amount of the plurality of scenes created by the scene detection means, calculating a similarity between the scenes, classifying the plurality of scenes into a plurality of groups based on the similarity between the scenes, A scene classification unit that creates a plurality of scene groups by assigning a scene group ID to the plurality of classified groups, and assigns the scene group ID that specifies a scene group to each of the plurality of scenes;
Extracting a set of the scene group IDs in which the scene group IDs repeatedly appear in the same order, creating a plurality of scene group groups by assigning a scene group ID to the extracted set of the plurality of scene group IDs, The number of times the plurality of scene group groups appear in the video data is counted and arranged in descending order of the number of appearances. For each of the scene group groups, the scene group group ID, the appearance order of the scene group ID, and the number of appearances in the input video data And outputs a set of a plurality of scenes along a time axis in which the order of appearance of the scene group IDs matches the order of appearance of the scene group IDs of the scene group group as a scene group. Assigning a scene group ID for specifying a scene group to the plurality of extracted scene groups, The appearance order of the loop IDs is made up of the scene group IDs of the scene groups that match the appearance order of the scene group IDs in the extracted scene group, and the position information of the first scene change of the extracted scene group. Scene group extracting means for outputting scene group information;
The scene group group is selected one by one in descending order of the number of appearances in the video data, and a sequence of scenes continuous on the time axis is selected from the remaining scenes not extracted as a scene group by the scene group extraction unit. Scenes having the same scene group ID as the scene group ID of the first scene of the selected scene group group in the sequence of the selected scenes. If there is a scene having the same scene group ID as the scene group ID of the last scene of the selected scene group, the scene having the same scene group ID as the scene group ID of the first scene of the selected scene group is selected. And the same scene group ID as the scene group ID of the last scene of the selected scene group. A set of a plurality of scenes ending with the scene having an ID is extracted as a scene group, a scene group ID for specifying a scene group is assigned to the extracted plurality of scene groups, and a scene group ID of the extracted scene group Generating scene group information including a scene group ID of the selected scene group and position information of the first scene change of the extracted scene group, and selecting all the scenes for which the extraction of the scene group is selected. Is repeated with respect to all the selected scene group groups, and one or more scenes along the time axis that are not extracted as scene groups even when the scene group information is used are used to generate scene group appearances. Determining a scene group in which no scene group group in the same order exists, assigning a scene group ID to the determined plurality of scene groups, For generating scene group information composed of a value indicating that there is no scene group group in which the appearance order of the scene group and the scene group match, and the determined position information of the first scene change of the scene group Determining means;
A video editing device comprising: a storage unit configured to store the scene group information in a scene group information database. Detects a scene change, which is the timing at which the video scene changes due to the input video data, divides the video data into multiple scenes,
Extracting a feature amount of each scene, generating a scene group in which each scene in the video data is classified into a plurality of groups according to the extracted feature amount, associating correspondence information specifying each scene group with each scene,
A set of scenes corresponding to a sequence of corresponding information appearing in the same arrangement multiple times on the time axis is specified and extracted as a scene group,
Matching the sequence of scene groups that were not targeted for extraction with the scene group, specifying the sequence of scene groups as scene groups according to the result of the matching,
A video editing method, comprising storing information as a result of classifying video data into scene groups. Detects a scene change, which is a timing at which a video scene is switched by an input video data or a timing at which an audio is switched, and generates position information for specifying a position of the scene change, and a plurality of positions sandwiched between the scene changes on the time axis. Divide the video data into sections, create scenes by assigning scene numbers to the divided sections in chronological order, and output the created plurality of scenes;
The feature amounts of the created plurality of scenes are extracted to calculate the similarity between the scenes, and the plurality of scenes are classified into a plurality of groups based on the similarity between the scenes. A plurality of scene groups are created by assigning a scene group ID to frames constituting a group, and the scene group ID for specifying a scene group is assigned to each of the plurality of scenes,
Extracting a set of the scene group IDs in which the scene group IDs repeatedly appear in the same order, creating a plurality of scene group groups by assigning a scene group ID to the extracted set of the plurality of scene group IDs, The number of times the plurality of scene group groups appear in the video data is counted and arranged in descending order of the number of appearances. For each of the scene group groups, the scene group group ID, the appearance order of the scene group ID, and the number of appearances in the input video data And outputs a set of a plurality of scenes along a time axis in which the order of appearance of the scene group IDs matches the order of appearance of the scene group IDs of the scene group group as a scene group. Assigning a scene group ID for specifying a scene group to the plurality of extracted scene groups, The appearance order of the loop IDs is made up of the scene group IDs of the scene groups that match the appearance order of the scene group IDs in the extracted scene group, and the position information of the first scene change of the extracted scene group. Output scene group information,
The scene group was selected one by one in descending order of the number of appearances in the video data, and a sequence of consecutive scenes on the time axis was selected one by one from the remaining scenes not extracted as a scene group. There is a scene having the same scene group ID as the scene group ID of the first scene of the selected scene group group in the sequence of the scenes, and the scene group of the selected scene group group is located behind the scene on the time axis. If there is a scene having the same scene group ID as the scene group ID of the last scene, the scene starts with the scene having the same scene group ID as the scene group ID of the first scene of the selected scene group group, and The scene having the same scene group ID as the scene group ID of the last scene of the scene group group A group of a plurality of ending scenes is extracted as a scene group, a scene group ID for specifying a scene group is assigned to the extracted plurality of scene groups, and a scene group ID of the extracted scene group and the selected scene group group The scene group information including the scene group ID of the scene group and the position information of the first scene change of the extracted scene group is generated. The scene group in which the order of appearance of the scene group matches one or more scenes along the time axis which is not extracted as a scene group even when the scene group information is used by repeating the scene group group A scene group having no group is determined, and a scene group ID is assigned to the determined plurality of scene groups. And generating a composed scene group information in the position information of the start of a scene change of the scene group scene group group is determined to a value that means it is absent the order of appearance matches,
A video editing method, wherein the scene group information is stored in a scene group information database. A video editing program mounted on a video editing device that edits video data and generates a scene group that is a collection of scenes,
On the computer,
A process of detecting a scene change, which is a timing at which a video scene changes according to input video data, and dividing the video data into a plurality of scenes;
A process of extracting a feature amount of each scene, generating a scene group in which each scene in the video data is classified into a plurality of groups according to the extracted feature amount, and associating correspondence information specifying each scene group with each scene; ,
A process of identifying and extracting a group of scenes corresponding to an array of corresponding information appearing in the same arrangement a plurality of times on the time axis as a scene group,
A process of matching the sequence of scene groups that were not the target of extraction with the extracted scene group, and including the sequence of scene groups in the scene group according to the result of the matching;
Storing a result of classifying the video data into scene groups in a storage device.

Images