JP2007336106A - Video image editing assistant apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video image editing assistant apparatus for allowing an ordinary user without any technology and knowledge of video image editing to effectively and efficiently edit a video content. <P>SOLUTION: The video image editing assistant apparatus supports a user in editing, when the user edits a video content to create a user-edited video image. The apparatus comprises a learning portion 2, a material receiving portion 10, a candidate generating portion 13, and an outputting portion 4. The learning portion 2 learns an edit information indicative of alignment of a plurality of substance components contained in a sample video image, which is an edited video content. The material receiving portion 10 receives input of a material video image, which is a material video content of the user-edited video image. The candidate generating portion 13 generates information indicative of a candidate for alignment of substance components at the user-edited video image, according to the edit information learned by the learning portion 2. The substance components are the ones contained in the material video image received by the material receiving portion 10 or the ones added to the user-edited video image. The outputting portion 4 outputs the information indicative of a candidate and generated by the candidate generating portion 13. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a video editing support apparatus that supports editing of video content including moving images and sounds.

  In recent years, with the improvement of information distribution technologies such as the Internet and digital television broadcasting, users who are not video specialists, that is, general users can obtain a large amount of various video contents.

  Furthermore, high-performance digital video cameras, personal computers (PCs), and the like are available at low cost, and an environment in which general users can easily store and browse video content is being prepared.

  In addition to handling video content in this way, software that can be easily edited by general users and can be edited on a PC has been developed and released one after another.

  Here, the video content is one type of multimedia content including a plurality of elemental media such as moving images, sounds, and subtitles, and is, for example, a movie.

  In addition, video editing is the work of creating a new video by connecting appropriate partial videos selected from video content to be edited (hereinafter referred to as “material video”). It is the work of stitching together while adding editing effects.

  A partial video is a predetermined unit of video that constitutes video content. The predetermined unit is, for example, a unit of video that is shot between pressing the shooting start button of the camera and pressing the shooting stop button, that is, a unit of video that is continuously recorded without interruption. It is.

  The sound added at the time of video editing includes music, sound effects, narration sound, and the like. The editing effect is a visual effect at the time of video switching, and includes cut, dissolve, fade-in, fade-out, and the like.

  That is, there are many types of partial videos included in the material video, and sounds and editing effects added during editing. Therefore, there are a huge number of combinations of sound and video and combinations of the same elemental media in time series.

  Therefore, the quality of the edited video content (hereinafter referred to as “edited video”) depends on the skill and knowledge of the editor.

  Therefore, research is being conducted to enable a general user who has no editing technique and editing knowledge to perform video editing effectively and efficiently.

For example, a technique for improving the efficiency of video editing by preparing a template for video editing in advance and allowing a user to insert a partial video into the template is disclosed (for example, see Non-Patent Document 1). .
XS. Hua, Z. Wang, and S. Li, "LazyCut-Content-Aware Template-Based Video Authoring," Proc. ACM Multimedia, 2005.

  However, in the above-described conventional technology, it is left to the user to create a template that presents how to combine partial videos and selection criteria for partial videos. Therefore, it is not possible to promote daily video editing for general users.

  There are also many studies related to automatic video editing, most of which focus on the composition of moving images. For example, research is being conducted on automatic video editing based on video grammar, such as shots with significantly different shot lengths cannot be connected.

  However, there are an infinite number of editing patterns based on such video grammars, but they are not formulated, including those that are set subjectively, and it is not easy to build a system that considers all editing patterns.

  In consideration of the above-described problems, an object of the present invention is to provide a video editing support apparatus for a general user who does not have video editing technology and knowledge to edit video content effectively and efficiently.

  In order to achieve the above object, a video editing support apparatus according to the present invention is a video editing support apparatus that supports editing when a user creates a video edited by editing a material video that is video content. Learning means for learning editing information, which is information relating to the arrangement of a plurality of entity elements included in the sample video that is the edited video content, material receiving means for receiving input of the material video, and learning by the learning means Candidates for generating information indicating candidates for arrangement of the entity elements included in the material video received by the material reception unit or the entity elements added to the user edited video according to the edited editing information in the user edited video Generating means; candidate output means for outputting information indicating the arrangement candidates generated by the candidate generating means; Provided.

  Thereby, editing information can be learned from a sample video produced by editing by a video editing specialist as a sample video. In addition, the user can be presented with information indicating candidates for arrangement of entity elements included in the material video or a plurality of entity elements added to the user-edited video.

  That is, a moving image to be placed at a certain time position on the user-edited video selected by the user, a time position to which the moving image selected by the user is to be placed, substantial elements such as additional sound and editing effect added to the moving image, and additional sound In addition, information indicating candidates such as time positions where editing effects and the like should be arranged can be presented to the user.

  That is, the user can effectively and efficiently produce the user edited video while referring to the editing technology and editing knowledge of the video editing specialist.

  The video editing support apparatus according to the present invention further includes an instruction receiving unit that receives an instruction from the user, and the candidate selection unit includes a predetermined unit entity element included in the material video. When an instruction to select is received, a candidate for a time position on the user-edited video where the instructed entity element is to be placed is selected from any time position on the user-edited video according to the editing information. Then, when the information indicating the arrangement candidate is generated and the instruction receiving unit receives an instruction to select a time position on the user-edited video, the entity element candidates to be arranged at the instructed time position May be selected from a plurality of entity elements included in the material video in accordance with the editing information to generate information indicating the arrangement candidate.

  Thus, for example, when the user selects one shot that he / she wants to include in the user-edited video, the video editing support apparatus, according to the editing information, shows information indicating candidates for time positions suitable for arranging the shot. It can be presented to the user.

  On the other hand, for example, when the user selects a time position where one shot is desired to be placed, the video editing support apparatus presents information indicating shot candidates suitable for placement at the time position to the user. Can do.

  Further, the video editing support apparatus of the present invention further includes material feature extraction means for extracting each signal feature of the moving image and the sound, which is an entity element included in the material video received by the material reception means, The learning means is an entity element included in the sample video, sample feature extraction means for extracting signal features of moving images and sounds, and analyzing the signal features extracted by the sample feature extraction means, Analysis means for obtaining information indicating a time-series configuration of each of the moving image and the sound included in the sample video and information indicating a synchronous configuration of the moving image and the sound, and the time obtained by the analyzing means The information indicating the sequence structure and the information indicating the synchronous structure are learned as the editing information, and the candidate generating means is the time series obtained by the analyzing means. By comparing the information indicating the configuration or the information indicating the synchronous configuration with the signal feature extracted by the material feature extraction means, the user edit video of a predetermined unit of moving image or sound included in the material video Information indicating arrangement candidates may be generated.

  As described above, the video editing support apparatus according to the present invention can learn the time-series configuration of the moving image and sound of the sample video and the synchronized configuration of the moving image and sound from the signal features of these entity elements.

  Furthermore, using the learned information, it is possible to present information indicating a moving image or sound arrangement candidate in the user edited video to the user.

  In other words, the video editing support apparatus can learn how the shots having what characteristics are arranged in the sample video, for example. In addition, it is possible to learn what sound is synchronized with what kind of shot.

  Furthermore, the information on the arrangement of such shots and sounds in the sample image is compared with the signal characteristics of the shots of the material video. As a result, information indicating a shot arrangement candidate in the user-edited video in accordance with the shot arrangement in the sample image can be presented to the user.

  Further, the analysis means indicates the relationship between a boundary between adjacent moving images divided into a predetermined unit and a boundary between adjacent sounds divided into a predetermined unit as information indicating the synchronization configuration. The information or the information indicating the relationship between the feature of the moving image and the feature of the sound added to the moving image may be obtained, and the analysis unit divides the information into the predetermined units as the information indicating the time-series configuration. Then, it is possible to obtain information indicating the relevance of the signal features of the adjacent moving images and the relevance of the signal features of the adjacent sounds divided into predetermined units.

  In this way, by learning the relationship between the boundary of the moving image and the boundary of the sound, for example, in the user-edited video, the arranged shot and the sound change point (segment boundary) can be automatically synchronized. it can. Further, for example, it is learned that one background music (BGM) is added across two consecutive shots in the sample video.

  In addition, by learning the relationship between the characteristics of the video and the characteristics of the sound added to the video, it is learned in the sample video that, for example, fast-beating music is added to a shot with intense movement. Is learned.

  Further, by learning the relationship between adjacent similar element media, for example, it is possible to learn features such as the shot lengths of consecutive shots are often similar.

  The video editing support apparatus of the present invention further includes sample receiving means for receiving the sample video and a source video that is a video before editing the sample video, and the sample feature extracting means further includes the source video The signal features of the moving image and the sound, which are entity elements included in the video, are extracted, and the analyzing means further includes the signal feature of the moving image or the sound included in the sample video and the moving image or the sound included in the source video By contrasting with the signal characteristics, information indicating what kind of characteristics of the source video is extracted as the video of the sample video may be obtained.

  Thereby, for example, it is possible to learn an editing pattern such as which shot position in the source video is extracted to become a sample video shot.

  The video editing support apparatus according to the present invention further includes an instruction receiving unit that receives the user instruction and an instruction received by the instruction receiving unit after the candidate output unit outputs information indicating the arrangement candidate. The editing execution means for obtaining the user-edited video by arranging two or more entity elements included in the material video, and the arrangement of the entity elements in the user-edited video obtained by the editing execution means, Video output means for visualizing and outputting the images in a form corresponding to the signal features extracted by the sample feature extracting means may be provided.

  Thereby, the user can be made to recognize effectively how to arrange shots and the like in the user edited video. That is, it is possible to further improve the efficiency of the user's editing work.

  Further, the storage device stores a plurality of additional audio signal features that are entity elements added to the user-edited video, and the candidate generation unit further includes editing information learned by the learning unit, By comparing the plurality of signal features stored in the storage means, information indicating candidates for arranging additional sounds to be added to the user-edited video is generated, and the candidate output means further includes the candidate generation Information indicating candidates for the arrangement of the additional sounds generated by the means may be output, and further comprising storage means for storing a plurality of signal characteristics of visual effects that are entity elements added to the user-edited video, The learning means is an entity element further included in the sample video by the analysis means analyzing the signal characteristics of the moving image extracted by the extraction means. By obtaining information relating to the arrangement of visual effects added to the video, the information relating to the arrangement of visual effects is learned as editing information, and the candidate generating means includes the editing information learned by the learning means, and the storage By comparing a plurality of signal features stored in the means, information indicating candidates for arranging visual effects to be added to the user edited video is generated, and the candidate output means further includes the candidate generation means Information indicating a candidate for the arrangement of the visual effects selected by may be output.

  Thus, the sound to be added to the user-edited video or the type of visual effect can be presented to the user together with the arrangement candidate.

  For example, it is possible to present to the user a candidate for a time position in a user-edited video to which music is to be arranged, or a candidate for music to be arranged so as to be synchronized with a certain shot in space.

  The video editing support apparatus according to the present invention further includes an instruction receiving unit that receives the user instruction and an instruction received by the instruction receiving unit after the candidate output unit outputs information indicating the arrangement candidate. The editing execution means for obtaining the user-edited video by arranging two or more entity elements included in the material video and the video output means for outputting the user-edited video obtained by the editing means Also good.

  Accordingly, it is possible to present to the user candidates for how to arrange moving images and the like generated according to the editing information obtained from the sample video, and to provide the user with an environment for performing actual editing processing.

  As described above, the video editing support apparatus of the present invention can learn from the sample video as editing information the editing technique or editing knowledge of the editor who performed the editing. Further, according to the editing information, candidates for arrangement of entity elements such as a moving image of the material video can be presented to the user.

  Furthermore, the present invention can be realized as a method in which the operations performed by the characteristic components in the video editing support apparatus of the present invention are steps, or can be realized as a program including those steps, or a CD in which the program is stored. It can also be realized as a storage medium such as a ROM or an integrated circuit. The program can also be distributed via a transmission medium such as a communication network.

  As described above, the present invention can learn editing information from a sample video. Thereby, for example, the editing technique or editing knowledge of the expert can be learned as editing information from the sample image produced by the video editing expert.

  Further, according to the editing information, candidates for arrangement of entity elements such as moving images on the user edited video can be presented to the user. That is, it is possible to guide the user's editing behavior in a more appropriate direction based on the learned video editing specialist's editing technique and the like.

  That is, the present invention can provide a video editing support apparatus for a general user who does not have video editing technology and knowledge to edit video content effectively and efficiently.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the configuration of the video editing support apparatus 1 according to the embodiment will be described with reference to FIGS.

  FIG. 1 is a diagram schematically illustrating the configuration and functions of a video editing support apparatus 1 according to the embodiment.

  The video editing support apparatus 1 is an apparatus for effectively and efficiently editing information by presenting information for editing to the user when the user edits the video content.

  As shown in FIG. 1, the video editing support apparatus 1 includes a learning unit 2 and an editing unit 3.

  The learning unit 2 is a configuration unit that learns editing information, which is information related to how elemental media such as moving images are arranged in the video content, from the video content produced by editing by a video editing specialist.

  The editing unit 3 is a component that guides the user's editing behavior on an interface such as the display device 30. Also, it has a function of outputting video content edited by the user's editing action.

  The edited video that is subject to editing information learning in the learning unit 2 is hereinafter referred to as a “sample video”. In the present embodiment, the learning unit 2 learns editing information from a movie preview video as a sample video. The video content being edited by the user and the video content that has been edited and output from the editing unit 3 are hereinafter referred to as “user edited video”.

  The learning unit 2 separates the sample video into moving images, sounds, and element media such as editing effects, and then extracts low-level features (signal features) of each element related to the sensory structure. Furthermore, the sensory structure of the sample video is learned from the extracted features.

  Hereinafter, the term “element” simply refers to an element medium such as a moving image. In addition, each of element media such as moving images and sounds included in the video content is an example of a substantial element in the video editing support apparatus of the present invention.

  Low-level features related to sensory structure include, for example, shot length, shot type (tight shot, middle shot, long shot, etc.), motion, brightness, and time series in the video included in the sample video It is a time position that is a position.

  Also, the volume, accent, and type (music, voice, sound effect, etc.) of the sound included in the sample video. Also, the type of editing effect added to the sample video, the appearance position on the time series, and the duration.

  FIG. 2 is a diagram schematically showing the sensory structure of the sample video that the learning unit 2 learns.

  The learning unit 2 learns the basic unit of the moving image as a shot and the time series pattern of the shot as a time structure of the shot, that is, a time series configuration. For example, information indicating the relevance of signal characteristics of adjacent shots is learned as a time-series configuration.

  As a result, for example, it is learned as a time-series composition of shots that shots having the same length in many cases are continuous and that the movement of the video is fast in the second half of the sample video.

  This time-series composition of shots is used by the editing unit 3 to generate information indicating candidates for time positions on the user-edited video where the shot selected by the user should be placed, for example.

  Similarly, the time structure of the sound stream, that is, the time series structure is learned. The time series structure of the sound stream in the sample video is used in the editing unit 3 to select additional sound candidates to be added to the shot selected by the user for placement in the user edited video, for example.

  Further, the learning unit 2 divides the sound stream included in the sample video corresponding to a moving image frame or shot unit. Furthermore, (1) the relationship between boundaries is learned. For example, information indicating the relevance of the boundaries, such as information indicating the coincidence / non-coincidence of the time positions between the moving image boundary and the sound boundary, is learned. (2) Learn the relationship between segments. For example, information such as what kind of sound is added to what kind of shot is learned.

  That is, the learning unit 2 learns the spatial structure of the moving image and the sound, in other words, the synchronized configuration of the moving image and the sound, for the two points.

  In the editing unit 3, the above (1) is, for example, when the shot has already been arranged at the time position where the user arranged the additional sound in the user edited video, and the change point (segment boundary) between the shot and the sound is automatically set. Used to synchronize automatically. The above (2) is used, for example, to select additional sound candidates to be arranged corresponding to shots already arranged in the user edited video.

  The synchronous configuration is information about what element media are combined at a certain point in time or period. That is, it is information about a combination of element media existing in the same space in time series.

  The learning unit 2 extracts, for example, the movement speed of the video in the shot and the beat speed of the music added to the shot as a synchronized configuration of the moving image and the sound. It learns that BGM with a fast beat is added.

  The learning unit 2 outputs the time-series configuration and the synchronous configuration of the elemental media such as moving images learned from the sample video to the editing unit 3 as editing information.

  The editing unit 3 receives an input of a material video that the user wants to edit from the user. Further, it receives an instruction to select a shot to be inserted into the user-edited video, an additional element that is an element medium to be added to the user-edited video, and a time position of the shot in the user-edited video. That is, various editing actions performed by the user are accepted.

  The editing unit 3 has a plurality of types of sound and editing effect data as additional elements to be added to the user edited video during video editing.

  The editing unit 3 can guide the user's next editing action on the interface according to the editing information of the sample video obtained from the learning unit 2 and the user's editing action.

  For example, when a user selects a shot as a shot to be arranged on the user edited video, a candidate position that is a candidate of a time position on the user edited video on which the shot is to be arranged is presented to the user according to the editing information. be able to. Further, it is possible to present the user with an additional sound that is most suitable for the shot, for example, a BGM candidate.

  FIG. 3 is a functional block diagram showing a functional configuration of the video editing support apparatus 1.

  As described above, the video editing support apparatus 1 includes the learning unit 2 and the editing unit 3, and further includes the output unit 4.

  The learning unit 2 includes a sample reception unit 20, a sample feature extraction unit 21, a sample analysis unit 22, and a model generation unit 23. The sample receiving unit 20 is a component that receives an input of a sample video such as a movie preview video.

  The sample feature extraction unit 21 is a component that extracts signal features of moving images and sounds, which are element media included in the sample video.

  The sample analysis unit 22 analyzes the signal feature extracted by the sample feature extraction unit 21 to obtain information indicating the time-series configuration of each of the moving image and the sound included in the sample video and the synchronized configuration of the moving image and the sound. It is a component which obtains the information shown.

  The model generation unit 23 uses a time / space model (hereinafter referred to as “edit model”) of the element media included in the sample video from the information indicating the time-series configuration obtained by the sample analysis unit 22 and the information indicating the synchronous configuration. .) Is generated and output to the editing unit 3. That is, the editing model as editing information is output to the editing unit 3.

  The editing unit 3 includes a material reception unit 10, a material feature extraction unit 11, a user input reception unit 12, a candidate generation unit 13, an effect data storage unit 14, an additional sound storage unit 15, and an editing execution unit 16. Have.

  The material reception unit 10 is a configuration unit that receives a material video input from a user. The material feature extraction unit 11 is a configuration unit that extracts signal features of moving images and sounds that are elemental media included in the material video.

  The material receiving unit 10 also receives additional sound input from the user and effect data that is data for adding an editing effect to the user edited video. These additional sound and effect data are respectively stored in an additional sound storage unit 15 and an effect data storage unit 14 which will be described later.

  The user input receiving unit 12 is a component that receives an input of an instruction such as a shot selection input from the input device 40 such as a keyboard.

  The candidate generation unit 13 is a configuration unit that generates information indicating candidates for arrangement in the user edited video of the elemental media included in the material video according to the editing information learned by the learning unit 2.

  Specifically, the candidate generation unit 13 should be arranged at a time position candidate on the user edited video where element media such as shots, additional sounds, and editing effects should be arranged, or at a certain time position on the user edited video. A candidate of elemental media such as a shot is generated as a candidate for arrangement of elemental media in the user edited video. That is, information indicating candidates for arrangement of element media in the user edited video is generated.

  The effect data storage unit 14 is a storage device that stores effect data. The effect data storage unit 14 stores a plurality of types of effect data.

  Specifically, the effect data storage unit 14 stores a plurality of types of visual effect data to be added to the user edited video as effect data, and further stores information indicating the characteristics of each visual effect. ing. Specifically, the information indicating the feature of the visual effect is a feature vector derived from the signal feature of the visual effect data. The feature amount vector indicates the type and duration of the visual effect. The editing effect is an example of the visual effect in the video editing support apparatus of the present invention.

  The additional sound storage unit 15 is a storage device that stores a plurality of types of additional sounds to be added to the user-edited video and information indicating the respective characteristics. Specifically, the information indicating the characteristic of the additional sound is a feature amount vector derived from the signal characteristic of the additional sound. This feature vector indicates the volume of the additional sound, the speed of the beat, and the like. The additional sound storage unit 15 stores music, sound effects, and the like as a plurality of types of additional sounds.

  The candidate generation unit 13 should be arranged in the user-edited video from the plurality of types of effect data and the plurality of types of additional sound stored in the effect data storage unit 14 and the additional sound storage unit 15, that is, A candidate for effect data and additional sound to be added to the user-edited video is selected.

  The editing execution unit 16 is a configuration unit that obtains a user-edited video by arranging and connecting elemental media such as moving images in shot units in accordance with a user instruction.

  The output unit 4 is a processing unit that outputs an editing screen, a user edited video, and the like to the display device 30 that is an interface between the user and the video editing support device 1. Note that the output unit 4 outputs sound to a speaker (not shown). The contents of the editing screen output from the output unit 4 and displayed on the display device 30 will be described later with reference to FIG.

  Next, the operation of the video editing support apparatus 1 according to the present embodiment will be described with reference to FIGS.

  FIG. 4 is a flowchart showing an outline of an operation flow related to editing support of the video editing support apparatus 1. An outline of operations related to editing support of the video editing support apparatus 1 will be described with reference to FIG.

  The sample video input to the video editing support device 1 is received by the sample receiving unit 20 (S1). The received sample video is analyzed by the sample feature extraction unit 21 and the sample analysis unit 22 (S2). The operation content of the video analysis will be described later with reference to FIG. The model generation unit 23 generates a sample video editing model, which is editing information obtained from the sample video (S3).

  The material video input by the user to the video editing support device 1 is received by the material receiving unit 10 (S4). The accepted material image is analyzed by the material feature extraction unit 11 (S5).

  Thereafter, according to the editing model generated by the model generation unit 23, the candidate generation unit 13 and the like provide editing support for efficiently obtaining a user edited video from the material video (S6). The details of the editing support operation will be described later with reference to FIG. The user produces a user edited video while receiving editing support, and the user edited video is output from the output unit 4 (S7).

  FIG. 5 is a flowchart showing an outline of an operation flow related to video analysis (S2 and S5 in FIG. 4) of the video editing support apparatus 1. The outline of the operation related to the video analysis of the video editing support apparatus 1 will be described with reference to FIG. Detailed description of video analysis performed by the video editing support apparatus 1 will be described later with reference to FIGS.

  Note that the video analysis for the sample video (S2 in FIG. 4) and the video analysis for the material video (S5 in FIG. 4) are the same as the main operation contents, so the video analysis for the sample video will be described as an example. .

  When the sample video is input, the sample feature extraction unit 21 detects a cut in the moving image included in the sample video (S10). Thereby, the moving image included in the sample video can be divided into shot sequences (S11). This is because one shot is formed between the cuts.

  Each shot obtained by dividing the moving image is divided into frame sequences (S12). A frame is a unit constituting a shot.

  The sample feature extraction unit 21 extracts a feature amount indicating a signal feature of each shot from each shot and each frame constituting the shot (S13).

  Further, a feature quantity vector representing these feature quantities as a multidimensional vector is calculated (S14). For example, as the feature amount vector, a four-dimensional feature amount vector of the shot length, the motion of the video in the shot, the average brightness of the shot, and the volume of the sound added to the shot is calculated.

  The sample analysis unit 22 analyzes each calculated feature vector. Specifically, by normalizing each feature quantity vector, the influence due to feature quantity variation is removed. Further, clustering is performed based on a predetermined similarity (S15). That is, the shots are grouped based on the feature similarity.

  In addition, the feature vector of each shot includes a component about sound. Therefore, it is possible to analyze what sound is arranged in time series in the sample video. Note that the sample feature extraction unit 21 and the sample analysis unit 22 also provide one-dimensional or more features for only the sound from the signal feature indicating the volume or the speed of the beat for the sound included in the sample video as well as the shot. The calculation of the quantity vector may be performed separately.

  Further, it is possible to extract features such as the type of editing effect added to the sample video from the feature quantity vector of each shot. That is, it is possible to analyze what editing effect is added to what shot. It should be noted that a component that clearly indicates the feature of the editing effect may be included as a component of the shot feature vector. Furthermore, a one-dimensional or more feature quantity vector for only the editing effect may be separately calculated from the signal characteristics indicating the type of editing effect and the start and end time positions.

  In this way, the sample analysis unit 22 determines what kind of shots and sounds are arranged in the time series in the sample video, and what kind of shots have the characteristics. It is possible to obtain information on whether the sound is related to the sound and the editing effect.

  That is, it is possible to obtain information indicating a time-series configuration of each of a plurality of elementary media such as a moving image included in the sample video and information indicating a synchronous configuration between different types of elementary media.

  Note that the above operation is also performed on the material feature extraction unit 11 for the material video, and each shot included in the material video is also divided into a plurality of groups based on the feature similarity.

  Thereafter, a symbol string representing the feature amount and the like of each shot is generated (S16) and output to the model generation unit 23.

  The model generation unit 23 generates an editing model that is editing information for one sample video by collecting information such as the feature amounts of each shot and their similarity obtained from one sample video. The generated editing model is output to the candidate generation unit 13.

  FIG. 6 is a diagram showing an example of an editing screen displayed on the display device 30 as a result of the operation flow shown in FIG.

  The editing screen 31 shown in FIG. 6 is a screen for visualizing the time / space structure of the user edited video and guiding the user's editing behavior according to the editing model.

  As shown in FIG. 6, the editing screen 31 includes a time input field 32, a material video timeline field 33, a shot selection field 34, an additional element selection field 35, a playback display field 36, and a story board 37. The structure display board 38 is displayed.

  The time input field 32 is a field for accepting an input of a time indicating the length when the user edited video is completed. For example, when the user desires to edit a one-hour material video and produce a five-minute user-edited video, “5 minutes” is input to the time input field 32 by the user.

  The material video timeline column 33 is a column for displaying a shot of the material video in a timeline. Each rectangle in the material video timeline column 33 represents one shot. For example, the type of shot is represented by color.

  In FIG. 6, a pattern such as a dot attached to each rectangle is represented by a color on the screen. The type of shot is, in other words, a characteristic of the shot.

  That is, rectangles with the same pattern represent shots having similar characteristics. As described above, the material feature extraction unit 11 groups the shots according to the feature similarity. As a result, the rectangle representing the shot can be displayed with a different color for each group.

  For example, the dots in the figure indicate red on the screen, and the rectangles with red belong to a group of shots with high brightness and high-luminance shots. In addition, other patterns are also given a color other than red, indicating that they belong to a group associated with a predetermined feature.

  Therefore, the user can visually grasp how and what type of shots are arranged in the material video.

  The expression contents of the pattern attached to the rectangle are the same in the shot selection field 34, the story board 37, and the structure display board 38 described below. Further, like the shot, the rectangle representing the additional sound is also given a pattern according to the type of the additional sound. That is, some or all of the colors are colored on the screen.

  Further, the expression contents of these patterns are the same in FIGS. Hereinafter, in the description of the diagrams representing these editing screens 31, rectangles representing shots and additional sounds are simply referred to as shots and additional sounds, respectively.

  The shot selection column 34 is a column for allowing the user to select a shot to be arranged in the user edited video. Each shot is displayed in groups. The user can select a shot to be included in the user-edited video and drag it onto the storyboard 37 described below to place the shot on the user-edited video.

  In FIG. 6 and FIGS. 8 to 12 to be described later, in order to clarify the features of the present invention, each rectangle in the shot selection field 34 is illustrated as displaying only a given color. Specifically, however, an image of the first frame of each shot is displayed except for a part of the peripheral edge of the rectangle. Thereby, the user can recognize the content of the shot corresponding to each rectangle. The same applies to the story board 37 described later.

  In one group, a plurality of shots are displayed in an overlapping manner. In this state, when any shot is clicked by the user, all shots belonging to the group are displayed.

  For example, each shot moves in the vertical direction or the horizontal direction so as to reduce the overlapping portion of the shots at least to the extent that the user can recognize the contents.

  Alternatively, shots other than the foremost shot move to the forefront of the group when the displayed portion is clicked by the user.

  That is, the user can recognize the contents of each shot regardless of which shot belongs to which group.

  The additional element selection column 35 is a column for allowing the user to select an additional element that is an element medium to be added to the user edited video. Various editing effects and additional sounds are prepared as such elemental media.

  The user can add an additional element to the user-edited video by selecting an additional element to be added to the user-edited video and dragging it to the storyboard 37 described below.

  The reproduction display column 36 is a column for reproducing each shot and a user-edited video during editing and after completion of editing, and allowing the user to confirm the contents of each.

  When the user selects a shot or an additional sound, the storyboard 37 allows the user to check the time series configuration and the synchronization configuration of the shot and the additional sound, and moves the shot and the additional sound, centering on the arrangement position. It is a column to make it.

  As shown in the figure, the story board 37 is composed of a video story column 37a and a sound story column 37b. The shots arranged in the video story column 37a are displayed, and the additional sounds arranged in the sound story column 37b are displayed.

  Thereby, the user can confirm not only how shots and additional sounds are arranged in a time series but also a synchronized configuration of any shot and additional sounds.

  The structure display board 38 is a column for allowing the user to confirm the entire structure of the user edited video. As shown in the figure, the structure display board 38 is composed of a video structure column 38a and a sound structure column 38b. The shots arranged in the video structure column 38a are displayed, and the additional sounds arranged in the sound structure column 38b are displayed.

  In the video structure field 38a and the sound structure field 38b, the relationship between the element media can be visualized by the position of the same type of element media in the vertical axis direction.

  For example, the position of each shot and each additional sound shown in the figure in the vertical axis direction is determined by the group to which the shot and the additional sound belong. That is, shots with the same color are displayed at the same height. In addition, additional sounds with the same color are displayed at the same height.

  By referring to the structure display board 38, the user can confirm the entire structure of the user edited video during editing and after editing is completed.

  In this embodiment, element media such as shots are represented by colored rectangles to visualize how element media are arranged. However, the arrangement of element media may be visualized by other methods.

  For example, the shots grouped according to the signal characteristics may be represented by images having different shapes for each group, so that the shot arrangement in the user edited video may be visualized and output. In other words, the arrangement of the element media in the user-edited video may be visualized and output by arranging the images in a mode corresponding to the signal characteristics of the element media.

  The operation of the video editing support apparatus 1 when the user edits the material video on the editing screen 31 displayed in this way will be described with reference to the editing screen 31 of FIGS. This will be described with reference to a display example.

  FIG. 7 is a flowchart showing a flow of operations related to editing support (S6 in FIG. 4) of the video editing support apparatus 1.

  The user input receiving unit 12 receives a user input (S20). Candidate generator 13 determines what selection the input indicates. As a result of the determination, if it is an element (element in S21), the degree of matching between the selected element and the element indicated in the edit model is calculated (S22).

  Based on the calculation result, the candidate generation unit 13 presents to the user, through the output unit 4, a candidate for a time position where the element selected by the user should be arranged, that is, the candidate position (S23).

  FIG. 8 is a diagram illustrating an example of the edit screen 31 when one shot is selected by the user. In FIG. 8, A1 to E1 attached to shots in the shot selection column 34 are shot identifiers.

  For example, as illustrated in FIG. 8, (1) when a shot C2 is selected by the user, the candidate generation unit 13 acquires the information. Further, the feature vector of the shot C2 received from the material feature extraction unit 11 and the time-series configuration of shots in the sample video or the synchronous configuration of shots and sounds shown in the editing model received from the model generation unit 23 are illuminated. By matching, that is, by contrasting, information indicating candidates for arrangement in the user edited video of the shot C2 is generated.

  Specifically, the distance between the feature vector of the shot C2 and each of the plurality of feature vectors shown in the editing model is calculated, and the shorter the distance, the higher the matching of the feature vector in the editing model is calculated. To do.

  As a result of the calculation, the time position of the feature vector whose degree of conformity satisfies a predetermined criterion is specified from the editing model. The time position in the user edited video corresponding to the time position in the specified sample video is presented to the user.

  For example, it is assumed that the total length of the sample video is 6 minutes 30 seconds, and the time position of the feature vector satisfying the predetermined criterion is 1 minute 30 seconds. In this case, the time position of the feature vector is a time position corresponding to 20% of the total length. Therefore, when the total length of the user edited video is 5 minutes, the time position in the user edited video corresponding to the time position is 1 minute corresponding to 20% of 5/5.

  Note that feature vectors such as shots of material video and sample video are normalized and clustered. That is, elements such as shots are grouped according to the characteristics of each element.

  Therefore, for example, a group in the sample video having the same or similar characteristics as the group in the material video to which the selected shot belongs is specified. It is assumed that one or more shots belonging to the specified group are shots that satisfy a predetermined standard as they are. Furthermore, the time position in the user edited video corresponding to the time position in the edited video of the one or more shots may be specified and presented to the user.

  On the editing screen 31, as shown in FIG. 8, (2) the corresponding position is displayed in the video story column 37a as a candidate position (represented by a region surrounded by a dotted line in the figure).

  As described above, when the user selects a shot, the video editing support apparatus 1 can present the user with candidate positions where the shot is to be arranged. This candidate position is selected by referring to an editing model created from a sample video produced by editing by a video editing specialist.

  In other words, the video editing support apparatus 1 can guide a user who is not a video editing expert to place a shot selected by the user at an appropriate time position without repeating unnecessary trial and error. it can.

  Next, as shown in the flowchart of FIG. 7, the video editing support apparatus 1 detects the movement of the element by the user (S26). Specifically, when the shot C2 is moved into the video story column 37a by the user, the editing execution unit 16 detects the movement. Also, information such as rewriting of the display contents of the edit screen 31 and the arrangement position of the shot C2 on the user edited video is recorded (S27).

  Thereafter, when there is no end instruction from the user (No in S28) and an instruction such as shot selection is input from the user, information for actual editing is recorded from the user input reception (S20) (S27). Is repeated. When there is an end instruction from the user (Yes in S28), the editing execution unit 16 performs an actual editing process (S29) according to the recorded information. That is, according to the recorded information, data such as shots selected by the user are connected to generate one user edited video file.

  FIG. 9 is a diagram illustrating an example of the editing screen 31 when one shot is moved to the story board 37 by the user.

  As shown in FIG. 9, (3) when the user moves the shot C2 to the candidate position in the video story column 37a, the movement is reflected in the (4) video structure column 38a.

  The rewriting of the image display is performed by the editing execution unit 16. Further, the editing execution unit 16 adds the data of the shot C2 to the data of the user edited video held therein so that the shot C2 is arranged at the time position of the user edited video.

  Also, as shown in the flowchart of FIG. 7, when the user selects a time position on the storyboard 37 (time position in S21), the candidate generation unit 13 sets an editing model corresponding to the time position. The degree of matching between the displayed element and the element displayed on the editing screen 31 is calculated (S24).

  Based on the result of the calculation, the candidate generation unit 13 presents to the user a candidate for an element to be placed at the time position selected by the user, that is, a candidate element, via the output unit 4 (S25).

  FIG. 10 is a diagram illustrating an example of the editing screen 31 when the time position is selected by the user.

  For example, as shown in FIG. 10, (1) when the user selects a time position on the video story column 37a (represented by a region surrounded by a dotted line in the figure), the candidate generator 13 , Get that information.

  Further, the feature quantity vector of the shot at the time position in the edit model corresponding to the selected time position is read from the edit model. The read feature quantity vector is compared with the feature quantity vector (hereinafter referred to as “material shot vector”) of each shot displayed in the shot selection field 34, and the degree of matching is calculated.

  Specifically, each distance between the read feature vector and the material shot vector is calculated, and the shorter the distance is, the higher the compatibility of the material shot vector is.

  As a result of the calculation, a shot whose fitness satisfies a predetermined standard is specified. As shown in FIG. 10, the identified shot is displayed as a candidate element by (2) blinking or the like on the screen.

  In the case of the display example shown in FIG. 10, shot B1 and shot C2 are displayed as candidate elements.

  Thereafter, when any shot is selected and moved onto the video story column 37a by the user, the video editing support device 1 detects the movement of the element by the user as shown in the flowchart of FIG. 7 (S26). ). Further, the edit execution unit 16 records information such as rewriting of the display contents of the edit screen 31 and the arrangement position of the shot C2 on the user edited video (S27).

  Thereafter, when there is no end instruction from the user (No in S28) and an instruction to select a time position or the like is input from the user, information for actual editing is recorded from the user input reception (S20) (S20). The steps up to S27) are repeated. When there is an end instruction from the user (Yes in S28), the editing execution unit 16 performs an actual editing process (S29) according to the recorded information and generates one user edited video file.

  As described above, the video editing support apparatus 1 allows the user to select candidate elements to be arranged at the time position not only when the user selects an element such as a shot but also when the user selects a time position in the user edited video. Can be presented.

  That is, it is possible to guide a user who is not a video editing expert to place a shot suitable for the time position at the time position selected by the user without repeating unnecessary trial and error.

  Further, the video editing support apparatus 1 can guide the user to place the additional element selected by the user at a suitable time position of the user edited video.

  FIG. 11 is a diagram illustrating an example of the edit screen 31 when one additional sound is selected by the user.

  For example, as illustrated in FIG. 11, (1) when “music 1” is selected by the user, the candidate generation unit 13 acquires the information. Furthermore, the feature amount vector of “Music 1” read from the additional sound storage unit 15 is compared with the plurality of feature amount vectors in the edit model received from the model generation unit 23, and the degree of matching is calculated.

  Specifically, the feature amount vector of “Music 1” is compared with the component representing the feature amount of the sound included in the plurality of feature amount vectors in the editing model.

  By this comparison, the feature vector in the editing model is specified that has a short distance from the feature vector of “Music 1”, that is, the degree of matching with the feature vector of “Music 1” satisfies a predetermined criterion. .

  Furthermore, a shot that has already been arranged in the user-edited video and has a feature vector whose distance from the specified feature vector is within a predetermined range is specified. That is, a shot to which “music 1” is to be added is specified. In addition, the time position of the shot is specified.

  It should be noted that the time position of the feature vector in the editing model that satisfies the predetermined standard for the degree of matching with the feature vector of “Music 1” is specified, and the time position in the user edited video corresponding to the time position May be specified. That is, the time position on the user-edited video where the music 1 ″ is to be placed may be specified without the presence of a shot.

  The time position on the user-edited video specified in this way is displayed as a candidate position in the (2) sound story column 37b as shown in FIG.

  Thereafter, when the user moves “Music 1” to the sound story column 37b, the editing execution unit 16 detects the movement. When the edit execution unit 16 detects the movement, it rewrites the display contents. Further, the data of “music 1” is added to the data of the user edited video held inside so that “music 1” is arranged at the time position of the user edited video.

  Further, the video editing support apparatus 1 can present the candidate position to the user even when the editing effect is selected by the user, as in the case where the additional sound is selected.

  FIG. 12 is a diagram illustrating an example of the editing screen 31 when the editing effect which is one additional element is selected by the user.

  For example, as illustrated in FIG. 12, (1) when “fade out” is selected by the user, the candidate generation unit 13 acquires the information. Further, the “fade-out” feature quantity vector read from the effect data storage unit 14 is compared with the plurality of feature quantity vectors in the edit model received from the model generation unit 23, and the respective matching degrees are calculated.

  Specifically, the “fade out” feature quantity vector and the component representing the feature quantity of the editing effect included in the plurality of feature quantity vectors in the editing model are compared.

  By this comparison, the feature vector in the edit model is identified that has a short distance from the “fade out” feature vector, that is, the degree of matching with the “fade out” feature vector satisfies a predetermined criterion.

  Furthermore, a shot that has already been arranged in the user-edited video and has a feature vector whose distance from the specified feature vector is within a predetermined range is specified. That is, a shot to which “fade out” is to be added is specified. In addition, the time position of the shot is specified.

  Using the time position on the user-edited video thus identified, as shown in FIG. 12, (2) in the video story column 37a, between the shot to which “fade out” should be added and the next shot, For example, a dotted vertical rectangle is displayed.

  Thereafter, when the user moves “Fade Out” to a vertically long rectangle in the video story column 37a, the editing execution unit 16 detects the movement. When the edit execution unit 16 detects the movement, it rewrites the display contents. Further, the fade-out effect data is added to the user-edited video data stored therein so that the fade-out is added to the shot immediately before the time position of the user-edited video.

  When an edit effect to be added at the beginning of a shot such as “fade in” is moved to the above-described vertically long rectangle, the edit effect is added to the shot immediately after the vertically long rectangle.

  When the user selects a vertically long rectangle arranged between shots on the video story field 37a of the editing screen 31, the video editing support apparatus 1 shots immediately before or after the vertically long rectangle. Editing effect candidates suitable for addition to the user can also be presented to the user.

  In this case, for example, a dotted vertical rectangle is displayed before and after the shot arranged in the video story column 37a. Further, the candidate generation unit 13 may specify an editing effect to be added to the shots from the feature vector and the editing model of the shots before and after the selected vertically long rectangle.

  Thereby, for example, when the shot immediately after the selected vertical rectangle is dark and the shot of the video is slow, one or more shots having the same characteristics in the sample video are specified. Further, one or more editing effects added to the start portion of the one or more shots are specified, and the specified editing effects are candidates for editing effects to be added to the shot immediately after the vertical rectangle selected by the user. Presented to the user.

  Also, the editing effect to be added to the end portion of the shot is specified from the characteristics of the shot immediately before the selected vertically long rectangle. Further, the specified editing effect is presented to the user as a candidate for an editing effect to be added to the shot immediately before the vertically long rectangle selected by the user.

  The user selects one editing effect from the one or more editing effects presented in this way, and moves it onto a vertically long rectangle. Thereby, the editing effect is added to the shot immediately before or after the vertically long rectangle according to the type of the editing effect.

  Further, when an editing effect is added to a certain shot and a sound is added to the shot, the sound is processed so as to be synchronized with the editing effect. For example, when “fade out” is added to the shot as an editing effect, the sound is also processed to fade out.

  An editing effect can also be determined based on the characteristics of the sound added to the shot.

  For example, in the user-edited video, it is assumed that sound has already been added to the shot immediately before the vertically long rectangle selected by the user, and the sound has faded out at the end of the shot. In this case, the material feature extraction unit 11 can detect that the volume is reduced at the end portion of the shot as the signal feature.

  In addition, the learning unit 2 learns that, as a feature of each shot of the sample video, if the added sound fades out, the moving image also fades out, that is, “Fade Out” is added as an editing effect. Assuming that

  In this case, the candidate generation unit 13 selects “fade out” as a candidate for the effect to be added to the shot on the user edited video based on the learning content by the learning unit 2 and is presented to the user via the output unit 4. .

  In other words, in video editing, the relationship between editing effects, which are elements of visual features, and sound features, which are auditory features, is important. Therefore, the video editing support apparatus 1 learns information indicating the relationship between the segment boundaries of sound and the segment boundaries of moving images from the sample video as information indicating the synchronization configuration of these boundaries. Furthermore, it is possible to guide the user's editing behavior in a more appropriate direction using the learning content.

  Note that two vertically long rectangles may be displayed side by side between each shot. That is, for example, when two shots A and B are arranged in time series in the order of AB, a vertically long rectangle immediately after A and a vertically long rectangle immediately before B may be displayed side by side. Thereby, the user can be made to explicitly recognize whether the editing effect selected by the user or the vertically long rectangle selected by the user corresponds to which shot of A and B.

  Similarly, when the time position is selected by the user on the sound story column 37b of the edit screen 31 and a shot has already been placed at the time position on the video story column 37a, the video editing support apparatus 1 The sound candidate suitable for adding to the shot can be presented to the user.

  In this case, the candidate generation unit 13 may specify the sound to be added to the shot from the feature vector and the edit model of the shot on the video story column 37a corresponding to the selected time position.

  Thereby, for example, when the shot at the selected position is bright and the shot has a strong motion, one or more shots having the same characteristics in the sample video are specified.

  Further, one or more types of sounds added to the one or more shots are specified, and the specified one or more types of sounds are compared with the sounds stored in the additional sound storage unit 15. Specifically, each feature vector is compared.

  Note that the sound specified here includes not only sound effects and music, but also “silence” indicating a state where there is substantially no sound.

  As a result of this comparison, one or more types of sounds similar to the specified one or more types of sounds are identified from the sounds stored in the additional sound storage unit 15. The specified one or more types of sounds are presented to the user as sound candidates to be placed at the time position selected by the user.

  Further, even when a time position is selected by the user on the sound story column 37b of the editing screen 31 and no shot is arranged at the time position on the video story column 37a, the video editing support device 1 can present a sound candidate suitable for the time position to the user.

  In this case, the candidate generation unit 13 may specify the sound to be placed at the time position from the time position corresponding to the selected time position and the edit model.

  Thereby, for example, when the selected time position is the time position immediately after the start of the user-edited video, the sound added to the video immediately after the start in the sample video is specified. For example, a gentle music having a signal feature such as a small volume is specified.

  Further, one or more music similar to the music is specified from the music stored in the additional sound storage unit 15 based on the signal characteristics of the specified music. The specified one or more pieces of music are presented to the user as music candidates to be placed at the time position selected by the user.

  In other words, the sound added to the video content is strongly dependent on the time position. For example, one example is that a gentle song is added to the start portion of the video content, and a song that makes the end portion feel exciting.

  Therefore, when it is useful to have only the time position selected by the user, select the sound candidates to be placed at that time position, and present to the user without considering what the video looks like Exists.

  Further, the additional sound and editing effect candidates can be presented to the user when the user places the shot on the user edited video.

  For example, when the user selects one shot displayed in the shot selection field 34, it is possible to specify a shot of the sample video having a feature close to that feature as described above. Therefore, the sound added to the sample video shot and the characteristics of the editing effect can also be specified.

  Therefore, from the characteristics of these sounds and editing effects, one or more additional sounds and editing effects having characteristics close to those characteristics are selected by the user and added sound and editing to be added to the shot moved to the video story column 37a. Can be presented as an effect candidate.

  As described above, even when the user selects an additional element, the video editing support apparatus 1 can generate information indicating candidates for time positions where the additional element should be arranged according to the editing model and present it to the user. it can. Similarly, even when the user selects a time position, information indicating additional element candidates to be placed at the time position can be generated according to the editing model and presented to the user.

  In other words, it is possible to guide a user who is not a video editing expert to arrange at a time position suitable for the additional element according to the additional element selected by the user without repeating unnecessary trial and error. it can. Further, according to the time position selected by the user himself / herself, the user can be guided to select an additional element suitable for the time position.

  As described above, when the user attempts to place one of the shots or the additional sound on the user edited video, the other may already be placed at the time position at which the user tried to place the shot.

  In this case, when the user executes the arrangement, the shot and the additional sound coexist at the same time position. In this case, it is considered that the shot length and the length of the additional sound (sound length) are often not the same. Therefore, in order to synchronize the shot and the additional sound, it is conceivable to match the shot length with the length of the additional sound added to the shot.

  When the shot length is shorter than the sound length, the video editing support apparatus 1 according to the present embodiment can synchronize the shot and the additional sound by cutting the additional sound according to the shot length. When the shot length is longer than the sound length, the shot and the additional sound can be synchronized by cutting the shot according to the sound length.

  FIG. 13 is a flowchart showing an outline of an operation flow of the video editing support apparatus 1 in consideration of shots or additional sounds that are already arranged.

  For example, when one shot is about to be placed at a certain time position on the video story column 37a by the user, when the shot is not placed at that time position (No in S30), the shot moves to that time position as it is. (S32).

  If a shot has already been placed at that time position (Yes in S30), the shot selected by the user is moved so as to overwrite the already placed shot (S31).

  Furthermore, when no additional sound is arranged at the time position in the sound story column 37b (No in S33), it is not necessary to adjust the length of the shot and the additional sound, and the processing is performed after performing the actual editing process. The operation related to is terminated.

  For example, when BGM is already arranged at the time position in the sound story column 37b (Yes in S33), the shot length and the BGM sound length are compared. This comparison is performed by the editing execution unit 16.

  As a result of the comparison, when the shot length is longer than the sound length (Yes in S34), the shot start position boundary and the BGM start position boundary are synchronized, and the shot is cut in accordance with the sound length. That is, the boundary of the shot end position and the boundary of the BGM end position are synchronized (S35).

  For example, when a one-minute shot is set to 30 seconds, shot data corresponding to other periods is cut so that 15 seconds to 45 seconds before that shot remain. That is, a shot having a length corresponding to the sound length is left around the center of the shot.

  For example, a shot having a length corresponding to the sound length may be left from the start of the shot, or for example, an arbitrary portion may be taken out to be a total of 30 seconds. In short, the shot length may be shortened to be the same length as the sound length.

  If the shot length is not longer than the sound length (No in S34), the actual editing process is performed and the operation related to the process is terminated. Specifically, when the shot length and the sound length are the same, the edit execution unit 16 stores the data of the user edited video stored therein so that the shot is arranged at the time position of the user edited video as it is. Append the shot data to.

  In addition, when the shot length is shorter than the sound length, for example, it is assumed that the sound length is 1 minute and the total length of one or a plurality of shots associated with the BGM is 30 seconds. In this case, when it is determined that the single or plural shots and the BGM are associated with each other, the BGM data is cut so as to leave only a portion corresponding to 30 seconds from the start, for example.

  If the BGM is associated with a plurality of shots and the total length of the plurality of shots is longer than the length of the BGM, the total number of shots is based on information indicating at which time position of the BGM the shot should be switched. It is determined whether or not to cut all shots so that the length falls within the length of the BGM.

  The information about the time position of the BGM at which the shot should be switched is learned from the sample video as information indicating the relationship between the sound segment boundary and the moving image segment boundary in the learning unit 2. is there.

  Further, the BGM is synchronized with the one or more shots, that is, the BGM data is added to the user edited video data so that the BGM is arranged at the same time position as the one or more shots. Will be added.

  Note that the start position of the BGM does not have to be the start position of the BGM music, and a position in the middle of the BGM may be handled as the start position according to a user's explicit instruction. The same applies to the case where sound effects other than music are added to the shot.

  Here, one shot included in the sample video may be obtained by extracting a part from one shot of the source video that is the source of the sample video or omitting a part thereof. is there.

  Therefore, the video editing support apparatus 1 learns how one shot of the sample video is extracted or omitted from the shot of the source video corresponding to the shot, and the material is determined according to the learned information. Support for shortening video shots may be provided.

  FIG. 14 is a diagram schematically showing a relationship between one shot of a sample video and a shot that is a source of the shot.

  For example, if one shot of the sample video (hereinafter referred to as “sample shot”) is taken as one shot of the movie preview video, the shot that is the source (hereinafter referred to as “source shot”) is one shot of the main movie. is there.

  The sample shot may be the corresponding source shot as it is, but as shown in FIG. 14, it may be produced by extracting only the necessary part from the source shot.

  In such a case, the video omission of the video performed by the editor who produced the sample shot from the video shot and sound signal characteristics of the source shot and the sample shot video and sound signal characteristics obtained from the source shot. Or it can learn how to extract.

  In this case, for example, in the learning unit 2, the sample receiving unit 20 receives input of the sample video and the source video. The sample feature extraction unit 21 calculates a feature vector such as the average luminance of the frame and the volume of sound added to the frame for each frame of one sample shot and the source shot corresponding to the sample shot.

  For example, a sample shot having a feature similar to the feature of the shot that the user is trying to shorten is selected as the sample shot that is a target for calculating the feature vector.

  Further, the sample analysis unit 22 normalizes the feature quantity vectors, and compares the feature quantity vectors of each frame of the source shot with the feature quantity vectors of each frame of the sample shot.

  Thereby, it is possible to learn what kind of signal characteristic of the source shot is extracted as the frame of the sample shot.

  For example, it is learned that a frame having a luminance higher than a certain luminance of the source shot is extracted as a sample shot frame. Further, for example, it is learned that a frame at a specific time position of the source shot is extracted as a frame of the sample shot.

  The model generation unit 23 outputs the information learned by the sample analysis unit 22 to the candidate generation unit 13 as an edit model for shot production.

  In the editing unit 3, the material feature extraction unit 11 calculates and normalizes a feature vector for each frame of a shot of a material video to be shortened (hereinafter referred to as “material shot”) in the same manner as the sample feature extraction unit 21. To do.

  The candidate generation unit 13 compares the feature vector for each frame of the material shot received from the material feature extraction unit 11 with the edit model received from the model generation unit 23.

  Thereby, for example, when the editing model indicates that a frame having a luminance equal to or higher than a certain luminance of the source shot is extracted as a frame of the sample shot, the candidate generating unit 13 includes the certain constant included in the material shot. A frame having a luminance higher than the luminance can be extracted and presented to the user as a frame candidate to be left as a material shot.

  The user can appropriately shorten the material shot by selecting a frame to be left as a material shot from the presented frame candidates.

  The comparison between the source video and the sample video as described above is also useful for learning the editing pattern of the sample video.

  FIG. 15 is a diagram schematically illustrating a relationship between a sample video and a source video that is a source of the sample video.

  As shown in FIG. 15, if the sample video is a movie trailer video, the source video is the main movie. Further, the sample video is composed of a plurality of shots selected from a plurality of shots included in the source video.

  Therefore, for example, information indicating at which time position each shot included in the sample video exists in the source video can also be regarded as one of the feature quantities of the shot.

  In this case, as in the case of comparing the source shot and the sample shot described above, the sample analysis unit 22 compares the signal characteristics of each shot of the source video and the sample video to determine what characteristics of the source video. It is possible to obtain information indicating whether or not a moving image has been extracted as a moving image of a sample video. For example, it is possible to learn which time position shot is extracted as a sample video shot.

  Further, the model generation unit 23 includes the information learned by the sample analysis unit 22 as one component in the feature vector of each shot of the sample video. Further, the editing model including these feature quantity vectors is output to the candidate generation unit 13.

  For example, when a certain time position is selected by the user, the candidate generation unit 13 specifies a feature quantity vector of a shot in the sample video corresponding to the time position from the edit model.

  Further, the time position in the source video where the shot was located is read out from the editing model from the specified feature amount. For example, assume that the read time position is a time position corresponding to 50% of the total length of the source video.

  In this case, the candidate generation unit 13 presents the three shots, the shot at the time position corresponding to 50% of the total length of the material video and the previous and subsequent shots, to the user as shot candidates to be arranged in the user edited video. To do.

  Or, considering other feature quantities, one or more shots in a material video having a shot at a time position corresponding to 50% of the total length of the material video and a feature quantity vector close to the feature quantity vector of the shot. Then, it is presented to the user as a shot candidate to be placed in the user edited video.

  The user selects and arranges a shot to be arranged at a time position selected by the user from one or more shots presented as candidates.

  As described above, the video editing support apparatus 1 can support the user's video editing also by using information indicating at which time position of the source video each shot of the sample video exists.

  Further, the movie editing support apparatus 1 may learn a plurality of types of editing information from a plurality of types of sample videos. For example, the editing information may be learned from a plurality of movie trailer videos of different genres such as action and comedy, a digest video of a sports game such as soccer, and the like.

  In this case, it is only necessary that the video editing support apparatus 1 has a function of storing the plurality of pieces of editing information and reading out necessary editing information according to an instruction from a user or the like.

  Thereby, for example, editing information more suitable for production of user-edited video can be selected from a plurality according to the content of the material video or according to the user's preference.

  The operations of the learning unit 2, the editing unit 3, and the output unit 4 included in the video editing support device 1 of the present embodiment are a central processing unit (CPU), a storage device, an interface for inputting and outputting information, and the like. And a program that causes the computer to perform predetermined processing.

  For example, the sample video input through the interface is stored in a predetermined memory. Further, the CPU executes a predetermined program to divide the sample video into shot sequences, extract feature quantities of each shot, generate an edit model, and the like.

  Further, the material video input via the interface is stored in a predetermined memory. Further, the CPU executes a predetermined program to divide the material video into shot sequences, extract feature amounts of each shot, and the like.

  Further, the CPU generates information indicating candidate elements or candidate positions by using the user instruction contents input through the interface and information such as an editing model. In addition, an edit screen including these candidate elements is generated and output to the display device via the interface according to instructions from the CPU.

  The video editing support apparatus of the present invention is also realized by such a computer operation.

  In addition, as described above, the video editing support apparatus 1 learns editing information by performing video analysis, which is analysis of moving images and sounds, on the sample video. Furthermore, selection of elemental media such as shots to be arranged in the user edited video and selection of time positions at which the elemental media should be arranged are guided according to the editing information.

  That is, the video editing support apparatus 1 can allow the user to use the editing technology and knowledge of the video editing specialist who created the sample video as editing information by analyzing the sample video.

  Accordingly, with reference to FIGS. 16 to 20, the video analysis performed by the video editing support apparatus 1 is performed while referring to experimental results and the like mainly focusing on theoretical aspects, including a part overlapping the above description. This will be described as an example.

  FIG. 16 is a flowchart showing the flow of operations related to video analysis of the video editing support apparatus 1.

  Taking the case where the video editing support apparatus 1 analyzes a movie preview video as an example, the operation of the video editing support apparatus 1 related to video analysis will be described.

  When a movie preview video is input to the video editing support device 1 (S40), a cut of the movie preview video is detected (S41) and divided into shot sequences (S42). Further, it is divided into frame sequences (S44).

  Next, in addition to the shot length, the intensity, variability, and range of the motion are extracted from the shot as the feature quantity related to the motion (S43, S45), and each shot is represented by a four-dimensional feature vector (S46).

  Furthermore, the obtained feature vector is normalized in order to remove the influence due to the difference in variation of these feature amounts in different videos (S47).

  The four-dimensional vectors obtained in this way are classified into classes based on similarity, and each shot is labeled with a class number as a symbol (S48).

  Finally, a shot time series in which all shots are represented by symbols is used as an observation series (S49), and time series pattern learning common to movie preview images is performed using a hidden Markov model (HMM) (S50).

  Details of the feature extraction from shots, labeling of shots, and learning methods using HMM will be described below.

(Shot feature extraction)
It is considered that the rhythm in the moving image, which is the sensory aspect of the image focused on in the learning method described below, is generated by changes in the moving image, such as the movement in the image and the frequency of camera switching. Therefore, the shot length and motion are used as the feature amount for the shot.

First, a two-dimensional histogram H (s, t) of a motion vector based on the motion vector shown in the following (Equation 1) obtained for the pixel (x, y) in the frame f by block matching (see Non-Patent Document 2). ) (S, t = 0, 1, 2,...).
G. Adiv, "Determining three-dimensional motion and structure from optical .flow generated by several moving objects," Proc. IEEE Trans. On PAMI, vol. 7, no. 4, pp. 384-401, 1985.

  Here, s represents the angle of the vector, and t represents the length of the vector. Here, block matching is very weak in a fast-moving frame or a frame with camerawork, and many motion vectors that do not match the actual motion appear.

  A common feature of such motion vectors is that they are detected as isolated vectors having different angles and lengths from surrounding motion vectors.

Therefore, the motion intensity v _i (f), the motion variability v _v (f), and the motion range v _a (f) are introduced as feature quantities considering the motion vector erroneously detected in this way.

First, the motion intensity v _i (f) is defined as follows.

This value is the sum of motion vectors excluding isolated vectors, and is considered to represent the actual magnitude of motion more faithfully. Also, the motion variability v _v (f) and the motion range v _a (f) are defined as follows.

Here, N (f) and N ₀ (f) are the total number of motion vectors in the frame f and the total number of motion vectors having a size of less than 1. Motion variability represents the percentage of isolated vectors detected. Many false detections due to block matching are observed between fast-moving frames, and many isolated motion vectors appear.

  Therefore, it can be seen that the greater the motion variability, the faster the motion in the target frame. The motion range represents the ratio of motion to the entire frame. When the background is moving, there are almost no motion vectors having a length of less than 1 in the entire frame, i.e., still points, and there are many still points in the still background.

FIG. 17 shows an example in which motion variability and a motion range are extracted from a frame in which camera work and object motion exist or do not exist. As shown in the figure, it is considered that the motion variability is related to the presence / absence of the motion of the object, and the motion range is related to the presence / absence of the camera work. Finally, by taking the average of the feature quantities of all the frames included in the shot s, the motion intensity v _i (s), the variability v _v (s), and the range v _a (s) in the shot s are obtained. A feature value is obtained.

The shot length l (s) is added to them, and the shot s is expressed by _a four-dimensional feature vector (l (s), v _i (s), v _v (s), v _a (s)). Here, it is necessary to normalize the obtained feature vectors in order to remove the influence due to the difference in variation of these feature amounts in different videos. Since both the motion variability and the motion range always fall within the range of 0 to 1, normalization is not necessary. Therefore, the motion intensity and the shot length are similarly normalized so as to fall within the range of 0 to 1. The motion intensity is normalized by the following equation so that the intermediate value is 0.5.

Where v _i (s _j ) and v ′ _i (s _j ) are the motion intensities of the j-th shot before and after normalization, and v _i (s _m ) is the middle of the motion intensities of all shots before normalization. Represents a value. However, since outliers may exist, 1 is set for extremely large data. The shot length is normalized by the following equation based on the ratio of each shot to the entire video.

Here, l (s _j ) and l ′ (s _j ) represent the shot lengths of the j-th shot before and after normalization, respectively.

(Label shots)
The same symbol is labeled on similar shots based on the obtained four-dimensional vector by feature amount extraction. By adding a symbol to a shot, the original shot time-series data can be expressed by a symbol string.

When labeling, each of the four feature quantity shot lengths, motion intensity, motion variability, and motion range is divided into N _i (1 ≦ i ≦ 4). At the time of division, a division point is determined every 100 / N _i % by the cumulative histogram of each data. That is, if the shot length is divided into three, it can be divided into three stages, short, medium and long. As a result, the shots are classified into a total of Π _i N _i classes.

(Learning with HMM)
The HMM has a plurality of states, and a transition probability is given between these states. Further, a symbol is output stochastically from each state. It is this output symbol sequence that can be observed, and the state of the model cannot be observed directly. FIG. 18 shows a conceptual diagram of the HMM.

  In general, the HMM is expressed by a parameter of λ (A, B, π) and is defined as follows.

V = {V _m } (m = 1,..., M): set of observation symbols.

Q = {q _n } (n = 1,..., N): a set of states.

O = O ₁ , O ₂ ,..., O _T : Observed symbol sequence (length T).

S = S ₁ , S ₂ ,..., S _T : State sequence (length T).

_St , _Ot : State and observation symbol at time t.

A = {a _ij | a _ij = P (S _{t + 1} = q _j | S _t = q _i )}: State transition probability distribution.

a _ij is the probability of transition from state q _i to q _j .

B = {b _j (O _t ) | b _j (O _t ) = P (O _t | S _t = q _j )}: Symbol output probability distribution.

b _j (O _t ) is the probability of outputting the symbol O _t in the state q _j .

π = {π _i | π _i = P (S ₁ = q _i )}: Initial state probability distribution. π _i is the probability that the transition will start from state q _i .

  In this description, it is assumed that the image is composed of a flow such as “contracting”, and a left-to-right type that can easily model a signal whose characteristics are continuously changed with time. Adopt HMM. Therefore, the initial state probability distribution is always constant and is as follows.

  In addition, since the Left-to-Right type HMM can only make a transition to the self and right states, the following constraint conditions are satisfied for the state transition probability distribution A.

  The reason for using HMM is not only to recognize time-series data without being affected by fluctuations in features, but also to abstract time-series data into model representation and generate time-series data from a model. It can be described with a mathematical model.

  In this description, generation of time-series data corresponds to generation of an ideal edited video shot time series, and is necessary for selection of partial video and determination of arrangement of partial video when producing edited video from material video. Become.

In addition, since HMM is a technique established as a very effective technique in the field of speech recognition, various algorithms for high-speed calculation have been developed. Here, the shot time series obtained in the previous section is used as the observation series {O _t }, and the parameters of HMM λ (A, B, π) are learned by the Baum-Welch algorithm.

(Experiment and evaluation)
The effect of the initial parameter values and the number of states on the learning result was verified using 20 preview images of action movies that are considered to have a relatively remarkable rhythm in the video.

  Here, one video is used as a test video, and the remaining video is used as a video for learning, and the observation probabilities in the HMM before and after learning of the shot time series obtained from the test video obtained by the forward algorithm are compared. To evaluate the learning effect. Cross-validation using a combination of 20 learning videos and test videos was performed.

  In addition, based on the acquired optimum initial parameter value and the number of states, the HMM after learning was evaluated using videos produced from the same source video by different editing methods.

In order to extract the feature amount, it is necessary to first divide the video into shots. As described above, it is possible to divide into shots by detecting cuts in the video, but in this experiment, shot division was performed manually. When labeling shots, N ₁ = N ₂ = 3, N ₃ = N ₄ = 2 and the four-dimensional feature space was divided into 36 classes.

  The HMMs before and after learning are λ (A, B, π) and λ ′ (A ′, B ′, π), respectively, and the observation probability at λ of the observation sequence {O} is P (O | λ). Since the observation probability varies exponentially depending on the length of each observation sequence, comparison between videos of different lengths is not possible. Therefore, the fitness defined as follows is introduced.

  Where T is the length of the observation sequence. This goodness of fit represents the observation probability of the observation sequence of unit length, and enables comparison between videos of different lengths. Furthermore, the learning result is evaluated based on the improvement rate defined as follows.

  However, Z ′ represents the degree of adaptation of the observation sequence to the model after learning, and the improvement rate corresponds to a change before and after learning of the observation probability for the same observation sequence {O}.

(Influence by initial parameter value)
First, the effect of initial parameter values on the experiment will be verified. Since the HMM used here is a Left-to-Right type, π is fixed among the initial values (A, B, π) of the HMM. Therefore, only A and B are changed. However, the number of states is 4 here.

  First, B is fixed and A is changed. Here, the symbol output probability distribution B is set as in (Equation 12) so that all symbols observed with the same probability are output from each state.

Further, in the state transition probability A, Δ _i = 2 is assumed, and the following form is obtained by the Left-to-Right type HMM.

The nine set randomly A (A 1 to A 9), and Table 1 shows the results for A ₁₀ set manually as follows.

A _10, because the picture is constructed generally a sequence of scenes, it is assumed that each state corresponds to a single scene, the transition probabilities for each scene in the video, a _ii> 0.9 and, a _ii >> a _{i (i + 1)} > a _{i (i + 2)}

The improvement rate is about 45 to 50% in all cases, and it is understood that the change of the initial value A has almost no influence on the result. Also, the A ₁₀ relatively high improvement rate has been obtained, each state of the HMM indicates a potential corresponding to the scene of the image.

Next, Table 2 shows the experimental results when A is fixed at A ₁₀ in the above experiment and B is set to 10 randomly set values. As for B, the effect on the improvement rate is relatively large, and it is necessary to examine the setting method in the future in consideration of the relationship with the video content of the class to which the shot belongs.

(Effect of number of states)
Next, in order to verify the influence of the number of states on the learning result, the number of states was changed to 2 to 7 and experiments were performed. However, Δ _i in (Expression 7) is 1 when the number of states is 2 and 3, and is 3 when the number of states is 6 and 7. FIG. 19 shows the average improvement rate obtained by preparing five sets of initial parameter values for each number of states and applying them to 20 different video combinations.

However initial parameter values, the value of five types set in the same manner as A ₁₀ for A, due to the high impact on the improvement rate for B, using the mean value in the same manner as (Equation 12).

  In FIG. 19, the horizontal axis represents the number of states, and the vertical axis represents the improvement rate. An average improvement rate of 50.2% was observed, but the improvement rate was relatively large when the number of states was 3 or 4, and the optimal number of states is considered to be 4. This is presumed to be because the video has a configuration such as “consolidation”. Here, FIG. 20 shows the relationship between the scene configuration based on the start and end of the video and the optimum state sequence output in the HMM with 4 states after learning.

  Since the HMM is a Left-to-Right type, the state transitions from 1 to 4. As shown in the figure, it can be seen that the state change approximates the scene change. Thus, the relevance with the video content of each state of HMM was confirmed. In the future, it will be necessary to clarify these relationships for all images.

(Evaluation of learning effect with different edited videos)
Finally, the learned HMM is verified using different videos produced by editing the same source video. For one movie preview video (Video 1), a short edited video manually created from the original movie was prepared separately as the source video. Here, the shots including the partial images included in the movie preview video are extracted from the movie, and these shots are arranged in the order in which they appeared in the source video (Video 2), and the shots are arranged in the order in which they appeared in the movie preview video (Video 3) was produced.

As initial parameters, A is set to A ₁₀ , B is set to an average value, and the number of states is set to 4. Table 3 shows the relevance ratio of each image to the model before and after learning and the improvement ratio obtained by learning.

  Video1, which is an actual video, showed an improvement rate of about 50% as in the experimental results so far, whereas the adaptability was hardly improved for Video2 and Video3.

  Here, Video 2 includes shots with the same content as Video 1 but appears in a different order, whereas Video 3 appears with the same content as Video 1 in the same order, so Video 3 has a slightly higher degree of fitness. Is considered high.

  However, each shot extracted from the source video used in Video 3 has a longer shot length and contains redundant content compared to the corresponding Video 1 shot, so the edited video does not resemble the actual video and the fitness is It did not improve so much.

  Thus, since it is considered that the HMM after learning represents the shot time-series pattern of the actual video well, for the user who is editing the video, so that the degree of matching with the HMM after learning is high, By guiding the shots to be selected and arranged, it is expected to be able to support production of edited videos similar to actual videos.

  The present invention can be applied to a video editing support apparatus that supports editing of video content. In particular, it is useful as a video editing support device, a video editing support program, a video editing support method, and the like for a general user who has no technical knowledge and knowledge of video editing to edit video content effectively and efficiently.

It is a figure which shows typically the structure and function of the video editing assistance apparatus of embodiment. It is a figure which shows typically the sensory structure of the sample image | video which a learning part learns. It is a functional block diagram which shows the functional structure of a video editing assistance apparatus. It is a flowchart which shows the outline | summary of the flow of operation | movement which concerns on the edit assistance of a video editing assistance apparatus. It is a flowchart which shows the outline | summary of the flow of operation | movement which concerns on the video analysis of a video editing assistance apparatus. FIG. 6 is a diagram illustrating an example of an editing screen displayed on the display device as a result of the operation flow illustrated in FIG. 5. It is a flowchart which shows the flow of the operation | movement which concerns on the edit assistance of a video editing assistance apparatus. It is a figure which shows an example of the edit screen when one shot is selected by the user. It is a figure which shows an example of the edit screen when one shot is moved to the storyboard by the user. It is a figure which shows an example of the edit screen when a time position is selected by the user. It is a figure which shows an example of the edit screen when an additional sound is selected by the user. It is a figure which shows an example of the edit screen when the edit effect is selected by the user. It is a flowchart which shows the outline | summary of the flow of operation | movement which considered the shot or sound already arrange | positioned of a video editing assistance apparatus. It is a figure which shows typically the relationship between one shot of a sample image | video, and the shot which is the source of the shot. It is a figure which shows typically the relationship between a sample image | video and the source image | video which is the source of the sample image | video. It is a flowchart which shows the flow of the operation | movement which concerns on the video analysis of the video editing assistance apparatus. It is a figure which shows the relationship between a motion variability and a motion range, a camera work, and the motion of an object. It is a figure for demonstrating a hidden Markov model. It is a figure which shows the result of the verification experiment about the influence which the number of states has on a learning result. It is a figure which shows the relationship between the scene structure of a movie preview image, and a HMM state series.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image | video editing assistance apparatus 2 Learning part 3 Editing part 4 Output part 10 Material reception part 11 Material feature extraction part 12 User input reception part 13 Candidate generation part 14 Effect data storage part 15 Additional sound storage part 16 Editing execution part 20 Sample reception part 21 Sample feature extraction unit 22 Sample analysis unit 23 Model generation unit 30 Display device 31 Edit screen 32 Time input column 33 Material video timeline column 34 Shot selection column 35 Additional element selection column 36 Playback display column 37 Storyboard 37a Video story column 37b Sound story field 38 Structure display board 38a Video structure field 38b Sound structure field 40 Input device

Claims (12)

A video editing support device that supports editing when a user edits a material video that is video content to produce a user-edited video,
Learning means for learning editing information, which is information on how to arrange a plurality of entity elements included in a sample video that is edited video content;
Material accepting means for accepting input of the material video;
According to the editing information learned by the learning means, candidates for arrangement in the user edited video of the entity elements included in the material video received by the material receiving means or the entity elements added to the user edited video are indicated. Candidate generation means for generating information;
A video editing support apparatus comprising: candidate output means for outputting information indicating the arrangement candidates generated by the candidate generation means. Furthermore, an instruction receiving means for receiving the user's instruction is provided,
The candidate selecting means includes
When the instruction accepting unit accepts an instruction to select an entity element of a predetermined unit included in the material video, the time position candidate on the user-edited video to which the instructed entity element should be placed is edited. According to the information, by selecting from any time position on the user-edited video, information indicating the arrangement candidate is generated,
When the instruction receiving unit receives an instruction to select a time position on the user-edited video, candidates for entity elements to be placed at the specified time position are included in the material video according to the editing information. The video editing support apparatus according to claim 1, wherein information indicating the arrangement candidate is generated by selecting from a plurality of entity elements. Furthermore, it comprises material feature extraction means for extracting the signal features of each of the moving image and the sound, which are entity elements included in the material video received by the material reception means,
The learning means includes
Sample feature extraction means for extracting signal features of moving images and sounds, which are entity elements included in the sample video;
By analyzing the signal features extracted by the sample feature extraction means, information indicating the time-series configuration of each of the moving image and the sound included in the sample video, and information indicating the synchronous configuration of the moving image and the sound, Analyzing means for obtaining
Learning the information indicating the time-series configuration and the information indicating the synchronous configuration obtained by the analysis unit as the editing information,
The candidate generation unit compares the information indicating the time-series configuration or the information indicating the synchronous configuration obtained by the analysis unit with the signal features extracted by the material feature extraction unit, thereby The video editing support apparatus according to claim 1, wherein information indicating candidates for arranging the predetermined unit of moving image or sound in the user-edited video is generated. The analysis means, as information indicating the synchronization configuration, information indicating a relationship between a boundary between adjacent moving images divided into a predetermined unit and a boundary between adjacent sounds divided into a predetermined unit; The video editing support apparatus according to claim 3, wherein information indicating a relationship between a feature of a moving image and a feature of a sound added to the moving image is obtained. The analysis means, as information indicating the time-series configuration, relates to the signal characteristics of each adjacent video divided into predetermined units and the signal characteristics of each adjacent sound divided into predetermined units. The video editing support apparatus according to claim 3, wherein information indicating the characteristics is obtained. Furthermore, it comprises a sample receiving means for receiving the sample video and a source video that is a video before editing the sample video,
The sample feature extraction unit further extracts signal features of each of moving images and sounds that are entity elements included in the source video,
The analysis means further has any characteristics of the source video by comparing the video or sound signal characteristics included in the sample video with the video or sound signal characteristics included in the source video. The video editing support apparatus according to claim 3, wherein information indicating whether the moving image is extracted as a moving image of a sample video is obtained. An instruction receiving means for receiving the user's instruction;
After the candidate output means outputs the information indicating the arrangement candidate, the user-edited video is obtained by arranging two or more entity elements included in the material video according to the instruction received by the instruction receiving means. Editing execution means;
Video output means for visualizing and outputting the arrangement of the entity elements in the user-edited video obtained by the editing execution means by arranging images in a manner corresponding to the signal features extracted by the sample feature extraction means; The video editing support device according to claim 3. Furthermore, a storage means for storing a plurality of signal characteristics of additional sound, which is an entity element added to the user edited video,
The candidate generation unit further compares the editing information learned by the learning unit with a plurality of signal features stored in the storage unit, thereby arranging the additional sounds to be added to the user edited video. Generate information that shows the candidates for
The video editing support apparatus according to claim 3, wherein the candidate output unit further outputs information indicating a candidate for arranging the additional sounds generated by the candidate generation unit. Furthermore, storage means for storing a plurality of signal characteristics of visual effects that are entity elements added to the user edited video,
The learning means relates to a method of arranging visual effects added to the moving image, which is an entity element included in the sample video, by the analysis means analyzing the signal characteristics of the moving image extracted by the extracting means. By obtaining information, the information on how to arrange the visual effects is learned as editing information,
The candidate generation means compares the editing information learned by the learning means with a plurality of signal features stored in the storage means, thereby arranging visual effect alignment candidates to be added to the user edited video. Generate information that indicates
The video editing support apparatus according to claim 3, wherein the candidate output unit further outputs information indicating candidates for arranging the visual effects selected by the candidate generation unit. An instruction receiving means for receiving the user's instruction;
After the candidate output means outputs the information indicating the arrangement candidate, the user-edited video is obtained by arranging two or more entity elements included in the material video according to the instruction received by the instruction receiving means. Editing execution means;
The video editing support apparatus according to claim 1, further comprising: a video output unit that outputs the user edited video obtained by the editing execution unit. A method for supporting editing when a user edits a material video as a video content to produce a user-edited video,
A learning step for learning editing information, which is information relating to the arrangement of a plurality of entity elements included in the sample video that is the edited video content,
A material receiving step for receiving input of the material video;
In accordance with the editing information learned in the learning step, a candidate for arranging the entity elements included in the material video received in the material reception step or the entity elements added to the user edited video in the user edited video is shown. A candidate generation step for generating information;
A candidate output step for outputting information indicating the arrangement candidate generated in the candidate generation step. A program for supporting editing when a user edits a material image by editing a material image as a video content,
A learning step for learning editing information, which is information relating to the arrangement of a plurality of entity elements included in the sample video that is the edited video content,
A material receiving step for receiving input of the material video;
In accordance with the editing information learned in the learning step, a candidate for arranging the entity elements included in the material video received in the material reception step or the entity elements added to the user edited video in the user edited video is shown. A candidate generation step for generating information;
A program for causing a computer to execute a candidate output step of outputting information indicating the arrangement candidate generated in the candidate generation step.

Images