JP2006014084A - Video editing apparatus, video editing program, recording medium, and video editing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize a summary video creating apparatus for creating a summary video image with which a viewer can accurately and easily grasp contents of entire video. <P>SOLUTION: A summary video creation apparatus 1 comprises: a shot analysis section 12 for recognizing features corresponding to a length of a duration time of a shot for each part of video based on video data 51; a video analysis section 13 for recognizing features corresponding to violence of a motion of the video for each part of the video based on the video data 51; a block extraction section 17 for specifying a block corresponding to intensified blocks (action block, tense block, relaxed block) in the video data 51 based on results of the recognitions; a subordination degree detection section 18 for detecting a subordination degree 63 among the intensified blocks based on the results of the recognitions; and a summary video generating section 19 for determining a portion to be adopted for a summary video image from the intensified blocks based on the results of the recognitions and a result of the detection. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a video editing apparatus for creating a summary video from a video having a story such as a movie or a TV drama, a video editing program, a computer-readable recording medium recording the video editing program, and a video editing method. .

  Due to the increase in communication speed on the Internet, video distribution and digital broadcasting are becoming more common, and video recorders with built-in HDDs have become popular, so users can acquire many videos via the Internet. It has become possible to accumulate and view them. For this reason, the user needs to select a desired video from many videos. One of the methods aimed at understanding the content and atmosphere of a video in a short time is to summarize the video.

  There are various types of video such as dramas, movies, sports, news, music programs, etc. Especially, movies and dramas are long, so if you can create a summary video that is easy to understand in a short time, It will be useful to you. For example, when browsing an accumulated movie, summary video is particularly useful when a movie critic wants to remember the contents of the movie when writing or introducing a review of a movie that he has watched in the past. The following technologies are known as video summarization techniques for movies.

  In Non-Patent Document 1, special events such as close-up of main characters, gunshots and explosions, titles and telops are detected and connected to create a summary video for the purpose of a movie trailer. In Non-Patent Document 2, attention is paid to a section having a high psychological impression of a drama, and a summary video in which a psychologically important part such as a start or end of music or a part where cuts frequently occur is cut out is created. Further, in Non-Patent Document 3, based on a User Attention Model created based on an element in which the viewer pays attention to the visual and auditory sense, a section where the viewer is likely to pay attention is adopted in the summary video.

  On the other hand, in Non-Patent Document 4, shots are clustered based on visual similarity, and the longest shot from each cluster is adopted as a summary video.

  In Non-Patent Document 5, a movie is structured into shots, story units, and scenes based on image and sound characteristics, and a summary video that takes into account the context of the movie is detected by detecting the dependency in each unit. Creating.

Patent Document 1 discloses a technique for extracting a video by using an evaluation value of the shot or scene created based on information assigned to each shot or scene.
WO00 / 40011 (International publication date July 6, 2000) R. Lienhart, S. Pfeiffer, W. Effelsberg, “Video Abstracting”, Communications of the ACM, Vol. 40, No. 12, pp. 55-62, Dec. 1997. Tsuyoshi Moriyama and Masao Sakauchi, “Generating Summary Video Based on the Psychological Content of Drama Video,” IEICE Transactions, Vol. J84-D-II, No. 6, pp. 1122-1131, Jun. 2001 . Yu-Fei Ma, Lie Lu, Hong-Jiang Zhang, Mingjing Li, “A User Attention Model for Video Summarization”, Proc. Of ACM Multimedia, pp. 533-542, Dec. 2002. Yihong Gong, Xin Liu, “Summarizing Video by Minimizing Visual Content Redundancies”, IEEE International Conference on Multimedia and Exposition, pp. 788-791, 2001. Kazuya Kato, Ikuo Yoshitaka, Masato Hirakawa, “Creating a Movie Summarizing the Context”, Information Processing Society of Japan, Vol. 2002, No. 25, pp. 25-30, Mar. 2002. By Daniel Arifon, Noriko Iwamoto, Taketo Deguchi, “Grammar of Cinema”, Kinokuniya, 1980. Akito Akutsu and Yoshinobu Tonomura, “Analysis Method of Video Using Projection and Application to Video Handling”, IEICE Transactions, Vol. J79-D-II, No. 5, pp. 675-686 , May 1996. Tomohiro Kawasaki, Ikuo Yoshitaka, Masato Hirakawa, Tadao Ichikawa, “Proposal of Music, Sound Effects Extraction and Impression Evaluation Techniques in Films”, IEICE Technical Report, MVE97-96, pp. 23-29, 1998.

  In the techniques disclosed in Non-Patent Documents 1 to 3, the sections in which specific features are detected are simply connected. Therefore, the summary video created by such a technique becomes a fragmentary video, and it is difficult to fully understand what happened in the video, and before and after the event. The summary video is difficult to understand.

  Further, in the technique disclosed in Non-Patent Document 4, only visually redundant shots are removed, and shots that are important for conveying the contents of video are not selected. Further, although the longest shot from each cluster is adopted as the summary video, it cannot be said that the longest shot is important in conveying the content of the video.

  In the technique disclosed in Non-Patent Document 4 above, the context is taken into consideration, but since all the shots in the dependency relationship are adopted in the summary video, there is a bias in the summary video, and the content of the story of the entire video It is difficult to know.

  In addition, in the technique disclosed in Patent Document 1, specific technical contents are disclosed in addition to the fact that subjective evaluation by an evaluator is disclosed regarding the provision of information used when creating an evaluation value. Is not disclosed.

  As described above, with the conventional technology, it is difficult to create a summary video that can accurately grasp the content of the video.

  The present invention has been made in view of the above problems, and an object thereof is to realize a video editing apparatus and a video editing method for creating a summary video that allows a viewer to accurately grasp the contents of the entire video. It is in.

  The video editing apparatus according to the present invention is a video editing apparatus that creates a summary video from a video including an emphasis section that can be specified based on the length of each shot constituting the video and the intensity of motion in the video. In order to solve the above problems, the shot recognition means for recognizing the feature corresponding to the length of the shot duration for each part of the video based on the video data, and the motion of the video for each part of the video based on the video data. Video recognition means for recognizing features according to the intensity of the image, enhancement section specifying means for specifying a section corresponding to the enhancement section in the video data based on the recognition results by the shot recognition means and the video recognition means, and the shot Based on the recognition results of the recognition means and the image recognition means, a dependency level detection means for detecting a dependency level between the emphasis sections, the shot recognition means, A recognition result by the image recognition unit, based on the detection result by the dependent degree detecting means is characterized by comprising a summary creation unit for determining the portion to be employed in the video summary from emphasis interval.

  The video editing method according to the present invention is a video editing method for creating a summary video from a video including an emphasis section that can be specified based on the length of each shot constituting the video and the intensity of movement in the video. In order to solve the above-described problem, a shot recognition process for recognizing a feature corresponding to the length of a shot duration for each part of the video based on the video data, and a video for each part of the video based on the video data. Video recognition processing for recognizing features according to the intensity of motion, and enhancement section identification processing for identifying a section corresponding to the enhancement section in the video data based on the recognition result by the shot recognition processing and the video recognition processing, Based on the recognition results of the shot recognition process and the video recognition process, a dependency level detection process for detecting a dependency level between the emphasis sections, and the shot recognition process. And a recognition result of the image recognition processing, based on the detection result of the subordinate level detection process is characterized in that it comprises a summarization process that determines the portion to be employed in the video summary from emphasis interval.

  In movies and television dramas that have a story, a technique called “movie grammar” is used for the purpose of emphasizing a specific meaning and intention when shooting or editing. In the grammar of the movie, an action section, a tight section, and a calm section are set as sections that are editorially emphasized so that the contents are effectively transmitted to the viewer. Here, the action section is a section in which short shots are continuous and the movement of the image tends to be intense, and the tense section is a section in which the length of the shot tends to be gradually reduced. A calm section is a section in which long shots continue and the motion of the video tends to be gradual.

  Also, according to the grammar of the movie, there may be a relationship between cause and effect (dependency relationship) between these sections, and the sections in the dependency relation can be combined to clearly communicate the contents. become.

  Therefore, in the above configuration and method, in order to create a summary video that accurately summarizes the entire video, the section emphasized on the editing is specified as an emphasized section, and the dependency relationship between the emphasized sections is considered, The part to be adopted in the summary video is determined.

  That is, in the above configuration and method, for each part of the video, in order to recognize the feature according to the length of the duration of the shot and the feature according to the intensity of the motion of the video, based on these, the action section, A tight section and a calm section can be specified as an emphasis section.

  Further, the degree of dependency between the emphasis sections (dependency degree) can be understood as a difference in the degree of characteristic properties (action degree, tightness degree, calmness degree) of each emphasis section. In the above-described configuration and method, for each part of the video, the feature according to the length of the duration of the shot and the feature according to the intensity of the motion of the video are recognized. It is possible to recognize the degree, the tightness degree, and the calmness degree, and to detect the degree of dependency between the emphasis sections.

  In the configuration and method described above, the degree of action, tightness, and calmness of each emphasis section can be recognized as described above, and the degree of dependency between each emphasis section can also be detected. Based on these, the portion to be adopted in the summary video is determined from the emphasis section.

  As a result, in the above configuration and method, a summary video that conforms to the grammar of the movie, that is, the emphasis sections that are emphasized in editing, and the dependency relationship between these emphasis sections are reflected, so that the entire content can be accurately viewed by the viewer. It is possible to create a summary video that is easy to grasp.

  In the video editing apparatus according to the present invention, in the video editing apparatus, the shot recognizing unit generates, as a recognition result, a feature amount indicating a duration of the shot and a feature amount indicating the length of the duration of the shot. The video recognition unit may generate a feature amount indicating the degree of intenseness of the motion of the video as a recognition result.

  In the above configuration, for each part of the video, a feature quantity indicating the duration of the shot, a feature quantity indicating the length of the duration of the shot, and a feature quantity indicating the severity of the motion of the video are generated. Here, the degree of the duration of the shot is the degree of the relative length of the shot of each part with respect to the whole image, and the degree of the intensity of the movement of the image is the relative degree of movement of each part with respect to the whole image. The degree of intensity.

  As described above, the action section as the emphasis section is a section in which short shots are continuous and the movement of the video tends to be intense, and in a tense section, the length of the shot tends to be gradually shortened. Since the long and continuous shots are slow and the movement of the video tends to be slow, the emphasis section can be specified by relatively simple calculation by using the above feature quantities. In addition, it is possible to detect the degree of dependency and determine the video portion to be adopted as the summary video.

  In the video editing apparatus according to the present invention, the video editing apparatus recognizes a feature corresponding to the duration of the instrument sound component included in the audio for each part of the video based on the audio data added to the video data. Voice recognition means is further provided, the enhancement section specifying means further specifies a section corresponding to the enhancement section in the video data based on a recognition result by the voice recognition means, and the dependency level detection means is further configured to recognize the voice recognition. Detecting the degree of dependency between each emphasis section based on the recognition result by the means, and the summary creating means further determines a portion to be adopted for the summary video from the emphasis section based on the recognition result by the voice recognition means. Is desirable.

  In many cases, audio is added to the video, and in this case, the characteristic properties of the action section and the calm section appear as the duration of the instrument sound component included in the audio. That is, the duration of the instrument sound component tends to be short in the action section, and the duration of the instrument sound component tends to be long in the calm section.

  Therefore, in the above configuration, for each part of the video, in addition to the characteristics according to the length of the duration of the shot and the characteristics according to the intensity of the motion of the video, according to the length of the duration of the instrument sound component Based on these features, the emphasis section is specified, the degree of dependency is detected, and the video portion to be used as the summary video is determined. As a result, a more accurate summary video can be created.

  In the video editing apparatus according to the present invention, in the video editing apparatus, the voice recognition unit may generate a feature amount indicating a degree of duration of a musical instrument sound component as a recognition result.

  In the above configuration, a feature amount indicating the degree of duration of the instrument sound component is generated for each part of the video. Here, the degree of duration of the instrument sound component is the degree of the length of the sound constituting the melody.

  As described above, the duration of the instrument sound component tends to be short in the action section, and the duration of the instrument sound component tends to be long in the calm section. Thus, it is possible to specify an emphasis section, detect the degree of dependency, and determine a video portion to be adopted as a summary video.

  The video editing apparatus according to the present invention further comprises subject detection means for detecting presence of a video subject for each part of the video based on the video data in the video editing device, and the summary creation means is further provided by the subject detection means. Based on the detection result, it is desirable to determine a portion to be adopted for the summary video from the emphasis section.

  An image subject is a character or various objects photographed so as to occupy a relatively large part of the image, and they are often larger than a certain size, composed of a certain range of hues, and The object has a large contrast. The main part of the video is an important part in conveying the video content to the viewer, and the summary video that preferentially adopts that part understands the video content compared to the video without considering it. It becomes easy.

  Therefore, in the above configuration, the presence of the video subject is detected for each part of the video, and a part to be adopted for the summary video is determined from the enhancement section based on the detection result. As a result, a more accurate summary video can be created.

  The present invention is a video editing program for operating the video editing apparatus, and can be realized as a video editing program for causing a computer to function as each of the means, and can be read by a computer recording the video editing program. It can also be realized as a simple recording medium.

  As described above, the video editing apparatus according to the present invention is based on video data, shot recognition means for recognizing a feature corresponding to the length of a shot duration for each part of the video, and each part of the video based on the video data. Video recognition means for recognizing features according to the intensity of motion of the video, and enhancement section specifying means for specifying a section corresponding to the enhancement section in the video data based on the recognition results by the shot recognition means and the video recognition means Based on the recognition results by the shot recognition means and the video recognition means, a dependency level detection means for detecting the degree of dependency between the emphasis sections, the recognition results by the shot recognition means and the video recognition means, and the dependency level detection Summarizing means for determining a portion to be adopted in the summary video from the emphasis section based on the detection result by the means.

  In addition, as described above, the video editing apparatus according to the present invention is based on video data, based on video data, based on video data based on shot recognition processing for recognizing features corresponding to the length of shot duration for each part of video. Video recognition processing for recognizing features according to the intensity of motion of the video for each part, and an emphasis section for identifying a section corresponding to the emphasis section of the video data based on the recognition result by the shot recognition process and the video recognition process A subordinate degree detection process for detecting a subordinate degree between each emphasis section based on a recognition process by the identification process, the shot recognition process and the video recognition process, a recognition result by the shot recognition process and the video recognition process, and the subordinate Based on the detection result of the degree detection process, and a summary creation process for determining a portion to be adopted in the summary video from the emphasis section.

  As a result, a summary video that conforms to the grammar of the movie, that is, a summary video that allows the viewer to accurately grasp the entire contents by reflecting the emphasis sections that are emphasized for editing and the dependency relationship between these emphasis sections. The effect is that it can be created.

  In the present invention, editing is performed so that the content is effectively conveyed to the viewer based on the “movie grammar” used by the producer for emphasizing a specific meaning or intention when shooting or editing a movie. An action section (action scene), a tight section (tight scene), a calm section (calm scene), and sections that are subordinate to these sections are extracted as the above-emphasized sections. Then, the shots in those sections are adopted as the summary video in descending order of importance so as to satisfy the constraint time. Therefore, not only the emphasized section but also the process leading to it can be included in the summary video. As a result, a method for creating a summary video in which the contents and context of the movie are easy to understand is realized.

  An embodiment of the present invention will be described with reference to FIGS. 1 to 15 as follows.

1. Processing Content 1.1 Movie Grammar Movies have techniques that are used by producers to emphasize specific meanings and intentions during filming and editing. This is called “movie grammar” (Non-Patent Document 6: written by Daniel Arihon, Noriko Iwamoto, Taketo Deguchi, “Grammar of Movie”, Kinokuniya, 1980).

  According to the grammar of the movie, the following points are stated as the characteristics of the action section, the tight section, and the calm section, which are editorially emphasized sections. In other words, the action section is a section in which short shots are continuous and the motion of the video is intense, the tight section is a section in which the shot length is gradually shortened, and the calm section is a series of long shots. In addition, this is a section where the motion of the video is slow. In addition, according to the grammar of the movie, it is stated that it is important to combine the sections having a relationship between cause and effect for effective content transmission.

1.2 Process Flow Based on the grammar of the movie, in order to effectively convey the contents of the story to the viewer, an action section, a tight section, and a calm section are extracted as sections that are emphasized for editing. At that time, in each shot, action property, tightness, and calmness are defined as shot properties based on the length of the shot and the intensity and gentleness of the motion of the image. A group of shots in which the values representing the properties are continuously high are defined as an action section, a tight section, and a calm section, respectively. By extracting these three sections and using them as candidates for creating a summary video in descending order of the value representing each property, it becomes possible to add a section that is editorially emphasized in the movie to the summary video. The summary video makes the content of the movie easy to understand.

  Here, a shot is a set of continuous frames taken from one camera. A cut is a shot boundary.

  It should be noted that when defining the action property, tightness, and calmness as the nature of the shot, it is desirable to consider the tempo of the music that is reproduced in synchronization with the shot.

  In addition, when determining whether or not to add the extracted section to the summary video, it is desirable to consider the presence of the subject (video subject). The subject's shot is an important shot for telling the audience what the story is about, and the summary video that is mainly based on that shot makes it easier to understand the content of the movie than when it is not taken into account. . Since there is a high possibility that the object emphasized in the image is the main subject, a shot in which an object having a certain size or more and the same color and a change in luminance is different from the surroundings is detected.

  Furthermore, when any two sections of an action section, a tight section, and a calm section are adjacent to each other, these sections have a dependency relationship that represents a cause and an effect. Therefore, the summary video including these two sections is easier to understand the context than the video not including. An average value of the action degree, the tightness degree, or the calmness degree is obtained in the extracted sections, and the difference is obtained in the preceding and following sections, thereby obtaining the degree of dependency in those sections. The reason why the degree of dependency is the difference between the values of the preceding and following sections is that the greater the difference in the properties before and after, the stronger the impression the viewer can have and the more effective the contents can be conveyed.

  Finally, when creating a summary video, select a video section that is uniformly a summary video from the entire movie, and divide the movie into n (= 20) equal parts to make it easier to understand the content of the story. Then, from the divided sections, the action video, the tightness, or the calmness are high so that the restriction time specified by the viewer is satisfied. As a summary video, a summary video that makes it easier to understand the content and context of the movie is created.

2. Definition of shot characteristics 2.1 Action characteristics 2.1.1 Action characteristics according to shot length In the action section, there is a feature that short shots are continuous. Find the value.

Assuming that the length of the shot at the k-th shot s _k is SL (s _k ) [seconds], a value SLV _A (s _k ) representing the action property according to the length of the shot at s _k is expressed by the equation (1). Define as follows. This is based on using a short shot to effectively convey the action to the viewer. If it is determined that the length of a certain shot is short, it is regarded as a shot representing the action, and the action property is 1 And Here, the reason why the action property according to the shot length is binary is that the fact that the shorter the shot length is, the higher the action property is, is not shown by the grammar of the movie.

However, Th _shot [seconds] is a threshold value indicating that the shot length is short, and SL _mean [seconds] is an average value of shot lengths of a whole movie. SL _mode [second] represents the mode of the shot length. However, the mode value is an intermediate value in 0.5 seconds in which the cumulative frequency of shots is obtained at intervals of 0.5 seconds and the frequency is maximum.

2.1.2 Actionability due to changes in the image Spatiotemporal projection image shown in Fig. 1 (Non-patent document 7: Akito Akutsu, Yoshinobu Tonomura, “Image analysis method using projection method and its application to image handling” ”, See IEICE Transactions, Vol. J79-D-II, No. 5, pp. 675-686, May 1996.) is an image that visualizes movements caused by objects and camera work in the video. Therefore, in Non-Patent Document 7, it is used when camerawork is detected.

  In the present embodiment, attention is paid to the appearance of a feature associated with the intensity of image movement in the spatiotemporal projection image, and the action property is obtained by detecting the feature. In the present embodiment, a spatiotemporal projection image in the horizontal direction is used. As shown in FIG. 1, in the horizontal spatiotemporal projection image, the arrangement of frames is taken in the horizontal direction (the f direction in FIG. 1, hereinafter referred to as “time axis direction”), and the arrangement of horizontal pixels in the video is taken in the vertical direction. (The x direction in FIG. 1, hereinafter referred to as “image scanning direction”).

  When the motion of the image is intense, as shown in FIGS. 2A and 2B, an edge in the image scanning direction appears on the spatiotemporal projection image.

Assuming that the number of edges in the image scanning direction in the spatiotemporal projection image at the shot s _k is E _v (s _k ), a value VTIV _A (s _k ) representing the action property by the spatiotemporal projection image is expressed as in Equation (2). Defined in In Formula (2), the intensity in the video is expressed as the number of edges that appear in unit time. This is based on the fact that the number of edges in the image scanning direction appearing in the spatio-temporal projection image increases as the movement in the video in the action section increases.

2.1.3 Action by music As shown in Fig. 3, by detecting instrument sound components showing frequency peaks along the time axis (horizontal axis) appearing on the sound spectrogram, the instrument sound components in a certain time interval are detected. It is possible to determine that music is flowing according to the number (Non-patent Document 8: Tomohiro Kawasaki, Ikuo Yoshitaka, Masato Hirakawa, Tadao Ichikawa, “Proposal of Extracting Music and Sound Effects and Impression Evaluation Techniques in Movies”, (See IEICE Technical Report, MVE97-96, pp. 23-29, 1998.)

  In the present embodiment, attention is paid to the fact that the feature of music appears in the duration of the instrument sound component, and the nature of the music is detected based on that time. Through experiments, it has been confirmed that the music flowing in the action section tends to have a short duration of instrument sound components. In addition, since musical instruments classified as bass in music serve as indices for knowing the tempo of music, attention is paid to musical instrument sound components in the frequency band that the bass plays. In movies, music played in an orchestra often flows, so the duration of instrument sound components in the frequency band (30-300 Hz) of the instrument that plays the bass in the orchestra is used as an index.

An action expressed by music, _assuming that the length of the instrument sound component at shot s _k is IL (s _k ) [seconds], and the threshold for determining that the duration of the instrument sound component is short is _ThinstA [seconds]. _A value MV _A (s _k ) representing sex is defined as in Equation (3). However, _ThinstA is a value obtained by experiments and is 1.24 [seconds].

2.1.4 Action Property Based on the value representing the action property of each feature obtained as described above, the action property degree Action (s _k ) in the shot s _k is expressed as Equation (4). Based on the three values obtained above, the degree of action at shot s _k is _obtained. However, not only certain elements always appear in the action section, but each element may be satisfied, so the average of each element The degree of action is sought.

2.2 Tension The length of a shot is gradually shortened in a tight section. A tight section is extracted based on the feature. In addition, the shorter the average time of shots in a tight section, the higher the sense of tightness. Therefore, Tension (s _k ) is defined as Equation (5), using this as the degree of tightness. However, SL _Tension length of the average value of shots in a tense period, n represents the number shots in tense section, m _i denotes the displacement from the k-th shot. Note that the degree of tightness is defined only in a tight section, that is, a section that satisfies the condition that the shot length gradually decreases.

2.3 Calmness 2.3.1 Calmness due to shot length Since there is a feature that long shots are continuous in a calm zone, it is extracted under the following conditions to obtain a value representing calmness.

Shot s _k values representative of the restless due the length of the shot with SLV _C (s _k) is defined as Equation (6). This is based on the fact that a long shot is used to effectively convey a calm atmosphere to the viewer. If it is determined that the length of a certain shot is long, it is regarded as a shot expressing a calm feeling. The calmness is 1. The reason why the calmness by the shot length is binary is that the longer the shot length, the higher the calmness is not indicated by the grammar of the movie.

2.3.2 Calmness due to movement in the image In a calm period, there is not much movement due to objects or camerawork in the video, so there are edges along the time axis direction on the spatiotemporal projection image . The calmness is defined by detecting the flatness of the edge. In this case, the flatness scale is a value representing calmness.

In order to obtain a measure of flatness in the shot s _k , a portion that becomes an edge on a spatiotemporal projection image is tracked, and the values shown in FIG. 4A are added according to the tracking order shown in FIG. Go.

  Specifically, it is as follows. First, edge enhancement in the time axis direction is performed on the spatiotemporal projection image, and a binarized image (time axis direction edge enhanced image) is created according to the presence or absence of an edge. Then, in this time axis direction edge enhanced image, a pixel corresponding to the edge is set as a target pixel, and the edge is traced in the time axis direction. In order to track the edge, the position where the pixel is first detected in accordance with the tracking sequence shown in FIG. Then, a numerical value (score) set as shown in FIG. 4A is acquired according to the position of the destination pixel with respect to the target pixel, and the above tracking is repeated using the destination pixel as a new target pixel. The scores obtained together with the tracking in this way are sequentially added, and the value obtained by dividing the addition result by the number of tracked pixels is used as a measure of flatness.

If the score addition result is Sum (s _k ) and the number of pixels to be tracked is N (s _k ), the value VTIV _C (s _k ) representing the calmness of the spatiotemporal projection image at the shot s _k is expressed by Equation (7). Define as follows. VTIV _C (s _k ) takes the maximum value 1 when the edge is a straight line in the time axis direction, and there is a portion that becomes a repeated edge at the position 7 or 9 in the tracking sequence of FIG. The minimum value is 0.

2.3.3 Calmness by music Calmness is determined by the duration of musical instrument sound components. Through experiments, it has been confirmed that music flowing in a calm section tends to have a long duration of instrument sound components.

_Assuming that the threshold for determining that the duration of the instrument sound component is long in the shot s _k is Th _instC [seconds], a value MV _C (s _k ) representing the calmness of music is defined as in Equation (8). However, _ThinstC is a value obtained by experiments and set to 1.40 [seconds].

2.3.4 Based on the value representing the restless due each feature determined by restlessness and higher, defined restless with the degree of shot s _k Calm the (s _k) as in Equation (9). Based on the three values obtained above, the degree of calmness in the shot s _k is _obtained , but only certain elements do not always appear in the calmed section, but each element may be satisfied. The average is obtained and the degree of calmness is assumed.

3. 3. Device Configuration and Processing Procedure 3.1 Device Configuration The block diagram of FIG. 5 shows the configuration of the summary video creation device 1 in this embodiment. The summary video creation device 1 includes a control unit 2, a storage unit 3, a data input unit 4, an operation unit 5, and a data output unit 6.

  The control unit 2 is configured by a computer having a central processing unit (CPU) that executes instructions of a predetermined program, a random access memory (RAM) that expands the program, and a read only memory (ROM) that stores programs and data. Has been. Then, the control unit 2 executes the video editing program, thereby performing the cut detection unit 11, the shot analysis unit 12, the video analysis unit 13, the audio analysis unit 14, the subject detection unit 15, the index generation unit 16, and the section extraction unit 17. , And functions as a dependency level detection unit 18 and a summary video generation unit 19.

  The video editing program can be supplied to the computer from a recording medium on which the program is recorded. The recording medium on which the video editing program is recorded may be configured to be separable from the computer, or may be incorporated in the computer. Even if this recording medium is mounted on a computer so that the recorded program code can be directly read by the computer, it can be read via a program reading device connected to the computer as an external storage device. It may be attached to.

  As the recording medium, for example, magnetic tape, flexible disk, hard disk, CD-ROM, MO, MD, DVD, CD-R, IC card, various ROMs and the like can be used.

  The control unit 2 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. That is, the video editing program may be supplied in the form of a carrier wave or a data signal sequence in which the program code is embodied by electronic transmission.

  In the present embodiment, it is assumed that the respective units of the control unit 2 are realized by a computer and a video editing program. However, the respective units of the control unit 2 may be configured by hardware.

  The storage unit 3 is configured by a hard disk, and stores video data supplied from the outside, data generated by processing executed by the control unit 2, and the like. Note that some of the various data illustrated in FIG. 5 as stored in the storage unit 3 may be stored in a RAM or the like inside the control unit 2 instead of being stored in the storage unit 3. The storage unit 3 is not limited to a hard disk, but may be any storage device that can store the data.

  The data input unit 4 is for inputting video data supplied from the outside to the summary video creation device 1 into the summary video creation device 1. The data output unit 6 is created by the summary video creation device 1. The summary video data is output to the outside of the summary video creation device 1.

  The operation unit 5 receives an operation input from the operator of the summary video creating apparatus 1 and outputs a signal corresponding to the operation input to the control unit 2.

  Details of the functions and operations of each unit of the summary video creation device 1 will be described below based on a flowchart.

3.2 Overall Flow The overall processing flow in the summary video creation apparatus 1 will be described based on the flowchart of FIG.

  First, when video data is input via the data input unit 4, it is stored as video data 51 in the storage unit 3 (step S1). Then, the cut detection unit 11 detects a cut included in the video based on the video data 51, and stores the cut position as the cut position 52 in the storage unit 3 (step S2). The cut position 52 can be represented by, for example, the elapsed time from the beginning in the video. Based on the cut position 52, the shot analysis unit 12 detects the length of each shot (step S3).

  Then, the video analysis unit 13 creates a spatiotemporal projection image 53 (see FIG. 2A) of the video based on the video data 51 and stores it in the storage unit 3 (step S4). Thus, the motion of the video is detected based on the spatiotemporal projection image 53 (step S6).

  Further, the sound analysis unit 14 creates a sound spectrogram 54 (see FIG. 3) of the sound added to the video based on the audio data included in the video data 51 and stores it in the storage unit 3 (step S4). ) The sound analysis unit 14 detects the nature of the music added to the video based on the sound spectrogram 54 (step S7).

  Further, the video analysis unit 13 detects the presence or absence of the subject in the video (step S8).

  Then, the index generation unit 16 generates the action degree, the tightness degree, and the calmness degree based on the detection results of steps S3, S5, and S7. A settled section is extracted (step S9). In addition, the dependency level detection unit 18 detects the dependency relationship of each section (step S10). Then, based on the section extracted in step S9 and the dependency of each section detected in step S10, a summary video is generated by adopting shots by the summary video generation unit 19 (step S11).

  Below, each said step S is demonstrated in detail. The cut detection process in step S2 and the sound spectrogram creation process in step S6 can use well-known processes, and thus detailed description thereof is omitted here.

3.3 Shot Length Detection A shot length detection process by the shot analysis unit 12 will be described based on the flowchart of FIG.

The shot analysis unit 12 calculates the shot length SL (s _k ) of each shot based on the cut position 52 (step S001).

When the calculated shot length SL (s _k ) is larger than the threshold Th _shot (S002), the shot analysis unit 12 determines that the calmness is high and sets SVL _C (s _k ) = 1 ( If the calculated shot length SL (s _k ) is smaller than the threshold Th _shot (S004), it is determined that the action property is high, and SVL _A (s _k ) = 1 (See step S005, equation (1)).

As described above, the shot analysis unit 12 includes the feature amount (SL (s _k )) indicating the shot duration, and the feature amounts (SVL _C (s _k ), SVL _A () indicating the length of the shot duration. s _k )). The length of the duration of a shot is the degree of relative length of each part shot with respect to the entire video. Note that SL (s _k ), SVL _C (s _k ), and SVL _A (s _k ) generated by the shot analysis unit 12 are stored in the storage unit 3 (not shown), and later the index generation unit 16 and sections Used for processing by the extraction unit 17.

3.4 Creation of Spatiotemporal Projection Image A process of creating a spatiotemporal projection image by the video analysis unit 13 will be described based on the flowchart of FIG.

  First, the video analysis unit 13 pays attention to each horizontal line of y = 30, 60, 90 in each frame (horizontal direction x = 160 pixels, vertical direction (y) = 120 pixels) in the video, and each horizontal line The average luminance line of each frame is created by averaging the luminance of the pixels at every pixel of the same x coordinate. Then, the average luminance lines are arranged in the time order of the frames to create a spatiotemporal projection image as shown in FIG. 2A (step S101).

  The video analysis unit 13 then, based on the created spatiotemporal projection image, a binary image in which the edge in the image scanning direction is emphasized (image scanning direction edge enhanced image) and a binary image in which the edge in the time axis direction is enhanced. (Time-axis direction edge enhanced image) is generated (steps S102 and S103).

3.5 Motion Detection A video motion detection process performed by the video analysis unit 13 will be described with reference to the flowchart of FIG.

The video analysis unit 13 uses the image scanning direction edge enhanced image created in step S102 of FIG. 8 to refer to the portion corresponding to each shot in the image scanning direction edge enhanced image, and 10 pixels existing in that portion. The number of edges configured as described above is calculated, and the result is set as the number of edges E _v (s _k ) of the shot (see formula (2)) (step S201). Then, based on the formula (2), a value VTIV _A (s _k ) representing the action property based on the motion of the image is calculated (step S202).

Next, the video analysis unit 13 uses the time-axis direction edge emphasized image created in step S103 of FIG. 8 to set an edge in the time axis direction in each portion corresponding to each shot in the time-axis direction edge emphasized image. While tracking, score addition is performed based on FIGS. 4A and 4B, and the result is Sum (s _k ) (see equation (7)) (step S203). Then, based on Expression (7), a value VTIV _C (s _k ) representing calmness based on the motion of the image is calculated (step S204).

In this way, the video analysis unit 13 generates feature quantities (VTIV _A (s _k ), VTIV _C (s _k )) indicating the degree of intenseness of video motion. The intensity of motion of the image is the degree of relative intensity of the movement of each part with respect to the entire image. Although not shown, VTIV _A (s _k ) and VTIV _C (s _k ) generated by the video analysis unit 13 are stored in the storage unit 3 and are used later for processing by the index generation unit 16.

3.6 Music Property Detection Music property detection processing by the voice analysis unit 14 will be described based on the flowchart of FIG.

The voice analysis unit 14 calculates the average value of the duration IL (s _k ) of the instrument sound component in each shot based on the sound spectrogram 54 (step S301). The calculation of the average value is performed by dividing the total duration of the instrument sound components in the total of 10 shots of 5 shots before and 4 shots after that shot by the number of shots 10 (formula (3) (See (8)).

When the calculated average value is larger than the threshold value _ThinstC (S302), the speech analysis unit 14 determines that gentle music is flowing and sets MV _C (s _k ) = 1 (step S303, equation (8)), and when the calculated average value is less than the threshold value Th _instA is (S304), it is determined that intense music is flowing and _{MV a (s k) = 1} ( step S305, equation (See (3)).

As described above, the voice analysis unit 14 generates feature quantities (MV _C (s _k ), _{M A} (s _k )) indicating the degree of the duration of music. The degree of duration of the instrument sound component is the degree of the length of the component that appears as a line segment by the instrument constituting the rhythm on the sound spectrogram, that is, the degree of the sound constituting the melody. Note that MV _C (s _k ) and MV _A (s _k ) generated by the voice analysis unit 14 are stored in the storage unit 3 (not shown), and are used later for processing by the index generation unit 16.

3.7 Detecting the subject When the brightness change is different from the surroundings in the image and there is an emphasized object, it is important because the shot is emphasized in conveying the contents. Therefore, the subject is detected in each shot as follows.

  Based on the flowchart of FIG. 11, the subject detection process by the subject detection unit 15 will be described.

The subject detection unit 15 performs the following process on the first frame (first frame) of each shot based on the video data 51 and the cut position 52. First, the image of the first frame is converted into a grayscale 16-gradation representation (step S401). As a result, the edge density is high on the image where the complex object exists, and this edge is detected (step S402).
Further, the main body detection unit 15 divides the first frame of 160 pixels × 120 pixels into blocks of 8 pixels × 6 pixels (step S403), and unifies the color of each block with the main colors in the block (step S404), Region division is performed in the HSV color system (step S405).

  Then, the subject detection unit 15 identifies a rectangular region where the subject may exist based on the distribution of blocks having a high edge density (step S406), and the number of blocks in the maximum region within the rectangular region is a predetermined threshold (for example, 15%) or more (step S407), it is determined that the subject exists, and the subject “present” is recorded in the subject presence / absence 59 for the shot (step S408).

3.8 Extraction of Emphasized Section Based on the flowchart of FIG.

First, the index generation unit 16 calculates the action degree and calmness degree of each shot. Specifically, the index generation unit 16 calculates the action degree and calmness degree based on the equations (4) and (9), respectively, and the calculated action degree and Action (s _k ) and calmness degree are calculated. Calm (s _k ) is stored in the storage unit 3 as the action degree 56 and the calmness degree 58, respectively (step S501). It should be noted that when the mathematical expressions (4) and (9) are calculated, SVL _A (s _k ) and SVL _C (s _k ) calculated by the shot analysis unit 12 and VTIV _A (s) calculated by the video analysis unit 13 are used. _k) and VTIV _C (s _k), using the calculated MV _a (s _k) and MV _C (s _k) by the voice analyzer 14.

  In addition, the action degree and calmness degree calculated for each shot are smoothed and stored in the storage unit 3 (step S502). Smoothing is performed by taking the average of the action degree and calmness degree of the shot of interest and a total of five shots, two shots before and after the shot. By smoothing in this way, it is possible to extract sections based on rough fluctuations in the degree of action and the degree of calmness, so a more desirable result can be obtained. Therefore, in the section extraction process, smoothed values are used as the action degree and the calmness degree.

  Next, the section extraction unit 17 extracts action sections, tight sections, and calm sections. For this purpose, the section extraction unit 17 performs the following processing for each shot.

  First, based on the shot lengths of the preceding and following shots including the shot of interest (focus shot), the shot of interest is included in a section in which the shot length gradually decreases (section that satisfies the if expression of Formula (5)). It is determined whether or not (step S503). If it is included, the shot of interest is stored in the storage unit 3 as a tight section 61 (step S504). In the above determination, a section in which only one shot is longer than the previous shot and other shots are gradually shortened may be regarded as a section in which the shot length is gradually shortened.

  When the shot of interest is not included in the section in which the length of the shot is gradually shortened, the action performance level 56 of the focus shot is equal to or higher than a predetermined threshold value, and the action performance level 56 of the shot of interest is calm. It is determined whether or not the condition of greater than 58 is satisfied (step S505). If the above condition is satisfied, the condition that the action performance degree 56 is greater than the calmness degree 58 after the shot of interest continues. The group of shots that are satisfied is stored in the storage unit 3 as the action section 60 (steps S506 to S509).

  If the condition of step S505 is not satisfied, the condition that the degree of calmness 58 of the target shot is equal to or greater than a predetermined threshold and the degree of calmness 58 of the target shot is larger than the action degree 56 is set. It is determined whether or not it is satisfied (step S510), and if the above condition is satisfied, the shot group that satisfies the condition that the calmness degree 58 is greater than the actionness degree 56 after the shot of interest is calm 62 is stored in the storage unit 3 (steps S511 to S514).

3.9 Detecting Dependencies of Sections If sections of different nature are consecutive, they become a dependency relationship between cause and effect. Therefore, it is possible to consider the context of the story by detecting their relationship.

  The video sections representing the cause and the result have a subordinate relationship, but since the properties are different, the impression can be strengthened by simultaneously adopting these sections in the summary video. Paying attention to the difference in the properties of the preceding and following sections, the difference in the average value of the degree of action, the degree of tension, or the degree of calmness is obtained and used as the degree of dependency (dependency). By obtaining the degree of dependency, it is a clue when determining which section to be adopted for the summary video from the section emphasized in editing and the preceding and following sections that are in a dependency relationship. As a result, it is possible to adopt a section having a strong dependency relationship with the section emphasized for editing as a summary video.

  Based on the flowchart of FIG. 13, the processing for detecting the dependency relationship between the sections will be described.

First, the index generation unit 16 calculates the tightness degree of each shot in the tight section. Specifically, the index generation unit 16 calculates the degree of tension based on the formula (5), and stores the calculated degree of tension Tension (s _k ) in the storage unit 3 as the degree of tension 57 (step) S601). It should be noted that SL (s _k ) calculated by the shot analysis unit 12 is used when calculating Equation (5).

  Next, the dependency level is detected by the dependency level detection unit 18. Therefore, the dependency level detection unit 18 performs the following process for each section.

  First, it is determined whether or not the section of interest (target section) is an action section (step S602).

  If it is an action section, it is further determined whether or not a tense section continues after the section of interest (step S603). If the tense section continues, the action property level of shots included in these two sections is determined. The difference between the average values of 56 is calculated, and the calculation result is stored in the storage unit 3 as the dependency 63 between the target section and the next section (step S604).

  If the attention section is not an action section, it is further determined whether or not a calm section continues after the attention section (step S605). If the calm section continues, the shots included in these two sections are determined. The difference of the average value of the action degree 56 is calculated, and the calculation result is stored in the storage unit 3 as the dependency 63 between the attention interval and the next interval (step S606).

  Even when the attention section is a tight section or a calm section, the degree of dependency 63 between the attention section and the next section is calculated and stored in the storage unit 3 in the same manner as in the action section. Steps S607 to S611, S612 to S616).

3.10 Generation of Summary Video A summary video generation process will be described based on the flowchart of FIG.

  First, when the user operates the operation unit 5, the restriction time of the summary video designated by the user is input (step S701). The constraint time is determined by designating any one of 5, 10, 15, 20, 25, and 30 minutes, for example.

  Next, the summary video generation unit 19 equally divides the video data into n (for example, n = 20) along the time axis (step S702). Then, the summary video generation unit 19 performs the following process for each of the divided n periods.

  First, the summary video generation unit 19 calculates the number of shots occupied by the action section, the tight section, and the calm section included in the period of interest (attention period) (step S703), and the ratio of the number of shots is calculated. Correspondingly, the time length (constraint time) of the action section, the tight section, and the calm section that are adopted for the summary video from the attention period is calculated (step S704).

  Then, the summary video generation unit 19 adopts shots in the following manner until the constraint time of the action section is satisfied in the action section included in the attention period. That is, among the shots that have not been adopted, the shot that has the subject and has the highest action degree is adopted (step S705), and the degree of dependency is high among the sections adjacent to the action section including the adopted shot. A process of selecting a section (step S706) and adopting a shot that is temporally closest to the action section including the adopted shot among the unadopted shots in the selected section (step S707) is performed. Repeat until the constraint time is satisfied.

  In addition, the summary video generation unit 19 selects shots in the tension period and the calm period included in the attention period as in the case of the action period (steps S708 to S710, S711 to S713).

  The summary video generation unit 19 stores the shot adopted as described above in the storage unit 3 as summary video data 64. The summary video data 64 may be data created by extracting and connecting portions corresponding to the adopted shots from the video data 51, but the portions corresponding to the adopted shots can be specified in the video data 51. Data indicating information may be used.

  Here, in order to generate the summary video, the description is based on the processing result by the voice analysis unit 14 and the detection result by the subject detection unit 15, and these are for generating a more accurate summary video. Although useful, even if these are omitted, it is possible to generate an accurate summary video.

4). Summary of Summary Video Creation Device As described above, in the summary video creation device (video editing device) 1, the shot analysis unit (shot recognition means) 12 uses the shot data for each part of the video based on the video data 51. Recognize features according to the size. Also, the video analysis unit (video recognition unit) 13 recognizes the characteristics of each part of the video according to the intensity of motion of the video based on the video data 51.

  Then, the section extraction unit (emphasis section specifying means) 17 recognizes the recognition results by the shot analysis unit 12 and the video analysis unit 13 (the action degree 56, the tightness degree 57, and the calmness generated by the index generation unit 16 based on them). Based on the degree of sex 58), a section corresponding to the emphasis section (action section, tight section, calm section) is specified in the video data. Further, the dependency level detection unit (dependency level detection means) 18 detects the level of dependency between the emphasis sections based on the recognition results by the shot analysis unit 12 and the video analysis unit 13.

  Then, the summary video generation unit (summary creation means) 19 uses the recognition result from the shot analysis unit 12 and the video analysis unit 13 and the detection result from the dependency level detection unit 18 to adopt the summary video from the enhancement section. To decide.

  As a result, the summary video creation device 1 reflects the summary video in accordance with the grammar of the movie, that is, the emphasis section emphasized on editing, and the dependency between these emphasis sections, so that the viewer can view the entire contents. It is possible to create a summary video that is easy to grasp accurately.

  Also, in the summary video creation device 1, the duration of the instrument sound component included in the audio for each part of the video is determined based on the audio data added to the video data 51 by the audio analysis unit (voice recognition means) 14. It is desirable to recognize the corresponding feature and use it for each processing in the section extraction unit 17, the dependency level detection unit 18, and the summary video generation unit 19.

  In many cases, audio is added to the video, and in this case, the characteristic properties of the action section and the calm section appear as the duration of the instrument sound component included in the audio. Therefore, in addition to the characteristics according to the length of the duration of the shot and the characteristics according to the intensity of the motion of the video, the characteristics according to the length of the duration of the instrument sound component are recognized and based on these. By specifying the emphasis section, detecting the degree of dependency, and determining the video portion to be adopted as the summary video, a more accurate summary video can be created.

  In the summary video creating apparatus 1, it is desirable that the subject detection unit (subject detection means) 15 detects the presence of the subject for each part of the video based on the video data 51 and uses it for processing in the summary video generation unit 19.

  The part where the subject is present becomes an important part in conveying the content of the video to the viewer, and the summary video that preferentially adopts that part is easier to understand than the one that does not consider it Become. Therefore, a more accurate summary video can be created by detecting the presence of the subject and determining a portion to be adopted for the summary video from the emphasis section based on the detection result.

5. Experiments and Evaluations 6 university students examined the summary video created by the summary video creation device 1 (Example) and the summary video created without considering the content and context (comparative example). He evaluated whether the contents of the movie and the flow of the story were easy to understand.

  As a comparative example, a summary video with the following cut frequency was created. For a frame every 5 seconds from the beginning of the movie, find the number of cuts in 10 seconds from there. The key frames are set in order from the frame having the largest number of cuts included in the 10 seconds. Here, the key frame is a frame to be focused on when creating a summary video. Shots that include key frames are taken as the first shot, and shots from the first shot until the total time exceeds 10 seconds are connected and adopted as a summary video. The process was repeated until the time length of the summary video reached the target time, and the selected sections were arranged in time order to obtain a summary video. The summary video of this comparative example is a video in which only shots with short shot lengths and strong impressions are connected.

  Summary of 5 and 10 minutes of two movies ("Speed 2" directed by Jan De Bon, 1997, action, "AI" directed by Steven Spielberg, 2001, science fiction / drama) as an example Have the subject watch the video and a summary video of 5 minutes and 10 minutes created as a comparative example, and make a five-level evaluation of the two aspects of ease of understanding the content of the story and ease of understanding the flow of the story I was asked to. The breakdown of 5 stages is that 5 is better in the example, 4 is better if the example is better, 3 is neither, 2 is rather better in the comparative example, 1 is better The comparative example is better.

  The format of the video data used is frame size 160 x 120 [pixel], frame rate 30 [frames / sec.], 24-bit color, audio format is sampling frequency 22.050 [kHz], quantization 8 bits, monaural is there.

  For subjects who have not seen the movie used in this experiment in order to evaluate whether the causal relationship between events and the story development are comprehensible summaries, read the movie synopsis in advance. The experiment was carried out with some understanding of the content of the story.

  The evaluation results are shown in FIG. In FIG. 15, the average evaluation values of six people are plotted. Overall, the embodiment is a summary video that is easy to understand, both in terms of content and flow. In the embodiment, an action section, a tight section, and a calm section are extracted as sections that are emphasized for editing, and a summary video is created by obtaining sections subordinate thereto. It is thought that a summary video that is easy to understand was created.

  In the present embodiment, a method of creating a summary video that makes it easier to understand the content of the story by considering the content and context of the movie has been proposed. By extracting an action section, a tight section, and a calm section based on the grammar of the movie, it is possible to include a section that has been editorially emphasized so that the contents can be effectively transmitted to the summary video. Furthermore, by obtaining a dependency relationship with these sections, it is possible to create a summary video without losing the connection between the previous and next stories.

  In creating a summary video of a movie, sound effects are considered to be an important factor, so it is desirable to create a summary video in consideration of sound effects.

  The present invention can be used to automatically create a summary video from a video having a story, such as a movie or a TV drama, and can be applied to, for example, a video viewing device provided to a viewer. The present invention can also be applied to an apparatus for creating an advertisement video provided to a producer.

It is drawing for demonstrating a spatiotemporal projection image. (A) is drawing which shows a spatiotemporal projection image, (b) is a drawing which shows the edge image which extracted the edge from the spatiotemporal projection image of (a). It is drawing which shows the example of a sound spectrogram. (A) is a figure which shows the value used for the calculation for calculating | requiring the flatness scale of an image | video, (b) is drawing which shows the order of the edge tracking performed in order to obtain | require the scale of an image | video flatness. It is a block diagram which shows the structure of the summary image | video production apparatus which concerns on one Embodiment of this invention. 6 is a flowchart showing an overall flow of summary video creation processing in the summary video creation device of FIG. 5. It is a flowchart which shows the specific content of the detection process of the shot length in FIG. It is a flowchart which shows the specific content of the creation process of the spatiotemporal projection image in FIG. It is a flowchart which shows the specific content of the motion detection process in FIG. It is a flowchart which shows the specific content of the detection process of the property of the music in FIG. It is a flowchart which shows the specific content of the detection process of the main body in FIG. It is a flowchart which shows the specific content of the extraction process of the area in FIG. It is a flowchart which shows the specific content of the detection process of the dependency relationship in FIG. It is a flowchart which shows the specific content of the production | generation process of the summary image | video in FIG. It is a graph which shows the evaluation result which compared the Example of this invention with the comparative example.

Explanation of symbols

1. Summary video creation device (video editing device)
2 control unit 3 storage unit 4 data input unit 5 operation unit 6 data output unit 11 cut detection unit 12 shot analysis unit (shot recognition means)
13 Video analysis unit (video recognition means)
14 Voice analysis unit (voice recognition means)
15 Subject detection unit (Subject detection means)
16 Index generation unit 17 Section extraction unit (emphasis section specifying means)
18 Dependency detector (Dependency detector)
19 Summary video generator (summary creation means)

Claims (8)

In a video editing device for creating a summary video from a video including an emphasis section that can be identified based on the length of each shot constituting the video and the intensity of movement in the video,
Shot recognition means for recognizing features according to the length of the duration of the shot for each part of the video based on the video data;
Video recognition means for recognizing features according to the intensity of motion of the video for each part of the video based on the video data;
Based on the recognition result by the shot recognizing unit and the video recognizing unit, the emphasis section specifying unit for specifying the section corresponding to the emphasis section in the video data;
Dependency degree detection means for detecting the degree of dependence between the emphasis sections based on the recognition results by the shot recognition means and the image recognition means;
Video comprising: summary creating means for determining a portion to be adopted for the summary video from the emphasis section based on the recognition result by the shot recognition means and the video recognition means and the detection result by the dependency level detection means. Editing device. The shot recognizing means generates, as a recognition result, a feature amount indicating the duration of the shot and a feature amount indicating the degree of the duration of the shot,
The video editing apparatus according to claim 1, wherein the video recognizing unit generates a feature amount indicating a degree of motion intensity of the video as a recognition result. Voice recognition means for recognizing a feature corresponding to the duration of the instrument sound component included in the voice for each part of the video based on the voice data added to the video data;
The enhancement section specifying means further specifies a section corresponding to the enhancement section in the video data based on the recognition result by the voice recognition means,
The dependency level detecting means further detects a dependency level between the emphasis sections based on a recognition result by the voice recognition means,
The video editing apparatus according to claim 1, wherein the summary creation unit further determines a portion to be adopted for the summary video from the emphasis section based on a recognition result by the voice recognition unit.   The video editing apparatus according to claim 3, wherein the voice recognition unit generates a feature amount indicating a degree of duration of the musical instrument sound component as a recognition result. Based on video data, further comprising subject detection means for detecting the presence of a video subject for each part of the video,
The video editing apparatus according to claim 1, wherein the summary creation unit further determines a portion to be adopted for the summary video from the emphasis section based on a detection result by the subject detection unit.   6. A video editing program for operating the video editing apparatus according to claim 1, wherein the video editing program causes a computer to function as each means.   A computer-readable recording medium on which the video editing program according to claim 6 is recorded. In a video editing method for creating a summary video from a video including an emphasis section that can be identified based on the length of each shot constituting the video and the intensity of movement in the video,
Shot recognition processing for recognizing features according to the length of the shot duration for each part of the video based on the video data;
Based on the video data, video recognition processing for recognizing features according to the intensity of motion of the video for each part of the video,
Based on the recognition result by the shot recognition process and the video recognition process, an emphasis section specifying process for specifying a section corresponding to the emphasis section in the video data;
Based on the recognition result by the shot recognition process and the video recognition process, a dependency level detection process for detecting a dependency level between the emphasis sections;
A summary creation process for determining a portion to be adopted in the summary video from the emphasis section based on the recognition result by the shot recognition process and the video recognition process and the detection result by the dependency detection process. Editing method.

Images