JP5129198B2 - Video preview generation device, video preview generation method, and video preview generation program - Google Patents

Description

  The present invention relates to a video preview generation apparatus and method for generating a video preview from a video material, and a video preview generation program used for realizing the video preview generation apparatus.

  For movies, a trailer for the movie is generated for use in advertising the movie.

  In the past, the generation of such a video preview has been performed manually, and since the workload is large, it is hoped that a technology that can automatically generate a video preview can be constructed. It is rare.

  Against this background, Non-Patent Document 1 below extracts video sections associated with the introduction text of the electronic program guide by associating the introduction text of the electronic program guide with the closed caption of the video. Thus, an invention of automatically generating a program introduction video based on this is disclosed.

Yoshihiko Kawai, Hideki Sumiyoshi, Nobuyuki Yagi, "Automatic Generation Method of Program Introduction Video Using Introductory Text in Electronic Program Guide", IEICE Transactions Vol.J91-D No.8 pp.2157-2165.

  Certainly, in Non-Patent Document 1, by extracting the introductory text of the electronic program guide and the closed caption of the video, the video section associated with the introductory text of the electronic program guide is extracted, and the program introduction video is based on that. The invention of automatically generating the is disclosed.

  However, the invention disclosed in Non-Patent Document 1 requires closed captioning of the video in order to automatically generate the program introduction video.

  From the above, according to the invention disclosed in Non-Patent Document 1, there is a problem in that it is not possible to automatically generate a notice for a video to which closed captions are not given.

  For example, closed captions are generally not assigned to unedited video. Thus, according to the invention disclosed in Non-Patent Document 1, there is a problem in that a preview of an unedited video cannot be automatically generated.

  In addition, although the electronic program guide is generally assigned to commercial video such as a television program, the actual situation is that other video is not provided.

  As can be seen from this, the invention disclosed in Non-Patent Document 1 cannot be said to be a method for automatically generating a video preview for a wide range of videos.

  The present invention has been made in view of such circumstances, and it is an object of the present invention to provide a new video notice generation technology that can automatically generate a notice of a video regardless of the attributes of the video such as commercial video and home video. And

  In order to achieve this object, the video preview generation device of the present invention realizes (1) a video section (one or a plurality of frames) of a video material in order to automatically generate a video preview from the video material. One or more video sections among a video section associated with an audio filler, a video section associated with excessive camera work, and a video section in which no captured video exists. And a filter means for removing the detected video section from the video material, and (2) a video section of the video material that has not been removed by the filter means as a processing target, and the audio enhancement level is higher than a predetermined threshold One or more video sections among a large video section, a video section in which the face size of the subject is larger than a predetermined threshold, and a video section in which the facial expression of the subject is larger than a predetermined threshold An opening generation means for detecting and generating the beginning of the video preview using the detected video section; and (3) a video section of the video material that has not been removed by the filter means and does not become the beginning. As a processing target, a video section in which the time change amount of the fundamental frequency of audio is larger than a predetermined threshold, and the degree of audio enhancement increases with time, and the time change amount of the increase is a predetermined threshold value. Main part generating means for detecting any one or more video sections in a larger video section and generating the main part of the video preview using the detected video section; and (4) generation of the head generating means. By combining the beginning and the main part generated by the main part generating means, a video notice generating means for generating a video notice is provided.

  In adopting this configuration, the beginning generation means synchronizes one of the tone, tempo, and enhancement degree of the musical sound with the timing of switching the image at the beginning, to the image at the beginning. The main part generating means may synchronize one of the tone, tempo, and enhancement degree of the musical sound with the video switching timing in the main part in order to synchronize the musical sound with the main part. May be granted.

  Each of the above processing means can also be realized by a computer program. This computer program is provided by being recorded on an appropriate computer-readable recording medium or provided via a network, and is used when implementing the present invention. The present invention is realized by being installed and operating on a control means such as a CPU.

In the video preview generation device of the present invention configured as described above, first, as a filtering process, the video section of the video material that is the target of video preview generation is processed,
(B) Video segment associated with audio filler (b) Video segment associated with excessive camerawork (c) Any one or more video segments (one The detected video section is removed from the video material.

  In this way, first, as a filtering process, a video section with a low video quality included in a video material to be generated as a video preview is detected, and the detected video section is removed from the video material.

Subsequently, the video section of the video material that has not been removed by the filtering process is processed.
(B) Video segment in which the degree of audio enhancement is greater than a predetermined threshold (b) Video segment in which the face size of the subject is greater than the predetermined threshold (c) Within the video segment in which the facial expression of the subject is greater than the predetermined threshold Any one or more video sections (which may be composed of one frame) are detected, and the head of the video preview is generated using the detected video sections.

  That is, the video section of the video material that has not been removed by the filtering process is set as a processing target, a video section having a large impact included in the video material to be processed is detected, and the detected video section is used to generate a large impact. The beginning of the video preview composed of video sections is generated.

Subsequently, the video section of the video material that was not removed by the filtering process, and the video section that did not become the beginning of the video preview,
(B) Video section in which the amount of time change in the fundamental frequency of audio is greater than a predetermined threshold (b) Video in which the degree of enhancement of audio increases with time and the amount of increase in time change is greater than the predetermined threshold Any one or more video sections (may be composed of one frame) in the section are detected, and a main part of the video preview is generated using the detected video section.

  In other words, a video section of a video material that has not been removed by the filtering process, and a video section that indicates a highlight scene included in the processing target video section with a video section that is not the beginning of the video preview as a processing target Is detected, and the detected video section is used to generate the main part of the video preview composed of the video section showing the highlight scene.

  Finally, a video trailer is generated by combining the beginning of the generated video trailer and the main part of the generated video trailer.

  In this way, in the present invention, without using the introduction text of the electronic program guide or the closed caption of the video, from the video material for which the video preview is to be generated, the beginning and the high-level composed of the video section having a high impact are recorded. Processing is performed so as to automatically generate a video trailer having a main part composed of video sections showing light scenes.

  Thus, according to the present invention, it is possible to automatically generate a preview of a video regardless of the attributes of the video such as commercial video and home video.

It is an apparatus block diagram of the image | video notification production | generation apparatus of this invention. It is a flowchart which the video preview production | generation apparatus of this invention performs. It is a flowchart which the video preview production | generation apparatus of this invention performs. It is a flowchart which the video preview production | generation apparatus of this invention performs. It is a flowchart which the video preview production | generation apparatus of this invention performs. It is explanatory drawing of the detection process of a camera work. It is explanatory drawing of an example of a junk shot. It is explanatory drawing when a musical sound is superimposed on a video preview.

  Hereinafter, the present invention will be described in detail according to embodiments. Here, the video section described below includes a case where the video section is configured by one frame.

  FIG. 1 illustrates an example of a device configuration of a video preview generation device 1 having the present invention.

  As shown in this figure, the video preview generation apparatus 1 of the present invention realizes automatic generation of a video preview from video material without using introductory text of an electronic program guide or closed captions of the video. The video material input unit 10, the video material storage unit 11, the filter unit 12, the video preview generation video material storage unit 13, the head generation unit 14, the main generation unit 15, and the video preview generation unit 16. A video preview storage unit 17 and a video preview output unit 18.

  Next, each of these processing units will be described.

  The video material input unit 10 inputs a video material to be generated as a video preview and stores it in the video material storage unit 11.

  The filter unit 12 uses the video material stored in the video material storage unit 11 as a processing target, and removes from the video material a low-quality video section that cannot be used to generate a video preview, and removes the video segment. The video material thus stored is stored in the video material storage unit 13 for generating a video preview.

  Video material often includes video and musically redundant scenes, and if you edit without removing these redundant scenes, a video preview containing unnecessary scenes will be generated. become. Such redundancies include voice fillers (speaking words such as “Ano”, a phenomenon peculiar to spoken language), excessive camera work, and junk shots (Junk Shot: video sections where there is no valid video shot). It seems to be related to the feature.

  Therefore, in order to remove such redundancy, the filter unit 12 (a) detects the appearance position of the audio filler included in the sound information of the video material, and specifies the video section associated with the appearance position. And (b) determining the camera work when the video material is shot, thereby detecting the excessive camera work appearance position and specifying the video section associated with the appearance position. An excessive camera work detection unit 121, and (c) a junk shot detection unit 122 that detects an appearance position of a junk shot included in the video material and identifies a video section associated with the appearance position, The video section specified by each of these detectors is processed to be removed from the input video material.

  The head generation unit 14 executes processing for automatically generating the head of a video preview composed of video sections having a large impact, with the video material stored in the video material storage unit 13 for video preview generation as a processing target.

  In order to execute the automatic generation process of the beginning, the beginning generation unit 14 determines the degree of enhancement of the sound included in the sound information of the video material, so that the degree of sound enhancement is higher than a predetermined threshold value. By detecting the appearance position that increases, and by determining the face size of the subject included in the video material, the voice enhancement degree determination unit 140 that identifies the video section associated with the appearance position, and the face size is determined. By detecting an appearance position that is larger than a predetermined threshold and identifying a video section associated with the appearance position, and (c) determining a facial expression of a subject included in the video material. A facial expression determination unit 142 that detects an appearance position where the facial expression is greater than a predetermined threshold and identifies a video section associated with the appearance position; and (d) a speech enhancement level determination unit according to a predetermined rule. 140 By assigning an order to the video sections specified by the face size determination unit 1411 / facial expression determination unit 142 and determining the timing of switching those video sections in a form that synchronizes with the previously prepared musical sound, And a musical sound assigning unit 143 for generating the beginning of the video preview.

  The main part generation unit 15 is composed of video sections indicating highlight scenes, with the video sections stored in the video preview generation video material storage section 13 being processed as video sections that are not the beginning of the video preview. A process for automatically generating the main video preview is executed.

  In order to execute the automatic generation process of the main part, the main part generation unit 15 determines (a) the temporal change amount of the basic frequency of the audio by determining the temporal change of the basic frequency of the audio included in the sound information of the video material. Detecting an appearance position where is greater than a predetermined threshold and identifying a video section associated with the appearance position; and (b) enhancement level of audio included in the sound information of the video material By determining the time change, the appearance position where the enhancement degree of the sound increases with time and the time change amount of the increase is larger than a predetermined threshold is detected and associated with the appearance position. A voice enhancement level pattern determination unit 151 that identifies a video section, and (c) according to a predetermined rule, the video frequency range determined by the fundamental frequency pattern determination unit 150 / sound enhancement level pattern determination unit 151 By giving the order Te, in the form of taking the tone to be prepared in advance synchronously, by determining the timing of switching of their video section, and a musical tone assigning unit 152 for generating a main picture notice.

  The video preview generation unit 16 generates a video preview by combining the beginning generated by the beginning generation unit 14 and the main part generated by the main part generation unit 15, and the generated video preview is stored in the video notice storage unit 17. To store.

  When there is a video output request, the video preview output unit 18 reads out and outputs a video preview with an output request from the video preview storage unit 17.

  2 to 5 show flowcharts executed by the video preview generation device 1 of the present invention configured as described above.

  Next, according to these flowcharts, processing executed by the video preview generation device 1 of the present invention configured as described above will be described.

[1] Overall Processing When there is a video preview generation request, the video preview generation apparatus 1 of the present invention first becomes a target for video preview generation in step S10 as shown in the flowchart of FIG. The video material is input and stored in the video material storage unit 11.

  Subsequently, in step S20, a predetermined filter process is performed on the video material stored in the video material storage unit 11 in order to remove a low-quality video section that cannot be used for generating the video preview, and the filter The processed video material is stored in the video material storage unit 13 for generating a video preview.

  Subsequently, in step S30, the head of the video preview is generated using the video section of the video material stored in the video material storage unit 13 for generating video preview.

  Subsequently, in step S40, the main part of the video preview is generated using the video section that is stored in the video preview generation video material storage unit 13 and does not become the beginning of the video preview.

  Subsequently, in step S50, a video preview is generated by combining the beginning generated in step S30 and the main part generated in step S40, and is stored in the video preview storage unit 17.

  Subsequently, in step S60, in response to the video preview output request, the video preview stored in the video preview storage unit 17 is output.

[2] Details of Step S20 Next, details of the filter processing executed in step S20 of the flowchart of FIG.

  When entering the process of step S20 in the flowchart of FIG. 2, the video preview generation apparatus 1 of the present invention first sets the video material stored in the video material storage unit 11 as a processing target as shown in the flowchart of FIG. In step S200, the appearance position of the audio filler is detected, the video section associated with the appearance position is specified, and in step S201, the video section is associated with the appearance position of the audio filler from the video material. Remove.

  Audio fillers often appear with utterances that are semantically unorganized, such as scenes that are uttered while thinking, and the video segments associated therewith are often video segments that are unnecessary for the generation of video previews. Therefore, the appearance position of the audio filler is detected, the video section associated with the appearance position is specified, and processing is performed to remove the specified video section from the video material.

  As a method for detecting the voice filler, for example, a time history of the fundamental frequency is measured, and when a large change is not seen in the fundamental frequency value for a predetermined time such as one second, the voice filler is Use the method of judgment. Here, for the extraction of the fundamental frequency, for example, the method shown in Reference Document 1 below can be used.

Reference 1: Sadahiro Furui, Digital Signal Processing, Tokai University Press, pp.57-63.
When the processing of step S200 and step S201 is executed in this way, subsequently, in step S202, an excessive camerawork appearance position is detected, a video section associated with the appearance position is specified, and subsequent step S203 is performed. Then, the video section associated with the excessive camerawork appearance position is removed from the video material.

  A video shot by excessive camera work has poor quality, and a video segment associated with the video is often an inappropriate video segment for generating a video preview. Therefore, an excessive camerawork appearance position is detected, a video section associated with the appearance position is specified, and processing is performed to remove the specified video section from the video material.

  Here, as a camera work detection method, for example, the method shown in Reference Document 2 below can be used.

Reference 2: Patent No. 3408117, Yukinobu Taniguchi, Akito Akutsu, Yoshinobu Tonomura, “Recording medium for recording camera operation estimation method and camera operation estimation program”
In this step S202, for example, as shown in FIG. 6, camerawork is detected by measuring how the angle of view at the time of shooting changes with time. FIG. 6 shows the transition of the lower left vertex of the angle of view with respect to the camera movement from time t0 to time t5.

  In this step S202, for example, A: regression line, B: quadratic regression curve, etc. are obtained for this transition, and the camerawork is stable when the reliability coefficient is equal to or greater than the threshold value R. If it is less than the threshold value R, it is determined that the camera work is excessive. Here, the threshold value R may be set to 0.7, for example, or may be determined by a cubic or higher regression curve.

  When the processing in step S202 and step S203 is executed in this way, subsequently, in step S204, the appearance position of the junk shot is detected, the video section associated with the appearance position is specified, and in step S205, The video section associated with the appearance position of the junk shot is removed from the video material.

  The video portion of the junk shot is not an effective video portion, and the video section associated therewith is a video section that is unnecessary for generating the video preview. Therefore, the appearance position of the junk shot is detected, the video section associated with the appearance position is specified, and processing is performed to remove the specified video section from the video material.

  As a junk shot, for example, as shown in FIG. 7, there is a case where the timing (reproduction stop position) at which a stop button is pressed after viewing a certain captured image deviates from the photographing end time. Further, when there is no photographed video, it may be displayed in blue or black, but if a section where nothing is recorded as data is detected, a junk shot can be extracted.

  In addition, for video materials for commercial use, a tool called a clapperboard that notifies the start of shooting may be used, or a screen called a color bar may be inserted. These are also positioned as junk shots, but for clappers, the use of clappers can be detected acoustically by preserving the pitch and frequency generated when the clappers are used in advance. A junk shot can be extracted by extracting a part. For color bars, if a color histogram is measured and it is composed of only 1 to N colors, the color bar screen can be detected by determining that it is a color bar. It can be extracted. Note that N may be set in advance, for example, 5 or the like.

  When the processing of step S204 and step S205 is executed in this way, subsequently, in step S206, the video material from which unnecessary video sections have been removed is stored in the video material storage unit 13 for video preview generation, and the flowchart of FIG. The process of step S20 is terminated.

  In this way, when the video preview generation apparatus 1 of the present invention enters the process of step S20 of the flowchart of FIG. 2, the video material stored in the video material storage unit 11 is executed by executing the flowchart of FIG. On the other hand, a filter process for removing a video section unnecessary for the generation of the video trailer is performed, and the video material as a result of the filter process is stored in the video material storage unit 13 for video trailer generation.

[3] Details of Processing in Step S30 Next, details of the processing for generating the head of the video notice executed in step S30 in the flowchart in FIG. 2 will be described according to the flowchart in FIG.

  As shown in the flowchart of FIG. 4, when the video preview generation apparatus 1 of the present invention enters the process of step S <b> 30 of the flowchart of FIG. 2, the video material stored in the video preview generation video material storage unit 13 is processed. First, in step S300, a video section in which the audio enhancement level is greater than a predetermined threshold is detected.

  Here, the speech enhancement degree may be obtained from the speech enhancement probability and the calm probability. Specifically, the probability ratio Ps (e) / Ps (n) between the emphasis probability Ps (e) and the calm probability Ps (n) for each audio sub-paragraph may be used as the audio enhancement level. The speech enhancement probability and the calm probability can be obtained using, for example, the method shown in Reference Document 3 below.

Reference 3: Japanese Patent No. 3803131, Kota Hidaka, Osamu Mizuno, Shinya Nakajo, “Speech Processing Method, Apparatus Using the Method, and Program”
Subsequently, in step S301, a frame in which the face size of the subject is larger than a predetermined threshold is detected, and thereby a video section (composed of one frame) in which the face size of the subject is larger than the predetermined threshold. May be detected).

  Here, as for the face size of the subject, a discriminator using Haar-like features by Adaboost learning may be used. A large number of weak classifiers may be of a cascade type, and this cascade type classifier may be applied by changing the size and position of the identification target to identify the face image area. This is described in Reference Document 4 below. If the face image area is obtained in this way, the face size can be measured.

Reference 4: Paul Viola, Michael J. Jones, Robust Real-Time Face Detection. International Journal of Computer Vision. Vol57, No.2, pp.137-154 (2004).
Subsequently, in step S302, a frame in which the richness of the facial expression of the subject is greater than a predetermined threshold (a frame having a facial expression on the face) is detected. A video section (which may be composed of one frame) that is larger than the threshold is detected.

  Here, the facial expression of the subject is performed by detecting whether or not there is an expression on the face, with a frame determined to have a face image as a processing target. For the detection of facial expressions, for example, the method shown in Reference Document 5 below can be used.

Reference 5: Kotsia I., Pitas I .: Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines. IEEE Transactions on Image Processing, Vol.16, No.1, pp. 172-187 (2007 ).
Subsequently, in step S303, the video section used for generating the head is selected from the video sections detected in steps S300 to S302, and the order is determined for the selected video section according to a predetermined rule. The beginning of the video preview is generated by arranging the video sections according to the determined order.

  For example, first, one or more frames are extracted as candidate frames at the beginning of (i) a high degree of speech enhancement, (ii) a large face size, and (iii) a rich expression. When candidate frames are continuous, a scene composed of these candidate frames is extracted. At this time, even if all of the conditions (i), (ii), and (iii) are not satisfied, for example, those that satisfy the conditions in the order of (i), (ii), and (iii) (Scene) may be extracted.

  Here, for example, the audio enhancement level is obtained for each scene (video sections such as audio sub-paragraphs and audio paragraphs described in Reference 3), and the audio enhancement levels are sorted in descending order to obtain the upper PE of the total number of scenes. A case where the value is% (the value of PE is preset to 10 or the like) is defined as high speech enhancement. As for the face size, a case where the face size is greater than or equal to the total PS% (PS value is preset to 20 or the like) is defined as large. As for facial expressions, smiles, surprises, sadness, etc. are defined as rich expressions.

  Subsequently, for example, by selecting a predetermined number of frames from the extracted candidate frames in the descending order of the speech enhancement degree, the frames used for generating the beginning are selected, and these frames are selected. For example, by determining the order of the frames in order of increasing voice enhancement or by determining the order in order of the shooting time, the frames are determined according to the determined order. The beginning of the video preview is generated by arranging them.

  Here, in this description, the frame has been described as a specific example, but the beginning of the video preview generated by the video preview generation apparatus 1 of the present invention may be a scene (video section) composed of a plurality of frames. In that case, the first scene may be generated by selecting the above conditions (i), (ii), and (iii) by the same method as described above in units of scenes.

  Candidate frames (scenes) selected according to the above conditions may include sections that look similar. If a plurality of such similar sections are included at the beginning of the video preview, the appearance information becomes redundant and increases the size of the video preview, which is inefficient.

  Therefore, it is preferable to execute a process of deleting the similar section in advance so that only one such similar section is included.

  Such processing can be performed using image information of candidate frames (scenes), for example. For example, image color information, texture information, shape information, and the like are extracted and vectorized, and one of two candidate frames (scenes) having a short vector distance is excluded from the candidate candidate frames (scenes). At this time, as to which of the two candidate frames (scenes) should be excluded, a frame with a lower priority may be excluded according to the priority based on the rules described above. For example, it is only necessary to exclude the one with the lower voice enhancement degree or the one with the slower shooting time.

  In this way, when the video preview generation apparatus 1 of the present invention enters the process of step S30 of the flowchart of FIG. 2, the video preview generation apparatus 1 stores the video preview generation video material storage unit 13 by executing the flowchart of FIG. The video section of the video material to be processed is detected, the video section with a large impact included in the video material is detected, and the beginning of the video preview composed of the video section with the high impact using the detected video section Is processed to generate.

  The video preview generation device 1 of the present invention does not assume that a musical sound is superimposed on the beginning when generating the beginning of the video preview according to the flowchart of FIG. Depending on the situation, the beginning may be edited so as to satisfy the expectation.

  In this case, the selected musical sound may be simply superimposed on the beginning, but may be superimposed in a manner that is synchronized with the tone and the degree of enhancement of the musical sound. For example, as shown in FIG. 8, the video section constituting the head may be inserted in accordance with the time when the enhancement degree sharply increases or the time when the tone changes.

  In other words, frames and scenes constituting the beginning are inserted at intervals of tone change. Also, a frame or scene constituting the beginning is inserted at a time when the emphasis degree reaches a peak. For the enhancement level, a threshold is set for the difference from the previous and subsequent i-frames and / or the enhancement level value, and the time of the enhancement level equal to or greater than the threshold is set as the insertion time of the frame or scene.

  Here, in FIG. 8, the tone change is indicated by an inverted triangle, and the enhancement degree is indicated by a curve. Although the degree of emphasis is originally a discrete value, this figure shows the discrete values connected by straight lines or curves (the discrete values can be approximated by spline curves).

  Regarding the change of the tone, the peak value of the amplitude per unit time T may be measured, and when the peak interval is about Tp, it may be determined that the tone has changed at the time interval of Tp. If a sound effect that gives a sense of urgency is indicated by an onomatopoeia, it can be expressed by “don don don ...” or the like. In such a case, it is assumed that the interval between the first “Don” and the next “Don” is constant, and for example, it appears at an interval such as Tp = 1 second. Therefore, the tone changes at the time interval of Tp. You may make it determine with it. Alternatively, the frequency per unit time T may be measured, and if a specific frequency appears at Tp intervals, it may be determined as a tone change.

  In this way, when music is superimposed on the beginning of the video preview, the original audio information exists in the scene used to generate the video preview, but this audio information is not necessarily required. It will not be. For example, by inserting a scene only for Visual, it becomes possible to give a sense of urgency and a sense of security in accordance with a sound effect (musical sound).

  Note that the musical sound to be superimposed on the beginning of the video preview is not limited to sound effects but may be music. For example, when the degree of emphasis is steep in a song, it may be a commonly called “rust” part. By inserting a frame or scene at the time of this “rust”, the sensitivity is the same as above. It is possible to realize rich editing.

[4] Details of Processing in Step S40 Next, details of the main image generation processing of the video preview executed in step S40 of the flowchart of FIG. 2 will be described according to the flowchart of FIG.

  When entering the process of step S40 in the flowchart of FIG. 2, the video preview generation device 1 of the present invention is a video material stored in the video preview generation video material storage unit 13 and does not become the beginning of the video preview. As shown in the flowchart of FIG. 5, first, in step S400, a video section in which the temporal change amount of the fundamental frequency of the audio is larger than a predetermined threshold is detected.

  Subsequently, in step S401, a video section in which the degree of enhancement of the sound increases with time and the time change amount of the increase is larger than a predetermined threshold is detected.

  Subsequently, in step S402, the video section used for generating the main part is selected from the video sections detected in step S401 and step S402, and the order is determined for the selected video section according to a predetermined rule. The main part of the video preview is generated by arranging the video sections according to the order.

  For example, first, a video segment in which (i) the amount of time variation of the fundamental frequency of the sound is large, (ii) the degree of enhancement of the sound increases with time, and the amount of time variation of the increase is large is selected as a candidate for this volume One or more video segments are extracted. At this time, even if all the conditions of (i), (ii), and (iii) are not satisfied, for example, those satisfying the conditions of (i) and (ii) in this order are extracted as candidate video sections of the main part. May be.

  Subsequently, for example, by selecting a predetermined number of video segments in descending order of the amount of temporal change in the fundamental frequency of the audio from the candidate video segments extracted in this way, the video segment used for generating the main program is selected. For example, the order of these video sections is determined by determining the order in descending order of the amount of time change in the fundamental frequency of the audio, or by determining the order from the earliest shooting time. Then, the main part of the video preview is generated by arranging the video sections according to the determined order.

  Here, the candidate video section selected according to the above-described condition may include a section that looks similar. If a plurality of such similar sections are included in the main part of the video notice, the appearance information becomes redundant and increases the size of the video notice, which is inefficient.

  Therefore, it is preferable to execute a process of deleting the similar section in advance so that only one such similar section is included.

  Such processing can be performed using, for example, image information of candidate video sections. For example, image color information, texture information, shape information, and the like are extracted and vectorized, and one of two candidate video sections having a short vector distance is excluded from the candidate video sections. At this time, as to which of the two candidate video sections to exclude, the one with the lower priority may be excluded according to the priority based on the rules described above. For example, it is only necessary to exclude the one with a smaller time variation of the fundamental frequency or exclude the one with a slower photographing time.

  In this way, when the video preview generation apparatus 1 of the present invention enters the process of step S40 in the flowchart of FIG. 2, the video preview generation apparatus 1 stores the video preview generation video material storage unit 13 by executing the flowchart of FIG. In the video material that is not the beginning of the video preview, the video segment indicating the highlight scene included in the video material is detected, and the detected video segment is used. Processing is performed so as to generate the main part of the video preview composed of the video section indicating the highlight scene.

  When generating the main part of the video preview according to the flowchart of FIG. 5, the video preview generation apparatus 1 of the present invention does not assume that the musical sound is superimposed on the main part, but by superimposing the musical sound on the main part, You may make it edit so that a main part may be filled with expectation.

  In this case, the selected musical sound may be simply superimposed on the main part, but may be superimposed in a form that is synchronized with the tone and enhancement degree of the musical sound. For example, as shown in FIG. 8, the video section constituting the main part may be inserted in accordance with the time when the emphasis is sharply increased or the time when the tone changes.

  In other words, video sections constituting the main part are inserted at intervals of tone change. Also, the video section constituting the main part is inserted at the time when the emphasis degree reaches a peak.

  In this way, when music is superimposed on the main part of the video preview, the original audio information exists in the scene used to generate the video preview, but this audio information is not necessarily required. It will not be. For example, by inserting a scene only for Visual, it becomes possible to give a sense of urgency and a sense of security in accordance with a sound effect (musical sound).

  Note that the musical sound to be superimposed on the main video preview is not limited to sound effects but may be music. For example, when the degree of emphasis is steep in a popular song, it may be a part of the so-called “rust”. By inserting a frame or scene at the time of this “rust”, It is possible to realize sensitive editing.

  The present invention can be applied when there is a request to generate a video preview. By applying the present invention, the video preview is automatically performed regardless of the video attributes such as commercial video and home video. Can be generated.

DESCRIPTION OF SYMBOLS 1 Image | video preview production | generation apparatus 10 Image | video material input part 11 Image | video material storage part 12 Filter part 13 Image | video material storage part for image | video notice generation 14 Head part generation part 15 Main part production | generation part 16 Image | video announcement generation part 17 Image | Unit 120 voice filler detection unit 121 excessive camera work detection unit 122 junk shot detection unit 140 voice enhancement level determination unit 141 face size determination unit 142 face expression determination unit 143 musical sound imparting unit 150 fundamental frequency pattern determination unit 151 voice enhancement level pattern determination unit 152 Musical sound giving part

Claims (5)

A video preview generation device that generates a video preview from video material,
One or more video sections among a video section associated with an audio filler, a video section associated with excessive camera work, and a video section in which no captured video exists, with the video section of the video material as a processing target Filter means for detecting the detected video section from the video material,
The video section of the video material that has not been removed by the filter means is processed, and the video section in which the audio enhancement level is larger than a predetermined threshold, the video section in which the subject face size is larger than the predetermined threshold, A head generation means for detecting any one or more video sections in a video section having a facial expression larger than a predetermined threshold, and generating a head of a video preview using the detected video section;
The video section of the video material that has not been removed by the filter means, and the video section that is not the beginning of the video section, the video section in which the time change amount of the fundamental frequency of the audio is greater than a predetermined threshold, Detecting one or more video segments in a video segment in which the audio enhancement level increases with time and the time change of the increase is greater than a predetermined threshold, and the detected video segment is A main part generating means for generating a main part of a video preview using,
Video preview generation means for generating a video preview by combining the beginning generated by the opening generation means and the main part generated by the main part generation means,
A featured video preview generator. In the video preview generation device according to claim 1,
Either one or both of the head generation means and the main part generation means give the sound to the video in such a manner that one of the tone, tempo, and enhancement degree of the sound is synchronized with the video switching timing. To do the
A featured video preview generator. A video preview generation method executed by a video preview generation device that generates a preview of the video from video material,
One or more video sections among a video section associated with an audio filler, a video section associated with excessive camera work, and a video section in which no captured video exists, with the video section of the video material as a processing target And detecting the detected video section from the video material,
The video section of the video material that has not been removed is processed, and the video section in which the audio enhancement level is larger than a predetermined threshold, the video section in which the subject face size is larger than the predetermined threshold, and the facial expression of the subject Detecting any one or more video segments within a video segment larger than a predetermined threshold, and generating the beginning of the video preview using the detected video segments;
The video section of the video material that has not been removed and the video section that did not become the beginning, the video section in which the temporal change in the fundamental frequency of the audio is greater than a predetermined threshold, and the audio enhancement The degree of time increases with time, and the time change amount of the increase is larger than a predetermined threshold, and any one or more video sections are detected, and the detected video section is used for video. The process of generating the main part of the notice,
A step of generating a video preview by combining the generated head portion and the generated main part,
A featured video preview generation method. In the video notice generation method according to claim 3,
Either or both of the process of generating the opening part and the process of generating the main part synchronize one of the tone, tempo, and enhancement degree of the musical sound with the timing of switching the video. To give a musical sound,
A featured video preview generation method.   A video preview generation program for causing a computer to function as means for constituting the video preview generation device according to claim 1.

Images