JP2016116073A - Video processing method, video processing apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To propose a video processing method capable of generating a summarized video of switching videos according to music in a proper operating mode, a video processing apparatus, and a program.SOLUTION: The video processing method includes: extracting a plurality of unit videos from an input video; generating editing information for switching the unit videos adopted from the extracted unit videos according to input music; and controlling an operation mode with a processor for the extracting processing of extracting the unit videos and for the adopting processing of adopting the unit video.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to a video processing method, a video processing device, and a program.

  In recent years, various types of cameras such as wearable cameras and action cameras have been widely used in sports and the like. Such a camera is often photographed continuously for a long time, and the composition tends to be monotonous. Therefore, the photographed image may be unbearable. Therefore, it is desired to develop a technique for generating a summary video that summarizes the highlights of the captured video.

  With respect to such a technique, for example, as disclosed in Patent Documents 1 and 2 below, a technique has been developed that has a plurality of operation modes for generating a summary video and generates a summary video in an appropriate operation mode. ing. Specifically, Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique that enables creation of a plurality of summary videos having different lengths.

JP 2002-271739 A JP 2005-11392 A

  However, the technique disclosed in Patent Document 1 enables generation of a plurality of summary videos having different lengths, and does not form diversity regarding the contents of the summary video. Therefore, the present disclosure proposes a new and improved video processing method, video processing apparatus, and program capable of generating a summary video that switches videos according to music in an appropriate operation mode.

  According to the present disclosure, a plurality of unit videos are extracted from an input video, and edit information for switching the unit video adopted from the extracted unit videos according to input music is generated. And a processor that controls an operation mode in an extraction process for extracting the unit video and an adopting process that employs the unit video.

  According to the present disclosure, the extraction unit that extracts a plurality of unit videos from the input video and the unit video adopted from the unit videos extracted by the extraction unit are switched according to the input music. There is provided a video processing apparatus including an editing unit that generates editing information for the above, and an operation mode control unit that controls an operation mode in the extraction unit and the editing unit.

  In addition, according to the present disclosure, the computer extracts, from the input video, a plurality of unit videos, and the unit video adopted from the unit videos extracted by the extraction unit into the input music. There is provided a program for functioning as an editing unit that generates editing information to be switched according to the operation mode, and an operation mode control unit that controls an operation mode in the extraction unit and the editing unit.

  As described above, according to the present disclosure, it is possible to generate a summary video that switches video according to music in an appropriate operation mode. Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is a figure for demonstrating the outline | summary of the video processing apparatus which concerns on this embodiment. It is a figure for demonstrating the outline | summary of the video analysis process performed in the video processing apparatus which concerns on this embodiment. It is a figure for demonstrating the outline | summary of the edit information generation process and summary video generation process which are performed in the video processing apparatus which concerns on this embodiment. It is a block diagram which shows an example of the logical structure of the video processing apparatus which concerns on this embodiment. It is a figure for demonstrating the extraction process of the unit image | video which concerns on this embodiment. It is a figure for demonstrating the setting process of the switching timing of the unit video which concerns on this embodiment. It is a figure for demonstrating an example of the operation mode of the video processing apparatus which concerns on this embodiment. It is a figure for demonstrating the selection process of the unit image | video which concerns on this embodiment. It is a figure for demonstrating the selection process of the unit image | video which concerns on this embodiment. It is a figure for demonstrating the setting process of the employment area which concerns on this embodiment. It is a figure for demonstrating the setting process of the employment area which concerns on this embodiment. It is a figure for demonstrating the setting process of the employment area which concerns on this embodiment. It is a flowchart which shows an example of the flow of the production | generation process of the summary image | video performed in the video processing apparatus which concerns on this embodiment. It is a block diagram which shows an example of the hardware constitutions of the information processing apparatus which concerns on this embodiment.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In the present specification and drawings, elements having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numerals. For example, a plurality of elements having substantially the same functional configuration are distinguished as necessary, such as the video processing devices 100A, 100B, and 100C. However, when there is no need to particularly distinguish each of a plurality of elements having substantially the same functional configuration, only the same reference numerals are given. For example, the video processing devices 100A, 100B, and 100C are simply referred to as the video processing device 100 when it is not necessary to distinguish them.

The description will be made in the following order.
1. Overview 2. Basic configuration Functional details 3.1. Unit video extraction process 3.2. Switching timing setting process 3.3. Operation mode determination processing 3.4. Unit video selection process 3.5. 3. Adopting section setting process Operation processing 5. Hardware configuration example Summary

<1. Overview>
First, an overview of a video processing apparatus according to the present embodiment will be described with reference to FIGS.

  FIG. 1 is a diagram for explaining an overview of a video processing apparatus 100 according to the present embodiment. FIG. 1 shows a user's operation using the video processing device 100 and a transition of processing performed in the video processing device 100, and time flows from left to right. As shown in FIG. 1, the video processing apparatus 100 generates a summary video 50 from a video 10 taken by a user. The summary video 50 is a digest version video that summarizes video shot by the user. The video processing apparatus 100 generates the summary video 50 by switching and connecting sections adopted from the captured video 10 using any adoption standard according to the input music 30. In the present specification, the video includes image (still image / moving image) data and audio data. Hereinafter, an outline of the generation process of the summary video 50 executed in the video processing apparatus 100 will be described.

  First, during a period in which shooting is performed by the user, the video processing apparatus 100 performs recording processing for recording the shot video 10 and video analysis processing for analyzing the video 10. For example, as the video analysis processing, the video processing device 100 analyzes user operations during shooting, performs image analysis such as smile detection, color detection, and motion vector detection, or subjects based on sensor information during shooting. Analyzing the operation of.

  Next, the video processing apparatus 100 performs editing information generation processing based on the video analysis result information 20 indicating the result of the video analysis processing and the input music 30. For example, the video processing apparatus 100 selects a unit video to be adopted as the summary video 50 from the video 10 by evaluating the video analysis result information 20 using an arbitrary adoption standard. The unit video is a series of video and is also called a shot. Then, the video processing apparatus 100 generates editing information 40 for switching the adopted unit video according to the music 30. The editing information 40 is information that defines which section of which music 30 is BGM (background music) and which unit video is switched at which timing. The video processing apparatus 100 analyzes the music 30 based on the music theory, and generates editing information 40 so that the unit video is switched at a timing according to the melody, rhythm, beat, or excitement of the music 30.

  Then, the video processing apparatus 100 performs a summary video generation process based on the editing information 40. For example, the video processing apparatus 100 generates the summary video 50 by switching and connecting the unit video specified by the editing information 40 at the specified timing with the music 30 specified by the editing information 40 as BGM. The video processing apparatus 100 can also reproduce, record, and transmit the summary video 50 to other devices.

  Note that the video analysis processing illustrated in FIG. 1 may be performed in parallel with shooting by the user or after shooting. In addition, the video analysis process, the editing information generation process, and the summary video generation process may be performed continuously or discontinuously. The video processing apparatus 100 may generate a summary video 50 in which a plurality of videos and music 30 are BGM.

  The outline of the generation process of the summary video 50 has been described above. Next, the generation process of the summary video 50 will be described in more detail with reference to FIGS.

  FIG. 2 is a diagram for explaining an overview of video analysis processing executed in the video processing device 100 according to the present embodiment. In the diagram shown in FIG. 2, the video 10 is a video of the user's day, and the video analysis result information 20 includes information indicating video attributes such as a highlight 21 and a scene segment 22. The video 10 includes a video arriving at the sea, a video surfing, a video during a break, a video at lunch, a video at a hotel, and a video at sunset. The highlight 21 is a section showing the highlight in the video 10. Highlights include, for example, specific scenes such as jumps and turns, smiles, lively scenes of cheering events, and important scenes in specific events such as wedding cake cuts and ring exchanges. The scene segment 22 is a section in which the video 10 is divided under a predetermined condition. For example, the scene segment 22 may be a section in which colors of the same system are divided based on colors. The scene segment 22 may be a section in which the same camera work is divided based on the camera work. In addition, the scene segment 22 may be a section that is divided on the basis of the date and time and is shot at a close date and time. In addition, the scene segment 22 may be a section that is divided based on a place and is shot at the same or near place. As an example, FIG. 2 shows an example in which the scene segment 22 is divided based on colors. The colors to be segmented may be, for example, white, blue, green, and red. The video processing apparatus 100 analyzes video attributes such as the highlight 21 and the scene segment 22 by video analysis processing.

  FIG. 3 is a diagram for explaining an overview of the editing information generation process and the summary video generation process executed in the video processing apparatus 100 according to the present embodiment. First, the video processing apparatus 100 extracts a series of videos having the same scene segment 22 as a unit video. Then, the video processing apparatus 100 adopts the unit video according to a predetermined policy while preferentially adopting the highlight 21 from the unit videos. For example, the video processing apparatus 100 may employ a unit video in which the scene segments 22 are dispersed in order to reduce visual bias. In addition, the video processing apparatus 100 may adopt unit videos in accordance with themes such as surfing and snowboard specified by the user. Specifically, in the case of surfing, the image processing apparatus 100 is configured so that the ratio of highlights such as turns during surfing is higher than that of meals, and the scenes of blue systems, places close to the sea, and time zones with high waves are high. Unit video may be adopted so that the proportion of segments increases. Further, the video processing apparatus 100 analyzes the music 30 (BGM) based on the music theory, and sets the timing for switching the unit video. Through these processes, the video processing apparatus 100 generates editing information 40 for switching the adopted unit video at the set timing. Then, the video processing apparatus 100 generates a summary video 50 based on the editing information 40. Note that the unit video included in the summary video 50 may or may not be along the time series.

  For example, the video processing apparatus 100 can be realized as a camera such as an action camera or a wearable camera. Cameras such as action cameras and wearable cameras are often taken continuously for a long time, and the composition tends to be monotonous. For this reason, it is desirable to edit the video shot by such a camera into a summary video that summarizes the highlights. However, since such a camera is often small or has a simple UI, it may be difficult to manually edit while checking the video. Therefore, it is desirable that an appropriate summary video is generated even if the composition is a monotonous video taken continuously for a long time. In this regard, the video processing apparatus 100 according to the present embodiment is a summary in which, even for such video, attributes are distributed and a shot including a highlight is switched according to BGM along a theme specified by the user. It is possible to generate a video. Note that the video processing apparatus 100 may be realized as a general video camera or the like, or may be realized as an information processing apparatus such as a PC (Personal Computer) separate from the camera or a server on a network.

  The overview of the video processing apparatus 100 according to the present embodiment has been described above. Next, a basic configuration example of the video processing apparatus 100 according to the present embodiment will be described with reference to FIG.

<2. Basic configuration>
FIG. 4 is a block diagram illustrating an example of a logical configuration of the video processing apparatus 100 according to the present embodiment. As shown in FIG. 4, the video processing apparatus 100 includes an input unit 110, a storage unit 120, an output unit 130, and a control unit 140.

(1) Input unit 110
The input unit 110 has a function of accepting input of various information from the outside. As shown in FIG. 4, the input unit 110 includes a sensor unit 111, an operation unit 112, a video acquisition unit 113, and a music acquisition unit 114.

(1.1) Sensor unit 111
The sensor unit 111 has a function of detecting the motion of the subject. For example, the sensor unit 111 may include a gyro sensor, an acceleration sensor, and a gravity sensor. A subject is a subject to be photographed and includes a photographer (user). The sensor unit 111 may include an arbitrary sensor such as a GPS (Global Positioning System), an infrared sensor, a proximity sensor, or a touch sensor. The sensor unit 111 outputs sensor information indicating the sensing result to the control unit 140. The sensor unit 111 may not be formed integrally with the video processing apparatus 100. For example, the sensor unit 111 may acquire sensor information from a sensor attached to the subject via wired or wireless communication.

(1.2) Operation unit 112
The operation unit 112 has a function of accepting user operations. For example, the operation unit 112 is realized by a button, a touch pad, and the like. The operation unit 112 can accept operations such as a zoom operation during shooting and a shooting mode setting operation. As the shooting mode, for example, a normal mode for shooting a moving image and a simultaneous shooting mode for simultaneously shooting a moving image and a still image can be considered. In addition, the operation unit 112 can accept an editing instruction for designating a section to be included in the summary video during or after shooting. The operation unit 112 outputs operation information indicating the content of the user operation to the control unit 140.

(1.3) Video acquisition unit 113
The video acquisition unit 113 has a function of acquiring video. For example, the video acquisition unit 113 is realized as an imaging device and outputs data of a captured image (moving image / still image) that is a digital signal. The video acquisition unit 113 may further include a microphone that collects ambient sound and acquires sound data converted into a digital signal via an amplifier and an ADC (Analog Digital Converter). In that case, the video acquisition unit 113 outputs video data accompanied by surrounding sounds.

(1.4) Music acquisition unit 114
The music acquisition unit 114 has a function of acquiring music data to be BGM of the summary video. For example, the music acquisition unit 114 is realized as a wired or wireless interface, and acquires music data from another device such as a PC or a server. As the wired interface, for example, a connector compliant with a standard such as USB (Universal Serial Bus) can be cited. As a wireless interface, for example, a communication device compliant with a communication standard such as Bluetooth (registered trademark) or Wi-Fi (registered trademark) can be cited. The music acquisition unit 114 outputs the acquired music data to the control unit 140.

(2) Storage unit 120
The storage unit 120 has a function of storing various information. For example, the storage unit 120 stores information output from the input unit 110 and information generated by the control unit 140.

(3) Output unit 130
The output unit 130 has a function of outputting various information. For example, the output unit 130 may have a function of playing back a summary video generated by a summary video generation unit 146 described later. In that case, the output unit 130 may include a display unit and a speaker. In addition, the output unit 130 may have a function of outputting editing information generated by the editing unit 144 described later. In that case, the output unit 130 may include a wired or wireless interface.

(4) Control unit 140
The control unit 140 functions as an arithmetic processing device and a control device, and controls the overall operation in the video processing device 100 according to various programs. As shown in FIG. 4, the control unit 140 includes a music analysis unit 141, a video analysis unit 142, an extraction unit 143, an editing unit 144, an operation mode control unit 145, and a summary video generation unit 146.

(4.1) Music analysis unit 141
The music analysis unit 141 has a function of analyzing the content of the input music. Specifically, the music analysis unit 141 performs analysis based on music theory for the music data acquired by the music acquisition unit 114.

  The music analysis unit 141 may analyze the music structure. For example, the music analysis unit 141 identifies a portion that satisfies a predetermined condition by analyzing the structure of music. For example, the music analysis unit 141 is based on a music theory, and includes an intro part, a melody (Verse) part, a chorus (also called chorus) part, an interlude part, and a solo part. ) Part, ending (Outro) part, etc. can be specified. The melody portion may be divided into A melody A and B melody B. Furthermore, the music analysis unit 141 may detect chord progression in each component of the identified music, and may identify a particularly important portion (section) in the chorus portion based on the detected chord signal. . In addition, the music analysis unit 141 may specify a section in which the vocal singing starts, a section having the highest vocal pitch, and the like as particularly important parts in the chorus part.

  Further, the music analysis unit 141 may analyze the rhythm of music. For example, the music analysis unit 141 analyzes music beats or measures bars. For example, in the case of four time signatures, four beats are included at equal intervals in one measure, and the first beat of these measures coincides with the beginning of the measure. The beat that coincides with the beginning of the measure is also referred to as the beat at the beginning of the measure.

  The music analysis unit 141 outputs music analysis result information indicating the analysis result to the editing unit 144. The music analysis result information includes, for example, information indicating the position of each component in the music data, particularly the position of an important part, the position of each beat, and the position of each bar.

(4.2) Video analysis unit 142
The video analysis unit 142 has a function of analyzing the content of the input video. Specifically, the video analysis unit 142 analyzes the content for the video data acquired by the video acquisition unit 113. Then, the video analysis unit 142 outputs video analysis result information indicating the analysis result of the video content to the extraction unit 143.

Highlight Detection For example, the video analysis unit 142 detects a highlight based on information input from the input unit 110, and outputs information indicating the detected highlight in the video analysis result information. As an example, an example will be described in which the video analysis unit 142 detects highlights related to subject motion, user operation, and face and smile.

  For example, the video analysis unit 142 detects a predetermined motion of the subject based on the sensor information acquired by the sensor unit 111. For example, the video analysis unit 142 can detect the movement of the subject such as jumping of the subject (jump), change of traveling direction (turn), running, acceleration, or deceleration based on the sensor information. The video analysis unit 142 may detect a predetermined motion of the subject by performing an image recognition process on the video data acquired by the video acquisition unit 113. Regarding subject motion detection processing, the video analysis result information may include information indicating the detected motion of the subject and information indicating a section in which the motion is detected in the video data.

  For example, the video analysis unit 142 detects a user operation based on the operation information acquired by the operation unit 112. For example, the video analysis unit 142 detects a predetermined operation such as a zoom operation or a shooting mode setting operation based on operation information acquired during shooting. Regarding the user operation detection process, the video analysis result information may include information indicating the detected user operation and information indicating a section in which the user operation is detected in the video data. In addition, the video analysis unit 142 detects an editing instruction based on operation information acquired during or after shooting. In this case, the video analysis result information may include information indicating a section designated as a section to be included in the summary video by the user.

  For example, the video analysis unit 142 performs image recognition processing on the video data acquired by the video acquisition unit 113 to detect the face and smile of the subject. Regarding the face and smile detection process, the video analysis result information may include information indicating a section and a region where the face and smile are detected in the video data, and the number of faces and smiles.

  For example, the video analysis unit 142 performs voice recognition processing on the video data acquired by the video acquisition unit 113 to detect a section where cheers are struck. Regarding the cheering detection process, the video analysis result information may include information indicating the section in which the cheer is detected in the video data and the volume.

  For example, the video analysis unit 142 detects an important scene in a specific event by performing an image recognition process on the video data acquired by the video acquisition unit 113. Important scenes include wedding cake cuts and ring exchange. Regarding the important scene detection processing, the video analysis result information may include information indicating the section in which the important scene is detected in the video data and the importance.

-Detection of information for a scene segment For example, the video analysis unit 142 detects information for a scene segment based on the information input by the input unit 110, and the information for the detected scene segment is a video analysis result. Output in the information. As an example, an example will be described in which the video analysis unit 142 detects information for a scene segment related to color, camera work, date / time, and location.

  For example, the video analysis unit 142 can detect the color of the video by performing image recognition processing on the video data acquired by the video acquisition unit 113. Specifically, the video analysis unit 142 analyzes YUV or RGB of the video and detects a color histogram for each frame or a plurality of frames. Then, the video analysis unit 142 detects a dominant color in each frame as the color of the frame. The identification information for identifying the detected color is also referred to as a color ID. Regarding the color detection process, the video analysis result information may include information indicating the color ID of each section.

  For example, the video analysis unit 142 can detect camerawork by performing image recognition processing on the video data acquired by the video acquisition unit 113. For example, the video analysis unit 142 detects camera work such as still, up and down, left and right by detecting a motion vector for each frame or for each of a plurality of frames. The identification information for identifying the detected camera work is also referred to as a camera work ID. Regarding the camera work detection process, the video analysis result information may include information indicating the camera work ID of each section.

  For example, the video analysis unit 142 can detect the shooting date and time acquired by a GPS included in the sensor unit 111 or a clock built in a camera or the like included in the video acquisition unit 113. Note that the identification information for identifying the detected shooting date / time is also referred to as a shooting date / time ID. It is assumed that the same shooting date / time ID is assigned to sections shot at the same or close date / time. Regarding the shooting date / time detection process, the video analysis result information may include information indicating the shooting date / time ID and the section of each shooting date / time segment.

  For example, the video analysis unit 142 can detect the location where the image was taken based on position information acquired by the GPS included in the sensor unit 111. The identification information for identifying the detected shooting location is also referred to as a shooting location ID. It is assumed that the same shooting location ID is attached to sections shot at the same or close locations. Regarding the shooting location detection process, the video analysis result information may include information indicating the shooting location ID of each section.

(4.3) Extraction unit 143
The extraction unit 143 has a function of extracting a plurality of unit videos from the input video. Specifically, the extraction unit 143 extracts a plurality of unit videos from the video data acquired by the video acquisition unit 113 based on the analysis result by the video analysis unit 142. Specifically, the extraction unit 143 extracts a series of videos having the same video attributes indicated by the analysis result information as unit videos.

  For example, the extraction unit 143 may extract a series of videos having the same scene segment as a unit video. Further, the extraction unit 143 may extract a video in which highlight is detected as a unit video. Specifically, the extraction unit 143 may extract a section in which a predetermined operation such as a subject jump is detected as one unit video. In addition, the extraction unit 143 extracts a section in which a predetermined operation such as a zoom operation or a shooting mode setting operation is detected, or a section designated as a section to be included in the summary video by the user as one unit video. May be. At that time, the extraction unit 143 may extract a section after zooming as a unit video if the operation is a zoom operation, or extract a section shot in the simultaneous shooting mode as a unit video if the operation is a shooting mode setting operation. Also good. In addition, the extraction unit 143 is a section in which the face or smile of the subject is detected, that is, a section in which the subject is detected to be in a predetermined state such as smiling or facing the camera, or before and after the section. May be extracted as one unit video. In addition, the extraction unit 143 may extract a section where cheers are boiling as one unit video. The extraction unit 143 may extract a section in which an important scene in a specific event is captured as one unit video. The extraction unit 143 may use a combination of these extraction criteria.

  The extraction unit 143 may set the degree of attention for the extracted unit video based on the analysis result by the video analysis unit 142. For example, the extraction unit 143 sets a high degree of attention to the unit video in the section corresponding to the highlight. Specifically, when the video analysis unit 142 analyzes that the motion of the subject in the shooting section of the unit video is a predetermined motion, the extraction unit 143 analyzes that the subject is in a predetermined state or When it is analyzed that a predetermined operation has been performed, a high degree of attention is set for the unit video. In addition, the extraction unit 143 sets a high degree of attention to the unit video when it is analyzed by the video analysis unit 142 that a cheer is generated in the shooting period of the unit video or when it is analyzed that the scene is an important scene. To do. As a result, a high degree of attention is set for the unit video corresponding to the section in which the predetermined motion such as the jump of the subject is detected. In addition, a high degree of attention is set for a unit video corresponding to a section in which the subject is detected to be in a predetermined state such as a smiling face or a face facing the camera. In addition, a high degree of attention is set for a unit video corresponding to a section in which a predetermined operation such as a zoom operation or a shooting mode setting operation is detected. In addition, a high degree of attention is set for the unit video corresponding to the section where the cheers are struck. Further, a high degree of attention is set for a unit video corresponding to a section in which an important scene in a specific event such as a wedding cake cut or ring exchange is detected. In addition, the extraction unit 143 may set a high degree of attention to a unit video corresponding to a section designated as a section to be included in the summary video by the user. Then, the extraction unit 143 sets a low degree of attention in cases other than those described above. Hereinafter, a unit video with a high degree of attention is also referred to as a highlight shot. A unit video with a low level of attention is also referred to as a sub-shot. The identification information for identifying the type of the extracted highlight shot is also referred to as a highlight ID. For example, as the highlight ID, a different ID can be set according to the type of highlight such as jump, zoom operation, cheer, important scene, or user-designated highlight.

(4.4) Editing unit 144
The editing unit 144 has a function of generating editing information for switching the unit video extracted by the extracting unit 143 according to the input music. For example, the editing unit 144 sets which section of which music is input as BGM. Then, the editing unit 144 divides the music to be BGM by the music analysis result by the music analysis unit 141, and assigns the unit video extracted by the extraction unit 143 to each section. As a result, in the summary video, the unit video is switched at the timing when the music is divided. When allocating unit videos, the editing unit 144 may determine all or part of the unit videos extracted by the extraction unit 143 as unit videos to be adopted as summary videos, and assign the adopted unit videos to each section. Note that the editing unit 144 assigns unit videos in the order of shooting times in principle. Of course, the editing unit 144 may assign a unit video without depending on the shooting time. In this way, the editing unit 144 generates editing information by setting which unit video is to be switched at which timing with which section of which music is input as BGM. Details of the processing by the editing unit 144 will be described in detail later.

(4.5) Operation mode control unit 145
The operation mode control unit 145 has a function of controlling operation modes in the extraction unit 143 and the editing unit 144. The operation mode control unit 145 controls the operation mode according to the unit video extraction result by the extraction unit 143 and the switching timing setting result by the editing unit 144. Details of processing by the operation mode control unit 145 will be described in detail later.

(4.6) Summary video generation unit 146
The summary video generation unit 146 has a function of generating a summary video composed of music and unit videos that are switched based on editing information. For example, the summary video generation unit 146 generates a summary video by switching and connecting the unit video specified by the editing information at the specified timing with the music specified by the editing information as BGM.

<3. Function details>
The basic configuration of the video processing apparatus 100 according to the present embodiment has been described above. Next, functions of the video processing apparatus 100 will be described in detail.

[3.1. Unit video extraction processing]
The extraction unit 143 extracts a plurality of unit videos from the video data acquired by the video acquisition unit 113 based on the analysis result by the video analysis unit 142. Specifically, the extraction unit 143 extracts a unit video according to the video attribute analyzed by the video analysis unit 142. For example, the extraction unit 143 extracts highlight shots and sub-shots from the video data based on information for scene segments and information indicating highlights. Hereinafter, the unit video extraction process based on the video analysis result will be described in detail with reference to FIG.

  FIG. 5 is a diagram for explaining unit video extraction processing according to the present embodiment. FIG. 5 schematically shows a process in which the extraction unit 143 extracts highlight shots 260A to 260E and sub-shots 270A to 270G. As shown in FIG. 5, the extraction unit 143 first generates a scene segment 210 based on information for a scene segment. For example, the extraction unit 143 generates a scene segment 210 by segmenting sections having the same color ID. The extraction unit 143 may use a plurality of pieces of information for a scene segment. For example, the extraction unit 143 generates a scene segment 210 by segmenting a section having the same color ID, camera work ID, shooting location ID, and shooting date / time ID. May be. Next, the extraction unit 143 associates the scene segment 210 with the highlight 220 and extracts highlight shots 240 </ b> A to 240 </ b> E from the input video 230. Then, the extraction unit 143 extracts a section divided by the scene segment 210 of the input video 230 as a sub-shot. However, the extraction unit 143 excludes a section that overlaps the highlight shot 240 and has a short time (for example, shorter than the length of the longest allocated section described later), extremely bright or dark, or a section where the camera work is not stable. The sub-shot 250 may be extracted. Hereinafter, the number of unit videos extracted by the extraction unit 143 based on the video result information, that is, the number of highlight shots and sub-shots is also referred to as the number of extractions.

[3.2. Switching timing setting process]
Based on the music analysis result information output from the music analysis unit 141, the editing unit 144 sets the switching timing of the unit video according to the input music. For example, the editing unit 144 generates editing information for switching the unit video extracted by the extracting unit 143 according to the component analyzed by the music analyzing unit 141, according to the measure, or according to the beat. May be. Specifically, the editing unit 144 divides the input music at a timing at which a component is switched, a timing at which a measure is switched, or a timing according to a beat, and sets a unit video switching timing at the divided position.

  For example, as the timing according to the beat, the editing unit 144 may generate editing information for switching the unit video for each beat. In this case, the unit video is switched with a sense of speed at a high tempo, and the viewer's emotion can be raised. However, the editing unit 144 may generate editing information for switching the unit video for every plurality of beats when the beat speed of the music exceeds the threshold value. For example, the unit video may be switched every two beats. Thereby, when the BGM is music with a fast tempo, the unit video is prevented from being switched too early, so that it is possible to avoid giving the viewer an impression that does not frustrate.

  For example, the editing unit 144 may set the number of times of switching of unit video according to the beat for each type of music structure analyzed by the music analysis unit 141. Specifically, the editing unit 144 may set the number of unit video switching operations according to the beat for each musical component such as an intro part and a chorus part. Further, the editing unit 144 may switch the unit video according to the beat at a part that satisfies the predetermined condition specified by the music analysis unit. Specifically, the editing unit 144 may switch the unit video in accordance with the beat in the chorus part, particularly in a part that is particularly important such as a part where the vocal starts and the part where the vocal pitch is highest. Good. Thereby, it is possible to switch the unit video according to the beat in accordance with the excitement of the BGM, and it is possible to excite the viewer's emotions more effectively.

  For example, the editing unit 144 may select whether or not to switch the unit video according to the beat in units of music measures analyzed by the music analysis unit 141. In this case, the unit video is switched according to the beat in units of measures. A person can be consciously or unconsciously listening to music with consciousness of measures and predicting development. Therefore, the switching of the unit video corresponding to the beat in the unit of measure is easily accepted by the viewer, so that the emotion of the viewer can be easily raised. Furthermore, the switching of the unit video according to the beat in the unit of measure is consistent with the switching of the unit video in the unit of measure. In addition, the editing unit 144 may separate bars for switching the unit video according to the beat. As a result, unit video switching according to beats is not performed in a plurality of consecutive bars, and excessive switching is prevented.

  A section in which music is divided at the set switching timing is also referred to as an allocation section below. In other words, setting the switching timing is equivalent to setting an allocation interval of how long each unit video is allocated to the summary video. The longest section among the allocation sections is also referred to as the longest allocation section below.

  The setting of the unit video switching timing described above may be set based on a preset probability table, for example. At that time, the editing unit 144 may follow rules such as switching unit videos and setting the length of the longest allocated section at the timing of switching of music components.

  In addition, it is desirable that the unit videos that are switched according to the beat within one measure are similar to each other. Thereby, it is possible to avoid giving a complicated impression to the viewer. “Similar to each other” means, for example, that at least one of the motion of the subject, the shooting date / time, the shooting location, the color, or the camera work is close. For example, it can be said that the unit video in which the color is the same and the camera work moves from right to left and the unit video from left to right are similar to each other. It can also be said that the unit videos in which the subject is jumping are similar to each other. Moreover, being similar to each other may indicate that a specific subject is included in the unit video, for example. For example, it can be said that unit videos including the same person or the same team are similar. Here, at least one of the unit images switched according to the beat in one measure may be adopted twice or more in one measure. For example, in the case of 4 beats, the unit video A, the unit video B, the unit video A, and the unit video B may be used in this order, or the unit video A, the unit video A, the unit video A, and the unit video A may be used in this order. May be. This makes it easier to avoid giving the viewer a complicated impression. Of course, the unit video that switches according to the beat in one measure may be different. For example, in the case of 4 beats, the unit video A, the unit video B, the unit video C, and the unit video D may be employed in this order.

  Hereinafter, with reference to FIG. 6, the setting process of the switching timing of the unit video based on the music analysis result will be specifically described.

  FIG. 6 is a diagram for explaining unit video switching timing setting processing according to the present embodiment. In FIG. 6, the component 320 of the area 310 used as BGM among music, and the switching timing 330 set are shown. The dividing line of the switching timing 330 shows the switching timing, and the section divided by the dividing line shows the allocation section. As shown in FIG. 6, the component 320 includes a melody portion, a chorus portion, and an ending portion. The music shown in FIG. 6 is quadruple music in which one bar 343 includes one bar 342 and three beats 341. In the example illustrated in FIG. 6, the editing unit 144 sets the unit video switching timing at the timing when the component 320 switches from the melody to the chorus, and at the timing when the component 320 switches from the chorus to the ending. Further, the editing unit 144 sets allocation sections 351A to 351D in units of one bar, sets allocation sections 352 in units of two bars, sets allocation sections 353 in units of three bars, and sets allocation sections 354 in units of one beat. It is set. Therefore, in the section 354, the unit video is switched for each beat. In this case, the longest allocation section 360 is 3 bars.

  Table 1 below shows the number of unit videos employed in the entire BGM and in each component in the example shown in FIG. 6 for each switching timing type (allocation section length).

  Note that when the unit video is switched for each beat, one unit video may be adopted a plurality of times, and therefore, the maximum number of unit videos to be selected is four. Referring to Table 1, in the example shown in FIG. 6, a total of up to 10 unit videos are adopted as summary videos. Further, in the example shown in FIG. 6, the longest allocation section is for three bars.

  As described above, the number of unit videos employed in the summary video is determined by the number of allocation sections determined by the switching timing set by the editing unit 144 based on the music analysis result information, that is, the number of music segments. In the following, the number of pieces of music divided by the editing unit 144 based on the music analysis result information is also referred to as an adopted number. For example, in the example shown in FIG. More specifically, if the content of switching according to the beat is unit video A, unit video B, unit video C, and unit video D, the number of adoption is ten. If the content of switching according to the beat is unit video A, unit video B, unit video A, and unit video B, the number of adoption is eight.

  The editing unit 144 may switch the unit video extracted by the extraction unit 143 at the switching timing set in the switching timing setting process. In addition, the editing unit 144 may change the switching timing set in the switching timing setting process. For example, the editing unit 144 may change the order of the allocation sections while maintaining the total number of allocation sections (corresponding to the number employed) and the number of allocation sections for each length set in the switching timing setting process. Such an example will be described later in the adopted section setting process.

[3.3. Operation mode decision processing]
The order of the switching timing setting process and the unit video extraction process described above is arbitrary.

  When the switching timing setting process is first, the unit video extraction process is restricted by the switching timing setting process. For example, the extraction unit 143 may be restricted to extract at least the number of adopted unit videos. Due to this restriction, unit videos are switched without overlapping in the summary video. Further, the extraction unit 143 extracts a unit video having a length equal to or longer than the longest allocation section (in the example illustrated in FIG. 6, three bars) so that each extracted unit video may be used at any timing. May be imposed. According to this restriction, any extracted unit video can be allocated to the longest allocation section.

  When the unit video extraction process is first, the switching timing setting process is limited by the unit video extraction process. For example, the editing unit 144 is restricted to set the switching timing so as to allocate a smaller number of unit videos than the number of unit videos extracted by the extraction unit 143. Due to this restriction, unit videos are switched without overlapping in the summary video. In addition, a limitation may be imposed that the editing unit 144 sets the switching timing so as to be an allocation section having a length corresponding to the length of each unit video extracted by the extraction unit 143. According to this restriction, a suitable allocation section can be assigned to each unit video extracted by the extraction unit 143.

  The operation mode control unit 145 can change the operation modes of the extraction unit 143 and the editing unit 144 in order to satisfy such a restriction. Hereinafter, a case where the switching timing setting process is performed first will be described.

  First, the operation mode control unit 145 operates the extraction unit 143 and the editing unit 144 with the operation mode as the normal processing mode (first operation mode). In the normal processing mode, the editing unit 144 sets the unit video switching timing using the music analysis result information as described above. Further, the extraction unit 143 extracts the unit video using the video analysis result information as described above.

  The operation mode control unit 145 selects at least one of the re-extraction process by the extraction unit 143 and the re-adoption process by the editing unit 144 according to the magnitude relationship between the number of extractions and the number of adoptions in the normal processing mode. It is determined whether or not to implement the change. The extraction processing here refers to the above-described unit video extraction processing. In addition, the adoption process here refers to the switching timing setting process described above. Regarding the magnitude relationship between the number of extractions and the number of adoptions, there is a limitation that the number of extractions is equal to or more than the number of adoptions as described above. The operation mode control unit 145 can satisfy the restriction by changing the operation mode when the restriction is not satisfied.

  For example, the operation mode control unit 145 determines not to change the operation mode when the employed number and the extracted number in the normal processing mode are equal or the extracted number is larger. That is, the operation mode control unit 145 determines not to change the operation mode when the number of extractions is equal to or greater than the number of adoptions. This is because the restriction that the number of extractions described above is equal to or greater than the number adopted is satisfied without changing the operation mode.

  On the other hand, the operation mode control unit 145 can change the operation mode to another operation mode when the number of extractions in the normal processing mode is smaller than the number of adoption. For example, the operation mode control unit 145 can change the operation mode to the division processing mode (second operation mode) or the retry processing mode (fifth operation mode).

  In the division processing mode, the extraction unit 143 divides at least one of the unit videos extracted in the normal processing mode into two or more unit videos. For example, the extraction unit 143 may set a unit video whose length exceeds a threshold among the unit videos extracted in the normal processing mode as a division target. In addition, the extraction unit 143 may determine the number of divisions so that the divided unit video is equal to or longer than the longest allocation section. Since the number of extractions increases according to the division processing mode, the restriction that the number of extractions is equal to or more than the number of adoptions can be satisfied.

  In the retry processing mode, the editing unit 144 sets the switching timing by dividing music at a predetermined interval. Further, the extraction unit 143 extracts a unit video obtained by dividing the video at a predetermined interval. For example, the editing unit 144 divides the input music at equal intervals or at preset intervals, and sets the timing of the division as the switching timing. In addition, the extraction unit 143 extracts the divided video as a unit video by dividing the input video at regular intervals or at preset intervals. That is, the extraction unit 143 extracts the unit video without considering the highlight. In the retry processing mode, the number of extractions and the number of extractions can be arbitrarily adjusted by adjusting the separation interval. Therefore, the restriction that the number of extractions is equal to or more than the number of adoptions can be satisfied.

  The operation modes described above will be compared and described with reference to FIG. FIG. 7 is a diagram for explaining an example of an operation mode of the video processing apparatus 100 according to the present embodiment. As shown in FIG. 7, in the normal processing mode, the video analysis result information and the music analysis result information are used, and a summary video with a video quality of “high” is generated. In the division processing mode, the video analysis result information is corrected and used. Specifically, as shown in FIG. 7, the unit video 410 extracted in the normal processing mode is divided into unit videos 411 and 412. Similarly, the unit video 420 is divided into unit videos 421, 422, and 423, and the unit video 430 is divided into unit videos 431, 432, and 433. In the division processing mode, a unit video that was originally one can be divided into a plurality of pieces and each can be adopted as a summary video. That is, since similar unit videos can be adopted for the summary video, the video quality of the summary video is “medium”. In the retry processing mode, the video analysis result information and the music analysis result information are ignored. Specifically, as shown in FIG. 7, the switching timing is equally spaced, and the unit video is obtained by dividing the input video at equal intervals. Therefore, the summary video generated in the retry processing mode is monotonous, and the video quality is “low”.

  The operation mode control unit 145 may change the operation mode to another operation mode other than the division processing mode and the retry processing mode when the number of extractions in the normal processing mode is smaller than the number of adoption. For example, the operation mode control unit 145 can change the operation mode to the longest allocation interval shortening processing mode (third operation mode) or the sub-shot condition relaxation processing mode (fourth operation mode).

  In the longest allocation section shortening processing mode, the editing unit 144 shortens the longest allocation section as compared with the normal processing mode. As a result, the extraction unit 143 extracts the unit video with a length equal to or longer than the longest allocation section shorter than the normal processing mode. In the example illustrated in FIG. 6, the extraction unit 143 extracts a unit video with a length of three bars or more in the normal processing mode. On the other hand, in the longest allocation section shortening processing mode, the extraction unit 143 extracts a unit video with a length of, for example, two bars or more. As a result, the extraction unit 143 can sub-extract and extract an image of a section that has only two bars in the normal processing mode and has not been extracted as a sub-shot because it is short. In this way, in the longest allocation section shortening processing mode, the number of extractions increases, so the restriction that the number of extractions is equal to or greater than the number of adoptions can be satisfied.

  In the sub-shot condition relaxation processing mode, the extraction unit 143 relaxes conditions related to the analysis result by the video analysis unit 142 for extracting the unit video as compared with the normal processing mode. For example, the extraction unit 143 extracts a unit video even if the time is short, extracts it as a unit video even if it is extremely bright or dark, or even if the camera work is not stable. Extract as video. In this way, in the sub-shot condition relaxation processing mode, the number of extractions increases, so that the restriction that the number of extractions is equal to or greater than the number of adoptions can be satisfied.

  The order of the operation modes listed above is arbitrary. For example, the operation mode control unit 145 may change the operation mode in the order of the division processing mode, the longest allocated section shortening processing mode, the sub-shot condition relaxation processing mode, and the retry processing mode after the normal processing mode. The operation mode control unit 145 may use any combination of the above-described operation modes. Furthermore, the operation mode control unit 145 may perform processing that employs all or part of the above-described operation modes in parallel, and may select an operation mode that provides the highest quality result.

[3.4. Unit video selection process]
(Overview)
The editing unit 144 selects a unit video to be used for the summary video from the unit videos extracted by the extraction unit 143. For example, the editing unit 144 prioritizes highlighting and selects unit videos for the number of adopted units. Hereinafter, the unit video selection process will be described with reference to FIGS.

  FIG. 8 is a diagram for explaining unit video selection processing according to the present embodiment. As shown in FIG. 8, the editing unit 144 first selects one or more sub-shots 510 as unit video candidates to be used for the summary video. The selected shot 520 is a unit video selected as a unit video candidate to be adopted for the summary video. The editing unit 144 may select the sub-shots 510 so that, for example, the scene segments are distributed and / or follow the theme specified by the user. For example, the editing unit 144 selects the sub-shots 510 in descending order of evaluation values based on an evaluation function described later. [1], [2], [3], [4], [5], [6], and [7] in the figure indicate the selection order using the evaluation function. In addition, it is assumed that the number of adoption is seven. As shown in FIG. 8, in the selected shot 520, the editing unit 144 arranges the selected unit videos along the time at which the selected unit videos were taken.

  FIG. 9 is a diagram for explaining unit video selection processing according to the present embodiment. As illustrated in FIG. 9, the editing unit 144 selects the highlight shot 530 as a candidate for a unit video to be adopted for the summary video. For example, the editing unit 144 can select the highlight shot 530 so that adjacent unit videos in the selected shot do not have the same highlight. For example, the editing unit 144 selects the highlight shots 530 in descending order of evaluation values based on an evaluation function described later. Further, the editing unit 144 removes the sub-shot 540 having a low priority from the already selected sub-shots instead of selecting the highlight shot 530. An example of the sub-shot 540 having a low priority is a sub-shot whose selection order is late. [1] and [2] in the figure indicate a selection order and a removal order using evaluation functions.

(Sub-shot evaluation function)
Hereinafter, an example of an evaluation function used for selecting a sub-shot will be described. For example, the editing unit 144 can select a sub-shot using an evaluation function shown in Equation 1 below.

W _si Si and W _ss Ss in Equation 1 are terms related to the scene segment. The symbol W _si and the symbol W _ss are weights of each term, and can be arbitrarily set by the editing unit 144. The symbol Si is a value (score) related to the segment ID of the scene segment. For example, the symbol Si is calculated based on the color ID, camera work ID, shooting date / time ID and / or location ID used for the scene segment. For example, to follow a preset theme, the score can be calculated to approach the percentage of segment IDs along the preset theme. Also, a score can be calculated such that each segment ID is evenly selected to reduce visual bias. The symbol Ss is a score related to the stability of the scene segment. The symbol Ss is calculated based on the color used for the scene segment and / or the stability of the camerawork (less time change). For example, a higher score may be calculated as the stability is higher. In addition, the editing unit 144 may add a term related to the selection source video file to Equation 1 in order to distribute the selection source video file. Further, the editing unit 144 may add a term related to the time to the selected shots before and after the selection to Equation 1 in order to disperse the variance of the shooting time.

  Each time the editing unit 144 selects one sub-shot, the evaluation function shown in Equation 1 is calculated for each unselected sub-shot, and the sub-shot having the highest evaluation value is selected. Note that the score of each symbol may vary in relation to the already selected sub-shot.

(Highlight Shot Evaluation Function)
In the following, an example of an evaluation function used for selecting a highlight shot will be described. For example, the editing unit 144 can select a highlight shot using an evaluation function shown in Equation 2 below.

W _hi Hi and W _hs Hs in Equation 2 are terms related to highlighting. The symbol W _hi and the symbol W _hs are the weights of each term, and can be arbitrarily set by the editing unit 144. The symbol Hi is a score related to the highlight ID. For example, the symbol Hi is calculated based on the highlight ID. For example, to follow a preset theme, the score can be calculated to approach the percentage of highlight IDs along the preset theme. Also, a score can be calculated such that each highlight ID is evenly selected to reduce visual bias. The symbol Hs is a score related to the value of the highlight shot. For example, if the symbol Hs is a snowboard jump, the higher the dwell time and the greater the amount of rotation, the higher the score can be calculated. Other symbols are the same as those in Equation 1 above.

  Each time the editing unit 144 selects one highlight shot, the evaluation function shown in Equation 2 is calculated for each unselected highlight shot, and the highlight shot with the highest evaluation value is selected. Then, the editing unit 144 removes sub-shots whose selection order is late from the already selected sub-shots. Note that the score of each symbol may vary depending on the relationship with the already selected highlight shot.

  By using the symbol Hi, the editing unit 144 can avoid, for example, continuous highlight shots of jumps. Note that the score related to the symbol Hi may be ignored for the highlight shot related to the section designated as the section to be included in the summary video by the user. In this case, for example, unit videos of jumps designated as highlights by the user can be continuous. In addition, the editing unit 144 can preferentially select a high-value highlight shot by using the symbol Hs.

Note that the editing unit 144 may set the number of highlight shots with the same highlight ID to be less than the preset number. For example, the editing unit 144 may select a highlight shot that satisfies the following mathematical formula. According to the following formula, for example, even if a highlight shot of a jump can be selected up to two times, a jump with a high score of the symbol Hs can be selected three times or more, and the score of the symbol Hs is low. For jumps, the number of selections can be less than two.
Highlight score Hs−attenuation coefficient × number of selections ≧ threshold value (Equation 3)

  Heretofore, an example of unit video selection processing has been described. In the above description, an example in which a sub-shot is first selected and then a highlight shot is selected has been described. However, the present technology is not limited to such an example. For example, the editing unit 144 may first select a highlight shot and then select a sub-shot. In that case, the editing unit 144 first selects a highlight shot, and selects the number of sub-shots obtained by subtracting the number of selected highlight shots from the number of adopted shots. In addition, the editing unit 144 may simultaneously select a highlight shot and a sub-shot. In that case, the editing unit 144 can apply a common evaluation function between the highlight shot and the sub-shot. For the sub-shots, there is no highlight ID and score (symbol Hi and Hs) regarding the value of the highlight shot, and therefore a common evaluation function can be obtained by setting the corresponding term to an arbitrary value (for example, 0). Can be applied.

[3.5. Adopting section setting process]
The editing unit 144 sets an adopted section corresponding to the content of the unit video in the unit video extracted by the extracting unit 143, and generates editing information for adopting the adopted section set for each of the plurality of unit videos. . For example, the editing unit 144 sets an adopted section to be adopted for the summary video according to the content of the unit video, and generates editing information for connecting the set adopted sections. Note that the position of the adopted section is a section adopted for the summary video among the unit videos. The adopted section may be the entire unit video or a part thereof.

  For example, the editing unit 144 may set the position of the adopted section in the unit video according to the content of the unit video. For example, the editing unit 144 adopts the adopted section according to the content of the unit video such as whether the unit video is a highlight shot or a sub-shot, and what attributes such as a highlight ID, a color ID, and a camera work ID are. Can be set. The position of the adopted section indicates the position of the section set as the adopted section among the unit videos in the entire unit video, and includes, for example, the first half, the middle board, or the second half of the unit video. Thereby, for example, a more appropriate section for exciting the viewer's emotion is set according to the content of the unit video and is adopted in the summary video.

  For example, the editing unit 144 may set the position of the adopted section in the unit video in accordance with the motion of the subject of the video analyzed by the video analysis unit 142. For example, a highlight shot related to a snowboard jump is assumed. For the unit video analyzed by the video analysis unit 142 as the subject's motion is a jump, the editing unit 144 is running, from running to running in the air, in the air, from landing in the air, or after landing from the landing. The employed section may be set at any position up to. In that case, the editing unit 144 can set an adopted section focusing on various notable highlights of the jump. As another example, a highlight shot relating to a snowboard turn (change in the direction of movement) is assumed. For the unit video analyzed by the video analysis unit 142 that the movement of the subject is a change of the moving direction, the editing unit 144 is either before the change to the change, during the change, or from the change to the after the change. An adopted section may be set at the position. In that case, the editing unit 144 can set an adopted section focusing on various notable highlights of the turn.

  For example, when the adopted section is set to two or more highlight shots of the same type (the same highlight ID), the editing unit 144 may distribute the positions of the adopted sections in each of the two or more highlight shots. For example, when the selected shot includes highlight shots related to jumps of a plurality of snowboards, the editing unit 144 is running, running from running to staying in the air, staying in the air, from staying in the air to after landing, or from landing to after landing For example, the positions of the adopted sections may be dispersed. Similarly, when the selected shot includes highlight shots related to a plurality of snowboard turns, the editing unit 144 determines the position of the adopted section from before conversion to during conversion, during conversion, or from conversion to after conversion. It may be dispersed. In that case, since the adopted sections are set from different viewpoints even for the same type of highlight shot, the viewer can appreciate the summary video without getting bored.

  For example, the editing unit 144 may generate editing information so as to connect the highlight shot and another type of highlight shot or sub-shot. For example, the editing unit 144 assigns highlight shots with the same highlight ID so as not to be continuous, or assigns sub-shots between them when they are consecutive. As a result, the summary video is inflected, so that the viewer can enjoy the summary video without getting bored.

  For example, the editing unit 144 may set the length of the highlight shot adoption section longer than the length of the sub-shot adoption section. For example, the editing unit 144 preferentially assigns highlight shots to long assignment sections. Accordingly, the viewer can view the highlight shot for a longer time, and thus can more effectively excite the viewer.

  Hereinafter, with reference to FIGS. 10 to 12, the setting process of the adopted section will be specifically described. FIGS. 10-12 is a figure for demonstrating the setting process of the employment area which concerns on this embodiment. In particular, FIG. 10 illustrates an example in which highlight shots are preferentially assigned to long assignment intervals.

  As shown in FIG. 10, the breakdown of the allocation section 710 set in the switching timing setting process is two allocation sections 711 for one bar unit, four allocation sections 712 for two bar units, and allocation for three bar units. Assume that the number of sections 713 is one. For example, the editing unit 144 preferentially allocates highlight shots to long allocation sections according to the rules shown in Table 2 below. Note that the rules shown in Table 2 below may be further subdivided according to the type of highlight, the type of scene segment, and the like.

  As shown in FIG. 10, the breakdown of the selected shot 720 is in the order of sub-shot 721A, highlight shot 722A, sub-shot 721B, highlight shot 722B, sub-shot 721C, sub-shot 721D, and highlight shot 722C. To do. The editing unit 144 generates editing information 730 for setting which unit video is switched at which timing by assigning an allocation section to each unit video as described below.

  First, the editing unit 144 assigns an allocation section 711A in units of one measure having the highest priority among the remaining allocation sections to the sub-shot 721A that is the first selected shot 720. Next, the editing unit 144 assigns the allocation section 713 in units of three measures having the highest priority among the remaining allocation sections to the highlight shot 722A that is the second selected shot 720. Next, the editing unit 144 assigns the allocation section 711B having the highest priority among the remaining allocation sections to the sub-shot 721B that is the third selected shot 720. Next, the editing unit 144 assigns the allocation section 712A in units of two measures having the highest priority among the remaining allocation sections to the highlight shot 722B that is the fourth selected shot 720. Next, the editing unit 144 assigns the allocation section 712B having the highest priority among the remaining allocation sections to the sub-shot 721C that is the fifth selected shot 720. Next, the editing unit 144 assigns the allocation section 712C having the highest priority among the remaining allocation sections to the sub-shot 721D that is the sixth selected shot 720. Finally, the editing unit 144 allocates the remaining 2-bar unit allocation section 712D to the highlight shot 722C that is the seventh selected shot 720.

  Note that the above-described assignment is performed for each component such as a melody. In this case, it is guaranteed that the unit video is switched at the timing when the component is switched, regardless of what assignment is performed in the component.

  Next, an example of setting an adopted section in one unit video will be described with reference to FIGS. 11 and 12. For example, as shown in FIG. 11, the editing unit 144 basically sets an adopted section 750 in the center portion of the unit video 740. On the other hand, as illustrated in FIG. 12, the editing unit 144 may set an adoption section 750 in the first half, the center, or the second half of the unit video 740 for a highlight shot such as a turn. Here, the length of the adopted section 750 set by the editing unit 144 corresponds to the length of the assigned section assigned to each unit video described with reference to FIG.

  The functions of the video processing apparatus 100 according to the present embodiment have been described above in detail. Next, an example of operation processing of the video processing apparatus 100 according to the present embodiment will be described with reference to FIG.

<4. Operation processing>
FIG. 13 is a flowchart illustrating an example of a summary video generation process executed in the video processing apparatus 100 according to the present embodiment.

  As shown in FIG. 13, first, in step S102, the music analysis unit 141 analyzes the input music. For example, the music analysis unit 141 analyzes the structure of music such as an intro part and a chorus part based on music theory, specifies a particularly important part in the chorus part, and analyzes beats and measures. .

  Next, in step S104, the video analysis unit 142 analyzes the input video. For example, the video analysis unit 142 detects a subject motion, detects a user operation, detects a face and a smile, detects a color, and detects camera work.

  Next, in step S106, the editing unit 144 sets unit video switching timing. For example, the editing unit 144 sets the switching timing for each beat, for each bar, or for each bar based on the music analysis result in step S102. At that time, the editing unit 144 can set so that switching according to the beat is performed in a particularly important portion of the chorus portion. By this step, the longest allocation section length is determined.

  Next, in step S108, the editing unit 144 calculates the number of unit videos (adopted number) adopted for the summary video. For example, the editing unit 144 calculates the number of adoption based on the number of allocation sections determined by the switching timing set in step S106. Specifically, the editing unit 144 calculates the number of adoption by subtracting the number of assigned sections from the number of assigned sections if the number of assigned sections is the same as the number of adopted sections when there is no overlap in the unit video.

  Next, in step S110, the extraction unit 143 extracts a unit video. For example, the extraction unit 143 extracts highlight shots and sub-shots based on the video analysis result in step S104. At that time, the extraction unit 143 extracts a unit video with a length equal to or longer than the longest allocation section among the allocation sections determined by the switching timing set in step S106. Further, the extraction unit 143 calculates the total number of extracted highlight shots and sub-shots as the number of extractions.

  Next, in step S112, the operation mode control unit 145 determines whether the number of extractions is equal to or greater than the number of adoptions.

  If it is determined that the number of extractions is not equal to or greater than the number of adoptions (S112 / NO), in step S114, the operation mode control unit 145 changes the operation mode. For example, the operation mode control unit 145 changes to the division processing mode when the change before the normal operation mode. Then, the process returns to step S106. As described above, the operation mode control unit 145 changes the operation mode and returns the process to step S106 until the number of extractions is equal to or more than the number of adoption. Note that in any operation mode, when the number of extractions does not exceed the number of adoptions, the video processing apparatus 100 may output an error and stop the processing.

  If it is determined that the extracted number is equal to or larger than the adopted number (S112 / YES), in step S116, the editing unit 144 selects a unit video to be adopted for the summary video. For example, the editing unit 144 adopts a unit video in which attributes are dispersed in order to reduce the visual bias from the unit videos extracted by the extraction unit 143, or follows the theme specified by the user. Select unit video. Note that the editing unit 144 may preferentially adopt the highlight shot as compared to the sub-shot.

  Next, in step S118, the editing unit 144 sets an adopted section for each unit video. For example, the editing unit 144 sets an adopted section to be adopted for the summary video among the unit videos selected in step S116. At this time, the editing unit 144 sets an adopted section at an appropriate position so that, for example, a section to be particularly noted is adopted in the summary video according to the content of the unit video. The editing unit 144 stores the processing result described above in the editing information.

  In step S120, the summary video generation unit 146 generates a summary video. For example, the summary video generation unit 146 generates a summary video by switching and connecting the unit video specified by the editing information at the specified timing with the music specified by the editing information as BGM.

  Heretofore, an example of the summary video generation processing flow according to the present embodiment has been described.

<5. Hardware configuration example>
Finally, the hardware configuration of the information processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 14 is a block diagram illustrating an example of a hardware configuration of the information processing apparatus according to the present embodiment. Note that the information processing apparatus 900 illustrated in FIG. 14 can realize the video processing apparatus 100 illustrated in FIG. 4, for example. Information processing by the video processing apparatus 100 according to the present embodiment is realized by cooperation of software and hardware described below.

  As illustrated in FIG. 14, the information processing apparatus 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904a. The information processing apparatus 900 includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, a communication device 913, and a sensor 915. The information processing apparatus 900 may include a processing circuit such as a DSP or an ASIC in place of or in addition to the CPU 901.

  The CPU 901 functions as an arithmetic processing device and a control device, and controls the overall operation in the information processing device 900 according to various programs. Further, the CPU 901 may be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, and the like. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like. For example, the CPU 901 can form the control unit 140 shown in FIG.

  The CPU 901, ROM 902 and RAM 903 are connected to each other by a host bus 904a including a CPU bus. The host bus 904a is connected to an external bus 904b such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 904. Note that the host bus 904a, the bridge 904, and the external bus 904b do not necessarily have to be configured separately, and these functions may be mounted on one bus.

  The input device 906 is realized by a device to which information is input by a user, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever. The input device 906 may be, for example, a remote control device using infrared rays or other radio waves, or may be an external connection device such as a mobile phone or a PDA that supports the operation of the information processing device 900. . Furthermore, the input device 906 may include, for example, an input control circuit that generates an input signal based on information input by the user using the above-described input means and outputs the input signal to the CPU 901. A user of the information processing apparatus 900 can input various data and instruct a processing operation to the information processing apparatus 900 by operating the input device 906. The input device 906 can form, for example, the operation unit 112 shown in FIG.

  The output device 907 is formed of a device capable of visually or audibly notifying acquired information to the user. Examples of such devices include CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, display devices such as lamps, audio output devices such as speakers and headphones, printer devices, and the like. For example, the output device 907 outputs results obtained by various processes performed by the information processing device 900. Specifically, the display device visually displays results obtained by various processes performed by the information processing device 900 in various formats such as text, images, tables, and graphs. On the other hand, the audio output device converts an audio signal composed of reproduced audio data, acoustic data, and the like into an analog signal and outputs it aurally. The display device and the audio output device can form, for example, the output unit 130 shown in FIG.

  The storage device 908 is a data storage device formed as an example of a storage unit of the information processing device 900. The storage apparatus 908 is realized by, for example, a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like. The storage device 908 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 908 stores programs executed by the CPU 901, various data, various data acquired from the outside, and the like. The storage device 908 can form, for example, the storage unit 120 shown in FIG.

  The drive 909 is a storage medium reader / writer, and is built in or externally attached to the information processing apparatus 900. The drive 909 reads information recorded on a removable storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs the information to the RAM 903. The drive 909 can also write information to a removable storage medium.

  The connection port 911 is an interface connected to an external device, and is a connection port with an external device capable of transmitting data by, for example, USB (Universal Serial Bus). The connection port 911 can form, for example, the music acquisition unit 114 shown in FIG.

  The communication device 913 is a communication interface formed by a communication device for connecting to the network 920, for example. The communication device 913 is, for example, a communication card for wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 913 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various communication, or the like. The communication device 913 can transmit and receive signals and the like according to a predetermined protocol such as TCP / IP, for example, with the Internet and other communication devices. The communication device 913 can form, for example, the music acquisition unit 114 illustrated in FIG.

  The network 920 is a wired or wireless transmission path for information transmitted from a device connected to the network 920. For example, the network 920 may include a public line network such as the Internet, a telephone line network, a satellite communication network, various LANs (Local Area Network) including Ethernet (registered trademark), a WAN (Wide Area Network), and the like. The network 920 may also include a dedicated line network such as an IP-VPN (Internet Protocol-Virtual Private Network).

  The sensor 915 is various sensors such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, an optical sensor, a sound sensor, a distance measuring sensor, and a force sensor. The sensor 915 acquires information on the state of the information processing apparatus 900 itself, such as the posture and movement speed of the information processing apparatus 900, and information on the surrounding environment of the information processing apparatus 900, such as brightness and noise around the information processing apparatus 900. . Sensor 915 may also include a GPS sensor that receives GPS signals and measures the latitude, longitude, and altitude of the device. The sensor 915 can form, for example, the sensor unit 111 shown in FIG. In the present embodiment, the sensor 915 may be separated from the information processing apparatus 900. For example, the sensor 915 may be attached to a subject, and the information processing apparatus 900 may acquire information indicating a result of sensing the subject by wired or wireless communication.

  The imaging device 917 photoelectrically converts imaging light obtained by the lens system including a lens system including an imaging lens, a diaphragm, a zoom lens, a focus lens, and the like, a drive system that performs a focus operation and a zoom operation on the lens system, and the lens system. A solid-state imaging device array that generates an imaging signal. The solid-state imaging device array may be realized by, for example, a CCD (Charge Coupled Device) sensor array or a CMOS (Complementary Metal Oxide Semiconductor) sensor array. The imaging device 917 outputs captured image data that is a digital signal. The imaging device 917 can form, for example, the video acquisition unit 113 illustrated in FIG.

  Heretofore, an example of the hardware configuration capable of realizing the functions of the information processing apparatus 900 according to the present embodiment has been shown. Each of the above components may be realized using a general-purpose member, or may be realized by hardware specialized for the function of each component. Therefore, it is possible to change the hardware configuration to be used as appropriate according to the technical level at the time of carrying out this embodiment.

  Note that a computer program for realizing each function of the information processing apparatus 900 according to the present embodiment as described above can be produced and mounted on a PC or the like. In addition, a computer-readable recording medium storing such a computer program can be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

<6. Summary>
The embodiment of the present disclosure has been described in detail above with reference to FIGS. As described above, the video processing apparatus 100 according to the present embodiment generates a summary video that can excite the viewer's emotions by switching an appropriate unit video at an appropriate timing according to music. Is possible.

  More specifically, the video processing apparatus 100 analyzes the beat of the input music, extracts a plurality of unit videos from the input video, and generates editing information for switching the extracted unit videos according to the beat. . As a result, the unit video is switched at a fast timing according to the beat, so that the viewer's emotion can be raised more effectively.

  In addition, the video processing apparatus 100 sets an adopted section corresponding to the content of the unit video in the extracted unit video, and generates editing information for adopting the adopted section set for each of the plurality of unit videos. As a result, the video processing apparatus 100 sets, for each section extracted as a candidate to be adopted for the summary video, a section to be actually used for the summary video in a section to be particularly appreciated among each of the extracted sections. It becomes possible to do. Therefore, for example, a more appropriate section is used for the summary video to excite the viewer's emotions.

  In addition, the video processing apparatus 100 controls an operation mode related to a process of extracting a unit video from the input video and a process of setting a timing for switching the unit video according to the input music. As a result, the video processing apparatus 100 can generate a summary video that switches videos according to music in an appropriate operation mode. Specifically, the video processing apparatus 100 can switch between different unit videos at the set switching timing by switching the operation mode so that the number of employed and the number of extracted is equal or the number of extracted is larger. it can.

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

  Each device described in this specification may be realized as a single device, or a part or all of the devices may be realized as separate devices. For example, in the functional configuration example of the video processing apparatus 100 illustrated in FIG. 4, the storage unit 120 and the control unit 140 are provided in a device such as a server connected to the input unit 110 and the output unit 130 via a network or the like. Also good.

  Further, the processes described using the flowcharts and sequence diagrams in this specification are not necessarily executed in the order shown. Some processing steps may be performed in parallel. Further, additional processing steps may be employed, and some processing steps may be omitted.

  Further, the effects described in the present specification are merely illustrative or exemplary and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
Extracting multiple unit videos from the input video;
Generating editing information for switching the unit video adopted from the extracted unit video according to the input music;
Controlling an operation mode in an extraction process of extracting the unit video and an adoption process of adopting the unit video by a processor;
Video processing method.
(2)
In controlling the operation mode, in the first operation mode, depending on the magnitude relationship between the number of extracted unit videos to be extracted and the number of adopted unit videos to be employed, the extraction process is performed again or again. The video processing method according to (1), wherein it is determined whether or not at least one of the adoption processes is performed by changing the operation mode.
(3)
The video processing method includes:
Analyzing the music content;
Analyzing the content of the video;
Further including
The extraction number is the number of unit videos extracted based on the analysis result of the video,
The video processing method according to (2), wherein the adopted number is a number obtained by dividing the music based on the analysis result of the music.
(4)
In controlling the operation mode, when the adopted number and the extraction number in the first operation mode are equal or the extraction number is larger, it is determined not to change the operation mode, The video processing method according to (3).
(5)
In controlling the operation mode, when the number of extraction in the first operation mode is less than the adopted number, the operation mode is set as the second operation mode,
In the extraction process, in the second operation mode, at least one of the unit videos extracted in the first operation mode is divided into two or more unit videos. Video processing method.
(6)
The video processing method according to (4) or (5), wherein in the extraction process, the unit video having a length equal to or longer than a longest section in which the music is divided is extracted.
(7)
In controlling the operation mode, when the number of extractions in the first operation mode is less than the number of adoption, the operation mode is a third operation mode,
The video processing method according to (6), wherein, in the third operation mode, the adoption process shortens a longest section in which the music is divided as compared to the first operation mode.
(8)
In controlling the operation mode, when the number of extraction in the first operation mode is less than the number of adoption, the operation mode is a fourth operation mode,
In the extraction process, in the case of the fourth operation mode, the conditions regarding the analysis result of the video for extracting the unit video are relaxed compared to the first operation mode. The video processing method according to any one of the above.
(9)
In controlling the operation mode, when the number of extraction in the first operation mode is less than the adopted number, the operation mode is set as a fifth operation mode,
In the adoption process, in the fifth operation mode, the music is divided at a predetermined interval. In the extraction process, in the fifth operation mode, the unit video obtained by dividing the video at the predetermined interval is The video processing method according to any one of (4) to (8), wherein the video processing method is extracted.
(10)
An extraction unit for extracting a plurality of unit videos from the input video;
An editing unit that generates editing information for switching the unit video adopted from the unit video extracted by the extraction unit according to input music;
An operation mode control unit for controlling an operation mode in the extraction unit and the editing unit;
A video processing apparatus comprising:
(11)
The operation mode control unit, in the first operation mode, according to the magnitude relationship between the number of unit videos extracted by the extraction unit and the number of unit videos adopted by the editing unit, The video processing apparatus according to (10), wherein it is determined whether or not at least one of a re-extraction process by the extraction unit and a re-adoption process by the editing unit is performed by changing the operation mode.
(12)
The video processing device includes:
A music analysis unit for analyzing the contents of the music;
A video analysis unit for analyzing the content of the video;
Further comprising
The number of extractions is the number of unit videos extracted by the extraction unit based on the analysis result by the video analysis unit,
The video processing apparatus according to (11), wherein the adopted number is a number obtained by dividing the music by the editing unit based on an analysis result by the music analysis unit.
(13)
The operation mode control unit determines not to change the operation mode when the adopted number and the extracted number in the first operation mode are equal to or greater than the extracted number. ).
(14)
Computer
An extraction unit for extracting a plurality of unit videos from the input video;
An editing unit that generates editing information for switching the unit video adopted from the unit video extracted by the extraction unit according to input music;
An operation mode control unit for controlling an operation mode in the extraction unit and the editing unit;
Program to function as.

10 video 20 video analysis result information 30 music 40 editing information 50 summary video 100 video processing apparatus 110 input unit 111 sensor unit 112 operation unit 113 video acquisition unit 114 music acquisition unit 120 storage unit 130 output unit 140 control unit 141 control unit 141 music analysis unit 142 Video analysis unit 143 Extraction unit 144 Editing unit 145 Operation mode control unit 146 Summary video generation unit

Claims (14)

Extracting multiple unit videos from the input video;
Generating editing information for switching the unit video adopted from the extracted unit video according to the input music;
Controlling an operation mode in an extraction process of extracting the unit video and an adoption process of adopting the unit video by a processor;
Video processing method.   In controlling the operation mode, in the first operation mode, depending on the magnitude relationship between the number of extracted unit videos to be extracted and the number of adopted unit videos to be employed, the extraction process is performed again or again. The video processing method according to claim 1, wherein it is determined whether or not to perform at least one of the adoption processes by changing the operation mode. The video processing method includes:
Analyzing the music content;
Analyzing the content of the video;
Further including
The extraction number is the number of unit videos extracted based on the analysis result of the video,
The video processing method according to claim 2, wherein the adopted number is a number obtained by dividing the music based on an analysis result of the music.   In the control of the operation mode, it is determined that the change of the operation mode is not performed when the adopted number and the extraction number in the first operation mode are equal to or greater than the extraction number. Item 4. The video processing method according to Item 3. In controlling the operation mode, when the number of extraction in the first operation mode is less than the adopted number, the operation mode is set as the second operation mode,
5. The video according to claim 4, wherein, in the extraction process, in the second operation mode, at least one of the unit videos extracted in the first operation mode is divided into two or more unit videos. 6. Processing method.   5. The video processing method according to claim 4, wherein, in the extraction process, the unit video having a length equal to or longer than a longest section in which the music is divided is extracted. In controlling the operation mode, when the number of extractions in the first operation mode is less than the number of adoption, the operation mode is a third operation mode,
The video processing method according to claim 6, wherein in the third operation mode, the adoption process shortens a longest section in which the music is divided as compared to the first operation mode. In controlling the operation mode, when the number of extraction in the first operation mode is less than the number of adoption, the operation mode is a fourth operation mode,
5. The video according to claim 4, wherein, in the extraction process, in the case of the fourth operation mode, the condition relating to the analysis result of the video for extracting the unit video is relaxed compared to the first operation mode. Processing method. In controlling the operation mode, when the number of extraction in the first operation mode is less than the adopted number, the operation mode is set as a fifth operation mode,
In the adoption process, in the fifth operation mode, the music is divided at a predetermined interval. In the extraction process, in the fifth operation mode, the unit video obtained by dividing the video at the predetermined interval is The video processing method according to claim 4, wherein extraction is performed. An extraction unit for extracting a plurality of unit videos from the input video;
An editing unit that generates editing information for switching the unit video adopted from the unit video extracted by the extraction unit according to input music;
An operation mode control unit for controlling an operation mode in the extraction unit and the editing unit;
A video processing apparatus comprising:   The operation mode control unit, in the first operation mode, according to the magnitude relationship between the number of unit videos extracted by the extraction unit and the number of unit videos adopted by the editing unit, The video processing apparatus according to claim 10, wherein it is determined whether or not at least one of a re-extraction process by the extraction unit and a re-adoption process by the editing unit is performed by changing the operation mode. The video processing device includes:
A music analysis unit for analyzing the contents of the music;
A video analysis unit for analyzing the content of the video;
Further comprising
The number of extractions is the number of unit videos extracted by the extraction unit based on the analysis result by the video analysis unit,
The video processing apparatus according to claim 11, wherein the adopted number is a number obtained by dividing the music by the editing unit based on an analysis result by the music analysis unit.   The operation mode control unit determines that the operation mode is not changed when the adopted number and the extracted number in the first operation mode are equal to or greater than the extracted number. The video processing apparatus described in 1. Computer
An extraction unit for extracting a plurality of unit videos from the input video;
An editing unit that generates editing information for switching the unit video adopted from the unit video extracted by the extraction unit according to input music;
An operation mode control unit for controlling an operation mode in the extraction unit and the editing unit;
Program to function as.

Images