JP2010245856A - Video editing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video editing device that can perform digest reproduction in which comparatively important scenes are collected, and scenes having same feature with little deviation as a scene do not continue. <P>SOLUTION: The video editing device calculates the evaluation of each scene, when generating a digest, according to attribute information added in the generation of video, and the attribute information obtained by analyzing the content of the video, and at the same time, finds the relevance between the scenes, and selects the scene to be extracted according to the relevance. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a video editing apparatus, and more particularly to a video editing apparatus that enables video digest (summary) playback.

  Patent Document 1 discloses an electronic camera that can perform digest reproduction in which various types of editing are performed on a captured moving image without input of a heavy editing operation. The electronic camera generates a random number, and performs a random editing process on the captured moving image based on the random number. Thereby, according to this electronic camera, it is possible to automatically perform various random edits and perform digest playback of a moving image without inputting an editing operation from the user.

  Japanese Patent Laid-Open No. 2004-228688 evaluates a scene based on metadata (attribute information) and, based on the evaluation result, captures a scene (summary video) that narrows down the number of scenes and clips of the captured video. An apparatus is disclosed.

JP 2005-260749 A JP 2008-227860 A

  However, the above-mentioned Patent Document 1 generates a digest (summary) video by randomly editing a shot video, and a digest is generated regardless of the video content of the extracted scene.

  Patent document 2 generates a digest based on the video content of each scene, but does not consider the relevance between a plurality of extracted scenes. Therefore, there is a possibility that a digest consisting only of the same kind of scenes as zooming up is generated.

  The present invention solves the above-described problems, and enables more suitable digest generation by considering the relationship between a plurality of extracted scenes when generating a digest. is there.

  The video editing apparatus of the present invention divides a video into scenes, generates attribute information for each scene, evaluation for each scene based on the attribute information, and relevance between a plurality of scenes, A scene analysis unit for extracting a scene to be reproduced and a reproduction information generation unit for generating reproduction information for recording information related to the scene to be reproduced are provided.

  As a result, scene extraction can be performed according to the content (attribute) between scenes, and a more suitable digest can be generated.

  Furthermore, the scene analysis unit may extract only one scene when the two scenes to be extracted have common attributes. As a result, scenes having the same attribute are not extracted and a digest is not generated.

  Furthermore, the scene analysis unit determines whether or not two scenes need to be extracted based on the relationship between the different attributes even when the two scenes to be extracted have different attributes. It may be.

  As a result, it is possible to extract scenes by determining the relevance between scenes depending on the type of attribute, and thus it is possible to generate a more suitable digest.

  According to the present invention, when generating a digest, it is possible to generate a more suitable digest by taking into account the relevance between a plurality of extracted scenes.

Video camera overview Hardware configuration diagram inside the video camera Functional configuration diagram inside the video camera Relationship diagram of clips, scenes, and frames of recorded video Diagram showing information that identifies a scene Diagram showing the relationship between attribute information and evaluation Figure showing the result of assigning an evaluation to a scene Figure extracted from a highly evaluated scene Diagram showing the relationship between zoom up / down and scene Example flowchart for determining relevance between scenes The figure which shows the relationship between the identification information of a scene, and the attribute to hold | maintain. Diagram showing the relationship between multiple attributes and related coefficients

(First embodiment)
<1. Configuration of video editing device>
In this embodiment, a description will be given using a video editing apparatus that edits video. Examples of the video editing device include, for example, a TV recording device that records a TV program and the like and digests and plays back the recorded program, a video camera (movie) that has a function of shooting video and digesting and playing back the captured video, and the like. There is. FIG. 1 is an external view of the video camera 100. In this embodiment, the video camera 100 will be described as a video editing apparatus.

  FIG. 2 shows an outline of the hardware configuration inside the video camera of FIG. The video camera 100 includes a lens group 200, an image sensor 201, a video ADC (Analog Digital Converter) 202, a video signal conversion IC 203, a CPU 204, a clock 205, a lens control module 206, an attitude detection sensor 207, an input button 208, a display 209, and a speaker. 210, output I / F (Interface) 211, compression / decompression IC 212, ROM (Read Only Memory) 213, RAM (Randam Access Memory) 214, HDD (Hard Disk Drive) 215, audio ADC (Anal Digital Converter 17) Microphone 2 , As a component.

  The lens group 200 adjusts light incident from the subject in order to form a subject image on the image sensor 201. Specifically, the focal length and zoom (image enlargement magnification) are adjusted by changing the distance between a plurality of lenses having various characteristics. These adjustments may be performed manually by a video camera photographer or automatically by control from the CPU 204 or the like described later.

  The image sensor 201 converts light incident through the lens group 200 into an electrical signal. A CCD, C-MOS, or the like can be used for the image sensor.

  The video ADC 202 converts an analog electric signal output from the image sensor 201 into a digital signal.

  The video signal conversion IC 203 converts the digital signal output from the video ADC 202 into a predetermined video signal such as NTSC or PAL.

  The CPU 204 controls the entire video camera 100. The types of control include, for example, the control of the focal length and zoom of the lens described above, the lens control for controlling the incident light to the image sensor 201 through the lens control module 206, the external from the input button 208, the posture detection sensor 207, and the like. These control algorithms such as input control for input and operation control of the compression / decompression IC 212 are executed by software or the like.

  The clock 205 outputs a clock signal serving as a reference for processing operation to a circuit such as the CPU 204 operating in the video camera 100. Note that the clock 205 may be a single clock or a plurality of clocks depending on an integrated circuit to be used and data to be handled. Further, an arbitrary multiple of the clock signal of one oscillator may be used.

  The lens control module 206 detects the state of the lens group 200 and operates the lens based on the control from the CPU 204. The lens control module 206 includes a lens control motor and a lens position sensor. The lens position sensor detects a distance or a positional relationship between a plurality of lenses constituting the lens group 200. Position information between the plurality of lenses detected by the lens position sensor is transmitted to the CPU 204. The CPU 204 transmits a signal for properly arranging a plurality of lenses to the lens control motor based on information from the lens position sensor and information from other components such as the image sensor 201. The lens control motor drives a motor that operates the lens based on a control signal transmitted from the CPU 204. As a result, the distance between the plurality of lenses of the lens group 200 is changed, and by adjusting the focal length and zoom of the lens, the incident light transmitted through the lens group 200 is the target object image on the image sensor 201. Tie.

  In addition to the above, the CPU 204 detects camera shake during video shooting with the video camera 100 using a lens position sensor, an attitude detection sensor described later, and the like, and performs control to drive the lens control motor, thereby preventing camera shake. It can also be executed by the lens control module 206.

  The posture detection sensor 207 detects the posture state of the video camera 100. The posture detection sensor 207 includes an acceleration sensor, an angular velocity sensor, an elevation angle / decline angle sensor, and the like. With these various sensors, the CPU 204 detects in what state the video camera 100 is shooting. Note that these sensors are preferably capable of detecting in three axial directions (vertical direction, horizontal direction, etc.) in order to detect the attitude of the video camera 100 in detail.

  The input button 208 is one of input interfaces used by the photographer of the video camera 100. Accordingly, the photographer can transmit various requests to the video camera 100, such as the start and end of shooting, and the insertion of a marking in the video being video-recorded.

  The display 209 is provided for use as an image captured by the video camera 100, a viewfinder at the time of image capturing, or the like. As a result, the photographer can check the photographed image on the spot. In addition, by displaying various information of the video camera 100, it is possible to convey more detailed information such as shooting information and device information to the photographer.

  The speaker 210 is used for audio output when playing back a captured video. In addition, it is possible to use the speaker 210 when the video camera 100 outputs a sound for conveying various kinds of information (for example, information related to photographing) to the photographer.

  The output I / F 211 is used to output video captured by the video camera 100 to an external device. Specifically, there are a cable interface for connecting to an external device with a cable, a memory card interface for recording a photographed video on a memory card, and the like. As a result, the captured video can be viewed using an external display larger than the display 209 provided in the video camera 100.

  The compression / decompression IC 212 converts the captured video or audio into a predetermined digital data format (encoding processing). Specifically, the captured video / audio data is subjected to encoding processing such as MPEG (Moving Picture Experts Group) or H264, and is converted (compressed) into a predetermined data format. Further, at the time of reproducing the captured data, the compression / decompression IC performs data processing for decompressing the video data in a predetermined data format and displaying it on the display 209 or the like.

  The ROM 213 records software programs processed by the CPU 204 and various data for operating the programs.

  The RAM 214 is used as a memory area used when executing a software program processed by the CPU 204. The RAM 214 may be used in common with the compression / decompression IC 212.

  The HDD 215 is used for the purpose of storing video data encoded by the compression / decompression IC 212. In addition to this, it is also possible to record reproduction information data to be described later.

  The audio ADC 216 converts the analog analog electrical data output from the microphone 217 into a digital signal.

  The microphone 217 converts the sound outside the video camera 100 into an electrical signal and outputs it.

  In the present embodiment, an example of the hardware configuration of the video camera 100 has been described above. However, the present invention is not limited to the above configuration. For example, the video ADC 202, the video signal conversion IC 203, and the like can be realized as a single integrated circuit, and a part of a software program executed by the CPU 204 is separately separated by hardware using an FPGA (Field Programmable Gate Array). It is also possible to realize.

  FIG. 3 shows a functional configuration diagram of the video camera 100. The video camera 100 includes, as a functional configuration, a lens unit 300, an imaging unit 301, a video AD conversion unit 302, a signal processing unit 303, a video signal compression unit 304, an imaging control unit 305, a video analysis unit 306, a lens control unit 307, an attitude. Detection unit 308, attribute information generation unit 309, multiplexing unit 310, storage unit 311, scene analysis unit 312, reproduction information generation unit 313, audio analysis unit 314, audio signal compression unit 315, digest reproduction unit 316, video signal decompression unit 317, a video display unit 318, an audio signal expansion unit 319, an audio output unit 320, an audio AD conversion unit 321, a microphone unit 322, and an external input unit 323.

  The lens unit 300 adjusts the focal length of the light incident from the subject and the zoom magnification (magnification magnification of the image). These are performed under the control of the lens control unit 307. The lens unit 300 corresponds to the lens group 200 in FIG.

  The imaging unit 301 converts the light transmitted through the lens unit 300 into an electrical signal. The imaging unit 301 outputs data in an arbitrary range on the imaging device under the control of the imaging control unit 305. In addition to video data, chromaticity space information of three primary colors, white coordinates, and gain information of at least two of the three primary colors, color temperature information, Δuv (delta uv), and gamma information of the three primary colors or luminance signals Etc. can also be output. In this case, these pieces of information are output to the attribute information generation unit 309. The imaging unit 301 corresponds to the imaging element 201 in FIG.

  The video AD conversion unit 302 converts the analog electrical signal from the imaging unit 301 into a digital signal according to predetermined processing contents. The video AD converter 302 corresponds to the video ADC 202 in FIG.

  The signal processing unit 303 converts the digital signal output from the video AD conversion unit 302 into a predetermined video signal format. For example, the video signal conforms to the number of horizontal lines, the number of scanning lines, and the frame rate specified by NTSC (National Television System Committee). The signal processing unit 303 corresponds to the video signal conversion IC 203 in FIG.

  The video signal compression unit 304 performs predetermined coding conversion on the digital video signal, and performs processing such as compression of data amount and coding suitable for video. Specific encoding conversion includes MPEG2, MPEG4, and H264 encoding methods. The video signal compression unit 304 corresponds to the compression function of the compression / decompression IC 212 in FIG.

  The imaging control unit 305 controls the operation of the imaging unit 301. The image pickup unit 301 controls the exposure amount at the time of shooting, the shooting (shutter) speed, sensitivity, and the like. These control information are also output to the attribute information generation unit 309. The imaging control unit 305 is one of control algorithms processed by the CPU 204 in FIG.

  The video analysis unit 306 extracts video features from the captured video signal. In the present embodiment, the color signal (for example, color distribution included in the video is detected), white balance, and when the video includes a human face, the video signal is detected by performing face detection. By analyzing, the feature of the video is extracted. The color distribution can be detected by confirming color information included in the data forming the video signal. Further, face detection can be realized by using pattern matching or the like. The video analysis unit 306 is one of algorithms processed by the CPU 204 in FIG.

  The lens control unit 307 controls the operation of the lens unit 300. Lens control includes zoom control, focus control, camera shake correction control, and the like. The lens control unit 307 controls the lens unit 300 and outputs the control information to the attribute information generation unit 309. The lens control unit 307 corresponds to the lens control module 206 in FIG.

  The posture detection unit 308 detects acceleration, angular velocity, elevation angle, depression angle, and the like of the video camera 100. The detected information is output to the attribute information generation unit 309 using the attitude of the video camera 100 and its change state as attribute information. It is desirable that acceleration and angular velocity can be detected in three directions, vertical and horizontal (two directions). The posture detection unit 308 corresponds to the posture detection sensor 207 in FIG.

  The attribute information generation unit 309 uses shooting information at the time of video shooting, external input information, and other information as attribute information (metadata). The following information can be considered as an example of attribute information.

・ Shooting start date and time (shooting start time)
・ Shooting end date and time (shooting end time)
・ Shooting time (playback time)
-Focal length-Zoom magnification-Exposure-Shooting speed-Photosensitivity-Color space information for the three primary colors-White balance-Gain information for at least two of the three primary colors-Color temperature information-Δuv (Delta uv)
・ Gamma information of three primary colors or luminance signals ・ Color distribution ・ Face recognition information ・ Camera posture (acceleration, angular velocity, elevation angle, depression angle, etc.)
・ Shooting time (shooting start time, end time)
・ Shooting index information ・ User input ・ Frame rate ・ Sampling frequency ・ Characteristic audio (input of specific sound)
・ Scene switching ・ Shooting status (fixed shooting using a tripod, etc., or shooting by the photographer in hand)
-Camera shake during video shooting-Face detection-Face recognition Also, information other than the above is attribute information if it is information related to the video.

  Information (secondary information) calculated from the various types of information is also included in the attribute information. As the secondary information, for example, camera work such as “pan” and “tilt” as attribute information is also attribute information from the camera posture (acceleration, angular velocity, elevation angle, depression angle, etc.) information when the video camera 100 is shooting. Further, the focal length and zoom magnification information can be used as attribute information as they are. In addition to this, information newly generated by combining and analyzing the above-described various information is also attribute information. The attribute information generation unit 309 generates attribute information by extracting and calculating information useful for scene evaluation from various types of information at the time of shooting.

  The multiplexing unit 310 multiplexes the encoded video data from the video signal compression unit 304, the encoded audio data from the audio signal compression unit 313, and the attribute information from the attribute information generation unit 309 and outputs the multiplexed data. The multiplexing unit 314 may be software executed by the CPU 204 in FIG. 2, or may be processed by the compression / decompression IC 212.

  The storage unit 311 temporarily holds the encoded video data, the encoded audio data, and the data multiplexed with the attribute information output from the multiplexing unit 310 for a long time. The storage unit 311 also holds the reproduction information generated by the reproduction information generation unit 313. The storage unit 315 corresponds to the HDD 215 or the RAM 214 in FIG.

  The scene analysis unit 312 evaluates each scene based on the attribute information generated by the attribute information generation unit 309 and selects a scene to be reproduced based on the result. The scene evaluation and selection method will be described in detail later.

  The reproduction information generation unit 313 generates a scene to be reproduced selected by the scene analysis unit 312 as reproduction information. This point will also be described later.

  The attribute information generation unit 309, the scene analysis unit 312 and the reproduction information generation unit 313 are processed as software by the CPU 204 in FIG.

  The voice analysis unit 314 extracts a characteristic sound from the voice data. The characteristic sounds here include, for example, a photographer's voice, pronunciation of a specific word, cheers, gunshots, and the like. These sounds can be identified by registering in advance the unique frequencies of these sounds (speech) and using a method such as discrimination based on the comparison result. In addition, other features such as the input level of the sound captured by the microphone are also detected. The voice analysis unit 314 is one of algorithms processed by the CPU 204 in FIG.

  The audio signal compression unit 315 converts the audio data output from the audio AD conversion unit 321 using a predetermined encoding algorithm. There are methods such as MP3 (MPEG Audio Layer-3) and AAC (Advanced Audio Coding) for encoding. The audio signal compression unit 315 is one of the compression functions in the compression / decompression IC 212 in FIG.

  Based on the reproduction information recorded in the storage unit 311, the digest reproduction unit 316 converts the video data and audio data of the multiplexed data recorded in the storage unit 311 into a video signal expansion unit 317 and an audio signal expansion unit 319. Are decoded and output from the video display unit 318 and the audio output unit 320, respectively. The digest playback unit 316 is a software processing algorithm executed by the CPU 204 of FIG.

  The audio AD conversion unit 321 converts the analog audio signal output from the microphone unit 322 into audio data of a digital signal. The audio AD conversion unit 321 corresponds to the audio ADC 216 in FIG.

  The microphone unit 322 converts ambient sounds into electrical signals and outputs them as audio signals. The microphone unit 322 corresponds to the microphone 217 in FIG.

  The external input unit 323 outputs various types of information received from the outside during video shooting, such as button input by a photographer, shooting index information received from the outside via communication, and the like. Note that the shooting index information is, for example, an identification number used for identifying each shooting such as a number for identifying a shooting scene or a number indicating the number of shooting times at the time of shooting a movie. The external input unit 323 corresponds to the input button 208 in FIG.

  With the above-described configuration, a video captured by the video camera 100 can automatically extract a preferable scene from the captured video based on the attribute information, and can reproduce only that portion.

  Note that the hardware configuration diagrams and functional diagrams of FIGS. 2 and 3 are one aspect of the present embodiment, and the present invention is not limited to this. For example, in FIG. 3, the scene analysis unit 312 and the reproduction information generation unit 313 read and process data recorded in the storage unit 311, but before the multiplexed data is recorded in the storage unit 311, compression is performed. It is also possible to perform processing such as scene analysis and reproduction information generation based on the recorded video signal, compressed audio signal, and attribute information at the time of shooting.

  Further, the relationship between the hardware configuration of FIG. 2 and the functional configuration of FIG. 3 described above is one aspect of the present embodiment, and the present invention is not limited to this. For example, the process of the imaging control unit 305 processed by the software algorithm of the CPU 204 may be performed by an independent hardware IC or the like.

<2. Analysis of captured scene, generation of playback information>
FIG. 4 is a diagram illustrating a configuration of an image captured by the video camera 100. The unit of the video imaged until the photographer gives an instruction to start shooting and gives an instruction to end shooting or pause shooting is “clip”. When the photographer repeats the start of shooting, the end of shooting, or the pause repeatedly, a plurality of clips are generated (FIG. 4A).

  One clip is composed of one or a plurality of “scenes”. A “scene” is a continuous video sequence in time. A scene can be set arbitrarily. For example, “1 clip” = “1 scene” may be set with a clip as one scene. In addition, a “scene” may be set on the boundary that the screen changes greatly. In this case, when the video analysis unit 306 calculates a motion vector between frames and the magnitude (change) of “motion” is larger than a predetermined value, the “scene” switching portion may be used. Further, a logical group may be set as one “scene” according to the content of the video. In this case, it is possible for the photographer to respond by inputting a logical break with an input button or the like. In this case, the “scene” in the “clip” can be configured with a clear intention of the photographer. In addition, the scene may be divided at regular intervals (FIG. 4B).

  A “scene” is composed of one or more “frames”. A “frame” is an individual image constituting a video (FIG. 4C).

  FIG. 5 shows an example in which the scene analysis unit 310 divides the clip into a plurality of scenes. The scene analysis unit 310 divides the clip based on the attribute information and the like as described above. In FIG. 5, each scene is defined by “start time” and “end time”, but the start and end of the scene may be defined by a frame number or the like.

  FIG. 6 is a diagram showing an example of the relationship between the attribute information used when the scene analysis unit 312 evaluates each scene and the evaluation. For example, if the clip-in (shooting start part) or clip-out (pre-shooting end part) part is considered as an introduction part or an important part of the picture, the logical meaning of the shot picture is high. It is inferred. In this example, the evaluation of clip-in (A) and clip-out (F) is 100. In addition, with regard to zoom-up (D) and zoom-down (G) as camera work at the time of shooting, the evaluation is set to 30 from the viewpoint of increasing the degree of attention to a specific subject. As described above, the scene analysis unit 312 has a numerical evaluation for each piece of attribute information in advance. In the example of FIG. 6, the higher the evaluation score, the higher the evaluation (preferred). The scene analysis unit 312 evaluates each scene based on the relationship between the attribute information and evaluation in FIG. When a plurality of pieces of attribute information are given to one scene, evaluations (evaluation points) assigned to the respective attribute information may be added. Further, when a plurality of pieces of attribute information are given to one scene, the evaluation (evaluation point) of the attribute having the highest evaluation among the attribute information may be used as the evaluation point of the scene. Furthermore, if various attributes included in the scene are taken into consideration, an average value of evaluation points of a plurality of attributes may be evaluated. Furthermore, when evaluating in more detail, you may evaluate for every flame | frame contained in a scene. Note that the evaluation need not be performed only on a preferable scene. For example, camera shake at the time of shooting may result in a video that is difficult for the viewer of the video to view. Therefore, a scene having such an attribute may be evaluated for a deduction (minus point).

  Note that the relationship between the attribute information and the evaluation in FIG. 6 is not limited to one. For example, the combination data of a plurality of attribute information / evaluation may be switched depending on a mode (for example, landscape shooting, person (portrait) shooting, still life shooting, etc.) in which the video camera 100 is shooting. Alternatively, a plurality of combination data may be provided in advance, and a plurality of data may be combined (addition of respective evaluation values at a certain ratio, etc.) depending on the shooting mode. In this case, the combination data of attribute information and evaluation can be dynamically changed by changing the composition ratio.

  FIG. 7 is a diagram showing a result of assigning an evaluation (priority) according to FIG. 6 to each scene of the video divided into the scene of FIG. FIG. 7 shows time (scene) on the horizontal axis and evaluation (priority) of each scene on the vertical axis.

  A in the vicinity of time 0 in FIG. 7 has an attribute of “clip in” because it is immediately after the start of shooting. According to FIG. 6, the attribute of “clip-in” has an evaluation (priority) of 100.

  The scene to which the symbol B is attached has an attribute of “extraction of specific sound”. The extraction of the specific voice is obtained by the above-described voice analysis unit 314 or the like. According to FIG. 6, the attribute of “extraction of specific speech” has an evaluation (priority) 70.

  The scene with the symbol C indicates an attribute that means that the photographer shoots by moving the camera body of the video camera 100 such as pan and tilt, and then shoots still. It is conceivable that such shooting can be judged to have a high evaluation because the photographer is very interested in (attention to) the subject when shooting at a standstill. According to FIG. 6, such an attribute has an evaluation (priority) 40.

  A scene to which a symbol D is attached is a scene in which the video camera is zoomed up and photographed. In FIG. 6, the zoom-up has an evaluation (priority) 30.

  The scene with the symbol E is a scene that is shot with the video camera zoomed down. In FIG. 6, zoom-down has the same evaluation (priority) 30 as zoom-up.

  It is also possible to change the evaluation value by zooming up and zooming down. For example, by setting the zoom-up higher than the zoom-down, the scene that is shot with the zoom-up, that is, the scene where the enlargement magnification of the video is large (the scene with the subject that is shot with the zoom) is highly evaluated ( (Priority) may be assigned. On the other hand, a relatively low evaluation (priority) can be assigned to a scene with a small video magnification.

  The scene with the symbol F has a “clipout” attribute because it is just before the end of shooting. According to FIG. 6, the attribute of “clipout” has an evaluation (priority) of 100.

  The scene to which G is attached is a scene in which the video camera performs shooting (camera work) with movement such as panning and tilting. In this case, evaluation (priority) 25 is assigned.

  A scene to which Z is not required is a scene with face detection. This indicates that a “face” of a person is detected in the subject video from the video signal. According to FIG. 6, face detection has an evaluation (priority) 80. It should be noted that “face recognition”, which is more technically advanced in face detection, may be used as a scene evaluation. Face recognition is a technique for identifying a specific face when there are a plurality of human faces in the subject video.

  As described above, the scene analysis unit 312 assigns an evaluation (priority) to each scene. In the example of FIG. 7, the evaluation is assigned in units of scenes. However, the scene analysis unit 312 may perform the above-described evaluation assignments in units of clips or frames.

  The scene analysis unit 312 further extracts only normally preferable scenes based on the evaluation assigned to each scene. A simple example is a method in which the highest evaluation included in each scene is adopted as a representative value, and only scenes having a high representative value are extracted. In the example of FIG. 7, only the scenes of the portions # 1, # 5, and # 8 are extracted.

  As for the extraction method, extraction can be performed from various viewpoints such as that the total playback time of the extracted scenes is within a predetermined time, or that the evaluation of the scene is more than a certain level.

  The reproduction information generation unit 313 generates reproduction information describing the procedure / method of video reproduction according to the scene extracted by the scene analysis unit 312. This reproduction information may be indicated by the start time and end time of a scene to be reproduced as shown in FIG. 8, for example. In this case, if a representative video screen in each scene (a video screen having the highest evaluation in the scene, etc.) is recorded separately, it is also effective for searching the reference screen.

  Note that the form of the reproduction information is not limited to that shown in FIG. 8, but may be other forms. For example, designation by a frame number is also possible. In addition, when the reproduction information generated by the reproduction information generation unit 313 is multiplexed with a video signal or audio signal encoded by the multiplexing unit 310 as a TS (Transport Stream) such as MPEG, it is used at the time of multiplexing. It is also possible to record the reproduction information using the time information (for example, time information of PTS or DTS). Similarly, in the case of H264, time information at the time of predetermined multiplexing may be used.

  Furthermore, when video data is recorded using a standard such as AVCHD (Advanced Video Code High Definition), which is used as a data recording method of some video cameras, a method of recording reproduction information in a PlayList file or the like is used. It may be used.

  As described above, it is possible to automatically generate a digest video (summarized video) more appropriately from the captured video.

<3. About scene extraction based on attribute information>
It is a simple method that the scene analysis unit 312 extracts the scenes from the captured video in order from the highest evaluation (priority) of each scene. However, if a digest is generated only with a scene extracted by this method, there is a high possibility that the scene is composed only of similar scenes, and there is a possibility that the digest is difficult for the viewer to see.

  In the example of FIG. 7, the number of scenes to be extracted can be up to five. In the above example, if five scenes are selected in descending order of representative values, scene # 1, scene # 4, and scene # 5 are selected. Scene # 6 and Scene # 8 are selected. Next, let us look at the video content of the captured video in FIG. 7 from the viewpoint of zooming up and zooming down. FIG. 9 is a diagram in which only the attribute information of zoom-up (D) and zoom-down (E) is extracted from FIG. According to this, it can be seen that scenes # 2 to # 6 are photographed with zoom-in. Therefore, in the digest extracted by the above method, the zoomed-up video continues for the continuous portion of scene # 4, scene # 5, and scene # 6. Therefore, all the scenes are zoomed-in images, and the digest images may be very difficult to see.

  Therefore, the scene analysis unit 312 according to the present embodiment performs scene extraction in consideration of the relationship between a certain scene and another scene. Specifically, the scene is extracted according to the flowchart shown in FIG. The scene extraction method in FIG. 10 will be described below.

  In S1010, it is determined whether or not extraction of a scene that is a digest candidate is completed from the captured video. In the above example, if five scenes have been extracted, it is determined that the process has been completed, and the process proceeds to S1030. If less than five scenes have been extracted, the process proceeds to S1020.

  In step S1020, a scene that is a digest candidate is extracted from the captured video. Specifically, a scene with a high evaluation is extracted from unextracted scenes. By repeating S1010 and S1020 as many times as necessary, the extraction of the scene necessary for the digest is completed.

  In S1030, if all the processes are completed for the relationships between the extracted scenes, the process of this flowchart ends. If not completed, the processing from S1040 is performed.

  In S1040, a first scene and a second scene are acquired from the extracted scene. The acquisition criterion for the first scene is that the scene with the highest evaluation among the scenes that have not yet been acquired as the first scene in the processing of S1040 to S1060 is preferentially acquired. The second scene acquisition criterion is to acquire a scene temporally adjacent to the first scene among scenes acquired as digest candidates.

  In the example of FIG. 7, scene # 1, scene # 4, scene # 5, scene # 6, and scene # 8 are extracted as digest candidates. Of these, scene # 1 and scene # 8 have the highest evaluation. In this case, the first scene is scene # 1. Then, the digest candidate scene that is continuous with the first scene is scene # 3.

  In S1050, the attribute information of the first scene (here, scene # 1) and the second scene (here, scene # 3) is acquired. Specifically, attribute information of clip-in (A) is acquired from scene # 1, and attribute information of zoom-up (D) and pan / tilt (G) is acquired from scene # 3.

  In S1060, it is determined whether the acquired attribute information is the same. In the above example, the clip-in (A) of the scene # 1, the zoom-up (D) and the pan / tilt (G) of the scene # 3 are all different, and thus are determined not to be the same. In this case, the process returns to S1030 again.

  When the process is returned to S1030 again and the same process is performed, the first scene and the second scene are acquired in S1040. In this case, since scene # 1 has already been acquired as the first scene, it is not an acquisition candidate. In this case, what is acquired as the first scene is the most evaluated scene # 8 except for the scene # 1. The second scene is scene # 6 temporally adjacent to the first scene (scene # 8 here). If the processes of S1050 and S1060 are similarly performed for the scene # 8 and the scene # 6, the same attribute information is not obtained, so the process of S1070 is not performed here and the process returns to S1030 again.

  When the process returns to S1030, the scene # 5 is acquired as the first scene, and the scene # 4 and the scene # 6 are acquired as the second scene in S1040. In this case, in S1050, the still image (C) and face detection / face recognition (Z) are taken as attribute information of the scene # 5, the specific sound is extracted from the scene # 4 (B), and the image is taken still (C ), Pan / tilt (G), the attribute information of specific sound extraction (B), zoom-down (E), and pan / tilt (G) is acquired from scene # 6.

  Here, looking at the scene # 4, the scene # 5, and the scene # 6 regarding the zoom control, as shown in FIG. 9, it can be seen that the zoom-up continues from the second half of the scene # to the second half of the scene # 6. . Therefore, scene # 4, scene # 5, and scene # 6 all have the zoom-up (D) attribute. When this is arranged, the contents shown in FIG. 11 are obtained.

  In this case, in S1060, the first scene (scene # 5) and the second scene (scene # 4) are stationary, and shooting (C) and zoom-up (D) have the same attributes. Further, the zoom-up (D) has the same attribute in the first scene (scene # 5) and another second scene (scene # 6). Therefore, in step S1070, scene # 4 and scene # 6 are deleted from the digest candidates.

  Even if the process returns to S1030 again, the check on the attribute information has already been completed for scene # 1, scene # 5, and scene # 8 remaining as digest candidates, and thus the process of this flowchart ends.

  As described above, the scene analysis unit 312 extracts a highly evaluated scene from the video as a digest candidate, and further selects a scene suitable for the digest based on the attribute information between the extracted scenes.

<4. About generation of reproduction information based on new evaluation>
The reproduction information generation unit 313 specifies a scene to be reproduced, extracted in consideration of the evaluation performed by the scene analysis unit 312 for each scene and the relationship between the scenes. As a result of the above example, FIG. 8 shows an example in which the scene analysis unit 312 specifies a scene to be reproduced by digest.

  Note that the reproduction information generated by the reproduction information generation unit 313 may not reproduce only the evaluated scene. For example, an evaluated scene may be reproduced at a normal speed, and a scene that has not been evaluated may be reproduced at a high speed. That is, the playback method may be changed between an evaluated scene (a highly evaluated scene) and a non-evaluated scene (a low evaluated scene).

  As described above, with the configuration described in this embodiment, when the video camera 100 generates a digest from a captured video, more appropriate digest generation can be performed by taking into account the relevance between a plurality of extracted scenes. It becomes possible.

  In the present embodiment, with regard to the scene extraction method in the scene analysis unit 312, once the scenes are extracted in descending order of evaluation, and the relationship between the extracted scenes is checked, the relationship between the scenes is determined. Although considered, it is not limited to this. For example, the scene may be extracted in consideration of the relevance with other scenes at the stage of extracting the scene from the video. In this case, the scene is extracted from the most evaluated scene, and when the second and subsequent scenes are extracted, the attribute information is identical with the scene that has already been extracted and is temporally adjacent. A determination method may be used.

In the present embodiment, the determination is made based on whether or not the attribute information is the same, but the present invention is not limited to this. For example, a relationship coefficient as shown in FIG. 12 may be determined in advance for the relationship between different attribute information, and the relationship between adjacent scenes may be determined based on this. In the example of FIG. 12, each attribute information is the same or shows how strongly related to other attribute information. The larger the coefficient value, the stronger the relationship. In the example of FIG. 12, when the zoom-in (D) is included in both of the adjacent scenes, it is indicated that the coefficient is 1.0, which indicates that the relationship is very strong. In contrast, the zoom-up (D) and the face detection / face recognition (Z) are given a coefficient of 0.1 because they are relatively weakly related. When judging the relevance between scenes, it is possible to digitize using these coefficients, and it is also possible to take a method such as not to adopt a relevance of a certain value or more as a digest. Although the description has been given by way of example between adjacent scenes, the present invention is not limited to this. For example, the digest may be generated in consideration of relevance with all other scenes constituting the digest. In this case, since the scene is extracted as the entire digest, it is possible to generate a digest that is coherent as a whole.

  In addition, in the present embodiment, the order of scenes at the time of digest is also determined based on the order of scenes constituting the video, but it is not necessary to be limited to this. In the description of the above embodiment, the scene # 1 is always placed at the head when the digest is generated, and the scene # 5 is not played back before the scene # 1. However, scene # 5 may be arranged at the head, and scene # 1 may be reproduced after scene # 5. In this case, when the attribute information is the same between adjacent scenes, the digest can be rearranged so that the digest can be easily read.

  Further, when generating a digest from a plurality of clips, the scene analysis unit 312 may determine the number of scenes to be extracted from each clip according to the length of the shooting time (reproduction time) of each clip. . In this case, since the scenes constituting the digest are extracted according to the shooting time (playback time) of each clip, it is possible to generate a digest in which scene extraction is performed by leveling from a plurality of clips. It becomes possible. In this case, by performing scene extraction in consideration of the relevance between scenes straddling between clips, the generated digest may have a more easily viewable content.

  As described above, as shown in the present embodiment, it is possible to generate a digest that is easier to view by considering attribute information between scenes when generating a digest.

  In this embodiment, the case of generating a digest (summary) video has been described as an example. However, the present invention can be similarly applied to a highlight video obtained by collecting scenes that rise as video content from a shot video. In this case, for the relationship between the attribute information and the evaluation, the one for highlight video generation is used, but the scene to be extracted is determined in consideration of the relationship with other scenes shown in this application. Can be implemented in the same manner.

  INDUSTRIAL APPLICABILITY The present invention can be used in product fields such as a video camera for generating video, a camera, a video player for viewing these videos, a television, and a video editor for editing these videos.

DESCRIPTION OF SYMBOLS 100 Video camera 200 Lens group 201 Image pick-up element 202 Image | video ADC
203 Video signal conversion IC
204 CPU
205 Clock 206 Lens Control Module 207 Posture Detection Sensor 208 Input Button 209 Display 210 Speaker 211 Output I / F
212 Compression / decompression IC
213 ROM
214 RAM
215 HDD
216 Audio ADC
217 Microphone 300 Lens unit 301 Imaging unit 302 Video AD conversion unit 303 Signal processing unit 304 Video signal compression unit 305 Imaging control unit 306 Video analysis unit 307 Lens control unit 308 Posture detection unit 309 Attribute information generation unit 310 Multiplexing unit 311 Storage unit 312 Scene analysis unit 313 Playback information generation unit 314 Audio analysis unit 315 Audio signal compression unit 316 Digest reproduction unit 317 Video signal expansion unit 318 Video display unit 319 Audio signal expansion unit 320 Audio output unit 321 Audio AD conversion unit 322 Microphone unit

Claims (3)

A video editing device for editing video,
An attribute information generating unit that divides the video into scenes and generates attribute information for each scene;
A scene analysis unit for extracting a scene to be reproduced based on the evaluation for each scene based on the attribute information and the relationship between a plurality of scenes;
A reproduction information generating unit for generating reproduction information for recording information about the scene to be reproduced;
A video editing apparatus comprising: When the scene analysis unit has common attribute information between two scenes due to the relationship between the scenes, the scene analysis unit extracts only one scene.
The video editing apparatus according to claim 1. When the scene analysis unit has different attribute information between the two scenes due to the relationship between the scenes, the scene analysis unit determines whether it is necessary to extract the two scenes based on the relationship between the different attribute information.
The video editing apparatus according to claim 1.

Images