JP2010263611A - Video photographing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve such a problem that scene extraction while recognizing characteristics of optical zooming and electronic zooming cannot be performed, when the scene extraction is automatically performed from a photographed video. <P>SOLUTION: A video photographing apparatus for photographing the video includes an attribute information generating portion which generates attribute information during photographing the video, a scene analyzing portion which evaluates the video at every scene based on at least one among pieces of information regarding a video magnification changing method of the video included in the attribute information, information regarding a video magnification, information regarding substantial resolution of the video and information regarding a temporal change rate of the video magnification, and a reproduction information generating portion which generates reproduction information of the video based on the evaluation. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a video photographing device for photographing a video or a video editing device for editing a video, and more particularly to a technique for automatically extracting an important scene based on a change in the magnification (zoom) of the video.

  Conventionally, shooting of movies, TV dramas and the like has been performed based on scenarios (scripts) created based on storyboards. A scenario has a title (name) and is composed of a plurality of scenes. A scene is composed of a plurality of cuts.

  The director performs according to the scenario, and actors such as actors, actresses, and extras perform according to the scenario description. As described above, in the shooting of a movie or drama by a professional, the actual content of each scene shown in the scenario is shot regardless of the shooting order. However, when non-professional people take pictures of daily life, there are usually no scenarios, leaving a happening experience as a video recording, and the enjoyment to watch later is the current video camera (VC) and digital It forms the market for still cameras (DSC).

  Currently, a commercially available video camera has a function that makes it easier to shoot a moving image. However, a function for easily editing and viewing captured moving images using metadata (attribute information) or the like has been studied but has not yet become widespread.

  For example, in a video camera, a method described in Patent Document 1 is conventionally known regarding a metadata input method and an editing system. Specifically, when metadata related to content is created or tagged, a keyword extracted in advance from a scenario or the like of the produced content is input by voice. Based on the scenario, dictionary fields are set and keywords are prioritized, and metadata is created by the voice recognition means. According to this method, even when metadata is given at intervals of several seconds, which is difficult by key input, it is possible to assign metadata efficiently by using voice recognition. A scene search can also be performed as a keyword for searching metadata.

  Moreover, what was described in patent document 2 is known as an apparatus which analyzes scenario information. The apparatus includes a scenario information storage unit, a reference information storage unit, a calculation unit, and an output unit. The information storage unit stores scenario information classified for each scene. The reference information storage unit stores reference information regarding data included in the scenario information. The calculation unit extracts data from the scenario information, calculates output information based on the extracted data and reference information, and outputs the output information to the output unit. As a result, in addition to shortening the planning time of the shooting schedule, shooting can be completed quickly by shooting according to the output shooting schedule, so that shooting costs can be reduced.

  In Patent Document 3, a scene is evaluated based on metadata (attribute information), and based on the evaluation result, a video (summary video) that easily generates a digest (summary video) in which the number of captured video scenes and clips is narrowed down An imaging device is disclosed.

JP 2004-153764 A JP 2004-362610 A JP 2008-227860 A

  In the devices and methods disclosed in Patent Document 1, Patent Document 2, and Patent Document 3 described above, camera work such as panning and zooming is performed between the start and end of shooting in video camera (movie) shooting. Based on the pattern, recorded sound, user operation, etc., metadata (attribute information) is generated and listed in a characteristic scene. A digest composed of related scenes can be created using the metadata in the list after the recording is completed.

  However, a specific method for handling attribute information is not described in zoom-up photography. As a zoom method, (1) an optical zoom method in which the size of a subject on the image receiving unit is changed by adjusting a zoom lens, and (2) a display area of a received image is enlarged (enlarged). Two methods are well known, namely, an electronic zoom method for expanding video. As for the electronic zoom method, there is a known problem that the resolution of an image is lowered according to the zoom magnification. Therefore, when evaluating a zoom-up scene, it is necessary to evaluate the scene in consideration of the characteristics of the zoom-up method.

  In view of the above problems, the present application performs a suitable scene evaluation on a video shot using both the optical zoom method and the electronic zoom method, and automatically performs scene extraction based on those evaluations. With the goal.

  The present invention is a video shooting device for shooting a video, an attribute information generation unit for generating attribute information at the time of video shooting, information on a video magnification change method included in the attribute information, and video magnification A scene analysis unit that evaluates the video for each scene based on at least one of information on the video, information on a substantial resolution of the video, and information on a temporal change rate of video magnification, and on the basis of the evaluation A reproduction information generation unit for generating reproduction information of the video.

  As a result, the video shooting apparatus can automatically extract a preferable scene from the shot video.

  The video imaging apparatus of the present invention further includes an imaging unit that converts light incident from a subject into an electrical signal, and a lens unit that adjusts the light incident on the imaging unit, and information on the video magnification changing method The information may indicate at least one of a magnification changing method by optical adjustment in the lens unit and a magnification changing method by signal processing based on an electric signal output from the imaging unit.

  As a result, the video photographing apparatus can distinguish between optical zoom and electronic zoom, and can evaluate a scene using a zoom method.

  Further, the scene analysis unit may evaluate the evaluation of the scene based on the information regarding the video magnification by comparing the video magnification of the video content with a predetermined magnification.

  Thereby, it is possible to evaluate the scene using the video magnification.

  The scene analysis unit may evaluate the evaluation of the scene based on the information regarding the substantial resolution of the video content by comparing the substantial resolution of the video content with a predetermined resolution. .

  This also makes it possible to evaluate the scene with the video resolution.

  The scene analysis unit may evaluate the evaluation of the scene based on the information on the time change rate of the video magnification by comparing the time change rate of the video content with a predetermined time change rate. .

  This makes it possible to evaluate the scene according to the enlargement change (zoom up / down) speed during video shooting.

  Further, the scene analysis unit may change the evaluation method according to the shooting situation of the video shooting device.

  This makes it possible to evaluate the scene according to the shooting state of the video shooting device.

  Further, the scene analysis unit may evaluate the video for each scene based on the information of the resolution complement processing.

  As a result, when a video enlarged by electronic zoom or the like is expressed in a pseudo high resolution by resolution complementation processing, the scene can be evaluated in consideration of this point.

  According to the present invention, it is possible to perform scene evaluation suitable for an image shot using both the optical zoom method and the electronic zoom method, and to extract a scene based on the evaluation.

Video camera external view Hardware configuration diagram inside the video camera Functional configuration diagram inside the video camera Relationship diagram of clips, scenes, and frames of recorded video Diagram showing information that identifies a scene Diagram showing the relationship between attribute information and evaluation Figure showing the result of assigning an evaluation to a scene The figure which shows the information contents of reproduction information Diagram showing the relationship between optical zoom and electronic zoom A graph showing a relationship example (first example) between video magnification and scene evaluation A graph showing a relationship example (second example) between video magnification and scene evaluation A graph showing a relationship example (third example) between video magnification and scene evaluation Diagram showing the relationship between evaluated scene and playback information The figure which shows the calculation method of the magnification of electronic zoom A graph showing an example of the relationship between the actual resolution of video and magnification Graph showing an example of the relationship between the magnification change speed and the scene evaluation A graph showing an example of the relationship between the presence / absence of a tripod and magnification and scene evaluation

(Embodiment 1)
<1. Configuration of video shooting device>
FIG. 1 is an external view of a movie or video camera (video imaging device) 100 that shoots video. In this embodiment mode, description is given using this video camera.

  FIG. 2 shows an outline of the hardware configuration inside the video camera of FIG. The video camera 100 includes a lens group 200, an image sensor 201, a video ADC (Analog Digital Converter) 202, a video signal conversion IC 203, a CPU 204, a clock 205, a lens control module 206, an attitude detection sensor 207, an input button 208, a display 209, and a speaker. 210, output I / F (Interface) 211, compression / decompression IC 212, ROM (Read Only Memory) 213, RAM (Randam Access Memory) 214, HDD (Hard Disk Drive) 215, audio ADC (Analog Digital Converter 2 microphone 17) A tripod sensor 218 as a component.

  The lens group 200 adjusts light incident from the subject in order to form a subject image on the image sensor 201. Specifically, the focal length and zoom (magnification magnification of the image) are adjusted by changing the distance between a plurality of lenses having various specifications. These adjustments may be performed manually by a video camera photographer or automatically by control from the CPU 204 or the like through a lens control module 206 described later.

  The image sensor 201 converts light incident through the lens group 200 into an electrical signal. A CCD, C-MOS, or the like can be used for the image sensor.

  The video ADC 202 converts an electrical signal output from the image sensor 201 from analog to digital.

  The video signal conversion IC 203 converts the digital signal output from the video ADC 202 into a predetermined video signal such as NTSC or PAL.

  The CPU 204 controls the entire video camera 100. The types of control include, for example, control of the focal length and zoom of the lens group 200 described above, lens control for controlling incident light to the image sensor 201 through the lens control module 206, input button 208, posture detection sensor 207, and the like. These control algorithms are executed by software or the like, such as control of input control with respect to the external input, operation control of the compression / decompression IC 212, and the like.

  The clock 205 outputs a clock signal serving as a reference for processing operation to a circuit such as the CPU 204 operating in the video camera 100. Note that the clock 205 may be a single clock or a plurality of clocks depending on an integrated circuit to be used and data to be handled. Further, an arbitrary multiple of the clock signal of one oscillator may be used.

  The lens control module 206 detects the state of the lens group 200 and operates the lens based on the control from the CPU 204. The lens control module 206 includes a lens control motor and a lens position sensor. The lens position sensor detects a distance or a positional relationship between a plurality of lenses constituting the lens group 200. Position information between the plurality of lenses detected by the lens position sensor is transmitted to the CPU 204. The CPU 204 transmits a signal for properly arranging a plurality of lenses to the lens control motor based on information from the lens position sensor and information from other components such as the image sensor 201. The lens control motor drives a motor that operates the lens based on a control signal transmitted from the CPU 204. As a result, the distance between the plurality of lenses of the lens group 200 is changed, and by adjusting the focal length and zoom of the lens, the incident light that has passed through the lens group 200 is the target object image on the image sensor 201. Will be tied.

  In addition to the above, the CPU 204 detects camera shake during video shooting with the video camera 100 using a lens position sensor, a posture detection sensor described later, and the like, and performs control to drive the lens control motor, thereby preventing camera shake. Can also be executed by the lens control module 206.

  The posture detection sensor 207 detects the posture state of the video camera 100. The posture detection sensor 207 includes an acceleration sensor, an angular velocity sensor, an elevation angle / decline angle sensor, and the like. With these various sensors, the CPU 204 detects in what state the video camera 100 is shooting. Note that these sensors are preferably capable of detecting in three axial directions (vertical direction, horizontal direction, etc.) in order to detect the attitude of the video camera 100 in detail.

  The input button 208 is one of input interfaces used by the photographer of the video camera 100. Accordingly, the photographer can transmit various requests to the video camera 100, such as the start and end of shooting, and the insertion of a marking in the video being video-recorded.

  The display 209 is provided for viewing images taken by the video camera 100 or for use as a shooting finder or the like. As a result, the photographer can check the photographed image on the spot. In addition, by displaying various information of the video camera 100, it is possible to convey more detailed information such as shooting information and device information to the photographer.

  The speaker 210 is used for audio output when playing back a captured video. In addition, it is possible to convey a warning output from the video camera 100 to the photographer with sound.

  The output I / F 211 is used to output video captured by the video camera 100 to an external device. Specifically, there are a cable interface for connecting to an external device with a cable, a memory card interface for recording a photographed video on a memory card, and the like. As a result, the captured video can be viewed using an external display larger than the display 209 provided in the video camera 100.

  The compression / decompression IC 212 converts the captured video or audio into a predetermined digital data format (encoding process). Specifically, the captured video / audio data is encoded (moving picture experts group), H264, etc., and converted (compressed) into a predetermined data format. Further, at the time of reproducing the captured data, the compression / decompression IC performs data processing for decompressing the video data in a predetermined data format and displaying it on the display 209 or the like.

  The ROM 213 records software programs processed by the CPU 204 and various data for operating the programs.

  The RAM 214 is used as a memory area used when executing a software program processed by the CPU 204. The RAM 214 may be used in common with the compression / decompression IC 212.

  The HDD 215 is used for the purpose of storing video data encoded by the compression / decompression IC 212 and other captured still image data. In addition to this, it is also possible to record reproduction information data to be described later.

  The audio ADC 216 converts the electrical data of audio output from the microphone 217 from analog to digital.

  The microphone 217 converts audio outside the video camera 100 into an electrical signal and outputs it.

  The tripod sensor 218 detects whether the video camera 100 is installed on a tripod. Specifically, it is possible to determine whether or not the video camera 100 is shooting using a tripod by providing a switch in a portion where the tripod is fixed.

  Although the hardware configuration of the video camera 100 has been described as described above, the present invention is not limited to the above configuration. For example, the video ADC 202, the video signal conversion IC 203, and the like can be realized as a single integrated circuit, and a part of a software program executed by the CPU 204 is separately separated by hardware using an FPGA (Field Programmable Gate Array). It is also possible to realize.

  FIG. 3 shows a functional configuration diagram of the video camera 100. The video camera 100 includes, as a functional configuration, a lens unit 300, an imaging unit 301, a video AD conversion unit 302, a signal processing unit 303, a video signal compression unit 304, an imaging control unit 305, a video analysis unit 306, a lens control unit 307, an attitude. Detection unit 308, attribute information generation unit 309, scene analysis unit 310, reproduction information generation unit 311, audio analysis unit 312, audio signal compression unit 313, multiplexing unit 314, storage unit 315, digest reproduction unit 316, video signal expansion unit 317, a video display unit 318, an audio signal expansion unit 319, an audio output unit 320, an audio AD conversion unit 321, a microphone unit 322, and an external input unit 323.

  The lens unit 300 adjusts the focal length of the light incident from the subject and the zoom magnification (magnification magnification of the image). These are performed under the control of the lens control unit 307. The lens unit 300 corresponds to the lens group 200 in FIG.

  The imaging unit 301 converts light transmitted through the lens unit 300 into an electrical signal. The imaging unit 301 outputs data in an arbitrary range on the imaging device under the control of the imaging control unit 305. In addition to video data, chromaticity space information of three primary colors, white coordinates, and gain information of at least two of the three primary colors, color temperature information, Δuv (delta uv), and gamma information of the three primary colors or luminance signals Etc. can also be output. In this case, these pieces of information are output to the attribute information generation unit 309. The imaging unit 301 corresponds to the imaging element 201 in FIG.

  The video AD conversion unit 302 performs analog / digital conversion on the electrical signal from the imaging unit 301 in accordance with predetermined processing contents. The video AD converter 302 corresponds to the video ADC 202 in FIG.

  The signal processing unit 303 converts the digital signal output from the video AD conversion unit 302 into a predetermined video signal format. For example, the video signal conforms to the number of horizontal lines, the number of scanning lines, and the frame rate specified by NTSC (National Television System Committee). The signal processing unit 303 corresponds to the video signal conversion IC 203 in FIG.

  The video signal compression unit 304 performs predetermined coding conversion on the digital video signal, and realizes data amount compression and the like. Specific examples include MPEG2, MPEG4, and H264 encoding methods. The video signal compression unit 304 corresponds to the compression function of the compression / decompression IC 212 in FIG.

  The imaging control unit 305 controls the operation of the imaging unit 301. For the imaging unit 301, the amount of exposure at the time of shooting, the shooting speed, sensitivity, and the like are controlled. These control information are also output to the attribute information generation unit 309. The imaging control unit 305 is one of control algorithms processed by the CPU 204 in FIG.

  The video analysis unit 306 extracts video features from the captured video signal. In the present embodiment, the color signal (for example, color distribution included in the video is detected), white balance, and when the video includes a human face, the video signal is detected by performing face detection. By analyzing, the feature of the video is extracted. The color distribution can be detected by confirming color information included in the data forming the video signal. Further, face detection can be realized by using pattern matching or the like. The video analysis unit 306 is one of algorithms processed by the CPU 204 in FIG.

  The lens control unit 307 controls the operation of the lens unit 300. Lens control includes zoom control, focus control, camera shake correction control, and the like. The lens control unit 307 controls the lens unit 300 and outputs the control information to the attribute information generation unit 309. The lens control unit 307 corresponds to the lens control module 206 in FIG.

  The posture detection unit 308 detects acceleration, angular velocity, elevation angle, depression angle, and the like of the video camera 100. These pieces of information are used for the purpose of detecting the posture of the video camera 100 and its change state. It is desirable that acceleration and angular velocity can be detected in three directions, vertical and horizontal (two directions). The posture detection unit 308 corresponds to the posture detection sensor 207 in FIG.

  The microphone unit 322 converts ambient sounds into electrical signals and outputs them as audio signals. The microphone unit 322 corresponds to the microphone 217 in FIG.

  The audio AD conversion unit 321 converts the analog audio signal output from the microphone unit 322 into digital audio data. The audio AD conversion unit 321 corresponds to the audio ADC 216 in FIG.

  The voice analysis unit 312 extracts a characteristic sound from the voice data. The characteristic sounds here include, for example, a photographer's voice, pronunciation of a specific word, cheers, gunshots, and the like. These sounds can be identified by registering in advance the unique frequencies of these sounds (speech) and using a method such as discrimination based on the comparison result. In addition, other features such as the input level of the sound captured by the microphone are also detected. The voice analysis unit 312 is one of algorithms processed by the CPU 204 in FIG.

  The audio signal compression unit 313 converts the audio data output from the audio AD conversion unit 321 using a predetermined encoding algorithm. There are methods such as MP3 (MPEG Audio Layer-3) and AAC (Advanced Audio Coding) for encoding. The audio signal compression unit 313 is one of the compression functions in the compression / decompression IC 212 in FIG.

  The external input unit 323 outputs various types of information received from the outside during video shooting, such as button input by a photographer, shooting index information received from the outside via communication, and the like. Note that the shooting index information is, for example, an identification number used for identifying each shooting such as a number for identifying a shooting scene or a number indicating the number of shooting times at the time of shooting a movie. The external input unit 323 corresponds to the input button 208, the tripod sensor 218, etc. in FIG.

The attribute information generation unit 309 uses shooting information at the time of video shooting, external input information, and other information as attribute information (metadata). The following information can be considered as an example of attribute information.
-Focal length-Zoom magnification-Exposure-Shooting speed-Sensitivity-Color space information of the three primary colors-White balance-Gain information of at least two of the three primary colors-Color temperature information-Δuv (Delta uv)
・ Gamma information of three primary colors or luminance signals ・ Color distribution ・ Face recognition information ・ Camera posture (acceleration, angular velocity, elevation angle, depression angle, etc.)
-Shooting time (shooting start time, end time)
・ Shooting index information ・ User input ・ Frame rate ・ Sampling frequency

  The attribute information also includes new information that characterizes the video scene calculated from the various information at the time of shooting (information calculated by combining various information at the time of shooting and analyzing them). . For example, it becomes possible for the video camera 100 to calculate camera work such as panning and tilting at the time of photographing from information on the camera posture (acceleration, angular velocity, elevation angle, depression angle, etc.). Further, the focal length and zoom magnification information can be used as attribute information as they are. The attribute information generation unit 309 generates attribute information by extracting and calculating information useful for scene evaluation from various types of information at the time of shooting.

  The scene analysis unit 310 evaluates each scene based on the attribute information generated by the attribute information generation unit 309 and selects a scene to be reproduced based on the result. The scene evaluation and selection method will be described in detail later.

  The reproduction information generation unit 311 generates the scene to be reproduced selected by the scene analysis unit 310 as reproduction information. This point will also be described later.

  The attribute information generation unit 309, the scene analysis unit 310, and the reproduction information generation unit 311 are processed as software by the CPU 204 in FIG.

  The multiplexing unit 314 multiplexes and outputs the encoded video data from the video signal compression unit 304, the encoded audio data from the audio signal compression unit 313, and the reproduction information from the reproduction information generation unit 311. The multiplexing unit 314 may be software executed by the CPU 204 in FIG. 2, or may be processed by the compression / decompression IC 212.

  The storage unit 315 temporarily or long-term holds the encoded video data, the encoded audio data, and the data multiplexed with the reproduction information output from the multiplexing unit 314. The storage unit 315 corresponds to the HDD 215, the RAM 214, or the like in FIG.

  The digest reproduction unit 316 reads the multiplexed data recorded in the storage unit 315, extracts reproduction information from the data, and in accordance with the reproduction information, the video data and audio data are converted by the video signal expansion unit 317 and the audio signal expansion unit 319. Each is decoded and output from the video display unit 318 and the audio output unit 320. The digest playback unit 316 is a software processing algorithm executed by the CPU 204 of FIG.

  With the above-described configuration, it is possible to automatically select a preferable scene extracted based on the attribute information and reproduce only that portion of the video shot by the video camera 100.

  Note that the hardware configuration diagrams and functional diagrams of FIGS. 2 and 3 are one mode for carrying out the present invention, and the present invention is not limited to this. For example, in FIG. 3, the scene analysis unit 310 and the reproduction information generation unit 311 are processed before recording data in the storage unit 315, but the compressed video signal and compressed audio signal are stored in the storage unit 315. Alternatively, the attribute information at the time of shooting may be multiplexed and recorded once, and after the data is read from the storage unit 315, scene analysis, generation of reproduction information, and the like may be performed.

  Also, the correspondence between the hardware configuration of FIG. 2 and the functional configuration of FIG. 3 is not limited to the above description. The above description is given as an embodiment for carrying out the present invention, and the functional configuration and the hardware configuration may be different.

<2. Analysis of captured scene, generation of playback information>
FIG. 4 is a diagram illustrating a configuration of an image captured by the video camera 100. The unit of the video imaged until the photographer gives an instruction to start shooting and gives an instruction to end shooting or pause shooting is “clip”. When the photographer repeats the start of shooting, the end of shooting, or the pause repeatedly, a plurality of clips are generated.

  One clip is composed of one or a plurality of “scenes”. A “scene” is a series of images that are logically connected. A scene can be set arbitrarily. For example, “1 clip” = “1 scene” may be set with a clip as one scene. In addition, a “scene” may be set on the boundary that the screen changes greatly. In this case, when the video analysis unit 306 calculates a motion vector between frames and the magnitude (change) of “motion” is larger than a predetermined value, the “scene” switching portion may be used. The “scene” may be divided based on other shooting information or the like. For example, the “scene” may be divided by a button input from the photographer. In this case, the “scene” in the “clip” can be configured with a clear intention of the photographer.

  A “scene” is composed of one or more “frames”. A “frame” is an individual image constituting a video.

  FIG. 5 shows an example in which the scene analysis unit 310 divides the clip into a plurality of scenes. The scene analysis unit 310 divides the clip based on the attribute information and the like as described above. In FIG. 5, each scene is defined by “start time” and “end time”, but the start and end of the scene may be defined by a frame number or the like.

  FIG. 6 is a diagram showing an example of the relationship between attribute information and evaluation used when the scene analysis unit 310 evaluates each scene. For example, if the clip-in (shooting start part) or clip-out (pre-shooting end part) part is considered as an introduction part or an important part of the picture, the logical meaning of the shot picture is high. It is inferred. In this example, the evaluation of clip-in (A) and clip-out (F) is 100. In addition, with regard to zoom-up (D) and zoom-down (G) as camera work at the time of shooting, the evaluation is set to 30 from the viewpoint of increasing the degree of attention to a specific subject. In this way, the scene analysis unit 310 has a numerical evaluation for each piece of attribute information in advance. In the example of FIG. 6, the higher the evaluation score, the higher the evaluation (preferred). The scene analysis unit 310 evaluates each scene based on the relationship between the attribute information and evaluation in FIG. When a plurality of pieces of attribute information are given to one scene, evaluations (evaluation points) assigned to the respective attribute information may be added. Further, when a plurality of pieces of attribute information are given to one scene, the evaluation (evaluation point) of the attribute having the highest evaluation among the attribute information may be used as the evaluation point of the scene. Furthermore, if various attributes included in the scene are taken into consideration, an average value of evaluation points of a plurality of attributes may be evaluated. Furthermore, when evaluating in more detail, you may evaluate for every flame | frame contained in a scene. Note that the evaluation need not be performed only on a preferable scene. For example, camera shake at the time of shooting may result in a video that is difficult for the viewer of the video to view. Therefore, a scene having such an attribute may be evaluated for a deduction (minus point).

  Note that the relationship between the attribute information and the evaluation in FIG. 6 is not limited to one. For example, the combination data of a plurality of attribute information / evaluation may be switched depending on a mode (for example, landscape shooting, person (portrait) shooting, still life shooting, etc.) in which the video camera 100 is shooting. Alternatively, a plurality of combination data may be provided in advance, and a plurality of data may be combined (addition of respective evaluation values at a certain ratio, etc.) depending on the shooting mode. In this case, the combination data of attribute information and evaluation can be dynamically changed by changing the composition ratio.

  FIG. 7 is a diagram showing a result of assigning an evaluation (priority) to each scene by the scene analysis unit 310. FIG. 7 shows time (scene) on the horizontal axis and evaluation (priority) of each scene on the vertical axis.

  A in the vicinity of time 0 in FIG. 7 has an attribute of “clip in” because it is immediately after the start of shooting. According to FIG. 6, the attribute of “clip-in” has an evaluation (priority) of 100.

  The scene to which the symbol B is attached has an attribute of “extraction of specific sound”. The extraction of the specific voice is obtained by the above-described voice analysis unit 312 or the like. According to FIG. 6, the attribute of “extraction of specific speech” has an evaluation (priority) 70.

  The scene with the symbol C indicates an attribute that means that the photographer shoots by moving the camera body of the video camera 100 such as pan and tilt, and then shoots still. It is conceivable that such a scene can be judged to have a high value as a video for a subject when shooting still. According to FIG. 6, such an attribute has an evaluation (priority) 40.

  The scene with the symbol D is a scene that is photographed by zooming up or down the video camera. In FIG. 6, the zoom up / zoom down has an evaluation (priority) 30. However, it is also possible to change the evaluation value by zooming up and down. For example, by setting the zoom-up higher than the zoom-down, the scene that is shot with the zoom-up, that is, the scene where the enlargement magnification of the video is large (the scene with the subject that is shot with the zoom) is highly evaluated ( (Priority) may be assigned. On the other hand, a relatively low evaluation (priority) can be assigned to a scene with a small video magnification.

  The scene with the symbol E is a scene in which the video camera has taken a picture (camera work) with movement such as panning and tilting. In this case, evaluation (priority) 25 is assigned.

  As described above, the scene analysis unit 310 assigns an evaluation (priority) to each scene. In the example of FIG. 7, the evaluation is assigned in units of scenes. However, the scene analysis unit 310 may perform the above-described evaluation assignments in units of clips or frames.

  The scene analysis unit 310 further extracts only a preferable scene based on the evaluation assigned to each scene. Specifically, only highly evaluated scenes are extracted. In the example of FIG. 7, only the scenes of the # 1 to # 3 portions are extracted. As for the extraction method, extraction can be performed from various viewpoints such as that the total playback time of the extracted scenes is within a predetermined time, or that the evaluation of the scene is more than a certain level.

  The reproduction information generation unit 311 generates reproduction information describing a video reproduction procedure / method according to the scene extracted by the scene analysis unit 310. This reproduction information may be indicated by the start time and end time of a scene to be reproduced as shown in FIG. 8, for example. In this case, if a representative video screen in each scene (a video screen having the highest evaluation in the scene, etc.) is recorded separately, it is also effective for searching the reference screen.

  Note that the management of reproduction information is not limited to the above contents, and other methods may be used. For example, designation by a frame number is also possible. In addition, when the reproduction information generated by the reproduction information generation unit 311 is multiplexed by the multiplexing unit 314 as a TS (Transport Stream) such as MPEG, the time information used at the time of multiplexing (for example, the time of PTS or DTS) It is also possible to record reproduction information using (information) etc. Similarly, in the case of H264, time information at the time of predetermined multiplexing may be used.

  Furthermore, when video data is recorded using a standard such as AVCHD (Advanced Video Code High Definition), which is used as a data recording method of some video cameras, a method of recording reproduction information in a PlayList file or the like is used. It may be used.

  As described above, a digest video (summarized video) can be automatically generated from the captured video.

<3. Handling zoom-up in attribute information>
The scene analysis unit 310 assigns a constant evaluation (priority) (30 in the example of FIG. 6) as attribute information for zooming up. However, the zoom-up function generally includes an “optical zoom” that changes the magnification of an image by controlling the lens of the lens unit 300, and an imaging unit 301 (imaging unit 301 to signal processing unit 303). There are two types of “electronic zoom” that enlarges an image by using a part of the video signal received in (1) as signal processing.

  FIG. 9 shows an example of using optical zoom and electronic zoom. In the range of low magnification (0 to 10 times), only the optical zoom is used, and in the region of 10 times or more, the optical zoom and the electronic zoom are used together. Since the electronic zoom is obtained by enlarging a part of the video received by the imaging unit 301 by signal processing as compared with the optical zoom, a substantial resolution of the video content (a part of the video content enlarged by the electronic zoom is The resolution at the time of image reception by the imaging unit 301 decreases as the enlargement magnification of the video content increases.

  The scene analysis unit 310 sets attribute information for zoom-up and its evaluation in consideration of the characteristics of optical zoom and electronic zoom. As shown in FIG. 9, the optical zoom and the electronic zoom will be described in the case where a zoom method to be used according to the enlargement magnification is determined in advance.

  The attribute information generation unit 309 collects various sensors and control information at the time of shooting. Based on the control information from the lens control unit 307 and the information from the imaging control unit 306 or the signal processing unit 303, the magnification information is obtained by the attribute information generation unit 309 from the operation contents of the optical zoom and the electronic zoom for each scene. To calculate.

  The scene analysis unit 310 determines scene evaluation (priority) based on the magnification information generated by the attribute information generation unit 309. FIG. 10 is a graph showing the relationship between magnification and evaluation. In this case, in the optical zoom range, the evaluation increases as the enlargement ratio increases. However, the evaluation decreases when entering an area with an enlargement magnification of 10 times or more at which the electronic zoom starts to act, and the evaluation becomes 0 at 20 times. This is because, in the electronic zoom, the substantial resolution of the captured image is lowered, and therefore, when the magnification at which the electronic zoom starts to act is reached, the evaluation is lowered according to the enlargement magnification. As a result, it is possible to evaluate the attribute information in consideration of the relationship between the degree of attention of the photographer and the reduction in image quality due to zoom-up. In the example of FIG. 10, compared with a predetermined enlargement magnification (threshold value), if it is larger than the threshold value (when electronic zoom is applied), the evaluation is reduced or not performed. When it is smaller than the threshold value (in the case of engineering zoom only), the attribute information is evaluated.

  Moreover, there exists an example shown in FIG. 11 as another evaluation method. In the example of FIG. 11, the scene analysis unit 310 increases the evaluation according to the enlargement magnification up to an enlargement magnification of 30 times in which the optical zoom and the electronic zoom are used together. After that, the evaluation is 0. This is because, when an image is enlarged by electronic zoom, the image quality deteriorates according to the enlargement magnification, but the image quality level does not decrease significantly in an area where the enlargement magnification of electronic zoom is small. Is to evaluate the scene. When the predetermined enlargement magnification is exceeded, the image quality level is remarkably lowered, and the scene is not evaluated. The case of FIG. 11 is an example when 30 times is set as a boundary of a predetermined magnification.

  It should be noted that enlargement by electronic zoom is performed between 10 and 30 times, and at this time, the resolution of the video acquired by the electronic zoom is set to a specified resolution (for example, display screen resolution, video recording). For example, a technique for converting the resolution to a resolution determined at the time of As a result, it is possible to record and display video captured using the electronic zoom at a higher resolution by complementing the resolution, and solve the problem that the substantial resolution, which is a drawback of the electronic zoom, is reduced. Is possible.

  Resolution complementation includes an individual frame forming an image, an intra-frame process in which processing is performed in each frame, and an inter-frame process in which processing is performed using a plurality of frames. Any method may be used for the above-described resolution complementation.

  As another evaluation method, there is an example shown in FIG. In the example of FIG. 12, the rate of increase in evaluation is changed in accordance with the enlargement magnification between an enlargement area range using only optical zoom and an enlargement area range where electronic zoom is used. In this case, the evaluation method is changed in relation to the effect of enlargement by optical zoom and enlargement by electronic zoom.

  As described above, by changing the evaluation of the scene at the time of zooming up according to the magnification of the image by the optical zoom and the electronic zoom, it becomes possible to extract the scene according to the content of the photographed image.

  In addition, when the attribute information generation unit 309 can acquire information related to the optical zoom and electronic zoom operations directly from the lens control unit 307 and the imaging control unit 305, the scene analysis unit performs scenes based on these video magnification change methods. Can also be evaluated. For example, the evaluation is high in the case of the optical zoom, but may not be evaluated (evaluation 0) when the electronic zoom is operating.

  In this case as well, it is possible to extract a scene according to the content of the video imaged as described above.

<4. About generation of reproduction information based on new evaluation>
The reproduction information generation unit 311 specifies a scene to be reproduced based on the evaluation performed by the scene analysis unit 310 for each scene. FIG. 13 shows an example of scene evaluation according to the above video magnification. FIG. 13 is a diagram illustrating a relationship between a scene, an enlargement magnification (zoom level), and a reproduced video. FIG. 13A shows the relationship between a scene and the magnification of the scene. The scene analysis unit evaluates the scene according to the magnification. In this example, a scene with an image magnification (zoom level) exceeding 30 times is not evaluated (evaluation is 0). In FIG. 13A, scenes 1, 2, and 4 have a magnification of 30 times or less, but since scene 3 is 50 times, scene analysis unit 310 does not evaluate scene 3. FIG. 13 (B)
FIG. 14 is a diagram illustrating scene positions corresponding to scenes 1 to 4 in FIG. FIG. 13B is a diagram illustrating a portion corresponding to a scene having magnification ratio (zoom level) attribute information based on the evaluation of the scene analysis unit 310 and a scene being evaluated. FIG. 13C is a diagram illustrating a video that is reproduced by the reproduction information generated by the reproduction information generation unit 311 in accordance with the evaluation of the scene analysis unit 310.

  Note that the reproduction information generated by the reproduction information generation unit 311 does not have to reproduce only the evaluated scene. For example, an evaluated scene may be reproduced at a normal speed, and a scene that has not been evaluated may be reproduced at a high speed. That is, the playback method may be changed between an evaluated scene (a highly evaluated scene) and a non-evaluated scene (a low evaluated scene).

  As described above, with the configuration described in this embodiment, the video camera 100 can perform suitable scene extraction from the captured video.

(Embodiment 2)
The difference between the present embodiment and the first embodiment is the evaluation of attribute information of optical zoom and electronic zoom. The other points are the same as those in the first embodiment, and thus description thereof is omitted.

  In the present embodiment, a method for evaluating a scene based on an enlargement magnification and a substantial resolution of a video accompanying zoom-up will be described.

  The attribute information generation unit 309 calculates the substantial resolution of the video based on information from the imaging control unit 305 or the video analysis unit 306. For example, when the video signal received by the imaging unit 301 is not subjected to electronic zoom signal processing by the subsequent video AD conversion unit 302, the signal processing unit 303, or the like, the substantial resolution of the video is the resolution of the imaging unit 301. Will be equal. When the electronic zoom is used, since a part of the image received by the imaging unit 301 is enlarged by signal processing, it is necessary to calculate the substantial resolution of the video. A specific example is shown in FIG. An imaging surface 1401 that receives light from the imaging unit 301 receives light that has passed through the lens unit 300 over the entire imaging surface. In the case of the electronic zoom, only the light signal received by the partial area 1402 which is a partial area of the light received by the imaging surface 1401 is enlarged to the size of the imaging surface 1401 by signal processing. Therefore, the substantial resolution of the video in this case can be said to be the resolution of this partial area. In the example of FIG. 14, the resolution is obtained by multiplying the resolution of the imaging surface 1401 by (a / A) × (b / B).

  FIG. 15 shows the relationship between the magnification and the substantial resolution in this case. When the image is magnified only by the optical zoom (up to 10 times), the image has the resolution of the entire imaging surface 1401, but when the enlargement process is performed by the electronic zoom, the substantial resolution of the image is reduced according to the magnification ratio. To do.

  The scene analysis unit 310 evaluates the scene using the substantial resolution of the video as attribute information. FIG. 15 is a graph showing an example of the relationship between the image magnification and the actual resolution. In FIG. 15, a scene is evaluated when the resolution is 60 or more, and is not evaluated when the resolution is less than 60 (evaluation 0). In this case, the scene analysis unit is evaluated as a scene if the magnification of the video is 30 times or less, and is not evaluated if it exceeds 30 times.

  As described above, in the present embodiment, it is possible to automatically perform scene extraction according to the video content by evaluating the scene based on the substantial resolution of the video.

  There are several methods for calculating the substantial resolution of the video. When calculating the enlargement ratio by analyzing the video signal itself, in one pixel constituting the video signal, display contents of the pixel and its neighboring pixels or other pixels within a certain distance from the pixel are displayed. The degree of approximation of display contents is calculated by comparison. By performing this over the entire screen of the video signal, the tendency of the degree of approximation of the entire screen is measured. As a result, it is possible to estimate the substantial resolution of the video.

  As another method, it is detected from the control contents from the imaging unit 301 to the signal processing unit 303 that the enlargement process of the video signal is performed, and the attribute information acquisition unit 309 receives the enlargement rate from the imaging control unit 307 and the like. It is also possible to acquire the actual resolution of the video accurately by acquiring the.

  In the present embodiment, the method of performing the evaluation based on the content of the attribute information and extracting from the highly evaluated scene has been described. However, the present invention is not limited to this. For example, only scenes with a low evaluation may be extracted, and only the scenes with a low evaluation may be excluded during video playback. In this way, the viewer can view the captured video without greatly changing.

(Embodiment 3)
The difference between the present embodiment and the first embodiment is the evaluation of attribute information. The other points are the same as those in the first embodiment, and thus description thereof is omitted.

  In the present embodiment, a method for evaluating a scene based on the amount of change in time unit of the video enlargement ratio at the time of zooming up and zooming down is shown. This is based on the idea that images shot with zoom-up and zoom-down operations performed extremely fast are difficult to see (low evaluation).

  FIG. 16 shows an example of the relationship between the enlargement change speed (zoom speed) and the scene evaluation (priority). In the example of FIG. 16, the highest evaluation (priority) is set when the enlargement magnification is changed at 2 times / second. When changing the enlargement magnification of a video at a higher speed, the evaluation (priority) is reduced as the speed of changing the magnification increases. This is an idea that when a viewer views a video, an enlargement change speed (zoom speed) of the video that is appropriate for the viewer is assumed, and the higher the rating is, the higher the rating is. Thereby, it is possible to automatically extract a scene more preferable for the viewer from the captured video.

  The enlargement change speed is determined based on the control content when the scene analysis unit 310 performs optical zoom control of the lens unit 300 by the lens control unit 307 and the control content of electronic zoom by the imaging unit 300 or the like. Alternatively, it is possible to calculate the enlargement change speed for each unit frame.

  As another method, it is also possible to calculate the enlargement speed from a captured video signal. In this case, the video analysis unit 306 calculates a motion vector between consecutive frames. In this case, when shooting by zooming up and zooming down is performed, a radial motion vector is calculated between successive frames with the center of the shot image as the starting point or the ending point. It is also possible to estimate the zoom-in / zoom-out enlargement change speed based on the magnitude of the motion vector at this time. As described above, the image change speed (zoom speed) can also be calculated by analyzing the video signal by the video analysis unit 306.

  As described above, the scene analysis unit 310 can evaluate (priority) the scene according to the image change speed. Based on this evaluation, the reproduction information generating unit 311 can generate reproduction information as in the first embodiment.

(Embodiment 4)
The difference between the present embodiment and the first embodiment is the evaluation of attribute information of optical zoom and electronic zoom. The other points are the same as those in the first embodiment, and thus description thereof is omitted.

  In this embodiment, an example will be described in which the scene analysis unit 310 performs scene analysis based on input information from the tripod sensor 218 illustrated in FIG.

  In general, when the enlargement magnification (zoom-up) at the time of shooting increases, there is a problem that camera shake at the time of shooting increases. The scene analysis unit 310 can detect camera shake during shooting based on various control information from the lens control unit 307 and the posture detection unit 308. Further, when the lens unit 300 has a camera shake correction function, camera shake can be detected by directly acquiring control information from the lens unit 300.

  These camera shakes can be corrected by controlling the lens unit 300 through the lens control unit 307 or the like. However, since these corrections are generally limited, there is a case where the correction cannot be made completely, particularly when the magnification for photographing is increased. In order to prevent camera shake correction at the time of shooting, it is generally desirable to perform fixed shooting in a stable state using a tripod or the like.

  Therefore, the scene analysis unit 310 performs evaluation for each scene based on the magnification ratio of the video and the attribute information of the tripod sensor 218. Specifically, when the tripod sensor, which is one of the attribute information, means “none”, the scene shown in FIG. 17A is evaluated. When the tripod use of the attribute information means “present”, the evaluation for each scene is performed as shown in FIG.

  The scene analysis unit 310 switches between the evaluation methods shown in FIGS. 17A and 17B depending on whether a tripod is used. In this evaluation method, when shooting with a large magnification, if the photographer holds the camera in his / her hand without a tripod, the video will be disturbed due to camera shake, and the video camera will be fixed to a tripod even if the magnification is large. In this case, the evaluation is performed from the viewpoint that the disturbance of the image is relatively small. Therefore, in the present embodiment, the evaluation for each scene is switched based on the enlargement magnification of the video and the use status of the tripod.

  As a result, it is possible to evaluate a shooting scene in consideration of shooting conditions, and it is possible to automatically extract a more preferable scene by the viewer.

  In the present embodiment, a tripod has been described as an example, but the present invention is not limited to this. A monopod other than a tripod may be used. In other words, if the video camera can identify whether the video camera is shooting in a fixed situation (stable situation) where there is no influence of camera shake, etc., or a situation where there is a risk of camera shake (unstable situation) Good. This is because the video photographing apparatus uses an external input (tripod sensor 218) as a method for identifying this, whereby the scene analysis unit 310 changes the evaluation method of the scene evaluation.

(Embodiment 5)
The difference between the present embodiment and the fourth embodiment is the evaluation of electronic zoom. The other points are the same as those in the fourth embodiment, and thus description thereof is omitted.

  In the present embodiment, a case will be described in which the scene analysis unit 310 performs different scene evaluations when the signal processing unit 303 performs the electronic zoom and when the resolution complementing process is not performed.

  When the signal processing unit 303 increases the resolution of the video captured using the electronic zoom using the resolution complement technology, the resolution is reduced due to the electronic zoom when the enlargement rate of the electronic zoom is relatively small. It becomes possible to compensate in a pseudo manner by resolution complementation technology. Therefore, in this case, as shown in FIG. 17B, the scene analysis unit 310 increases the evaluation of the scene as the enlargement magnification is increased up to a predetermined resolution.

  On the other hand, when the resolution complementing technique is not used, the substantial resolution of the video also decreases as the enlargement magnification is increased by the electronic zoom. Therefore, the evaluation is performed as shown in FIG. In this case, for example, the substantial change in the resolution of the video is as in the case where the resolution complementing process shown in FIG.

  This makes it possible to evaluate a scene depending on whether or not the electronic zoom is accompanied by a resolution complementation technique, and the scene analysis unit 310 can automatically perform a more preferable scene extraction.

  Note that the evaluation methods (particularly evaluation preferences) described in all of the above embodiments are all examples and are not limited to the above evaluation methods. For example, the evaluation method may be changed according to the type of video preferred by the viewer.

  In addition, it is naturally possible to evaluate by combining a plurality of evaluation methods shown in the above embodiments.

  Furthermore, another attribute information (secondary attribute information) may be generated based on the attribute information calculated by the scene analysis unit 310, and the scene may be evaluated based on the attribute information. By generating new attribute information from a plurality of attribute information collected by the attribute information generation unit, it becomes possible to determine a more advanced scene.

  In the above-described embodiment, the video photographing apparatus has been described. However, the present invention is not limited to this. For example, the present invention can be similarly applied to a video editing apparatus that edits a video shot by a video shooting apparatus. In this case, the video editing apparatus can be realized by adopting a method of receiving attribute information for each scene by receiving only video and audio captured from the video imaging apparatus and analyzing only the video and audio. Furthermore, when the video editing apparatus can acquire not only video and audio, but also attribute information and the like, the scene can be evaluated based on the acquired attribute information in the same manner as in the above embodiment.

  The present invention can be realized by a video photographing device for photographing a video or a video editing device for editing a video. In particular, it is possible to automatically extract a scene having a higher value as a video by using attribute information relating to an enlargement ratio changing (zoom) operation at the time of video shooting.

DESCRIPTION OF SYMBOLS 100 Video camera 200 Lens group 201 Image pick-up element 202 Image | video ADC
203 Video signal conversion IC
204 CPU
205 Clock 206 Lens Control Module 207 Posture Detection Sensor 208 Input Button 209 Display 210 Speaker 211 Output I / F
212 Compression / decompression IC
213 ROM
214 RAM
215 HDD
216 Audio ADC
217 Microphone 218 Tripod sensor 300 Lens unit 301 Imaging unit 302 Video AD conversion unit 303 Signal processing unit 304 Video signal compression unit 305 Imaging control unit 306 Video analysis unit 307 Lens control unit 308 Posture detection unit 309 Attribute information generation unit 310 Scene analysis unit 311 Playback Information Generation Unit 312 Audio Analysis Unit 313 Audio Signal Compression Unit 314 Multiplexing Unit 315 Storage Unit 316 Digest Playback Unit 317 Video Signal Expansion Unit 318 Video Display Unit 319 Audio Signal Expansion Unit 320 Audio Output Unit 321 Audio AD Conversion Unit 322 Microphone Part 323 External input part

Claims (7)

A video shooting device for shooting video,
An attribute information generating unit for generating attribute information at the time of video shooting;
Based on at least one of information on the video magnification change method of the video included in the attribute information, information on the video magnification, information on the substantial resolution of the video, and information on the temporal change rate of the video magnification. A scene analysis unit for evaluating the video for each scene;
Based on the evaluation, a reproduction information generation unit that generates reproduction information of the video,
A video shooting device comprising: An imaging unit that converts incident light from an object into an electrical signal;
A lens unit for adjusting light incident on the imaging unit;
Further comprising
The information on the video magnification changing method is information indicating at least one of a magnification changing method by optical adjustment in the lens unit and a magnification changing method by signal processing based on an electrical signal output from the imaging unit. is there,
The video imaging device according to claim 1. The scene analysis unit evaluates the evaluation of the scene based on the information on the video magnification by comparing the video magnification of the video content with a predetermined magnification;
The video imaging device according to claim 1. The scene analysis unit evaluates the evaluation of the scene based on the information regarding the substantial resolution of the video content by comparing the substantial resolution of the video content with a predetermined resolution;
The video imaging device according to claim 1. The scene analysis unit evaluates the evaluation of the scene based on the information about the time change rate of the video magnification by comparing the time change rate of the video content with a predetermined time change rate;
The video imaging device according to claim 1. The scene analysis unit changes the evaluation method according to a shooting situation of the video shooting device.
The video imaging device according to claim 1. The scene analysis unit evaluates the video for each scene based on the information of resolution complement processing.
The video imaging device according to claim 1.