JP2013533669A - Video summary instruction metadata storage - Google Patents

Abstract

  A method for storing a video summary relating to a digital video time series photographed by a digital video photographing apparatus, the step of photographing a digital video time series having a plurality of video frames by a digital video photographing apparatus, and the digital video time series In a processor-accessible memory, identifying one or more key video fragments corresponding to a group of video frames from a stored digital video time series, and combining the key video fragments A step of generating a summary; and a step of specifying a storage location of the video memory in the processor-accessible memory by storing metadata indicating a video frame group corresponding to the video summary in association with the stored digital video time series, and Have To provide a method.

Description

  The present invention relates to digital video processing, and more particularly to a method for generating a digital video summary.

  Although there are many digital photographing devices that can take still images as well as video images, managing digital video content tends to be a tedious task. This is because the thumbnail image of the first frame in the video image is generally used as a visual symbol of the video content. Since it is difficult to estimate the video content sufficiently with a thumbnail image, there are cases where it is necessary to view the entire video image in order to know what event is reflected in the obtained video image. Users prefer to watch a concise summary of the video image rather than watching the entire lengthy video image.

  Digital video also has practical problems related to sharing. In many digital photographing devices, the recording speed is 30 to 60 frames / sec and the spatial resolution is 1920 × 1080 pixels or more, so even if it is compressed, a considerable amount of data is obtained. It cannot be shared.

  It is also possible to create a shortened version (summary) that can be shared more easily by manual operation of video images using video editing software. However, manual video editing is often a lengthy and tedious task and is painful for most users. On the other hand, there is an automatic video summary generation algorithm, that is, an algorithm for analyzing a video image obtained by shooting and generating the summary. However, since the video image needs to be decoded when the summary is generated by analyzing the video image, such an algorithm is very complicated. That is, when such an algorithm is executed on a digital photographing device, a summary corresponding to the video image just obtained by photographing cannot be immediately viewed. This disadvantage is hindered in allowing quick confirmation and sharing of video images obtained by shooting.

International Publication No. 2007/122541 Pamphlet European Patent No. 2063635 US Pat. No. 5,818,439 US Patent Application Publication No. 2007/0182861 US Patent Application Publication No. 2007/0237225 US Pat. No. 3,971,065 US Pat. No. 4,642,678 US Pat. No. 4,775,574 US Pat. No. 5,189,511 US Pat. No. 5,493,335 US Pat. No. 5,562,621 US Pat. No. 5,668,597 US Pat. No. 5,995,095 US Pat. No. 6,192,162 US Pat. No. 6,292,218 US Pat. No. 6,833,865 US Pat. No. 6,934,056 US Pat. No. 7,035,435 US Pat. No. 7,046,731 US Pat. No. 7,403,224 US Pat. No. 7,409,144 US Pat. No. 7,483,618 US Pat. No. 7,420,277 US Patent Application Publication No. 2004/0052505 US Patent Application Publication No. 2005/0191729 US Patent Application Publication No. 2007/0183497 US Patent Application Publication No. 2009/0007202

MA, YF et al., "A Generic Framework of User Attention Model and its Application in Video Summarization," IEEE Transactions on Multimedia, IEEE Service Center, Piscata Way, NJ, US, vol.7, no.5, 1 October 2005 (2005-10-01), pages 907-919, XP01113970, ISSN: 1520-9210, DOI: 10.1109 / TMM.2005, 854410 MA, YF et al., "A User Attention Model for Video Summarization," Proceedings 10th ACM International Conference on Multimedia, Juanles-Pins, France, Dec.1-6, 2002, vol.Conf.10, 1 December 2002 (2002 -12-01), pages 533-543, XP001175055, DOI: 10.1145 / 641107,641116, ISBN: 978-1-58113-620-3 Divakaran, A, et al., "Video Browsing System for Personal Video Recorders," Proceedings of SPIE, The International Society for Optical Engineering SPIE, USA, vol.4861, 1 January 2002 (2002-01-01), pages 22- 25, XP009092815, ISSN: 0277-786X, DOI: 10.117 / 12.470201 Zhang Tong, "Intelligent Keyframe Extraction for Video Printing," Proceedings of SPIE, The International Society for Optical Engineering SPIE, USA, vol.5601, 1 January 2004 (2004-01-01), pages 25-35, XP009093166, ISSN: 0277-786X, DOI: 10.1117 / 12.572474

  Thus, it is possible to provide a system and method capable of generating a video summary in a digital photographing apparatus, and in particular, to provide a technique that requires a short time from the end of video photographing to the generation of a video summary on the digital photographing apparatus. It is hoped that.

Here, the method according to the present invention is a method for storing a video summary relating to a digital video time series shot by a digital video shooting device,
Photographing a digital video time series having a plurality of video frames with a digital video photographing device;
Storing the digital video time series in a processor accessible memory;
Identifying one or more key video fragments corresponding to a group of video frames from a stored digital video time series;
Generating a video summary by combining key video fragments;
Identifying the storage location of the video memory in the processor-accessible memory by storing metadata indicating a video frame group corresponding to the video summary in association with the stored digital video time series; and
Have

  The present invention has the advantage that since the video summary is stored as metadata in the digital video file, it is not necessary to encode the video summary into a separate file. The video summary can be conveniently used by smart video players that can understand the metadata that points to the video summary and ignored by video players that do not.

  Furthermore, in a form in which the video summary is stored in the same digital video file in association with the original digital video time series, the video summary is copied or shared together when the digital video time series is copied or shared. There is an advantage that.

It is a high-level conceptual diagram which shows the component of the video summary production | generation system which concerns on one Embodiment of this invention. 3 is a flowchart of a video summary generation method according to an embodiment of the present invention. 5 is a flowchart of a user feedback-based video summary generation method according to an embodiment of the present invention. 5 is a flowchart of a metadata summary type video summary generation method according to an embodiment of the present invention. 5 is a flowchart of a video summary display method according to an embodiment of the present invention.

  In the following, a preferred embodiment of the present invention will be described in detail with respect to what is generally implemented as a software program. As obvious to those skilled in the present technical field (so-called persons skilled in the art), it is also possible to realize hardware equivalent to such software. Since image manipulation algorithms and systems are well known, the following description focuses on algorithms and systems that make up or work directly with the systems and methods of the present invention. Other examples of such algorithms and systems, and related image signal generation or processing hardware or software can be selected from systems, algorithms, members, and elements known in this technical field. Omitted. With reference to the following description of the system of the present invention, software that is useful for the implementation of the present invention but lacks specific illustrations, suggestions, and descriptions may also be implemented in accordance with the prior art and what is commonly known by those skilled in the art.

  A computer program for executing the method according to the present invention includes a magnetic disk (eg, hard disk, floppy disk), a magnetic recording medium such as a magnetic tape, an optical recording medium such as an optical disk, an optical tape, and a machine-readable barcode. Computer readable recording media including solid state electronic storage devices such as random access memory (RAM), read only memory (ROM), ie, a computer controlling one or more computers to embody the method of the present invention It can be stored in various tangible devices or media that can be used to store programs.

  The present invention also includes a configuration in which the embodiments described in the present application are combined. If there is a description such as “specific example”, the configuration can be adopted in at least one of the embodiments of the present invention. What is referred to as “one embodiment” or “specific example” in one place is not necessarily the same as what is referred to as “one embodiment” or “specific example” in another place. On the other hand, they may not be mutually exclusive unless explicitly stated or obvious to a person skilled in the art. There is no meaning limited to the gist, except for “method”, “methods” and the like. It should be noted that the word “or” is used in a non-exclusive sense unless explicitly stated or contextually.

  Since digital cameras with imaging devices, associated signal acquisition / processing circuits, displays, etc. are well known, the following description focuses on the elements that make up or directly cooperate with the method and apparatus according to the present invention. I will decide. Elements not specifically shown or described in the present application may be selected from those known in the present technical field. Some of the described embodiments take the form of software. With reference to the following description of the system of the present invention, software that is useful for the implementation of the present invention but lacks specific illustrations, suggestions, and descriptions may also be implemented in accordance with the prior art and what is commonly known by those skilled in the art.

  The following description of the digital camera will be easily understood by those skilled in the art. As is obvious, the configuration described below can be modified in various ways for the purpose of cost reduction, function addition, camera performance improvement, and the like.

  FIG. 1 shows a block configuration of a digital photo system having a digital camera 10 capable of video shooting according to an embodiment of the present invention. Preferably, the camera 10 is battery-driven and portable and can be easily held by the user during shooting or image review. A digital image obtained by photographing with the camera 10 is filed and stored in the image memory 30. In the present application, the terms “digital image”, “digital image file” and the like are used to include various digital images and their files regardless of whether they are still images or video images.

  The digital camera 10 of this embodiment has both a video shooting function and a still shooting function. The present invention can also be implemented in the form of a digital video camera that can only shoot video, or in the form of a digital music player (eg, MP3 player), a mobile phone, a GPS receiver, a personal digital assistant (PDA), and other functions. it can.

  The digital camera 10 includes a lens 4 and an adjustable aperture and adjustable shutter 6 associated therewith. In the present embodiment, the lens 4 is a zoom lens, and a zoom / focusing motor driver 8 is provided as means for controlling it. The lens 4 focuses light from a scene (not shown) on the image sensor 14, specifically on a single-chip color CCD image sensor or color CMOS image sensor. The lens 4 is a kind of optical system that can generate an image of a scene on the sensor 14. In addition to this, the present invention can also be implemented in a form using an optical system having a fixed focal length lens and having a variable or fixed focal point.

  The output of the image sensor 14 is converted into digital data by an analog signal processor (ASP) and an analog / digital (A / D) converter 16 and temporarily stored in a buffer memory 18. The image data temporarily stored in the memory 18 is operated by the processor 20 according to an embedded software program in the firmware memory 28, for example, firmware. In the present embodiment, the ROM type memory 28 that permanently holds the software program is used. However, the present invention can be implemented also in a form in which a memory whose contents can be modified, such as a flash EPROM, is used as the memory 28. Can do. In the latter case, an external device is connected via the wired interface 38 or the wireless modem 50, and the software program in the memory 28 is updated through the connection, and the image sensor calibration data, user setting data, etc. are lost even when the camera is turned off. The memory 28 can be used to store no data. Although not shown, in this embodiment, a program memory is attached to the processor 20, and the software program in the memory 28 is copied there and executed by the processor 20.

  As can be understood, the processor 20 has various functions. The realization of these functions can include one or more programmable processors such as a digital signal processor (DSP), one or more custom circuits such as a custom integrated circuit (IC) for digital cameras, programmable processors and custom circuits. Combinations and the like can be used. As can also be understood, some or all of the components shown in FIG. 1 can be connected to the processor 20 via a common data bus. For example, the processor 20, the buffer memory 18, the image memory 30, and the firmware memory 28 may be connected via a common data bus.

  The processed image data is stored in the image memory 30. As is obvious, the memory 30 may take various forms known to those skilled in the art including a removable flash memory card, a built-in flash memory chip, a magnetic memory, an optical memory, and the like. The memory 30 may be configured to have a built-in flash memory chip and a standard interface compatible with a removable flash memory card. As the memory card, a secure digital (SD (registered trademark)) card, a micro SD (registered trademark) card, a compact flash (CF (registered trademark)) card, a multimedia card (MMC), an xD (registered trademark) card, a memory stick Etc. can be used.

  The image sensor 14 is controlled in accordance with various clock signals, such as a low select signal and a pixel select signal, issued to the timing generator 12 so as to synchronize with the operation of the ASP and A / D converter 16. In this example, since the size of the sensor 14 is 12.4 megapixels (4088 × 3040 pixels), still image data of about 4000 × 3000 pixels can be generated. Further, since a color filter array is superimposed on the copying sensor 14 and a pixel array in which pixel groups having different colors are mixed is formed, a color image can be obtained with the sensor 14. The pixel color arrangement in the pixel array can take various patterns. One example is a well-known Bayer as described in Patent Document 6 (inventor: Bayer, name: Color imaging array, the contents of which are incorporated herein by reference), the assignee of which is the assignee of the present application. It is a pixel color arrangement pattern in which a color filter array is formed. As another example, Patent Document 25 (inventor: Compton and Hamilton, filing date: July 28, 2007, name: Image Sensor with Improved Light Sensitivity), assigned to the assignee of the present application. There is a pixel color arrangement pattern described in the above). These are all examples, and there are various patterns that can be used as the pixel color arrangement pattern.

  As can be understood, the image sensor 14, the timing generator 12, and the ASP and A / D converter 16 can be manufactured as separate ICs, or can be manufactured as a single IC following the customary practice of CMOS image sensors. It is also possible. Such an IC can be provided with a part of the functions shown in FIG.

  As a drive mode of the image sensor 14 by the timing generator 12, there is first a first mode in which a time series with motion of low resolution image data is generated. At the time of video image shooting or previewing / compositioning prior to still shooting, HD (registered trademark) resolution image data of 1280 × 720 pixels, VGA resolution image data of 640 × 480 pixels, etc. Sensor image data with a small number of rows is generated under this mode.

  For sensor image data for preview, processing for combining pixel values between adjacent pixels of the same color, processing for ignoring some pixel values, processing for combining pixel values for one color while ignoring pixel values for other colors Etc. can be applied. Patent Document 15 (inventor: Parulski, et al., Name: an electronic camera for initiating capture of still images while previewing motion images) that starts still shooting during video image preview; You may make it perform the process as described in this application).

  As a driving mode of the image sensor 14 by the timing generator 12, there is a second mode in which high-resolution still image data is generated. The resulting final sensor image data is 12 megapixel high resolution image data having a resolution of, for example, 4000 × 3000 pixels. When the scene brightness is high, the pixel values of the pixels in the sensor 14 are used as the final image data. When the scene brightness is low, the signal intensity and thus the sensor 14 internal class is used to increase the ISO (registered trademark) speed. A pixel value binned (combined) between color pixels is used as final image data.

  At that time, the processor 20 controls the zoom / focusing motor driver 8 by issuing a control signal so that the focal length setting becomes appropriate and the light from the scene is focused on the image sensor 14. The exposure level of the sensor 14 is controlled by controlling the f / number and exposure time by the adjustable diaphragm and adjustable shutter 6, by controlling the exposure cycle of the sensor 14 by the timing generator 12, and by the gain setting by the ASP and A / D converter 16. It is controlled by controlling the ISO (registered trademark) speed setting. The processor 20 also controls the flash 2 to illuminate the scene.

  In the first mode described above, Patent Document 12 (inventor: Parulski et al., Name: progressive scan image sensor) that automatically assigns an image onto a progressive scan image sensor (Electronic Camera with Rapid). As described in (Automatic Focus of an Image upon a Progressive Scan Image Sensor), the contents of which are incorporated herein by reference), the lens 4 of the digital camera 10 can be automatically focused by the through-the-lens method. This is because the zoom / focusing motor driver 8 is used to change the focal position of the lens 4 within a range from the closest focal position to the infinity focal position, and sharpness in the central area of the image taken by the image sensor 14. The processing is executed by the processor 20 determining the focal position where the value exhibits a peak, that is, the best focal position. The focal length corresponding to this best focus position can be used afterwards for several purposes including automatic setting of the appropriate scene mode, etc., so it is stored as metadata in the image file along with other lens setting information and camera setting information. Is done.

  The processor 20 displays on the image display 32 the low-resolution color image temporarily stored in the display memory 36 and the created menu. The display 32 is an active matrix color liquid crystal display (LCD), but other types of displays including organic light emitting diode (OLED) displays can be used. A video image output signal from the digital camera 10 is supplied to a video display 46, specifically a flat panel HDTV display, via a video interface 44. In the video shooting mode and the preview mode, the digital image data read from the buffer memory 18 is manipulated by the processor 20, and a series of motion preview images are displayed in color on the image display 32 in principle. In the image display mode, an image is displayed on the image display 32 based on the image data in the digital image file stored in the image memory 30.

  On the image display 32, a graphical user interface that can be operated by a user input via the user controller 34 is displayed. The controller 34 is used for setting various camera modes such as a video shooting mode, a still shooting mode, and an image display mode, and for instructing a start of a still shooting and a video recording. In this embodiment, when the user half-presses the shutter button which is a kind of the controller 34, the first mode is shifted to the above-described first mode, and the still image is previewed. When the user fully presses the shutter button, the second mode is shifted to the still shooting. Is done. The controller 34 is also used for powering on the camera, operating the lens 4 and starting the imaging process. The controller 34 may take the form of a button, a rocker switch, a joystick, a rotary dial, any combination thereof, a touch screen superimposed on the display 32, or the like. You can add several status displays and image displays.

  The camera can also be set to the timer mode using the user controller 34. In the permission of the timer mode, after the user fully presses the shutter button, still shooting by the processor 20 is started after a slight delay time, for example, 10 sec.

  The processor 20 is further connected to an audio codec 22 that receives an audio signal from the microphone 24 and supplies an audio signal to the speaker 26. These members can be used not only for recording / reproduction of audio tracks, but also for recording / reproduction of video image time series, still images, and the like. It is also possible to configure the digital camera 10 as a multi-function device such as a camera-equipped mobile phone, and use the microphone 24 and the speaker 26 as call means.

  In the present embodiment, the speaker 26 is also used as a part of the user interface. Specifically, the operation of the user controller 34 or the designation of a specific mode is notified by various audible signals emitted from the speaker 26. In the present embodiment, voice recognition is further performed using the microphone 24, the audio codec 22, and the processor 20. Therefore, the user can input to the processor 20 by a voice command instead of operating the controller 34. The speaker 26 is also used as a means for notifying the user of the arrival of a telephone call. For this notification, a standard ring tone stored in the firmware memory 28 is used. If a custom ring tone has been downloaded into the image memory 30 via the wireless network 58, the custom ring tone can be used. Further, although not shown, a vibration device may be provided so that the arrival of a telephone call can be notified in a silent mode, that is, a non-audible mode.

  Since the digital camera 10 of the present embodiment includes the accelerometer 27, information regarding camera motion can be obtained therefrom. Preferably, since this accelerometer 27 can detect linear acceleration and angular acceleration for each of the three orthogonal axes, it can acquire information for a total of six dimensions.

  The processor 20 also performs further processing on the image data obtained from the image sensor 14 to convert it into sRGB (registered trademark) image data, compresses it, and compresses the final image file, for example, the well-known Exif (registered trademark). ) -Generate an image file in JPEG format and store the file in the image memory 30.

  The digital camera 10 can be connected via a wired interface 38 to an interface / charger 48 and thus to a desktop or portable computer 40 in the home or office. In this example, the interface 38 that conforms to the well-known USB 2.0 interface specification is used. Therefore, power can be supplied from the interface / charger 48 to the secondary battery group in the camera 10 (not shown) via the interface 38.

  The digital camera 10 can also be connected to the wireless network 58 via the wireless frequency band 52 via the wireless modem 50. The radio interface protocol to which the modem 50 conforms is, for example, a well-known Bluetooth (registered trademark) radio interface, a well-known IEEE 802.11 radio interface, or the like. Images that have arrived at the computer 40 can then be registered with the photo service provider 72, for example, Kodak (registered trademark) EasyShare (registered trademark) gallery, via the Internet 70. The image registered in the provider 72 can also be accessed from other types of devices (not shown).

  In the present invention, the wireless modem 50 is connected to a mobile phone network (not shown) such as a 3GSM (registered trademark) network via a radio frequency link such as a wireless link, and the digital image file in the digital camera 10 is transmitted onto the Internet 70. Can be implemented. The sent digital image file is received by the computer 40 or the photo service provider 72.

  Next, a method according to an embodiment of the present invention will be described with reference to FIG. In this method, first, a digital video time series having a plurality of video frames is photographed in a digital video time series photographing step 210 using a digital video photographing apparatus such as a digital camera 10.

  At the time of digital video time series shooting, a feature amount determination step 220, that is, a step of determining one or more feature amounts through analysis of a video frame group or a part thereof is also executed. As feature amounts obtained by the discrimination, first, there are feature amounts related to video attributes, including color characteristics of video frames and presence / absence of faces in video frames. In addition to the amount of global motion between successive video frames and the amount of local motion between corresponding elements in successive video frames, the feature quantities related to motion can also be obtained. The global motion generally corresponds to the movement of the photographing apparatus, whereas the local motion corresponds to the movement of the subject in the scene. As can be understood by a so-called person skilled in the art, the above-described feature amounts are examples, and other types of feature amounts can be determined through analysis of video frames.

  The feature quantities that can be discriminated in step 220 include audio-related feature quantities. For example, feature quantities such as signal strength in the time domain, signal strength in a specific frequency band, and the like are recorded through the microphone 24 during digital video time series shooting and processed by the audio codec 22 as one or more audio samples. It can be determined by subjecting it to analysis.

  The feature quantity that can be discriminated in step 220 includes a feature quantity related to the apparatus setting. For example, the feature amount is a command given by the user via the user controller 34 for adjusting the position of the zoom lens 4 under the control of the zoom / focusing motor driver 8. This type of feature quantity can be determined by analyzing the settings of the digital video photographing apparatus at the time of digital video time series photographing. Since the magnification of an image changes even with digital zoom, the digital zoom may be separately determined as a feature amount.

  The feature quantity that can be discriminated in step 220 includes a feature quantity that indicates a measurement result related to the movement of the digital camera 10. This type of feature quantity can be determined by analyzing accelerometer data obtained from the accelerometer 27 at the time of digital video time series shooting, for example, as a reinforcement or alternative to motion-related feature quantities derived based on video frame data. Can be used.

  The feature quantity that can be discriminated in step 220 includes an analysis result of data generated by applying the video encoding process in the processor 20. Examples of the analysis target data include motion vector information generated in a motion estimation process executed during the video encoding process. In most video encoding processes, such motion estimation processing is routinely performed as part of the normal processing chain.

  After the feature amount determination for each video frame is completed, the video frame is compressed in the digital video time series compression step 230. The video compression algorithm used is the MPEG standard, H.264. The H.263 standard and other standards that are known to those skilled in the art. The compressed video frame is accommodated by a file format wrapper for a video file provided by a container, specifically, Apple (registered trademark) QuickTime (registered trademark).

  In the compressed version digital video time series storage step 240, the compressed digital video time series is stored in the processor-accessible memory, for example, the image memory 30. The stored compressed digital video time series includes video information and audio information.

  The feature quantity discriminated in the feature quantity discrimination step 220 is stored as, for example, metadata related to the stored compressed digital video time series. The metadata can be stored using, for example, a user data atom defined in the Apple (registered trademark) QuickTime (registered trademark) file format specification.

  Alternatively, the feature quantity determined in the feature quantity determination step 220 may be stored in another file associated with the digital video time series stored in the compressed state.

  Further, when the digital video time series is stored in a compressed state, the feature amount determined in the feature amount determination step 220 may not be stored in the permanent storage memory. In this case, the feature amount is discarded at the end of the video summary generation algorithm.

  After the video shooting operation and the storage of the compressed digital video time series are thus completed, in the key video fragment specifying step 250, the key video fragment representing the digital video time series is specified. That is, by automatically analyzing various feature quantities using a processor, one or a plurality of key video fragments including video frames in several digital video time series can be obtained without expanding the stored compressed digital video time series. Identified. In principle, a key video fragment is a collection of multiple video frames that are consecutive in the digital video time series. Therefore, the key video fragment is specified individually by a combination of the start frame number and end frame number or key video fragment length. can do.

  As a technique for specifying the key video fragment in the video time series based on the related feature values, for example, various techniques known in this technical field are used. One example is a technique for generating individual key video fragments by first identifying several key video frames and selecting a portion of the video time series to encompass the individual key video frames. As described in Patent Document 26 (inventor: Luo et al., Which is incorporated herein by reference), a technique for selecting a key video frame based on an in-video motion calculated by digital motion estimation, or Patent Document 4 (Invention). (Luo et al., Which is incorporated herein by reference), a method for selecting a key video frame based on the characteristics of motion in video obtained as data from an accelerometer associated with the video photographing apparatus is described in this book. In carrying out the invention, the present invention can be used to specify a key video fragment based on a feature amount discrimination result.

  If the feature quantity is stored in another file associated with the stored compressed digital video time series, the feature quantity to be analyzed in the key video fragment specifying step 250 is read from the file.

  When the feature quantity is stored as metadata related to the stored compressed digital video time series, the video frame group related to the stored compressed digital video time series is expanded in the key video fragment specifying step 250. Instead, the feature quantity to be analyzed is extracted from the compressed digital video time series file. It should be noted that the extraction of the feature quantity stored as metadata relating to a stored compressed digital video time series should not be regarded as an extension of the compressed digital video time series. Rather than decompressing the stored compressed digital video time series, rather, the data in the compressed bit stream used when reconstructing a series of video frames according to the compressed digital video time series, e.g. Decoding video information and header information.

  One advantage of the present invention is that key video fragments can be identified without decompressing the stored compressed digital video time series. For this reason, the identification of the key video fragment and the generation of the video summary that follows the key video fragment are completed without much time from the end of the shooting operation. In conventional techniques where decompression is used as a means of extracting individual video frames from a stored compressed digital video time series, the length of time required to perform the decompression task has become a problem.

  The identification of the key video fragment can be performed entirely depending on the result of the analysis on the feature amount determined at the time of shooting the digital video time series, or the information extracted from the stored compressed digital video time series. Can be executed in combination with the feature value obtained by the analysis. In the latter case, the information contained in the stored compressed digital video time series may have to be partially decoded.

  For example, a case where a key video fragment is specified by using audio information extracted from a stored compressed digital video time series together with a feature amount obtained by analysis. However, the audio information generally has a small ratio to the entire compressed digital video file, and can be extracted more quickly than the expansion of the pixel data constituting the video frame. Also, if audio feature-related feature values could not be generated at the time of shooting, or there was no period available for processing of audio video-related video processing and encoding, as well as audio attribute-related and other feature values. Even if the audio information is extracted from the stored compressed digital video time series, the audio information can be used for key video fragment identification. This represents a compromise between speed and performance. That is, by using the audio information, the performance of the key video fragment specifying step 250 can be enhanced in exchange for the overall increase in the processing time required for specifying the key video fragment.

  The same applies to the case where the key video fragment is specified by using the video information extracted from the stored compressed digital video time series together with the feature amount obtained by the analysis. If the entire video time series is expanded, the merit of discriminating the video attribute-related feature amount at the time of shooting is largely negated, so that the video frames constituting the stored compressed digital video time series are decoded. It is desirable to make the number as small as possible. As can be understood by a so-called person skilled in the art, if a frame is encoded independently of other frames, the frame can be efficiently decoded from the compressed digital video time series. This also represents a compromise between speed and performance. That is, by using the video information, the performance of the key video fragment specifying step 250 can be enhanced in exchange for the overall increase in the processing time required for specifying the key video fragment.

  In executing the key video fragment identification step 250, user input via the user controller 34 may be received and various attributes of the video summary may be controlled accordingly. For example, the user receives designation from the user regarding the length of the video summary, the minimum time length of individual key video fragments, the total number of key video fragments, and the like.

  In the key video fragment identification step 250, pending US patent application Ser. No. 12 / 786,471, assigned to the assignee of the present application (inventor: Dever, name: Method for Determining Key Video Frames) May be used. This method derives a temporal change in importance by analyzing a digital video time series, generates a time distortion expression of the digital video time series based on the result, and converts the time distortion expression into a plurality of distortion isometric lengths. A key video frame is selected for each long period of time such as distortion by analyzing the video frames within a long period of time such as each distortion. The degree of importance is information reflecting characteristics such as global motion and local motion provided in the digital video time series, in particular information derived based on the result of feature amount discrimination in the feature amount discrimination step 220. The time distortion expression is a weighted expression such that one of the video frames in the digital video time series is lengthened and the other is shortened. The key video frame selection is preferably performed by analyzing a feature amount determined in the feature amount determination step 220 and related to a video frame group within a long period, such as a corresponding distortion. For example, a video frame that satisfies the condition that the degree of local motion in the central area is short after the zoom-in operation is completed is selected as a key frame.

  After discriminating key video frames, individual key video fragments may be specified by selecting a group of video frames before and after each key video frame. Specifically, a key video fragment may be specified by selecting a video frame group belonging to a total period of 4 seconds from 2 seconds before and 2 seconds after the key video frame.

  The key video fragments can also be identified in such a manner that the key video frames are ranked and only the group of key video frames associated with the highest rank key video frame is generated. Ranking for the key video frames can be performed by determining the camera movement pattern by analyzing the digital video time series and determining the rank of the key video frame based on the camera movement pattern. For example, by analyzing the features related to the global motion among the features determined in the feature determination step 220, a global motion trajectory indicating the transition of the camera fixed region in the digital video time series can be obtained. Areas where the camera is fixed at a higher ratio than the total length of video shooting, that is, video frames corresponding to the high-frequency fixed areas are given higher ranks. This ranking process may be performed as an iterative process of selecting the highest rank key video frame for each round, and in each round of the process, a key video frame representing the same fixed area as that of the selected key video frame. It is sufficient to preferentially select a key video frame representing a fixed area other than that. After ranking the key video frames, the key video fragments may be identified so that the highest rank key video frame is included.

  As described above, a key video fragment can be identified by selecting a group of video frames from before to after each key video frame. Alternatively, the key video fragment can be identified by setting conditions relating to the total time length of the video summary and the minimum allowable time length of the key video fragment, and selecting to satisfy these conditions. It is possible to impose further conditions, for example that the speaker's voice is not interrupted at the beginning or end of the key video fragment.

  After identifying the key video fragment, a video summary generation step 260 generates a video summary. That is, a video summary is generated by joining key video fragments together. In this embodiment, the key video fragments are combined in an order that matches the appearance order in the digital video time series.

  In the video summary representation storage step 270, the video summary representation is stored in the processor accessible memory. The video summary expression stored in the processor-accessible memory is, for example, frame indication metadata indicating a digital video time-series video frame group composing the video summary. The frame indication metadata is data that can be stored in association with the stored compressed digital video time series, and indicates the start and end frames of the key video fragments that compose the video summary. In this form, the amount of physical memory required for storing the video summary expression can be reduced to that required for storing the frame instruction metadata.

  The video summary representation stored in the processor-accessible memory is alternatively a fused video time series corresponding to the video summary. This can be newly generated by extracting a video frame group composing the specified key video fragment from the stored compressed digital video time series and fusing the video frames together. In that case, it is sometimes necessary to partially decode the compressed version of the digital video time series or to compress the fused version of the video time series to generate a compressed version of the video time series. In order to include not only video data but also audio data in the fused video time series, it is also necessary to extract the audio data from the compressed digital video time series.

  When the generated compressed video summary is stored in the processor-accessible memory, the compressed video summary is stored in a digital video file different from that of the corresponding compressed digital video time series. Can do. In this form, the digital video file becomes a video summary expression, so that the video summary expression can be viewed or shared independently of the original compressed digital video time series. It is desirable that the format of the digital video file as a video summary expression be a format that can be played back by a standard video player.

  When generating a compressed video summary by compressing the extracted video frames, for example, by re-sampling the video frames, the spatial resolution is changed from the previous value to a new value, and a new It is desirable to generate a compressed video summary by compressing a video frame group related to the spatial resolution. This resampling is useful when sharing videos taken at high spatial resolution. This is because a compressed version video summary with a small number of video frames included and a low spatial resolution of the video frames, that is, a video summary that is smaller in size and easier to share than a compressed version digital video time series can be obtained. A low spatial resolution video summary can be generated without stretching the entire high resolution video summary. Only the video frames needed to generate the video summary need to be decompressed.

  Similarly, when the extracted video frames are compressed to generate a compressed video summary, the video frames that compose the video summary are resampled along the time axis, and the time resolution is changed from the previous value to a new value. It can also be changed to a value.

  FIG. 3 shows a video summary generation method according to another embodiment of the present invention, in which the user can preview the generated video summary and request the user to update the video summary generation result through setting adjustment. The method is shown. Digital video time series shooting step 210, feature quantity determination step 220, digital video time series compression step 230, compressed digital video time series storage step 240, key video fragment identification step 250, video summary generation step 260 and video summary expression storage Steps 270 are executed in the same manner as described with reference to FIG. 2, but in this embodiment, the video summary generated in the video summary generation step 260 is displayed in the video summary display step 262 prior to its storage. The user's approval / disapproval regarding the video summary is confirmed in the user approval / disapproval determination step 264. If the user indicates satisfaction, the process proceeds to the video summary expression storage step 270 and the same processing as in FIG. 2 is executed. If the user indicates dissatisfaction, one or more settings are made. A new video summary is generated after user adjustment in step 266. User settings can be adjusted via the user controller 34. The target can include setting of parameter groups such as the video summary time length, the minimum time length of key video fragments, the number of key video fragments included in the video summary, and the like. After the setting adjustment by the user is completed, a video summary based on the new user setting is newly generated in the key video fragment specifying step 250 and the video summary generating step 260. As will be appreciated by those skilled in the art, the user can repeatedly perform video summary previews and setting adjustments until a satisfactory video summary is obtained.

  FIG. 4 shows a video summary generation method according to another embodiment of the present invention in which data specifying a video summary is stored as metadata related to a stored digital video time series. First, in the digital video time series photographing step 410, a digital video time series having a plurality of video frames is photographed by the digital video photographing apparatus. In the digital video time series storage step 420, the digital video time series is stored in the processor-accessible memory. The feature amount determined in the feature amount determination step 220 in FIG. 2 may or may not be stored together with the digital video time series.

  Next, in a key video fragment specifying step 430, one or a plurality of key video fragments composed of one or a plurality of video frames included in the stored digital video time series are specified. This step 430 is executed based on the stored feature amount, for example, following the key video fragment specifying step 250 described with reference to FIG. The step 430 may include a process of directly analyzing a frame group constituting a stored digital video time series according to a video analysis algorithm. In that case, performing step 430 would require decompressing the digital video time series according to the analytical needs. Various methods known in this technical field can be used to identify the key video fragment, including those described above with reference to FIG.

  In the video summary generation step 440, as in the video summary generation step 260 in FIG. 2, a video summary is generated by combining the identified key video fragments. In the video summary instruction metadata storage step 450, the metadata indicating the video frame group corresponding to the video summary is stored in association with the stored digital video time series, whereby the location of the video summary in the processor accessible memory is determined. Identified.

  The key video fragment specifying step 430, the video summary generation step 440, and the video summary instruction metadata storage step 450 are different from those in which the digital video time series shooting step 410 and the digital video time series storage step 420 are executed. It can also be executed on the processor 20 or above. For example, the image memory 30 in which the digital video time series is stored in step 420 is a removable memory card, the digital video time series is carried to another device using the memory card, and the steps 430 to 450 are executed by the device. It may be. For example, the digital video time series is carried to another digital video photographing apparatus similar to that shown in FIG. 1, or the digital video time series is loaded into a system such as another apparatus such as the computer 40 shown in FIG. 1 or a video editing system. However, a video summary may be generated by processing there.

  The key video fragment specifying step 430 includes a process of extracting one or a plurality of video frames from the stored digital video time series, and a process of discriminating a feature amount relating to the video frames by analyzing the extracted video frames. Can be included. The step 430 further includes a process of extracting one or a plurality of audio samples from the stored digital video time series, and a process of discriminating a feature amount related to the audio samples by analyzing the audio samples. It can also be. Analysis of these feature quantities helps to identify key video fragments.

  The video summary indication metadata storage step 450 may be configured to store frame indication metadata indicating a group of video frames in the digital video time series constituting the video summary. Storing the video summary as frame indication metadata according to this guideline is beneficial in that the amount of physical memory required to store the video summary remains at the minimum amount required to store the frame indication metadata. . Preferably, the frame indication metadata can be stored in association with the stored compressed digital video time series. For example, if a video summary expression indicating a group of start and end frames of a frame constituting a key video fragment in a video summary is stored as metadata in a file related to a stored compressed digital video time series. Good. The frame instruction metadata may be stored in a file different from the stored compressed digital video time series file.

  In the video summary instruction metadata storage step 450, metadata indicating the audio sample composing the key video fragment among the audio samples composing the video summary may be stored.

  In the video summary instruction metadata storage step 450, metadata indicating the video transition effect applied to the boundary between the identified key video fragments may be stored. Examples of video transition effects include fading from one key video fragment to the next key video fragment (cross dissolve effect), fading from one key video fragment to the next key video fragment via a white screen or black screen, etc. Can be specified. Other transition effects, for example, special effects such as a cross wipe effect, a circle expansion / contraction effect, a horizontal / vertical blind effect, and a checkboard transition effect can also be designated. As will be understood by those skilled in the art, these are merely examples, and various other transition effects can be used in the practice of the present invention. By using the video transition effect, it is possible to obtain a video summary with high overall visual quality without sudden transitions between segments in the digital video time series.

  In the video summary instruction metadata storage step 450, metadata indicating the audio transition effect applied to the boundary between the key video fragments may also be stored. As the audio transition effect, for example, fading from sound to silence or fading from silence to sound can be specified. By using such an audio transition effect, it is possible to obtain a video summary with a high overall auditory quality without sudden transitions between audio segments in a digital video time series.

  FIG. 5 shows a flow of a video summary display method executed on the video playback system according to an embodiment of the present invention. The illustrated method is suitable for displaying a video summary whose location is specified by metadata, not a video summary stored as a digital video file that can be displayed directly.

  First, in the data reading step 510, data related to the stored video summary, specifically, data indicating the video frame composing the stored digital video time series is read. For example, data indicating the video frame that composes the video summary indicates which metadata is stored in the same digital video file where the digital video time series is stored, particularly the video frame that composes the video summary. Extracted from metadata. Alternatively, it is extracted from another file associated with the stored digital video time series.

  Next, in a video frame extraction step 520, the video frames that make up the key video fragment and thus the video summary are extracted from the stored digital video time series. In the case where the digital video time series is stored in a compressed state following the usual practice, in step 520, the digital video time series is also decompressed.

  In the video summary generation step 530, a video summary is generated based on the video frame extraction result for each key video fragment. The simplest implementation of this step 530 is to simply arrange the extracted video frames to produce an uninterrupted video clip.

  After the video summary is generated, in the video summary display step 540, the video summary is displayed on the screen of the soft copy display. As the soft copy display, for example, a browsing screen on a digital video photographing device, a display connected to a computer, or a screen of a television receiver can be used.

  The data reading step 510 may be configured to obtain an indicator indicating an audio sample group corresponding to the video summary. In this case, in the video frame extraction step 520, it is only necessary to extract the audio sample group composing the stored digital video time series from the audio sample group suitable for the video summary.

  In the data reading step 510, an indicator indicating the video transition effect used in the video summary, an indicator indicating the audio transition effect used in the video summary, and the like may be obtained. If an indicator indicating the video transition effect is obtained in the data reading step 510, the desired transition is obtained by modifying the extracted video frame group in accordance with the transition effect related to the indicator in the video summary generation step 530. A video summary edited to present can be generated. Therefore, for example, when data including an instruction to apply a black transition fading video transition effect to 15 frames located at the end of the key video fragment in the video summary is read, Are extracted from the stored digital video time series and the data for those frames is modified according to a slow blackening fading scheme. In addition, the 15 frames located at the beginning of the next key video fragment are extracted, and the data related to those frames are corrected according to the slow normal video return fading mode.

  When the soft copy display has one or more speakers for audio output, it is also possible to apply an audio transition effect prior to audio output from the speakers. For example, when data with an instruction to apply a silence fading audio transition effect is read to 8000 audio samples located at the end of a key video fragment, the effect should be applied first. A group of audio samples is extracted from the stored digital video time series, and the data associated with those audio samples is modified according to a slow silence fading scheme. In addition, 8000 audio samples located at the beginning of the next key video fragment are extracted, and the data related to those audio samples is modified according to the slow normal audio return fading mode.

  In the present invention, as a video summary generation method, metadata indicating a video frame group or an audio sample group included in the video summary, or metadata indicating a video or audio transition effect applied thereto is stored in a digital video time series. A method such as storing in the same digital video file as before can be used. In the present invention, as the video summary display system, since the video summary is composed, it is necessary to extract from the stored digital video time series, the video frame and the audio sample, the video display on the display and the speaker. A system may be used that reads and identifies the corresponding metadata to identify video and audio transition effects to be applied to the data prior to audio output. Such a method and system has the advantage that the video summary can be displayed without storing the video summary in a separate video file.

  The device configuration of the video display may be devised so that the user can select whether to watch the original video or the video summary. For example, a plurality of playback buttons may be provided on a digital video camcorder, divided into those for original video time-series playback and those for video summary playback. The video summary may be played in response to the operation of the fast forward button. In other words, video summary reproduction can be used as a high-speed vertical moving means in the video time series in place of the conventional fast-forward function. Compared to traditional fast-forwarding, which is just time sampling of frames within a video time series, video summary playback is more useful in that it can present to the user only the part of the video time series that the user would want to watch It is a thing.

  The video playback system described with reference to FIG. 5 extracts metadata related to a video summary from a stored digital video time series, and processes the metadata to obtain a video summary version of the digital video time series. It is a smart video player in the sense that it can be determined whether or not to display. In addition, the user can be presented with options regarding whether he wants to watch the entire digital video timeline or to watch the video summary version. On the other hand, a general video player cannot recognize the relationship between the video summary and the corresponding metadata. However, even a general video player can read the stored original digital video time series for display. That is, in a general video player, the metadata accompanying the stored digital video time series is ignored. The entire original digital video time series can be played back.

  As shown in FIGS. 1 and 2, the present invention has an embodiment of a digital video shooting system (eg, digital camera 10) that acquires a digital video time series by shooting and generates a corresponding video summary. . The system of the present embodiment includes an image sensor 14 for acquiring a video frame, an optical system (for example, a lens 4 and an adjustable aperture and adjustable shutter 6 associated therewith) that causes an image of a scene to appear on the image sensor, a video A processor 20 that generates a summary and stores it in a processor-accessible memory, an image display 32 for browsing digital video time series, and a soft copy display that represents any of the stored digital video time series and video summary expressions. Means (for example, a user controller 34) for allowing the user to select whether to display the image on the upper side. This system may take the form of, for example, a digital video camera, a digital camera having both a still shooting mode and a video shooting mode, a laptop with a webcam, or a desktop computer.

  The processor 20 in the digital video shooting system executes software responsible for the steps of the method shown in FIG. Specifically, in response to an instruction from the user, the processor 20 executes digital video time-series shooting having a plurality of video frames (digital video time-series shooting step 210), and in parallel with the shooting, One or more feature quantities are discriminated by automatically analyzing part or all of the video frames in the series (feature quantity discrimination step 220), and the digital video time series is compressed (digital video time series compression step). 230), and the compressed digital video time series obtained thereby is stored in the processor-accessible memory (compressed digital video time series storage step 240), and the feature amount is not expanded without decompressing the stored compressed digital video time series. A key consisting of video frames within a digital video time series by automatically analyzing One or a plurality of video fragments are identified (key video fragment identification step 250), a video summary is generated by combining the key video fragments (video summary generation step 260), and the video summary representation is processed by the processor. Store in accessible memory (video summary expression storage step 270).

  The step of automatically analyzing a part or all of the audio sample group in the digital video time series and determining one or a plurality of feature amounts is performed in parallel with the digital video time series shooting in the processor 20 in the digital video shooting system. May be executed.

  The processor 20 in the digital video shooting system executes the step of storing the determined feature quantity in the processor-accessible memory, particularly the step of storing the feature quantity as metadata relating to the stored compressed digital video time series. You may do it.

  Storing the determined feature quantity in the processor-accessible memory, in particular, storing the feature quantity in another file associated with the stored compressed digital video time series. The processor 20 may be executed.

  The step of automatically analyzing the feature quantity and user input to identify one or more key video fragments may be executed by the processor 20 in the digital video shooting system. User input can be obtained via the user controller 34. As user input, conditions relating to the time length of the video summary, the minimum time length of the key video fragment in the video summary, the number of key video fragments in the video summary, and the like can be used.

  The step of storing the video summary representation in the processor accessible memory may be performed by the processor 20 in the digital video shooting system. For example, it is a step of generating metadata indicating a video summary composing among video frames in a digital video time series, and storing the metadata in association with the stored compressed digital video time series.

  The video summary is displayed on the screen of the soft copy image display 32 or the external video display 46 in accordance with a user operation on the user controller 34 provided in the digital video shooting system. In the case where the video summary is stored in a form in which the metadata that composes the video summary among the video frames in the digital video time series is indicated by metadata, the video summary is extracted and displayed in the manner shown in FIG. Specifically, the digital video shooting system behaves as a smart video player and extracts video and audio data appropriate for the metadata.

  It is also possible to configure the user interface so that the user can view the video summary and determine whether it is possible. Since the user may not like the video summary, it is desirable to provide a corresponding means in the digital video shooting system by using the user controller 34 so that the user himself can adjust the settings relating to the video summary. User settings that can be adjusted include the video summary time length, the minimum time length of key video fragments in the video summary, the number of key video fragments in the video summary, and the like. It is also possible to configure the system so as to automatically generate a plurality of video summary candidates by changing the processing contents and setting used in the key video fragment specifying step 250. In this configuration, since the user can select an appropriate video summary from a plurality of candidates, the probability that a satisfactory video summary is not generated can be suppressed, and the effect of the present invention can be enhanced as a whole.

  The above digital video shooting system quickly generates and displays a group of video summaries without decompressing the stored compressed digital video time series and without generating a new file by encoding the video summary. There is an advantage that you can.

  In a configuration where the video summary is stored in a separate file, a smart video display that can extract the video summary by sharing the video summary on the network and interpreting the metadata in the compressed digital video time series is implemented. It is possible to more suitably perform video summary reference with a device that has not been performed. In this case, it is desirable to store the video summary expression generated by the processor 20 in the digital video shooting system in a digital video file in a format that can be played back by a general video player. The video summary may be configured to always form another file, or the user operates the user controller 34 to specify the storage form of the video summary, for example, when the user operates the share button on the user interface. The summary may be stored in a separate file.

  In a digital video shooting system of a type in which the video summary is stored in a separate file, the processor 20 may execute the step of converting the video summary representation into a digital video file and storing it in a processor accessible memory. For example, a video frame group corresponding to a video summary is extracted by expanding a part or the whole of a stored compressed digital video time series, and a compressed video summary is generated by compressing the video frame group. The processor 20 executes software for storing the compressed video summary in the processor-accessible memory. Further, a step of extracting a part or the whole of the stored compressed digital video time series to extract an audio sample group corresponding to the video summary, compressing the audio sample group and incorporating it into the compressed video summary. May be executed by the processor 20 in the digital video shooting system.

  Decreasing the spatial or temporal resolution of the digital video time series from its original value prior to sharing is beneficial when sharing video files on a computer network. A video frame group having a spatial or temporal resolution different from the original value is obtained by re-sampling the extracted video frame group by the processor 20 in the digital video shooting system. One of the advantages of the present invention is that it can thus generate a low-resolution video summary without decompressing the entire original compressed digital video time series. Since only compressed digital video data related to the video summary generation needs to be decompressed, it is possible to perform higher-speed code conversion from the original compressed digital video time series to the video summary.

  When sharing a video file on a network, the size of the compressed digital video time series can be reduced by increasing the aggressiveness of compression. The extracted video frame group is compressed by the processor 20 in the digital video shooting system in accordance with a compression setting more aggressive than that of the stored compressed digital video time series, so that a more strongly compressed video summary is obtained. Can be generated. One advantage of the present invention is that it is possible to generate a more strongly compressed video summary without decompressing the entire original compressed digital video time series.

  Constructing a digital video shooting system in the form of a digital video camera having a user interface in addition to the image sensor 14, optical system (lens 4), processor 20 and soft copy display (image display 32), in particular a stored compressed version It is also possible to have a configuration that presents the user with options on whether to display the digital video time series on the softcopy display or to display the stored video summary. In this case, immediately after shooting a video with the digital video camera, the user can view a summary version of the video, ask the video summary to make corrections at will, or save the video summary in a separate file. Can be saved and shared.

  The digital video camera can be connected to an external soft copy display to display a stored compressed digital video time series or stored video summary on it for viewing by the user.

  Providing a soft copy display as a component of an independent video viewing system accessible to the stored digital video time series and video summary, and the stored digital video time series and stored video summary It is also possible to provide a user interface on the digital camera that presents options to the user regarding which to display on the softcopy display.

  The processor 20 in the digital video shooting system may be configured to generate the video summary while applying the video transition effect between the key video fragments. The result of the video transition effect can be calculated at the time of display, which is advantageous in the configuration in which the location of the video summary is specified by the metadata relating to the stored compressed digital video time series.

  The processor 20 in the digital video shooting system may analyze the key video fragment and automatically select some of the video transition effects. Depending on the video content, it is up to the automatic analysis of key video fragments whether the whitening fading transition effect is appropriate, the black fading transition effect is appropriate, or the other video transition effect is appropriate. Utilizing the information obtained is useful in identifying video transition effects between key video fragments that provide the optimal visual effect.

  Audio transition effects can be handled similarly. What is necessary is just to make the processor 20 in a digital video imaging system generate | occur | produce a video summary, applying the audio transition effect between key video fragments. The result of the audio transition effect can be calculated at the time of display.

  A form in which the processor 20 in the digital video photographing system executes a modified version of the key video fragment specifying step 250, for example, a feature amount is automatically analyzed and the result is used in combination with information in the stored compressed digital video time series. It is also possible to implement the present invention in a form that specifies fragments. At the expense of speed, this form allows identification of key video fragments that make up a video summary based on more information. In many cases, the processor 20 cannot derive a desired feature amount because there is no time margin at the time of shooting. Even in such a case, it is possible to extract information useful for deriving the key video fragment by partially decompressing the stored compressed digital video time series. For example, audio information in the stored compressed digital video time series can be extracted, and video information in the stored compressed digital video time series can be extracted.

  The present invention also has a feature that manual trimming can be supported. Manual trimming is an editing function provided in many digital video shooting apparatuses, and by using this, the user can trim the start point and end point of the video arbitrarily based on the review result of the shot video. In accordance with the present invention, a video summary composed of a single key video fragment may be generated, in which case the start and end points of the video summary are presented as recommendations for manual trimming.

  It should be noted that the procedure of discriminating feature amounts at the time of shooting and using the feature amount discrimination results after shooting is also suitable for uses other than video summary generation. It is recognized that configurations employing other methods including key video frame discrimination based on feature amounts are also within the technical scope of the present invention.

  Many digital video compression algorithms are used for encoding by dividing a digital video time series into a plurality of video frame groups. Each group of video frames consists of one independently encoded video frame (I frame) and several predictive encoded video frames (P frames) that require information about one or more other video frames during decoding. ). Among them, one I frame precedes, followed by several P frames generated by prediction based on that I frame. It is an I frame in the latter that appears when a certain video frame group ends and the next video frame group starts. In such a compression method, an I frame is used as an access point that reaches the inside of a compressed digital video time series, and a frame group starting from the I frame can be extracted. Specifically, the entire encoding target video frame group is extracted from the compressed digital video time series only by decoding header information indicating the position and number of compressed bytes constituting the encoding target video frame group. The code can be converted into a video summary. Therefore, when generating a video summary, it is useful to obtain that the first frame of each key video fragment is an I frame, or to obtain no fractional number in the number of target video frames included in the fragment. That is. By imposing these constraints, it is possible to generate a video summary without much decompressing the original compressed digital video time series.

  Computer program products for executing the method according to the present invention include magnetic recording media such as magnetic disks (eg floppy disks), magnetic tapes, optical recording media such as optical disks, optical tapes, machine-readable barcodes, RAM, ROM Various tangibles that can be used to store one or more recording media including a solid-state electronic storage device such as a computer program for controlling one or more computers so that the method of the present invention is embodied. It can be stored in a device or medium.

  1 flash, 4 lens, 6 adjustable aperture and adjustable shutter, 8 zoom / focus motor driver, 10 digital camera, 12 timing generator, 14 image sensor, 16 ASP and A / D converter, 18 buffer memory, 20 processor , 22 Audio codec, 24 Microphone, 25 Pressure sensor, 26 Speaker, 27 Accelerometer, 28 Firmware memory, 30 Image memory, 32 Image display, 34 User controller, 36 Display memory, 38 Wired interface, 40 Computer, 42 Tilt sensor 44 Video interface 46 Video display 48 Interface / charger 50 Wireless modem 52 Radio frequency band 58 Wireless network 70 Internet, 72 Photo service provider, 210, 410 Digital video time series shooting step, 220 Feature value determination step, 230 Digital video time series compression step, 240 Compressed digital video time series storage step, 250, 430 Key video fragment identification step 260, 440, 530 Video summary generation step, 262, 540 Video summary display step, 264 User acceptance / denial determination step, 266 User setting adjustment step, 270 Video summary expression storage step, 420 Digital video time series storage step, 450 Video summary Instruction metadata storing step, 510 data reading step, 520 video frame extracting step.

Claims (11)

A method for storing a video summary of a digital video time series taken with a digital video camera comprising:
Photographing a digital video time series having a plurality of video frames with a digital video photographing device;
Storing the digital video time series in a processor accessible memory;
Identifying one or more key video fragments corresponding to a group of video frames from a stored digital video time series;
Generating a video summary by combining key video fragments;
Identifying the storage location of the video memory in the processor-accessible memory by storing metadata indicating a video frame group corresponding to the video summary in association with the stored digital video time series; and
Having a method.   The method of claim 1, wherein the metadata includes metadata indicating a group of audio samples corresponding to a video summary.   The method of claim 1, comprising storing metadata indicative of a video transition effect applied to a boundary between key video fragments.   The method of claim 1, comprising storing metadata indicative of an audio transition effect applied to the boundary between key video fragments. A system for displaying video frames corresponding to a video summary,
A soft copy display used to display video frames,
A processor;
And the processor is
A step of reading data indicating a video frame group corresponding to the video summary among the data associated with the stored digital video time series;
Extracting a video frame group corresponding to the video summary from the stored digital video time series;
A display step for displaying a video frame group corresponding to the video summary on the soft copy display;
Running system.   6. The system according to claim 5, wherein in the reading step, data indicating an audio sample group corresponding to a video summary is also read.   7. The system according to claim 6, wherein a group of audio samples corresponding to the video summary is extracted from the stored digital video time series.   6. The system according to claim 5, wherein in the reading step, data indicating a video transition effect used in a video summary is also read.   9. The system according to claim 8, wherein, in the display step, a video transition effect is applied to a video frame group prior to display.   6. The system according to claim 5, wherein the listed data associated with the stored digital video time series is metadata relating to the stored digital video time series.   6. The system of claim 5, wherein the listed data associated with the stored digital video time series is stored in a separate file.

Images