JP2008060622A - Video editing system, video processing apparatus, video editing device, video processing method, video editing method, program, and data structure - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate metadata holding a feature quantity of video data or another video data synchronized with the video data, synchronously with the video data or to edit video data on the basis of the metadata. <P>SOLUTION: In the video editing system, metadata 103 is generated from second video data 102 synchronized with first video data 101 by a video processing apparatus 110. The metadata 103 holds the feature quantity of the second video data 102 synchronously with the first video data 101 while managing it on the time base. Positions meeting an extraction condition are searched in the metadata 103 by a position search part 122 of a video editing device 120 to generate an edit list 104. The video images are extracted from the first video data 101 by a video extraction part 123 in accordance with the edit list 104 to generate edited video data 105. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a video editing system, and in particular, a video editing system, a video processing apparatus, or a video editing apparatus that edits video data based on the metadata, and processing in these, The present invention relates to a method, a program for causing a computer to execute the method, and a data structure used for them.

  A QuickTime file format is known as one of recording formats in the recording / reproducing apparatus. This QuickTime file format is a file format for handling multimedia data. Real data of video data (video data and audio data) is held in a media data atom (also referred to as movie data), and management information thereof is stored. Stored in a movie atom (also called a movie resource). This allows video data to be edited "non-destructively" without having to modify the actual data directly. File formats based on this QuickTime file format include ISO base media file format, MPEG4 (MP4) file format, MJ2 (Motion JPEG2000) file format, and AVC (Advanced Video Coding: MPEG4-part10) file format.

In such a file format divided into an actual data storage unit and a management information storage unit, a method of editing with a data structure called an edit atom that refers to the original data from the outside and performs time axis management on reproduction It has been known. For example, a video recording apparatus has been proposed that uses such an edit atom to mark a video being recorded in a non-destructive manner (see, for example, Patent Document 1).
Japanese Patent Laying-Open No. 2005-303943 (FIG. 5)

  However, in a movie file composed of edit atoms, only one final editing result is stored, and in order to perform editing under other conditions, the process must be restarted from the beginning. For example, when analyzing some feature quantity in video data or other video data synchronized with the video data, and editing based on the analysis result, it will start from the analysis of the feature quantity every time the editing condition is changed. There is a problem in processing efficiency.

  Therefore, the present invention generates metadata that holds the feature amount of the video data or other video data synchronized with the video data in synchronization with the video data, or stores the video data based on the metadata. The purpose is to edit.

  The present invention has been made to solve the above problems, and a first aspect of the present invention is a video acquisition means for acquiring second video data synchronized with the first video data managed in time series. A video analysis unit that analyzes a feature quantity in the second video data; and a metadata generation unit that generates metadata that holds the feature quantity in synchronization with the first video data. Is a video processing apparatus. This brings about the effect that the feature quantity of the second video data synchronized with the first video data is held in the metadata synchronized with the first video data.

  In the first aspect, the feature amount may be a facial expression included in the second video data. Types of facial expressions such as laughter, surprise, anger, and sleepiness can be expressed.

  In the first aspect, the image processing device further includes a first imaging unit that captures the first video data, and the video acquisition unit is configured to capture the first video data and the second video data simultaneously. Second imaging means for imaging the image may be included. This brings about the effect | action that the image | video regarding the 1st image data currently imaged is imaged as 2nd image data.

  In the first aspect, the image processing device further includes a reproducing unit that reproduces the first video data, and the video obtaining unit is configured to simultaneously reproduce the first video data by the reproducing unit. An imaging means for imaging data may be included. This brings about the effect | action that the image | video regarding the 1st image data currently reproduced | regenerated is imaged as 2nd image data.

  Further, in the first aspect, the image processing device further includes a reproduction unit that reproduces the first video data, and the video acquisition unit converts the first video data reproduced by the reproduction unit into the second video. Video input means for inputting as data may be included. Thereby, the effect | action that the 1st video data currently reproduced | regenerated is input as 2nd video data as it is brought about is brought about.

  In the first aspect, the first video data and the metadata may be recorded in a media data atom format in the QuickTime format.

  Further, the second aspect of the present invention provides a time series that meets a predetermined condition after acquiring metadata that holds the feature amount in the second video data synchronized with the first video data managed in time series. A position search means for searching the upper position and generating the result as search information, and a video extraction for extracting a portion corresponding to the matching time-series position from the first video data based on the search information And a video editing apparatus. As a result, the corresponding portion of the first video data is extracted and edited nondestructively with respect to the position on the time series where the feature amount included in the metadata matches a predetermined condition.

  Further, in this second aspect, when the position search means acquires management information for managing the metadata, and the management information indicates that the feature amount is not retained in the metadata. The metadata may not be acquired. This brings about the effect of avoiding access to meaningless metadata.

  In addition, according to a third aspect of the present invention, video acquisition means for acquiring second video data synchronized with the first video data managed in time series, and a feature amount in the second video data are analyzed. A video analysis unit, a metadata generation unit that generates metadata that holds the feature amount in synchronization with the first video data, and a time-series position that matches a predetermined condition is searched from the metadata. Position search means for generating the result as search information, and video extraction means for extracting a portion corresponding to the matching time-series position from the first video data based on the search information. Is a video editing system characterized by Accordingly, there is an effect that the first video data is edited nondestructively with the metadata holding the feature amount of the second video data as an intermediate state.

  According to a fourth aspect of the present invention, the feature amount in the first video data managed in time series and the second video data synchronized with the first video data is synchronized with the first video data. A data structure having metadata stored therein, wherein the computer acquires the metadata, searches the metadata for a position in time series that matches a predetermined condition, and searches for the result The data structure is characterized in that it is generated as information and a portion corresponding to the matching time-series position is extracted from the first video data based on the search information. As a result, the metadata is held as an intermediate state for editing the first video data.

  According to the present invention, with respect to video data or other video data synchronized with the video data, metadata that retains the feature amount in synchronization with the video data is generated, or video data is generated based on the metadata. An excellent effect of being able to edit can be achieved.

  Next, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration example of a video editing system 100 according to an embodiment of the present invention. The video editing system 100 analyzes the feature quantity of the video editing apparatus 120 that outputs the edited video data 105 obtained by editing the first video data 101 based on the metadata 103, and the second video data 102, and outputs the metadata. And a video processing apparatus 110 that outputs 103.

  The first video data 101 and the second video data 102 are video data managed in time series, and may include audio data in addition to moving image data. The first video data 101 is video data to be edited by the video editing device 120. The second video data 102 is synchronized with the first video data 101 and can be imaged simultaneously with the imaging or reproduction of the first video data 101. Further, the first video data 101 and the second video data 102 may have the same content.

  The metadata 103 is synchronized with the first video data 101 and holds the feature amount of the second video data 102 while managing it on the time axis. As described later, the facial expression included in the second video data 102 is assumed as the feature amount.

  The video processing device 110 includes a video acquisition unit 111, a video analysis unit 112, and a metadata generation unit 113. The video editing apparatus 120 includes an extraction condition reception unit 121, a position search unit 122, and a video extraction unit 123.

  The video acquisition unit 111 acquires the second video data 102. The video acquisition unit 111 may be a video camera that captures an image with an optical lens, or may be an input terminal for inputting an electronic signal.

  The video analysis unit 112 analyzes the feature amount in the second video data 102 acquired by the video acquisition unit 111. For example, the video analysis unit 112 extracts a face image included in the second video data 102 as a feature amount, and determines the facial expression. A known technique can be used as a processing procedure for determining the facial expression. For example, a technique (for example, Japanese Patent Application Laid-Open No. 2005-266984) that determines a smile based on a ratio of a line segment connecting an external eye corner point and a line segment connecting a mouth corner point in a face image, and a reference image for each face component A technique (for example, Japanese Patent Application Laid-Open No. 2004-46591) has been proposed in which an evaluation point is provided and an average value is calculated to determine the facial expression of the subject.

  The metadata generation unit 113 generates the metadata 103 that holds the feature amount analyzed by the video analysis unit 112. The generated metadata 103 holds the feature amount of the second video data 102 corresponding to each time of the first video data 101. Thus, for example, what facial expression is included in the second video data 102 at each time of the first video data 101 can be held as an intermediate state for editing.

  The extraction condition receiving unit 121 receives input of extraction conditions. For example, if the facial expression is a condition, a condition such as whether to extract a smile or a surprised face is accepted. These conditions can be combined with logical product (AND) or logical sum (OR).

  The position search unit 122 searches the metadata 103 based on the extraction condition received by the extraction condition reception unit 121. Thereby, a position on the time series that matches the extraction condition is obtained and held as the edit list 104.

  The video extraction unit 123 extracts a portion corresponding to the position on the time series that matches the extraction condition from the first video data 101 based on the edit list 104, and outputs it to the edited video data 105.

  FIG. 2 is a diagram illustrating a first configuration example of the video processing apparatus 110 according to the embodiment of the present invention. In the first configuration example, the recording unit 210 includes an imaging unit 211, a video processing unit 212, a video compression unit 213, a file generation unit 214, a writing unit 215, an imaging unit 216, and a video analysis unit 217. A recording control unit 218 and a recording medium 219.

  The imaging unit 211 captures the subject as the first video data 101. The video processing unit 212 performs processing such as effect processing on the video captured by the imaging unit 211. The video compression unit 213 compresses the video processed by the video processing unit 212.

  The imaging unit 216 captures the face of the photographer who is capturing the subject (or the viewer who is viewing the subject) as the second video data 102. The video analysis unit 217 analyzes the video imaged by the imaging unit 216. That is, the video analysis unit 217 analyzes the facial expression of the photographer included in the video. This analysis result becomes metadata 103.

  The file generation unit 214 includes a file including the video compressed by the video compression unit 213 (first video data 101) and the analysis result (metadata 103) of the facial expression of the photographer analyzed by the video analysis unit 217. Each is generated as a predetermined file format.

  The writing unit 215 writes the file generated by the file generation unit 214 to the recording medium 219. As the recording medium 219, a disk-shaped recording medium such as a hard disk or a semiconductor recording medium such as a memory stick can be assumed.

  The recording control unit 218 controls the recording operation on the recording medium 219 in the recording unit 210.

  In this first configuration example, the imaging unit 211 captures the subject as the first video data 101, and at the same time, the imaging unit 216 captures the facial expression of the photographer as the second video data 102. The metadata 103 is generated by analysis.

  FIG. 3 is a diagram illustrating a usage mode according to the first configuration example of the video processing apparatus 110 according to the embodiment of the present invention. In FIG. 3A, the subject 501 is imaged by the camera 521 on the front surface of the video camera device 520, and the face of the photographer 502 is imaged by the camera 522 on the operation surface of the video camera device 520. Accordingly, the metadata can be generated by analyzing the facial expression of the photographer 502 inside the video camera device 520.

  In addition, the video camera device 520 may be connected to the network 510. Through the network 510, the video of the subject 501 captured by the camera 521 on the front surface of the video camera device 520 can be distributed to the television device 530 in FIG. 3B or the computer device 540 in FIG.

  A video camera device 531 is connected to the television device 530 in FIG. 3B, and the face of the viewer 503 is captured. Accordingly, the metadata can be generated by analyzing the facial expression of the viewer 503 inside the television device 530 or the video camera device 531.

  A camera 541 is provided in front of the computer apparatus 540 in FIG. 3C, and the face of the viewer 504 is captured. Thus, the metadata can be generated by analyzing the facial expression of the viewer 504 inside the computer device 540.

  As described above, in the first configuration example of the video processing apparatus 110, the second video data 102 is imaged simultaneously with the imaging of the first video data 101, and the metadata 103 is generated from the second video data 102. To do. Since the facial expression included in the second video data 102 is considered to reflect some reaction to the video of the first video data 101, the feature amount is stored in the metadata 103 as an intermediate state. The first video data 101 is to be used for editing.

  FIG. 4 is a diagram illustrating a second configuration example of the video processing apparatus 110 according to the embodiment of the present invention. In the second configuration example, the recording unit 210 includes an imaging unit 211, a file generation unit 214, a writing unit 215, a video analysis unit 217, a recording control unit 218, and a recording medium 219. . In addition, the playback unit 220 includes a display unit 221, a video processing unit 222, a video expansion unit 223, a file decoding unit 224, a reading unit 225, and a recording medium 229.

  The recording medium 229 records the first video data 101 in a predetermined file format. The reading unit 225 reads a file including the first video data 101 from the recording medium 229. The file decryption unit 224 decrypts the file read by the reading unit 225. The video decompression unit 223 decompresses the compressed video in the file decrypted by the file decryption unit 224. The video processing unit 222 performs processing such as effect processing on the video expanded by the video expansion unit 223.

  The display unit 221 displays the video output from the video processing unit 222. As a result, the first video data 101 recorded on the recording medium 229 is reproduced and displayed on the display unit 221.

  The imaging unit 211 images the subject as the second video data 102. The video analysis unit 217 analyzes the video imaged by the imaging unit 211. That is, the video analysis unit 217 analyzes the facial expression of the viewer included in the video. This analysis result becomes metadata 103.

  The file generation unit 214 generates a file including the analysis result (metadata 103) of the facial expression of the viewer analyzed by the video analysis unit 217 as a predetermined file format.

  The writing unit 215 writes the file generated by the file generation unit 214 to the recording medium 219. The recording control unit 218 controls the recording operation on the recording medium 219 in the recording unit 210.

  As the recording media 219 and 229, a disc-shaped recording medium such as a hard disk and a semiconductor recording medium such as a memory stick can be assumed, but both may be different types of recording media.

  In the second configuration example, the display unit 221 reproduces and displays the first video data 101, and at the same time, the imaging unit 211 captures the facial expression of the viewer as the second video data 102, and analyzes the facial expression. Thus, the metadata 103 is generated.

  FIG. 5 is a diagram illustrating a usage mode according to the second configuration example of the video processing device 110 according to the embodiment of the present invention. In FIG. 2A, a playback display screen is displayed on the operation surface of the video camera device 520, and the face of the viewer 505 is captured by the camera 522 on the operation surface. Thereby, metadata can be generated by analyzing the facial expression of the viewer 505 inside the video camera device 520.

  A video camera device 531 is connected to the television device 530 in FIG. 5B, and the face of the viewer 503 is captured. Accordingly, the metadata can be generated by analyzing the facial expression of the viewer 503 inside the television device 530 or the video camera device 531.

  A camera 541 is provided in front of the computer apparatus 540 in FIG. 5C, and the face of the viewer 504 is captured. Thus, the metadata can be generated by analyzing the facial expression of the viewer 504 inside the computer device 540.

  As described above, in the second configuration example of the video processing apparatus 110, the second video data 102 is captured simultaneously with the reproduction of the first video data 101, and the metadata 103 is generated from the second video data 102. To do. Since the facial expression included in the second video data 102 is considered to reflect some reaction to the video of the first video data 101, the feature amount is stored in the metadata 103 as an intermediate state. The first video data 101 is to be used for editing.

  FIG. 6 is a diagram illustrating a third configuration example of the video processing apparatus 110 according to the embodiment of the present invention. In the third configuration example, the recording unit 210 includes a video input unit 206, a file generation unit 214, a writing unit 215, a video analysis unit 217, a recording control unit 218, and a recording medium 219. Yes. Further, the playback unit 220 includes a video processing unit 222, a video expansion unit 223, a file decoding unit 224, a reading unit 225, and a recording medium 229.

  The configuration of the playback unit 220 is the same as that of the second configuration example, except that the display unit 221 is omitted and the output of the video processing unit 222 is supplied to the recording unit 210 as it is.

  The configuration of the recording unit 210 is the same as that in the second configuration example, except that a video input unit 206 that inputs video from the playback unit 220 is provided instead of the imaging unit 211.

  That is, in the third configuration example, the video (first video data 101) reproduced by the reproduction unit 220 is supplied as it is as the input video (second video data 102) of the recording unit 210 and is included therein. The metadata 103 is generated by analyzing facial expressions.

  FIG. 7 is a diagram illustrating a configuration example of a camera-integrated imaging apparatus that is an example of the video processing apparatus 110 according to the embodiment of the present invention. The imaging apparatus includes an imaging unit 301, a video processing unit 330, a video compression unit 341, a compression control unit 342, a recording medium access unit 351, a drive control unit 352, an operation reception unit 360, and a display unit 370. And a system control unit 390.

  The imaging unit 301 images a subject and outputs it as video data. The video processing unit 330 performs effect processing on the video data output from the imaging unit 301. The video compression unit 341 compresses the video data processed by the video processing unit 330. The compression control unit 342 controls compression processing in the video compression unit 341.

  The recording medium access unit 351 performs writing and reading with respect to the recording medium 309. The drive control unit 352 controls writing and reading by the recording medium access unit 351.

  The operation accepting unit 360 accepts an operation input by a user, and various buttons, GUI (Graphical User Interface), and the like are assumed. The display unit 370 displays an image being picked up or reproduced, or various messages to the user.

  The system control unit 390 controls the entire imaging apparatus and can be realized by, for example, a microprocessor. The system control unit 390 controls the start and stop of video recording, the elapsed time information of recording, and the like according to the operation input received by the operation receiving unit 360, and controls the display on the display unit 370 for the user. Further, the system control unit 390 exchanges information with the camera control unit 329 and the compression control unit 342, and performs writing control on the recording medium 309 via the drive control unit 352.

  The imaging unit 301 includes a zoom lens 311, an iris (aperture) 312, a focus lens 313, a filter 314, an image sensor 321, an A / D converter 322, a camera signal processing circuit 323, and a detection unit. 324, a zoom control unit 325, an angular velocity sensor 326, and a camera control unit 329.

  The zoom lens 311 is a lens for performing zoom (enlargement) processing. The iris 312 is an aperture for adjusting the amount of light from the subject. The focus lens 313 is a lens for focusing on a subject. The filter 314 is a filter for removing infrared rays.

  The imaging element 321 is a photoelectric conversion element that converts light supplied from the optical lens group into an electrical signal, and can be realized by, for example, a CCD (Charge Coupled Devices). By this image sensor 321, the image of the subject is extracted as three video signals corresponding to, for example, three primary colors of RGB (red, green, blue).

  The A / D converter 322 converts an analog electric signal supplied from the image sensor 321 into a digital signal. The camera signal processing circuit 323 performs signal processing such as white balance that defines a white reference for the digital signal converted by the A / D converter 322.

  The detection unit 324 performs various types of detection processing upon receiving feedback of the video signal subjected to signal processing by the camera signal processing circuit 323. For example, auto focus (AF) detection for automatically focusing on a subject, auto exposure (AE) detection for automatically performing exposure, and for automatically performing white balance Auto white balance (AWB) detection is performed.

  The zoom control unit 325 controls the zoom process by moving the zoom lens 311 in accordance with an operation input from the user. The angular velocity sensor 326 detects the angular velocity of the imaging apparatus, and detects the degree of camera shake using, for example, a gyroscope.

  The camera control unit 329 controls the imaging unit 301. For example, the camera control unit 329 performs control so as to reduce the deterioration in image quality by performing camera shake correction on the camera shake detected by the angular velocity sensor 326. The camera control unit 329 performs control of video input from the image sensor 321, control of processing in the detection unit 324, control of processing in the zoom control unit 325, and the like.

  FIG. 8 is a diagram showing a structure example of a file format based on the QuickTime file format (hereinafter referred to as QuickTime base file format). In this file format, the content of the file is divided into an actual data storage unit and a management information storage unit that stores location information necessary for referring to the actual data. In the QuickTime file format, the actual data storage unit is called a media data atom (type name: 'mdat'), and the management information storage unit is called a movie atom (movie atom, type name: 'moov'). “Atom” may also be expressed as “box”. A movie atom may be expressed as a movie resource, and a media data atom may be expressed simply as media data or movie data.

  These media data atom and movie atom may be included in the same file or may be separated into different files. For example, as shown in FIG. 8, a media data atom 612 including media data of a moving image (V1 or the like) or sound (A1 or the like) and a movie atom 611 that refers to the media data atom 612 are stored in the same file 610. Alternatively, a movie atom 621 that refers to the media data atom 612 may be stored in another file 620. A file having the former format is called a self-contained file, and a file having the latter format is called an external reference file. Therefore, the movie atom can store management information indicating a relative path or an absolute path of an external file including a media data atom to be externally referenced.

  The media data atom 612 stores, for example, audio data encoded by a compression encoding method based on MPEG1 audio (MPEG1 Audio Layer2) and image data encoded by a compression encoding method conforming to the MPEG2 Video (MPEG2 Video) standard. Is done. The encoding method is not limited to these. For example, in the case of video data, motion JPEG (Motion JPEG), MJ2 (Motion JPEG2000), MPEG4 (MP4), AVC (Advanced Video Coding: MPEG4-part10), audio If it is data, Dolby AC3 (Dolby AC3), ATRAC (Adaptive TRansform Acoustic Coding), etc. may be sufficient, and it is also possible to store the linear data which has not been compression-encoded.

  FIG. 9 is a diagram showing a hierarchical structure in the QuickTime file format. Actual data in the media data atom ('mdat') is divided into minimum management units called samples, and a collection of an arbitrary number of samples is called a chunk. The movie atom ('moov'), which is management information of the media data atom ('mdat'), stores the sample size, the chunk storage location, the display time of each sample, and the like.

  The movie atom ('moov') is composed of a movie header atom ('mvhd'), a track atom ('trak'), and the like.

  The movie header atom ('mvhd') is a part that holds movie atom header information, and indicates the characteristics of the entire movie. For example, the period, time scale, creation date, etc. of the entire movie are included as items.

  A track atom ('trak') stores different types of data such as sound, video, and text in different tracks. In this figure, a track header atom ('tkhd') and a video track atom are shown. , An edit atom ('edts'), a user data atom ('udta'), and a media atom ('mdia'). Although the audio track atom is omitted, it has the same configuration as in the case of video.

  The track header atom ('tkhd') is a part that holds the track atom header information, and indicates the characteristics of the track. For example, the number of video pixels, sound volume, creation date, and the like are included as items.

  The edit atom ('edts') holds track editing information as an edit restore tom ('elst'). This edit atom will be described in detail with reference to FIG.

  The user data atom ('udta') includes arbitrary information defined by the user as necessary. For example, it is possible to hold a movie window position, a playback method, creation information, and the like. This user data atom holds movie user data in a list format.

  The media atom ('mdia') stores information regarding actual data actually used in the track. That is, the media atom stores information on the entire medium, information on handling of media data, information on the configuration of the media, and the like. Actual data is divided into minimum management units called samples, and a collection of an arbitrary number of samples is called a chunk. In the media atom, the sample size, the chunk storage location, the display time of each sample, and the like are stored.

  The media atom includes a media header atom ('mdhd'), a media handler atom ('hdlr'), a media information atom ('minf'), and the like.

  The media header atom ('mdhd') is a part that holds the header information of the media atom, and indicates the characteristics of the entire medium.

  The media handler atom ('hdlr') holds information regarding handling for each medium.

  The media information atom ('minf') holds information expressed by the media type. This media information atom is composed of a video media information header atom ('vmhd'), a data handler atom ('hdlr'), a data information atom ('dinf'), a sample table atom ('stbl'), and the like. The

  The video media information header atom ('vmhd') holds header information related to video media in the video track. In the case of an audio track, a sound media information header atom ('smhd') that holds header information related to sound media is included instead of the video media header atom ('vmhd').

  The data handler atom ('hdlr') holds information relating to handling of video media.

  The data information atom ('dinf') holds information related to the storage location of actual data that is actually referred to. The data information atom includes a data reference atom ('dref') that holds information regarding the storage method, storage location, and file name of the actual data to be referenced.

  The sample table atom ('stbl') holds information about a sample that is a minimum management unit of actual data of the medium. The sample table atom includes a sample description atom ('stsd'), a time-to-sample atom ('stss'), a sample size atom ('stsz'), a sample-to-chunk atom ('stsc'), and a chunk It consists of an offset atom ('stco') or the like.

  The sample description atom ('stsd') holds information regarding the compression method and characteristics of each sample. Time vs. sample atom ('stss') holds the relationship between each sample and time. The sample size atom ('stsz') holds the data amount of each sample. Sample-to-chunk atom ('stsc') holds the relationship between a chunk and the samples that make up that chunk. A chunk offset atom ('stco') holds an offset from the file head to the head position of each chunk.

  In the embodiment of the present invention, not only the first video data 101 is held in the QuickTime base file format, but also the metadata 103 is held in the QuickTime base file format. Thereby, the feature amount of the second video data 102 can be held in the metadata 103 while being managed on the time axis.

  FIG. 10 is a diagram showing an example of a file storage format according to the embodiment of the present invention.

  FIG. 10A is a diagram illustrating a configuration example of a video file 630 including the first video data 101. The video file 630 includes media data 632 having the first video data 101 and a movie resource 631 for managing the media data 632. The media data 632 includes each sample 633 of the first video data 101, and these are managed by the movie resource 631.

  FIG. 10B is a diagram illustrating a configuration example of the metafile 640 including the metadata 103. The metafile 640 includes media data 642 having the metadata 103 and movie resources 641 for managing the media data 642. The media data 642 includes each sample 643 of the metadata 103, and these are managed by the movie resource 641. In addition, the movie resource 641 is also managed by one time axis in a format that externally refers to the media data 632 as well.

  FIG. 10C is a diagram illustrating the flow on the time axis of the video track 650 and the meta track 660 realized by the video file 630 and the meta file 640. Here, for simplification, only the video track 650 is shown in the video signal and audio signal of the video file 630.

  In the video track 650, the samples 653 are arranged on the time axis. In the meta track 660, the samples 663 are arranged on the time axis in synchronization with the sample 653 of the video track 650. For example, when a smile is extracted as a feature amount in a section from time t1 to t2 of the second video data 102, that fact is recorded in a section from time t1 to t2 of the meta track 660. Similarly, when a surprised face is extracted as a feature amount in a section from time t3 to t4 of the second video data 102, the fact is recorded in a section from time t3 to t4 of the meta track 660.

  That is, the meta track 660 indicating the metadata 103 is synchronized with the video track 650 indicating the first video data 101 via the second video data 102.

  Although omitted in the figure, information indicating that there is no expression is recorded in the meta track 660 in a section where there is no expression in the classification of facial expressions.

  FIG. 11 is a diagram showing an example of the hierarchical structure of the meta track in the embodiment of the present invention. This meta track has basically the same configuration as the video track described in FIG. However, it differs from a video track in that it has a track reference atom ('tref') under the track and a track input map atom ('map') under the media atom.

  The track reference atom ('tref') holds information for designating a reference relationship with the source track (first video data 101). Therefore, the track reference atom includes a track reference type atom ('ssrc') that designates a track ID unique to the track stored in the track header atom ('tkhd') of the track to be designated. The number of track IDs included in this track reference type atom matches the number of source tracks.

  The track input map atom ('imap') holds information relating to the source track, and is configured by a data structure called a QT atom structure in QuickTime. The track input map atom includes one or more track input QT atoms ('in') packed by a container having a QT atom container ('sean') as the highest level atom. The number of track input QT atoms matches the number of source tracks.

  The track input QT atom ('in') holds an input type QT atom ('ty') and a data source type QT atom ('dtst'). The input type QT atom ('ty') specifies that the source track is video media. The data source type QT atom ('dtst') gives a unique name to the source track.

  FIG. 12 is a diagram showing a description example of a movie atom including a meta track in the embodiment of the present invention. In this example, only one video track atom (video) is included as a source track (first video data 101). Therefore, only one track input QT atom ('in') in the meta track atom is included.

  FIG. 13 is a diagram showing another description example of the movie atom including the meta track in the embodiment of the present invention. FIG. 14 is a diagram showing a description example of the meta track atom in the movie atom of FIG. In this description example, two video track atoms are included as a source track (first video data 101) (video 1 and video 2). Therefore, as shown in FIG. 14, two track input QT atoms ('in') in the meta track atom are also included.

  FIG. 15 is a diagram showing a description example of the meta description sample description atom ('stsd') in the embodiment of the present invention. In this description example, M meta sample description entries (M is an integer of 1 or more) are included. The number of metasample description entries matches the number of feature types. For example, if there are two types of feature amounts, that is, an expressionless face and a smile, the number of metasample description entries is two.

  In the figure, the numbers in parentheses indicate the number of bytes in each field.

  The meta sample description entry has a structure in which a stream descriptor atom is extended and added to the sample description entry in QuickTime. The data format field in the sample description entry is originally for specifying the effect. In the embodiment of the present invention, this field is used for expansion. As a result, the extension can be performed while maintaining compatibility with the normal QuickTime file format.

  FIG. 16 is a diagram showing an example of the data format field in the embodiment of the present invention. As shown in this figure, the data format field is originally for designating an effect. In the figure, the types indicated by lowercase letters are the effect types defined in QuickTime. For example, the type name “brco” designates the effect of changing the brightness indicating brightness and the contrast indicating the width of black and white in the image.

  On the other hand, the type indicated by the capital letter is an effect type not defined in QuickTime. In the embodiment of the present invention, by specifying the type name “UDEF” indicating user-defined metadata in the bottom column of the figure in this data format field, the meaning uniquely expanded as metadata is provided. It shows that it has.

  FIG. 17 is a diagram illustrating a description example of the stream descriptor atom ('strd') according to the embodiment of the present invention. This stream descriptor atom holds fields of size (Size), type (Type), version (Version), and flags (Flags), as with other atom structures in QuickTime.

  The size field holds the size of the entire stream descriptor atom including this size field. The type field holds “strd” as the type name of the stream descriptor atom. The version field and the flag group field are reserved for future expansion, and all zeros are set here.

  As described below, the stream descriptor atom ('strd') further holds three fields of data format (Data Format), user defined meta type (User Defined Meta Type), and parameter flag (Parameter Flag). is doing.

  The data format field is a field that holds the same format as the data format field of the metasample description entry described with reference to FIG. 16, and indicates the type name “UDEF” in the embodiment of the present invention.

  As shown in FIG. 18, the user-defined metatype field holds a 2-byte owner ID (Owner ID) and a 2-byte meta ID (Meta ID). The owner ID is an ID assigned to each manufacturer, and thus each manufacturer can use a unique extension definition by using a meta ID. These owner IDs and meta IDs are valid only when the data format field indicates the type name “UDEF”. For example, as shown in FIG. 19, as a meta ID, types such as smile, surprise, angry, and sleepy can be expressed as facial expressions.

  The reason that the detailed metadata type is defined separately in the two fields of the owner ID and the meta ID is that it is closed in the manufacturer and the metadata type is managed and operated without duplication. If such a distinction is not provided, names may be duplicated between manufacturers to be newly defined, or management of application order may become complicated. Therefore, the unique definition metadata type specified in the data format field on the metasample description entry side only indicates that it is broadly classified as “UDEF”, and the detailed unique extended definition metadata type. Indicates the metadata type defined by which manufacturer by combining both the owner ID and meta ID fields.

  The parameter flag field is a field indicating whether or not the effect by metadata is effective. For example, as shown in FIG. 20, 1 bit out of 16 bits is used to indicate whether the effect of metadata is “valid” or “invalid”. As a result, when the parameter flag field in the movie resource ('moov') indicates "invalid", it is determined that the media data ('mdat') has no effect without accessing the media data ('mdat'). And the processing load can be reduced.

  FIG. 21 is a diagram illustrating a sample description example of media data (metadata 103) of a meta track according to the embodiment of the present invention. The meta track samples are stored in the media data atom for each sample in the same manner as the video track samples.

  Here, it is possible to indicate the number of faces (face_number) extracted in the corresponding sample of the second video data 102 and the degree of facial expression of each face. For example, the degree of smile, the degree of surprise, the degree of anger, and the degree of sleepiness can be shown.

  FIG. 22 is a diagram showing an example of a hierarchical structure of a file including metadata in the embodiment of the present invention. In this example, it is assumed that there is only one source track (first video data 101) (source track 1). The track header atom ('tkhd') of the movie resource ('moov') of the source track 1 holds the track ID “# 1”. The track header atom of the movie resource of the meta track (meta data 103) holds “# 2” that is the track ID.

  In the meta-track movie resource, the track ID “# 1” of the source track 1 is held in the track reference type atom (“ssrc”) of the track reference atom (“tref”).

  Also, in the track input map atom ('imap') of the media atom ('mdia'), 'video' representing the video media is set as the input type QT atom ('ty'), and the data source type QT atom (' The name “srcA” of the source track 1 is set as “dtst”).

  Also, in the metatrack movie resource, two types “meta_type1” and “meta_type2” are defined by the meta ID of the sample description atom ('stsd').

  In this example, four meta samples are provided in the media data ('mdat') of the meta track. There is only one source track, and all indicate the same source track 1. Further, meta samples # 1 and # 3 indicate “meta_type1”, and meta samples # 2 and # 4 indicate “meta_type2”.

  FIG. 23 is a diagram illustrating the relationship between the source track (first video data 101) and the meta track (metadata 103) in the example of FIG. In this example, there is only one source track, and all are the same source track 1 ('srcA').

  Here, when “meta_type2” is extracted as a specific section, only this specific section is extracted from the source track, and other sections are treated as unnecessary sections.

  FIG. 24 is a diagram showing another example of a hierarchical structure of a file including metadata in the embodiment of the present invention. In this example, it is assumed that there are two source tracks (first video data 101) (source track 1 and source track 2). The track header atom ('tkhd') of the movie resource ('moov') of the source track 1 holds the track ID “# 1”. The track header atom ('tkhd') of the movie resource of the source track 2 holds “# 2” that is the track ID. Also, the track header atom ('tkhd') of the movie resource of the meta track (meta data 103) holds the track ID “# 3”.

  In the meta track movie resource, the track reference type atom ('ssrc') of the track reference atom ('tref') has the track ID "# 1" of the source track 1 and the track ID "# 2" of the source track 2 respectively. Is retained.

  The track input map atom ('imap') of the media atom ('mdia') includes two track input QT atoms ('in'), and the first track input QT atom is 'Video' representing video media is set as the input type QT atom ('ty'), and the name 'srcA' of the source track 1 is set as the data source type QT atom ('dtst'). In the second track input QT atom, “video” representing video media is set as the input type QT atom, and the name “srcB” of the source track 2 is set as the data source type QT atom.

  Also, in the metatrack movie resource, two types “meta_type1” and “meta_type2” are defined by the meta ID of the sample description atom ('stsd').

  In this example, four meta samples are provided in the media data ('mdat') of the meta track. There are two source tracks, metasamples # 1 and # 2 refer to 'srcA', and metasamples # 3 and # 4 refer to 'srcB'. Further, meta samples # 1 and # 3 indicate “meta_type1”, and meta samples # 2 and # 4 indicate “meta_type2”.

  FIG. 25 is a diagram showing the relationship between the source track (first video data 101) and the meta track (metadata 103) in the example of FIG. In this example, there are two source tracks, metasamples # 1 and # 2 refer to 'srcA', and metasamples # 3 and # 4 refer to 'srcB'.

  Here, when “meta_type2” is extracted as a specific section, only this specific section is extracted from the source track, and other sections are treated as unnecessary sections.

  FIG. 26 is a diagram showing a relationship example between the source track (first video data 101) and meta track (meta data 103) and the edit track (edit video data 105) in the embodiment of the present invention. In this example, the source track starts at time 0 and ends at time ts6. The meta track is synchronized with the source track, indicates that a smile is detected in the section from time ts1 to time ts2, indicates that a surprised face is detected in the section from time ts3 to time ts4, and starts from time ts5. This indicates that a smile is detected in the section at time ts6.

  In a movie file that has not been edited, the time axis of the source track time often has a one-to-one correspondence with the time axis of the media time. Therefore, here, time ts1 = time tm1, time ts2 = time tm2, Time ts3 = time tm3, time ts4 = time tm4, time ts5 = time tm5, time ts6 = time tm6.

  Here, if it is assumed that the source track is edited using a section showing a smiling face or a surprised face in the meta track as an extraction condition, an edit track as shown in the figure is generated as the output. That is, the portion of the source track in the section from the time ts1 to the time ts2 of the source track time becomes the section from the time 0 to the time te1 of the edit track time in the edit track, and the source in the section from the time ts3 to the time ts4 of the source track time The track portion is a section from time te1 to time te2 of the edit track time in the edit track, and the source track portion in the section from time ts5 to time ts6 of the source track time is from time te2 of the edit track time in the edit track It becomes the section of time te3.

  FIG. 27 is a diagram illustrating a description example of edit atoms ('edts') in the QuickTime file format. As with the stream descriptor atom ('strd') described with reference to FIG. 17, this edit atom holds fields of size, type, version, and flag group. In this edit atom, “edts” is held as a type name.

  This edit atom further holds an edit restore tom ('elst'). Similar to the edit atom, this edit restore atom holds fields of size, type, version, and flag group. In this edit restore tom, “elst” is held as the type name. The edit restore tom further includes N (N is an integer of 1 or more) edit list entries (Edit List Entry) and the number (Number of Entries).

  Each of the edit list entries includes a segment duration, a media time, and a media rate.

  FIG. 28 schematically shows the edit atom ('edts'). As shown in FIG. 28A, the edit atom 680 includes a size 681 indicating the size of the edit atom 680, a type 682 indicating that the edit atom 680 is an edit atom, a version 683 of the edit atom, and an unused flag group 694. And an edit restore tom 570.

  The edit restore tom 690 includes a size 691 indicating the size of the edit restore tom 690, a type 692 indicating to be an edit restore tom, an edit restore tom version 693, an unused flag group 694, and an edit list table. 696 and the number of entries 695 in the edit list table 696.

  The edit list table 696 is composed of the number of entries indicated by the number of entries 695. As shown in FIG. 28B, each entry of the edit list table 696 includes a segment period 697, a media time 698, and a media rate 699. A segment period 697 indicates the period of the corresponding editing unit. The media time 698 indicates the start time in the media data atom of the corresponding editing unit. When the media time 698 indicates “−1”, it means that the corresponding editing unit does not exist in the media data atom. The media rate 699 indicates the time ratio at the time of reproduction, and indicates “1.0” when the reproduction time does not change between the time axis in the media data atom and the time axis after editing.

  FIG. 29 is a diagram showing an example of the contents of the edit restore tom ('elst') in the example of FIG. FIG. 29A shows the contents of the edit restore tom before editing. In the state before editing, there is only one entry. The segment period indicates ts6 of the entire period. The media time indicates 0 at the start time. The media rate 699 is “1.0”.

  FIG. 29B shows the contents of the edited restore tom after editing. After editing, three entries are generated.

  The first entry indicates the first smile period te1 (= tm2-tm1). The media time indicates the start time tm1 of the first smile. The media rate 699 is “1.0”.

  The second entry indicates the surprised face period te2-te1 (= tm4-tm3). The media time indicates the start time tm3 of the surprised face. The media rate 699 is “1.0”.

  The third entry indicates the second smile period te3-te2 (= tm6-tm5). The media time indicates the start time tm5 of the second smile. The media rate 699 is “1.0”.

  FIG. 30 is a diagram showing another example of the relationship between the source track (first video data 101) and the meta track (meta data 103) and the edit track (edit video data 105) in the embodiment of the present invention. In this example, the source track and the meta track have the same relationship as in the example of FIG.

  Here, if it is assumed that the source track is edited using the section showing smile in the meta track as an extraction condition, an edit track as shown in the figure is generated as the output. That is, the portion of the source track in the section from the time ts1 to the time ts2 of the source track time becomes the section from the time 0 to the time te1 of the editing track time in the editing track, and the source in the section from the time ts5 to the time ts6 of the source track time. The track portion is a section from the time te1 to the time te2 of the edit track time in the edit track.

  FIG. 31 is a diagram showing an example of the contents of the edit restore tom ('elst') in the example of FIG. This figure shows the contents of Edit Restore Tom after editing. After editing, two entries are generated.

  The first entry indicates the first smile period te1 (= tm2-tm1). The media time indicates the start time tm1 of the first smile. The media rate 699 is “1.0”.

  The second entry indicates the second smile period te2-te1 (= tm6-tm5). The media time indicates the start time tm5 of the second smile. The media rate 699 is “1.0”.

  The edit restore tom (edit list 104) generated in this way is supplied to the video extraction unit 123, where the source track (first video data 101) to the editing track (edited video data 105) are supplied. Generated. That is, non-destructive editing that does not destroy the original source track can be realized.

  Next, the operation of the video editing system in the embodiment of the present invention will be described with reference to the drawings.

  FIG. 32 is a diagram illustrating an example of a processing procedure performed by the video processing device 110 according to the embodiment of the present invention. First, the video of the second video data 102 is acquired by the video acquisition unit 111 (step S911). The second video data 102 is synchronized with the first video data 101 and may be captured simultaneously with the imaging of the first video data 101 as in the example of FIG. Thus, it may be captured at the same time as the reproduction of the first video data 101, or may have the same content as the first video data 101 as in the example of FIG.

  When the video is acquired in step S911, the feature amount in the acquired second video data 102 is analyzed by the video analysis unit 112 (step S912). For example, a facial image included in the second video data 102 is extracted as a feature quantity, and the facial expression is determined.

  Then, metadata 103 for managing the feature amount analyzed in step S912 on the time axis is generated by the metadata generation unit 113 (step S913). The generated metadata 103 holds the feature amount of the second video data 102 corresponding to each time of the first video data 101.

  The processing procedure of these steps S911 to S913 is repeated until all the images of the second video data 102 are processed (step S914).

  FIG. 33 is a diagram showing an example of a processing procedure performed by the video editing apparatus 120 in the embodiment of the present invention. First, the extraction condition receiving unit 121 receives input of extraction conditions (step S921). Then, the position in the metadata 103 is searched according to the extraction condition (step S930). Thereby, a position on the time series that matches the extraction condition is obtained and held as the edit list 104.

  Based on the edit list 104, a portion corresponding to the position on the time series that matches the extraction condition from the first video data 101 is extracted by the video extraction unit 123 and output as the edited video data 105 (step S923). .

  FIG. 34 is a diagram showing an example of a processing procedure in the position search process (step S930) of FIG. First, a metadata movie resource (meta track in FIG. 11) is acquired (step S931). Then, the effectiveness of the effect, that is, the effectiveness of recording the feature amount is determined based on the parameter flag (see FIG. 20) included in the sample description atom ('stsd') of the movie resource (step S932).

  If it is determined in step S932 that it is “valid”, a sample of media data corresponding to the metadata is acquired (step S933). As a result, if the feature amount indicated by the metadata sample matches the extraction condition (step S934), the corresponding section (segment) is registered as an entry in the edit restore tom (edit list 104) (step S935). ).

  On the other hand, if it is determined as “invalid” in step S932, the media data of the metadata is not acquired, and the processing relating to the sample ends.

  The processing procedures in steps S931 to S935 are repeated until all the samples of the metadata 103 are processed (step S936).

  As described above, according to the embodiment of the present invention, the metadata 103 is generated by the video processing device 110 from the second video data 102 synchronized with the first video data 101. This metadata 103 is synchronized with the first video data 101 and holds the feature amount of the second video data 102 while managing it on the time axis. Further, according to the embodiment of the present invention, the position matching the extraction condition in the metadata 103 is searched by the position search unit 122 of the video editing apparatus 120, and the edit list 104 is generated. In accordance with the edit list 104, a video is extracted from the first video data 101 by the video extraction unit 123, and edited video data 105 is generated. That is, according to the embodiment of the present invention, the first video data 101 can be edited nondestructively with the metadata 103 as an intermediate state.

  The embodiment of the present invention is an example for embodying the present invention and has a corresponding relationship with the invention-specific matters in the claims as shown below, but is not limited thereto. However, various modifications can be made without departing from the scope of the present invention.

  That is, in claim 1, the video acquisition unit corresponds to, for example, the video acquisition unit 111, the imaging units 211 and 216, or the video input unit 206. The video analysis means corresponds to the video analysis unit 112 or 217, for example. The metadata generation means corresponds to, for example, the metadata generation unit 113 or the file generation unit 214.

  Further, in claim 3, the first imaging unit corresponds to the imaging unit 211, for example. The second imaging unit corresponds to the imaging unit 216, for example.

  Further, in claim 4, reproduction means corresponds to the reproduction unit 220, for example. An imaging unit corresponds to the imaging unit 211, for example.

  Further, in claim 5, reproduction means corresponds to the reproduction unit 220, for example. The video input means corresponds to the video input unit 206, for example.

  Further, in claim 7, the position search means corresponds to the position search unit 122, for example. The video extraction means corresponds to the video extraction unit 123, for example.

  Further, in claim 9, the video acquisition unit corresponds to, for example, the video acquisition unit 111, the imaging units 211 and 216, or the video input unit 206. The video analysis means corresponds to the video analysis unit 112 or 217, for example. The metadata generation means corresponds to, for example, the metadata generation unit 113 or the file generation unit 214. The position search means corresponds to the position search unit 122, for example. The video extraction means corresponds to the video extraction unit 123, for example.

  Further, in claims 10 and 11, the video acquisition procedure corresponds to, for example, step S911. The video analysis procedure corresponds to, for example, step S912. The metadata generation procedure corresponds to step S913, for example.

  Further, in claims 12 and 13, the metadata acquisition procedure corresponds to, for example, step S933. The position search procedure corresponds to, for example, step S934. The video extraction procedure corresponds to step S923, for example.

  In claim 14, the first video data corresponds to, for example, the first video data 101. The second video data corresponds to the second video data 102, for example. The metadata corresponds to the metadata 103, for example.

  The processing procedure described in the embodiment of the present invention may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program May be taken as

It is a figure which shows the example of 1 structure of the video editing system 100 in embodiment of this invention. It is a figure which shows the 1st structural example of the video processing apparatus 110 in embodiment of this invention. It is a figure which shows the usage condition by the 1st structural example of the video processing apparatus 110 in embodiment of this invention. It is a figure which shows the 2nd structural example of the video processing apparatus 110 in embodiment of this invention. It is a figure which shows the usage condition by the 2nd structural example of the video processing apparatus 110 in embodiment of this invention. It is a figure which shows the 3rd structural example of the video processing apparatus 110 in embodiment of this invention. It is a figure which shows the structural example of the camera integrated imaging device which is one Example of the video processing apparatus 110 in embodiment of this invention. It is a figure which shows the structural example of QuickTime base file format. It is a figure which shows the hierarchical structure in a QuickTime file format. It is a figure which shows an example of the preservation | save format of the file in embodiment of this invention. It is a figure which shows an example of the hierarchical structure of the meta track | truck in embodiment of this invention. It is a figure which shows the example of description of the movie atom containing the meta track | truck in embodiment of this invention. It is a figure which shows the other description example of the movie atom containing the meta track | truck in embodiment of this invention. It is a figure which shows the example of description of the meta track atom in the movie atom of FIG. It is a figure which shows the example of description of the sample description atom ('stsd') of the meta track | truck in embodiment of this invention. It is a figure which shows an example of the data format field in embodiment of this invention. It is a figure which shows the example of description of the stream descriptor atom ('strd') in embodiment of this invention. It is a figure which shows the example of description of the user defined metatype field in embodiment of this invention. It is a figure which shows the bit field structural example of meta ID in embodiment of this invention. It is a figure which shows the bit field structural example of the parameter flag field in embodiment of this invention. It is a figure which shows the example of description of the sample of the media data of the meta track | truck in embodiment of this invention. It is a figure which shows an example of the hierarchical structure of the file containing the metadata in embodiment of this invention. It is a figure which shows the relationship between the source track in the example of FIG. 22, and a meta track. It is a figure which shows the other example of the hierarchical structure of the file containing the metadata in embodiment of this invention. FIG. 25 is a diagram illustrating a relationship between a source track and a meta track in the example of FIG. 24. It is a figure which shows the example of a relationship between the source track and meta track in the embodiment of this invention, and an edit track. It is a figure which shows the example of a description of the edit atom ('edts') in a QuickTime file format. It is the figure which represented typically the edit atom in a QuickTime file format. FIG. 27 is a diagram illustrating a content example of an edit restore tom ('elst') in the example of FIG. 26. It is a figure which shows the other example of a relationship between the source track in the embodiment of this invention, a meta track, and an edit track. It is a figure which shows the example of the content of the edit restore tom in the example of FIG. It is a figure which shows an example of the process sequence by the video processing apparatus 110 in embodiment of this invention. It is a figure which shows an example of the process sequence by the video editing apparatus 120 in embodiment of this invention. It is a figure which shows an example of the process sequence in the position search process (step S930) of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Video editing system 101 1st video data 102 2nd video data 103 Metadata 104 Edit list 105 Edit video data 110 Video processing apparatus 111 Video acquisition part 112 Video analysis part 113 Metadata generation part 120 Video editing apparatus 121 Extraction conditions Reception unit 122 Position search unit 123 Video extraction unit 206 Video input unit 210 Recording unit 211 Imaging unit 212 Video processing unit 213 Video compression unit 214 File generation unit 215 Writing unit 216 Imaging unit 217 Video analysis unit 218 Recording control unit 219 Recording medium 220 Playback unit 221 Display unit 222 Video processing unit 223 Video expansion unit 224 File decoding unit 225 Reading unit 229 Recording medium 301 Imaging unit 309 Recording medium 311 Zoom lens 312 Iris 313 Focus lens 314 Filter 321 Image sensor 322 A / D converter 323 Camera signal processing circuit 324 Detection unit 325 Zoom control unit 326 Angular velocity sensor 329 Camera control unit 330 Video processing unit 341 Video compression unit 342 Compression control unit 351 Recording medium access unit 352 Drive Control unit 360 Operation receiving unit 370 Display unit 390 System control unit 501 Subject 502 Photographer 503 to 505 Viewer 510 Network 520, 531 Video camera device 521, 522, 541 Camera 530 Television device 540 Computer device

Claims (14)

Video acquisition means for acquiring second video data synchronized with the first video data managed in time series;
Video analysis means for analyzing the feature quantity in the second video data;
A video processing apparatus comprising: metadata generation means for generating metadata that holds the feature amount in synchronization with the first video data.   The video processing apparatus according to claim 1, wherein the feature amount is a facial expression included in the second video data. Further comprising first imaging means for imaging the first video data;
The video processing apparatus according to claim 1, wherein the video acquisition unit includes a second imaging unit that captures the second video data simultaneously with the imaging of the first video data. Replay means for replaying the first video data;
The video processing apparatus according to claim 1, wherein the video acquisition unit includes an imaging unit that captures the second video data simultaneously with the reproduction of the first video data by the reproduction unit. Replay means for replaying the first video data;
2. The video processing apparatus according to claim 1, wherein the video acquisition means includes video input means for inputting the first video data reproduced by the reproduction means as the second video data.   The video processing apparatus according to claim 1, wherein the first video data and the metadata are recorded in a media data atom format in a QuickTime format. Search for a position on the time series that matches a predetermined condition after obtaining metadata holding the feature amount in the second video data synchronized with the first video data managed in time series, and search for the result Position search means for generating information;
A video editing apparatus, comprising: a video extracting unit that extracts a portion corresponding to the matching time-series position from the first video data based on the search information.   The position search means acquires management information for managing the metadata, and does not acquire the metadata when the management information indicates that the feature amount is not retained in the metadata. The video editing apparatus according to claim 7. Video acquisition means for acquiring second video data synchronized with the first video data managed in time series;
Video analysis means for analyzing the feature quantity in the second video data;
Metadata generation means for generating metadata for holding the feature amount in synchronization with the first video data;
A position search means for searching a position on the time series that matches a predetermined condition from the metadata and generating the result as search information;
A video editing system comprising: a video extracting unit that extracts a portion corresponding to the matching time-series position from the first video data based on the search information. A video acquisition procedure for acquiring second video data synchronized with the first video data managed in time series;
A video analysis procedure for analyzing a feature amount in the second video data;
A video processing method comprising: a metadata generation procedure for generating metadata that holds the feature amount in synchronization with the first video data. A video acquisition procedure for acquiring second video data synchronized with the first video data managed in time series;
A video analysis procedure for analyzing a feature amount in the second video data;
A program that causes a computer to execute a metadata generation procedure for generating metadata that holds the feature amount in synchronization with the first video data. A metadata acquisition procedure for acquiring metadata that retains feature quantities in the second video data synchronized with the first video data managed in time series;
A position search procedure for searching for a position on a time series that matches a predetermined condition in the metadata and generating the result as search information;
A video editing method comprising: extracting a portion corresponding to the matching time-series position from the first video data based on the search information. A metadata acquisition procedure for acquiring metadata that retains feature quantities in the second video data synchronized with the first video data managed in time series;
A position search procedure for searching for a position on a time series that matches a predetermined condition in the metadata and generating the result as search information;
A program causing a computer to execute a video extraction procedure for extracting a portion corresponding to the matching position on the time series from the first video data based on the search information. First video data managed in time series;
A data structure comprising metadata for holding a feature amount in second video data synchronized with the first video data in synchronization with the first video data,
A computer acquires the metadata, searches a position on the time series that matches a predetermined condition in the metadata, generates a result as search information, and based on the search information, the first information A data structure, wherein a portion corresponding to the matching time-series position is extracted from video data.

Images