JP2007052626A - Metadata input device and content processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that the time and labor of access and reading is required for performing access along the time records of a recording medium in order to quickly perform access to desired content section, and to perform viewing or editing work as for a medium in which content such as an image or voice is recorded. <P>SOLUTION: When inputting relevant metadata to image data, the complicate extraction of a text input or the like by a keyboard is performed during editing work to easily retrieve an image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

The present invention is suitable for a method of generating and inputting metadata from contents (video, audio, data) that can be acquired at the time of camera shooting, and an editing system that speeds up searching and cueing.

Conventionally, editing of camera-captured content involves selecting and combining original content (video, audio, data) recorded on a master medium (tape, disk, etc.) according to the intention of the content creator. It is a work that requires a great deal of labor and time. The amount of work and the time required for editing work vary greatly depending on the content field and content, such as broadcasting, business, or home. When editing news and sports programs in the broadcasting field, it takes a lot of effort to extract video content in seconds from material tapes, and there is a problem with the credibility of whether the extracted content is the most suitable scene. Remains. Furthermore, when taking a picture of a specific person following sports or athletic meet, there is a problem that the subject to be shot out of the viewfinder if the subject moves quickly or the photographer cannot concentrate on the shooting task. Also, since there is no established method for automatically adding keywords related to shooting scenes (scenes) as tags, or for adding tags with simple actions, you can immediately access the scene you want from shooting content, or It is difficult to find and watch instantly. Not only viewing but also editing, it takes a lot of work to grasp the whole picture of the shot content, and as a result, it can be used in some fields such as broadcasting where editing can cost content production. It has been restricted.
Conventionally, the metadata input method described in Patent Document 1 is known. In order to provide program additional data relating to each scene of the recorded video program, information for identifying the corresponding scene is input from the input form.
As a conventional editing support system, the one described in Patent Document 2 is known. This is metadata that shows various information related to recording. Using the metadata that is serial number, cassette number, and comment, character information related to the recorded video is obtained, and thus the character string is searched and desired. It is a system that makes editing work more efficient by searching for the video of the scene.
Therefore, a metadata input method and an editing system described in Patent Document 3 are known. First, characters included in an image recorded at the time of recording are detected, character recognition is performed on the detected characters, character data is generated as metadata, and is attached to the recorded image data. And, by using metadata automatically generated at the time of recording in editing work, labor and time required for inputting metadata are not required. In addition, since it is possible to recognize text data written in clappers and memos, it is easy to give information directly related to editing as metadata, so that the metadata can be used as it is for editing work.
JP 2002-152694 A JP 2001-292407 A JP 2005-39534 A

However, in the conventional method described above, the metadata must be input manually by an input form while watching the video after recording. Therefore, there is a problem that labor and time for input are required. In addition, when the metadata added to the image generates a character string signal from the serial number, cassette number, etc., there is a problem that it is difficult to pinpoint information necessary for editing directly. . Further, there is a problem that metadata cannot be directly generated from information other than characters, voice, a person or an object.

Recognizes images of people and objects included in video (contents of video, audio, and data) recorded during recording, recognizes audio such as keywords included in audio, converts each recognition result into character data, and metadata Is generated. Then, this metadata is associated with the recorded content. By using metadata automatically generated at the time of recording and metadata generated by a simple operation for cueing and editing, the work efficiency can be greatly improved.

  According to the above invention, a person or an object included in the video of the content (video, audio, data) recorded at the time of recording is recognized, voice such as a keyword included in the audio is recognized, and each recognition result is converted into character data. Convert to generate metadata. Then, this metadata is associated with the recorded content. The metadata automatically generated at the time of recording and the metadata generated by a simple operation can be used for cueing and editing work, and the work efficiency can be greatly improved.

  Furthermore, when a person is recognized by the image recognition means, the clothing worn by the person or the bag he / she holds is registered in the image database and associated with the person, so that inquiries during viewing can be answered. Search can be performed easily.

  In addition, by providing means for automatically recognizing copyrighted items such as music and people and people's movements, you can call up copyright-related items for each scene that makes up the complete package edited from the content material, Display or add metadata such as copyright management source, contact information, usage conditions and copyright fee to each copyright-related item. You can easily create a list of copyright processing required for a complete package. Therefore, the reuse of content is promoted.

  The user can also view the thumbnail images displayed, select the video clip that he / she wants to edit, select the scene to be used in each clip, generate a playlist by changing the order, and perform digest playback, etc. It becomes.

  Further, by providing the playlist output means, it is possible to output the playlist to the outside and output only AV contents according to the playlist from the external device. Thus, it is possible to perform digest playback from a remote playlist.

  Further, by adding the metadata time correction means, it is possible to remove frame or field errors between the playlist and the AV content by reproduction by the playlist. As a result, users such as movie photographers and filmmakers can synchronize metadata and video as expected, improving the editing efficiency of AV signals, and improving the accuracy and sophistication of video expression. be able to.

(Embodiment 1)
FIG. 1 is an explanatory diagram of the present invention, and shows a model of a system that creates video data, audio data, and metadata on a recording medium (or buffer memory) in a camera. Reference numeral 101 denotes a camera, reference numeral 102 denotes a camera lens unit, reference numeral 103 denotes a camera microphone, and reference numeral 104 denotes a camera object (landscape, an animal such as a person or a pet, an object such as a car or a building). Reference numeral 105 denotes data photographed by a camera, and is composed of video data 106, audio data 107, and metadata 108. Reference numeral 109 denotes a data sequence photographed by the camera, and video, audio, and metadata are arranged on the time axis. The metadata is handled as text data in text format, but may be data in binary format.

  Here, the data sequence 109 includes the extracted scene # 1 to scene # 5. Reference numeral 111 denotes a data sequence in which scenes # 1 to # 5 are connected by editing. The user can display a list of scenes on the TV 112 in the order of the edited data sequence by remote control using the remote controller 110.

  Reference numeral 113 denotes a metadata input button, which includes three buttons. By pressing the metadata input button at an important scene while shooting with the camera, the important shooting scene (scene) can be marked (marking function). The mark indicating the important scene is also metadata, and a scene with a mark can be called by mark search after shooting. Three buttons, for example, the first button is for registering important scenes, the second button is for mode switching to enable button operation or switch to character input mode, and the third button is for canceling registration , Use each. It is also possible to switch to a mode in which the period during which the first button is pressed is registered as an important scene. Furthermore, the mode can be switched to a mode in which 5 seconds before and after the first button is pressed, or a total of 15 seconds, 5 seconds before and 10 seconds after, is registered as an important scene. If there are three buttons, it can be used for many functions depending on the combination of the type, timing, and length of the button to be pressed.

  Here, the time length of the scenes from # 1 to # 5 is arbitrary. The user can rearrange the scenes by selecting the start position (time) and end position (time) or length of each scene from the data sequence that is the photographing material taken by the camera. When each scene is displayed on a TV monitor or the like, the frame (or field) video at the head or the head after the head of the scene can be represented as a video representing the scene.

  FIG. 2 is a diagram for explaining how the camera 101 handles video signals, audio signals, and metadata. 201 is a video signal and audio signal input means from the camera, 202 is a voice and image recognition unit means, 203 is a voice recognition means, 204 is an image recognition means, 205 is information about the voice of the user and the importance set by the user. Reference numeral 206 denotes camera sensor information input means. Reference numeral 207 denotes metadata generation / synchronization / management means for generating metadata, synchronizing with video and audio, and managing metadata.

  208 is MPEG-2 compression or H.264. This is an AV signal compression / recording control means that performs H.264 compression, converts it into a format for recording on a recording medium, and records it on the recording medium. Reference numeral 209 denotes a recording medium that also operates as a buffer memory. The AV data file directory 210 includes AV data files, the title list / playlist / navigation data file directory 211 includes title lists / playlists / navigation data files, and metadata. It has a metadata file directory 212 containing files.

  Reference numeral 213 denotes a dictionary group which includes a plurality of field-specific dictionaries (dictionary A, dictionary B, dictionary C). 214 is an addition / deletion management unit for dictionary registration data, 215 is an image recognition unit for specifying a person or object existing in an image included in a video signal, and 216 is an image recognition database (describes characteristics of a person, an animal, or an object) (Database) 217 is an AV signal reproduction management means, 218 is a clock, 219 is a management control means, and 220 is an AV signal output means.

  The operation of FIG. 2 will be described. In FIG. 2, the video signal and the audio signal photographed by the camera 101 are respectively input to the AV signal input means 201. The video signal and the audio signal input to the AV signal input unit 201 are divided into a plurality of systems and buffered (temporarily held), and then respectively to the audio and image recognition unit unit 202 and the AV signal compression / recording control unit 208. Is output.

  The voice and image recognition unit means 202 includes a voice recognition means 203 and an image recognition means 204 for detecting an image, and recognizes an input voice and detects an image included in the video.

  Here, the voice recognition unit 203 selects an arbitrary dictionary from the plurality of dictionaries A, B, C, etc. in the dictionary group 213 by a user button input or the like, and the word group registered in the selected dictionary Perform voice recognition using data. Examples of the dictionary A, dictionary B, and dictionary C include the vocabulary and the number of words for registered words set for each sports field such as baseball, soccer, basketball, or for events such as athletic meet, birthday party, wedding ceremony, etc. Select a dictionary that is narrowed down by selecting content suitable for the writing field. Therefore, if a field in which recognition is performed before performing speech recognition is selected, erroneous recognition in the speech recognition operation can be reduced and the recognition rate can be improved. In addition, each dictionary can add and delete the field-specific dictionary itself, and can add and delete registered words in each field-specific dictionary via the dictionary registration data addition / deletion management unit 214. For example, parents can add the names of their children and acquaintances' children to athletic meet dictionaries. The children's names are converted into text by speech recognition and associated with video as metadata (also known as linking) ) By recording, by specifying the name of the child at the time of playback (for example, selecting from the registered metadata list displayed on the TV screen at the time of playback), it is possible to quickly access (quick access) the image showing the child . As described above, in speech recognition, the recognition speed and accuracy can be improved by performing two steps of selecting a field and registering a keyword in a dictionary narrowed down by field.

  The image recognition means 204 for detecting an image detects and recognizes a meaningful image in a video frame or a video field (abbreviated as video) whose video signal constitutes one picture. The meaning of an image in the present embodiment is a meaningful object image in a video whose video signal constitutes one picture. The image recognition means 204 includes a person, a human face, an animal (a pet such as a dog or a cat), a vehicle such as a car, a train or an airplane, a building such as a house or a building, or a sign as a meaningful object in the video. There are road views, tourist attractions, countryside, mountains, and town scenery. Information about these meaningful objects in the video is input from an image recognition database 216 describing the characteristics of people, animals, and things. For example, when recognizing a human face, a human face in a video (video frame or video field) is recognized, and an area where the human face exists in the video is recognized as, for example, a square area or a round area.

  For example, a square area or a round area where the recognized human face exists is attached with metadata such as “non-specific person number 1” and “non-specific person number 123”. It should be noted that non-specific person numbers can be reduced by assigning the same non-specific person number to the recognition area related to the face of a person who is recognized as the same person in successive images. In addition, the non-specific person number can be reduced by adding an additional function for assigning a non-specific person number only when it is detected over a certain time such as 1 second or 3 seconds. As a result, it is possible to eliminate the recognition of a person who seemed to flicker only for a part of the time against the photographer's intention.For the timing of generating metadata, when the metadata creation button is pressed by the user, It is also good. Furthermore, the non-specific person number can be further reduced by adding a function of adding the non-specific person number only when the size is larger than a specific size depending on the position on the video screen. For example, if the number of pixels is VGA size (horizontal 640 pixels, vertical 480 pixels), in the middle of the screen (horizontal 320 pixels, vertical 240 pixels), detect only when the facial area is vertical, horizontal 60 pixels or more, In the area at the edge of the screen, there is a method of detecting when the face area is vertical and horizontal 40 pixels or more. Thereby, the detection speed (calculation speed) and accuracy of the person recognized in the middle of the screen can be increased.

  As described above, the voice and image recognition unit means 202 inputs text information obtained by voice recognition and image recognition to the metadata generation / synchronization / management means 207 as metadata. The metadata generation / synchronization / management unit 207 receives time information from the clock 218 and performs time management (time synchronization) of video, audio, and metadata in cooperation with the AV signal compression / recording control unit 208.

  The information input to the metadata generation / synchronization / management means 207 includes not only the metadata input from the voice and image recognition unit means 202 but also from the important scene setting button or the still image acquisition button attached to the camera. There are metadata from user information input means 205 that accepts button input information, and sensor information input means 206 that accepts various parameter inputs representing the operating state of the camera. Here, examples of parameters representing the operation state of the camera include a position using GPS or a position sensor or an acceleration sensor of a mobile phone, information on the direction and inclination (elevation angle) of the camera, and a lens 102 used by the camera 101. Exposure information such as the type of lens, zoom magnification, and aperture.

  The video signal, audio signal, and various metadata input to the AV signal compression / recording control unit 208 are associated with each other with the associated video signal, audio signal, and metadata. This linking is performed based on time information (time code, which may be a video frame or a video field for accuracy) in which video, audio, and metadata are generated, but data position information in a stream or file is used. The association (association) may be performed.

  AV signal compression / recording control means 208 uses MPEG-2 (or ITU-T standard H.262) and H.264 as video compression. A compression engine of the H.264 / AVC system and MPEG-2 AAC (Advanced Audio Coding) and MPEG-1 Layer 3 (MP3) compression engines are incorporated as audio compression. The compression engine is not limited to these, and the DV (IEC 61834) method, the DVCPRO method (SMPTE 314M), the DivX Video method (www.divx.com), the XviD method, the WMV9 (Windows (registered trademark) Media Video 9) method. (Www.microsoft.com)) and other types of compression engines can also be selected. In this embodiment, H.264 is used as video compression. Assume that the H.264 / AVC format (hereinafter abbreviated as AVC) and MPEG-2 AAC (hereinafter abbreviated as AAC) are selected as audio compression.

Here, the AVC method and AAC method selected in this embodiment will be described. FIG. 3 is a diagram for explaining in more detail the configuration of the video and audio compression engine and its peripheral processing means in the AV signal compression / recording control means 208 of FIG. As typical constituent elements in FIG. 3, a video encoding unit 301, a VCL (Video Coding Layer) -NAL (Network Abstraction Layer) unit buffer 302, an AAC-based audio encoding unit 303, a PS (Parameter Set) buffer 304, A VUI (Video Usability Information) buffer (305), an SEI (Supplemental Enhancement Information) buffer 306, a non-VCL-NAL unit buffer 307, an MPEG-TS mapping processing means 308, and the like. As shown in FIG. 3, the input video signal is converted into data in the VCL NAL unit format. Also, audio signals, metadata, external input PS (Parameter Set) data, external input VUI (Video Usability Information) data, and external input SEI (Supplemental Enhancement Information) data are converted into data in the Non VCL NAL unit format. These VCL NAL unit format data and Non VCL NAL unit format data are converted into MPEG-2 TS format and output. H. The explanation about the H.264 / AVC system includes, for example, “H.264 / AVC textbook”, supervision by Satoshi Okubo, and Impress Co., Ltd. MPEG-TS (Moving Picture Experts Group, Transport Stream) signals are defined in IEC 61883-4. MPEG-TS is a collection of a plurality of MPEG transport packets (abbreviated as TS packets). The TS packet is a 188-byte fixed-length packet whose length is consistent with the ATM cell length (of 53 bytes, the ATM payload is 47 bytes), and applicable when performing error correction coding such as Reed-Solomon codes. Has been determined in consideration of.
The TS packet includes a 4-byte fixed-length packet header, a variable-length adaptation field (adaptation field), and a payload (payload). PID (packet identifier) and various flags are defined in the packet header. The type of TS packet is identified by this PID. Adaptation_field and payload can be either only one or both, and the presence / absence can be identified by a flag (adaptation_field_control) in the packet header. The adaptation_field has information transmission such as PCR (Program_Clock_Reference) and a stuffing function in the TS packet for making the TS packet have a fixed length of 188 bytes. In the case of MPEG-2, the PCR is a time stamp of 27 MHz, and the PCR value is referred to reproduce the reference time at the time of encoding by the STC (System Time Clock) of the decoder. The clock of the time stamp added to each TS packet is, for example, equal to the MPEG system clock frequency, and the packet transmitting apparatus further receives the TS packet, and from the time stamp added to the received TS packet, A clock recovery means for recovering the MPEG system clock by removing transmission jitter added to the Program Clock Reference (PCR) by network transmission is provided.
In MPEG-2 TS, the STC of the decoder has a PLL motive function by PCR. In order to stabilize the PLL synchronization operation, the PCR transmission interval is determined to be within 100 msec in the MPEG standard. An MPEG-PES packet containing individual streams such as video and audio is divided into a plurality of TS packet payloads having the same PID number and transmitted. Here, the head of the PES packet is configured to start from the head of the TS packet.
Since a transport stream can be transmitted by mixing a plurality of programs, table information indicating the relationship between a program included in the stream and program elements such as video and audio streams constituting the program is included. Used. This table information is called PSI (Program Specific Information) and uses tables such as PAT (Program Association Table) and PMT (Program Map Table). PSI such as PAT and PMT is transmitted by being arranged in a payload in a TS packet in units called sections.
Since PAT specifies the PID of the PMT corresponding to the program number, and the PMT describes the video, audio, additional data, and PID of the PCR included in the corresponding program, refer to the PAT and PMT. Thus, TS packets constituting the target program can be extracted from the stream. References regarding TS include, for example, CQ Publisher, TECH I Vo. 4. “All of image & audio compression technology (essential technology in the Internet / digital television and mobile communication era)”, supervised by Hiroshi Fujiwara, Chapter 6, “MPEG system for multiplexing images and audio” It is explained.

  Regarding the logical hierarchical structure regarding PSI and SI, examples of processing procedures, and examples of channel selection processing, “Channel selection technology in digital broadcast receivers”, Miyake et al., Sanyo Electric Technical Report, VOL. 36, JUNE 2004, No. 74, pages 31-44.

Incidentally, the metadata is input to the SEI buffer 306. Here, the metadata is stored in the User Data Unregistered SEI of the SEI. As types of metadata, in addition to the metadata described above, metadata obtained by converting general data into metadata, or metadata obtained from SI (Service Information; program arrangement information) by receiving digital broadcasting, Metadata such as EPG information obtained from EPG providers, metadata such as EPG obtained from the Internet, metadata associated with AV content (movies such as still images, audio, and clips) taken by individuals There is. As the metadata format, for example, there are properties and attributes as standard specifications of UPnP and UPnP-AV, such as http: // upnp. org and can be expressed in a description language such as XML (Extensible Markup Language) or BML (Broadcast Markup Language). http: // upnp. In org, for example, “Media Architecture V 1.0”, “Content Directory: 1 Service Template Version 1.01”, “MediaServer V 1.0 and MediaRenderer V 1.0”, “Media“ V ” "MediaRenderer V 1.0", "ConnectionManager V 1.0", "ContentDirectory V 1.0", "RenderingControl V 1.0", "AVTransport V 1.0", "UPnP ^TM AV Architecture V. 83", etc. The specification is published. As for metadata standards, there are metadata formats determined by EBU P / Meta, SMPTE KLV, TV Anytime, MPEG7, etc. The standardization trend of metadata for
In addition, in order for photographers such as movies, content creators, or content copyright holders to add value to each metadata and collect usage fees according to the usage and frequency of users who use the content, You can associate metadata that gives value. The metadata giving value to each metadata may be given as an attribute of the metadata or may be given as an independent property. For example, information related to the recording device and recording conditions, that is, the device ID of the movie, the photographer of the movie, the content creator, or the metadata created and registered by the content copyright holder is highly valuable and requires a license. In the case of thinking, the use of the metadata can take a configuration in which a configuration for executing a process of permission for use by authentication is incorporated in the present invention.
For example, a file obtained by encrypting moving image content shot by the user is created, and the encrypted file is uploaded to a server on the Internet. The description of the encrypted file and some images can be made public so that those who like it can purchase it. In addition, when a valuable news source can be recorded, it is possible to adopt a configuration for auctioning among news departments of a plurality of broadcasting stations.
By using these metadata, it is possible to search for desired content from many AV contents, classify it into a library, increase recording time, perform automatic display, sell content, and use content efficiently. It becomes possible. In order to extend the recording time, the resolution of low-value video content should be reduced, or only audio and still images (for example, MPEG I picture or H.264 IDR picture may be extracted). It can be realized by taking a configuration such as only a still image.

  Now, returning to FIG. The MPEG-TS signal generated by the AV signal compression recording control unit 208 is recorded (or temporarily stored) in the AV data file directory 210 in the recording medium (or buffer memory) 209. Here, by using a semiconductor memory, optical disk (DVD-RAM, DVD-R, BD, etc.), HDD (hard disk drive) as the recording medium (or buffer memory) 209, quick access becomes possible. It is easy to modify or add some data, for example, metadata. The title of the MPEG-TS signal is recorded in the title list / playlist / navigation data file directory 211 in the recording medium 209 by the AV signal compression / recording control means 208. Further, the metadata of the MPEG-TS signal is recorded in the metadata directory 212 in the recording medium 209 by the AV signal compression / recording control means 208.

  Next, a method for identifying a person who is an image detected by the image recognition unit 204, for example, a person (person) face, in the AV data file recorded on the recording medium 209 will be described. In the metadata file 212, meta information indicating which image (video frame or video field) of which AV data file recorded in the AV data file the image detected by the image recognition means 204 is located at. Is recorded. The AV signal reproduction control means 217 receives information on the data position for identifying a person from the metadata file and calls the corresponding image data from the AV data file. Then, the called image data is input to the image recognition means 215 for identifying a person. The image recognition means 215 uses the image recognition database (database describing the characteristics of people, animals, and objects) 216 to determine who is referred to and adds the determination result to the metadata file. To do. As the determination result of this person, when the original metadata is the above-mentioned “non-specific person number 123”, a person determination result, for example, “Jiro Tanaka” is added. If the person name confirmed later by the user and the person's face are wrong, the user can correct the name to “Ichiro Tanaka” through the button input of the management control means 219. In addition, even with button input, character input can be realized as is common in recent mobile phones. Further, the number of persons in one image (video frame or video field) is not limited to one person, and any number of persons can be detected as long as they are equal to or larger than the minimum size of the detection area. That is, a plurality of person names such as “Ichiro Tanaka”, “Ayuko Suzuki”, and “Natsuko Kato” can exist simultaneously in one image. Depending on the setting of the metadata generation / synchronization / management means 207, the number of persons in one video (video frame or video field) is specified by a specific number, for example, 5 for a specific file or scene in the file. Can be limited to people. Thereby, it is possible to reduce the data capacity of person data and the processing load.

  It should be noted that the object recognized by the image recognition means 215 is not a human face, but refers to each database file in the image recognition database 216 (a database describing the characteristics of a person, animal or object) such as characters, animals and cars. Recognize and identify the image and add it to the metadata file.

  For example, if you recognize a person, you can register his / her clothes, ties, glasses, hats, watches, shoes, bags and bags you have in the image database and associate them with that person, A search corresponding to the inquiry can be easily executed. In this case, a person can be a UPnP property, and clothes, ties, glasses, hats, watches, shoes, and a bag or bag that the person is wearing can be defined as attributes of the person property.

  In addition, by providing means for automatically recognizing copyrighted items such as music, people and people's movements, the work of each scene that composes a complete packet (completed packet, completed content) edited from the content material Rights-related items can be called out and displayed, and metadata such as copyright management source, contact information, usage conditions and copyright fees can be added to each copyright-related item. This makes it possible to prepare a list of necessary copyright processing for the editing material and the complete package if copyright processing is required for the photographed content. Therefore, the reuse of content is promoted.

  When reproducing a video from the recording medium 209, the user accesses the AV signal reproduction control unit 217 from the management control unit 219, and selects a reproduction file from the recorded file title. When the user cannot specify a file to be reproduced from a plurality of AV files, the user inputs a search keyword from the user interface (user button input) of the management control means 219 and makes an inquiry to the AV reproduction control means 217. The AV playback control means 217 searches the title list / playlist / navigation data file directory 211 and the metadata file directory 212 for metadata that matches all or part of the input keyword, and the result is, for example, text information. And the thumbnail of the corresponding video are superimposed on the output video of the AV signal output means 220. As a result, the user can check the search result on the TV screen as a pair of text and thumbnail. Here, for the notification of the search result to the user, a general display such as a finder attached to the camera may be used.

Now, after specifying the AV data to be reproduced from the title list and the metadata search result, the user calls and reproduces the target AV file. When a specific scene in the AV file is immediately accessed, the user can immediately access and play the scene to be accessed by using the metadata associated with the AV signal via the management control unit 219. can do.
(Embodiment 2)
Next, a second embodiment of the present invention will be described. In the following, description of the same parts as those in the first embodiment will be omitted, and only different parts will be described. FIG. 4 is an explanatory diagram of the second embodiment. In FIG. 4, a playlist generation / management means 401 is newly added.

  The playlist generation / management means 401 generates a new AV file by selecting arbitrary AV signal parts from a plurality of AV files and freely combining (editing) them. For this generation method, the management control means 219 uses a time axis based on the scene specified by the metadata for the file specified by the button input by the user and the scene that the user considers important but not specified by the metadata. Accordingly, thumbnails are displayed on the output signals of the AV signal output means 220 (see 112 in FIG. 1). The thumbnail is the head image (or thumbnail representative image) of a video clip having a specific length such as the length specified by the user. The user looks at the displayed thumbnail image, selects a video clip to be edited, selects a scene to be used in each clip, changes the order, and generates a new video file. What is actually generated in this operation is a so-called playlist as a combination of clip cut-out position information from the file, and this playlist is registered in the title list / playlist / navigation data file directory 211. If a playlist is used in this way, a compact file without an extra AV signal can be virtually generated.

  When reproducing a video from the recording medium 209, the user accesses the AV signal reproduction control unit 217 from the management control unit 219 and selects a reproduction file from the recorded file title and playlist. If the user cannot specify the title or playlist of a file to be played from a plurality of AV files, the user inputs a search keyword from the user interface (user button input) of the management control means 219, and AV playback control is performed. Query means 217. The AV playback control means 217 searches the title list / playlist / navigation data file directory 211 and the metadata file directory 212 for metadata that matches all or part of the input keyword, and the result is, for example, text information. And the thumbnail of the corresponding video are superimposed on the output video of the AV signal output means 220. As a result, the user can check the search result on the TV screen as a pair of text and thumbnail. Here, for the notification of the search result to the user, a general display such as a finder attached to the camera may be used.

Now, in addition to the above-described title list and metadata search results, AV data to be reproduced can be identified from the playlist, and after the identification, the target AV file is called and reproduced. When a specific scene in the AV file is immediately accessed, the user can immediately access and play the scene to be accessed by using the metadata associated with the AV signal via the management control means 219. .
(Embodiment 3)
Next, a third embodiment of the present invention will be described. In the following, description of the same parts as those of the second embodiment will be omitted, and only different parts will be described. FIG. 5 is an explanatory diagram of the third embodiment. In FIG. 5, a playlist output unit 501 is newly added. The playlist output unit 501 outputs the playlist generated in the second embodiment and registered in the title list / playlist / navigation data file directory 211.

  When playing back video from the recording medium 209, the user accesses the AV signal playback control means 217 from the management control means 219, and selects a playback file from the recorded file title and playlist. If the user cannot specify the title or playlist of a file to be played from a plurality of AV files, the user inputs a search keyword from the user interface (user button input) of the management control means 219, and AV playback control is performed. Query means 217. The AV playback control means 217 searches the title list / playlist / navigation data file directory 211 and the metadata file directory 212 for metadata that matches all or part of the input keyword, and the result is, for example, text information. And the thumbnail of the corresponding video are superimposed on the output video of the AV signal output means 220. As a result, the user can check the search result on the TV screen as a pair of text and thumbnail. Here, for the notification of the search result to the user, a general display such as a finder attached to the camera may be used.

  Now, after specifying the AV data to be reproduced from the above-mentioned title list, playlist, metadata search result, etc., the user outputs the target playlist from the playlist output means.

  The merit of outputting a playlist in this way will be described with reference to FIG. In FIG. 6, it is assumed that user 1 and user 2 are connected via a network. Regardless of the type of network, it may be an IP-based home network or the Internet.

  Consider a case where the user 2 accesses the remote movie 101 from the remote controller 609 of the local TV 608 via a network. When the user 2 views the recorded file in the movie 101 remotely, if the AV signal can be viewed in accordance with the playlist in the movie 101, the data transfer amount is small because there is no extra data in the playlist file before editing. Thus, the load on the device and the network can be reduced. That is, it is possible to more efficiently view the AV signal edited by deleting unnecessary signals from the raw shooting signal in the movie.

  Also, the user 1 can upload the AV file, metadata regarding the AV file, and the playlist to the server 606 in the network 605, the user 1 local AV recorder 602, and the user 2 local AV recorder 602. Accordingly, another user (for example, user 3) can access the server 606, the AV recorder 602, and the AV recorder 602 to efficiently transmit and receive AV signals according to the playlist.

  Here, if a different playlist is devised than the playlist watched by the user 2 or another user (for example, the user 3), the user 2 or another user (for example, the user 3) will create a new playlist. Is uploaded to the server 606, the AV recorder 602, and the AV recorder 602. Thereby, many edit titles (playlists) can be generated from a limited number of AV signals (contents), and the AV signals (contents) can be viewed from various viewpoints. It is also possible to jointly perform so-called network-type video editing and production via a network.

  Also, another application (usage method) is born by using the playlist. For example, the movie 101 is converted into a low-resolution AV signal by the AV signal reproduction control means 217 and output to the mobile phone 601 together with the metadata, and the video editing is performed using the metadata in the form phone 601 to edit the EDL (or (Playlist) is sent to the movie 101. The TV 603 can access the movie 101, select a playlist, and delete an unnecessary signal from a raw shooting signal in the movie to view a finely edited AV signal.

  Also, by uploading (or downloading) AV signals according to the playlist to the AV recorder 602, the server 605, and the AV recorder 607, users connected to the network are edited with a higher degree of completeness via the network. AV content signals can be viewed efficiently.

  The present invention can also be developed for viewing AV content using a playlist. For example, when the network 605 in FIG. 7 is the Internet, the user 1 publishes a blog (an alias for Blog, Weblog) format on a server 606 on the Internet, and accesses a plurality of users who have registered by accessing the blog site. AV content addition / update information can be notified in RSS (RDF Site Summary) format. Here, the user 1 publishes AV content and a plurality of playlists corresponding to the AV content. For example, a digest version of the AV content, a simplified version, a complete version, a raw content before editing, and the like are attached to each playlist, and a user who views the AV content selects a favorite playlist and selects AV. Content can be viewed. This can be considered as a system in which EPG (Electronic Program Guide) distribution in digital broadcasting is extended to broadcast distribution media on the Internet. The digest version is effective for paying or free content distribution on 1-segment broadcasting or a mobile phone, and the simplified version, complete version, or pre-editing content is effective for paying or free content distribution on the WEB. Since the user 1 can notify many users over the Internet, even a small business or individual can open an Internet-based broadcasting station for audio and video. In addition, the user can view AV content on the Internet using so-called podcasting, which is a mechanism for automatically collecting AV content on the Internet using RSS feeds.

  Further, the movie 101 has a function of connecting to the Internet and acting as a server (movie camera 101 with an IP network connection function). In this case, the user 1 can live-distribute the content being photographed together with the metadata via the Internet. In other words, the user 1 can quickly convert live shooting audio and video by voice recognition, image recognition, and button input, and publish it as an XML document on the Internet via RSS. It can be carried out.

(Embodiment 4)
Next, a fourth embodiment of the present invention will be described. In the following, description of the same parts as those of the third embodiment will be omitted, and only different parts will be described. FIG. 7 is an explanatory diagram of the fourth embodiment. In FIG. 7, metadata time correction means 701 is newly added. The playlist output unit 501 outputs the playlist generated in the second embodiment and registered in the title list / playlist / navigation data file directory 211.

  When playing back video from the recording medium 209, the user accesses the AV signal playback control means 217 from the management control means 219, and selects a playback file from the recorded file title and playlist. If the user cannot specify the title or playlist of a file to be played from a plurality of AV files, the user inputs a search keyword from the user interface (user button input) of the management control means 219, and AV playback control is performed. Query means 217. The AV playback control means 217 searches the title list / playlist / navigation data file directory 211 and the metadata file directory 212 for metadata that matches all or part of the input keyword, and the result is, for example, text information. And the thumbnail of the corresponding video are superimposed on the output video of the AV signal output means 220. This allows the user to check the search results as a text / thumbnail pair on the TV screen. However, if there is a time lag that is not intended by the movie shooter between the metadata and the thumbnail, the search result will be shifted before editing or viewing. Need to be corrected. Therefore, the user corrects (trims) the time lag between the metadata and the thumbnail in units of video frames or fields by inputting a button from the management control unit 219 while viewing the thumbnail for the specified metadata. At this time, the metadata time correction unit 701 corrects the time information (time code or data position information) of the video signal associated with the metadata by the amount designated by the user, and associates the metadata and the video signal with the new time information. As a result, users such as movie photographers and film directors can synchronize metadata and video as expected. Therefore, the editing efficiency of the AV signal can be improved, and the accuracy and sophistication of the video expression can be improved.

  Further, according to the present invention, playlist distribution of broadcast programs can be performed as a new usage method of playlists. For example, when the movie 101 has a built-in TV tuner and records a TV broadcast, the user 1 edits the recorded TV program as described above to generate a playlist. In this case, the user 1 publishes only the recorded program title and playlist to a server on the Internet. When another general user (for example, user 123) is recording the same program as user 1, user 123 downloads the playlist generated by user 1 to record the TV recorded by himself (user 123). The user can watch the program recorded by himself in an unknown story tailored according to the playlist generated by the user 1. For example, it is possible to generate a digest list for sports programs, a headline playback for news, and a playlist that collects only commercials. Here, the issue is time synchronization, but it is possible with current technology to match the clock accuracy of a movie, server or AV recorder within the video frame. For example, in Japanese digital broadcasting, common time information can be generated from a TOT (Time Offset Table) signal defined by the ARIB standard. In the case of analog broadcasting, it can be discriminated from the characteristics of the standard radio wave, the received video frame, and audio.

  Further, the present invention can be applied not only to television recording but also to movie creation, movie content on the Internet, and content creation for mobile terminals.

  The metadata can be attached to the content as text data, the metadata can be attached to the content as binary data, or the metadata can be attached to the content as a watermark.

  Also, the metadata can be concoded as a watermark embedded in the image data, recorded, reproduced, transmitted and received, and then decoded. In the above description, recording and storage on the same medium are taken as an example. However, metadata and video data may be stored separately in two or more associated media. Further, as long as the medium is associated, either metadata only, video data only, or both metadata and video data may be stored.

The present invention is suitable for a method of generating and inputting metadata from contents (video, audio, data) that can be acquired at the time of camera shooting, and an editing system that speeds up searching and cueing.

Model diagram of camera photographing, photographing data editing system based on photographing data and metadata of the present invention Explanatory drawing of 1st Example of this invention H. Explanatory drawing of handling of metadata in H.264 compression Explanatory drawing of 2nd Example of this invention Explanatory drawing of 3rd Example of this invention Model diagram showing an example in which the editing system of the present invention is applied to a network Explanatory drawing of 4th Example of this invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Camera 102 Camera lens part 103 Camera microphone 104 Camera object 105 Data captured by camera 106 Video data 107 Audio data 108 Metadata 109 Data sequence captured by camera 110 Remote control 111 Scene # 1 to scene Data sequence connecting up to # 5 112 TV (TV)
113 Button for entering metadata

Claims (20)

For recording means of content containing either video, audio or data,
A field-specific dictionary selection means for selecting and adding a field-specific tag for classifying the recorded content,
A dictionary of each field selected by the field-specific dictionary selection means;
Speech recognition means for converting speech contained in the content into character data by speech recognition referring to the dictionary, or speech recognition means for selecting external input speech and converting it to character data by speech recognition referring to the dictionary, or At least one means of image recognition means for recognizing a person or object in the video included in the content and converting it into character data;
Means for associating the character data with the content as metadata;
A metadata input device comprising: The field-specific dictionary selecting means specifies the field by speech recognition referring to a dictionary including the field as a word, or specifies the field by an input from a button input or a keyboard input or an external interface input. The metadata input device according to claim 1. 2. The metadata input device according to claim 1, wherein words to be registered in the dictionary are added, replaced, or deleted by button input, keyboard input, voice recognition of voice input from a microphone, or external interface input. . The content recording means includes a voice recognition operation button,
The voice recognition means, a means for attaching an identification flag to the output voice of the voice selection means when the voice recognition operation button is pressed;
2. The metadata input device according to claim 1, wherein a speech recognition operation is performed for speech in a separately designated time range including a position with the identification flag. The voice recognition operation performed on the voice in the specified time range including the position with the identification flag is determined according to the CPU computing capability that the content recording means can assign to the voice recognition means. 5. The metadata input device according to claim 4, wherein the metadata input device is executed asynchronously after recording with respect to the voice at the indicated position. 6. The metadata that is asynchronously executed and recognized after the recording is added, replaced, or deleted by button input, keyboard input, voice recognition of voice input from a microphone, or external interface input. Metadata input device. A human face or animal or object detection means included in the video signal of the content, a human face feature recognition means detected by the detection means, a human face feature data dictionary, and a human face Means for identifying a person from the feature of the person's face extracted by the feature recognition means of the person's face with reference to a dictionary of feature data and converting it to character data; and means for associating the character data with the content as metadata The metadata input device according to claim 1, further comprising: 9. The metadata input device according to claim 8, wherein the human face detecting means adds an individual identification flag to each detected human face. 9. The metadata input device according to claim 8, wherein said human face detection means tracks the detected human face, and adds one identification flag and information on the length of the identification time to the same person. . If the person cannot be specified by the means for specifying the person and converting it into character data, the metadata is stored as a non-specific person, and the person is input by button recognition, keyboard input, voice recognition input from a microphone, or external interface input. 8. The metadata input device according to claim 7, wherein the metadata is registered as a specific person after the data is added. When new person data is input to the human face feature data dictionary, person recognition is performed on the non-specific person or a person having features similar to the features of the new person data. 8. The metadata input device according to claim 7, wherein if it is made, the person information is set as newly recognized person information. The content recording means includes a button for face recognition operation,
The human face detection means includes means for attaching an identification flag to the video when the face recognition operation button is pressed;
8. The metadata input device according to claim 7, wherein a face recognition operation included in a video of a separately designated time range including the position with the identification flag is performed. The face recognition operation performed on the video in the specified time range including the position with the identification flag is determined according to the CPU computing capability that the content recording means can assign to the face recognition means. 13. The metadata input device according to claim 12, wherein the metadata input device is executed asynchronously after recording with respect to the face included in the video at the indicated position. A means for calling metadata generated by the metadata input device according to claim 1 by button input, keyboard input, voice recognition of voice input from a microphone or external interface input, and associated with the metadata. A content processing apparatus comprising means for calling a new video (frame or field) and newly associating the metadata with another video (frame or field). A priority is added to the metadata generated by the metadata input device according to claim 1 or claim 7, and each video having a separately specified length is extracted in descending order of the priority of the metadata, What is claimed is: 1. A content processing apparatus comprising: means for generating content having a length specified by button input, keyboard input, voice recognition of voice input from a microphone, or external interface input. 8. A content processing apparatus comprising: means for generating an edit list using metadata generated by the metadata input device according to claim 1; and means for editing content in accordance with the edit list. . 15. The content processing apparatus according to claim 14, wherein the length of the video associated with the metadata is specified by button input, keyboard input, voice recognition of voice input from a microphone, or external interface input. The content processing apparatus according to claim 15 or claim 16 or claim 17,
A content processing apparatus comprising means for outputting generated content in a file format. Means for generating content title, table of contents information or inclusion metadata information from the edit list for generating the file format content, title of the content, table of contents information, metadata included in the content, or playlist of the content 20. The content processing apparatus according to claim 19, further comprising means for disclosing the information to another user via a network. 20. The content processing according to claim 19, wherein at least the playlist is published on a server on the Internet, and information related to reproduction of AV content is notified to a user accessing the server in an RSS (RDF Site Summary) format. apparatus.

Images