JP2008227860A - Device for photographing content - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for photographing content which can create a digest easily by narrowing down the number of metadata below a predetermined number, in the order of priority, thereby narrowing down the number of scenes and clips. <P>SOLUTION: With reference to parameters contained in any one of video image and sound being photographed and recorded or the operational information of a photographic device, a scene information creating section creates scene information by detecting a characteristic scene. A supplementary information adding section adds any one among the type, priority, start time, finish time or representative time as the supplementary information, based on a predetermined rule; a scene list describing section describes the scene information and its supplementary information on a scene list; and a scene sorting section sorts the scene information within a predetermined numbers of pieces, based on the priority from among a plurality of pieces of scene information described on the scene list, after ending recording operation onto an information storage medium. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a content shooting apparatus, and more particularly, in a movie shooting, from the start to the end of clip recording, it is easy to easily delete a digest composed of content from which unnecessary parts are deleted or a scene that is presumed to be important. The present invention relates to a content photographing apparatus that can be created.

  Conventionally, shooting of movies, TV dramas, and the like has been performed based on scenarios (scripts) created based on storyboards and the like. A scenario has a title (name) and is composed of a plurality of scenes. A scene is composed of a plurality of cuts. The director performs according to the scenario, and actors such as actors, actresses, and extras perform according to the scenario description. In live performances and concerts, performances are performed in the order of the scenes shown in the scenario.

  On the other hand, in movies, dramas and the like, it is rare that shooting is performed in the order of the scenes indicated in the scenario.

  The creation and editing of metadata in the prior art will be described. As a metadata input method and editing system, the one described in Patent Document 1 is known. Specifically, when metadata related to content is created or tagged, a keyword extracted in advance from a scenario or the like of the produced content is input by voice. Based on the scenario, dictionary fields are set and keywords are prioritized, and metadata is created by the voice recognition means. According to this method, even when metadata is given at intervals of several seconds, which is difficult by key input, it is possible to assign metadata efficiently by using voice recognition. A scene search can also be performed as a keyword for searching metadata.

  Moreover, what was described in patent document 2 is known as an apparatus which analyzes scenario information. The apparatus includes a scenario information storage unit, a reference information storage unit, a calculation unit, and an output unit. The information storage unit stores scenario information classified for each scene. The reference information storage unit stores reference information regarding data included in the scenario information. The calculation unit extracts data from the scenario information, calculates output information based on the extracted data and reference information, and outputs the output information to the output unit.

The scenario information analysis apparatus configured as described above automatically calculates and outputs output information such as an appropriate shooting schedule based on the data included in the scenario and the reference information related to the data. Information can be analyzed. As a result, in addition to shortening the planning time of the shooting schedule, shooting can be completed quickly by shooting according to the output shooting schedule, so that shooting costs can be reduced.
Japanese Patent No. 3781715 JP 2004-362610 A

  In the conventional apparatuses and methods represented by the above-described Patent Document 1 and Patent Document 2, during the movie shooting, from the start to the end of clip recording, camera work patterns such as pan and zoom, recorded sound, and user Based on the operation or the like, metadata is generated and listed in a characteristic scene.

  A digest composed of related clips can be created using the metadata in the list after the recording is completed. Although it depends on the content, the content digest is generally preferably 1/3 or less of the whole (original shooting amount). However, in the above method, depending on the content, the number of metadata becomes enormous, and more than half of the photographed content may remain in the created digest. In such a case, a long digest that is not a very digest is completed. This is because in the conventional configuration, when a digest is generated from captured content, the number of scenes and clips constituting the digest cannot be reduced.

  In addition, there are many user requests for creating digests from various viewpoints. Specifically, digests that focus on camera work, digests that focus on sound, digests that focus on user button input, digests that fit within the desired length of time, and multiple shooting clips For example, digestion may be performed by selecting only a clip having a large number of scenes (for example, a clip having a high priority). However, the conventional technology cannot automatically generate a user's favorite digest that satisfies such a demand.

  In view of the above problems, the present invention can easily generate a digest with a reduced number of scenes and clips by narrowing the number of metadata to a preset number or less in order of high priority or low priority. Another object of the present invention is to provide a content photographing apparatus that can automatically generate a favorite digest.

In order to achieve the above object, the present invention records content including any one of video, audio, and data in combination with scene information on an information storage medium, and accesses a specific part of the content by referring to the scene information. A device,
Scene information generating means for generating scene information by detecting a characteristic scene with reference to a parameter included in either video / audio to be captured and recorded or operation information of the imaging device;
Means for adding any one of the type, priority, start time, end time, and representative time as auxiliary information to the scene information according to a predetermined rule;
Scene list description means for describing the scene information and its auxiliary information in a scene list;
And a scene selection means for selecting scene information within a predetermined number based on the priority from a plurality of scene information described in the scene list after the recording operation to the information storage medium is completed.

  The scene selection means preferably includes at least one of a high priority scene selection means for selecting scene information in descending order of priority and a low priority scene selection means for selecting scene information in order of low priority.

  The high-priority scene selection unit may select a high-priority scene from the photographed content excluding the scene selected by the low-priority scene selection unit.

The content photographing device includes:
Setting means for the scene information selection method;
Second priority means for calculating a combination of a plurality of scene information described in the scene list description memory and adding a second priority to the scene information by a calculation method determined by the setting means;
Auxiliary information adding means for adding at least one of a start time, an end time, and a representative time of the characteristic scene having the second priority as auxiliary information of the scene information;
Second priority for generating a second priority list by selecting scene information within a predetermined number in descending order of the second priority from a plurality of scene information held as a list in the scene list description memory It is desirable to further include a degree list generating means.

  It is desirable to further include second priority list description means for describing the second priority list in a reference file at the time of reproduction of photographed content.

  It is preferable that the content photographing apparatus further includes a skip unit that refers to the second priority list at the time of reproduction to skip to a point that refers to a start point, a representative point, or an end point of a characteristic scene.

  The content photographing apparatus preferably further includes a scene reproducing unit that refers to the second priority list and reproduces characteristic scene sections in a specified order.

  The content photographing apparatus further includes a telop display unit that superimposes the description of the characteristic scene as a telop on the reproduced video when reproducing the characteristic scene sections in a specified order. desirable.

The content photographing device includes:
Digest method designation means for inputting at least one of priority specification of a characteristic scene to be digested, specification of a characteristic scene type to be digested, specification of a digest time length, specification of a reduction ratio to a content digest,
Means for generating a reference file at the time of reproduction according to the digest generation method designated by the digest method designation means with reference to a second priority list which is auxiliary data of the reference file at the time of reproduction;
It is desirable to further comprise registration means for registering the reproduction-time reference file in the reproduction object list.

  The content photographing apparatus further includes a telop display unit that superimposes the description of the characteristic scene as a telop on the reproduced video when reproducing each characteristic scene section when reproducing with reference to the reproduction reference file. desirable.

The content photographing device includes:
Digest method designation means for inputting at least one of priority specification of a characteristic scene to be digested, specification of a characteristic scene type to be digested, specification of a digest time length, specification of a reduction ratio to a content digest,
File generation means for generating a file as a set of designated characteristic scenes according to the digest generation method specified by the digest method specifying means with reference to the second priority list as auxiliary data of the reference file at the time of reproduction It is desirable to further comprise.

The content photographing device includes:
Digest method designation means for inputting at least one of priority designation of a characteristic scene to be digested, designation of a characteristic scene type to be digested, designation of a digest time length, and designation of a reduction ratio to a content digest,
It further comprises reproduction means for joining and reproducing characteristic scene sections not designated according to the digest generation method designated by the digest method designation means with reference to the second priority list which is auxiliary data of the reference file at the time of reproduction. Is desirable.

The content photographing device includes:
Digest method designation means for inputting at least one of priority designation of a characteristic scene to be digested, designation of a characteristic scene type to be digested, designation of a digest time length, and designation of a reduction ratio to a content digest,
File generation means for generating a file in which shooting sections not specified at the time of reproduction are collected according to the digest generation method specified by the digest method specifying means with reference to the second priority list which is auxiliary data of the reference file at the time of reproduction; It is desirable to further provide.

  The content photographing apparatus refers to the second priority list that is auxiliary data of the reference file at the time of reproduction, thereby performing normal reproduction in a section designated at the time of reproduction, and a photographing period not designated at the time of reproduction is referred to as “reproduction speed. It is desirable to further include a playback means for performing “playback changed from normal” or “playback of video that has been processed from playback video”.

  The content photographing apparatus preferably further includes reproduction display means for performing slow reproduction, high-speed reproduction, skip reproduction, or still image display of a photographed video, and performs the “reproduction with the reproduction speed changed from normal”.

  The content photographing apparatus further includes a video generation means for generating a video obtained by removing a roll and a sway of a captured video caused by camera work, and performs the “reproduction of a video obtained by processing a reproduced video” Is desirable.

The content photographing device includes:
File generation means for generating a recommended playback reference file composed of a scene having a predetermined priority or a scene having a specific camera work from a playback reference file registered in the playback object list; ,
It is desirable to further comprise a registration means for registering the recommended playback reference file in the recommended playback object list.

The content photographing device includes:
Means for generating BGM at the time of reproduction of the reproduction-time reference file registered in the reproduction object list;
It is desirable to further include changing means for changing at least one of the BGM melody, timbre, and tempo in the vicinity of the characteristic scene change of the reference file at the time of reproduction.

  According to the content shooting apparatus of the present invention, it is possible to easily delete unnecessary portions of a shot content and generate a digest, particularly in video shooting.

(First embodiment)
With reference to FIG. 1, the operation of the content photographing apparatus according to the first embodiment of the present invention will be described. That is, a system model that generates video data, audio data, and metadata on a recording medium (or buffer memory) inside the camera 101 that is a content photographing apparatus, and provides a digest playback function and the like with reference to the metadata. An example will be described.

  In FIG. 1, reference numeral 101 indicates a camera, reference numeral 102 indicates a lens unit of the camera 101, reference numeral 103 indicates a microphone of the camera 101, and reference numeral 104 indicates a photographing target of the camera 101. Note that the imaging target 104 is, for example, a landscape, an animal such as a person or a pet, a car, or a building.

  Reference numeral 114 indicates a metadata input button, reference numeral 105 indicates data captured by the camera 101, and reference numeral 106 indicates an AV stream data file 106 including metadata. Reference numeral 107 indicates metadata such as shooting scene information IS (scene number, cut number, take number, adoption of the recorded take, non-adoption, hold). Reference numeral 109 indicates a remote controller for the camera 101. The user operates the metadata input button 114 and the remote controller 109 to input the metadata 107 to the camera 101. The image sensor used in the camera 101 is preferably composed of a CCD, C-MOS, or the like.

  Reference numeral 108 indicates a data sequence photographed by the camera 101. In the data sequence 108, video data, audio data, and metadata 107 are arranged on the time axis. The metadata 107 is handled as text data in text format, but may be data in binary format. The data sequence 108 includes clips # 1 to # 5 in a specific scene.

  Reference numeral 110 indicates a data sequence in which clip # 1 to clip # 5 are connected by editing. Reference numeral 111 denotes a television that can be connected to the camera 101. Reference numeral 112 indicates a connection cable for transmitting a signal from the camera 101 to the television 111. Reference numeral 113 indicates a connection cable for transmitting a signal from the television 111 to the camera 101. The user can display the content or digest (or summary content) from which unnecessary portions are deleted on the television 111 via the signal cable 112 by operating the remote control 109 at a location away from the camera 101.

  Reference numeral 115 denotes a microphone that detects sound and inputs it to the camera 101 as an audio signal, similarly to the microphone 103. Reference numeral 115 indicates a microphone built in the camera 101. However, the microphone 115 is connected to the camera 101 with a cable or the like to record the sound far away from the camera 101 compared to the case where the microphone 103 and the microphone 117 are directly attached to the camera 101 and record the sound near the camera 101. Used for. As will be described later, the microphone 115 can use an optical sensor instead of the microphone.

  A list display by the television 111 will be briefly described. On the screen of the television 111, the horizontal axis represents the passage of time, and the valid part (valid scene) and invalid part (invalid scene) of each clip are displayed.

The effective part is, for example,
・ Fixed scenes after panning and zooming, and ・ Scenes characterized by voices such as cheers and applause.
On the other hand, the invalid part is, for example,
-Scenes with large image shake due to camera shake (generally "grabbing"),
・ Out-of-focus scene, pan / tilt / zoom too early,
・ Scenes where the screen is black due to backlight,
・ Scenes with howling,
A scene shooting the ground, and
・ It consists of scenes where the camera cap is closed.

  In the example of FIG. 1, the representative clips of each of the three valid portions in the list display of the television 111 are displayed on the screen as representative thumbnails. This representative clip may be the first frame of each effective portion or a representative frame in the middle of the effective portion. Also, priority is assigned to each valid part and invalid part, and a digest can be generated by selecting only scenes having a specific priority.

  The metadata input button 114 described above is preferably composed of three buttons. When the user operates the metadata input button 114 in an important scene while shooting with the camera, the important shooting scene (clip) can be marked (referred to as “marking function”). The mark indicating the important clip is also metadata 107, and by using this metadata 107, the clip (the video of the head or the representative frame of the clip, or the thumbnail video thereof) marked by the mark search after shooting is used. ) Can be called quickly. The three buttons of the metadata input button 114 are, for example, the first button for registering important clips, the second button for mode switching for enabling button operation or switching to the character input mode, and the like. The first button is used for canceling registration.

  It is also possible to switch to a mode in which the period during which the first button is pressed is registered as an important clip. Further, the mode can be switched to a mode in which 5 seconds before and after the first button is pressed, or a total of 15 seconds, 5 seconds before and 10 seconds after, are registered as important clips. If there are three buttons, many functions can be realized by combining the type of button to be pressed, timing, and pressing length.

  The shooting scene information IS input as the metadata 107 is associated with a clip time code (for example, a time code expressed in 32 bits at a clock frequency of 27 MHz). The metadata 107 associated with the time code is generated as new metadata 107 by electronically associating it with a clapperboard sound or recorded content within the main body of the camera 101. As a result, not only the immediate access to the time when the clapper is sounded but also the deletion of unnecessary recorded data before the time when the clapper is sounded and the rearrangement of scenes and cuts where the recording results are adopted can be easily performed. For example, when shooting an athletic meet, a frame image at the start timing of a long distance competition such as a game (short distance competition), a relay, a tug of war, and a ball insertion can be called immediately.

  In this specification, a content shot during a period from the start of shooting to the end of shooting or from a start of shooting to a shooting pose is defined as a clip. The user can rearrange the clips by designating the start position (time) and end position (time) or length of each clip based on the sequence of the material photographed by the camera. When each clip is displayed on a TV monitor or the like, the most characteristic of a clip, such as a frame (or field) video from the beginning or the beginning to the end of the clip, or a fixed image before or after panning or zooming, etc. A frame can be designated as a video representing the clip.

  In addition, movie button operations such as recording, pause, and stop, and the photographer's voice detected by the microphone 115 can be registered as metadata associated with (marked) a specific time code of the clip. As an example of a photographer's voice, there is meta-information related to a photographing target. Specific examples include the shooting date (date, morning, day, night, etc.), shooting method (lens, camera, shot, light source, etc.), event participants (line of sight, movement, facial expression, tension, makeup, costume status) ), Words (keywords such as ad lib), and other points of interest such as sound, etc.

  Next, the internal configuration and operation of the camera 101 will be described with reference to FIG. Inside the camera 101 are a zoom control unit 201, a focus control unit 202, an exposure control unit 203, an image sensor 204, a shutter speed control unit 205, a camera microcomputer 206, an absolute tilt sensor 207, an angular velocity sensor 208, front / rear / left / right / vertical. Acceleration sensor 209, user input system 210, camera signal processing unit 211, audio processing system 212, H.P. A H.264 encoder 213, a recording medium 214, and an output interface 215 are provided.

  The camera microcomputer 206 includes scene information generation means (index generation means) that detects unnecessary scenes and important scenes. Detection of unnecessary scenes and important scenes is realized by performing specific calculations on data such as pan, tilt, zoom, focus, and audio input level of the imaging apparatus.

  The zoom control unit 201 controls the zoom operation of the lens unit 102. The focus control unit 202 controls the focus operation of the lens unit 102. The exposure control unit 203 controls the exposure adjustment operation of the lens unit 102. A shutter speed control unit 205 controls the shutter speed adjustment operation of the image sensor 204. The absolute tilt sensor 207 detects the absolute tilt of the camera 101 in the horizontal / vertical direction. An angular velocity sensor 208 detects the angular velocity of the camera 101 in the horizontal / vertical direction. The acceleration sensor 209 detects the front / rear / left / right / vertical acceleration of the camera 101.

  The user input system 210 receives a user operation with a button or the like and generates an instruction signal. The audio processing system 212 receives input from the built-in microphone 117, the external microphone 103, or the microphone 115. H. The H.264 encoder 213 detects clapperboard sound from the sound input to the sound processing system 212 and generates clapperboard sound detection metadata.

  The operation parameters of the image sensor 204 include chromaticity space information of three primary colors, white coordinates, and gain information of at least two of the three primary colors, color temperature information, Δuv (delta uv), and three primary colors or luminance signals. There is at least one image sensor operation data for gamma information. In the present embodiment, as an example, chromaticity space information of three primary color points, gain information of R (red) and B (blue) of three primary colors, and gamma curve information of G (green) are handled as metadata. Shall. If the chromaticity space information of the three primary color points is known, the range in which color reproduction in the color space is possible is known. If the gain information of R (red) and B (blue) among the three primary colors is known, the color temperature can be determined. Further, if the G (green) gamma curve information is known, the gradation expression characteristic can be known. Note that a dedicated sensor for detecting color temperature may be provided, and color temperature information may be received from the sensor.

  Lens zoom information, lens focus information, lens exposure information, image sensor shutter speed information, horizontal / vertical absolute tilt information, horizontal / vertical angular velocity information, front / rear / left / right / vertical acceleration information, user's information Input button information, scene number, cut number, take number, information on adoption, non-adoption, hold, etc. of the recorded take, chromaticity space information of the three primary colors, R (red) and B (blue) of the three primary colors Gain information and G (green) gamma curve information are handled as metadata 107 (referred to as camera meta) in the camera microcomputer 206.

  Information (image data) captured by the image sensor 204 is subjected to processing such as pixel defect correction and gamma correction in pixel units or block units composed of a plurality of pixels by the camera signal processing unit 211. After being compressed by the H.264 encoder 213, it is stored in the recording medium 214 together with the camera meta described above. H. The AV output of the H.264 encoder 213 and the camera meta output of the camera microcomputer 206 are output from the output interface 215, respectively.

Next, metadata created in the movie camera will be described with reference to FIG. H. As an example of real-time metadata mapped to SEI of H.264 stream, for example,
・ Metadata associated with AV content shot with a camera such as a movie ・ Metadata generally converted to metadata ・ Metadata obtained from SI (Service Information) of digital broadcasting ・ EPG obtained from an EPG provider Metadata such as information-There is metadata such as EPG obtained from the Internet.

As metadata associated with AV content shot with a camera, for example,
-Button information pressed by the user in an important scene (identification information such as a number can be added)
・ Shooting data Note that the imaging data includes the operation mode of the image sensor, backlight compensation, aperture / exposure information, focus, shutter speed information, color temperature, white balance, zoom, elevation angle, ground shooting, swaying shaking, swaying, panning / Tilt / zoom (abbreviated as PTZ), howling, camera cap closed, and camera posture (horizontal / vertical absolute tilt information, horizontal / vertical angular velocity information, front / back / left / right / vertical) Acceleration information, etc.).
・ Time code (video frame, audio frame)
There are video and audio format information such as shooting frame rate and recording frame rate.

Non-real time metadata includes, for example,
・ Menu information ・ Title list (representative events, events registered by the user)
・ Scene number, cut number, take number, information on adoption / non-adoption / holding of the recorded take ・ Video block brightness, color information ・ Image recognition data (detection and recognition of faces, people, pets, etc.)
・ Voice input level (maximum input level for a specified period of a specified channel)
・ Voice recognition data ・ Chromaticity space information of the three primary color points of the image sensor, white coordinates, gain information of at least two of the three primary colors, color temperature information, Δuv (delta uv) and other operation data of the imaging system Input by communication (text of scenario etc. is input from XML, binary data format file is input from external interface)
-Gamma information of 3 primary colors or luminance signals-Still images-Thumbnails etc. Necessary ones of the above metadata are selected and used. As a description format of metadata, there are a property (property) and an attribute (attribute) introduced in UPnP and UPnP-AV. These are available at http: // upnp. org, and can be operated efficiently to perform design in consideration of the use of text and XML (Extensible Markup Language).

  In addition, in order that a photographer such as a movie, a content creator, or a copyright holder of content adds value to each metadata and collects usage fees according to the usage details and frequency of users who use the content, You can associate metadata that gives value. Metadata giving value to each metadata may be given as an attribute of the metadata or may be given as an independent property.

Hereinafter, an example of information regarding the recording device and the recording condition will be described. If the device manufacturer identification ID, model identification ID, movie photographer, content creator, or content copyright holder creates and registers the value of the metadata and thinks that a license is required, For the use of data, an efficient operation can be performed by incorporating a configuration for executing a process of permissioning use by authentication into the present invention.
In this case, the photographer may create a file in which the captured video content is encrypted, upload the file to a server on the Internet and publish it, or upload and announce the encrypted file. You can make a structure that people can buy. In addition, when content that has news value such as a happening video can be recorded, it can be auctioned between a plurality of broadcasting stations. By using metadata, it is possible to efficiently classify and search for increasing contents.

Next, the following four methods in the video compression method (H.264 / AVC method) and the audio compression method (AAC method) will be described as an embodiment with reference to FIG.
・ Real-time metadata mapping method ・ Unnecessary scene detection from real-time metadata ・ Important scene detection and scene information (called scene index, tag, or metadata) mapping method

  FIG. 4 is a more detailed illustration of the video and audio compression engine and its peripheral processing means in the AV signal compression / recording control unit inside the camera 101 in FIG. 4, reference numeral 401 indicates a video encoding unit, reference numeral 402 indicates a VCL (Video Coding Layer) -NAL (Network Abstraction Layer) unit buffer, and reference numeral 403 indicates an audio encoding unit based on the AAC scheme. ing.

  Reference numeral 404 indicates a PS (Parameter Set) buffer, reference numeral 405 indicates a VUI (Video Usability Information) buffer, reference numeral 306 indicates a SEI (Supplemental Enhancement Information) buffer, and reference numeral 407 indicates a non-VCL-NA. Reference numeral 408 denotes an MPEG-PES packet generator. Reference numeral 409 indicates an MPEG-TS (MPEG Transport Packet) generator, reference numeral 410 indicates an ATS (Arrival Time Stamp) packet generator, and reference numeral 411 indicates an EP-map generator.

  As shown in FIG. 4, the video signal is temporarily stored in the VCL-NA unit buffer 402 after being converted into data in the VCL-NA unit format by the video encoding unit 401. The voice signal, the external input PS data, and the external input VUI data are converted into data of the Non VCL-NA unit format by the voice encoding unit 403, the PS buffer 404, and the VUI buffer 405, respectively, and then the non-VCL-NA unit buffer. It is temporarily held at 407. Similarly, real-time metadata such as pan, tilt, zoom, focus, and audio input level (maximum value of input level for a specified period of a specified channel) of the imaging apparatus is H.264. After being mapped to the SEI user data unregistered SEI message of H.264 / AVC, it is further converted into non-VCL-NA unit format data by the SEI buffer 406 and temporarily stored in the non-VCL-NA unit buffer 407.

  The video signal (4a) is input to the video encoding unit 401 and the face / person detection means 408. The face / person detection unit 408 detects the position / size / number of human faces and outputs detection data (4q) to the metadata generation unit 409 of scene information. The audio signal (4b) is input to the audio encoding unit 403 and the scene information metadata generation means 409. External input PS data (4c), external input VUI data (4d), time code (4e), shooting / recording frame rate (4f), time data (4g) when the user presses the operation button, backlight correction / aperture Data (4h), color temperature / white balance data (4i), focus data (4j), zoom data (4k), gyro sensor yaw / roll / pitch data (4m), elevation angle / ground shot detection data (4n) The state data (4p) of the closing state of the lens cap of the camera is input to the scene information metadata generation means 409, respectively.

  The scene information metadata generation unit 409 includes a howling detection unit 410, an unnecessary scene detection unit 411, an important scene detection unit 412, and a real-time data selection / mapping unit 413. The howling detection means 410 detects howling based on the audio signal (4b).

Here, an example of unnecessary scene detection will be described. The unnecessary scene detection unit 411 detects the metadata (UA, UB, UC, UD, UE, UF, UG) of the unnecessary scene shown in FIG. In addition,
UA represents the state in which the camera cap is closed,
UB represents a state where camera shake and wobble are large,
UC represents out of focus, out of focus,
UD represents pan / tilt / zoom too early,
UE represents the state of being backlit,
UF represents a state in which there is howling, and UG represents a state in which the ground is being photographed.

  The above-stated image represented by the above-mentioned metadata is recognized as an unnecessary scene in almost all countries as long as it is a person who has a common sense or emotion. This is probably because the values for unacceptable things are generally based on human physiological perception and cognitive mechanisms, and are almost universal.

  Specifically, the metadata UA, UB, UC, UD, UE, UF, and UG respectively have a priority (point) of UA (50 points), UB (40 points), UC (25 points), Weighted as UD (20 points), UE (40 points), UF (35 points), UG (25 points).

  In FIG. 7, as an unnecessary scene selection algorithm, an algorithm that handles not only a single unnecessary scene but also treats it as a single continuous unnecessary scene when the interval between a plurality of unnecessary scenes is 60 frames or less is used. . In other words, (single unnecessary scene section) + (a plurality of unnecessary scene sections with an interval of 60 frames or less) can be defined as the “unnecessary scene section definition formula”.

  In addition, when the interval between a plurality of unnecessary scenes is within 60 frames, the reason why these plurality of unnecessary scenes are handled as one unnecessary scene is that if the cut-off video is connected within 60 frames, the video becomes busy and uncomfortable. Because. As the scene information of the unnecessary scene, the type of metadata having the highest priority constituting the unnecessary scene, the time and period of the unnecessary scene (the length of the unnecessary scene), and the marker information of the playlist as the detailed description of the scene information ( It can be used for marker skipping) and auxiliary data associated with the marker information selection algorithm.

  The number of unnecessary scenes can be determined in advance according to the content length. For example, the number of unnecessary scenes can be determined within 5 per minute and up to 100. In addition, it is possible to specify the type of unnecessary scene and the reduction rate for each content. In addition, when a stable video is shot using a tripod or the like, the unnecessary scene detection function can be manually turned off for shooting.

  As an example of unnecessary scene deletion, in FIG. 5, two sections (scenes) that are more shaken than the photographed content are deleted, and three sections (scenes) that are not shaken are combined to form one unswinged video sequence. Show the procedure. It should be noted that unnecessary sections can be deleted by a similar method due to causes other than shaking.

Next, an important scene detection example will be described. The important scene detection means 412 detects the metadata (A, B, C, D, E, F) of the important scene shown in FIG. 8 as follows.
Regarding the metadata (A), an algorithm for extracting a fixed part of 1 to 3 seconds as an important part at the beginning of the clip (CLIP-IN) or the end of the clip (CLIP-OUT) can be created. For example, 100 points.

  Regarding the metadata (B), an algorithm for extracting the sound detection metadata 107 (the metadata portion that detects the input from the main microphone and the sub microphone, the clapperboard sound, the pistol emission sound, the cheer, etc.) as an important part. Can be created). For example, 70 points.

  As for metadata (C), a fixed (still) scene after panning and tilting of the camera can be detected from the output of the gyro. For example, 40 fixed scenes (a portion where the photographer is interested in shooting a fixed video, an Interest portion) are 40 points.

  With regard to the metadata (D), it is possible to detect a zoomed-up and zoomed-down portion by monitoring the temporal change of the zoom value of the camera. More specifically, it is possible to create an algorithm that extracts a fixed part of 1 to 3 seconds before and after zooming up and zooming down as an important part. For example, 30 points.

  Regarding metadata (E), panning and tilting of the camera can be detected from the output of the gyro. For example, 25 points.

  For metadata (F), the captured image obtained from the image sensor is divided into a plurality of blocks, and information on whether the hue and chromaticity of each block are within a predetermined hue and chromaticity range is detected. it can. For example, a human face can be detected from the size and shape of the detection block and the skin chromaticity. Furthermore, it is possible to detect a human face more accurately from the shape and skin color of the detection block of the fixed image after panning, tilting and zooming. ). For example, 50 points.

  It should be noted that the level described above may be slightly different depending on the level of recognition as an important scene for those who have general sensibility and emotions but who are familiar with shooting and camerawork. It is. In other words, people who are accustomed to shooting shoot with camerawork compliant with video shooting techniques, but amateurs do not have such knowledge, so they often shoot subjects without thinking about camerawork. Because. However, it is customary to gradually become familiar with camera work, which is generally considered good, while receiving advice and being able to do it while an amateur is shooting.

  FIG. 6 shows a procedure for constructing one digest by extracting two fixed sections (scenes) after zoom-in and zoom-down from the captured content as an example of extracting an important scene and generating a digest. Note that extraction of important scenes and digest generation due to factors other than zoom-up and zoom-down can be performed in the same manner.

  A, B, C, D, E, and F are priorities (each has a point) and are weighted. In FIG. 8, an unnecessary scene is an important scene represented by one of metadata of A, B, C, D, E, and F, or a video frame, where N is an integer and is within a window of (N) frame. The metadata having the highest priority among a plurality of important scenes present is represented.

For example, when a 300-frame-wide window is used, a “priority point calculation formula” from a plurality of important scenes present in the window is as follows:
Define (priority of the scene with the highest priority within the window of the 300 frame section) + (value obtained by weighting the priority of the other priority scenes with a certain coefficient), and "priority of a plurality of important scenes" Can be calculated. For example, the priority weight of priority scenes other than the highest priority scene is set to 0.2.

In the example shown in FIG. 8, D (30 points), C (40 points), E (25 points), and E (25 points) are included in the important scene detection window. Therefore, the priority total value is
0.2 × 30 + 40 + 0.2 × 25 + 0.2 × 25 = 6 + 40 + 5 + 5
= 56
It becomes.

  Note that the scene information of the important scene includes the type of metadata (A, B, C, D, E, F) having the highest priority constituting the important scene, the time and period of the important scene (the length of the important scene). Is used as detailed description of scene information for playlist marker information (can be used for marker skipping), auxiliary data associated with marker information, and the like. Here, the number of important scenes can be determined in advance according to the content length. In addition, the reduction rate of content due to important scenes can be determined in advance.

  For example, in the example of FIG. 8, the metadata having the highest priority in the window W1 for a certain period is set as the second metadata used for generating the digest. In this case, Interest metadata C representing a fixed state, which is present in about 800 frames, is selected. At this time, according to a rule determined in advance, an event type, an event priority, an event start time, an end time, and a representative time are added to the metadata as attributes. By referring to the second metadata and the attribute start time and end time of the event, and the representative time, a 5 second shooting scene from zoom-up (3 seconds) to fix (2 seconds), for example, can be obtained. It can be expressed by one piece of metadata.

Therefore, a digest video can be generated by designating the order of priority of the metadata or the type of event of the metadata.
Also, by specifying the metadata priority order or metadata event type (for example, zoomed-in part) and referring to the metadata position, a meaningful part ( For example, skip playback (skip playback using metadata) can be performed on the zoomed-in portion.

  In FIG. 8, the second metadata is selected only when the total value of the metadata having priority in the window for a certain period exceeds a preset value (for example, 250). It is also possible to add a mode to set as follows.

  Further, the selection of the high priority scene can be performed more stably by first selecting the high priority scene from the content excluding the low priority scene. For example, even if it is effective as a high priority scene in the fixed part after zooming up, it may be out of focus, and such a low priority scene can be preferentially excluded.

  Similarly, even if it is effective as a high-priority scene in the fixed part after zooming up, the screen may become black due to backlighting, and such a low-priority scene can be preferentially excluded. . In addition, even if the pistol firing sound in the athletic club is detected and effective as a high-priority scene, the zoom-up is too early or panning is too early, so scenes that cannot be viewed are low. Priority scenes can be excluded preferentially.

  As described above, as a first step, a playlist composed of scenes that have been stably photographed excluding low priority scenes is generated. Next, as a second stage, a high priority scene is selected from stable scenes. For example, a high priority scene is added to the playlist created in the first stage, and important scenes are selected from contents that are not unnecessary. Through the above-described steps, it is possible to select important scenes that are more visually stable without shaking blurring.

  Note that unnecessary scene metadata (UA, UB, UC, UD, UE, UF, UG) has a negative meaning of being unnecessary, and may be processed with a minus sign. However, even in this case, the above-mentioned calculation algorithm for unnecessary scenes (processing for putting a plurality of unnecessary scenes into one unnecessary scene) and the calculation algorithm for important scenes (representing a representative scene from a plurality of important scenes, the priority of the representative scene) The process of calculating the degree) is a different process and cannot be realized with the same algorithm.

  A scene having an intermediate value between a high priority scene (important scene) and a low priority scene (unnecessary scene) can be handled as an ordinary scene (general scene).

  The real-time data selection / mapping means 413 maps the above-described real-time metadata to SEI, so that it is possible to recalculate the scene information of unnecessary scenes and important scenes from this data even in another playback device such as a personal computer. it can. This is effective when an algorithm for recalculating scene information of unnecessary scenes and important scenes is updated.

  Based on the VCL-NA unit format data output from the VCL-NA unit buffer 402 and the Non VCL-NA unit format data output from the Non VCL-NA unit buffer 407, an MPEG-PES packet (FIG. 9 ( C)) to generate an 188-byte MPEG-TS (see FIG. 9D). Next, a 4-byte header including a time stamp is added to each MPEG-TS to generate a 192-byte ATS packet (see FIG. 9E).

  This time stamp indicates, for example, the time at which each MPEG-TS packet arrives at a processing block that generates an ATS packet. Generally, the clock of the time stamp is 27 MHz, and when all 4 bytes are used as a time stamp, 30 bits out of 4 bytes are used as a time stamp, and the remaining 2 bits are used as a flag for protecting contents. is there.

  Further, the PTS (Presentation Time Stamp) of the first picture of each GOP (Group of Picture) included in the stream and the serial number of the first ATS in the first picture of each GOP are output as a pair as an EP-MAP. Since PTS and DTS (Decode Time Stamp) are included in the header of the PES packet, extraction is easy.

  The serial number of the first ATS in the first picture of each GOP is a number obtained by sequentially counting the number of ATSs from the stream head, with the serial number of the ATS at the head of the stream being 1. The EP-MAP defined as a pair of PTS and ATS serial number of the first picture of each GOP is used for playback using a playlist and stream editing.

  These H.C. The H.264 / AVC format is described in detail in books such as “H.264 / AVC textbook”, supervised by Okubo Satoshi, published by Impress Corporation. The MPEG-TS signal is defined by IEC 61883-4.

  References related to TS such as PAT and PMT of MPEG-TS include, for example, CQ Publisher, TECH I Vo. 4. “All of image & audio compression technology (essential technology in the Internet / digital television and mobile communication era)”, supervised by Hiroshi Fujiwara, Chapter 6, “MPEG system for multiplexing images and audio” It is explained.

  Also, as an example of the hierarchical structure of PSI and SI, an example of processing procedure, and a channel selection process, “Channel selection technology in a digital broadcast receiver”, Miyake et al., Sanyo Electric Technical Report, VOL. 36, JUNE, 2004, No. 74, pages 31 to 44.

  Next, referring to FIG. An example of the H.264 / AVC file structure will be described. The left square frame is a directory structure, and the right square frame is a file structure, which is configured on an information recording medium such as an SD card, DVD-R, DVD-RAM, or BD-RE.

  In FIG. 9, the square frame on the left shows the directory structure. Under the root, there are a reference file (ref.file) and “PlayList”, “CLIP”, and “STREAM” directories. Under the “PlayList” directory, there is a “* .pls” file that is a playlist (file). In the “CLIP” (clip) directory, a “* .clp” file that is a clip file exists. Under the “STREAM” directory, there is a “* .ats” file that is a stream file composed of ATS (192 bytes).

  In FIG. 9, the right square frame describes the file structure. The index file manages content title information and manages a plurality of chapter information. The playlist manages a plurality of playback part information (Part # (n), where n is a natural number). The clip file has an EP map. An EP map is a cross-reference map table of PTS and ATS sequence numbers of ATS packets that make up a stream file, and performs mutual conversion between time code and data position, and is indispensable for playlist playback and stream file editing. is there.

  As described above, the title is associated with the playlist file, the playlist file is associated with the clip file, and the clip file is associated with the stream file based on the ATS packet.

  The real-time metadata and the non-real-time metadata described above are mapped to SEI and then converted into an ATS stream file. Also, metadata generated by calculating from real-time or non-real-time metadata is mapped as scene information of the clip as additional information to the part information or auxiliary area of the playlist. That is, the list having the priority is mapped as auxiliary data of the playlist file that is referred to when the captured content is reproduced. This has a great feature that the scene information metadata of the clip can be referred to only by looking at the data of the playlist file.

  Therefore, it is possible to immediately access (skip) the start point or the representative point of the event in the photographed content such as an unnecessary scene or an important scene with reference to the additional information of the playlist by the content playback device. In addition, by referring to the metadata list that is auxiliary data of the playlist file that is referenced during playback, the specified event section (event section generated by referring to the start point to end point of the event) is played in order. it can.

  When generating a digest composed of important scenes, a digest method specifying means for inputting the priority of the scene, the specification of the scene type, the specification of the digest time length, or the reduction rate specification to the digest, It is also possible to generate a new playlist file by referring to the list of scenes that is auxiliary data of the playlist.

  Further, by referring to the playlist, the section designated as the important scene is normally reproduced, and the other scenes can be reproduced at a high speed. In addition, by referring to the playlist, the section designated as an unnecessary scene can be played back at high speed, and other scenes can be played back at normal speed. Also, by referring to the playlist, the section designated as an unnecessary scene is played back by displaying the detected representative scene and the still image that has been captured and registered in advance for 3 seconds, and other scenes are played back at normal speed. You can also

  Especially for shooting contents such as children's music concerts, I don't want to see unnecessary images that are shaking or out of focus, but I want to hear performances such as piano and chorus continuously. In consideration of this, it is possible to switch the video to be played back to a representative scene or a video (such as a blue sky, a building, a flower, or a child's face) that has been captured and registered in advance and can continuously reproduce only the audio.

  By referring to the playlist, the section designated as an unnecessary scene has a means for generating a video from which the swaying and shaking of the captured video caused by camera work is removed, and the section designated as an unnecessary scene is It is possible to generate and display a video from which image shaking has been removed by image processing, and to reproduce other scenes at normal speed.

  By referring to the playlist, a playlist composed of scenes having a priority higher than a predetermined value or scenes having a specific camera work may be newly generated and registered in the title.

  By referring to the playlist, a BGM according to the type of each scene is generated and played back, and the BGM melody, tone, and tempo are changed in the vicinity of the scene change to play back more artistic and cultural content. You can also.

  Referring to FIG. The configuration of the H.264 / AVC stream will be described. FIG. The GOP structure of the H.264 / AVC stream. FIG. 10B shows that each picture is composed of VCL and Non-VCL NAL units. NAL (video) is a video NAL unit, NAL (Audio) is an audio NAL unit, and NAL (SEI) is a SEI NAL unit. The above-described real-time metadata can be inserted into NAL (SEI).

  From the experimental results, it is not necessary to insert pan, tilt, lens zoom information, lens focus information, and the like of the imaging device into all pictures in the GOP structure, and even if the time code is thinned out every two frames, The knowledge that if it can be restored, camera work such as panning, tilting, zooming, and focusing at a speed suitable for normal viewing can be restored.

  FIG. 10C shows the structure of the PES packet, and a PES packet header is added to the picture data data shown in FIG. The PES packet header can include MPEG PTS / DTS as a header option. H. From the viewpoint of H.264, a PES packet is handled as 1 AU (Access Unit). In this example, as shown in FIG. 10D, the PES packet is divided every 188 bytes to generate an MPEG-TS packet. FIG. 10E shows that an ATS packet is formed by adding a 4-byte header including a time code to an MPEG-TS packet.

Next, an example of editing the shot content will be described with reference to FIG. First, as the file handling mode,
Mode A: Original shooting content Mode B: Unnecessary scene content (less than 30 scenes)
Mode C: Content without unnecessary scenes (30 or more scenes)
Mode D: Contents of important scenes (50 or more scenes)
There are four types.

There are the following three steps as editing steps. In particular,
Step 1) Take out only the content of the unnecessary scene from the original photographed content file and confirm it. If OK, the unnecessary scene is deleted.
Step 2) Generate content without unnecessary scenes.
Step 3) Further, from the content without unnecessary scenes, content indicated as important scenes by metadata is generated.

  Here, Step 1, Step 2, and Step 3 can be realized only by operating the playlist file without changing the ats file.

  In addition, when the contents of an unnecessary scene are reproduced in mode B, it is possible to notify the user by inserting under what conditions the video is unnecessary in the video.

  Further, content without unnecessary scenes in mode C and content of important scenes in mode D can be created as one complete packet file.

  FIG. 12 is an explanatory diagram of a case where the operation of FIG. 11 is performed on a movie for generating scene information, a personal computer (PC) using the scene information, a DVD recorder, and a television. For example, mode A shown in FIG. 11 is used for movies, mode B, mode C and mode D are used for personal computers, and mode C and mode D are used for DVD recorders and televisions.

  Since the present invention can easily generate contents and digests without unnecessary parts, it can be used for content photographing apparatuses represented by home videos and the like.

Model diagram of content photographing apparatus according to an embodiment of the present invention Explanatory drawing of the internal configuration of the camera shown in FIG. Diagram showing an example of metadata classification Illustration of thin information generation block from metadata Explanatory drawing of the procedure which deletes two areas with larger shaking than a photographed content, and composes a video sequence without shaking by combining three sections without shaking. Explanatory drawing of the procedure which extracts two fix sections after zooming up and zooming down from a photographed content and constitutes one digest Illustration of generation of unnecessary scene information Illustration of generation of important scene information Illustration of directory structure and file structure H. Illustration of H.264 picture structure and conversion method to MPEG-TS Illustration of content editing concept Example of editing content between devices

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Camera 102 Camera lens part 103 Camera microphone 104 Camera photographing target 105 Data 106 photographed by camera 106 AV stream file data 107 Metadata 108 Data sequence photographed by camera 109 Remote control 110 Scenes # 1 to # 5 by editing Data sequence 111 connecting TVs (TV)
112 signal connection cable 113 signal connection cable 114 metadata input button (important scene registration button, still image shooting button)
115, 117 Microphone 201 Zoom control unit 202 Focus control unit 203 Exposure control unit 204 Image sensor 205 Shutter speed control unit 206 Camera microcomputer 207 Absolute tilt sensor 208 Angular velocity sensor 209 Acceleration sensor 210 User input system 211 Camera signal processing unit 212 Audio processing system 213 H.I. H.264 encoder 214 Recording medium 215 Output interface 401 Video encoding unit 402 VCL-NA unit buffer 403 Audio encoding unit 404 PS buffer 405 VUI buffer 406 SEI buffer 407 Non VCL-NA unit buffer 408 Face / person detection means 409 Scene information Metadata generation means 410 Howling detection means 411 Unnecessary scene detection means 412 Important scene detection means 413 Real-time data / selection mapping means

Claims (18)

A content photographing device that records content including any of video, audio, or data in an information storage medium in combination with the scene information, and accesses a specific part of the content with reference to the scene information,
Scene information generating means for generating scene information by detecting a characteristic scene with reference to a parameter included in either video / audio to be captured and recorded or operation information of the imaging device;
Auxiliary information adding means for adding any one of the type, priority, start time, end time, and representative time as auxiliary information to the scene information according to a predetermined rule;
Scene list description means for describing the scene information and its auxiliary information in a scene list;
Content comprising scene selection means for selecting scene information within a predetermined number based on the priority from a plurality of scene information described in the scene list after the recording operation to the information storage medium is completed Shooting device.   2. The scene selection unit according to claim 1, wherein the scene selection unit includes at least one of a high priority scene selection unit that selects scene information in descending order of priority and a low priority scene selection unit that selects scene information in descending order of priority. Content shooting device.   3. The content photographing apparatus according to claim 2, wherein the high priority scene selection unit selects a high priority scene from the shooting content excluding the scene selected by the low priority scene selection unit. Setting means for the scene information selection method;
Second priority means for calculating a combination of a plurality of scene information described in the scene list description memory and adding a second priority to the scene information by a calculation method determined by the setting means;
Auxiliary information adding means for adding at least one of a start time, an end time, and a representative time of the characteristic scene having the second priority as auxiliary information of the scene information;
Second priority for generating a second priority list by selecting scene information within a predetermined number in descending order of the second priority from a plurality of scene information held as a list in the scene list description memory The content photographing apparatus according to claim 1, further comprising a degree list generation unit.   5. The content photographing apparatus according to claim 4, further comprising second priority list description means for describing the second priority list in a reference file at the time of reproduction of photographed content.   5. The content photographing apparatus according to claim 4, further comprising skip means for skipping to a point referring to a start point, a representative point, or an end point of a characteristic scene by referring to the second priority list during reproduction.   The content photographing apparatus according to claim 5, further comprising a scene reproducing unit that reproduces characteristic scene sections in a designated order by referring to the second priority list.   6. The content shooting according to claim 5, further comprising: a telop display means for superimposing a description of the characteristic scene as a telop on the reproduced video when reproducing the characteristic scene sections in a specified order. apparatus. Digest method designation means for inputting at least one of priority specification of a characteristic scene to be digested, specification of a characteristic scene type to be digested, specification of a digest time length, specification of a reduction ratio to a content digest,
Means for generating a reference file at the time of reproduction according to the digest generation method designated by the digest method designation means with reference to a second priority list which is auxiliary data of the reference file at the time of reproduction;
6. The content photographing apparatus according to claim 5, further comprising registration means for registering the reproduction reference file in a reproduction object list.   6. The content shooting according to claim 5, further comprising: a telop display means for superimposing a description of the characteristic scene as a telop on the reproduced video when reproducing each characteristic scene section when reproducing with reference to the reproduction reference file. apparatus. Digest method designation means for inputting at least one of priority specification of a characteristic scene to be digested, specification of a characteristic scene type to be digested, specification of a digest time length, specification of a reduction ratio to a content digest,
File generation means for generating a file as a set of designated characteristic scenes according to the digest generation method specified by the digest method specifying means with reference to the second priority list as auxiliary data of the reference file at the time of reproduction The content photographing device according to claim 5, further comprising: Digest method designation means for inputting at least one of priority designation of a characteristic scene to be digested, designation of a characteristic scene type to be digested, designation of a digest time length, and designation of a reduction ratio to a content digest,
And further comprising a reproducing means for joining and reproducing characteristic scene sections not specified according to a digest generation method specified by a digest method specifying means with reference to a second priority list which is auxiliary data of the reference file at the time of reproduction. Item 6. The content photographing device according to Item 5. Digest method designation means for inputting at least one of priority designation of a characteristic scene to be digested, designation of a characteristic scene type to be digested, designation of a digest time length, and designation of a reduction ratio to a content digest,
File generation means for generating a file in which shooting sections not specified at the time of reproduction are collected according to the digest generation method specified by the digest method specifying means with reference to the second priority list which is auxiliary data of the reference file at the time of reproduction; The content photographing device according to claim 5, further comprising:   By referring to the second priority list, which is auxiliary data of the reference file at the time of reproduction, normal reproduction is performed in a section specified at the time of reproduction, and “reproduction with the reproduction speed changed from normal” is performed at a shooting period not specified at the time of reproduction. The content photographing apparatus according to claim 5, further comprising reproduction means for performing “reproduction of a video obtained by processing a reproduction video”.   The content photographing apparatus according to claim 14, further comprising reproduction display means for performing slow reproduction, high-speed reproduction, skip reproduction, or still image display of a photographed video, and performing the “reproduction with the reproduction speed changed from normal”.   15. The content according to claim 14, further comprising video generation means for generating a video obtained by removing a roll and a shake of a shot video caused by camera work, and performing the “reproduction of a video obtained by processing a reproduction video”. Shooting device. File generation means for generating a recommended playback reference file composed of a scene having a predetermined priority or a scene having a specific camera work from a playback reference file registered in the playback object list; ,
The content photographing apparatus according to claim 9, further comprising registration means for registering the recommended reproduction reference file in a recommended reproduction object list. Means for generating BGM at the time of reproduction of the reproduction-time reference file registered in the reproduction object list;
10. The content photographing apparatus according to claim 9, further comprising changing means for changing at least one of a BGM melody, timbre, and tempo in the vicinity of a characteristic scene change of the reference file during reproduction.

Images