JP2007101945A - Apparatus, method, and program for processing video data with audio - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus, a method, and a program for processing video data with audio that can automatically display data including a voice in the form of characters in forms matching contents of words and scenes. <P>SOLUTION: A video/audio signal analysis part 48 converts a human voice which can be converted into characters from audio data 62 in video data 60 with audio into characters through speech recognition processing and outputs the characters as utterance content information. Further, the video/audio signal analysis part 48 acquires speaker information by acquiring voice feature quantity information, utterance time information, a speaker identifier for identifying a speaker, and position coordinates of the speaker on a screen. A metadata generation part 50 stores the utterance time information, utterance content information, speaker information, voice feature quantity information, etc., in metadata in specified format (e.g. xml). The metadata are saved in a video/audio signal recording part 46 in specified format (e.g. MPEG-2, AVI, etc.). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a video data processing apparatus with audio, a video data processing method with audio, and a program for processing video data with audio, and more particularly to a technique for characterizing audio of video data with audio.

2. Description of the Related Art Conventionally, a technique for displaying a video person and audio in association with each other has been proposed. For example, Patent Document 1 discloses a video display method for displaying voiced data in correspondence with a speaker in a video with high accuracy.
JP 2004-56286 A

  However, the video display method disclosed in Patent Document 1 described above performs face recognition processing of video information to detect the appearance timing at which a speaker of the dialogue appears in the video, and subtitles corresponding to the dialogue based on the appearance timing. Is inserted into video information, and a method for synchronizing video and audio when there is no speaker on the screen has not been disclosed. In addition, in the video display method of the above-mentioned Patent Document 1, it is not possible to recognize a dialog, the contents of a scene, and the atmosphere, and automatically convert characters into a style suitable for the scene for display.

  The present invention has been made in view of such circumstances, and a video data processing apparatus with audio and a video with audio that can automatically display data in which voice is converted into text and a format that matches the contents of the scene. An object is to provide a data processing method and a program for processing video data with audio.

  In order to achieve the above object, a video data processing apparatus with audio according to claim 1, a data acquisition means for acquiring video data with audio including video data and audio data synchronized with the video data, and the audio data Utterance content information generating means for generating utterance content information by converting the text into speech, utterance time information acquisition means for acquiring utterance time information indicating the time at which the audio data is uttered in the video data, the utterance content information and the utterance It is characterized by comprising metadata creating means for creating metadata including time information, and recording means for recording the audio-added video data and the metadata in association with each other.

  According to the video data processing apparatus with audio according to the first aspect, the utterance content information and the utterance time information obtained by converting the audio data included in the video data with audio can be stored as metadata. By using this metadata, it is possible to easily create telops, scripts, scenarios, minutes of meetings, and the like.

  The video data processing apparatus with audio according to claim 2 further comprises speaker identification means for analyzing the video data and the audio data and identifying a speaker who has uttered the audio, according to claim 1, The creating means is characterized in that the utterance content information and the identification information of the utterer are associated and recorded in the metadata.

  According to the video data processing apparatus with audio according to the second aspect, in addition to the above-described operation, the identification information of the speaker who has spoken can be stored as metadata.

  According to a third aspect of the present invention, there is provided the video data processing apparatus with audio according to the first or second aspect, wherein the speaker identifying means analyzes the video data and the audio data and displays the video data of the speaker who has made the voice. The apparatus further comprises speaker position information acquisition means for acquiring position information on the screen, wherein the metadata generation means records the utterance content information and the position information of the speaker in association with each other and records them in the metadata. And

  According to the video data processing apparatus with audio according to the third aspect, in addition to the above-described operation, the position information of the speaker can be stored as metadata.

  The video data processing apparatus with audio according to claim 4 further comprises audio feature quantity acquisition means for analyzing the audio data and acquiring the audio feature quantity according to claims 1 to 3, wherein the metadata generation means Is characterized in that the utterance content information and the audio feature quantity are associated with each other and recorded in the metadata.

  According to the video data processing apparatus with audio according to the fourth aspect, the audio feature quantity can be stored as metadata in addition to the utterance content.

  According to a fifth aspect of the present invention, there is provided the video data processing apparatus with audio according to the fourth aspect, wherein the audio feature amount acquisition unit acquires at least one information of the size, level, inflection, or tone of the audio. To do. Claim 5 lists the audio feature quantities of claim 4.

  A video data processing apparatus with audio according to claim 6 is the information acquisition apparatus according to claim 1, wherein reproduction display means for reproducing and displaying the video data with audio and information for acquiring utterance content information and utterance time information from the metadata. Means, telop creation means for creating a telop based on the acquired utterance content information, and telop insertion means for inserting the telop when reproducing the video data with audio based on the acquired utterance time information. It is characterized by providing.

  According to the video data processing apparatus with audio according to the sixth aspect, it is possible to create a telop from the metadata of the video data with audio and display it when reproducing the video data with audio.

  According to a seventh aspect of the present invention, there is provided a video data processing apparatus with audio according to the third aspect, wherein reproduction display means for reproducing and displaying the video data with audio, utterance content information, utterance time information, and speaker position information from the metadata. Information acquisition means for acquiring, telop generation means for generating a telop based on the acquired utterance content information, and telop insertion for inserting the telop when reproducing the video data with audio based on the acquired utterance time information And insertion position adjusting means for adjusting the insertion position of the telop based on the acquired position information of the speaker.

  According to the video data processing apparatus with audio according to claim 7, the correspondence between the inserted telop and the speaker is easily understood by adjusting the insertion position of the telop based on the position information of the speaker acquired from the metadata. Can be displayed.

  According to an eighth aspect of the present invention, there is provided a video data processing apparatus with audio according to claim 4 or 5, wherein reproduction display means for reproducing and displaying the video data with audio, utterance content information, utterance time information, and audio feature amounts from the metadata. Information acquisition means for acquiring, telop generation means for generating a telop based on the acquired utterance content information, and telop insertion for inserting the telop when reproducing the video data with audio based on the acquired utterance time information And character attribute changing means for changing the character attribute of the telop according to the voice feature amount.

  According to the video data processing apparatus with audio according to claim 8, for example, the font size of a telop corresponding to large (small) audio is increased (small), or the telop font corresponding to strong (weak) audio is used. By thickening (thinning) the telop, it is possible to create a telop rich in expressiveness according to the audio feature amount.

  According to a ninth aspect of the present invention, there is provided the video data processing apparatus with audio according to the eighth aspect, wherein the character attribute changing means includes a font, a font size, a character color, a background color, a character decoration of the telop according to the audio feature amount. It is characterized in that at least one of a column, parentheses, or a symbol attached to the telop, such as a balloon, an exclamation mark, or a question mark, is changed. The ninth aspect lists the character attributes of the eighth aspect.

  The audio-added video data processing method according to claim 10 includes a data acquisition step of acquiring audio-added video data including video data and audio data synchronized with the video data, and utterance content information by characterizing the audio data. Utterance content information generating step, utterance time information acquisition step of acquiring utterance time information indicating a time at which the audio data is uttered in the video data, and metadata including the utterance content information and the utterance time information. It comprises a metadata creating step to create, and a recording step for recording the audio-added video data and the metadata in association with each other.

  An audio-added video data processing program according to claim 11 is a data acquisition function for acquiring audio-added video data including video data and audio data synchronized with the video data; Utterance content information generation function for generating information, utterance time information acquisition function for acquiring utterance time information indicating the time at which the audio data is uttered in the video data, and metadata including the utterance content information and the utterance time information And a recording function for recording the video data with audio and the metadata in association with each other.

  Software and firmware including the audio data processing program according to claim 11 in addition to a personal computer (PC), a video playback device (video deck, television), a digital camera, a mobile phone or other device having a video playback function. By applying this, it is possible to realize the video data processing apparatus with audio and the video data processing method with audio of the present invention.

  According to the present invention, utterance content information and utterance time information obtained by converting audio data included in video data with audio into characters can be stored as metadata. By using this metadata, telops, scripts, scenarios, minutes of meetings, etc. can be easily created.

  Preferred embodiments of a video data processing apparatus with audio, a video data processing method with audio, and a program for processing video data with audio according to the present invention will be described below with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating a main configuration of an imaging apparatus including an audio-equipped video data processing apparatus according to an embodiment of the present invention. An imaging apparatus 10 illustrated in FIG. 1 is, for example, an electronic camera, a digital camera, or a digital video camera having a moving image shooting function.

  The CPU 12 is connected to each part in the imaging device 10 via the bus 14 and is a general control unit that controls the operation of the imaging device 10 based on an operation input from the operation switch 16 or the like. The operation switch 16 includes a power switch, a release switch 16A, a cross key, and the like, and receives an operation input from the user. The release switch 16A is configured in a two-stage manner, and the automatic focus (AF) and automatic exposure control (AE) are activated in the state of “half-press (S1 = ON)” in which the release switch 16A is lightly pressed to stop, and AF and AE Is locked, and shooting is executed in the state of “full press (S2 = ON)”, which is further pressed from “half press”.

  The memory 18 includes a ROM that stores programs to be processed by the CPU 12 and various data necessary for control, an SDRAM that is a work area in which the CPU 12 performs various arithmetic processes, and an image processing area.

  The external communication interface (external communication I / F) 20 is a device for connecting to a network or an external output device (for example, a personal computer, a television, a display, a printer, an external recording device), and the like. Send and receive data. The data transmission / reception method is, for example, the Internet, a wireless LAN, a wired LAN, IrDA, Bluetooth, or the like.

  The imaging element 24 is an element that receives light that has entered through the optical system (lens) 22 and converts it into an electrical signal, and is, for example, a CCD. This electrical signal is amplified by a preamplifier (not shown), converted into a digital signal by an A / D converter 26, and input to the video processing unit 28.

  The imaging apparatus 10 according to the present embodiment includes a plurality of operation modes including a shooting mode for shooting a video (still image and moving image) and a playback mode for displaying and playing back the video. The operation mode is set by the operation input from.

  In the shooting mode, the electric signal output from the image sensor 24 is processed by the video processing unit 28 to generate video data for checking the angle of view (through image), which is displayed on the video display unit (monitor) 30. When shooting a still image, when the release switch 16A is operated to shoot a still image, the electrical signal output from the image sensor 24 is processed by the video processing unit 28 to create still image data for storage. Is done. This storage still image data is stored in the recording medium 32 in a predetermined file format. Here, the recording medium 32 is, for example, a semiconductor memory, a video tape, a hard disk drive (HDD), a DVD, or the like. It should be noted that voice can be input through the microphone 34 and the still image data and voice can be stored in association with each other.

  On the other hand, when shooting a moving image, acquisition of sound is started by the microphone 34 when shooting of the moving image is started by the release switch 16A. Then, the moving image data for storage is generated by the video processing unit 28 and the audio data for storage is generated by the audio processing circuit 36. The video data and audio data for storage are converted into video data with audio in a predetermined file format (for example, MPEG format or AVI format) and stored in the recording medium 32.

  On the other hand, in the playback mode, at the time of playback of a still image, the still image data stored in the recording medium 32 is read by the video processing unit 28 to create still image data for display and displayed on the monitor 30. At the time of reproducing a moving image, the moving image data stored in the recording medium 32 is read out by the video processing unit 28, and the moving image data for display is created and displayed on the monitor 30, and the audio associated with the moving image data is displayed. Data is read out and output from the speaker 38. As described above, the monitor 30 is used as an electronic viewfinder for checking the angle of view at the time of shooting, and is used for displaying the shot video data (still image data, moving image data).

  Next, processing for generating and assigning metadata from audio-attached video data imaged by the imaging device 10 will be described with reference to FIG. FIG. 2 is a functional block diagram showing a flow of video data processing with audio in the imaging apparatus 10. The recording instruction / control unit 40 shown in FIG. 2 is a functional block including a release switch 16A for instructing recording start and the CPU 12. The operation input from the release switch 16A causes the external video / audio input unit 42 and the video / audio input unit 42 from the CPU 12 to operate. A start signal for moving image shooting is output to the audio signal encoding unit 44. The external video / audio input unit 42 is a functional block including the optical system 22, the image sensor 24, and the microphone 34, and the video / audio signal encoding unit 44 is a functional block including the video processing unit 28 and the audio processing circuit 36. is there. The video and audio electrical signals output from the external video / audio input unit 42 include audio data 62 and video data 64 as shown in FIG. 3 by the video / audio signal encoding unit 44 (moving image codec). It is converted into video data 60 with audio in a predetermined format and stored in the video / audio signal storage unit 46 (memory 18, recording medium 32).

  Next, the audio / video signal 60 is read from the video / audio signal storage unit 46, and the audio / data 62 in the audio / video data 60 is extracted by the video / audio signal analysis unit 48. The video / audio signal analysis unit 48 converts a voice of a person that can be converted into characters from the extracted audio data 62 into a character by voice recognition processing, and outputs it as utterance content information. It also recognizes voice feature quantities such as the magnitude, height, inflection, and tone of the voice, classifies the voice into a predetermined texture, and outputs it as voice feature quantity information.

  Further, the video / audio signal analyzing unit 48 acquires the utterance time information in which the above-mentioned characterized sound is emitted. This time information is, for example, an information (frame) number that identifies a frame of video data (moving image) at the start and end of an utterance, an utterance start time and an end time, and the like. Further, the video / audio signal analysis unit 48 analyzes the video data 64 to detect a speaker corresponding to the utterance content, and acquires a speaker identifier for identifying the speaker and a position coordinate of the speaker on the screen. And output as speaker information. The metadata generation unit 50 stores the above utterance time information, utterance content information, speaker information, voice feature amount information, and the like in metadata of a predetermined file format (for example, xml format). This metadata includes information as shown in FIG. 4, and is stored in the recording medium 32 of the video / audio signal recording unit 46 in a predetermined format (for example, MPEG-2 or AVI format).

  FIG. 5 is a diagram showing an example of metadata in xml format, and FIG. 6 is a diagram showing a schema of xml. In the example shown in FIG. 5, in the utterance time information, the start time and end time of the transcribed utterance content are described in the voice tag by a frame number, and the utterance content information is described by a text tag. The speaker information (person) is described in the name attribute by, for example, the name of the speaker (“◯ × Δ male”). In the example shown in FIG. 5, since there is no speaker on the screen, the pos attribute is omitted or blank. In addition to the narration, the type attribute of the voice feature amount information (tone) can be set to, for example, a laughing voice, a crying voice, a loud voice, a secret voice, or the like.

  Next, a metadata storage format will be described. FIG. 7 is a diagram illustrating an example of storing metadata in the MPEG format. As shown in FIG. 7, in the MPEG-2 format, a video stream 64 ′ including video data 64, an audio stream 62 ′ including audio data 62, and a metadata stream 66 ′ including metadata 66 are defined in the standard. (Called a pack, which is a series of data units of 2,048 kb in one example), is recorded as a single file 68 by interleaving.

  FIG. 8 is a diagram illustrating an example of storing metadata in the AVI format. In FIG. 8, “RIFF AVI” indicates the entire AVI file. “LIST hdr” is a header area of the AVI file, and includes two header areas “LIST str” for video and audio. In this embodiment, original extension data streams “strd” and “strn” indicated by thick frames are provided in the header area “LIST strl” for video, and the metadata in the xml format shown in FIG. 5 is included in this stream. Save as binary data. Thereby, metadata can be saved in the AVI file.

  Hereinafter, a method for processing video data with audio will be described with reference to FIG. FIG. 9 is a flowchart illustrating a method for processing video data with audio according to an embodiment of the present invention. First, the video data with audio 60 is read from the video / audio signal storage unit 46, buffered by a certain amount, and analysis of the audio data 62 is started (step S10). In step S10, the data amount of the video data with audio 60 to be buffered can be adjusted. Here, the value of the amount of data to be buffered may be an approximate break in the context when the audio data 62 to be analyzed is converted into text. For example, based on the data that the normal speed utterance in Japanese is about 400-500 words per minute, the amount of data that can contain one syllable is calculated backward, and the initial amount of data to be buffered It may be a value.

  Next, the speech content of the voice data 62 is converted into text by voice recognition (step S12). In step S12, for example, a voice of a person (speaker), an animal voice, surrounding sounds, sound effects, and the like are extracted from the sound data 62 in the video data with sound 60, and a human voice dictionary and sound effect dictionary are created. The extracted human voice data and sound effect data are converted into characters. Furthermore, the voice data is analyzed, the voice is classified for each speaker based on the voice feature quantity such as voiceprint and utterance speed, and stored as utterance content information. Note that the method of characterizing the audio data 62 is not limited to the above.

  Since it is necessary to synchronize the text data (speech content information) with the time information in units of frames, it is further analyzed and decomposed into appropriate punctuation such as one syllable or one sound. The frame number or time information synchronized with the utterance time information is stored as the utterance time information, for example, the utterance time of the utterance content “It was summer” from frame 0 to frame 10 or from 0:00 to 0:05 (See FIG. 5).

  In step S12, if the utterance end time of the transcribed audio data is earlier than the end time of the buffered video data 64 and the remaining video data with the buffer is left, the video data with audio of the next loop is generated. The buffering start position of 60 should be matched with the end time of the voice data that has been characterized this time.

  If the utterance is not included in the audio data 62 analyzed in step S12 (No in step S14), there is no audio data that has been transcribed, so the process returns to step S10 and the processing is continued on the remaining data of the audio-attached video data 60. To do.

  On the other hand, if the speech data 62 analyzed in step S12 includes an utterance (Yes in step S14), the speech data that has been transcribed is converted into speech feature quantities (voice volume, pitch, inflection, tone, etc.) of the speech signal. Tone analysis is performed (step S16). Here, tone analysis refers to categorizing into a tone texture (for example, laughing voice, secret voice, loud voice, etc.) prepared in advance. In step S16, numerical data (size, frequency, etc.) representing the tone of the voice is also recorded.

  Next, the speaker is analyzed based on the voice data that has been converted to text (step S18). In step S18, for example, the video data 64 is analyzed, and a person region in which a person is shown is extracted for each frame of the video data 64. The video feature amount of the person is calculated, and the person is estimated based on the video feature amount. Here, the video feature amount includes, for example, average density, highlight (lowest density), shadow (highest density), histogram, and the like. The estimation of a person is performed by estimating the person group such as the gender, age and occupation of the person. For example, gender estimation is performed by extracting a face region (hair) from the person region and, based on these extraction results, when the volume of the hair region is large, when the hair region is elongated and long hair, or below the torso If the shape of the clothing extracted from the contour shape pattern matching seems to be a skirt, and if the clothing color is red or pink, or if the face area is extracted, the presence or absence of makeup and the use of lipstick It can be presumed that the woman is a woman by comprehensively judging from the presence / absence of the child and the presence / absence of wearing the accessory. Further, the age can be estimated as an adult, a junior high school student, an elementary school student, an infant, or the like by calculating the height of the subject person from the display video and the size. Alternatively, when the volume of the extracted hair region is small or the hair color is white, it is estimated that the person is an elderly person. Moreover, occupation estimation can be performed by clothes, for example. For example, if the suit type is high from the shape, density, and color of clothing, it can be estimated as a salaried worker group, and if it is thought that it is a uniform type from the shape and color of clothing, the estimation result of gender and age It can be estimated that the students include junior and senior high school students. Note that the person layer estimation method described here is merely an example, and is not limited thereto.

  Then, statistics on the occurrence status of timings appearing in the same scene are taken for the number N of person regions estimated from the video data 64 and the number M of persons estimated from the voice of the audio data 62. At this time, when the person layer estimation result based on the video feature amount and the person layer estimation result based on the audio feature amount contradict each other, the statistical processing does not count up. For example, the analysis result of the video data 64 is a case where the audio data is a woman although the man is male. Also, in the video data 64, both male candidates and female candidates are extracted, and when the audio is only female candidates, only the female candidates in the video are counted up. At this time, the movement of the mouth of the person in the video may be detected, and the degree of coincidence with the utterance timing may be used for weighting the degree of coincidence between the video and the audio so as to obtain statistics.

  Then, this statistical processing is performed at regular time intervals and tabulated. As the fixed time interval, for example, an interval of 10 minutes may be actually divided, or if the video data 64 is a TV program recorded, it may be divided within one program or commercial, You may separate them with chapters. From the results of the statistics thus obtained, the association between the utterer detected from the video data 64 and the utterance content is determined based on the combination of the high correlation between the human estimation based on the video and the human estimation based on the voice, and the utterance is determined. Identify the speaker who made the call. At this stage, a contradiction check between the person estimation result obtained by analyzing the video data 64 and the person estimation result obtained by analyzing the audio data 62 may be performed.

  Then, a speaker identifier (for example, female A, old woman A, etc.) consisting of a character string that can specify a speaker such as a person name, gender, and age is assigned to the specified speaker, and the specified speaker belongs to the speaker. It is stored as speaker information including the position coordinates of the person area and the speaker identifier.

  In the present embodiment, a speaker database (DB) is provided in the memory 18, and the face area, person name, nickname, voiceprint, etc. of the speaker are stored in advance in the speaker DB. And the extracted video feature amount of the person may be compared to specify the speaker.

  Next, metadata including the utterance time information, utterance content information, utterer information (speaker identifier and utterer position coordinates), voice feature amount information, and the like is generated (step S20). In step S20, first, metadata is generated based on the utterance content information and the utterance time information, and the speaker information and voice feature information are also described in the metadata.

  Next, when there is unprocessed audio-added video data (Yes in step S22), the process returns to step S10 to continue the processing. Then, when there is no unprocessed video data with audio (No in step S22), the generation of metadata is terminated, and the generated metadata is stored by an appropriate method (step S24). For example, as shown in FIGS. 7 and 8, the metadata may be stored in the same file as the audio-added video data 60 in MPEG-2 or AVI format. The video data 60 with audio 60 may be stored in association with each other as an xml file.

  According to the present embodiment, it is possible to add and store metadata including utterance content information obtained by characterizing voice data. By using this metadata, telops, scripts, scenarios, minutes of meetings, etc. can be easily created.

  Next, the video reproduction function of the imaging device 10 including the above-described audio-equipped video data processing device will be described with reference to FIGS. FIG. 10 is a block diagram showing the main configuration of the video playback function unit of the video data processing apparatus with audio. As shown in FIG. 10, the video playback function section of the video data processing apparatus with audio includes a video / audio signal recording section 46, a playback instruction control section 70, a video / audio signal decoding / playback section 72, a metadata reading section 74, a telop. A generation display unit 76 and an external video / audio output unit 78 are provided.

  The playback instruction control unit 70 allows a user, such as a playback switch for instructing playback of video data, a stop switch for instructing playback stop, a pause switch, a rewind / fast forward switch, a menu switch, a remote controller, etc. An operation member for performing the operation is included, and a control signal is sent to each block of the video reproduction function unit in response to an operation input from each operation member.

  The video / audio signal decoding / reproducing unit 72 reads the video data with audio designated by the operation input from the reproduction instruction control unit 70 from the video / audio signal recording unit 46 and decodes the video signal and the audio signal. The metadata reading unit 74 reads the metadata of the video data with audio designated by the operation input from the reproduction instruction control unit 70 and outputs it to the telop generation display unit 76. The telop generation / display unit 76 reads the utterance content information and the utterance time information from the metadata, and instructs the video / audio signal decoding / playback unit 72 to insert a telop of the utterance content information into the frame corresponding to the utterance time information. Output. The telop generation display unit 76 reads the speaker information from the metadata, specifies the position of the speaker in all frames corresponding to the utterance time information, and outputs a command for designating the insertion position of the telop. Further, the telop generation display unit 76 uses the telop font, font size, character color, background color, character decoration, brackets attached to columns or telops, balloons, exclamation marks, question marks, etc. based on the audio feature information. Outputs a command that specifies. The video / audio signal decoding / playback unit 72 inserts a telop into the decoded video signal based on a command from the telop generation / display unit 76, and outputs it to the external video / audio output unit 78 together with the decoded audio signal. The external video / audio output unit 78 includes an image display unit 30 that displays video, a speaker 38 that outputs audio, a video / audio output terminal, and the like. Play audio.

  Next, the flow of processing for reproducing audio-added video data to which metadata is attached will be described with reference to FIG. FIG. 11 is a flowchart showing a flow of processing for reproducing audio-added video data to which metadata is added.

  First, when video data with audio to be reproduced is selected by the reproduction instruction control unit 70, the metadata is read by the metadata reading unit 74 (step S30). Here, when the metadata is stored and pushed in the same file as the video data with audio as shown in FIGS. 7 and 8, the metadata reading unit 74 reads the metadata from the video data with audio. If the metadata is a separate file from the video data with audio and is recorded in association with the URL or the like, the metadata reading unit 74 associates with the specified video data with audio. Get metadata file.

  Next, from the utterance content information (text tag in FIG. 5) included in the read metadata, the text data of the utterance content is read to generate telop character data (step S32). Further, the speaker information and the voice feature amount information are acquired from the metadata, and the text attribute of the telop is set (step S34). In step S34, for example, the text color of the telop is changed for each speaker (speaker identifier) so that the utterance content of the same speaker can be identified by the telop color. Further, the character attribute is changed according to the voice feature amount. For example, the telop font, font size, text color, background color, text decoration, column change according to the volume, height, and inflection of the voice, or the sign such as parenthesis, speech balloon, exclamation mark, question mark, etc. Attached to the telop. This makes it possible to create an effective telop according to the nature of the voice. In steps S32 and S34, the user may operate the operation switch 16 while looking at the screen to manually correct or add a telop character or change a character attribute setting.

  Further, the insertion position and size of the telop are adjusted according to the speaker information included in the metadata (step S36). For example, if the position of the speaker on the screen is described in the metadata, the insertion position and size of the telop are adjusted so that the person speaks according to the position of the speaker on the screen. . For example, depending on the position coordinates of the speaker, the speaker's dialogue shown on the left is inserted on the left, and the speaker's dialogue shown on the right is inserted on the right. Note that the position of the speaker's face and mouth is stored in the metadata, or is detected by the video data processing apparatus with audio 120, and a speech bubble is displayed near the speaker's face area, and a telop is included in the speech bubble. May be inserted. Further, the telop may be recorded so that the position of another speaker or the size of the person area is recorded in the metadata so that the telop does not overlap another person appearing in the same frame. When the speaker's position coordinates are not included in the speaker information, that is, when the speaker is not in the screen, for example, the background area is detected by video analysis, and the telop is set so that it falls within the background area. It is preferable to calculate the position and size of. In step S36, the user may manually change the insertion position and size of the telop by operating the operation switch 16. Moreover, you may display a telop with speaker information (name attribute information).

  Next, a telop is inserted in the video based on the character attribute, insertion position, size, etc. determined as described above, and video data with audio is reproduced (step S38). When reproducing video data with audio, the processing shown in FIG. 12 may be continued to create and display a telop in real time, or the telop may be cached by prefetching metadata before reproduction. Sometimes it may be displayed.

  According to the video data processing apparatus with audio of the present embodiment, intelligent telop is automatically adjusted by adjusting the insertion position, size, and character attribute of the telop according to the position of the speaker in the video data and the audio characteristics. Can be created and displayed automatically.

  In the above embodiment, the telop is easily created using the metadata. However, the method of using the metadata is not limited to this. For example, it is possible to easily create a telop, script, scenario, meeting minutes, etc. using the metadata by connecting a printer to the video data processing apparatus with audio.

  In the present embodiment, an example of an imaging apparatus including a video data processing apparatus with audio has been described. However, for example, an apparatus having a function of reproducing an image, such as a personal computer, a video recorder, or a hard disk recorder, may be used. A video data processing apparatus with audio can be applied.

  FIG. 12 is a block diagram showing another embodiment of a video data processing apparatus with audio. A video data processing apparatus with audio 100 shown in FIG. 12 is, for example, a personal computer, a video recorder, a hard disk recorder, or the like, with audio input via a recording medium 114, a video input terminal, or an audio input terminal (not shown). This is a device that generates and assigns metadata for video data, television programs, and the like.

  As shown in FIG. 12, the CPU 102 is connected to each block in the audio-added video data processing apparatus 100 via a bus 104, and performs overall control of each block based on an operation input from the operation unit 106 or the like. It is a control unit. The operation unit 106 includes a keyboard, a mouse, and other operation members, and outputs a signal to the CPU 102 in response to operation inputs from these operation members. The external storage device 108 is a device that stores programs processed by the CPU 102 and various data necessary for control, and is, for example, a hard disk device (HDD). The memory control unit 110 is controlled by the CPU 102 and writes data to the main memory 112 and the recording medium 114 and reads data from the main memory 112 and the recording medium 114. The main memory 112 is a main storage device of the video data processing apparatus with audio 100, for example, a semiconductor memory. The main memory 112 is an SDRAM that is a work area when the CPU 102 reads out programs and various data from the external storage device 108 and performs various arithmetic processes and the like, and a storage area that stores contents displayed on the display monitor. VRAM etc. are provided. The recording medium 114 records video. The user can input a desired video to the video data processing apparatus with audio 100 via the recording medium 114. The video / audio signal analysis unit 116 and the metadata generation unit 118 are the same as those in FIG.

The block diagram which shows the main structures of an imaging device provided with the video data processing apparatus with an audio | voice which concerns on one Embodiment of this invention. Functional block diagram showing the flow of video data processing with audio in the imaging apparatus 10 Block diagram showing video data with audio Table showing examples of information included in metadata Diagram showing an example of metadata in xml format Diagram showing xml schema The figure which shows the example which preserve | saves metadata in MPEG format The figure which shows the example which preserve | saves metadata in AVI format The flowchart which shows the processing method of the video data with audio | voice which concerns on one Embodiment of this invention. Block diagram showing the main configuration of the video playback function section of the video data processing apparatus with audio Flowchart showing a flow of processing for reproducing audio-added video data to which metadata is added Block diagram showing another embodiment of a video data processing apparatus with audio

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Imaging device, 12 ... CPU, 14 ... Bus, 16 ... Operation switch, 18 ... Memory, 20 ... External communication interface (external communication I / F), 22 ... Optical system (lens), 24 ... Imaging device, 26 ... A / D converter, 28 ... Video processing unit, 30 ... Video display unit (monitor), 32 ... Recording medium, 34 ... Microphone, 36 ... Audio processing circuit, 38 ... Speaker, 40 ... Recording instruction / control unit, 42 ... External video / audio input unit, 44 ... video / audio signal encoding unit, 46 ... video / audio signal storage unit, 48 ... video / audio signal analysis unit, 50 ... metadata generation unit, 60 ... video data with audio, 62 ... audio data, 64 ... video data, 70 ... reproduction instruction control unit, 72 ... video / audio signal decoding / reproduction unit, 74 ... metadata reading unit, 76 ... telop generation display unit, 78 ... external video / audio output unit, 1 DESCRIPTION OF SYMBOLS 0 ... Video data processing apparatus with an audio | voice, 102 ... CPU, 104 ... Bus, 106 ... Operation part, 108 ... External storage device, 110 ... Memory control part, 112 ... Main memory, 114 ... Recording medium, 116 ... Video / audio signal Analysis unit, 118 ... metadata generation unit

Claims (11)

Data acquisition means for acquiring video data with audio including video data and audio data synchronized with the video data;
Utterance content information generation means for generating utterance content information by characterizing the voice data;
Utterance time information acquisition means for acquiring utterance time information indicating a time at which the audio data is emitted in the video data;
Metadata creation means for creating metadata including the utterance content information and the utterance time information;
Recording means for associating and recording the video data with audio and the metadata;
A video data processing apparatus with sound, comprising: The apparatus further comprises speaker identification means for analyzing the video data and the voice data and identifying a speaker who has uttered the voice,
The audio-added video data processing apparatus according to claim 1, wherein the metadata creating means records the utterance content information and the identification information of the speaker in association with each other. The speaker identification means further comprises speaker position information acquisition means for analyzing the video data and voice data and acquiring position information on a screen displaying the video data of the speaker who has spoken the voice,
The audio-added video data processing apparatus according to claim 1 or 2, wherein the metadata creation means records the utterance content information and the position information of the utterer in association with each other. A voice feature quantity acquisition unit for analyzing the voice data and acquiring the voice feature quantity;
4. The video data processing apparatus with audio according to claim 1, wherein the metadata creating unit records the utterance content information and the audio feature quantity in association with each other. 5.   5. The video data processing apparatus with audio according to claim 4, wherein the audio feature amount acquisition unit acquires at least one information of a size, a height, an inflection, or a tone of the audio. Reproduction display means for reproducing and displaying the video data with audio,
Information acquisition means for acquiring utterance content information and utterance time information from the metadata;
Telop creating means for creating a telop based on the acquired utterance content information;
A telop insertion means for inserting the telop when reproducing the video data with audio based on the acquired utterance time information;
The video data processing apparatus with audio according to any one of claims 1 to 5, further comprising: Reproduction display means for reproducing and displaying the video data with audio,
Information acquisition means for acquiring utterance content information, utterance time information and speaker position information from the metadata;
Telop creating means for creating a telop based on the acquired utterance content information;
A telop insertion means for inserting the telop when reproducing the video data with audio based on the acquired utterance time information;
An insertion position adjusting means for adjusting the insertion position of the telop based on the acquired position information of the speaker;
The video data processing apparatus with audio according to claim 3, further comprising: Reproduction display means for reproducing and displaying the video data with audio,
Information acquisition means for acquiring utterance content information, utterance time information and voice feature amount from the metadata;
Telop creating means for creating a telop based on the acquired utterance content information;
A telop insertion means for inserting the telop when reproducing the video data with audio based on the acquired utterance time information;
Character attribute changing means for changing the character attribute of the telop according to the voice feature amount;
The video data processing apparatus with audio according to claim 4 or 5, characterized by comprising:   The character attribute changing means, according to the audio feature amount, the telop font, font size, character color, background color, character decoration, column, parenthesis, speech balloon attached to the telop, exclamation mark, question mark, etc. 9. The video data processing apparatus with audio according to claim 8, wherein at least one of the codes is changed. A data acquisition step of acquiring video data with audio including video data and audio data synchronized with the video data;
Utterance content information generating step for generating utterance content information by characterizing the voice data;
An utterance time information acquisition step of acquiring utterance time information indicating a time at which the audio data is emitted in the video data;
A metadata creation step for creating metadata including the utterance content information and the utterance time information;
A recording step of recording the audio-added video data and the metadata in association with each other;
A method of processing video data with audio, comprising: A data acquisition function for acquiring video data with audio including video data and audio data synchronized with the video data;
An utterance content information generation function for generating utterance content information by characterizing the voice data;
An utterance time information acquisition function for acquiring utterance time information indicating a time at which the audio data is emitted in the video data;
A metadata creation function for creating metadata including the utterance content information and the utterance time information;
A recording function for associating and recording the video data with audio and the metadata;
A computer program for processing video data with sound, characterized in that a computer is realized.

Images