JP4599244B2 - Apparatus and method for creating subtitles from moving image data, program, and storage medium - Google Patents

Description

  The present invention relates to a technique for automatically generating and editing captions from moving image data.

  In recent years, devices such as digital video cameras and digital cameras that have the ability to shoot movies with digital data have become widespread, and as a result, images taken are edited with digital video cameras and digital cameras, and the videos that have been shot are processed. More and more people enjoy original images. In addition, an increasing number of people take captured videos to a personal computer, edit them on a personal computer, and create an original video by adding a title composition or adding a caption.

  On the other hand, in public TV broadcasting, the effect is produced by displaying the speech of the speaker as a caption supervision, and a service that displays the speech of the speaker as a caption supervision for people who are deaf There are also many places.

  In this way, there is an increasing need for the captured video to display the speaker's dialogue as a caption superimposition with the video data.

  When importing a speaker's dialogue into a movie as a subtitle for a recorded movie, text data is usually entered as dialogue in a normal editing software and displayed as a speech bubble near the speaker, or as a movie It is edited so that it will be displayed as a caption subtitle at the bottom of the screen.

  This editing process is time consuming.You must play the video, input the result of the editor's listening to the input as necessary text data from the editing software, and edit it so that a phrase is displayed only for the time required for playback. I must. Also, when determining who is the speaker and changing the display position of subtitle supermarket and the color of subtitles, it is necessary for the editor to individually specify the position, color, etc. It was time consuming.

As a technique for performing these operations easily and efficiently, application of the following published technique can be considered. That is,
For a method of detecting a face area from a photographed image and displaying text data created in advance as a speech bubble near the mouth of the detected face (for example, claim 10 of Patent Document 1) or a voice input terminal such as a microphone A method in which a speaker is associated, voice to text is automatically converted, the face of the speaker is detected, and the converted text data is displayed in the vicinity of the speaker's face as a speech bubble (for example, Patent Literature Two claims 2) have been proposed. As a technology that supports these, a method for identifying a specific face from the feature amount of the face region (for example, Patent Document 3) or extracting a feature amount included in the input speech, and a pattern with a pre-registered speech feature amount A method for matching (for example, Patent Document 4) and a method for creating a minutes by converting text from input speech (for example, Patent Document 5) have been proposed.
JP 2002-176619 A JP 2003-339034 A JP-A-8-063597 Japanese Unexamined Patent Publication No. 6-083382 JP-A-8-194492

  However, in the above technique, in order to easily create subtitles, it is possible to create text data from the speech data of the speaker and display the supertitle as a speech bubble near the speaker's face. The speaker's voice is not associated with the speaker's face, and it is necessary to specify the speaker in advance. Therefore, after specifying the speaker in advance, editing was performed so that the caption superimpose was displayed near the speaker's face, so the task of automatically identifying the speaker and adding a speech balloon to the predetermined speaker could not. That is, in editing work, it is necessary to specify a speaker and then edit it. The present invention aims to solve such a problem.

In order to solve the above-described problem, the present invention is an apparatus for creating captions from original moving image data including images and sounds, and a face detection unit that detects a feature amount of a face from an image portion of the original moving image data, A voice identification unit that detects a voice feature amount from a voice portion of the original moving image data, a face feature amount detected by the face detection unit, and a voice feature amount detected by the voice identification unit are prepared in advance. A speaker specifying means for specifying a speaker in comparison with a voice feature amount for identifying a voice of a speaker and a face feature amount for identifying a face of the speaker, and specifying a face position of the specified speaker A position specifying unit, a voice recognition unit that recognizes a character string from the specified voice of the speaker, and generates text data of the character string; a face position obtained by the position specifying unit; and the voice recognition unit Generated text Based on the chromatography data, and blowout creating means to create a balloon data for displaying the text data string uttered from the identified said speaker on the display screen, the balloon data to the original video data And a moving image creating means for creating new moving image data, wherein the speech balloon creating means generates a speech balloon corresponding to the identified speaker with respect to the speech balloon data created by the speech balloon creating means. Speech balloon editing means for displaying a balloon editing screen for editing at least one of shape, color, pattern, size, and character color, size, and font; An image display area for displaying data, a text display area for editing the balloon data, and voice recognition by the voice recognition means A voice recognition operating unit for, viewing including and a reproduction operation portion for performing reproduction of the sound, the balloon creation means has been recognized recognized faces the voice of the speaker in the speaker identification means When it is not possible or when the speaker disappears from the display screen, instead of the balloon data corresponding to the speaker's face position, only a character string is displayed as a subtitle super in the area below the display screen. Create data .

Further, the present invention is a method for creating subtitles from original moving image data including images and sounds, a face detection step of detecting a facial feature amount from an image portion of the original moving image data, and sound of the original moving image data A speech identification step for detecting a feature amount of speech from a portion; a feature amount of a face detected in the face detection step; and a feature amount of a speech detected in the speech identification step. A speaker specifying step of specifying a speaker in comparison with a voice feature amount for identifying a voice and a face feature amount for identifying a face of the speaker, and a position specifying step of specifying the face position of the specified speaker A speech recognition step of recognizing a character string from the specified voice of the speaker and generating text data of the character string, a face position obtained by the position specification step, and a text generated by the speech recognition step Specific based on data The a balloon creation process to create a balloon data for displaying the text data of the spoken character string within the display screen from the speaker, the new moving image data by adding the balloon data to the original video data Creating a moving image creating step, the balloon creating step, for the balloon data created by the balloon creating step, the shape, color, pattern, size of the balloon corresponding to the specified speaker, And a balloon editing step for displaying a balloon editing screen for editing at least one of the color, size, and font of the character, wherein the balloon editing screen is an image display area for displaying the new moving image data. A text display area for editing the balloon data, and a voice recognition operation unit for executing voice recognition by the voice recognition step , Look-containing and a reproduction operation portion for performing reproduction of the sound, the balloon creation process, when it recognizes the voice of the speaker in the speaker identification step unrecognized face or speaker said display When the screen disappears, data for displaying only a character string as a subtitle super is created in the area below the display screen, instead of the balloon data corresponding to the speaker's face position .

  Note that the present invention can also be realized as a program for causing a computer to execute a method for creating subtitles from moving image data including images and sound, or a computer-readable storage medium storing the program.

  According to the present invention, the speaker is identified from the face and voice of the input video data, and the balloon data is created from the speaker position and the voice of the corresponding speaker. Can be displayed, making it easy to create and edit speech balloons and subtitles.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

[First Embodiment]
FIG. 1 is a block diagram for realizing an automatic speech balloon creation / editing processing function according to an embodiment of the present invention.

  Reference numeral 101 denotes a moving image input unit which inputs a moving image video signal. The input video signal is input as digital data. The input video signal is sent to the face detection unit 103 and the image data unit 111. The face detection unit 103 extracts a human face from the input video signal and calculates the feature amount. As for the face detection algorithm, known techniques such as skin color detection, eye-nose-mouth detection, and face contour detection are used. Regarding the calculation of the feature amount, a known feature amount calculation is used in which the positional relationship and size of the eyes and nose and mouth, the ratio to the face contour, and the like are combined with the detected human face. Further, the face detection unit 103 determines the size of the face, the position of the mouth, and the orientation of the face, and these pieces of information are also sent to the speaker specifying unit 107 as face feature amounts.

  An audio input unit 102 inputs an audio signal of a moving image. The input audio signal is input as digital data. The input voice signal is sent to the voice identification unit 104, the voice recognition unit 105, and the voice data unit 113. The voice identification unit 104 calculates a voice feature amount from the input voice signal. As for a speech recognition algorithm, a known feature amount calculation combining a speech frequency characteristic, a voice strength characteristic, and the like is used.

  The feature amount calculated by the face detection unit 103 and the feature amount calculated by the voice identification unit 104 are sent to the speaker specifying unit 107. The speaker identification unit 107 compares and refers to the individual voice feature quantity and face feature quantity registered in the voice / face correspondence unit 106 with respect to the feature quantities sent from the face detection unit 103 and the voice identification unit 104. To identify the speaker. When a plurality of persons are recognized by the face detection unit 103, a plurality of face feature amounts are sent, and when a plurality of persons are recognized by the voice identification unit 104, a plurality of sound feature amounts are sent. When these multiple feature quantities are sent, the feature quantities are combined, and the voice feature quantity registered in the voice / face correspondence unit 106 is compared with the face feature quantity to refer to a plurality of speakers. It is also possible to specify. When a speaker is specified by the speaker specifying unit 107, information on the face position and size of each speaker is sent to the position specifying unit 109, and speaker identification information is sent to the voice recognition unit 105. .

  Reference numeral 105 denotes a voice recognition unit, which performs voice recognition on a voice signal input from the voice input unit 102 when the speaker corresponds to a specific speaker based on the speaker identification information notified from the speaker specifying unit 107. . The speech recognition algorithm uses known techniques such as removal of ambient noise, feature extraction, and phoneme detection using an acoustic model.

  Information decomposed into phonemes by the speech recognition unit 105 is sent to the text unit 108, where lexical analysis and grammatical analysis are performed, and text data of the uttered character string is generated. The speech text conversion algorithm in the text conversion unit 108 uses known techniques such as lexical analysis, grammatical analysis, and dictionary lookup. The voice information converted into text is sent to the position specifying unit 109 together with the speaker information.

  The voice recognition unit 105 and the text conversion unit 108 can perform voice recognition and text conversion for each speaker for a plurality of speakers. As a result, even when a plurality of persons are talking in a state where a plurality of persons are shown at the same time, voice information converted into text for each speaker is generated.

In the position specifying unit 109, the speaker's face position and size information sent from the speaker specifying unit 107 and the text-formed voice information and speaker information sent from the text forming unit 108 are used to determine the speaker's face. The position specifying information generated by combining the face position (the display position of the speech bubble or the display position of the caption subtitle) and the audio text information is sent to the speech bubble creation unit 112. Further, the position specifying unit 109 determines the position specifying information on whether to generate a speech bubble at the mouth or to display it as a supertitle by determining the direction and size of the speaker's face and the duration of voice utterance. For example, since the case vary greatly with respect to the utterance duration magnitude of the voice of the speaker's face performs a zoom operation, it is thought that the screen and put the balloon in the mouth is difficult to see, display the subtitles at the bottom of the screen To do. If the speaker turns backward during the voice utterance duration, the video continuity of the speaker is determined to track the speaker, and the balloon display position is moved from the mouth to the tip of the head.

  In addition, when the speaker moves greatly in the screen during the voice utterance duration, it is necessary to move the speech balloon a lot and it may be difficult to see the screen, so a caption subtitle is displayed at the bottom of the screen. Also, if the speaker moves off the screen during the duration of the speech, or disappears from the screen, such as when the speaker is hidden behind the screen, a speech bubble is displayed at the mouth when the speaker is inside the screen, and the speaker disappears from the screen. Displays subtitle super at the bottom of the screen. The relationship between the speaker and the speech balloon or the caption subtitle mentioned here is an example, and other combinations may exist.

  When the position specifying information specified by the position specifying unit 109 is sent to the balloon creating unit 112, the balloon or the caption super display for displaying the balloon or the caption super display position in the position specifying information and the voiced text information. Balloon data is created. The balloon data created here is described using metadata. In the metadata tag, a start frame and an end frame, duration, effect, font, attributes (font color and background color, transparency), and a balloon shape are designated. The tag shown here is an example, and does not limit the tag type of the present embodiment. Since the speech balloons and subtitles to be created are described in metadata, the editing work can be performed not on the image data but on the text data editing base to facilitate the editing work.

  The speech balloon data created by the speech balloon creation unit 112 is sent to the moving image creation unit 114 in synchronization with the data in the image data unit 111 and the audio data unit 113, and is collected into a moving image format. Typical video format standards include Motion JPEG, MPEG, and the like.

  Reference numeral 110 denotes a synchronization unit that synchronizes the video signal of the moving image and the audio signal, and supplies them to the face detection unit 103, the audio identification unit 104, and the audio recognition unit 105, respectively. The face detection unit 103 calculates the face recognition start time and frame number (hereinafter referred to as a time code) from the synchronization signal, moves the face (moving amount per time), and the time (face Information such as time code that can no longer be recognized is generated. In addition, the voice identification unit 104 recognizes and identifies the voice of the speaker from the synchronization signal, thereby calculating the utterance start time code and the utterance end time code. The voice recognition unit 105 recognizes the voice of the speaker from the synchronization signal, and calculates the time code when the recognition is started as a word and the end time code of the utterance. These synchronization signals make it possible to determine the speaker's face image, balloon start display time code, display duration, and display position. Even if the processing time is different due to the difference in speed, it is possible to eliminate the deviation between the speaker's face, voice and speech balloons. A synchronization signal is also sent from the synchronization unit 110 to the image data unit 111 and the audio data unit 113, and the image and the audio are synchronized when creating a moving image file.

  FIG. 2 shows the configuration of the video recording / editing apparatus 200 having the automatic speech balloon creation / editing processing function shown in FIG.

  A camera unit 201 outputs captured image data as an analog signal, is converted into dot-sequential digital data by an A / D conversion unit 202, and is sent to an image processing unit 203. In the image processing unit 203, color processing, luminance processing, and the like are performed from the video signal sent in point-sequential order and sent to the above-described automatic balloon creation / editing processing unit 100.

  Reference numeral 204 denotes a microphone unit that acquires an audio signal at the same time as photographing, and sends it to the A / D converter 205 as analog audio data. The A / D conversion unit 205 converts analog audio data into digital data in accordance with the sampling period and sends the digital data to the audio signal processing unit 206. Data processed by the audio signal processing unit 206 is sent to the automatic speech balloon creation / edit processing unit 100 as time-series digital data.

  Reference numeral 207 denotes a control device that controls the entire apparatus 200, and includes a control microcomputer (CPU), a program storage memory (ROM, flash memory, RAM, etc.), a data storage memory (RAM), etc. Controls each block and the entire device.

  Reference numeral 208 denotes an operation member of the apparatus 200, which includes various switches, levers, buttons, and the like, and includes detection members such as a user interface member of the apparatus 200 and a sensor in the apparatus. By operating these operation members, it is possible to start / stop shooting, playback, various settings, and editing operations.

  Reference numeral 210 denotes a recording device that records moving image data with a balloon created by the automatic balloon creation / editing processing unit 100. The recording device 210 is composed of recording means that can be incorporated or detached, such as a hard disk, a memory card, a magneto-optical storage medium, and the like. Depending on the storage medium, the moving image data may be stored as raw moving image data or may be recorded as a file format.

  A moving image / balloon synthesis processing unit 211 receives moving image data read from the recording device 210 or moving image data output from the automatic balloon creation / editing processing unit 100. The position information recorded in the balloon data from the timing at which the frame is displayed according to the start time code recorded in the balloon data while synchronizing the video data and audio data of the movie with the input movie data An actual speech balloon is created based on the effect, attribute, font, color, speech balloon shape, etc., and the image is synthesized. The moving image / balloon composition processing unit 211 can display the characters in a time series in order to change the shape of the balloon in time series depending on the effect of the fixed form and characters from the fixed thing. It is also possible to sequentially change the font color and the background color of the balloon in time series.

  Reference numeral 209 denotes a synchronization unit. A synchronization signal is provided from the synchronization unit 209 to the camera unit 201, the A / D conversion unit 202, and the image processing unit 203, and is used as a sampling rate of the video signal. A synchronization signal is supplied from the synchronization unit 209 to the audio system A / D conversion unit 205 and the audio signal processing unit 206, and is used as a sampling rate of the audio signal. A synchronization signal is supplied from the synchronization unit 209 to the automatic speech balloon creation / editing processing unit 100, and the video signal and the audio signal are synchronized and supplied to the synchronization unit 110. A synchronization signal is supplied from the synchronization unit 209 to the moving image / balloon synthesizing process 211, the video signal and the audio signal at the time of moving image reproduction are synchronized, and the frame is displayed according to the start time code recorded in the balloon data From the above, it is used as a timing signal for displaying / erasing a balloon or applying an effect.

  The moving image signal configured by the moving image / balloon synthesis processing unit 211 is supplied to the display device 213 as a video signal and also supplied to the speaker 212 as an audio signal. As a result, when a registered person's voice is output from the speaker 212, a moving image with a speech balloon or a caption supervision is displayed on the display device 213.

  FIG. 3 is an external view of the video recording / editing apparatus of FIG.

  Reference numeral 300 denotes a video recording / editing device main body. Reference numeral 301 denotes a photographing button, and photographing is started / stopped by pressing this button. When the automatic speech balloon creating / editing function described above is valid, a speech balloon and a caption supertitle are automatically created and recorded when shooting is started. Reference numeral 302 denotes an eyepiece (viewfinder), which allows a photographer to check a captured image. Reference numeral 303 denotes a photographing lens, and photographing is performed through this lens. Reference numeral 304 denotes a liquid crystal finder / reproduction screen, which can confirm an image being photographed, a reproduced image, and various settings. If the automatic speech balloon creation / editing function is valid, the captured image is automatically displayed with a speech balloon and a caption subtitle. In addition, a playback image to which a balloon or a caption supertitle is added is also displayed. Reference numeral 305 denotes an operation switch that performs various setting operations and operations such as playback, fast forward, and rewind.

  Next, the flow of the automatic speech balloon creation / editing process will be described with reference to FIGS.

  FIG. 4 shows processing when time elapses in the right direction with the horizontal direction as the time axis.

  As the video signal, a video (FIG. 5) showing only the child A and a video (FIG. 6) showing the two children A and B are input to the moving image input unit 101. A video in which only A child is shown is a period 401, and a video in which two children A and B are shown is a period 402. On the other hand, the voice signal is “Good morning A child” in the voice of child A, “Good morning B child” in the voice of child B, and “C child. Good morning A child, B child in the voice of child C. "Is input to the voice input unit 102. The period during which child A is speaking is 403, the period during which child B is speaking is 404, and the period during which child C is speaking is 405. During other periods, background audio is input to the audio input unit 102.

  In the video period 401 in which only the child A is shown, the face detection unit 103 calculates the face feature amount, face direction, mouth position, and the like of the child A, and sends them to the speaker specifying unit 107. In the voice / face correspondence unit 106, a child A face feature amount and voice feature amount, a child B face feature amount and voice feature amount, and a combination of child C face feature amount and voice feature amount are registered. In the video period 401, the speaker identification unit 107 recognizes that the child A exists in the screen by comparing the face feature amount registered in the voice / face correspondence unit 106 against the face feature amount of the child A. ing.

  On the other hand, with respect to the voice signal input to the voice input unit 102, the voice identification unit 104 calculates a voice feature amount at a constant period and sends it to the speaker specifying unit 107. The speaker specifying unit 107 collates the voice feature amount registered in the voice / face correspondence unit 106 against the voice feature amount. In the period 403 during which the child A is speaking, the voice identification unit 104 calculates the voice feature amount of the child A, and the speaker identifying unit 107 performs the voice / face correspondence unit 106 on the voice feature amount of the child A. The child A is recognized as a speaker by collating the voice feature values registered in. As described above, the speaker specifying unit 107 recognizes that the child A exists in the screen in the period 401 and that the child A exists in the screen and is a speaker in the period 403. The speaker specifying unit 107 sends identification information indicating that the child A is a speaker to the voice recognition unit 105 during the period 403. The identification information sent here includes information (period 403) of a period during which the child A is a speaker and data including a pre-registered voice feature amount of the child A. The voice recognition unit 105 extracts the A child voice information in the period 403 from the voice signal sent from the voice input unit 102 based on the sent identification information, performs voice recognition, and obtains the phoneme of the A child voice. Extract. With respect to the phoneme data for the voice of child A extracted by the voice recognition unit 105, the text conversion unit 108 converts the content that A child uttered in the period 403 to “Good morning A child”. Since it takes time for the speaker identification unit 107 to specify the speaker, the voice identification unit 104 and the voice recognition unit 105 accumulate (store) the voice for a certain period of time, and the speaker identification unit 107 identifies the speaker specified by the speaker identification unit 107. Going back to the utterance start time, voice recognition and voice recognition can be performed again.

  The speaker specifying unit 107 recognizes that the child A is present in the screen during the period 403 and can also specify the position of the face and mouth. The text unit 108 can also convert the content uttered by the child A into text. Completed. Based on these pieces of information, the position specifying unit 109 determines the position of the balloon as the mouth of the child A, and sends the position specifying information to the balloon creating unit 112.

  The speech balloon creation unit 112 creates metadata for displaying a speech balloon of A child's utterance content “Good morning A child” at the mouth of A child based on the position specifying information sent. Reference numeral 406 denotes a description of the metadata. Here, when creating a speech bubble, the content is easy to read by setting a holding time and displaying the speech bubble for a while after the speech is finished so that the speech data does not disappear at the same time that the child A utters. It is also possible to do. It is also possible to perform syllable division in the speech recognition unit 105 and to display text sequentially in units of words or speech in the text unit 108. Furthermore, it is also possible to allocate a duration in units of words according to the utterance time, and display the text sequentially according to the time in units of text characters. In addition, when a speaker is registered in the voice / face correspondence 106 in advance, it is also possible to create a speech balloon having characteristics for each speaker by associating the shape, color, and effect of the text and speech balloon. .

  Next, a video period 402 in which A child and B child are shown will be described.

  The face detection unit 103 calculates the facial feature amount, face orientation, mouth position, etc. of child A, and the facial feature amount, face orientation, mouth position, etc. of child B. 107. In the video period 402, the speaker specifying unit 107 collates the face feature amount registered in the voice / face correspondence unit 106 against the face feature amount of the child A and the face feature amount of the child B, and the child A and the child B. Are aware that they are on the screen.

  In the period 404 in which the child B is uttering, the speech identification unit 104 calculates the speech feature amount of the child B, and the speaker specifying unit 107 performs the speech / face correspondence unit 106 on the speech feature amount of the child B. The child B is recognized as a speaker by collating the voice feature values registered in. Also, during the period 405 when the child C is speaking, the voice identification unit 104 calculates the voice feature amount of the child C, and the speaker identification unit 107 performs voice / face correspondence on the voice feature amount of the child C. The voice feature quantity registered in the unit 106 is collated to recognize that the child C is a speaker. In this way, the speaker identifying unit 107 recognizes that the child B is present in the screen and the child B is the speaker in the period 404, and the child C is not present in the screen in the period 405. Recognizes that he is a speaker.

  During the period 404, the speaker specifying unit 107 sends identification information indicating that the child B is a speaker to the voice recognition unit 105. The speech recognition unit 105 extracts the B child speech information in the period 404 from the speech signal sent from the speech input unit 102 based on the sent identification information, performs speech recognition, and obtains the phoneme of the B child speech. Extract. With respect to the phoneme data for the B child voice extracted by the voice recognition unit 105, the text conversion unit 108 converts the text “Good morning B child” uttered by the B child in the period 404. In the same manner, the content “C child. Good morning A child, B child.” Uttered by child C in period 405 is converted into text.

  The speaker specifying unit 107 recognizes that the child B is present in the screen during the period 404 and can also specify the position of the face and mouth, and the texting unit 108 converts the content of the child B to the text. Has also been completed. Based on these pieces of information, the position specifying unit 109 determines the position of the balloon as the mouth of the child B, and sends the position specifying information to the balloon creating unit 112.

  On the other hand, the speaker identifying unit 107 recognizes that the child C does not exist in the screen in the period 405, and the text portion 108 has completed the text conversion of the content uttered by the child C. Based on these pieces of information, the position specifying unit 109 determines that the caption subtitle is not the balloon but the bottom of the screen, and sends the position specifying information to the balloon creating unit 112. In the video of the present embodiment, when the speaker disappears from the screen while speaking, or when facing backwards, the size change or the position change in the speaker's screen is not shown. In such a case, the above-described processing may be performed.

  The speech balloon creating unit 112 displays a speech balloon of the child B's utterance content “Good morning B child” on the mouth of the child B, based on the position specifying information sent during the period 404 when the child B spoke. Create metadata. Reference numeral 407 denotes a description form of the metadata. In addition, based on the position specifying information sent for the period 405 when C child uttered, the subtitle superimposition of C child's utterance content “C child. Good morning A child, B child.” Is displayed at the bottom of the screen. Create metadata for. Reference numeral 408 denotes a description form of the metadata.

  The speech balloon data created in time series by the speech balloon creation unit 112 in this manner is the moving image data generated by the moving image creation unit 114 based on the synchronization signals provided from the image data unit 111, the audio data unit 113, and the synchronization unit 110. Assembled as.

  When the moving image data assembled in this way is sent to the moving image / balloon synthesis processing unit 211, the video and the audio signal are reproduced in synchronization, the video signal is sent to the display device 213, and the audio signal is sent to the speaker 212. It is done. In the video period 401 in which the child A is shown and no voice of a registered person is detected, no speech balloon is generated (section 1).

  In section 1, an image showing the child A is displayed as shown in FIG. In the video period 401 in which the child A is shown and in the period including the period 403 in which the child A utters (the display period defined by the balloon creation unit 112), a balloon 406 is generated and combined with the video signal and displayed. (Section 2).

  In section 2, A child is reflected as shown in FIG. 6, and a balloon is displayed from the mouth of A child. In the video period 402 in which the child A and the child B are shown, in a state where the voice of the registered person is not detected, a balloon is not generated (section 3).

  In section 3, an image showing A child and B child is displayed as shown in FIG. In a video period 402 in which A child and B child are shown, a balloon 407 is generated and combined with the video signal in a period including the period 404 in which the B child is uttered (display period defined by the balloon creating unit 112). Are displayed (section 4).

  In section 4, A child and B child are reflected as shown in FIG. 8, and a balloon is displayed from the mouth of B child. In the video period 402 in which A child and B child are shown, when the child B's utterance is completed and the display period defined by the balloon creating unit 112 has passed, the registered person's voice is not detected, A balloon is not generated (section 5).

  In section 5, an image showing child A and child B is displayed as shown in FIG. In the video period 402 in which the child A and the child B are shown, and in the period including the period 405 in which the child C utters (the display period defined by the balloon creation unit 112), the caption super 408 is generated and synthesized with the video signal. And displayed (section 6).

  In section 6, A child and B child are shown as shown in FIG. 9, and a caption super is displayed at the bottom of the screen.

  As described above, according to the present embodiment, dialogue is performed by voice recognition of a speaker by performing face detection and voice identification on the voice / face correspondence data of a person registered in advance and specifying the speaker. Automatic text conversion makes it easy to create speech balloons and subtitles.

[Second Embodiment]
FIG. 17 exemplifies a moving image data editing apparatus including software that realizes an automatic speech balloon creation / editing processing function according to an embodiment of the present invention.

  In this embodiment, the moving image data editing apparatus is realized by a personal computer 600 including a display device 601, a keyboard 602, and a mouse 603, but a video recording / editing apparatus (magnetic tape recording type, magneto-optical disk recording type, An optical recording disk recording type, a magnetic disk recording type, etc.) and an editing-dedicated device may be used.

  FIG. 12 exemplifies a display screen displayed on the display device 601 by the function of software that implements the automatic speech balloon creation / editing process of the present embodiment.

  Reference numeral 501 denotes an area for displaying a video image to be edited. An area 502 displays a list of speakers registered in the voice / face correspondence unit 106. Reference numerals 503, 504, and 505 denote areas for displaying information for each registered speaker. Reference numeral 506 denotes a state in which the speaker is present in the image. The video signal input through the moving image input unit 101 is recognized by the speaker specifying unit 107 for the moving image displayed in the video area 501. Shows the person. That is, the video area 501 indicates that a person currently registered in the speaker information 503 is shown. Reference numeral 508 denotes a speaker's utterance state display. A speaker in which an audio signal input through the audio input unit 102 is voice-recognized by the speaker specifying unit 107 for a moving image displayed in the video area 501. Is shown. That is, it is indicated that the person currently registered in the speaker information 503 is speaking as a speaker. Reference numeral 507 denotes a slider for scrolling the speakers in the speaker list display area 502. Reference numeral 509 denotes a slider indicating the position in the moving image data displayed in the video area 501. By moving the slider lever, the moving image data can be moved to an arbitrary position. Reference numeral 510 denotes the level of an audio signal input to the audio input 102 of moving image data. Reference numeral 511 denotes a time code in the moving image data when a speaker registered in the speaker specifying unit 107 is detected (detection start time). Reference numeral 512 denotes a time code in the moving image data at the time when the utterance of the speaker registered in the speaker specifying unit 107 ends (detection end time). Reference numeral 513 denotes a time code in the moving image data of the video displayed in the current video area 501. Reference numeral 514 denotes an operation state of the application. Application states include voice / face identification and voice recognition / text conversion (writing). Reference numeral 515 denotes a voice / face identification start button. By pressing the button, voice / face identification is started and speaker identification is performed. Reference numeral 516 denotes a preview button, which can automatically reproduce a balloon or a caption subtitle created automatically or edited by the user together with moving image data. The image of the screen given here is an example for explaining the present embodiment, and does not limit the function of the present embodiment.

  Next, the operation of the software of the present embodiment will be described with reference to a flowchart and a display screen example.

  FIG. 10 is a flowchart showing voice / face correspondence data registration processing by software that realizes an automatic speech balloon creation / edit processing function. FIG. 13 is an example of a display screen in the voice / face correspondence registration process.

  Note that it is necessary to associate the audio feature quantity with the face feature quantity when executing the processing by this software.

  First, when voice / face correspondence registration is started (S100), a voice / face correspondence registration screen 520 is displayed, which is a person name input step (S101). In the person name input step (S101), the person name is input into the person name input field 521 of the voice / face correspondence registration screen 520. Subsequently, a face image capturing step (S102) is performed in order to register the facial feature amount of the person. In the face image capture step (S102), a face image is captured by pressing the face image capture button 526, and the captured image is displayed in the face display area 522, and the face feature amount calculation step (S103) is executed. Is done. Subsequently, an audio capturing step (S104) is performed in order to register the audio feature amount of the person. In the sound capturing step (S104), the sound is captured by pressing the sound capturing button 527, the level of the captured sound is displayed in the sound level display area 525, and the sound feature amount calculating step (S105) is executed. The In the present embodiment, the facial feature amount registration for face identification and the voice feature amount registration for voice identification are executed only once, but may be executed a plurality of times. For example, when acquiring facial feature values, by calculating feature values in the front direction, the left-right diagonal direction, and the up-down diagonal direction, the identification rate can be improved even if the speaker faces away from the front side. Regarding the voice feature amount, the identification rate under various situations can be improved by calculating the feature amount while changing the strength of a plurality of words and voices.

  When the calculation of the face feature amount and the voice feature amount is completed, a balloon setting step (S106) and a caption super setting step (S107) are performed. In the balloon setting step (S106), a balloon property setting item S110 is set. When the balloon property setting button 523 is pressed, a balloon property setting screen 530 is displayed. In the setting screen, tabs 531, 532, and 533 are provided for each item of the balloon property setting item S110, and the tab of the item to be set is selected and each item is set. In FIG. 13, a setting screen for a tab 531 for selecting a balloon shape is displayed. A plurality of shapes that can be set are shown in the selection list 535, and a desired shape is selected from these shapes. Similarly, in the caption super setting step (S107), a caption super property setting item S111 is set. When the caption super property setting button 524 is pressed, a caption super property setting screen is displayed, and the caption super property setting item S111 is set. The balloon property setting item S110 and the caption super property setting item S111 according to the present embodiment are examples, and other setting items may be included, and the content of the proposal is not limited.

  When the calculation of the voice / face feature amount, the speech balloon setting, and the subtitle super setting are completed, the recording confirmation step (S108) is performed. If the recording is acceptable, the voice / face correspondence recording step (S109) is performed and the voice / face correspondence is performed. Registered in the unit 106.

  FIG. 11 is a flowchart showing balloon creation (S120) and balloon editing (S140).

  When the balloon creation (S120) is started, a moving image input step (S121) is first executed. For example, a file (F) is selected and an existing moving image file is read, and a file (F) is selected and a moving image is input from an external input (external moving image playback device, video camera, VCR, DVD player, etc.). Is read.

  When the input of the moving image is determined, a speaker detection start step (S122) is executed. When the speaker detection start button 515 is pressed on the screen of FIG. 12, the video data of the moving image data specified in the moving image input step (S 121) is input to the moving image input unit 101 and the audio data is input to the audio input unit 102. The input video signal is sent to the face detection unit 103 and the speaker identification unit 107. The input voice signal is sent to the voice identification unit 104 and the speaker identification unit 107.

  When speaker detection is started in the speaker detection start step (S122), the speaker specifying step (S123) is performed. In the speaker specifying step (S123), the status display 514 becomes “speaker specifying”. In the speaker specifying step (S123), the facial feature amount and voice feature amount of the moving image input to the speaker specifying unit 107 and the speaker facial feature amount and voice feature amount registered in the voice / face correspondence unit 106 are used. Are identified and the speaker is identified. When the speaker identifying unit 107 detects a speaker whose voice feature amount matches, the application is notified of the utterance start time code, the utterance end time code, the speaker person name, and the face recognition state, and the speaker detection end step ( S124) is executed.

  In the speaker detection end step (S124), the input from the moving image data to the moving image input unit 101 and the voice input unit 102 is stopped, and the speaker specifying process of the speaker specifying unit 107 is also stopped. Further, the utterance start time code 511 and the utterance end time code 512 in which the speaker is specified are displayed. Furthermore, a speaker's in-image presence state 506 and a speaker's utterance state 508 are displayed according to the speaker's person name and face recognition state. In FIG. 12, child A is shown on the screen, and the voice of child A is the voice of “Good morning child A”.

  When the speaker detection is finished (S124), a voice recognition step (S125) is started. In the voice recognition step (S125), the status display 514 becomes “during voice recognition”. In the voice recognition step (S125), the voice signal of the time interval is input again from the moving image data to the voice input unit 102 using the A child's voice start time code and voice end time code, and the voice recognition unit 105 performs voice recognition. The text conversion step (S125) is performed by the text conversion unit 108. In this embodiment, the time interval is read from the moving image data, but the voice input unit 102 or the voice identification unit 105 holds the voice data for a certain period in the past, and the voice data is used for voice recognition and text conversion. May be performed.

  When the speech recognition step (S125) and the text conversion step (S126) are completed, a speech balloon automatic creation step (S127) is executed. In the speech balloon automatic creation step (S127), if the speaker is present on the screen due to the speaker person name and the face recognition state identified in the speaker identifying step (S123), a speech balloon is displayed. If it doesn't exist, it automatically creates a caption. In the automatic speech balloon creation step (S127), the position specifying unit 109 determines the display position of the speech balloon or caption supervision from the speaker person name and face recognition state detected by the speaker specifying unit 107. The display position determined by the position specifying unit 109 and the voice information converted to text by the text converting unit 108 in the text conversion step (S125) are input to the speech bubble generating unit 112, and a speech bubble or a caption subtitle is generated. When the speech balloon creating unit 112 creates a speech balloon, the speech balloon property setting speech balloon shape, speech balloon background color, character font, character color, speech balloon corresponding to the speaker person name registered in the voice / face correspondence unit 106 Balloon data is created based on transparency, effect, and display hold time. Also, when creating a caption super in the speech balloon creation unit 112, the caption super background color, character font, character color of the caption super property setting corresponding to the speaker person name registered in the voice / face correspondence unit 106, The speech balloon data is created as the caption subtitle based on the caption caption transparency, effect, and display holding time.

  A speech balloon display step (S128) is executed based on the speech balloon data created in the speech balloon automatic creation step (S127). In the balloon display step (S128), when a balloon is displayed, a balloon 540 is created and displayed in the video area 501 of FIG. 14 based on the balloon data. Also, in the balloon display step (S128), when displaying a caption super, a caption super 560 is created and displayed in the video area 501 of FIG. 15 based on the balloon data.

  After the balloon display step (S128), a balloon editing step (S129) is executed. In the speech balloon editing step (S129, S140), the character confirmation step (S141), the character correction step (S144), and the speech balloon setting change step (S145) for the speech balloon data created in the speech balloon automatic creation step (S127). ) To edit the balloon data.

  Hereinafter, the balloon editing process and the caption super editing process will be described.

FIG. 14 shows a balloon edit screen. When the balloon editing step S140 is executed and the balloon data is a balloon, a balloon editing screen 541 is displayed. The balloon edit screen 541 includes an image confirmation area 542, a text display / edit area 543, a slider 544 for displaying / moving the display position during the balloon display period, a balloon property setting 545, a speaker person name 546, and a utterance start time. It includes a code 547, an utterance end time code 548, a voice re-recognition button 549, a voice playback button 550, and a confirmation button 551. FIG. 14 shows a balloon edit screen in a state where A child is uttering “Good morning A child”. In the character confirmation step (S141), the text is acquired from the balloon data and displayed in the text display / edit area 543. The user determines whether there is a correction in the correction confirmation step (S142). If correction is necessary, the sound reproduction button 550 is pressed as necessary, and sound is reproduced from the moving image data from the utterance start time code 547 to the utterance end time code 548 in the sound reproduction step (S143). it can. Further, by moving the slider 544, it is possible to reproduce sound from an arbitrary position during the utterance period. The user can edit / correct the text displayed in the text display / edit area 543 in the character correction step (S144) while listening to the voice. Further, voice recognition (S125) and text conversion (S126) can be performed again by the voice re-recognition button 549. If the display text content of the speech balloon is confirmed, a speech balloon setting changing step (S145) is executed as necessary. In the balloon data, the contents of the balloon property setting of child A registered in the voice / face correspondence 106 are copied. By changing the contents set in the balloon property setting 545, the balloon property setting of individual balloon data can be changed. The balloon property setting changed here is effective only for the balloon of “Good morning A child”, and does not affect the balloon property setting of the child A registered in the voice / face correspondence 106. When the balloon editing is completed, the confirmation button 551 is pressed to complete the balloon editing step S129.

FIG. 15 is a subtitle editing screen. When the speech balloon editing step S140 is executed and the speech balloon data is caption super, the caption super editing screen 561 is displayed. The subtitle super editing screen 561 includes an image confirmation area 562, a text display / edit area 563, a slider 564 for displaying / moving the display position during the balloon display period, a subtitle super property setting 565, a speaker person name 566, an utterance It includes a start time code 567, an utterance end time code 568, a voice re-recognition button 569, a voice playback button 570, and a confirmation button 571. FIG. 15 is a caption super editing screen in a state where C child is uttering “C child. Good morning A child, B child”. In the character confirmation step (S141), the text is acquired from the balloon data and displayed in the text display / edit area 563. The user determines whether there is a correction in the correction confirmation step (S142). If correction is necessary, the sound reproduction button 570 is pressed as necessary, and sound is reproduced from the moving image data from the utterance start time code 567 to the utterance end time code 568 in the sound reproduction step (S143). it can. Further, by moving the slider 564, it is possible to reproduce sound from an arbitrary position during the utterance period. The user can edit / correct the text displayed in the text display / edit area 563 in the character correction step (S144) while listening to the voice. Further, voice recognition (S125) and text conversion (S126) can be performed again by the voice re-recognition button 569. When the display text content of the caption superimposition is confirmed, a balloon setting changing step (S145) is executed as necessary. The contents of the subtitle super property setting for the child C registered in the voice / face correspondence 106 are copied in the balloon data. By changing the content set in the subtitle super property setting 565, the subtitle super property setting of individual balloon data can be changed. The subtitle super property setting changed here is effective only for the subtitle super of “C child. Good morning A child, B child”, and the subtitle super of C child registered in the voice / face correspondence 106. Property settings are not affected. When the balloon editing is completed, the confirmation button 571 is pressed to complete the balloon editing step (S129).

  In the speech balloon editing step (S129), when the speaker is present in the screen, the speech balloon 540 of FIG. 14 is displayed in the video area 501, but the speech balloon 540 is designated to indicate the position, orientation, and size of the speech balloon. Adjustments can be made. It is also possible to designate the balloon 540 and change it to a subtitle supermarket. The balloon editing procedure and screen described here are merely examples, and the editing procedure and screen are not limited by the present invention. For example, the process from detection of a speaker to creation of an automatic speech balloon may be performed on the entire moving image data, and then an individual speech balloon or caption super editing operation may be performed.

  When the speech balloon editing step (S129) is completed, the speech balloon edited in the preview display step (S130) can be confirmed. In FIG. 12, when a preview button 516 is pressed, a preview screen is displayed.

  FIG. 16 shows a preview screen.

  Reference numeral 580 denotes a video area for displaying an image obtained by combining a video and a balloon. Reference numeral 581 denotes a time code of the video displayed in the video area 580. Reference numerals 582 to 586 denote operation buttons for performing reproduction. 582 is a move button to the immediately preceding utterance start time code, 583 is a rewind button, 584 is a play button, 585 is a fast forward button, and 586 is a move button to the immediately following utterance start time code. Reference numeral 587 denotes a balloon information window, which can display balloon information in an arbitrary range in the moving image data using the slider 592. Reference numeral 588 denotes a time code scale, which displays the existence start time code, the existence end time code, the utterance start time code, and the utterance end time code in the image of the registered speaker. In the example shown in the figure, 01:12:20 14 is the time code when the child A started to appear, 01:12:21 05 is the child A start time code, and 01:12:24 12 is the child A end time code. , 01:12:26 02 is the time code when the A child and the B child start to appear, and 01:22:27 15 is the utterance start time code of the B child. Reference numeral 589 denotes an index image in the presence start time code and utterance start time code of the registered speaker. Reference numeral 590 denotes a balloon information display, which indicates the text information displayed in the balloon and the display time of the balloon. The balloon display time is a time obtained by adding the display holding time to the utterance time. Reference numeral 591 denotes a caption super information display, which shows the text information displayed in the caption super and the display time of the caption super. The caption super display time is a time obtained by adding the display holding time to the utterance time.

  On the preview screen, by pressing the play button 584, playback is performed with a balloon from the current time code, and the contents and effects of the balloon can be confirmed. By pressing the rewind button 583, playback is performed in the reverse direction. The rewinding speed can be increased by pressing twice or more. When the fast forward button 584 is pressed, playback is performed in the forward direction. Fast-forward speed can be increased by pressing twice or more. A move button 582 to the immediately preceding utterance start time code can return to the time code at which the speaker started uttering immediately before the current time code. The move button 586 to the utterance start time code immediately after the current time code can fast forward to the time code at which the speaker started uttering. These buttons are assigned to move to the utterance start time code of the speaker, but as buttons to move to the utterance start time code, presence end time code, utterance start time code, utterance end time code in the speaker image May also be assignable.

  On the preview screen, by designating an arbitrary time code or index image 589 of the time code scale 588, an image of the designated time code can be called with a balloon or a caption superimposition.

  On the preview screen, the balloon edit screen 541 can be called by selecting the text portion of the balloon information display 590. By moving the left end of the text portion of the balloon information display 590, the balloon display start time code can be adjusted back and forth. It is also possible to adjust the display end time code of the balloon forward and backward by moving the right end of the text portion of the balloon information display 590. On the preview screen, the caption super editing screen 561 can be called by selecting the text portion of the caption super information display 591. By moving the left end of the text portion of the caption super information display 591, it is also possible to adjust the display start time code of the caption supermarket forward and backward. By moving the right end of the text portion of the caption super information display 591, it is also possible to adjust the display end time code of the caption supermarket forward and backward. Furthermore, it is also possible to switch from the balloon display to the caption super display by moving the text part of the balloon information display 590 to the caption super information display 591.

  The speech balloon edited in the preview display step (S130) is confirmed (S131). If correction is necessary, the flow returns to the speech balloon editing step (S129) again. If correction is not necessary, the editing completion confirmation step (S132) is performed. If there are more speech balloons, the process returns to the next speaker detection start step (S122) to detect the next speaker. When the editing completion confirmation step (S132) is completed, the speech balloon creation is completed. In the moving image creation step (S133), the image data 111, the sound data 113, and the speech balloon data are collectively created as moving image data by the moving image creation unit 114. . The created moving image data is stored in a moving image output step (S134). For example, the file (F) is selected and written as a new moving image file, or the file (F) is selected and the moving image is output to an external output (external moving image recording device, video camera, video deck, DVD recorder, etc.). Write.

  According to the above embodiment, the speaker is identified from the face and voice of the input video data, and the balloon data is created from the speaker position and the voice of the corresponding speaker. Speech balloons can be displayed, making it easier to create and edit speech balloons and subtitles.

  In addition, since a speaker can be identified and a speech bubble and a caption superimpose can be created simultaneously with the shooting of a moving image, it is easy to edit the speech balloon and the caption supervision after photographing. In addition, since the speaker can be specified simultaneously with the input of the moving image from the outside and the speech balloon and the caption subtitle can be created, the speech balloon and the caption supervision after the image input can be easily edited.

[Other Embodiments]
The embodiment according to the present invention has been described in detail using specific examples. However, the present invention can take an embodiment as a system, apparatus, method, program, storage medium (recording medium), or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to each functional block or flowchart shown in the drawings) that realizes the functions of the above-described embodiment is directly or remotely supplied to the system or apparatus. This includes the case where the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  As a recording medium (storage medium) for supplying the program, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, an MO, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, and a nonvolatile memory card ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, a client computer browser is used to connect to an Internet homepage, and the computer program of the present invention itself or a compressed file including an automatic installation function is downloaded from the homepage to a recording medium such as a hard disk. Can also be supplied. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of the processes and performing the processing.

  Furthermore, after the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a block diagram for implement | achieving the automatic speech balloon creation and edit processing function of embodiment which concerns on this invention. It is a figure which shows the structure of the video recording / editing apparatus which has the automatic speech balloon creation / edit processing function shown in FIG. FIG. 3 is an external view of the video recording / editing apparatus of FIG. 2. It is the figure which showed a mode that the image | video, an audio | voice, a speech balloon, a supertitle | superscript, and a synthesized image were produced | generated in the automatic speech balloon creation / edit process. It is a figure which shows the image image (A child) produced in an automatic speech balloon preparation / edit process. It is a figure which shows the image image (A child and a speech balloon) produced in automatic speech balloon creation and edit processing. It is a figure which shows the image image (A child and B child) produced in automatic speech balloon production / edit processing. It is a figure which shows the image image (A child, B child, and a speech balloon) produced in automatic speech balloon production / edit processing. It is a figure which shows the image image (A child, B child, and subtitle super) created in an automatic speech balloon preparation / edit process. It is a flowchart showing the voice / face correspondence data registration processing in the automatic speech balloon creation / edit processing function. It is a flowchart which shows speech balloon creation and speech balloon editing. It is a figure which shows the example of a display screen at the time of performing automatic speech balloon preparation and edit processing. It is a figure which shows the example of a display screen at the time of performing the audio | voice and face corresponding | compatible registration process of FIG. It is a figure which shows the example of an edit screen at the time of performing speech balloon edit processing. It is a figure which shows the example of an edit screen at the time of performing a caption super editing process. It is a figure which shows the example of a display screen at the time of previewing an automatic balloon production | generation / editing process result. It is a figure which shows the moving image data editing apparatus provided with the software which implement | achieves the automatic speech balloon creation and edit processing function of embodiment which concerns on this invention.

Explanation of symbols

100 Automatic speech balloon creation / editing processing section
101 Video input section
102 Audio input section
103 Face detector
104 Voice identification part
105 Voice recognition unit
106 Voice / face support
107 Speaker Identification Department
108 Texting Department
109 Positioning part
110 Synchronization part
111 Image data section
112 Callout generator
113 Audio data section
114 Moving image generator
200 Video recording / editing equipment
201 Camera section
202 Video A / D converter
203 Image processing unit
204 Microphone input section
205 Voice A / D converter
206 Audio signal processor
207 Controller
208 Control members
209 Synchronization part
210 Recording device
211 Moving image / balloon composition processing part
212 Speaker
213 display
300 Video recording / editing equipment
301 Shooting button
302 Eyepiece (viewfinder)
303 Photo lens
304 LCD viewfinder, playback screen
305 Operation buttons
406 Speech balloon (child A)
407 Speech balloon (child B)
408 Subtitle Supermarket (C Child)
501 video area
503,504,505 Speaker information
510 audio signal level
511 Voice start time code
512 utterance end time code
513 Current display video time code
514 Operating status
515 Start button
516 Preview button
520 Voice / Face Registration Screen
521 Person name input field
522 Face display area
525 Audio level display area
526 Face image import button
527 Audio capture button
530 Callout property setting screen
541 Speech bubble edit screen
542 Image confirmation area
543 Text display / edit area
544 slider
549 Voice recognition button
550 Audio playback button
551 Confirm button
561 Subtitle Super Setting Screen
562 Image confirmation area
563 Text display / edit area
564 Slider
569 Voice recognition button
570 Audio playback button
571 Confirm button
580 video area
581 Video time code
582 Button to move to the last utterance start time code
583 Rewind button
584 Play button
585 Fast-forward button
586 Button to move to the next voice start time code
588 timecode scale
589 Index Image
590 Callout information display
591 Subtitle super information display
592 slider
600 personal computer
601 Display device
602 keyboard
603 mouse

Claims (22)

An apparatus for creating subtitles from original video data including images and sound,
Face detection means for detecting a facial feature amount from an image portion of the original moving image data;
A voice identifying means for detecting a voice feature amount from a voice portion of the original moving image data;
The feature amount of the face detected by the face detection unit and the feature amount of the speech detected by the speech identification unit are identified with the voice feature amount for identifying the voice of the speaker prepared in advance and the face of the speaker. A speaker identification means for identifying a speaker in comparison with a facial feature,
Position specifying means for specifying the face position of the specified speaker;
Voice recognition means for recognizing a character string from the identified voice of the speaker and generating text data of the character string;
Based on the face position obtained by the position specifying unit and the text data generated by the voice recognition unit, the text data of the character string uttered from the specified speaker is displayed in the display screen. A balloon creating means for creating balloon data;
A moving image creating means for creating new moving image data by adding the balloon data to the original moving image data;
The speech balloon creating means has a speech balloon shape, color, pattern, size, and character color, size, and font corresponding to the specified speaker for the speech balloon data created by the speech balloon creating means. Having a balloon editing means for displaying a balloon editing screen for editing at least one of them;
The balloon editing screen includes an image display area for displaying the new moving image data, a text display area for editing the balloon data, and a voice recognition operation unit for executing voice recognition by the voice recognition means. When, seen including and a reproduction operation portion for performing reproduction of the sound,
The speech balloon creating means recognizes a speech corresponding to the speaker's face position when the speaker's voice is recognized by the speaker identifying means but the face cannot be recognized or when the speaker disappears from the display screen. An apparatus for generating data for displaying only a character string as a subtitle super in the area below the display screen instead of data . Further comprising synchronization means for managing from the start of speaking to the end of speaking by synchronizing the image and sound;
The speech balloon creating means is based on the face position obtained by the position identifying means, the text data generated by the speech recognition means, and the time from the start of speech to the end of speech obtained by the synchronization means. The apparatus according to claim 1, wherein the apparatus creates data. The face detecting means detects a face direction and a mouth position from the image portion;
The apparatus according to claim 1, wherein the position specifying unit specifies the face direction and mouth position of the speaker so that the balloon data can be displayed in accordance with the face direction.   The apparatus according to claim 1, wherein the balloon creating unit changes a balloon size and a character size in accordance with the face position and size specified by the position specifying unit. When the speech balloon creation unit cannot recognize the speaker's face between the start of speech and the end of speech, the balloon creating unit displays only a character string as a subtitle super in the area below the display screen from the point when the speech cannot be recognized. apparatus according to claim 2, characterized in that to create the data for.   The speech balloon creation means tracks the face of the speaker by the face detection means between the start of the utterance and the end of the utterance, and if the face of the speaker cannot be recognized, the head is traced to finish the utterance. The apparatus according to claim 2, wherein balloon data up to is created. When the speaker moves beyond a preset movement amount in the display screen between the start of the utterance and the end of the utterance, the speech balloon creating means is a character string as a subtitle super in the area under the display screen. apparatus according to claim 2, characterized in that to create the data for display only.   The apparatus according to claim 1, wherein the character string of the balloon data is described in a language including metadata described by text data. The apparatus according to any one of claims 1 to 8 , further comprising moving image input means capable of capturing the moving image data or inputting the moving image data from outside. It further comprises recording means for recording moving image data,
The moving image creating means generates the balloon data from the original moving picture data input by the moving picture input means, and sequentially records new moving picture data created by adding the balloon data to the recording means. The apparatus of claim 9 . A method for creating subtitles from original video data including images and sounds,
A face detection step of detecting a feature amount of a face from an image portion of the original moving image data;
A voice identification step of detecting a voice feature amount from a voice portion of the original moving image data;
The feature amount of the face detected in the face detection step and the feature amount of the speech detected in the speech identification step, the speech feature amount for identifying the voice of the speaker prepared in advance, and the face of the speaker A speaker identification step for identifying a speaker in comparison with a facial feature to be identified;
A position specifying step for specifying the face position of the specified speaker;
A speech recognition step of recognizing a character string from the voice of the identified speaker and generating text data of the character string;
Displaying text data of a character string uttered from the specified speaker on the display screen based on the face position obtained by the position specifying step and the text data generated by the voice recognition step A speech balloon creation process for creating speech balloon data;
A moving image creation step of creating new movie data by adding the balloon data to the original movie data,
In the speech balloon creation process, the speech balloon data created in the speech balloon creation process has a speech balloon shape, color, pattern, size corresponding to the specified speaker, and character color, size, font type. Having a speech balloon editing step for displaying a speech balloon editing screen for editing at least one of them,
The balloon editing screen includes an image display area for displaying the new moving image data, a text display area for editing the balloon data, and a voice recognition operation unit for executing voice recognition by the voice recognition step. When, seen including and a reproduction operation portion for performing reproduction of the sound,
In the speech balloon creating step, when the voice of the speaker is recognized in the speaker specifying step but the face cannot be recognized or the speaker disappears from the display screen, the speech balloon corresponding to the speaker's face position is used. A method for generating data for displaying only a character string as a subtitle super in the lower area of the display screen instead of data . Further comprising a synchronization step of synchronizing the image and the sound to manage from the start of speaking to the end of speaking.
In the balloon creating step, the balloon is based on the face position obtained by the position identifying step, the text data generated by the voice recognition step, and the time from the utterance start to the utterance end obtained by the synchronization step. The method according to claim 11 , wherein data is created. In the face detection step, a face direction and a mouth position in the image data are detected,
12. The method according to claim 11 , wherein in the position specifying step, the direction of the speaker's face and the position of the mouth are specified so that the balloon data can be displayed in accordance with the direction of the face. 12. The method according to claim 11 , wherein, in the balloon creating step, the balloon size and the character size are changed in accordance with the face position and size specified in the position specifying step. In the speech balloon creation step, if the speaker's face cannot be recognized between the start of speech and the end of speech, only the character string is displayed as the subtitle super in the area below the display screen from the point when the speech cannot be recognized. the method of claim 12, wherein the creating the data to. In the balloon creating step, the face of the speaker is tracked by the face detection step between the start of the utterance and the end of the utterance, and when the face of the speaker cannot be recognized, the head is tracked and the utterance is ended. 13. The method according to claim 12 , wherein the balloon data up to is created. In the step of creating the speech balloon, if the speaker moves beyond the movement amount set in advance on the display screen from the start of the utterance to the end of the utterance, only a character string as a subtitle super in the area below the display screen the method of claim 12, wherein the creating data for displaying. The method according to claim 11 , wherein the character string of the balloon data is described in a language including metadata described by text data. The method according to any one of claims 11 to 18 , further comprising a moving image input step of capturing the moving image data or inputting the moving image data from outside. A recording step of recording the moving image data in the recording means;
In the moving image creation step, the balloon data is generated from the original moving image data input in the moving image input step,
The method according to claim 19 , wherein in the recording step, new moving image data created by adding the balloon data is sequentially recorded in the recording means. A program for causing a computer to execute the method according to any one of claims 11 to 20 . A computer-readable storage medium storing the program according to claim 21 .

Images