JP2005124169A - Video image contents forming apparatus with balloon title, transmitting apparatus, reproducing apparatus, provisioning system, and data structure and record medium used therein - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a forming apparatus of video image contents wherein the relation between an utterance person and a title is intelligible and the whole screen is easy to see; a transmitting apparatus; a reproducing apparatus, a providing system, and a data structure and a recording medium which are used in them. <P>SOLUTION: The contents forming apparatus 1 forms balloon data required for offering video image contents with titles by balloons. In the balloon data, at least one information out of information on time duration which should display a balloon, information on a region which displays the balloon, information on a profile of the balloon and information on title characters to be inserted in the balloon is stored. The contents transmitting apparatus 2 multiplexes the balloon data and contents data and makes a broadcast instrument 3 broadcast them. The contents reproducing apparatus 4 analyzes the balloon data, generates a signal for a balloon image and a signal for title characters, synthesizes the signals with a video image picture and makes a content indicating apparatus 5 display a video image with a title by the balloon. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a video content creation device, a transmission device, a playback device, a providing system, and a data structure and a recording medium used therein, and more specifically, a video content creation device with captions, a transmission device, and a playback device. The present invention relates to a providing system and a data structure and a recording medium used in the providing system.

  Conventionally, in order to understand the contents of foreign language movies, conversations by characters have been displayed in native language characters on the top, bottom, left and right of the video screen. This allowed viewers to understand what the characters were talking in in a foreign language. In addition, recently, for reasons of production, even if the characters are talking in their native language, the contents of the characters spoken by characters are displayed around the top, bottom, left, and right of the screen. There is also a broadcast. In addition to the content of the characters' conversation, characters may be displayed around the top, bottom, left and right of the screen for explanation of the screen. Thus, the characters displayed around the top, bottom, left and right of the screen are called subtitles. By displaying the subtitles in the video, the viewer can understand the conversation of the characters in the video and understand the content of the video.

  In recent years, various methods have been proposed in order to make it easy to understand the relationship between a speaker on a screen and subtitles. For example, when the speaker is a female, there is a method in which the color of the subtitle is warm, and when the speaker is a male, the color of the subtitle is cold or the name of the speaker is displayed.

In addition, in order to make it easy to visually understand the relationship between a speaker and subtitles on the screen, a method of displaying subtitles at the mouth of the speaker has also been proposed (see Patent Document 1). The apparatus according to Patent Document 1 three-dimensionally calculates the position of a speaker on the screen, the position of the speaker's mouth, and the body direction of the speaker. Further, the apparatus calculates the direction in which the speaker on the screen is speaking three-dimensionally. The device displays the utterance character on the reference line by making the sound generation direction two-dimensionally on the display screen as a reference line.
JP-T 9-505671

  In general, even when there are subtitles, the viewer outputs audio and recognizes features such as the level of audio along with the subtitles to recognize who the speaker is. Therefore, when conventional subtitles are used, if the sound is completely erased, the viewer cannot understand who is speaking on the screen. This is particularly problematic when there are multiple speakers on the screen.

  In addition, it is possible to suggest who the speaker is by changing the color of the characters as in the past, but this method only gives the viewer a hint about who the speaker is. If the audio is muted, the viewer may not be able to fully understand the speaker.

  In addition, the name of the speaker can be displayed by displaying the name of the speaker, but there are significant disadvantages such as an increase in the number of characters.

  Furthermore, as in Patent Document 1, the method of displaying subtitles along the reference line from the mouth of the speaker also hides the faces of characters other than the speaker or hides important screens. There is a problem such as being loose.

  As described above, the conventional video display method using subtitles has a problem that it is difficult to understand the relationship between the speaker and the subtitle, and even if the relationship between the speaker and the subtitle is easy to understand, The screen was difficult to see.

  SUMMARY OF THE INVENTION Therefore, an object of the present invention is to create a video content creation device, a transmission device, a playback device, a providing system, and a data structure and a recording medium used in them, in which the relationship between a speaker and subtitles is easily understood and the entire screen is easy to see Is to provide.

  A further object of the present invention is to provide a video content creation device, a transmission device, a playback device, a providing system, and a system for creating a video content that makes it easy to understand the relationship between a speaker and subtitles and to easily view the entire screen even if the voice is turned off. It is to provide a data structure and a recording medium to be used.

  In order to solve the above problems, the present invention has the following features. The present invention is a content creation apparatus for creating data necessary for providing video content with subtitles by speech balloons. The content creation device includes speech balloon time extraction means, speech balloon area determination means, speech balloon image determination means, subtitle character determination means, and speech balloon data creation means. The balloon time extracting means extracts a time for displaying the balloon in the video based on the video content data as the original data. The balloon area determining means determines a balloon area suitable for displaying a balloon in the video at the time extracted by the balloon time extracting means. The balloon image determining unit determines a balloon image to be combined with the balloon region determined by the balloon region determining unit. The subtitle character determining means determines a subtitle character to be combined with the balloon image determined by the balloon image determining means. The speech balloon data creating means creates speech balloon data by converting at least one of the information regarding the time for displaying the speech balloon, the information regarding the speech balloon area, the information regarding the speech balloon image, and the information regarding the subtitle character into data. The speech balloon data created by the speech balloon data creating means is reproduced together with the video content data, thereby providing video content with subtitles by the speech balloon.

  Preferably, the balloon area determination unit detects a change in color tone in the image based on the video content data, extracts a portion having a flat color tone, and sets a frame included in the flat portion as a balloon region. The balloon image determination means may use an image having a size that displays the subtitle characters in the frame as the balloon image.

  Further, preferably, the balloon area determining means may determine the balloon area by changing the extracted frame based on an instruction from the user. The balloon image determining means may change the shape of the balloon image based on an instruction from the user. The subtitle character determining means may change the subtitle character based on an instruction from the user.

  Further, the subtitle character determining means determines whether or not the number of subtitle characters per unit time in the time when the speech balloon is to be displayed is equal to or greater than a predetermined number. The user should be notified.

  Preferably, the subtitle character determination means may change the attribute of the subtitle character based on an instruction from the user.

  Furthermore, the content creation device may include multiplexing means for multiplexing the video content data and the balloon data created by the balloon data creation means. Furthermore, the content creation device may further include multiplexed data transmission means for transmitting the data multiplexed by the multiplexing means via the network. Furthermore, the content creation device may further include package media storage means for storing the data multiplexed by the multiplexing means in the package media.

  Furthermore, the content creation device may include a volume determination unit that determines the volume of the sound being played back of the video content data. At this time, the subtitle character determining means may change the attribute of the subtitle character according to the sound volume determined by the sound volume determining means.

  Furthermore, the content creation device may include a face size extraction unit that extracts the size of a person's face in the video based on the video content data. At this time, the balloon image determining means may determine the start point of the balloon image according to the face size extracted by the face size extracting means.

  Preferably, the video content data is encoded by the MPEG (Moving Picture Experts Group) method, and the balloon data is described by XML (extensible Markup Language).

  The present invention is also a content transmission apparatus for transmitting data necessary for providing video content with subtitles by speech balloons. The content transmission apparatus includes balloon data acquisition means, video content data acquisition means, multiplexing means, and transmission means. The balloon data acquisition means includes information relating to the time for displaying the balloon in the video based on the video content data as the original data, information relating to the area in which the balloon is displayed in the video, information relating to the shape of the balloon in the area, and Balloon data in which at least one piece of information is converted into data among the information about the subtitle characters to be inserted is acquired. The video content data acquisition means acquires video content data. The multiplexing means, the balloon data acquired by the balloon data acquisition means, and the video content data acquired by the video content data acquisition means are multiplexed. The transmission means transmits the data multiplexed by the multiplexing means.

  For example, the transmission means may transmit multiplexed data to a broadcasting device for wireless broadcasting, or to a content playback device for playing back video content data and balloon data via a network. Multiplexed data may be transmitted.

  The present invention is a content package mediating device for packaging data necessary for providing video content with captions by speech balloons. The content package media conversion apparatus includes balloon data acquisition means, video content data acquisition means, multiplexing means, and storage means. The balloon data acquisition means includes information relating to the time when the balloon should be displayed in the video based on the video content data as the original data, information relating to the area in which the balloon is displayed in the video, information relating to the shape of the balloon in the area, and Balloon data in which at least one piece of information is converted into data among the information about the subtitle characters to be inserted is acquired. The video content data acquisition means acquires video content data. The multiplexing unit multiplexes the balloon data acquired by the balloon data acquisition unit and the video content data acquired by the video content data acquisition unit. The storage means stores the data multiplexed by the multiplexing means in the package medium.

  The present invention is also a content playback device for playing back video content with subtitles by a balloon. The content reproduction apparatus includes balloon data acquisition means, video content data acquisition means, balloon signal generation means, caption character signal generation means, video signal generation means, and synthesis transfer means. The balloon data acquisition means includes information relating to the time when the balloon should be displayed in the video based on the video content data as the original data, information relating to the area in which the balloon is displayed in the video, information relating to the shape of the balloon in the area, and Balloon data in which at least one piece of information is converted into data among the information about the subtitle characters to be inserted is acquired. The video content data acquisition means acquires video content data. The balloon signal generation means generates a signal related to the balloon image based on the balloon data. The subtitle character signal generating means generates a signal related to the subtitle character based on the balloon data. The video signal generation means generates a video signal based on the video content data. The composition transfer means synthesizes the speech signal generated by the speech signal generation means, the caption character signal generated by the caption character signal generation means, and the video signal generated by the video signal generation means, and transfers the composite signal to the display device.

  Further, the content reproduction apparatus may include a compositing presence / absence command unit that commands the compositing / transfer unit to synthesize the speech signal and the subtitle character signal with the video signal. At this time, when receiving a command for synthesizing the balloon signal and the subtitle character signal with the video signal from the synthesis presence / absence command unit, the synthesis transfer unit transfers the synthesized signal to the display device and receives a command for not synthesizing. In this case, only the video signal may be transferred to the display device.

  Furthermore, the content reproduction apparatus includes a volume measuring unit that measures the volume of surrounding sounds, and a volume threshold value determining unit that determines whether the volume of sound measured by the volume measuring unit exceeds a threshold value. Good. At this time, the compositing presence / absence command means may instruct the compositing / transfer means to determine whether or not to composit according to the determination result of the volume threshold value determining means.

  Preferably, the synthesis presence / absence command means gives an instruction to synthesize to the synthesis transfer means when the volume threshold judgment means judges that the volume of the surrounding sound does not exceed the threshold, and further outputs the voice It is better not to output audio to the audio output device.

  Further, the synthesis presence / absence command means may give the synthesis transfer means a command to synthesize when the volume threshold judgment means judges that the volume of the surrounding sound exceeds the threshold.

  Furthermore, the content playback apparatus may include a moving speed measuring unit that measures the moving speed of the device itself. At this time, the compositing presence / absence command means determines whether or not the moving speed measured by the moving speed measuring means exceeds a predetermined threshold value.

  Further, the synthesis presence / absence command means may instruct the synthesis transfer means whether or not to synthesize the speech signal and the subtitle character signal with the video signal in accordance with an instruction from the user.

  Further, the subtitle character signal generation means may generate a normal subtitle character signal for displaying the subtitle character on either the top, bottom, left or right of the screen based on the balloon data in accordance with an instruction from the user. At this time, when the subtitle character signal generation unit generates the normal subtitle character signal, the synthesis transfer unit may synthesize only the normal subtitle character signal and the video signal and transfer the synthesized signal to the display device.

  Preferably, the synthesis transfer unit may synthesize the speech balloon signal, the caption character signal, and the video signal for each frame.

  Furthermore, preferably, the content reproduction apparatus may include a display unit that displays the combined video based on the combined signal transferred from the combined transfer unit.

  The present invention is also a computer-readable recording medium on which data having a structure for displaying video content with subtitles by a balloon is recorded on a computer device. The recording medium has a structure for storing information related to the time for displaying the speech balloon in the video based on the video content data as the original data, and information related to the area for displaying the speech balloon in the video corresponding to the information related to the time. Corresponding to the structure for storing the text and information about the time, the structure for storing the information about the shape of the speech bubble in the area, and the information about the subtitle character to be inserted into the speech balloon corresponding to the information about the time It is preferable that data having a structure to be stored is stored.

  Preferably, the structure for storing information relating to time may be composed of a structure for storing information indicating the start time of captions and a structure for storing information indicating time to continue captions.

  In addition, the present invention is a data structure as described above for displaying video content with captions by a balloon on a computer device.

  The present invention is also a content providing system for providing video content with subtitles by a speech balloon, wherein the information regarding the time to display the speech balloon in the video based on the video content data as the original data, the speech balloon in the video A content creation device that creates speech balloon data in which at least one of the information related to the area to be displayed, the information related to the shape of the speech balloon in the region, and the information related to the subtitle character inserted in the speech balloon is converted into data, and the content creation device The content providing means for multiplexing the speech balloon data and the video content data created by and providing the multiplexed data as video content, and the video with subtitles by the speech balloon based on the multiplexed data provided from the content providing means Content And a content reproduction apparatus which live.

  The content providing means may transmit the multiplexed data to the content reproduction device by wireless broadcasting, may transmit the multiplexed data to the content reproduction device by network distribution, or may multiplex through the package media. The structured data may be provided to the content reproduction apparatus.

  According to the present invention, subtitle characters can be inserted and displayed in a speech balloon portion in video content, so that the relationship between a speaker and subtitles can be easily understood. Further, since the subtitle characters are displayed in the balloon portion, the screen is easy to see. The speech bubble has a starting point, and the starting point indicates who the speaker is. Therefore, even if no voice is output, the user can associate the speaker with the subtitle character. The content of the video can be recognized. This is more useful in a quiet place where no sound can be output, or a place where the surrounding sound is loud and it is difficult to hear the sound from the speaker. Further, by incorporating it into a mobile communication terminal, the contents of the video can be recognized without listening to sound from headphones or the like.

  In addition, since the balloon is provided in a portion where the color tone is flat, it is possible to avoid a situation where an important portion is hidden on the screen. In addition, since the area for displaying the balloon image can be changed according to the user's instruction, it is possible to consciously prevent the important part from being hidden. Moreover, since the shape of the speech balloon image can be changed, an appropriate speech balloon can be selected according to the content of the speech of the speaker. For example, a cloud-like balloon can be used for things that are in mind. In addition, since the subtitle characters can be changed, it is possible to add processing when emphasis is desired.

  When the number of subtitle characters is large, the user is automatically notified, so that an appropriate subtitle character can be created.

  By using MPEG data as the video content data and using XML-compliant data as the balloon data, the affinity can be increased and contribution to standardization can be achieved.

  Since the content reproduction apparatus can control the output of audio and the display of subtitle characters according to the volume of surrounding sounds, the output corresponding to the place is automatically provided.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the overall configuration of a broadcasting system for broadcasting video content with captions using a speech balloon according to an embodiment of the present invention. In FIG. 1, the broadcast system includes a content creation device 1, a content transmission device 2, a broadcast device 3, a content reproduction device 4, and a content display device 5. In FIG. 1, only one content creation device 1, content transmission device 2, broadcast device 3, content reproduction device 4, and content display device 5 are shown for simplicity, but there may be two or more. Good.

  The content creation device 1 synthesizes data indicating a list of subtitle characters matching the video based on the content data stored in advance (hereinafter referred to as subtitle list data), and video with subtitles using speech bubbles on the video based on the content data. Create speech balloon data to be used.

  The content transmission device 2 acquires content data and speech balloon data from the content creation device 1 and multiplexes these data as multiplexed data via a network such as a local line, public network, Internet, radio network, etc. 3 to send. The content creation device 1 and the content transmission device 2 exist on the content creator side such as a production company, for example. Here, the multiplexed data is transmitted to the broadcasting device 3 via the network. However, the multiplexed data is stored in a recording medium such as a DVD so that the broadcasting device 3 can read the multiplexed data. It may be.

  The broadcast device 3 receives the multiplexed data sent from the content transmission device 2 and broadcasts it via the antenna. The broadcast device 3 is present on the broadcast provider side such as a television station, for example.

  The content reproduction device 4 receives and analyzes the multiplexed data transmitted from the broadcasting device 3 and causes the content display device 5 to display a video with captions using a balloon. The content display device 5 displays a video with captions using a balloon in response to a signal sent from the content playback device 4. The content reproduction device 4 and the content display device 5 exist in the home of a viewer or the like, for example.

  FIG. 2 is a block diagram illustrating a functional configuration of the content creation device 1. In FIG. 2, the content creation device 1 includes a data creation control unit 11, an input unit 12, a display output unit 13, a time count unit 14, and a storage unit 15.

  The input unit 12 is an input device such as a mouse, a keyboard, a touch panel, and a joystick, and inputs operation information input by the user to the data creation control unit 11 by operating these devices.

  The storage unit 15 is a recording device such as a hard disk. The storage unit 15 stores content data, subtitle list data, balloon shape data, and balloon data.

  Content data is encoded stream data obtained as a result of encoding video and audio by an encoding method such as MPEG (Moving Picture Experts Group).

  The subtitle list data stores subtitle characters and information about the time for displaying them. FIG. 3 is a diagram illustrating an example of a data structure of caption list data. As illustrated in FIG. 3, for example, a caption start time, a caption duration, and a caption character are registered in the caption list data. Here, the subtitle start time is information indicating how much time has elapsed from the beginning of the content and the corresponding subtitle character is to be displayed. The subtitle duration is information indicating the time during which the corresponding subtitle character is continuously displayed. In FIG. 3, an example of subtitle list data for displaying the subtitle character “I agree with your idea” for 0 minute 2 seconds from the start of the content at 0 hour 24 minutes 30 seconds 15 frames. Show. The subtitle list data shown in FIG. 3 is an example, and the present invention is not limited to this. Also, the number of frames per second is not particularly limited.

  The balloon shape data is data defining the shape of the balloon. For example, the balloon shape data is associated with the name of the balloon shape and information regarding the balloon shape.

  FIG. 4 is a diagram illustrating an example of the data structure of the balloon data. As shown in FIG. 4, for example, the speech balloon data includes, for each subtitle start time, subtitle duration, subtitle character expansion speed, subtitle character attributes, speech balloon range, speech balloon start point, speech balloon shape, corresponding to the content data name. , And subtitle characters are described. The caption start time and caption caption duration are information relating to the time for which a speech balloon should be displayed. The subtitle character development speed, the subtitle character attribute, and the subtitle character are information on the subtitle character. The speech balloon range and the speech balloon start point are information relating to the speech balloon area in the video suitable for displaying the speech balloon. The balloon shape is information regarding a balloon image to be combined with the balloon area. The speech balloon data may be data obtained by converting at least one of the information regarding the time for displaying the speech balloon, the information regarding the speech balloon area, the information regarding the speech balloon image, and the information regarding the subtitle character into data. For example, the balloon data is described in a meta language. Here, the subtitle start time, subtitle duration, and subtitle characters are the same as those of the subtitle list data. The subtitle character expansion speed is information indicating the speed at which subtitle characters are sequentially displayed from the beginning of the sentence within the subtitle duration. The subtitle character attribute is information indicating the font type of the subtitle character, the color of the subtitle character, the background and transparency of the subtitle character, the type of the subtitle character frame, and the like. The balloon range is information indicating the position on the screen where the balloon is synthesized. The balloon start point is information indicating the position on the screen where the balloon starts. The balloon shape is information indicating the name of the balloon registered in the balloon data.

  As described above, the balloon data has a structure for causing the computer device to display video content with captions by the balloon. Such a structure includes a structure for storing information related to the time for which a speech balloon should be displayed in the video based on the video content data serving as the original data (for example, the above-described caption start time and caption duration), and information related to the time. Corresponding to the structure for storing information (for example, the above-mentioned speech balloon range and speech balloon starting point) in the video, and the information regarding the time, the shape of the speech balloon in the area Corresponding to the structure for storing information (for example, the above-mentioned balloon shape) and the information about time, information about the subtitle character to be inserted into the balloon (for example, the above-mentioned subtitle character expansion speed, subtitle character attribute, subtitle character) And a structure for storing. In the present embodiment, the structure for storing information related to time is composed of a structure for storing information indicating the start time of captions and a structure for storing information indicating time to continue captions. . Data having such a structure can be stored in a computer-readable recording medium.

  The time count unit 14 measures time. The display output unit 13 displays a video and an image for creating a speech balloon according to a signal from the data creation control unit 11, and outputs a sound.

  The data creation control unit 11 reproduces the content data, detects the rise time and fall time of the audio, and recognizes the caption start time and the caption duration time. The data creation control unit 11 creates subtitle list data in association with the recognized subtitle start time and subtitle duration and the subtitle characters input by the user via the input unit 12 and stores the subtitle list data in the storage unit 15. The data creation control unit 11 refers to the caption list data, detects the sound rise time, and causes the display output unit 13 to display and output the video and audio at the display time. The data creation control unit 11 synthesizes a balloon shape with the displayed video and synthesizes a subtitle character in the balloon shape. If the final decision is made by the user as a result of the synthesis, the data creation control unit 11 creates balloon data at the caption start time. The data creation control unit 11 generates final speech balloon data by combining speech balloon data created at each caption start time. The data creation control unit 11 stores the generated final balloon data in the storage unit 15.

  FIG. 5 is a block diagram illustrating a functional configuration of the content transmission device 2. In FIG. 5, the content transmission apparatus 2 includes a multiplexing control unit 21, an operation unit 22, an error correction code addition unit 23, a digital modulation unit 24, and a transmission unit 25.

  The operation unit 22 is an input device such as a mouse or a keyboard, and inputs information related to content data to be broadcast to the multiplexing control unit 21 in accordance with a user instruction.

  Based on the information from the operation unit 22, the multiplexing control unit 21 reads out the content data desired by the user from the storage unit 15 of the content creation device 1 and the corresponding balloon data and multiplexes the two data. Hereinafter, the multiplexed data is referred to as multiplexed data.

  The error correction code adding unit 23 adds an error correction code to the multiplexed data multiplexed by the multiplexing control unit 21. The digital modulation unit 24 digitally modulates the multiplexed data to which the error correction code is added. The transmission unit 25 transmits the digitally modulated multiplexed data to the broadcasting device 3. The content data and the balloon data may be multiplexed in advance by the content creation device 1. Moreover, the transmission function of multiplexed data may be included in the content creation device.

  The broadcast device 3 converts the multiplexed data sent from the content transmission device 2 into a radio wave and radiates it. Since the internal configuration of the broadcasting device 3 is the same as a known one, detailed description thereof is omitted.

  FIG. 6 is a block diagram showing a functional configuration of the content reproduction apparatus 4. In FIG. 6, the content reproduction apparatus 4 includes a reproduction control unit 41, an operation unit 42, a composition transfer unit 43, a time count unit 44, a balloon shape storage unit 45, a reception unit 46, a demodulation unit 47, And an error correction unit 48.

  The receiving unit 46 receives radio waves broadcast by the broadcasting device 3. The demodulator 47 demodulates the radio wave received by the receiver 46. The error correction unit 48 refers to the error correction code included in the multiplexed data demodulated by the demodulation unit 47 and corrects the error.

  The operation unit 42 is an input device such as a remote controller or button switch for the user to control the operation of the content reproduction apparatus 4. The time count unit 44 counts the time for reproducing the content data. The balloon shape storage unit 45 stores balloon shape data, like the storage unit 15 of the content creation device 1.

  The reproduction control unit 41 reads content data from the multiplexed data that has been error-corrected by the error correction unit 48, and synthesizes and forwards video and audio signals (hereinafter referred to as video signals and audio signals) for each frame. 43. Further, the reproduction control unit 41 reads out the balloon data from the multiplexed data that has been error-corrected by the error correction unit 48, and outputs the balloon data from the balloon shape storage unit 45 according to the information regarding the balloon shape included in the balloon data. Data related to the shape is read, a signal related to the balloon image (hereinafter referred to as a balloon signal) is generated, and is sent to the synthesis transfer unit 43. Note that the balloon signal may be the same over a plurality of frames, but here, the reproduction control unit 41 sends the balloon signal to the synthesis transfer unit 43 for each frame. The reproduction control unit 41 creates a signal related to the subtitle character to be inserted into the balloon (hereinafter referred to as a subtitle character signal) according to the subtitle character included in the balloon data, and sends it to the composition transfer unit 43. send. The receiving unit 46 may be outside the content reproduction device 4.

  The composition transfer unit 43 synthesizes the signals sent from the reproduction control unit 41 for each frame, and transfers them to the content display device 5.

  FIG. 7 is a block diagram showing a functional configuration of the content display device 5. In FIG. 7, the content display device 5 includes a display output device unit 51 and a drive circuit unit 52. The display output device unit 51 is a cathode ray tube, a liquid crystal display, a speaker, or the like. The drive circuit unit 52 causes the display output device unit 51 to reproduce video and audio in accordance with the synthesized signal and audio signal transmitted from the content reproduction device 4.

  FIG. 8 is a flowchart showing the operation of the content creation device 1. 9A to 9D are diagrams illustrating examples of display contents in the content creation device 1. Hereinafter, the operation of the content creation device 1 will be described with reference to FIG. 8 and FIGS. 9A to 9D.

  First, the data creation control unit 11 of the content creation device 1 reads out desired content data stored in the storage unit 15 in accordance with an instruction from the user input via the input unit 12, and displays a display output unit. The video is displayed on 13 and the sound is output (step S101).

  Next, the data creation control unit 11 determines whether or not the voice rising timing comes by voice recognition (step S102). If the voice rise timing has not arrived, the data creation control unit 11 proceeds to the operation of step S104. On the other hand, when the voice rise timing has arrived, the data creation control unit 11 uses the voice rise timing as the caption start time and the elapsed time until the voice falls as the caption continuation time, and the caption corresponding to the voice during this time Let the user enter characters. The data creation control unit 11 stores the subtitle start time, the subtitle duration, and the subtitle character as part of the subtitle list data in the storage unit 15 (step S103), and proceeds to the operation of step S104. At this time, the user may input subtitle characters by division.

  In step S104, the data creation control unit 11 determines whether or not the reproduction of the content data has been completed. If not all have been completed, the data creation control unit 11 returns to the operation of step S102 and continues to create subtitle characters from the next voice rising timing. On the other hand, when all the processes are completed, the data creation control unit 11 combines the caption list data created at any time in step S103 into one, finally creates caption list data for the content, and stores it in the storage unit 15. (Step S105), the process proceeds to Step S106.

  In step S106, the data creation control unit 11 refers to the caption list data and recognizes the caption start time and the caption duration time in order from the top. Next, the data creation control unit 11 refers to the content data and causes the display output unit 13 to reproduce video and audio from the caption start time to the caption duration time recognized in step S106 (step S107).

  Next, the data creation control unit 11 calculates the flatness of the color in the video from the caption start time to the caption continuation time, and extracts a portion having a similar color tone (hereinafter referred to as a flat portion) (step). S108). Next, the data creation control unit 11 recognizes a quadrangle that is large enough to fit into the extracted flat portion (step S109). Next, the data creation control unit 11 causes the display output unit 13 to synthesize and display the image at the start time of the caption so that the recognized rectangle is represented by a dotted frame (hereinafter referred to as a square frame) (step S110). At this time, the data creation control unit 11 displays the four corners of the square frame with black circles. FIG. 9A is a diagram illustrating an example of a screen displayed in step S110. As shown in FIG. 9A, the square frame Sa is displayed so that the color tone is maximally accommodated in the flat portion Fa. The frame may be other than a rectangle.

  Next, the data creation control unit 11 causes the display output unit 13 to display a display asking the user whether or not the square frame represented in step S110 may be the range for displaying the balloon, and if there is a correction instruction from the user The square frame corresponding to the instruction is set as a range for displaying the balloon (step S111). At this time, the data creation control unit 11 temporarily stores the coordinates of the four corners of the square frame in a memory (not shown). Further, correction by the user is performed by the input unit 12. For example, the user modifies the size and / or position of the square frame by positioning the mouse pointer on the four sides or four corners of the square frame and dragging the mouse pointer. Since such a method is well known in the field of image software and the like, further detailed description is omitted.

  Next, the data creation control unit 11 recognizes a human face from the video (step S112). There are various recognition methods at this time, but the data creation control unit 11 can recognize the face portion of the person based on the skin color, the shape of the face, and the like. Since such a method is known in the field of image recognition, further detailed description is omitted.

  Next, the data creation control unit 11 obtains the area of the recognized face portion and determines whether or not the area exceeds a predetermined threshold (step S113). If so, the data creation control unit 11 detects the mouth portion and causes the display output unit 13 to display a reference line from the mouth toward the diagonal line intersection of the square frame (hereinafter referred to as the center of the square frame). The balloon start point candidates are displayed (step S114), and the operation proceeds to step S116.

  On the other hand, if not exceeding, the data creation control unit 11 recognizes the center part of the face, causes the display output unit 13 to display the reference line from the center part toward the center of the rectangular frame, and starts the balloon on the reference line. Are displayed (step S115), and the operation proceeds to step S116. FIG. 9B is a diagram illustrating an example when a balloon start point candidate is displayed in step S115. As shown in FIG. 9B, the balloon start point Pa is displayed on a reference line La drawn from the center of the face toward the center of the square frame Sa. Thus, the data creation control unit 11 determines the start point of the balloon image according to the face size.

  In step S116, the data creation control unit 11 corrects the balloon start point in accordance with an instruction from the user via the input unit 12, and stores the coordinates of the final balloon start point in a memory (not shown). Then, the operation proceeds to step S117. When there is no correction by the user, the data creation control unit 11 stores the coordinates that are candidates for the balloon start point.

  In step S117, the data creation control unit 11 reads from the storage unit 15 data related to a balloon shape that is preset as a standard balloon shape, and if necessary, puts it in the square frame determined in step S111. The size of the balloon shape is changed so as to be accommodated to the maximum, and the balloon image after the size change is displayed on the display output unit 13 within the rectangular frame. FIG. 9C is a diagram illustrating an example of a balloon image displayed in step S117. As shown in FIG. 9C, the balloon image Ba is displayed so as to be contained within the square frame Sa.

  Next, the data creation control unit 11 corrects the balloon image in accordance with an instruction from the user (step S118). Specifically, the shape, size, direction, etc. of the balloon are corrected. Such correction is performed in such a manner that the user selects a desired shape from a dialog box indicating the shape of the balloon. The size correction is performed by dragging a balloon that is being displayed. There are various other correction methods.

  If the correction by the user is completed or there is no correction, the data creation control unit 11 determines a final balloon image (step S119). At this time, the data creation control unit 11 temporarily stores a name indicating the shape of the balloon image in a memory (not shown). If the size of the balloon image has been changed, the data creation control unit 11 uses the coordinates of the four corners of the minimum-sized square frame surrounding the balloon whose size has been changed as a range for displaying the balloon. Change the coordinates of the four corners stored in memory.

  Next, the data creation control unit 11 reads out the subtitle character at the subtitle start time from the subtitle list data, and inserts the subtitle character into the determined balloon (step S120). At this time, the data creation control unit 11 instructs the display output unit 13 to display the subtitle characters frame by frame from the beginning until the subtitle duration time elapses from the subtitle start time. At this time, the data creation control unit 11 determines the subtitle character expansion speed. The subtitle character development speed is defined by showing step by step how many subtitles are newly displayed in one frame. For example, it is defined that six characters are newly displayed in one frame at normal speed. The data creation control unit 11 also temporarily stores the subtitle character expansion speed in the memory. FIG. 9D is a diagram illustrating an example of a display when a caption character is inserted. As shown in FIG. 9D, the subtitle character Ca is displayed in the balloon image Ba.

  Next, the data creation control unit 11 corrects the caption character in accordance with an instruction from the user (step S121). Here, it is assumed that the character type, character color, subtitle background, subtitle transparency, subtitle frame type, subtitle character emphasis, and the like can be corrected as subtitle character attributes. The data creation control unit 11 temporarily stores these subtitle character attributes in the memory. Note that the content creation device 1 may include a volume determination unit that determines the volume of audio during reproduction of video content data. At this time, the content creation device 1 may change the attribute of the subtitle character according to the volume determined by the volume determination unit. For example, when the volume is high, the content creation device 1 may enlarge the caption character or change the color.

  Next, the data creation control unit 11 reads the information temporarily stored in the memory, and subtitle duration, subtitle character expansion speed, subtitle character attribute, balloon range (at the four corners of the square frame) so as to correspond to the subtitle start time. Coordinates), the coordinates of the balloon start point, the balloon shape, and the subtitle characters are stored in the storage unit 15 (step S122).

  Next, the data creation control unit 11 determines whether or not all the contents have been completed (step S123). If not completed, the data creation control unit 11 returns to the operation of step S106 and continues creating the balloon data for each caption start time. On the other hand, if completed, the data creation control unit 11 integrates the balloon data for each subtitle start time created so far into one, and creates final balloon data corresponding to the desired content data. Then, the data is stored in the storage unit 15 (step S124), and the process is terminated.

  FIG. 10 is a diagram illustrating an example of the balloon data finally created. FIG. 10 shows an example in which balloon data is described in a format compliant with XML (extensible Markup Language) in order to maintain compatibility with the MPEG data format used as content data and facilitate standardization. Yes. As shown in FIG. 10, subtitle character expansion speed, subtitle duration, speech bubble range, speech bubble start point, speech bubble shape, and subtitle characters are defined in the speech bubble data for each caption start time. In FIG. 10, the subtitle character attributes are unified as a whole, but may be defined for each subtitle start time.

  FIG. 11 is a flowchart showing the operation of the content transmission device 2. Hereinafter, the operation of the content transmission device 2 will be described with reference to FIG.

  First, the multiplexing control unit 21 of the content transmission device 2 reads out desired content data stored in the storage unit 15 of the content creation device 1 in accordance with an instruction from the user via the operation unit 22 (step S201). ). Next, the multiplexing control unit 21 reads balloon data corresponding to the content data from the storage unit 15 (step S202). Next, the multiplexing control unit 21 multiplexes the read content data and balloon data (step S203). As a multiplexing method, any method such as embedding balloon data in the header portion of the content data may be used.

  Next, the error correction code adding unit 23 adds an error correction code to the multiplexed data (step S204). Next, the digital modulation unit 24 digitally modulates the multiplexed data to which the error correction code is added (step S205). Next, the transmission unit 25 transmits the digitally modulated data to the broadcasting device 3 (step S206), and ends the process.

  FIG. 12 is a flowchart showing the operation of the content reproduction apparatus 4. 13A to 13D are diagrams illustrating examples of the contents of the video signal, the speech balloon signal, and the subtitle character signal created by the content reproduction device 4. Hereinafter, the operation of the content reproduction apparatus 4 will be described with reference to FIG. 12 and FIGS. 13A to 13D.

  First, the content reproduction apparatus 4 demodulates the signal received by the reception unit 46 by the demodulation unit 47, corrects the error by the error correction unit 48, and then inputs the signal to the reproduction control unit 41 (step S301). Next, the reproduction control unit 41 reads content data from the error-corrected multiplexed data, and sends a video signal and an audio signal necessary for reproduction of the content data to the synthesis transfer unit 43 for each frame ( Step S302). The reproduction control unit 41 sends the video signal and the audio signal to the synthesis transfer unit 43 for each frame in parallel with the following operation. FIG. 13A is a diagram illustrating an example of the content of a video signal. As shown in FIG. 13A, in step S302, only information related to video and audio that does not include information related to speech balloons is transferred.

  Next, the reproduction control unit 41 reads out the balloon data from the multiplexed data and recognizes the caption start time and the caption duration time (step S303). Next, the playback control unit 41 determines whether or not the caption start time has arrived based on the information from the time count unit 44 (step S304). If the subtitle start time has not arrived, the playback control unit 41 proceeds to the operation of step S312.

  On the other hand, when the subtitle start time has arrived, the playback control unit 41 recognizes the range on the screen into which the speech balloon is to be inserted based on the speech balloon range stored in the speech balloon data (step S305). Next, the reproduction control unit 41 reads out information related to the balloon shape designated from the balloon shape storage unit 45 based on the balloon shape stored in the balloon data, and falls within the range obtained in step S305. Next, the size of the balloon image is determined (step S306). Next, the reproduction control unit 41 creates a balloon signal that displays a balloon image in the determined range and size, and sends the balloon signal to the composition transfer unit 43 (step S307). At this time, the shape of the speech balloon portion does not change during the caption duration, but the playback control unit 41 is in parallel with other operations in order to facilitate synchronization with the video signal and the caption character signal. A balloon signal is sent to the synthesis transfer unit 43 for each frame. FIG. 13B is a diagram illustrating an example of the content (balloon image) of the balloon signal. As shown in FIG. 13B, the balloon signal provides information relating only to the image of the balloon portion.

  Next, the reproduction control unit 41 recognizes the number of frames existing during the caption duration based on the caption duration stored in the balloon data (step S308). Next, the reproduction control unit 41 divides the number of subtitle characters by the number of frames obtained in step S308, recognizes the number of subtitle characters to be displayed per frame, and displays subtitle characters per frame. A character signal is created (step S309) and sent to the composition transfer unit 43 (step S310). FIG. 13C is a diagram illustrating an example of an image based on the caption character signal in the first frame. FIG. 13D is a diagram illustrating an example of an image based on a caption character signal in the second frame. As shown in FIGS. 13C and 13D, subtitle characters to be displayed are gradually added during the subtitle duration by the subtitle character signal.

  Next, the playback control unit 41 determines whether or not all the frames in the caption duration time have ended (step S311). If not finished, the playback control unit 41 returns to the operation of step S308, creates a subtitle character signal necessary for the next frame, and sends it to the synthesis transfer unit 43. On the other hand, if it has been completed, the playback control unit 41 determines whether or not the content has ended (step S312). If the content has not ended, the playback control unit 41 returns to the operation of step S304 and continues to transfer the balloon signal and the caption character signal from the next caption start time. On the other hand, when the content has ended, the playback control unit 41 ends the process.

  FIG. 14 is a diagram illustrating the operation of the composition transfer unit 43 of the content reproduction apparatus 4. FIG. 15A and FIG. 15B are diagrams showing display examples on the content display device 5. Hereinafter, the operation of the composite transfer unit 43 will be described with reference to FIGS. 14, 15A, and 15B.

  First, the composition transfer unit 43 receives a video signal for one frame sent from the reproduction control unit 41 (step S401). Next, the synthesis transfer unit 43 receives the speech balloon signal and the caption character signal for one frame sent from the reproduction control unit 41, and synthesizes the video signal, the speech balloon signal, and the caption character signal (step S402). ) And the audio signal are transferred to the content display device 5 (step S403), the process returns to step S401, and the process proceeds to the next frame.

  The content display device 5 that has received the signal from the composition transfer unit 43 displays a part of the subtitles as shown in FIG. 15A in the first frame, and as shown in FIG. 15B in the second frame. A subtitle combining the subtitle in the first frame and the subtitle in the second frame is displayed.

  As described above, according to the embodiment of the present invention, subtitle characters can be inserted and displayed in the balloon portion in the video content, so that the relationship between the speaker and the subtitle can be easily understood. Further, since the subtitle characters are displayed in the balloon portion, the screen is easy to see.

  In the content reproduction device and the content display device in the above-described embodiment, even if the sound is muted, it is clear at a glance who is speaking by looking at the balloon start point. Therefore, the content reproduction apparatus and content display apparatus in this embodiment can be effectively used to understand the content of video content even in an environment where no sound can be produced. Thereby, the user can enjoy video content without using a device such as headphones.

  For example, if the content playback device and content display device according to this embodiment are installed as fixed information terminals in a quiet place such as a library, a hospital, or a public facility where sound should not be output, it will cause inconvenience to those around you. You can enjoy video content without spending time. In this case, the content reproduction device and the content display device can be easily realized by a personal computer or the like. In addition, even in an environment where the sound is difficult to hear due to the surrounding sounds being noisy, if installed as an outdoor advertising device or public guidance service device, the video content can be seen without listening to the sound by watching the subtitles in a balloon. Can enjoy.

  In the above embodiment, the content reproduction device and the content display device are separate devices, but the two devices can be integrated and miniaturized so that they can be carried. With such a portable information terminal, you can enjoy video content even in an environment where you have to consider sound as a manner (for example, in a train or bus, in a ship, in a library, a hospital, or in an airplane) be able to. As described above, the present invention has a wide range of utility values.

  In the above embodiment, the content playback device and the content display device have been described as if they were separate devices, but one function may be included in the other. Also, the content creation device and the content transmission device may include one function in the other.

  As described above, in order to have a wide range of useful values, it is better that the content playback device (including the content display device built-in) further has the following functions: .

  For example, the content reproduction device may be able to switch the presence / absence of a balloon display in accordance with a user instruction. In this case, specifically, when the user gives an instruction not to display the speech balloon, the reproduction control unit of the content reproduction apparatus synthesizes only the video signal and the audio signal to the synthesis transfer unit. You can order it.

  Further, the content reproduction apparatus may automatically switch the presence / absence of a balloon display. For example, the content reproduction apparatus further includes a volume measuring unit that measures the surrounding volume. The content reproduction apparatus compares the volume output from the speaker measured by the volume measurement unit with the surrounding volume. As a result of the comparison, when the surrounding sound volume is equal to or higher than a predetermined threshold, the playback control unit of the content playback apparatus instructs the synthesis transfer unit to switch to synthesis by subtitle display by a balloon, and stops the audio output from the speaker. As a result, when the surrounding sound becomes loud, the balloon caption is automatically displayed, so that the video content can be enjoyed even in an environment where the sound is not easily transmitted.

  Also, the playback control unit of the content playback device automatically sets the manner mode when the surrounding volume is lower than a predetermined threshold, stops the audio output from the speaker, and causes the synthesis transfer unit to perform synthesis by subtitle display by a balloon. It may be instructed to switch. Accordingly, when a mobile terminal such as a mobile phone or a PDA is used, if the surroundings are quiet, the manner mode is automatically set and the video content can be enjoyed.

  In addition, the content reproduction apparatus further includes a movement speed measurement unit that calculates an acceleration sensor, a Doppler effect of received radio waves, and the like, and measures how fast the device is moving. When the moving speed measured by the speed measuring unit is faster than the walking speed, the playback control unit of the content playback device determines that the vehicle is driving or is on a vehicle, and switches to subtitle display using a speech balloon as the manner mode. Alternatively, the combination transfer unit may be instructed.

  Note that the content reproduction apparatus may be able to switch between conventional subtitle display and subtitle display using a balloon in accordance with an instruction from the user. Specifically, when the subtitle display in the conventional manner is instructed by the user, the content playback apparatus is designated by the subtitle start time by referring only to the subtitle start time, subtitle duration, and subtitle character information of the balloon data. The subtitle character signal is generated so that the subtitle characters are arranged on either the top, bottom, left, or right of the screen from the current time to the subtitle duration. Then, the synthesis transfer unit synthesizes the subtitle character signal and the video signal and causes the content display device to display the synthesized signal. As a result, conventional subtitles can also be displayed.

  Note that the content creation device may register information for emphasizing characters in the caption list data according to the sound pressure level when creating the caption list data. In this case, specifically, the content creation device includes a sound pressure detection device that detects sound pressure using a piezoelectric sensor or the like, and if the average of the sound pressure during the caption duration is larger than a threshold value, the content is enlarged. The attribute is registered in the caption list data. If the attribute is small, the attribute for reducing the text may be registered in the caption list data.

  If the subtitle characters cannot be entered in the balloon part because the subtitle duration is short, the content creation device notifies the user by displaying on the display output unit with a mark or the like indicating that the subtitle characters cannot be entered. To do. When this notification is given, the user may change the size of the balloon part, the content of the subtitle characters, and the like. Whether or not subtitles can be inserted into the balloon portion by the content creation device is determined by whether or not the number of subtitle characters per unit time (for example, one frame) in the subtitle duration is greater than or equal to a predetermined number. Good. If the number is greater than or equal to the predetermined number, the content creation device may notify the user to change the subtitle character on the assumption that the subtitles cannot be entered.

  If the number of subtitle characters is large, only the subtitle characters that can be entered into the speech balloon portion are displayed, and then the subtitle characters that have not been entered can be newly inserted into the same speech balloon portion. Specifically, this can be easily realized by creating a subtitle character signal composed of subtitle characters that cannot be included when the content reproduction apparatus creates a subtitle character signal in step S309.

  Ideally, the balloon shape data should be standardized. However, if the balloon shape data used between the content creation device and the content playback device is different, the content playback device uses the standard data as the balloon shape data. Use it. What kind of standard data should be used can be determined in advance.

  In the above embodiment, the content creation apparatus creates the caption list data separately from the balloon data, but may create the caption list data together with the balloon data. Specifically, the content creation device may register the balloon shape and the subtitle character at the same time when the voice rising position is recognized.

  In the above embodiment, the caption list data is created immediately before creating the balloon data, but the caption list data may be created in advance separately from the balloon data.

  In the above embodiment, first, the content creation device automatically selects the balloon shape, and then the user makes corrections if necessary, but the content creation device does not allow the user to make corrections, All of the balloon data may be automatically created, or all the balloon data may be manually created.

  In the above embodiment, the content data and the balloon data are broadcast. However, the system for providing the content is not limited to this.

  FIG. 16 is a diagram showing the overall configuration of a system for providing content data and speech balloon data via the Internet. As shown in FIG. 16, the content transmission device 2a may transmit data in which content data and speech balloon data are multiplexed to the content reproduction device 4a via the Internet 3a. In this case, the content creation device 1 and the content display device 5 have the same configuration as in the above embodiment. The content transmission device 2a may transmit packets of multiplexed data according to TCP / IP via the Internet. The content playback apparatus 4a may receive multiplexed data transmitted via the Internet in units of packets.

  FIG. 17 is a diagram showing the overall configuration of a system for storing and distributing multiplexed data obtained by multiplexing content data and balloon data in a package medium such as a DVD. As shown in FIG. 17, the package media converting apparatus 2b stores the multiplexed data in a recording medium such as a DVD to form a package media. The package media reaches the viewers via the distribution system 3b. The package media playback device 4b reads multiplexed data stored in the package media and plays back video content with balloon subtitles.

  The video content creation device, the transmission device, the playback device, the providing system, and the data structure and recording medium used in them according to the present invention are images in which the relationship between a speaker and subtitles is easily understood and the entire screen is easy to see. Content can be provided, which is useful in the field of content creation.

The block diagram which shows the whole structure of the broadcast system for broadcasting the video content with a subtitle using the speech balloon concerning one Embodiment of this invention. The block diagram which shows the functional structure of the content creation apparatus 1 The figure which shows the example of a data structure of subtitle list data The figure which shows the example of a data structure of speech balloon data The block diagram which shows the functional structure of the content transmitter 2 The block diagram which shows the functional structure of the content reproduction apparatus 4. The block diagram which shows the functional structure of the content display apparatus 5 Flow chart showing the operation of the content creation device 1 The figure which shows an example of the display content in the content creation apparatus 1 The figure which shows an example of the display content in the content creation apparatus 1 The figure which shows an example of the display content in the content creation apparatus 1 The figure which shows an example of the display content in the content creation apparatus 1 The figure which shows an example of the balloon data finally created Flow chart showing operation of content transmission device 2 A flowchart showing the operation of the content reproduction apparatus 4 The figure which shows an example of the image based on the video signal produced by the content reproduction apparatus 4 The figure which shows an example of the content of the image based on the balloon signal produced by the content reproduction apparatus 4 The figure which shows an example of the image based on the caption character signal produced by the content reproduction apparatus 4 The figure which shows an example of the image based on the caption character signal produced by the content reproduction apparatus 4 The figure which shows operation | movement of the synthetic | combination transfer part 43 of the content reproduction apparatus 4. The figure which shows the example of a display in the content display apparatus 5 The figure which shows the example of a display in the content display apparatus 5 The figure which shows the whole structure of the system for providing content data and speech balloon data via the internet The figure which shows the whole structure of the system for storing and distribute | circulating the multiplexed data with which content data and balloon data were multiplexed in package media, such as DVD

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Content production apparatus 2, 2a Content transmission apparatus 2b Package media conversion apparatus 3 Broadcast apparatus 3a Internet 3b Distribution system 4, 4a Content reproduction apparatus 4b Package media reproduction apparatus 5 Content display apparatus 11 Data creation control part 12 Input part 13 Display output part 14, 44 Time count unit 15 Storage unit 21 Multiplexing control unit 22, 42 Operation unit 23 Error correction code addition unit 24 Digital modulation unit 25 Transmission unit 41 Playback control unit 43 Composite transfer unit 45 Balloon shape storage unit 46 Reception unit 47 Demodulation Unit 48 error correction unit 51 display output device unit 52 drive circuit unit

Claims (34)

A content creation device for creating data necessary to provide video content with subtitles by a speech balloon,
In the video based on the video content data as the original data, a balloon time extracting means for extracting a time for displaying the balloon,
In the video at the time extracted by the balloon time extracting means, a balloon area determining means for determining a balloon area suitable for displaying a balloon,
A balloon image determining unit for determining a balloon image to be combined with the balloon region determined by the balloon region determining unit;
Subtitle character determining means for determining a subtitle character to be combined with the balloon image determined by the balloon image determining means;
Balloon data creation means for creating balloon data by converting at least one of the information about the time to display the balloon, the information about the balloon area, the information about the balloon image, and the information about the subtitle character into data. And
The content creation device, wherein the speech balloon data created by the speech balloon data creation means is reproduced together with the video content data to provide video content with captions by speech balloons. The balloon region determining means detects a change in color tone in the image based on the video content data, extracts a portion having a flat color tone, and sets a frame included in the flat portion as a balloon region,
The content creation apparatus according to claim 1, wherein the balloon image determination unit sets an image having a size that displays the subtitle characters in the frame as a balloon image.   The content creation apparatus according to claim 2, wherein the balloon area determination unit determines the balloon area by changing the extracted frame based on an instruction from a user.   The content creation apparatus according to claim 2, wherein the balloon image determination unit changes the shape of the balloon image based on an instruction from a user.   The content creation device according to claim 2, wherein the subtitle character determination unit changes the subtitle character based on an instruction from a user.   The subtitle character determining means determines whether or not the number of subtitle characters per unit time in the time when the speech balloon is to be displayed is equal to or greater than a predetermined number. The content creation device according to claim 5, wherein the content creation device notifies the user to do so.   The content creation device according to claim 2, wherein the subtitle character determination unit changes an attribute of the subtitle character based on an instruction from a user.   The content creation apparatus according to claim 1, further comprising a multiplexing unit that multiplexes the video content data and the balloon data created by the balloon data creation unit.   The content creation device according to claim 8, further comprising multiplexed data transmission means for transmitting data multiplexed by the multiplexing means via a network.   The content creation device according to claim 8, further comprising package media storage means for storing data multiplexed by the multiplexing means in a package medium. Furthermore, a volume determination means for determining the volume of the sound during reproduction of the video content data is provided,
The content creation device according to claim 1, wherein the subtitle character determination unit changes the attribute of the subtitle character according to the volume determined by the volume determination unit. Furthermore, a face size extracting means for extracting the size of a person's face in the video based on the video content data is provided,
The content creation apparatus according to claim 1, wherein the balloon image determination unit determines a start point of the balloon image according to a face size extracted by the face size extraction unit. The video content data is encoded by the MPEG (Moving Picture Experts Group) system,
The content creation apparatus according to claim 1, wherein the balloon data is described in XML (extensible Markup Language). A content transmission device for transmitting data necessary for providing video content with subtitles by a speech balloon,
Information about the time to display a speech balloon in the video based on the video content data as the original data, information about the area where the speech balloon is displayed in the video, information about the shape of the speech balloon in the area, and subtitle characters to be inserted into the speech balloon Balloon information acquisition means for acquiring balloon data in which at least one piece of information is converted into data,
Video content data acquisition means for acquiring the video content data;
A multiplexing unit that multiplexes the balloon data acquired by the balloon data acquisition unit and the video content data acquired by the video content data acquisition unit;
A content transmission apparatus comprising: transmission means for transmitting data multiplexed by the multiplexing means.   15. The content transmitting apparatus according to claim 14, wherein the transmitting unit transmits multiplexed data to a broadcasting apparatus for wireless broadcasting.   15. The content transmission apparatus according to claim 14, wherein the transmission means transmits multiplexed data to a content reproduction apparatus for reproducing the video content data and balloon data via a network. . A content package mediating device for packaging data necessary for providing video content with subtitles by a speech balloon,
Information about the time to display a speech balloon in the video based on the video content data as the original data, information about the area where the speech balloon is displayed in the video, information about the shape of the speech balloon in the area, and subtitle characters to be inserted into the speech balloon Balloon information acquisition means for acquiring balloon data in which at least one piece of information is converted into data,
Video content data acquisition means for acquiring the video content data;
A multiplexing unit that multiplexes the balloon data acquired by the balloon data acquisition unit and the video content data acquired by the video content data acquisition unit;
A content package media conversion apparatus comprising: storage means for storing data multiplexed by the multiplexing means in a package medium. A content playback device for playing back video content with subtitles in a balloon,
Information about the time to display a speech balloon in the video based on the video content data as the original data, information about the area where the speech balloon is displayed in the video, information about the shape of the speech balloon in the area, and subtitle characters to be inserted into the speech balloon Balloon information acquisition means for acquiring balloon data in which at least one piece of information is converted into data,
Video content data acquisition means for acquiring the video content data;
A balloon signal generating means for generating a signal related to the image of the balloon based on the balloon data;
Subtitle character signal generating means for generating a signal related to subtitle characters based on the balloon data;
Video signal generating means for generating a video signal based on the video content data;
Combining and transferring means for synthesizing the balloon signal generated by the balloon signal generating means, the caption character signal generated by the caption character signal generating means, and the video signal generated by the video signal generating means, and transferring the combined signal to the display device A content playback device comprising: And further comprising a synthesis presence / absence command means for commanding the synthesis transfer means whether to synthesize the speech signal and the subtitle character signal with the video signal,
When the synthesis transfer means receives an instruction to synthesize the speech signal and the subtitle character signal with the video signal from the synthesis presence / absence command means, the synthesis transfer means transfers the synthesis signal to the display device and does not synthesize. 19. The content reproduction apparatus according to claim 18, wherein when receiving a command, only the video signal is transferred to the display device. further,
A volume measuring means for measuring the volume of surrounding sounds;
Volume threshold judgment means for judging whether or not the volume of sound measured by the volume measurement means exceeds a threshold;
20. The content reproduction apparatus according to claim 19, wherein the synthesis presence / absence commanding unit commands the synthesis transfer unit to determine whether or not to perform synthesis in accordance with a determination result of the volume threshold value determination unit.   The synthesis presence / absence command means, when the volume threshold value judgment means judges that the volume of surrounding sounds does not exceed the threshold value, gives a command to synthesize to the synthesis transfer means, and further outputs the voice 21. The content reproduction apparatus according to claim 20, wherein the audio output apparatus does not output audio.   21. The composition presence / absence command means, when the volume threshold value judgment means judges that the volume of surrounding sounds exceeds the threshold value, gives a command to synthesize to the composition transfer means. The content reproduction device described. Furthermore, it has a moving speed measuring means for measuring the moving speed of its own device,
The compositing presence / absence command means determines whether or not the moving speed measured by the moving speed measuring means exceeds a predetermined threshold value, and if so, gives a command for compositing to the combining and transferring means. The content playback device according to claim 19.   The composition presence / absence command means instructs the composition transfer means to synthesize the speech signal and the subtitle character signal with the video signal in accordance with an instruction from a user. The content reproduction apparatus described in 1. The subtitle character signal generating means generates a normal subtitle character signal for displaying a subtitle character on either the top, bottom, left or right of the screen based on the balloon data in accordance with an instruction from the user,
When the subtitle character signal generating unit generates the normal subtitle character signal, the synthesizing and transferring unit synthesizes only the normal subtitle character signal and the video signal and transfers the composite signal to the display device. The content reproduction apparatus according to claim 18.   The content reproduction apparatus according to claim 18, wherein the synthesis transfer unit synthesizes the speech balloon signal, the caption character signal, and the video signal for each frame.   The content reproduction apparatus according to claim 18, further comprising display means for displaying the synthesized video based on the synthesized signal transferred from the synthesized transfer means. A computer-readable recording medium on which data having a structure for displaying video content with subtitles by a speech balloon is recorded on a computer device,
A structure for storing information about a time when a speech balloon should be displayed in a video based on video content data as original data;
Corresponding to the information about the time, a structure for storing information about a region for displaying a balloon in the video,
Corresponding to the information about time, a structure for storing information about the shape of the balloon in the region;
A recording medium on which data having a structure for storing information on subtitle characters to be inserted into a balloon is recorded in correspondence with the information on time. The structure for storing information about the time is:
A structure for storing information indicating the start time of subtitles;
29. The recording medium according to claim 28, comprising a structure for storing information indicating a duration of subtitles. A data structure for displaying video content with subtitles by a balloon on a computer device,
A structure for storing information about a time when a speech balloon should be displayed in a video based on video content data as original data;
Corresponding to the information about the time, a structure for storing information about a region for displaying a balloon in the video,
Corresponding to the information about time, a structure for storing information about the shape of the balloon in the region;
A data structure having a structure for storing information on subtitle characters to be inserted into a balloon in correspondence with the information on time. A content providing system for providing video content with subtitles by a speech balloon,
Information about the time to display a speech balloon in the video based on the video content data as the original data, information about the area where the speech balloon is displayed in the video, information about the shape of the speech balloon in the area, and subtitle characters to be inserted into the speech balloon A content creation device that creates speech balloon data in which at least one piece of information is converted into data,
Content providing means for multiplexing the balloon data created by the content creation device and the video content data, and providing the multiplexed data as video content;
A content providing system comprising: a content reproducing device that reproduces video content with subtitles by a balloon based on multiplexed data provided from the content providing means.   32. The content providing system according to claim 31, wherein the content providing means transmits the multiplexed data to the content reproduction device by wireless broadcasting.   32. The content providing system according to claim 31, wherein the content providing means transmits the multiplexed data to the content reproduction device by network distribution.   32. The content providing system according to claim 31, wherein the content providing means provides the multiplexed data to the content reproduction device via a package medium.

Images