JP2021090172A - Caption data generation device, content distribution system, video reproduction device, program, and caption data generation method - Google Patents

Abstract

To eliminate a delay difference of video/voice and a caption.SOLUTION: A caption data generation device comprises: a first acquisition part which acquires a caption text obtained by converting a voice of a broadcast program into a text, while listening to the voice of the broadcast program by an operator; a second acquisition part which acquires a voice text obtained by recognizing the voice of the broadcast program and converting a result into a text and a voice time of the voice of the broadcast program; a comparison part which compares the caption text with the voice text and detects a coincident word or paragraph; and a caption data generation part which generates caption data obtained by associating the caption text with the voice time of the voice text including a word or paragraph coincident with the word or paragraph of the caption text.SELECTED DRAWING: Figure 2

Description

The present invention relates to a subtitle data generation device, a content distribution system, a video playback device, a program, and a subtitle data generation method, and in particular, subtitle data generation that eliminates a difference in output timing between video and audio (actually spoken timing) and subtitles. The present invention relates to an apparatus, a content distribution system, a video reproduction apparatus, a program, and a subtitle data generation method.

Traditionally, TV programs with subtitles have been broadcast in TV program broadcasting. Such subtitled broadcasting is particularly useful for hearing-impaired people to enjoy television programs. In recent years, subtitle broadcasting in live broadcasting and the like has also been attempted. In order to deal with subtitled broadcasting in live broadcasting and the like, there are roughly two methods.

One is called a real-time subtitle method, which is a method in which an operator listens to the voice of a broadcast program, inputs characters from a keyboard, and converts the text into text (for example, Patent Document 1).

Another method is a voice recognition method that converts the voice of a broadcast program into text by using voice recognition technology.

Japanese Unexamined Patent Publication No. 2001-188649

However, in the real-time subtitle method, since the operator listens to the voice of the broadcast program and inputs characters from the keyboard, the text is accurate, but there is a problem that it is delayed by about 5 to 20 seconds from the timing of actually speaking. There is. In order to solve this, a method of delaying the video / audio by a certain fixed time and synchronizing it with the subtitles can be considered, but the delay time is not a fixed time but changes, so it is not realistic.

On the other hand, since the voice recognition method is performed automatically without human intervention, the time for converting to text is short and the text is acquired in almost real time, but there is a problem that the text is not accurate compared to the real-time subtitle method. is there.

Therefore, the present invention has been invented in view of the above problems, and it is possible to display a subtitle in which the content of the audio is accurately converted into text in synchronization with the video and audio that are the source of the subtitle. It is an object of the present invention to provide a subtitle data generation device, a content distribution system, a video playback device, a program, and a subtitle data generation method.

One aspect of the present invention is a first acquisition unit that acquires a subtitle text in which the audio of the broadcast program is converted into text while the operator listens to the audio of the broadcast program, and an audio recognition of the audio of the broadcast program and converts it into text. A second acquisition unit that acquires the audio text and the audio time of the audio of the broadcast program, a comparison unit that compares the subtitle text with the audio text, and detects a matching word or phrase. It is a subtitle data generation device including a subtitle data generation unit that generates subtitle data in which the subtitle text is associated with the audio time of the audio text including the word or phrase matching the word or phrase of the subtitle text.

One aspect of the present invention includes a separation unit that separates broadcast program data into video data, audio data, and subtitle text in which the audio of the broadcast program is converted into text while the operator listens to the audio of the broadcast program. A voice recognition unit that recognizes voice data and converts it into text to generate voice text, a first acquisition unit that acquires the subtitle text, and a second acquisition unit that acquires the voice text and the voice time of the voice data. An acquisition unit, a comparison unit that compares the subtitle text with the audio text to detect a matching word or phrase, and an audio including the subtitle text and a word or phrase that matches the word or phrase of the subtitle text. A distribution content that includes at least the video data, the audio data, and the subtitle data by delaying the video data and the audio data for a predetermined time with the subtitle data generation unit that generates the subtitle data associated with the audio time of the text. Distribution content that generates data This is a content distribution system that includes a data generation unit.

One aspect of the present invention is a storage unit that stores a broadcast program, a separation unit that reads the broadcast program from the storage unit and separates the video file, an audio file, and subtitle data, and a subtitle text from the subtitle data. A first acquisition unit for acquiring the data, a second acquisition unit for acquiring the audio text obtained by recognizing the audio of the audio file and converting it into text, and the audio time of the audio of the broadcast program, and the subtitle text. The comparison unit that detects a matching word or phrase by comparing the voice text with the voice text, the subtitle text, and the voice time of the voice text including the word or phrase matching the word or phrase of the subtitle text are associated with each other. It is a video reproduction device including a subtitle data generation unit that generates the subtitle data, and a display control unit that superimposes the subtitle text of the subtitle data on the video of the video file corresponding to the audio time of the subtitle data.

One aspect of the present invention includes a first acquisition process of acquiring a subtitle text in which the audio of the broadcast program is converted into text while the operator listens to the audio of the broadcast program, and voice recognition of the audio of the broadcast program and conversion into text. A second acquisition process for acquiring the voice text and the voice time of the voice of the broadcast program, a comparison process for comparing the subtitle text with the voice text, and a comparison process for detecting a matching word or phrase. It is a program for causing a computer to execute a subtitle data generation process for generating subtitle data in which the subtitle text and the audio time of the audio text including the word or phrase matching the word or phrase of the subtitle text are associated with each other.

In one aspect of the present invention, while the operator listens to the audio of the broadcast program, the subtitle text obtained by converting the audio of the broadcast program into text is acquired, and the audio of the broadcast program is recognized and converted into text. The audio time of the audio of the broadcast program is acquired, the subtitle text is compared with the audio text, a matching word or phrase is detected, and the subtitle text matches the word or phrase of the subtitle text. It is a subtitle data generation method that generates subtitle data associated with the audio time of an audio text including a word or a phrase.

The present invention eliminates the difference in output timing between the video and audio (actually spoken timing) and the subtitle, and creates a subtitle in which the content of the audio is accurately converted into text with the video and audio that is the source of the subtitle. It can be displayed in synchronization.

FIG. 1 is a diagram showing an overall configuration example of the content distribution system according to the present embodiment. FIG. 2 is a block diagram showing a functional configuration example of the content distribution device 1. FIG. 3 is a diagram for explaining the operation of the content distribution device 1. FIG. 4 shows an example of subtitles displayed on the viewing terminal 3 after the display timing is corrected in the above-described embodiment. FIG. 5 shows a display example of connected subtitles. FIG. 6 is a block diagram of the content distribution device 1 configured by the computer system. FIG. 7 is a block diagram of the recorder according to the second embodiment.

Embodiments of the present invention will be described.

<First Embodiment>
FIG. 1 is a diagram showing an overall configuration example of the content distribution system according to the present embodiment. As shown in FIG. 1, the content distribution system includes a content distribution device 1, a voice recognition device 2, a viewer's viewing terminal 3, and a communication line 4.

The content distribution device 1 is a server system configured to include a single server device, a plurality of server devices, a storage device, and the like. The content distribution device 1 receives a broadcast program and distributes a moving image (broadcast program) with subtitles to the viewing terminal 3. In this embodiment, a simultaneous distribution system that is distributed at the same time as broadcasting such as terrestrial digital broadcasting is assumed.

The voice recognition device 2 is a Web API server, and uses existing voice recognition technology to convert received voice data into text. The existing voice recognition technology used by the voice recognition device 2 has a correct answer rate of about 80 to 90%, and although it lacks accuracy, it is possible to convert voice data into text in almost real time. The textualized voice data is hereinafter referred to as voice text. In the present embodiment, the content distribution device 1 is assumed to be a server connected to the communication line 4, but the content distribution device 1 may include a voice recognition device 2.

The viewing terminal 3 is a terminal that supports a method such as HTTP Live Streaming (HLS) and can watch a moving image with subtitles, and is connected to a communication line 4 via a wireless communication base station or the like to be connected to the content distribution device 1. Data communication can be performed. The viewing terminal 3 is, for example, a smartphone, a mobile phone, a portable game device, a stationary home-use game device, a business-use game device, a personal computer, a tablet computer, a controller of a stationary home-use game device, or the like. Basically, there are a plurality of viewing terminals 3, and each viewer operates them.

The communication line 4 means a communication path capable of data communication. That is, the communication line 4 includes a telephone communication network, a cable network, a communication network such as the Internet, as well as a LAN by a dedicated line (dedicated cable) for direct connection and Ethernet (registered trademark), and the communication method is wired. / Regardless of wireless. Then, the content distribution device 1 and the voice recognition device 2 can be connected to the communication line 4 and can communicate with each other, and the content distribution device 1 and the viewing terminal 3 can be connected to the communication line 4 and can communicate with each other. Communication is possible.

Next, the configuration of the content distribution device 1 will be described. FIG. 2 is a block diagram showing a functional configuration example of the content distribution device 1.

The content distribution device 1 includes a separation unit 11, a real-time caption data acquisition unit 12, a voice text acquisition unit 13, a text comparison unit 14, a subtitle data generation unit 15, and a distribution content data generation unit 16.

The separation unit 11 receives the content file of the terrestrial digital broadcasting. The content file can include a video file (hereinafter, may be referred to as “original video file”) and an audio file in which subtitles are not superimposed on the video, and ancillary data (Ancillary).

The video file and the audio file also include information on the PTS (presentation time stamp), which is the display timing.

Ancillary data includes subtitle data. The subtitle data includes text (character string) superimposed on the video, display timing of each character string (PTS: display start timing and end timing), display position of each character string (position in the video), and the like. The subtitle data may indicate the character string, display timing, display position, and the like for each of a plurality of video formats in which at least one of the resolution and the aspect ratio is different from each other. Examples of the video format include HD (high definition video), SD (standard definition television), One Seg (mobile phone), and the like.

In the present embodiment, it is assumed that the subtitle data is generated by the real-time subtitle method generated by the operator listening to the audio of the broadcast program. Therefore, it should be noted that the display timing of the subtitle text superimposed on the video is delayed compared to the original video / audio.

The separation unit 11 receives the content file and separates the video file, the audio file, and the ancillary data. The separation unit 11 outputs the audio file as audio data (including PTS acquired from video or audio) to the audio recognition device 2. Further, the separation unit 11 outputs the ancillary data to the real-time subtitle data acquisition unit 12. Further, the separation unit 11 outputs the video file and the audio file to the distribution content data generation unit 16.

The real-time subtitle data acquisition unit 12 separates the subtitle data from the ancillary data, acquires the subtitle data (subtitle text and PTS), and outputs the subtitle text and PTS to the text comparison unit 14.

The voice text acquisition unit 13 acquires the voice text from the voice recognition device 2, associates the acquired voice text with the PTS of the voice text, and outputs the voice text to the text comparison unit 14. If the voice PTS cannot be acquired from the video or voice data, the acquisition time at which the voice text is acquired from the voice recognition device 2 can be used instead of the voice text PTS. This is because the voice recognition device 2 can convert the voice data into text in almost real time, so that the acquisition time of the voice text is almost the same as the PTS of the actual voice.

The text comparison unit 14 compares the subtitle text acquired by the real-time subtitle data acquisition unit 12 with the voice text acquired by the voice text acquisition unit 13, and detects a word or phrase that matches the subtitle text and the voice text. As a method of detection, morphological analysis is performed on the subtitle text and the voice text, the text is divided into words or phrases, and then the matching text is detected for each word or phrase. It should be noted that detection may be performed between all subtitle texts and voice texts, but when detecting voice texts including words or phrases in a certain subtitle text, voices having a PTS of 20 seconds before and after the PTS of the certain subtitle texts. It may be limited to text. By doing so, the detection range can be narrowed and the detection process can be speeded up.

Further, the text comparison unit 14 determines the correspondence between the subtitle text and the voice text. Regarding the correspondence between the subtitle text and the audio text, if the words or phrases of the audio text and the words or phrases of the subtitle text match more than a predetermined number, it is determined that the audio text and the subtitle text correspond to each other. .. The predetermined number is about 70 to 80% of the number of all words or phrases in the speech text, considering the current accuracy of speech recognition. However, this is just an example, and it may be changed in a timely manner according to the accuracy of the voice recognition used.

The correspondence between the subtitle text and the voice text is as follows: when one voice text corresponds to one subtitle text, when multiple voice texts correspond to one subtitle text, to multiple subtitle texts. On the other hand, it is conceivable that one voice text corresponds to it.

Based on the comparison result (correspondence between the subtitle text and the audio text) of the text comparison unit 14, the subtitle data generation unit 15 includes the subtitle text and the audio text including the word or phrase matching the word or phrase of the subtitle text. Generates new subtitle data by associating with PTS.

Regarding the association between the subtitle text and the PTS of the audio text, the following cases are assumed depending on the correspondence between the subtitle text and the audio text.

(1) When one voice text corresponds to one subtitle text In principle, the subtitle text is associated with the PTS of the voice text corresponding to this subtitle text.

(2) When multiple voice texts correspond to one subtitle text When one subtitle text contains words or phrases of multiple matching voice texts, the PTS time of the corresponding multiple voice texts Of these, the earliest time is selected as the start time, the latest time is selected as the end time, these times are adopted as PTS, and this PTS is associated with one subtitle text.

(3) When one voice text corresponds to a plurality of subtitle texts When one voice text corresponds to a plurality of subtitle texts, the number of the corresponding multiple subtitle texts is used for one voice text. Divide the PTS time. For example, if one voice text corresponds to two subtitle texts and the PTS of one voice text is 15: 05: 10-15: 05: 15, 15: 05: 10-15: 05: Divide into 12 and 15:05:13-15:05:15, associate 15:05:10-15:05:12 with the subtitle text with the early PTS time, and 15 with the subtitle text with the late PTS time. Associate: 05: 13-15: 05: 15. Note that this is an example, and for example, the PTS may be divided according to the number of characters in each subtitle text.

However, it is also possible that there is no voice text corresponding to the first or last word or phrase of the subtitle text, such as when there is an erroneous recognition in voice recognition in all of (1), (2), and (3). For example, if there is no matching word in the first word of the subtitle text, it is determined how many words precede the matching word in the subtitle text. Then, the voice text is traced back by the number of words that match the subtitle text, and the PTS of the voice text including the word is adopted. By doing so, it is possible to prevent deviation from the video and audio that are the source of the subtitle text.

Then, the subtitle data generation unit 15 has a table of subtitle data in which the PTS and the subtitle text are associated as described above.

It takes some processing time to create a table of subtitle data, but simultaneous distribution is generally sent with a delay of about 30 seconds from on-air such as terrestrial digital broadcasting, and during that 30 seconds, By creating a subtitle table synchronized with video / audio, it is possible to transmit video / audio / subtitles in synchronization during simultaneous distribution.

The distribution content data generation unit 16 receives the video file and the audio file from the separation unit 11 and converts the video file and the audio file into a method such as HTTP Live Streaming (HLS). Further, the distribution content data generation unit 16 reads the subtitle data from the subtitle data generation unit 15 and converts it into a subtitle format such as Web Video Text Tracks (WebVTT).

The subtitle data generation device 5 is composed of a real-time subtitle data acquisition unit 12, a voice text acquisition unit 13, a text comparison unit 14, and a subtitle data generation unit 15.

Subsequently, the operation of the content distribution device 1 will be described. FIG. 3 is a diagram for explaining the operation of the content distribution device 1.

The content distribution device 1 receives a content file of terrestrial digital broadcasting. The content file can include a video file and an audio file (including PTS) in which subtitles are not superimposed on the video, and ancillary data. Ancillary data includes subtitle data. The subtitle data includes at least a character string superimposed on the video and a display timing (PTS: display start timing) of each character string. However, the display position (position in the image) of each character string may be included.

The separation unit 11 separates the subtitle data from the ancillary data and outputs the subtitle data to the real-time subtitle data acquisition unit 12. Further, the separation unit 11 outputs the voice file as voice data (including PTS) to the voice recognition device 2.

The real-time subtitle data acquisition unit 12 separates the subtitle data from the ancillary data, acquires the subtitle data (subtitle text and PTS), and outputs the subtitle text and PTS to the text comparison unit 14. In the example of Fig. 3, the subtitle text and PTS are "15:05:20-15:05:22 Actor XX and actress □□ (8 seconds delay with the video)" and "15:05. : 27-15: 05: 29 I reported my marriage today. (11 seconds behind the video) "," 15:05:38-15:05:40 Many actors and other friends were blessed. (18 seconds behind the video) ”and“ 15: 05: 47-15: 05: 49, the topic changed and the weather forecast. (15 seconds behind the video) ”. It should be noted that the part in parentheses indicates the delay time with the actual video for the purpose of explanation described later, and is not included in the actual data.

On the other hand, the voice text acquisition unit 13 acquires the voice text from the voice recognition device 2, associates the acquired voice text with the time stamp of the voice, and outputs the voice text to the text comparison unit 14. In the example of Fig. 3, the voice text and time stamp are "15: 05: 12-15: 05: 14 with actor XX Mountain" and "15: 05: 14-15: 05: 16 actress □□". "Yamaga", "15:05:16-15:05:18 I reported blood stains today.", "15:05:20-15:05:22 Actors, etc.", "15: 05:23-15:05:25 We were staying in large numbers. "And" At 15:05:32-15:05:34, the topic is the weather forecast in the river. "

The text comparison unit 14 performs morphological analysis on the subtitle text acquired by the real-time subtitle data acquisition unit 12 and the voice text acquired by the voice text acquisition unit 13, and detects words that match on a word-by-word basis.

The results of morphological analysis for the real-time subtitles in the example of FIG. 3 are as follows. Each word is written in parentheses.
・ Subtitle text of 15:05:20-15:05:22: (actor) (of) (〇〇) (san) (and) (actress) (of) (□□) (san) (ga)
・ Subtitled text at 15:05:27-15:05:29: (Today) (Marriage) () (Report) () (Done) (Done) (.)
・ Subtitled text at 15:05:38-15:05:40: (actor) (friend) (etc.) (many) (ni) (blessing) (was) (was) (.)
・ Subtitle text of 15: 05: 47-15: 05: 49: (in) (topic) (ga) (changed) (weather) (forecast) (is) (.)

The results of the morphological analysis of the voice text in the example of FIG. 3 are as follows. Each word is written in parentheses.
・ 15: 05: 12-15: 05: 13 voice text: (actor) (of) (〇〇) (mountain) (and)
・ 15: 05: 14-15: 05: 15 voice text: (actress) (of) (□□) (mountain) (ga)
・ Voice text of 15: 05: 16-15: 05: 18: (Today) (Blood stain) () (Report) () (Done) (Done) (.)
・ Voice text of 15: 05: 20-15: 05: 22: (actor) (friend) (etc.)
・ 15: 05: 23-15: 05: 25 voice text: (many) (ni) (accommodation) (was) (was) (.)
・ Voice text of 15:05:32-15:05:34: (Well) (Topic) (Ga) (River) (De) (Weather) (Forecast) (.) (.)

The text comparison unit 14 detects a match or a mismatch with respect to the voice text having the PTS for 20 seconds before and after the one subtitle text as a reference. In the example of Fig. 3, first, "Subtitle text of 15:05:20-15:05:22: (actor) (of) (〇〇) (san) (and) (actress) (of) (□□) (San) (ga) "and the words in the voice text that match the words are detected. Then, "15:05:12-15:05:13 voice text: (actor) (no) (〇〇) (mountain) (and)" is the subtitle text "(san)" and the voice text "(san)". (Mountain) ”is different, but the other parts are the same. Similarly, "15: 05: 14-15: 05: 15 voice text: (actress) (of) (□□) (mountain) (ga)" is the subtitle text "(san)" and the voice text. It is different from "(mountain)", but the other parts are the same.

Therefore, the voice text "Actor OO Mountain" is a part of the voice "Actor OO and actress □□" by the voice recognition device 2 at 15:05:12-15:05:13. Can be inferred to be a text version. Similarly, the voice text "Actress □□ Yamaga" is one of the voices "Actor XX and Actress □□ ga" at 15:05:12-15:05:13 by the voice recognition device 2. It can be inferred that the part is a text version.

When the same detection process is performed, the following correspondence between the voice text and the subtitle text can be obtained.
・ 15: 05: 12-15: 05: 13 voice text "Actor XX Mountain" and 15: 05: 14-15: 05: 15 voice text "Actress □□ Mountain", 15: 05: 20-15: 05: 22 The subtitle text "Mr. XX of the actor and Mr. □□ of the actress" correspond.
・ 15: 05: 16-15: 05: 18 voice text "I reported blood stains today." And 5:05:27-15:05:29 subtitle text "I reported my marriage today. "." Corresponds.
・ 15:05:20-15:05:22 voice text "Actors, etc." and 15:05:23-15:05:25 voice text "A lot of people were staying.", 15:05 : 38-15: 05: 40 The subtitle text "I was blessed by a lot of actors and friends." Corresponds.
・ 15: 05: 32-15: 05: 34 voice text "The topic is the weather forecast in the river." And the subtitle text "15: 05: 47-15: 05: 49 The topic changes and the weather forecast." Is. "

The text comparison unit 14 outputs such a result to the subtitle data generation unit 15.

The subtitle data generation unit 15 receives the comparison result of the text comparison unit 14, and generates the subtitle data by associating the subtitle text with the PTS of the audio text that matches the subtitle text.

For example, the subtitle data generation unit 15 has the audio text "Actor XX Mountain" at 15:05:12-15:05:13 and the audio text "Actress □" at 15:05:14-15:05:15. From the correspondence between "□ Yamaga" and the subtitle text "Actor XX and actress □□ ga" at 15:05:20-15:05:22, the earliest time of the audio text is "15:05". Create a PTS (15:05:12-15:05:15) with ": 12" as the start time and "15:05:15" as the end time, which is the latest time of the audio text, and this and the subtitle text. Generate subtitle data by associating "actor XX with actress □□". By such an association, the subtitle text that is originally delayed by 8 seconds from the video / audio can be changed to the original video / audio time.

As shown in FIG. 3, such an association is performed for all subtitle data. As a result, the following table of subtitle data is created.
15:05:12-15:05:15: Actor 〇〇 and actress □□
15:05:16-15:05:18: I reported my marriage today.
15:05:20-15:05:25: It was blessed by many actors and others.
15:05:32-15:05:34: Then the topic changes and it is the weather forecast.

As mentioned above, it takes some processing time to create a table of subtitle data, but simultaneous distribution is generally sent with a delay of about 30 seconds from the on-air of terrestrial digital broadcasting, so that time. It is quite possible to create a table of subtitle data inside.

Next, the distribution content data generation unit 16 accumulates the video file and the audio file received from the separation unit 11 for the time required to generate the subtitle data, and converts the video file and the audio file into a method such as HTTP Live Streaming (HLS), for example. To do.

Further, the distribution content data generation unit 15 refers to the subtitle data table and converts the subtitle text into a subtitle format such as Web Video Text Tracks (WebVTT). An example of converting the subtitle data shown in the above-mentioned subtitle data table into the WebVTT subtitle format is shown.
15:05:12-> 15:05:15
Actor 〇〇 and actress □□
15:05:16-> 15:05:18
I reported my marriage today.
15:05:20-> 15:05:25
It was blessed by many actors.
15:05:32-> 15:05:34
Then the topic changes and it is the weather forecast.

The distribution content data generation unit 16 generates content data for distribution and transmits it to the viewing terminal 3.

In the present embodiment, the text content is accurate with respect to the original video or audio, but the output timing is delayed with respect to the output timing of the original video or audio, and the original subtitle data. Although the text content is not very accurate with respect to the video or audio, audio recognition subtitle data having an output timing similar to the output timing of the original video or audio is used. Then, by comparing the subtitle text of the real-time subtitle data with the audio text of the voice recognition subtitle data, the correspondence between the subtitle text and the audio text is investigated, and based on the correspondence, the subtitle text whose content is accurate and the content are accurate. New subtitle data is generated by associating the output timing of the voice recognition subtitle data, which is the same output timing as the output timing of the original video or audio.

With such a configuration, in the present embodiment, subtitles in which the content of the video or audio is accurately converted into text can be displayed in synchronization with the video and audio.

In the present embodiment, an example of using the subtitle text and the PTS which is the display time of the subtitle text from the real-time subtitle data has been described, but the PTS of the subtitle text is not indispensable and the PTS of the subtitle text is not used. However, the present invention is feasible.

<Modification 1 of the first embodiment>
Next, a modification 1 of the present embodiment will be described.

Since real-time subtitles input by the operator are transmitted one after another from words or phrases determined with priority on speed, there are many cases where very short subtitles of about 2 to 10 characters are transmitted.

Further, since the displayed subtitles usually need to be displayed for about 2 seconds, short subtitles are switched and displayed in a short time. FIG. 4 shows an example of subtitles displayed on the viewing terminal 3 after the display timing is corrected in the above-described embodiment. As shown in FIG. 4, short subtitles are switched and displayed in a short time. In this way, the short subtitles being switched and displayed in a short time have a problem that the subtitles are difficult to see. On the other hand, the subtitles of television broadcasting can display 42 characters (21 characters x 2 lines) per page.

Therefore, in the first modification of the embodiment, the subtitle data generation unit 15 will explain an example in which short subtitles (words and phrases) are connected to make the subtitles easier to see.

The subtitle data generation unit 15 of the modification 1 of the embodiment connects short subtitles (words and phrases) up to 42 characters and changes the display timing (PTS) of the subtitles. However, if the subtitle is cut off at a place where the sentence is meaningless, it will be difficult to read. Therefore, the subtitles are concatenated so that the subtitle ends within 42 characters and at a predetermined delimiter. Here, the predetermined delimiter is typically a punctuation mark ("." Or "."), But is not limited to this. For example, a colon (:), a semicolon (;), an exclamation mark (!), A question mark (?), Etc. may be used.

Explaining a specific operation, the subtitle data generation unit 15 concatenates the subtitles so that the subtitle ends within 42 characters and at a predetermined delimiter after the association between the voice text and the time stamp is completed. Here, a predetermined delimiter will be described as a punctuation mark (“.” Or “.”).

The subtitle data generation unit 15 holds the following table at the end of the association between the voice text and the time stamp.
15:05:12-15:05:15: Actor 〇〇 and actress □□
15:05:16-15:05:18: I reported my marriage today.
15:05:20-15:05:25: It was blessed by many actors and others.
15:05:32-15:05:34: Then the topic changes and it is the weather forecast.

The subtitle data generation unit 15 searches for subtitles that can be linked to the first subtitle, "Mr. XX of the actor and Mr. □□ of the actress." The condition is that the subtitle ends with a punctuation mark ("." Or ".") Within 42 characters. Then, the next subtitle "I reported my marriage today." Can be concatenated, but if you concatenate to the next subtitle "I was blessed by a lot of actors and friends.", It exceeded 42 characters. , Cannot be connected.

Therefore, the first subtitle "Actor XX and actress □□" and the next subtitle "I reported my marriage today." Were connected, and one subtitle was "With actor XX." Actress □□ reported her marriage today. ” Then, as the PTS of this subtitle, the PTS of the first subtitle and the PTS of the next subtitle are concatenated. That is, (15:05:12-15:05:18).

Similarly, search for subtitles that can be linked to the subtitle "I was blessed by many actors." The condition is that the subtitle ends with a punctuation mark ("." Or ".") Within 42 characters. Then, the next subtitle "The topic changes and it is the weather forecast." Can be linked. Therefore, the first subtitle "I was blessed by many actors and others." And the next subtitle "The topic changed and the weather forecast." Were connected, and one subtitle was "Blessed by many actors and others." Then the topic changed and it was a weather forecast. " Then, as the PTS of this subtitle, the PTS of the first subtitle and the PTS of the next subtitle are concatenated. That is, (15:05:20-15:05:34).

As a result, the subtitle data generation unit 15 has the following table. Although "/" indicates a line break, if the 21st character and the 22nd character are one word, a line break may be performed at the break of the word before the word.
15:05:12-15:05:18: Actor 〇〇 and actress □□ reported their marriage today.
15:05:20-15:05:34: It was blessed by many actors and others. Then / the topic has changed and it is the weather forecast.

FIG. 5 shows a display example of connected subtitles. In this way, since the subtitles that are organized to some extent are displayed for a long time, the viewer can watch the video while understanding the meaning of the subtitles.

The modification 1 of the first embodiment is particularly effective when one voice text corresponds to a plurality of subtitle texts in the above-mentioned correspondence relationship between the subtitle text and the voice text. This is because by combining a plurality of subtitle texts into one subtitle text, the PTS of the audio text can be used without being divided.

<Modification 2 of the first embodiment>
A modified example 2 of the first embodiment will be described.

Specifically, the content distribution device 1 described above can be realized by a computer system having a processor that performs various arithmetic processes and the like. FIG. 6 is a block diagram of the content distribution device 1 configured by the computer system.

As shown in FIG. 6, the content distribution device 1 includes a processor 101, a memory (ROM or RAM) 102, an input device (keyboard, mouse, touch panel, etc.) 103, a communication device 104, and a storage device (hard disk, semiconductor disk, etc.) 105. It can be configured by the computer 100 having the computer 100. The storage device 105 may be physically provided externally to the computer 100 and may be connected to the computer 100 via a network such as a LAN.

In the content distribution device 1, the program stored in the storage device 105 is loaded into the memory 102 and executed by the processor 101, so that the separation process 111, the real-time subtitle data acquisition process 112, the audio text acquisition process 113, and the like. The text comparison process 114, the subtitle data generation process 115, and the distribution content data generation process 116 are realized.

Here, the separation process 111 corresponds to the separation unit 11, the real-time caption data acquisition process 112 corresponds to the real-time caption data acquisition unit 12, the voice text acquisition process 113 corresponds to the voice text acquisition unit 13, and the text comparison process 114. Corresponds to the text comparison unit 14, the subtitle data generation process 115 corresponds to the subtitle data generation unit 15, and the distribution content data generation process 116 corresponds to the distribution content data generation unit 16.

<Second Embodiment>
The technical idea of generating new subtitle data by associating the text of the real-time subtitle data whose content described in the above-described embodiment is accurate with the display timing of the voice recognition subtitle data whose display timing is not delayed is , It can be applied not only to the distribution side that distributes the content, but also to the recorder that records and watches the TV broadcast. Therefore, in the second embodiment, an example in which the subtitle data generation device 5 in the above-described embodiment is applied to the recorder will be described.

FIG. 7 is a block diagram of the recorder according to the second embodiment.

The recorder 50 includes a storage unit 51 and display control in addition to a separation unit 11, a subtitle data generation device 5 including a real-time subtitle data acquisition unit 12, a voice text acquisition unit 13, a text comparison unit 14, and a subtitle data generation unit 15. A unit 52 is provided.

The storage unit 51 stores, for example, data recorded from terrestrial digital broadcasting.

The separation unit 11 reads the data stored in the storage unit 51, separates the subtitle data from the ancillary data, outputs the subtitle data to the real-time caption data acquisition unit 12, and outputs the audio, as in the above-described embodiment. The file is output to the voice recognition device 2 as voice data (including PTS).

The real-time subtitle data acquisition unit 12 separates the subtitle data from the ancillary data, acquires the subtitle data (subtitle text and PTS), and outputs the subtitle text and PTS to the text comparison unit 14.

The voice text acquisition unit 13 acquires the voice text from the voice recognition device 2, associates the acquired voice text with the time stamp of the voice, and outputs the voice text to the text comparison unit 14.

The text comparison unit 14 performs morphological analysis on the subtitle text acquired by the real-time subtitle data acquisition unit 12 and the voice text acquired by the voice text acquisition unit 13, and detects words that match on a word-by-word basis. Then, the text comparison unit 14 determines the correspondence between the subtitle text and the voice text.

Based on the comparison result of the text comparison unit 14, the subtitle data generation unit 15 associates the subtitle text with the PTS of the audio text including the word or phrase matching the word or phrase of the subtitle text, and creates new subtitle data. Generate.

The display control unit 52 receives the video file and the audio file from the separation unit 11, temporarily stores the video file, reads the subtitle data from the subtitle data table of the subtitle data generation unit 15, and displays the video at the same time as the PTS of the subtitle data. Superimpose subtitle text on. Then, the video and audio on which the subtitle text is superimposed are output to a display device such as a display.

With this configuration, even a recorder that has recorded a television broadcast can display subtitles in which the audio content is accurately converted into text in synchronization with the video and audio without delay. In addition, as in the present embodiment, in the case of recording a broadcast and viewing it later, or in the case of so-called chase playback, the process of generating new subtitle data is performed after on-air such as terrestrial digital broadcasting, so that it is sufficiently permissible. Processing is possible within the time range, and video, audio, and subtitles can be played back in synchronization.

Although the present invention has been described above with reference to preferred embodiments, the present invention is not necessarily limited to the above embodiments, and can be variously modified and implemented within the scope of the technical idea.

1 Content distribution device 2 Voice recognition device 3 Viewing terminal 4 Communication line 5 Subtitle data generation device 11 Separation unit 12 Real-time subtitle data acquisition unit 13 Voice text acquisition unit 14 Text comparison unit 15 Subtitle data generation unit 16 Distribution content data generation unit 50 Recorder 51 Storage unit 52 Display control unit 100 Computer 101 Processor 102 Memory 103 Input device 104 Communication device 105 Storage device

Claims (11)

A first acquisition unit that acquires subtitle text that is a text version of the audio of the broadcast program while the operator listens to the audio of the broadcast program.
A second acquisition unit that acquires the voice text obtained by voice-recognizing the voice of the broadcast program and converting it into a text, and the voice time of the voice of the broadcast program.
A comparison unit that compares the subtitle text with the voice text and detects a matching word or phrase.
A subtitle data generation device including a subtitle data generation unit that generates subtitle data in which the subtitle text is associated with the audio time of the audio text including the word or phrase matching the word or phrase of the subtitle text. The subtitle data generation device according to claim 1, wherein the audio time is a PTS (presentation time stamp) of a broadcast program. The subtitle data generation device according to claim 1, wherein the voice time is an acquisition time when the voice text is acquired. When the subtitle text includes a plurality of words or phrases of the voice text, the subtitle data generation unit associates the earliest and latest voice times of the plurality of voice texts with the subtitle text. The subtitle data generation device according to any one of 1 to 3. The subtitle data generation device according to any one of claims 1 to 4, wherein the subtitle data generation unit connects two or more of the subtitle texts so as to be within a predetermined number of characters. A separation unit that separates the broadcast program data into video data, audio data, and subtitle text in which the audio of the broadcast program is converted into text while the operator listens to the audio of the broadcast program.
A voice recognition unit that recognizes the voice data and converts it into text to generate voice text,
The first acquisition unit for acquiring the subtitle text and
A second acquisition unit that acquires the voice text and the voice time of the voice data,
A comparison unit that compares the subtitle text with the voice text and detects a matching word or phrase.
A subtitle data generation unit that generates subtitle data in which the subtitle text and the audio time of the audio text including the word or phrase matching the word or phrase of the subtitle text are associated with each other.
A content distribution system including a distribution content data generation unit that delays the video data and the audio data for a predetermined time and generates distribution content data including at least the video data, the audio data, and the subtitle data. The content distribution system according to claim 6, wherein the audio time is a PTS (presentation time stamp) of a broadcast program. The content distribution system according to claim 6, wherein the voice time is an acquisition time when the voice text is acquired. A storage unit that stores broadcast programs,
A separation unit that reads the broadcast program from the storage unit and separates it into a video file, an audio file, and subtitle data.
The first acquisition unit that acquires the subtitle text from the subtitle data, and
A second acquisition unit that acquires the voice text obtained by voice-recognizing the voice of the voice file and converting it into a text, and the voice time of the voice of the broadcast program.
A comparison unit that compares the subtitle text with the voice text and detects a matching word or phrase.
A subtitle data generation unit that generates subtitle data in which the subtitle text and the audio time of the audio text including the word or phrase matching the word or phrase of the subtitle text are associated with each other.
A video playback device including a display control unit that superimposes the subtitle text of the subtitle data on the video of the video file corresponding to the audio time of the subtitle data. The first acquisition process of acquiring the subtitle text in which the audio of the broadcast program is converted into text while the operator listens to the audio of the broadcast program,
A second acquisition process for acquiring the voice text obtained by voice-recognizing the voice of the broadcast program and converting it into a text, and the voice time of the voice of the broadcast program.
A comparison process for detecting a matching word or phrase by comparing the subtitle text with the voice text,
A program that causes a computer to execute a subtitle data generation process for generating subtitle data in which the subtitle text is associated with the audio time of the audio text including the word or phrase matching the word or phrase of the subtitle text. While the operator listens to the audio of the broadcast program, the subtitle text obtained by converting the audio of the broadcast program into text is acquired.
The voice text obtained by voice-recognizing the voice of the broadcast program and converting it into a text and the voice time of the voice of the broadcast program are acquired.
The subtitle text is compared with the voice text to detect a matching word or phrase.
A method for generating subtitle data that generates subtitle data in which the subtitle text is associated with the audio time of an audio text including a word or phrase that matches the word or phrase of the subtitle text.

Images