JP2009088990A - Reception apparatus, television broadcast playback method, and television broadcast playback program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To display captions in synchronization with the video and audio even in a case where information for displaying captions is not contained in television broadcast. <P>SOLUTION: A television receiver includes: a playback unit 41 which receives a broadcast wave of television broadcast containing video signals, audio signals and caption information and delays by a predetermined time and plays back the video signals and the audio signals contained in the broadcast wave of the television broadcast; a character recognizing unit 23 which extracts character information from the video signals; an audio recognizing unit 25 which extracts character information from the audio signals; and a determination unit 27 which determines timing to display the caption information according to a correlation of the caption information and the extracted character information. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a receiving apparatus, a television broadcast reproduction method, and a television broadcast reproduction program, and more particularly to a receiving apparatus that receives a television broadcast including subtitle information, a television broadcast reproduction method and a television for reproducing the television broadcast. It is related with John broadcast reproduction program.

  In recent years, live broadcast programs such as news and sports programs in television broadcasts may be given subtitle information that expresses sound in characters. Since the caption information is created by recognizing the sound of the broadcast program on the broadcast station side, the caption information is added to the broadcast signal with a delay of several seconds to several tens of seconds with respect to the broadcast video.

  Conventionally, in order to synchronize subtitles and video, the broadcast station adds a delay time to the video of the subtitle character string to the broadcast signal, and delays the video by a predetermined time on the broadcast signal reception side. And a subtitle character string are displayed at a timing determined based on the delay time of the subtitle character string with respect to the video (for example, Patent Documents 1 to 3).

In these conventional techniques, there is a problem in that the subtitle and video cannot be synchronized on the receiving side unless a broadcast signal including a time difference between the video and subtitle is transmitted on the broadcasting station side.
JP 2004-207721 A JP 2006-211636 A Japanese Patent Laid-Open No. 2006-324779

  The present invention has been made to solve the above-described problems, and one of the objects of the present invention is to provide video and audio even when information for displaying subtitles is not included in a television broadcast. It is to provide a receiving apparatus capable of displaying subtitles synchronously.

  Another object of the present invention is a television broadcast reproduction method capable of synchronizing and displaying subtitles in video and audio even when the information for displaying subtitles is not included in the television broadcast, and It is to provide a television broadcast reproduction program.

  In order to achieve the above-described object, according to one aspect of the present invention, a receiving apparatus is a receiving apparatus that receives a broadcast wave of a television broadcast including a video signal, an audio signal, and caption information. A reproduction means for reproducing a video signal and an audio signal included in a broadcast wave of John Broadcast with a predetermined delay, a character information extraction means for extracting character information from at least one of the video signal and the audio signal, subtitle information, Determining means for determining the timing for displaying the subtitle information based on the correlation with the extracted character information.

  According to this aspect, character information is extracted from at least one of the video signal and the audio signal, and the timing for displaying the caption information is determined based on the correlation between the caption information and the character information. For this reason, it is possible to synchronize the caption information with the video and audio of the television broadcast reproduced with a predetermined time delay. As a result, it is possible to provide a receiving apparatus that can display subtitles in synchronism with video and audio even when the information for displaying the subtitles is not included in the television broadcast.

  Preferably, the character information extracting means includes voice recognition means for outputting a character string recognized by voice recognition of the voice signal as character information.

  Preferably, the determining means further includes a display means for displaying the character string when the timing for displaying the caption information cannot be determined based on the correlation between the caption information and the character string output by the voice recognition means.

  According to this aspect, it is possible to display a character string that is included in the audio signal but not included in the caption information.

  Preferably, the character information extraction means further includes character recognition means for outputting a character string recognized by character recognition of the video signal as character information, and the determination means is a character output by the caption information and voice recognition means. If the timing for displaying the caption information cannot be determined based on the correlation with the column, the timing for displaying the caption information is determined based on the correlation between the caption information and the character string output by the character recognition means.

  According to this aspect, the character string included in the subtitle information can be displayed in synchronization with the video.

  Preferably, the character information extraction means includes character recognition means for outputting a character string recognized by character recognition of the video signal as character information.

  According to this aspect, it is possible to display a character string that is included in the video signal but not included in the caption information.

  Preferably, the character information extracting means includes vowel recognition means for outputting an array of vowels as character information by recognizing the movement and shape of the lips included in the video signal.

  According to another aspect of the present invention, a television broadcast reproduction method includes a step of receiving a broadcast wave of a television broadcast including a video signal, an audio signal, and caption information, and a broadcast wave of the television broadcast. Based on a correlation between the step of reproducing the video signal and the audio signal delayed by a predetermined time, the step of extracting the character information from at least one of the video signal and the audio signal, and the subtitle information and the extracted character information Determining the timing for displaying the caption information.

  According to this aspect, there is provided a television broadcast reproduction method capable of displaying subtitles in synchronization with video and audio even when the information for displaying the subtitles is not included in the television broadcast. be able to.

  According to still another aspect of the present invention, a television broadcast reproduction program receives a broadcast wave of a television broadcast including a video signal, an audio signal, and caption information, and a broadcast wave of the television broadcast. Based on the correlation between the step of reproducing the included video signal and the audio signal with a predetermined time delay, the step of extracting the character information from at least one of the video signal and the audio signal, and the subtitle information and the extracted character information And determining the timing for displaying the subtitle information.

  According to this aspect, there is provided a television broadcast reproduction program capable of displaying subtitles in synchronization with video and audio even when the information for displaying the subtitles is not included in the television broadcast. be able to.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  FIG. 1 is a functional block diagram showing an outline of functions of a television receiver in one embodiment of the present invention. Referring to FIG. 1, a television receiver 1 as a receiving device receives a broadcast wave of a television broadcast and outputs a broadcast signal, a control unit 20 for processing the broadcast signal, subtitles, and the like. Includes a first buffer memory 31 that temporarily stores video signals, a second buffer memory 33 that temporarily stores video signals and audio signals, and a reproduction unit 41 that reproduces broadcast signals.

  The tuner 10 selects / receives a broadcast wave and outputs a broadcast signal including a video signal, an audio signal, and a caption signal to the control unit 20. The tuner 10 may be an analog broadcast tuner or a digital broadcast tuner. As the tuner 10, a tuner 10 suitable for the type of broadcast wave is used.

  The control unit 20 receives a video signal, an audio signal, and a caption signal, a character recognition unit 23 to which the video signal is input, an audio recognition unit 25 to which the audio signal is input, and a caption signal. , A determination unit 27 that determines the timing for displaying the subtitles, and a reproduction control unit 29 for controlling the reproduction unit 41.

  The broadcast signal includes an audio signal, a video signal, and a caption signal. The separation unit 21 separates the broadcast signal into an audio signal, a video signal, and a caption signal, outputs the video signal to the character recognition unit 23, outputs the audio signal to the voice recognition unit 25, and determines the caption signal 27. And the audio signal and the video signal are stored in the second buffer memory 33. The caption signal includes a character string.

  The separating unit 21 records the audio signal and the video signal in the second buffer memory 33. However, the separating unit 21 may store the broadcast signal received by the tuner in the second buffer memory 33 as it is. In the case of broadcasting, the partial TS may be stored.

  The character recognition unit 23 performs character recognition on the video included in the video signal input from the separation unit 21. Although all the frames included in the video signal may be recognized as characters, all the frames may not be recognized as characters. For example, a frame extracted at a predetermined time interval or a frame with a drastic change in video may be extracted so that characters are recognized for each image of one frame. In character recognition from an image (video), a character region is determined from the image, and a character is specified by pattern matching the determined character region. For character recognition, a technique widely known in OCR or the like may be used. The character recognition unit 23 numbers the extracted character strings in the order in which the character strings are extracted from the video signal. Here, the i-th character string extracted by the character recognition unit 23 is referred to as a video character string VS (i) (i is a positive integer).

  When the character recognition unit 23 recognizes the frame and extracts the video character string VS (i), the video character string VS (i) and the frame from which the video character string VS (i) is extracted are reproduced. The set with the video time VT (i) is output to the determination unit 27. Here, for the sake of explanation, time will be described as relative time. The relative time indicates the time as the elapsed time since the time when the broadcast signal reproduction is started is set to zero. The video time VT (i) includes at least a time (start time) at which the first frame including the video character string VS (i) is reproduced.

  The voice recognition unit 25 recognizes voice included in the voice signal input from the separation unit 21. The voice recognition unit 25 recognizes voice in a period sandwiched between silence periods and extracts a character string. As a speech recognition method, for example, a method using an acoustic (phoneme) model having acoustic features and a language model having linguistic features is used. Hidden Markov Models (HMMs) are widely used as acoustic models, and HTK is famous as a tool for creating HMMs. Julius is known as an open source large vocabulary continuous speech recognition engine using HTK. The speech recognition method is not limited to this, and a conventionally known method may be used. The voice recognition unit 25 assigns numbers to the extracted character strings in the order in which the character strings are extracted from the voice signal. Here, the j-th character string extracted by the speech recognition unit 25 is referred to as a speech character string AS (j) (j is a positive integer).

  When the voice recognition unit 25 extracts the voice character string AS (j), the voice character string AS (j) and the voice time AT (j) in which the voice from which the voice character string AS (j) is extracted is reproduced. Are output to the determination unit 27. The voice time AT (j) includes a start time at which the voice from which the voice character string AS (j) is extracted is played back and an end time at which the playback ends.

  The determination unit 27 receives a set of the video character string VS (i) and the video time VT (i) from the character recognition unit 23, and the voice character string AS (j) and the voice time AT (j) from the voice recognition unit 25. And a subtitle signal is input from the separation unit 21. The determination unit 27 assigns numbers to the character strings included in the caption signal in the order in which they are input. Here, the kth input character string is referred to as a caption character string SS (k) (k is a positive integer). Further, the time when the subtitle character string SS (k) is input is temporarily set as the subtitle time ST (k).

  The determination unit 27 compares the subtitle character string SS (k) with at least one audio character string AS (j) or video character string VS (i) in the order of input, and reproduces the subtitle character string SS (k). The caption time ST (k) is determined. The determination unit 27 stores a set of the subtitle character string SS (k) and the subtitle time ST (k) determined for the subtitle character string SS (k) in the first buffer memory 31.

  More specifically, the determination unit 27 compares the subtitle character string SS (k) with at least one speech character string AS (j) in the order in which the subtitle character string SS (k) is input, and determines one having a predetermined correlation. To do. For example, if an audio character string AS (J) having a predetermined correlation with the subtitle character string SS (K) is determined in the audio character string AS (j), the subtitle time for reproducing the subtitle character string SS (K) is determined. ST (K) is determined to be the same value as the voice time AT (J). Thereby, since the subtitle character string SS (K) can be displayed in the audio time AT (J), the subtitle character string SS (K) can be synchronized with the audio.

  Further, the determination unit 27 determines the subtitle character string SS (K) if the audio character string AS (J) having a predetermined correlation with the subtitle character string SS (K) is not determined in the audio character string AS (j). Compared with at least one video character string VS (i), one having a predetermined correlation is determined. For example, if the video character string VS (I) having a predetermined correlation with the subtitle character string SS (K) is determined in the video character string VS (i), the subtitle time for reproducing the subtitle character string SS (K) is determined. ST (K) is determined to be the same value as the video time VT (I). Thereby, since the subtitle character string SS (K) can be displayed at the video time VT (I), the subtitle character string SS (K) can be synchronized with the video.

  If the video character string VS (I) having a predetermined correlation with the subtitle character string SS (K) is not determined in the video character string VS (i), the determination unit 27 reproduces the subtitle character string SS (K). Subtitle time ST (K-1) and subtitle time for reproducing subtitle character string SS (K-1) and subtitle character string SS (K + 1) before and after subtitle character string SS (K), respectively. Set during ST (K + 1). Thereby, even when the timing for displaying the subtitle character string SS (K) is not determined from the audio signal or the video signal, the subtitle character string SS (K) can be displayed at an appropriate timing.

  Further, the determination unit 27 converts the speech character string AS (J), which has no predetermined correlation with any of the subtitle character strings SS (k) among the sound character strings AS (j), into a new subtitle character string SS. Set. The new subtitle character string SS is inserted between two subtitle character strings SS having a subtitle time ST before and after the audio time AT (J) of the audio character string AS (J). The subtitle time ST of the new subtitle character string SS is set to the same value as the audio time AT (J). Thereby, although it is not contained in subtitle information, the character string contained in an audio | voice signal can be displayed synchronizing with an audio | voice.

  In addition, the determination unit 27 converts the video character string VS (I) that has no predetermined correlation with any of the subtitle character strings SS (k) in the video character string VS (i) to a new subtitle character string SS. Set. The new subtitle character string SS is inserted between the two subtitle character strings SS having the subtitle time ST before and after the video time VT (I) of the video character string VS (I). The subtitle time ST of the new subtitle character string SS is set to the same value as the video time VT (I). Thereby, although not included in the caption information, the character string included in the video signal can be displayed in synchronization with the video.

  The playback control unit 29 controls the playback unit 41. The reproduction unit 41 reads the subtitle character string SS (k) and the subtitle time ST (k) from the first buffer memory 31 according to the instruction input from the reproduction control unit 29, and the video signal and audio from the second buffer memory 33. Read the signal. The reproduction unit 41 reproduces the video signal and the audio signal from the second buffer memory 33 after a predetermined time has elapsed since they were stored. As a result, the video signal and the audio signal are reproduced with a predetermined time delay after the broadcast signal is received.

  The reproduction unit 41 displays the subtitle character string SS (k) read from the first buffer memory 31 when the time for reproducing the video signal and the audio signal is the subtitle time ST (k). Thereby, a subtitle character string can be displayed in synchronization with video display and audio output. The subtitle character string SS (k) is displayed superimposed on the video signal. An image of the subtitle character string SS (k) may be synthesized with the video signal, or may be displayed using an on-screen function of a display such as OSD.

  The first buffer memory 31 and the second buffer memory 33 are DRAM (Dynamic RAM) or SDRAM (Symchronous Dynamic RAM). Further, it may be a semiconductor memory such as a flash memory.

  The control unit 20 loads and executes a program stored in the flash memory 51. Note that the program executed by the control unit 20 is not limited to being recorded in the flash memory 51, but may be recorded in another recording medium.

  The program here includes not only a program directly executable by the control unit 20 but also a source program, a compressed program, an encrypted program, and the like.

  In this example, the character recognition unit 23 extracts the video character string VS (i) from the video signal. However, instead of the character recognition unit 23, the movement and shape of the lips included in the video signal are recognized. Accordingly, a vowel recognition unit that outputs the vowel array as character information may be provided. The arrangement of vowels output by the vowel recognition unit can be processed in the same manner as the video character string VS (i) output by the character recognition unit 23. In addition, the vowel recognition unit and the character recognition unit 23 may be provided so that either one of the character strings output from both may be the video character string VS (i). Both may be used.

  FIG. 2 is a functional block diagram illustrating detailed functions of the determination unit. Referring to FIG. 2, determination unit 27 includes a correlation calculation unit 61 and a selection unit 63. The correlation calculation unit 61 receives a set of the video character string VS (i) and the subtitle character string SS (k) or a set of the audio character string AS (j) and the subtitle character string SS (k) from the selection unit 63. Is done. The correlation calculation unit 61 obtains the correlation between the subtitle character string SS (k) and the video character string VS (i), and outputs the correlation value XC (k, i) to the selection unit. Further, the correlation between the subtitle character string SS (k) and the voice character string AS (j) is obtained, and the correlation value XC (k, j) is output to the selection unit.

There are various methods for calculating the correlation value, and any of them may be used. Here, an example in which sequences of speech phonemes are compared is shown. Note that the correlation value calculation method is not limited to the method of comparing sequences for each phoneme of speech. FIG. 3 is a diagram for explaining comparison between two phoneme sequences. Here, the phoneme sequence of the speech character string AS (j) is a _i (konbanha), and the phoneme sequence of the subtitle character string SS (k) is b _i (konbanha). In this case, the cross-correlation function is given by However, a _i b _{i + τ} is “1” when a _i and b _{i + τ} match, and is “0” when they are different. _Τ that maximizes C _τ indicates the strongest correlation.

図３に示す音素系列ａ_iおよび音素系列ｂ_ｉの場合には、τ＝０から順番に相関値Ｃ_τを計算し、Ｃ_τがしきい値Ｔを超えれば、その時のτが最も相関の強いことを示す。図３では、τ＝０でしきい値Ｔを超えるので、τ＝１以上の相関値Ｃ_τを計算する必要はない。したがって、音素系列ａ_ｉおよび音素系列ｂ_ｉの時間のずれＴ５−Ｔ０が音声文字列ＡＳ(ｊ)と字幕文字列ＳＳ（ｋ）のずれとなる。

In the case of the phoneme sequence a _i and phoneme sequence b _i shown in FIG. 3, the correlation value C _τ is calculated in order from τ = 0, and if C _τ exceeds the threshold value T, τ at that time is the most correlated. Indicates strong. In FIG. 3, since the threshold value T is exceeded when τ = 0, there is no need to calculate a correlation value C _τ of τ = 1 or more. Therefore, the time difference T5-T0 between the phoneme sequence a _i and the phoneme sequence b _i is the difference between the speech character string AS (j) and the subtitle character string SS (k).

  The correlation calculation unit 61 calculates the correlation value XC = Cτ sequentially from τ = 0, and outputs the correlation value XC to the selection unit 63 when XC exceeds the threshold value T. If there is no XC exceeding the threshold value, nothing is output to the selector 63. Alternatively, a value indicating that there is no XC exceeding the threshold value may be output. Here, 0 is output as a value indicating that there is no XC exceeding the threshold. If a strong correlation value XC cannot be obtained, the phoneme sequence of the subtitle character string SS (k) may be composed of only the vowel part from which the consonant part is omitted to obtain the correlation. The series used for obtaining the correlation value XC may use syllables, word units, etc. in addition to phonemes. Further, the correlation value XC is calculated by shifting either one of the voice character string AS (j) or the subtitle character string SS (k) and using the correlation value XC exceeding the threshold value, If it is clear that the character string AS (j) and the subtitle character string SS (k) are not shifted, the correlation value XC obtained without shifting either the voice character string AS (j) or the subtitle character string SS (k). May be used. Further, when obtaining the correlation, it is not necessary to compare all of the voice character string AS (j) and the subtitle character string SS (k), and the comparison is limited to a predetermined number of characters such as the first 10 characters. May be. Thereby, the time for calculating the correlation value XC can be shortened.

  The combination for which the correlation calculation unit 61 calculates the correlation value is determined by the selection unit 63. Returning to FIG. 2, the selection unit 63 receives the video character string VS (i) and the video time VT (i) from the character recognition unit 23, and the voice character string AS (j) and the voice time AT from the voice recognition unit 25. (J) is input, and the subtitle character string SS (k) and subtitle time ST (k) included in the subtitle signal input from the separation unit 21 are input.

  The selection unit 63 sets the caption character string SS (k) as a processing target in the order in which it is input. Here, a case where the subtitle character string SS (K) is a processing target will be described as an example. First, the selection unit 63 sets the speech character string AS (J) having the speech time AT (J) as the comparison target before the caption time ST (K) of the caption character string SS (K) and the closest time. To do. Then, the correlation calculation unit 61 calculates the correlation between the subtitle character string SS (K) and the voice character string AS (J). If the correlation value XC calculated by the correlation calculation unit 61 exceeds the threshold value T, the correlation calculation unit 61 outputs the correlation value XC, the selection unit 63 selects the speech character string AS (J), The corresponding speech character string AS (J) corresponding to the subtitle character string SS (K) is determined. Then, the subtitle time ST (K) for reproducing the subtitle character string SS (K) is changed to the same value as the audio time AT (J) of the audio character string AS (J) determined as the corresponding audio character string.

  On the other hand, if the correlation value XC calculated by the correlation calculation unit 61 does not exceed the threshold value T, the correlation calculation unit 61 outputs 0 as a value indicating that there is no correlation value XC exceeding the threshold value T, The selection unit 63 determines the speech character string AS (J-1) immediately before the speech character string AS (J) as a processing target, and causes the correlation calculation unit 61 to obtain a correlation value. If the speech character string AS (Jm) (m is a positive integer) to be processed has already been determined as the corresponding speech character string, the speech character correlated with the subtitle character string SS (K) It is determined that the column AS (j) does not exist.

  Next, the selection unit 63 first compares the video character string VS (I) with the video time VT (I) as the closest time before the subtitle time ST (K) of the subtitle character string SS (K). Set the target. Then, the correlation calculation unit 61 calculates the correlation between the subtitle character string SS (K) and the video character string VS (I). If the correlation value XC calculated by the correlation calculation unit 61 exceeds the threshold value T, the correlation calculation unit 61 outputs the correlation value XC, and the selection unit 63 converts the video character string VS (I) into the subtitle character string. The corresponding video character string corresponding to SS (K) is determined. Then, the subtitle time ST (K) for reproducing the subtitle character string SS (K) is changed to the same value as the video time VT (I) of the video character string VS (I) determined as the corresponding video character string.

  When the caption time ST (K) includes a start time and an end time, and the video time VT (I) includes only the start time, the start time of the caption time ST (K) is set as the start of the video time VT (I). The time is set, and the end time may be set to a time after a predetermined time has elapsed from the start time. Further, the start time of the next caption time ST (K + 1) may be set to the end time of the caption time ST (K).

  On the other hand, if the correlation value XC calculated by the correlation calculation unit 61 does not exceed the threshold value T, the correlation calculation unit 61 outputs 0 as a value indicating that there is no correlation value XC exceeding the threshold value T, The selection unit 63 determines the video character string VS (I-1) immediately before the video character string VS (I) as a processing target, and causes the correlation calculation unit 61 to obtain a correlation value. If the video character string VS (In) (n is a positive integer) to be processed has already been determined as the corresponding video character string, the video character correlated with the subtitle character string SS (K) It is determined that the column VS (i) does not exist.

  If the video character string VS (I) having a predetermined correlation with the subtitle character string SS (K) is not determined in the video character string VS (i), the selection unit 63 reproduces the subtitle character string SS (K). Subtitle time ST (K-1) and subtitle time ST (K-1) for reproducing subtitle character string SS (K-1) and subtitle character string SS (K + 1) before and after subtitle character string SS (K), respectively. Set to ST (K + 1).

  FIG. 4 is a flowchart illustrating an example of the flow of synchronization processing. The synchronization process is a process executed by the control unit 20 when the control unit 20 executes a program recorded in the flash memory 53. 7 to 11 are time charts showing temporal arrangements of the audio character string AS, the video character string VS, and the subtitle character string SS. T0 is the time when video or audio playback starts. For example, referring to FIG. 7, the start time of speech time AT (1) corresponding to speech character string AS (1) is time T0, and the end time is time T1. The start time of the video time VT (1) corresponding to the video character string VS (1) is time T0, and the end time is time T1. The start time of the caption time ST (1) corresponding to the caption character string SS (1) is time T1, and the end time is time T2.

  Hereinafter, the flow of the synchronization process shown in FIG. 4 will be described with reference to FIGS. Referring to FIG. 4, control unit 20 sets variables k, j, i, amin, and vmin to initial values “1”, respectively (step S01). The variable k is used to specify the caption character string SS (k) and the caption time ST (k) to be processed. The variable j is used to specify the speech character string AS (j) and the speech time AT (j) to be processed. The variable i is used to specify the video character string VS (i) and the video time VT (i) to be processed. The variable amin is used to specify the last speech character string AS to be compared with the subtitle character string SS. The variable vmin is used to specify the last video character string VS to be compared with the subtitle character string SS. By executing step S01, “1” is set in the variable k, so the first subtitle character string SS (1) and subtitle time ST (1) are set as processing targets.

  In the next step S02, the audio time AT (J) that is temporally earlier than the caption time ST (k) and is closest to the caption time ST (k) is selected from the audio time AT (j). . The voice character string AS (J) is the first voice character string to be compared with the subtitle character string SS (k) among the plurality of voice character strings. Since it goes back to the past in order from the closest to ST (k), it becomes the first comparison target. For example, referring to FIG. 7, the voice character string AS (1) is the closest to the time T1 before the start time T1 of the caption time ST (1), so the voice time AT (1) is selected. Next, “J” selected in step S02 is set in variable j (step S03), and the process proceeds to step S04.

  In step S04, a correlation value XC between the subtitle character string SS (k) and the voice character string AS (j) is calculated. Then, the correlation value XC is compared with the threshold value T (step S05). If correlation value XC exceeds threshold value T, the process proceeds to step S06; otherwise, the process proceeds to step S11.

  In step S06, the audio time AT (j) is set as the caption time ST (k). If the correlation value XC between the subtitle character string SS (k) and the audio character string AS (j) exceeds the threshold value T, the subtitle character string SS (k) is similar to the audio character string AS (j). For this reason, the subtitle character string SS (k) is synchronized by being displayed at the audio time AT (j) when the audio character string AS (j) is reproduced. Here, FIG. 8 shows a time chart when the correlation value XC between the caption character string SS (1) and the voice character string AS (1) shown in FIG. At this time, the audio time AT (1) is set to the caption time ST (1).

If _τ of correlation value C _τ is not “0” when obtaining correlation value XC, subtitle time ST (k) is set to a value obtained by increasing or decreasing audio time AT (j) by τ.

  In the next step S06A, an unassigned character string insertion process for a speech character string is executed. FIG. 5 is a flowchart showing an example of the flow of unassigned speech character string insertion processing. Referring to FIG. 5, in step S31, the number M = (j−1) −amin + 1 of the speech character strings AS to which the subtitle character string SS is not assigned before the speech character string AS (j) is calculated. In step S32, it is determined whether or not there is a voice character string AS to which no subtitle character string SS is assigned before the voice character string AS (j) by checking whether M is positive. . If M is positive, it is determined that there is a voice character string AS to which no subtitle character string SS is assigned before the voice character string AS (j), and the process proceeds to step S33. It is determined that there is no voice character string AS, and the process returns to the synchronous process.

  In steps S33 to S40, an audio character string AS to which no subtitle character string SS is assigned is inserted between the subtitle character string SS (k) and the immediately preceding subtitle character string SS (k-1). When SS is an array, the subtitle character string SS after SS (k) needs to be moved down by the number M of unassigned speech character strings, so that the subtitle character string after SS (k) in steps S33 to S36. All SSs are copied, and in subsequent steps S37 to S40, a speech character string AS to which no subtitle character string SS is assigned is inserted. If the SS is configured as a bi-directional linked list, only the process of rewriting and inserting the pointer to the next list of SS (k−1) and the pointer to the list before SS (k) is sufficient. .

  In the next step S41, M is added to N and k, respectively. Step S41 and step S09 in FIG. 4 are not necessary in the case of a bidirectional linked list, and it is necessary to change the conditional expression such as step S10. A detailed example in the case of a bi-directional linked list is obvious to those skilled in the art and will not be repeated here.

  Returning to FIG. 4, in step S07, the variable amin is set to a value obtained by adding 1 to the variable j. In step S06, since the audio time AT (j) is assigned to the caption time ST (k), the last audio character string AS (amin) to be compared is set to the audio time AT next to the audio time AT (j). This is because the voice character string AS (j + 1) of (j + 1) is set. Thereby, the number of voice character strings to be compared with one subtitle character string SS (k) can be limited, and the processing speed can be increased.

  On the other hand, when the process proceeds to step S11, the correlation value XC between the subtitle character string SS (k) and the voice character string AS (j) does not exceed the threshold value T. In this case, the variable j is decremented by 1 in order to change the subject of comparison with the subtitle character string SS (k) to the voice character string AS (j−1) immediately before the voice character string AS (j). (Step S11). Then, it is determined whether or not the variable j is smaller than the variable amin in order to determine whether or not it is before the last phonetic character string AS (amin) to be compared (step S12). If variable j is smaller than variable amin, the process proceeds to step S13; otherwise, the process returns to step S04.

  When the process proceeds to step S13, it is determined that the subtitle character string SS (k) to be processed and the voice character string AS (j) to be compared do not exist. In this case, the subtitle character string SS (k) is compared with the video character string VS (i). Steps S13 to S18 are the same as steps S02 to S07 except that the comparison targets are different, and steps S19 and S20 are the same as steps S11 and S12. Therefore, the description will not be repeated here. FIG. 6 shows the unassigned video character string insertion process executed in step S17A. The unassigned video character string insertion process shown in FIG. 6 is different from the unassigned audio character string insertion process shown in FIG. 5 only in the character string to be inserted, and therefore description thereof will not be repeated here.

In step S20, when i <vmin, it is determined that the subtitle character string SS (k) to be processed and the video character string VS (i) to be compared do not exist. In this case, the caption time ST (k) of the caption character string SS (k) to be processed is determined by one of the following methods (step S21).
(1) The audio time AT of the audio character string AS or the video time VT of the video character string VS giving the maximum value among all the correlation values XC calculated for the caption character string SS (k) to be processed. , Subtitle time ST (k) is set.
(2) Subtitle time ST (1) to subtitle time ST (k-1) of subtitle character string SS (1) to subtitle character string SS (k-1) before the subtitle character string SS (k) to be processed The subtitle time ST (k) is increased / decreased by the average of the time (shift time) at which is changed.
(3) Increase or decrease the caption time ST (k) by a predetermined time.

  When the subtitle time ST (k) of the subtitle character string SS (k) is determined in step S21, in the next step S22, the audio character string AS of the audio time AT after the determined subtitle time ST (k) is determined. The variables amin and vmin are updated so that the video character string VS of the video time VT after the determined subtitle time ST (k) is to be compared. Thereafter, the process proceeds to step S08.

  In the time chart shown in FIG. 7, when the correlation value XC between the subtitle character string SS (2) and the audio character string AS (2) exceeds the threshold value T, the audio time AT ( 2) is set. A time chart at this time is shown in FIG. The subtitle character string SS (2) is displayed and synchronized with the time when the audio character string AS (2) is reproduced.

  Furthermore, for the subtitle character string SS (3) from the state shown in FIG. 9, the closest sound character string AS (4) in time before the subtitle time ST (3) is the object of comparison. If the correlation value XC does not exceed the threshold value T, the phonetic character string AS (3) is to be compared. When the correlation value XC between the subtitle character string SS (3) and the audio character string AS (3) exceeds the threshold value T, the audio time AT (3) is set as the subtitle time ST (3). The time chart auto at this time is shown in FIG. The subtitle character string SS (3) is displayed and synchronized with the time when the audio character string AS (3) is reproduced. In this way, the subtitle character string SS (3) is compared not only with the audio character string AS (4) but also with the previous audio character string AS (3) that goes back further. ) Can be displayed at an appropriate position, and can be accurately synchronized with the sound.

  Returning to FIG. 4, in step S08, the subtitle character string SS (k) is divided as necessary. When the length of the voice character string AS (j) is different from the length of the subtitle character string SS (k), the longer character string of the voice character string AS (j) and the subtitle character string SS (k) is divided. To do. When the phonetic character string AS (j) is divided, the second half phonetic character information becomes the j + 1th phonetic character string AS (j + 1). The voice time AT (j + 1) in the second half may be the same as the voice time AT (j) in the first half, but if the time is known for each phoneme or character, the divided voice character string AS (j + 1) The time of the first phoneme or character may be AT (j + 1). Further, in this division processing, when the character string is an array, it is necessary to copy all the subsequent character strings. However, if it is configured as a bidirectional linked list, it is only necessary to rewrite the front and rear pointers.

  When the subtitle character string SS (k) is divided, the number of subtitle character strings SS increases, so that the constant N indicating the total number of subtitle character strings SS is changed to a value obtained by adding 1 (N = N + 1). The subtitle time ST (k + 1) in the latter half is the same as the next subtitle time ST (k + 2) or is delayed for a certain time when the time is unknown for each phoneme or character. For example, in FIG. 8, the correlation value XC between the audio character string AS (2) and the subtitle character string SS (2) is higher than the threshold value T, and the audio character string AS (2) is the subtitle character string SS. When the length is shorter than (2), the subtitle character string SS (2) is divided as shown in FIG. In this case, the subtitle character string SS (3) in FIG. 11 corresponds to the latter half of the subtitle character string SS (2) in FIG. 8, and the subtitle character string SS (4) in FIG. It corresponds to 3).

  In the next step S09, in order to change the caption character string SS (k) to be processed to the next caption character string SS (k + 1), 1 is added to the variable k, and the process proceeds to step S10. In step S10, it is confirmed whether the variable k does not exceed the constant N. The constant N indicates the total number of subtitle character strings SS. If the variable k does not exceed the constant N, the process returns to step S02, and the same processing as described above is repeated with the next caption character string SS (k) as the processing target. If the variable k exceeds the constant N, it is determined that there is no more subtitle character string SS and the process is terminated.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

<Appendix>
(1) The determination unit includes a corresponding character information selection unit that selects one corresponding character information corresponding to the caption information from at least one character information output by the character information extraction unit,
2. The time when the subtitle information is displayed is determined when the reproduction unit reproduces the portion of the audio signal or the video signal from which the corresponding character information has been extracted by the character information extraction unit. Receiver.
(2) The determination unit includes a corresponding character string selection unit that selects a corresponding character string whose correlation with the caption information is a predetermined value or more from at least one of the character strings output by the voice recognition unit,
The receiving apparatus according to claim 2, wherein when the portion of the audio signal from which the corresponding character string has been extracted by the character information extracting unit is reproduced by the reproducing unit, the timing for displaying the caption information is determined.
(3) The determination unit includes a corresponding character string selection unit that selects a corresponding character string whose correlation with the caption information is a predetermined value or more from at least one of the character strings output by the character recognition unit,
6. The reception according to claim 4 or 5, wherein when the portion of the video signal from which the corresponding character string is extracted by the character information extraction unit is reproduced by the reproduction unit, the timing for displaying the caption information is determined. apparatus.
(4) The determination unit includes a corresponding vowel array selection unit that selects a corresponding vowel array whose correlation with the caption information is a predetermined value or more from at least one array of the vowels output by the vowel recognition unit. ,
The receiving apparatus according to claim 6, wherein when the portion of the video signal from which the corresponding vowel array is extracted by the character information extracting unit is reproduced by the reproducing unit, the timing for displaying the caption information is determined.

It is a functional block diagram which shows the outline | summary of the function of the television receiver in one of embodiment of this invention. It is a functional block diagram which shows the detailed function of a determination part. It is a figure for demonstrating the comparison of two phoneme series. It is a flowchart which shows an example of the flow of a synchronous process. It is a flowchart which shows an example of the flow of an unassigned speech character string insertion process. It is a flowchart which shows an example of the flow of an unassigned video character string insertion process. It is a 1st time chart which shows temporal arrangement | positioning with the audio | voice character string AS, the video character string VS, and the caption character string SS. It is a 2nd time chart which shows temporal arrangement | positioning with the audio | voice character string AS, the video character string VS, and the caption character string SS. It is a 3rd time chart which shows temporal arrangement | positioning with the audio | voice character string AS, the video character string VS, and the caption character string SS. It is a 4th time chart which shows temporal arrangement | positioning with the audio | voice character string AS, the video character string VS, and the caption character string SS. It is a 5th time chart which shows temporal arrangement | positioning with the audio | voice character string AS, the video character string VS, and the caption character string SS.

Explanation of symbols

  1 TV receiver, 10 tuner, 20 control unit, 21 separation unit, 23 character recognition unit, 25 speech recognition unit, 27 determination unit, 29 playback control unit, 31 first buffer memory, 33 second buffer memory, 41 playback Part, 53 cards, 61 correlation calculation part, 63 selection part.

Claims (8)

A receiving device for receiving a broadcast wave of a television broadcast including a video signal, an audio signal, and caption information,
Reproduction means for reproducing a video signal and an audio signal included in a broadcast wave of a television broadcast with a predetermined time delay;
Character information extraction means for extracting character information from at least one of the video signal and the audio signal;
And a determination unit configured to determine a timing for displaying the caption information based on a correlation between the caption information and the extracted character information.   The receiving apparatus according to claim 1, wherein the character information extraction unit includes a voice recognition unit that outputs a character string recognized by voice recognition of the voice signal as the character information.   The determination means further includes a display means for displaying the character string when the timing for displaying the caption information cannot be determined based on the correlation between the caption information and the character string output by the voice recognition means. The receiving device according to claim 2 provided. The character information extraction means further includes character recognition means for outputting a character string recognized by character recognition of the video signal as the character information,
If the determination means cannot determine the timing for displaying the subtitle information based on the correlation between the subtitle information and the character string output by the voice recognition means, the subtitle information and the character recognition means output The receiving device according to claim 2, wherein timing for displaying the caption information is determined based on a correlation with the character string to be performed.   The receiving apparatus according to claim 1, wherein the character information extraction unit includes a character recognition unit that outputs a character string recognized by character recognition of the video signal as the character information.   The receiving apparatus according to claim 1, wherein the character information extracting unit includes a vowel recognition unit that outputs an array of vowels as the character information by recognizing a lip movement and shape included in the video signal. Receiving a broadcast wave of a television broadcast including a video signal, an audio signal, and caption information;
Reproducing a video signal and an audio signal included in a broadcast wave of the television broadcast with a predetermined delay;
Extracting character information from at least one of the video signal and the audio signal;
Determining a timing for displaying the subtitle information based on a correlation between the subtitle information and the extracted character information. Receiving a broadcast wave of a television broadcast including a video signal, an audio signal, and caption information;
Reproducing a video signal and an audio signal included in a broadcast wave of the television broadcast with a predetermined delay;
Extracting character information from at least one of the video signal and the audio signal;
The television broadcast reproduction program which makes a computer perform the step which determines the timing which displays the said caption information based on the correlation with the said caption information and the extracted said character information.

Images