JP4561736B2 - Content playback device - Google Patents

Description

  The present invention relates to a content reproduction apparatus that displays video synchronized with a video signal input from the outside.

  In the performance by the karaoke apparatus, the lyrics telop is displayed in synchronization with the performance of the music and the background video (for example, Patent Document 1). This is because the performance track for performance and the lyrics track for displaying the lyrics telop are previously stored in the song data in parallel.

JP 2000-99044 A

  In recent years, not only karaoke apparatuses but also television broadcasts and the like are often used to display lyrics in accordance with the singer's singing. Also, in television broadcasting, not limited to lyrics, a news program or the like in which a person's remarks are written in text is displayed as a telop and displayed on the screen as a superimpose (telop). It has been widely performed to display.

  All of these are playback and display of a video that has been synthesized in advance with a telop. For example, live performances of live performances (so-called live broadcasts) are performed in real time. The telop was not generated synchronously.

  An object of the present invention is to provide a content reproduction apparatus capable of reproducing a video such as a lyrics telop in synchronization with a video signal input by streaming.

According to the first aspect of the present invention, there is provided a content data storage unit for storing content data in which reference video time-series data and playback video time-series data are recorded in association with each other in time series, and a video signal for inputting a video signal from the outside an input unit, by a call to execute the comparison process for detecting the synchronization position by comparing the image motion and motion video time-series data for the reference image of said video signal, reproduction clock synchronized with the video signal A content reproduction apparatus comprising: a clock generation unit that generates a video reproduction unit that reproduces the reproduction video time-series data based on a reproduction clock generated by the clock generation unit ;
An auxiliary information input unit that inputs auxiliary information that is information indicating the position or speed of the reproduction clock from the outside is further provided, and the clock generation unit compares the reference video time-series data based on the auxiliary information. Alternatively, the speed of the reproduction clock is corrected .

  According to a second aspect of the present invention, in the first aspect of the present invention, the video signal input unit further includes a video synthesis unit that synthesizes and outputs the video signal input by the video signal input unit and the video signal reproduced by the video reproduction unit. Features.

  According to a third aspect of the present invention, in the second aspect of the invention, the video signal further includes a video delay unit that delays the video signal by a time required for generating the reproduction clock by the clock generation unit and then inputs the delayed video signal to the video synthesis unit. It is characterized by.

  According to a fourth aspect of the present invention, in the first to third aspects of the present invention, an audio signal input unit that inputs an audio signal synchronized with the video signal from the outside, and an audio signal input from the audio signal input unit is used as the clock. And an audio signal delay unit that delays and re-outputs the time required for generating the reproduction clock by the generation unit.

  According to a fifth aspect of the present invention, in the first to fourth aspects of the present invention, the content data stored in the content data storage unit is audio time-series data associated with the reference video time-series data in time series. In addition, an audio reproduction unit that reproduces the audio time-series data based on the reproduction clock generated by the clock generation unit is further provided.

  According to a sixth aspect of the present invention, in the first to fifth aspects of the invention, the video time-series data stored in the content data storage unit is sequence data including video data to be displayed and timing data indicating the display timing. The video reproduction unit includes a sequence processing unit for reproducing the sequence data.

  In a seventh aspect of the present invention, in the first to fifth aspects of the invention, the video time-series data stored in the content data storage unit is video data including a moving image, and the video reproduction unit stores the video data A video playback unit for playback is included.

  The invention according to claim 8 is the invention according to any one of claims 1 to 7, wherein the content data storage unit further stores device control time series data for controlling the external device in time series, and the reproduction generated by the clock generation unit An external device control unit that reads out the device control time-series data based on a clock and outputs a control signal is further provided.

[Action]
In the present invention, a reproduction clock synchronized with the video signal is generated based on the video signal and various auxiliary information, and the reproduction video time-series data is reproduced based on the reproduction clock, whereby the video synchronized with the video signal ( And audio). Note that the video signal may be output for playback via this device, or may be played back by another path that bypasses this device.

  In addition, since the process of generating a reproduction clock synchronized accurately is a complicated and heavy process, a certain amount of time is required until a clock synchronized with the signal is generated after the video signal is input. By delaying the input video signal and audio signal by this time and outputting them, it becomes possible to more accurately synchronize the reproduced video and the delayed video signal.

  According to the present invention, playback video time-series data can be played back in synchronization with the input video signal. For example, when karaoke apparatus stores karaoke song data in which a background video of the person himself / herself is combined, a live performance video can be input and a lyrics telop can be reproduced and displayed in synchronization with the video.

Embodiments of the present invention will be described with reference to the drawings.
<< first embodiment >>
FIG. 1 is a block diagram of a content reproduction apparatus according to the first embodiment of the present invention. This content playback device is a device that plays back and outputs video sequence data (Display Sequence Data) 9 in synchronization with a video signal input from a video input unit (Video In) 24. In the following description, a case where lyrics synchronized with video / audio of live performance are displayed as an example using song data and background video used for performance of karaoke music will be described as an example.

The content playback apparatus has a video input unit (Video) to which a video signal is input from the outside.
In) 24, an audio input unit 1 to which an audio signal is input from the outside, an auxiliary information input unit (Aux Info) 2 to which auxiliary information is input, an audio output unit (Audio Out) 11 to which an audio signal is output, and an image An output video output unit (Video Out) 12, reference video data (Video Data for reference) 8 and a storage unit (Song Data) 7 for storing karaoke song data including the video sequence data 9 are input. A signal processing unit 6 for synchronizing the video signal and the video, a video sequencer (Graphic Sequencer) 10 for reproducing the video sequence data 9, and a video reproduced by the video sequencer 10 and the input video signal. Video mixer (SuperIm ose) is equipped with a 26.

  When displaying the lyrics of a live performance using karaoke music data having a background image of the person himself / herself, the video signal input from the video input unit 24 is a live image of the performer of the live performance. Live video is distributed via public broadcast networks, the Internet, and the like. The reference video data 8 is a video image that captures the singing of the original singer of the karaoke song. The video sequence data 9 is a lyrics track for displaying lyrics. The sequence data is composed of event data that is information for executing a tone generation operation and a mute operation, and timing data that indicates the read timing of the event data.

  The signal processing unit 6 includes a synchronization control unit (Time Alignment) 3, a clock generation unit (Clock Generator) 4, a video delay unit 25, and an audio delay unit (Delay) 5. The synchronization control unit 3 compares the video signal input from the video input unit 24 with the video image of the reference video data 8, and compares the position and speed based on the auxiliary information input from the auxiliary information input unit 2. The reproduction position information (timing information) and tempo information synchronized with the input video signal are generated. The clock generation unit 4 generates a reproduction clock based on the reproduction position information (timing information) and tempo information generated by the synchronization control unit 3. The video delay unit 25 delays the video signal input from the video input unit 24 by a predetermined time and inputs the video signal to the video output unit 12. The audio delay unit 5 delays the audio signal 1 input from the audio input unit 1 by a predetermined time and inputs the audio signal 1 to the audio output unit 11.

  The synchronization control unit 3 analyzes the video for each frame of the video signal input from the video input unit 24, analyzes the action of the live performer and the movement of the mouth that are being shot, and is stored in the storage unit 7. The video for each frame of the reference video data 8 is analyzed, and the movement of the original singer and the movement of the mouth are analyzed. What is necessary is just to detect a face and a body by the method described in Unexamined-Japanese-Patent No. 2003-196655, 2000-222576, etc., and to compare the motion of the detected face and body. Then, based on the comparison result, the input video signal and the reference video data are compared for each frame, the corresponding position on the time axis that most closely approximates the movement, mouth movement, etc. is detected, and the song data at the corresponding position is detected. Is time performance information of the music. That is, it detects which position in the music data the performance sound of the live performance is playing.

  By detecting the performance position, that is, synchronous detection, the position of the currently input video signal on the time axis in the video image of the reference video data is determined, and the progress speed, that is, the tempo of the video signal is determined. Then, the position on the time axis, that is, performance position information and timing information (information on when the beat timing is passed) and tempo information are output to the clock generation unit 4. Note that the tempo and beat timing may be determined using, for example, the technique described in Japanese Patent Laid-Open No. 9-16171.

  This clock signal is input to the video sequencer 10. The video sequencer 10 reads the video sequence data 9 which is a lyrics telop in accordance with the clock signal and reproduces the video of the lyrics telop. Since the clock signal is synchronized with the externally input video signal, that is, the video of the live performer as described above, the lyrics telop reproduced by the video sequencer 10 and output from the video output unit 12 is live. It will be synchronized with the performance.

  It should be noted that some processing time is required for the synchronization detection by the synchronization control unit 3 and the clock signal generation by the clock generation unit 4, so that the video signal input from the video input unit 24 and the input from the audio input unit 1 are input. The audio signals thus output are delayed by the processing time by the video delay unit 25 and the audio delay unit 5, respectively, and output. As a result, the lyrics telop reproduced by the video sequencer 10 and the input video signal and audio signal can be more accurately synchronized. Note that the video delay unit 25 and the audio delay unit 1 are not indispensable, and the input signal is output as it is when the synchronization does not require high accuracy or when the processing time required for the clock signal generation process is very small. You may do it.

Here, with reference to FIG. 2, FIG. 3, the display method of the song data of a karaoke song and a lyrics telop is demonstrated.
In FIG. 2, the song data includes a header, a musical sound track for playing a karaoke song, a guide melody track for generating a guide melody, and a lyrics track for displaying a lyrics telop, as shown in FIG. It consists of a mark track or the like on which jump marks indicating the song separation positions are written. On the mark track, for example, jump marks indicating the first, second, chorus, climax, intro, interlude, ending and the like are written.

Each track is described according to the MIDI format. For example, a musical tone track and a guide melody track are composed of event data such as note-on event data and note-off event data and timing data indicating the read timing of each event data, as shown in FIG. The note-on event data includes pitch data, and designates the pitch and volume of a musical sound generated by the note-on. This musical sound continues until the corresponding note-off event data is read out.
The timing data can be composed of duration data indicating the time interval between the event data, absolute time data indicating the absolute time from the start time of the music, and the like.

  The music track and guide melody track event data consists of note event data that indicates the pitch, volume, on / off, etc. of the music as described above. / Muted. The musical sound track is composed of a plurality of tracks (parts) for generating musical sounds of a large number of musical instruments, and the guide melody track is composed of single melody MIDI data for guiding the singing melody.

  The lyrics track is sequence data that implements various data for displaying the karaoke song name and lyrics, and is composed of lyrics display data read based on the timing data as shown in FIG. .

  The lyric display data includes all data related to lyric display for one line, display on timing, display off timing, character string data (including the number of display points, display coordinates, character spacing data, etc.), color 1 data , Color 2 data, lyric color change data, etc.

  With reference to FIG. 5B, the display mode of the lyrics telop by the lyrics display data will be described. In the graph of this figure, the vertical axis represents time and the horizontal axis represents the x-coordinate of the lyrics telop (monitor screen). Prior to ton when the performance of the song reaches the location of the lyrics, the lyrics for one line are displayed in advance in color 1. The display continues in color 1 until the performance of the song progresses to the lyrics. When the performance of the song reaches the location of the lyrics, the display color is changed (wiped) from color 1 to color 2 in order from the left as the song progresses. After the wipe is completed, the telop is erased after being displayed in color 2 for a while (te).

  The color change data of the lyrics is composed of a plurality of plotting data indicating the main points of the wipe (time-x coordinate) curve as shown in FIG. The video sequence unit 10 performs color change (wipe) by interpolating the plotting data with a quadratic curve.

  Here, referring again to FIG. 1, it is difficult for the synchronization control unit 3 to determine the performance position and to accurately determine the synchronization only with the video signal input from the video input unit 24. Therefore, reliable performance position determination and synchronization determination are performed using the auxiliary information input from the auxiliary information input unit 2.

  The auxiliary information is, for example, information input by a staff member at a live venue, information detected from a live video, or the like. In the live venue, a plurality of staff members including a cameraman and a voice staff are engaged for live broadcasting, and one of them inputs auxiliary information for synchronized playback on the karaoke apparatus. The auxiliary information includes start information indicating that the song has started, stop information indicating that the song has been paused, end information indicating that the song has ended, and indicating that the performance position of the song jumps from normal progress. Jump information and the like. In the case of live performance, there is a case where the third chorus is omitted due to time restrictions, or the chorus is repeated in accordance with the audience's reaction, and the performance is different from the recorded normal performance. In such a case, where the attendant at the venue has jumped (or prediction of where to jump) is input as jump information. The jump position may be specified by a jump mark in the music data, but may be specified by a time axis value in the music data. As described above, the jump marks include marks indicating the first, second, rust, climax, intro, interlude, and ending, and are attached to the time axis value in the song data according to each song. The It is preferable that the jump mark is originally added to the song data as shown in FIG. 2, but for the song data without the jump mark, the person in charge at the live venue uses the jump mark as auxiliary information at the start of the song. The track data may be distributed.

  In addition, a beat signal input manually (tapping) at the start of a song or when the tempo changes is also sent as auxiliary information input by a live venue staff. This beat signal is not highly accurate and is not sent continuously from the beginning to the end of the song, so it cannot be used as a tempo clock signal as it is, but the tempo determination and beat timing are not It can be used as information for determining.

  The synchronization control unit 3 refers to the auxiliary information as described above input from the auxiliary information input unit 2 to determine where the current performance position is, reads out the surrounding reference video data, Contrast with the input video signal. In addition, the beat signal and intermittent tempo that are input intermittently are used to grasp the approximate beat timing and tempo. By contrast, synchronization is facilitated.

  In the content playback apparatus of this embodiment, video sequence data 9 such as lyrics telop is stored as playback video time-series data. However, video data (Video Data) is stored in the video sequence data 9 and the video data is stored in the video sequence data 9. You may make it reproduce | regenerate synchronizing with a reproduction | regeneration clock signal. In this case, a video playback unit (Video Player) is provided as a functional unit for playing back the video data instead of the video sequencer 10. The video playback unit preferably uses a variable frame rate playback device (playback software).

<< Second Embodiment >>
FIG. 4 is a block diagram of a content reproduction apparatus according to the second embodiment of the present invention. In FIG. 4, the same components as those of the first embodiment shown in FIG.

  In the content reproduction apparatus of this embodiment, the storage unit 7 stores external sequence data (Aux Sequence Data) 17 for controlling external devices in addition to the video sequence data 9. An external sequencer 18 is provided as a functional unit for sequencing the external sequence data 17. The control data sequenced by the external sequencer 18 is output from the control signal output unit 19 to the external device.

  The control signal output unit 19 is connected to a production device such as an automatic performance piano, lighting equipment, fireworks, and fountains. By controlling such a device in synchronization with a video signal input from the video input unit 24, a live piano is played in synchronization with a live performance, lighting is changed, or fireworks are ignited. Can be added.

<< Third Embodiment >>
FIG. 5 is a block diagram of a content reproduction apparatus according to the third embodiment of the present invention. In FIG. 5, the same components as those of the first embodiment shown in FIG.

  The content reproduction apparatus of this embodiment includes an audio reproduction unit 21 as a functional unit that reproduces the audio data 14. The audio data 14 may be sequence data such as MIDI data, or may be audio waveform data such as PCM or MP3. When the playback audio data 14 is sequence data such as MIDI, the audio playback unit 21 has functions of a sequencer and a sound source. When the reproduction audio data 14 is audio waveform data, the audio reproduction unit 21 includes a decoder.

  The audio signal of the audio data 14 reproduced by the audio reproducing unit 21 is input to the mixer 22. The audio signal input from the audio input unit 1 and delayed by the delay unit 5 is also input to the mixer 22. The mixer 22 mixes the input audio signal and the reproduced audio signal and outputs them from the audio output unit 11.

  Accordingly, it is possible to reproduce a video such as a lyrics telop in synchronization with the input live video, and further reproduce and output a sound signal in synchronization with the input live sound.

<< Reading >>
In the second embodiment, the example in which the external device control function by the external sequence data 17 and the external sequencer 18 is added to the first embodiment. However, the external device control function may be provided in the third embodiment. Any form is within the technical scope of the present invention.

  In the above embodiment, the live performance relay has been mainly described as an example. However, the present invention is not limited to the live performance, and any method can be used as long as the video is reproduced in synchronization with the video signal. It can be applied to such situations. For example, the present invention can be applied to a caption display device for television, a multilingual subtitle display device for movies, a language learning device, and the like.

  In the above embodiment, an example in which an audio signal or a video signal is transmitted through a network or a broadcast is shown. However, the video signal, the audio signal, and auxiliary information are stored in a medium or storage such as a DVD or HDD, It is also possible to use the content reproduction apparatus in such a form that a reproduced one is input as a video signal, an audio signal, and auxiliary information.

  In the above embodiment, the content playback apparatus is installed on the local (karaoke box) side. However, the content playback apparatus is installed on the distribution side (for example, a live venue) to synthesize video synchronized with the video signal. And may be distributed.

The block diagram of the content reproduction apparatus which is 1st Embodiment of this invention The figure which shows the structure of the music data memorize | stored in the content reproduction apparatus The figure explaining the display system of the lyrics telop The block diagram of the content reproduction apparatus which is 2nd Embodiment of this invention The block diagram of the content reproduction apparatus which is the 3rd Embodiment of this invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Audio input part 2 Auxiliary information input part 3 Synchronization control part 4 Clock generation part 5 Delay part 6 Signal processing part 7 Storage part 8 Reference video data 9 Video sequence data 10 Video sequencer 11 Audio output part 12 Video output part 14 Audio data 17 External sequence data 18 External sequencer 19 Control signal output unit 21 Audio reproduction unit 22 Mixer 24 Video input unit 25 Video delay unit 26 Video mixer

Claims (8)

A content data storage unit for storing content data in which reference video time-series data and playback video time-series data are recorded in association with each other in time series;
A video signal input unit for inputting a video signal from the outside ;
By the this running the comparison process for detecting the synchronization position by comparing the image motion and motion video time-series data for the reference image of said video signal, a clock for generating a reproduction clock synchronized with the video signal A generator,
A video reproduction unit for reproducing the reproduction video time-series data based on the reproduction clock generated by the clock generation unit;
A content playback device comprising:
An auxiliary information input unit for inputting auxiliary information which is information indicating the position or speed of the reproduction clock from the outside ;
The clock generation unit corrects the comparison position of the reference video time-series data or the speed of the reproduction clock based on the auxiliary information.
Content playback device.   The content reproduction apparatus according to claim 1, further comprising a video synthesis unit that synthesizes and outputs a video signal input by the video signal input unit and a video signal reproduced by the video reproduction unit.   The content reproduction apparatus according to claim 2, further comprising a video delay unit that delays the video signal by a time required for generating a reproduction clock by the clock generation unit and then inputs the video signal to the video synthesis unit. An audio signal input unit for inputting an audio signal synchronized with the video signal from the outside;
An audio signal delay unit that delays and re-outputs the audio signal input from the audio signal input unit by a time required to generate a reproduction clock by the clock generation unit;
The content reproduction apparatus according to claim 1, 2 or 3, further comprising: The content data stored in the content data storage unit further includes audio time-series data associated with the reference video time-series data in time series,
5. The content reproduction apparatus according to claim 1, further comprising an audio reproduction unit that reproduces the audio time-series data based on a reproduction clock generated by the clock generation unit. The video time-series data stored in the content data storage unit is sequence data including video data to be displayed and timing data indicating the display timing thereof,
The content reproduction apparatus according to claim 1, wherein the video reproduction unit includes a sequence processing unit that reproduces the sequence data. The video time-series data stored in the content data storage unit is video data including a moving image,
The content reproduction apparatus according to claim 1, wherein the video reproduction unit includes a video reproduction unit that reproduces the video data. The content data storage unit further stores device control time-series data for controlling external devices in time series,
8. The content reproduction apparatus according to claim 1, further comprising an external device control unit that reads out the device control time-series data based on the reproduction clock generated by the clock generation unit and outputs a control signal. .

Images