JP2010112981A - Remote demonstration reproducing method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a remote demonstration reproducing method or the like capable of halving delay in both sounds occurring when mixing demonstrations at A point and B point distant from each other compared to heretofore. <P>SOLUTION: A device at A point and a device at B point mutually transmit guide data for reproducing the same guide via a communication network, and the device at A point and the device at B point simultaneously reproduce the same guide based on the mutually received guide data. The device at A point and the device at B point accept inputs of demonstration signals from demonstrators who look and listen to the reproduced guide and mutually transmit the demonstration signals accepted to be inputted via the communication network. The device at A point and the device at B point mix and reproduce the mutually received demonstration signals and the demonstration signals directly inputted to the devices themselves. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a method of mixing demonstrations at point A and point B that are remote and reproducing at each point.

  A conventional example of this type will be described with reference to FIG. A communication karaoke system is installed at each of the remote locations A and B, and each communication karaoke system performs data communication via a communication network. When a singer at point A and a singer at point B perform a duet, first, the communication karaoke system at point A starts karaoke performance of the music (hereinafter referred to as guide performance) (intro start). The singer at point A sings a song while listening to this guide performance (vocal in).

  In addition, the communication karaoke system at the point A simultaneously transmits the guide performance reproduced by itself to the karaoke system at the point B via the communication network. However, a communication time lag T1 occurs until the guide performance reaches point B.

  The communication karaoke system at point B plays immediately after receiving a guide performance sent from point A (intro start). However, the guide performance at point B always proceeds at a timing delayed by T1 from the guide performance reproduced at point A due to the influence of the communication time lag.

  The singer at point B sings a song in time with the timing of the guide performance played at point B (vocal in). The communication karaoke system at point B transmits the singing voice of the singer at point B to the communication karaoke system at point A via the communication network. However, a communication time lag T2 occurs until the singing voice at point B reaches point A.

  The communication karaoke system at point A immediately reproduces the singing voice received from point B and delivers it to the singer's ear at point A. However, when viewed from the timing of the guide performance at the point A, the singer's singing voice at the point B is accompanied by a delay of T1 + T2. Depending on the degree of this delay, the singer at the point A senses that the singing voice at the point B is heard with a delay, and feels difficult to sing. In the human sense, it is generally said that a delay exceeding 0.2 seconds is perceived as an unacceptable delay, and a delay within 0.2 seconds is acceptable. Such a delay is preferably shortened.

Therefore, in Patent Document 1, for example, a synchronization signal is periodically exchanged between the A point and the B point during the performance of karaoke, thereby controlling the timing of karaoke at both points. However, since performance timing control according to the synchronization signal is necessary during performance, the performance may be interrupted unnaturally.
JP 2005-77485 A

  The present invention improves the inconvenience of such a conventional example, and in particular, a remote demonstration playback method that can halve the delay of both sounds that occur when mixing the demonstration of A point and B point that are remote. The issue is to provide. It is an object to realize this without synchronization control during the demonstration.

  In order to solve the above problems, the present invention adopts the following configuration.

(1) Based on the same video or / and audio guide, a method for mixing and reproducing demonstration signals respectively input from points A and B that are remote from each other, The device at the B point transmits the guide data for reproducing the same guide to each other via the communication network, and the device at the A point and the device at the B point are the same based on the received guide data. And the device at point A and the device at point B receive a demonstration signal input from a performer who views the reproduced guide, and the demonstration signal is received via a communication network. The step of transmitting to each other, and the device at the point A and the device at the point B mix and reproduce the demonstration signal received mutually and the demonstration signal directly input to the own device. Remote demonstration reproducing method comprising the steps, a.

  In the present invention, video and audio guides are transmitted from the A point to the B point and from the B point to the A point, and the mutually received guides are reproduced simultaneously at the A point and the B point. The performer at point A performs each performance while viewing the guide reproduced at point A, while the performer at point B views the guide reproduced at point B. Each demonstration is sent to each other from point A to point B and from point B to point A. At each point, the performance input to the device and the performance received from the other party are mixed and reproduced.

  At each point, the time lag between the timing of the guide and the timing of the performance received from the other party is only one way of the communication path, so the time lag can be halved compared to the conventional case. Further, since the synchronous control of the points A and B is not performed during the demonstration, there is no possibility that the guide is unnaturally interrupted during the demonstration as in the conventional example.

  The present invention can be used as follows. For example, a karaoke can be played as a guide, and a remote performer can sing and duet along with the karaoke. You can also play a rhythm or metronome as a guide, and a remote performer can play and play an instrument along with the rhythm or metronome, or you can combine the performance of the instrument with a song. In addition, a movie image can be played as a guide, and a remote performer can perform post-recording and at-recording in accordance with this image.

(2) In the method of (1), the device at the point A and the device at the point B mutually transmit imaging data obtained by imaging the performer at each point by the imaging means via a communication network; And the device at the point B combine the received image data received with each other and the image data directly input to the own device, and reproduces the synthesized video. Method.

  In the present invention, the camera image at point A is transmitted to point B, and the camera image at point B is transmitted to point A, and is combined with the camera image input at each point. For this reason, for example, it is possible to combine the performer at point A and the performer at point B into one video.

(3) In the method of (1), in the step of transmitting the demonstration signal to each other, the demonstration signal is preferentially transmitted on a channel different from the guide data.

  In the present invention, since the guide and the demonstration are transmitted on different channels and the transmission of the demonstration is prioritized over the transmission of the guide, the delay of the demonstration with respect to the guide can be minimized.

(4) A one-point demonstration reproduction device that mixes and reproduces demonstration signals respectively input from point A and point B that are remote from each other based on the same video and / or audio guide, A guide transmission function for transmitting guide data for reproducing the guide to a device at a partner point via a communication network, and a guide that is the same as the partner point is reproduced at the same time as the partner point based on the guide data received from the device at the partner point. A reception guide reproduction function, a demonstration signal transmission function that receives an input of a demonstration signal from a performer who views the reproduced guide, and transmits the demonstration signal that has received the input to a device at a partner point via a communication network; A mixing function that mixes and reproduces the performance signal received from the device at the location and the performance signal directly input to the device itself. Remote demonstration reproducing apparatus characterized.

  In this invention, there exists the same effect as the remote demonstration reproduction | regeneration method mentioned above.

  According to the present invention, it is possible to halve the delay of both sounds that occur when the remote demonstrations of point A and point B are mixed. Further, since the synchronous control of the points A and B is not performed during the demonstration, there is no possibility that the guide is unnaturally interrupted during the demonstration as in the conventional example.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. As described above, the present invention can be used for various purposes such as karaoke, ensemble, and after-recording. However, the present embodiment will be described by taking the case of using the karaoke as an example.

  First, an outline of the present embodiment will be described based on FIG. In the present embodiment, the data of the guide (karaoke video and performance) reproduced at the B point is transmitted from the A point. At the same time, guide data to be reproduced at point A is transmitted from point B. Both data are the same data. At the point A, the guide data transmitted from the point B is streamed and reproduced. At the point B, the guide data transmitted from the point A is reproduced by streaming. Streaming playback at both points starts simultaneously. A communication time lag T1 occurs until the guide data reaches the partner point, but it is offset by streaming playback at points A and B at the same time.

  The intro starts at point A and point B at the same time, and then the singer at point A starts singing while listening to karaoke (vocal in). At the same time, the singer at point B starts to sing while listening to karaoke. At this point, the timings of both songs are the same.

  The song sung at point A is distributed live at point B. The song sung at point B is distributed live at point A. At the point A, the song at the point A and the song distributed live from the point B are mixed and reproduced. At the B point, the song at the B point and the song distributed live from the A point are mixed and reproduced. Since a communication time lag T2 is generated when song data is transmitted between the AB points, the song at the B point is reproduced with a delay of T2 at the A point, and the song at the A point is reproduced with a T2 delay at the B point. However, as compared with the conventional example shown in FIG. 6 with a delay of T1 + T2, the delay can be halved.

  FIG. 2 is a configuration diagram of the communication karaoke system (demonstration playback device) in the present embodiment. Since the system at the point A and the system at the point B have the same configuration, the configuration of the system at the point A will be described. The communication karaoke system includes karaoke control means 1, remote duet control means 2, performance start synchronization means 3, and video / performance buffer 4. The communication karaoke system also includes a vocal buffer 5, a video composition unit 6, and a composite video buffer 7. Further, the communication karaoke system includes a buffer reproduction means 8, a remote communication means 9, and a music database (DB) 10. In addition to this, the communication karaoke system includes a microphone 11, a digital camera 12, a speaker 13, a display monitor 14, an operation panel 15, and a remote control interface (I / F) 16.

  The karaoke control means 1 provides basic karaoke functions such as karaoke song selection, key adjustment, tempo adjustment, and performance stop according to inputs from the karaoke operation panel 15 or remote control I / F 16.

  The remote duet control means 2 realizes a duet between the singer at the point A and the singer at the point B which are remote, and the configuration will be described in detail later.

  The performance start synchronization means 3 makes preparations for starting playback of the karaoke at the point A and the karaoke at the point B at the same time.

  The video / performance buffer 4 temporarily stores guide (karaoke video and performance) data sent from the point B.

  The vocal buffer 5 temporarily stores vocal data (demonstration signal) sent from the point B.

  The video composition means 6 synthesizes the video photographed by the digital camera 12 at the point A and the video photographed by the digital camera at the point B.

  The synthesized video buffer 7 temporarily stores the video data synthesized by the video synthesizing means 6.

  The buffer playback means 8 performs streaming playback of the data temporarily stored in the video / performance buffer 4, vocal buffer 5, and composite video buffer 7.

  The remote communication means 9 performs peer-to-peer data communication with the remote communication means of the communication karaoke system at point B.

  The music DB 10 stores a large number of music guide (karaoke video and performance) data in a streaming file format.

  The singing voice of the singer (demonstrator) is input to the microphone 11. The digital camera 12 takes a picture of a singer. The speaker 13 is connected to an amplifier and a mixer (not shown), and mixes and outputs the song input from the microphone 11 and the sound (guide performance and the song at the point B) output from the buffer reproducing means 8. The display monitor 14 displays the video (karaoke video, synthesized video of the singer) output from the buffer playback means 8. The operation panel 15 includes buttons for inputting basic karaoke instructions such as karaoke music selection, key adjustment, tempo adjustment, and performance stoppage to the karaoke control means 1. The remote control I / F 16 performs short-range wireless communication such as infrared communication with a remote control including a button capable of inputting the same instruction as the operation panel 15.

  The communication karaoke system at point B is configured in the same way as the communication karaoke system at point A. Regarding the configuration of the communication karaoke system at point B, among the configurations of the communication karaoke system at point A, the points A and B It shall be read as a point. The communication karaoke system at the point A and the communication karaoke system at the point B perform peer-to-peer communication via a communication network N such as the Internet.

  FIG. 3 is a configuration diagram of the remote duet control means 2. The remote duet control means 2 includes an A / D converter 21, a live audio encoder 22, a live audio buffer 23, a live video encoder 24, a live video buffer 25, a streaming processing unit 26, and a channel distribution processing unit 27. And a main control unit 28.

  The A / D converter 21 converts the audio signal input from the microphone 11 into a digital signal.

  The live audio encoder 22 encodes audio input from the microphone into streaming data.

  The live audio buffer 23 temporarily stores audio data encoded by the live audio encoder.

  The live video encoder 24 encodes the video data of the singer imaged by the digital camera 12 into streaming data.

  The live video buffer 25 temporarily stores the video data encoded by the live video encoder 24.

  The streaming processing unit 26 time-divides the data of the streaming file 10f of the music selected for performance from the music DB 10 into data pieces, and guides (karaoke video and performance) to the header (packet header) of each divided data piece. The data of channel (CH) 1 is added as a data channel, and streaming output is performed. In addition, the streaming processing unit 26 time-divides the data temporarily stored in the live audio buffer 23 into data pieces, adds CH2 data as a vocal channel to the header (packet header) of each divided data piece, Perform streaming output. The streaming processing unit 26 time-divides the data temporarily stored in the live video buffer 25 into data pieces, and adds CH3 data as a camera video channel to the header (packet header) of each divided data piece. , Perform streaming output. At this time, the streaming processing unit 26 controls the ratio of data to be output for streaming for each channel according to a preset setting, and preferentially transmits vocal data to the guide data.

  The channel distribution processing unit 27 receives the CH1 to CH3 data pieces streamed from the point B via the remote communication means 9, the CH1 data to the video performance buffer 4, and the CH2 data to the vocal buffer 5. The data of CH3 is input to the video composition means 6, respectively.

  The main control unit 28 controls the operation timing of each of the above elements.

  Here, the karaoke control means 1, the remote duet control means 2, the performance start synchronization means 3, the video composition means 6 and the buffer reproduction device 8 are realized by an information processing apparatus such as a CPU executing a program. The video / performance buffer 4, vocal buffer 5, composite video buffer 7 and music DB 10 are formed in a storage area of a storage device such as an HDD or RAM. The remote communication means 9 is a communication device such as a NIC that is connected to the communication network N and is suitable for peer-to-peer communication with a communication karaoke system at point B. Each unit including the remote duet control unit 2, the video composition unit 6, and the buffer reproduction unit 8 may be configured by hard logic.

  Next, the operation of the present embodiment will be described with reference to FIG.

(S1) A performance start synchronization process is executed between the communication karaoke system at point A and the communication karaoke system at point B. The performance start synchronization means 3 pings the communication karaoke system at the point B via the remote communication means 9, and stores half of the time taken from the transmission of the test packet to the reception of the response as the estimated communication time lag time. To do.

(S2: Guide transmission function) Next, the performance start synchronization means 3 at the point A transmits a command instructing the performance start to the communication karaoke system at the point B, and at the same time waits for the estimated communication time lag, Simultaneously with waiting for the communication time lag prediction time, transmission of karaoke guide data selected in advance is started toward point B. When the predicted communication time lag has elapsed at point A, a performance start command arrives at the communication karaoke system at point B, and the communication karaoke system at point B receives the performance start command and simultaneously transmits the same karaoke guide data. Start towards point A.
At this time, the streaming processing unit 26 time-divides the guide data of the music streaming file 10f, adds CH1 data to the header of the divided data piece (packet header), and communicates with the other party via the remote communication means 9. Stream to karaoke system.

(S3) The communication karaoke system at the point B receives the guide data streamed from the point A one after another. A communication time lag T1 occurs until the guide data transmitted from the point A reaches the point B. Since the guide data at the points A and B are reproduced at the same time, the communication time lag T1 is a relative delay between the points AB. Does not cause. The guide data transmitted from the point A is received by the remote communication means 9 at the point B. The channel distribution processing unit 27 at the point B determines the channel identification information added to the data pieces to be received one after another, and since CH1 is added to the guide data, it is temporarily stored in the video / performance buffer 4.

(S4: Reception guide reproduction function) The buffer reproduction means 8 at the point B reproduces guide data accumulated one after another in the video / performance buffer 4, outputs karaoke performance sound from the speaker 13, and also includes karaoke subtitles, etc. Are displayed on the display monitor 14.

(S5) The singer at point B sings a song while watching the guide output from the speaker 13 and the display monitor 14.

(S6) The singing voice is input from the microphone 11, converted into a digital signal by the A / D converter 21, encoded into streaming data by the live audio encoder 22, and temporarily stored in the live audio buffer 23.
In addition, the digital camera 12 at the B point photographs the singer at the B point. Video data output from the digital camera 12 is encoded into streaming data by the live video encoder 24 and temporarily stored in the live video buffer 25.

(S7: Demonstration signal transmission function) The B point streaming processing unit 26 continuously time-divides the guide data from the point of S2, adds CH1 as channel identification information to the header of the divided data piece, and The singer's vocal data temporarily stored in the live audio buffer 23 is time-divided, CH2 is added to the header of the divided data piece as channel identification information, and the singer's video data temporarily stored in the live video buffer 25 And CH3 is added as channel identification information to the header of the divided data pieces, the data pieces of each channel are serially arranged, and are streamed to the communication karaoke system at point A via the remote communication means 9. At this time, the streaming processing unit 26 adjusts the ratio of the data pieces to be transmitted to increase the ratio of the data pieces of the vocal data to the data pieces of the guide data (for example, 1: 2). Thus, the vocal data is preferentially transmitted with respect to the guide data. In this way, it is possible to minimize the delay of vocals received and reproduced from point B with respect to the guide performance reproduced at point A.

(S8) The channel distribution processing unit 27 at the point A receives the data pieces streamed out from the point B one after another via the remote communication means 9, determines the channel identification information added to the header of each data piece, The CH1 data piece is inputted to the video / performance buffer 4, the CH2 data piece is inputted to the vocal buffer 5, and the CH3 data piece is inputted to the video composition means 6, respectively. At this time, a communication time lag T2 is generated until the data transmitted from the B point reaches the A point.

(S9) The video composition means 6 at the point A synthesizes the video input from the digital camera 12 at the point A and the camera video at the point B input from the channel distribution processing unit 27, and temporarily stores them in the composite video buffer 8. accumulate. As a method of video composition, for example, the stage background of a singer at each point of AB is set to blue, and the video is synthesized by chroma key technology.
FIG. 5 is an image composition image. The video of the singer at the point A and the video of the singer at the point B may be combined and the lyrics subtitles may be superimposed and displayed on this screen. Alternatively, the display area for the karaoke guide video and the display area for the camera video may be divided and displayed simultaneously.

(S10: mixing function) The buffer reproduction means 8 at the point A is a mixer for the karaoke guide performance sound temporarily stored in the video / performance buffer 4 and the vocal sound of the singer B point temporarily stored in the vocal buffer 5. The sound is mixed and output from the speaker 13. The voice of the singer at point A input from the microphone 11 at point A is also mixed directly through the mixer and output from the speaker 13. In the mixer, effects such as echo can be applied as appropriate.

  Since the guide performances at the points A and B are performed at the same time, the song of the singer at the point A and the song of the singer at the point B transmitted from the point B are equal to the communication time lag T2 at the point A. Is output from the speaker with a delay of only. However, the vocal delay can be reduced to about half compared to the conventional example described with reference to FIG.

  Also, the buffer playback means 8 at the point A arranges the karaoke guide video temporarily stored in the video / performance buffer 4 and the camera composite video temporarily stored in the composite video buffer on a single screen on the display monitor 14. indicate.

  In the above operation, a guide is transmitted from point A, a song is sung at point B based on this guide, vocal data of point B is transmitted to point A, and a song at point A and a point B are transmitted at point A. The explanation is centered on the operation of mixing and playing the song. Exactly the same operation is performed simultaneously with the points A and B reversed. About the reverse operation | movement, it can know by replacing A point and B point.

  As described above, according to the present embodiment, it is possible to suppress the delay of vocals transmitted from the opponent point to about half of the conventional example in FIG. 6 with respect to the guide performance reproduced at one point. In addition, since the synchronous control of the A point and the B point is not performed during the demonstration, there is no possibility that the guide is unnaturally interrupted during the demonstration as in the conventional example. Further, by transmitting the vocal data transmitted from one point to the opponent point with priority over the guide data, the delay of the vocal with respect to the guide performance played at the opponent point can be minimized. Also, by synthesizing and playing back the camera video at one point and the other point, it is possible to excite the image of the singer at point A and the singer at point B being together.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. For example, the streaming data transmission function for each channel may not necessarily be installed, and the camera video composition function may not necessarily be installed. The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

It is explanatory drawing for demonstrating the outline | summary of embodiment of this invention. It is a block diagram of the communication karaoke system which is embodiment of this invention. It is a block diagram of the remote duet control means shown in FIG. It is a flowchart which shows operation | movement of embodiment shown in FIG. It is an image figure of the camera image | video synthetic | combination process shown in FIG. It is explanatory drawing for demonstrating a prior art example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Karaoke control means 2 Remote duet control means 3 Performance start synchronization means 4 Video / performance buffer 5 Vocal buffer 6 Video composition means 7 Composite video buffer 8 Buffer playback means 9 Remote communication means 10 Music database 11 Microphone 12 Digital camera 13 Speaker 14 Display Monitor 15 Operation panel 16 Remote control interface

Claims (4)

Based on the same video or / and audio guide, a method of mixing and playing demonstration signals input from points A and B that are remote from each other,
The device at point A and the device at point B mutually transmit guide data for reproducing the same guide via a communication network; and
The device at point A and the device at point B simultaneously play back the same guide based on the received guide data;
The device at the point A and the device at the point B receive an input of a performance signal from a performer who views the reproduced guide, and transmit the performance signal that has received the input to each other via a communication network;
The device at the point A and the device at the point B mix and reproduce the demonstration signal received from each other and the demonstration signal directly inputted to the own device;
A remote demonstration playback method characterized by including: The method of claim 1, wherein
The device at the point A and the device at the point B mutually transmit imaging data obtained by imaging the performer at each point by the imaging means via a communication network;
The device at the point A and the device at the point B synthesize the image data received from each other and the image data directly input to the own device, and reproduce the synthesized video;
A remote demonstration playback method characterized by including: The method of claim 1, wherein
In the step of transmitting the demonstration signals to each other, the demonstration signal is preferentially transmitted on a channel different from the guide data. A one-point demonstration playback device that mixes and reproduces demonstration signals respectively input from point A and point B that are remote from each other based on the same video or / and audio guide,
A guide transmission function for transmitting guide data for reproducing the guide to a device at a partner point via a communication network;
A reception guide reproduction function for reproducing the same guide as the other party at the same time as the other party based on the guide data received from the device at the other party;
A demonstration signal transmission function for receiving a demonstration signal input from a performer who views the reproduced guide, and transmitting the demonstration signal that has received the input to a device at a partner point via a communication network;
A mixing function that mixes and reproduces the demonstration signal received from the device at the other party and the demonstration signal input directly to the own device,
A remote demonstration playback device characterized by comprising:

Images