JP2022028570A - Recorded data reproduction method and system - Google Patents

Abstract

To support an online ensemble among players in remote locations, enabling a realistic physical feeling through reducing an effect of a network delay as possible.SOLUTION: A sound/video distribution technology with a play position and time specified allows players to perform a chorus or a concert without any special skill, through synchronizing online a performance timing, even staying in remote locations if timing when a sound/video distribution server 104 distributes data is different. Each terminal is capable of displaying a musical score distributed from a musical score provision server 101, as well as capable of reproducing a sound or a video in which the provided musical score is played, generating a pace of the chorus or the concert, so as to allow each player to synchronize the timing through the sound or the video simultaneously reproduced at each terminal.SELECTED DRAWING: Figure 2

Description

The present invention relates to a recorded data reproduction method and system, and more specifically, to a recorded data reproduction method and system capable of providing and managing musical scores and supporting an online ensemble via a network.

Conventionally, when performing jointly online, for example, by playing or singing along with the delivered video, there is a network delay or communication delay, and it is possible to play with other performers even if there is some deviation. It is done. However, in order to match the timing of performances with each other in such an environment and to obtain the feeling of playing as if they were playing in a realistic ensemble, a high performance technique considering network delay is required. In recent years, with the development of the network environment, the network delay has been reduced, but when trying to synthesize multiple audio and video that played music, the deviation is easily perceived even with normal listening ability, and it is realistic. It is difficult to get a feel for it.

Despite these technical challenges, in recent years there has been an increasing need for ensemble and chorus by remote performers, so to record each performer's performance, though not in real time, to compensate for delays. The method of editing is also realized. In addition, for the purpose of providing an ensemble device or the like that reduces discomfort based on the delay time on the network, the first performer at the first point and the second point different from the first point are connected via the network. It is placed at the first point to realize an ensemble with the second performer, and corresponds to the biometric signal generated due to the intention of the first performer to make a movement to generate a rhythmic sound. A biometric signal acquisition unit that acquires biometric information, and a transmission unit that transmits biometric information to a second point via a network and then transmits music sound information corresponding to the sound generated by the movement of the biometric signal. Including, at the second point, an ensemble device has been proposed in which the stimulus corresponding to the biological information is presented to the second performer earlier than the sound corresponding to the musical sound information is reproduced. There is no mechanism that makes it easy to create, ensemble, and publish performance sounds, and to easily create and present music together. Application software on mobile terminals, etc. that allows other users to view the displayed audio signal and superimpose a new audio signal while outputting the audio. In addition, the case of connecting the musical instrument and the mobile terminal by wire is also included in the means. (See, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2015-20801

However, the conventional score display system only makes the performer illusion by physical stimulation and plays at an earlier timing by the amount of network delay, and has only the effect of reducing the sense of incongruity. When there are two or more people, there is a problem that it is difficult to absorb the delay. Further, there is a problem that even if the music can be edited so as to compensate for the delay after the performance is completed, the performance cannot be performed in real time.

The present invention has been made in view of the above-mentioned conventional problems, and the performance position is displayed on the score according to the reproduction of the sound or the moving image, and the sound played according to the reproduction is synthesized or the moving image is formed. The purpose is to provide a recorded data reproduction method and system that supports an online ensemble between performers in remote areas where the effects of network delays can be reduced as much as possible and a realistic experience can be obtained by displaying the music side by side. do.

The invention according to claim 1 is a recorded data reproduction method for supporting an online ensemble via a network, in which a score transmission step of transmitting a score to a terminal via the network and a first performance related to the score. The recorded data is played back from the specified position on the score displayed on the terminal, and is performed by superimposing the playback step of displaying the playback position on the score and the first performance played in the playback step. , A performance acquisition step in which a second performance related to a musical score is acquired by a plurality of terminals and transmitted via a network, and a plurality of second performances transmitted in the performance acquisition step are reproduced as a first performance. It is characterized by having an output step for outputting together with.

The invention according to claim 2 records the first performance corresponding to the designated position on the score when the designation of the start position of the performance is received from the terminal in the recording data reproduction method according to claim 1. The data reproduction time is acquired according to the correspondence information generated from the score and the data recording the first performance, and is transmitted to the terminal to reproduce the data recording the first performance from the acquired reproduction time. It is characterized by further providing a reproduction step to be performed.

The recorded data reproduction method according to claim 1, wherein the data in which the first performance is recorded is audio data obtained by recording the previous performance. ..

The invention according to claim 4 is the recorded data reproduction method according to any one of claims 1 to 3, wherein the data recording the first performance is moving image data obtained by recording the first performance. It is characterized by that.

The invention according to claim 5 is characterized in that, in the recorded data reproduction method according to any one of claims 1 to 4, the reproduction step starts reproduction at a different reproduction start time for each terminal.

According to the present invention, the score transmission step of transmitting the score to the terminal via the network and the data recording the first performance related to the score are reproduced from the designated position on the score displayed on the terminal. , The playback step of displaying the playback position on the score and the second performance related to the score, which is performed on top of the first performance played in the playback step, are acquired by multiple terminals and transmitted via the network. Since it includes a performance acquisition step to be performed and an output step to output a plurality of second performances transmitted in the performance acquisition step together with the reproduced first performance, the performance was performed in accordance with the recorded reproduction. By synthesizing audio and displaying videos side by side, it is possible to support an online ensemble between performers in remote locations where the effects of network delays can be reduced as much as possible to obtain a realistic experience.

It is a whole system block diagram of one Embodiment of this invention. It is a figure for demonstrating the principle of each embodiment of this invention. It is a functional block diagram of the server of one Embodiment of this invention. It is a figure which shows the association of the measure on the musical score and the corresponding play | reproduction position in one Embodiment of this invention. It is a flowchart which shows the process of acquiring the association between the bar on the musical score and the corresponding reproduction position of one Embodiment of this invention. It is a figure for demonstrating the display of the position on the musical score corresponding to the reproduced image on the moving image of one Embodiment of this invention. It is a figure for demonstrating the process which manages the display of the musical score on the terminal screen of another embodiment of this invention. It is a figure which shows the flow of a series of services at the time of reproducing and supporting the recorded voice of one Embodiment of this invention. It is a figure which shows the flow of a series of services at the time of playing back and supporting the recorded moving image of one Embodiment of this invention. It is a flowchart which shows the process which supports the online ensemble by reproducing the recorded data of one Embodiment of this invention. It is a figure which shows the flow of a series of services at the time of reproducing and supporting the recorded voice of another embodiment of this invention. It is a figure which shows the flow of a series of services at the time of playing back and supporting the recorded moving image of another embodiment of this invention. It is a flowchart which shows the process which supports the online ensemble by reproducing the recorded data of another embodiment of this invention.

Hereinafter, embodiments of the recorded data reproduction method and system of the present invention will be described with reference to the drawings. Even in different drawings, the same reference numerals are used to indicate the same processing and configuration.

In this embodiment, when providing an electronic score via the Internet and then viewing a performance video of a song posted on a video posting site, the start position of video playback can be specified, or the entire performance of the image being played can be specified. In a system that displays the position in the music, it assists a plurality of performers in executing an online ensemble. In the following embodiments, the score or the total score is used, but the performance is not limited to this. Any form can be supported as long as some drawing showing the progress of the above can be displayed on the terminal. Here, basically, the following embodiment will be described on the premise that some of the performers who uploaded the audio or video are associated with the score used for the performance and set in the server in advance. This embodiment can be implemented as long as the music being played and the score are almost the same. For example, a person who recognizes the audio or video distributed for some reason can set it in the server in advance in association with the score corresponding to the music being played, or by other methods known in the present technical field. You can associate the score with the audio / video.

Further, in each of the following embodiments, the sound or the moving image obtained by recording or recording the performance performed in advance is included in the "data recording the performance" of the present invention, and the sound is voiced via the network. Choruses or ensemble that are supported by playing or playing a video are included in the "online ensemble" of the present invention, but are not limited to all that human beings perform in multiple timed manner using themselves or some tool. Performance is included.

The online ensemble support system of the present invention supports ensemble and chorus by performers in a plurality of remote locations, but as shown in FIG. 2, basically common audio data that generates chorus and ensemble pace. The audio / video distribution server 104 distributes the moving image data to each performer 201 to 203, and the score providing server 101 distributes the score of each performer to each performer's terminal and refers to it. It is possible to output a chorus or ensemble with a performance timing by the performer, for example, on the terminal 114. Here, if there is no big difference in the network environment of each performer's terminal, that is, if there is no big difference in the communication delay between each terminal and the server, it is time to deliver the voice or video from the server 104 to each terminal. And since the difference in the time for uploading the recorded data from each terminal is small, the communication time between the server and the terminal may be the same for each terminal. Specifically, the audio / video distribution server 104 distributes data to each terminal at the same time and outputs it, and each performer plays in accordance with this, so that the timing of each performance can be adjusted. Therefore, an online ensemble can be realized (first embodiment). On the other hand, if the network environment is different for each terminal and the communication delay is different, it is possible to adjust the communication delay for each terminal to match the distribution timing, but it is difficult to adjust if the network delay changes each time the performance starts. Become. Therefore, in this case, by acquiring the delay time for each terminal in advance, determining the time difference equal to or longer than the delay time, and setting the time difference at the performance start time of each performer, the communication delay of each terminal is not affected. Each performer can perform at the same timing (second embodiment).

As described above, in each embodiment, the timing at which the audio / video distribution server 104 reproduces the data is different, but each performer can use the audio / video distribution technique for specifying the performance position and time described in detail below. You can perform choruses or ensemble without special skills by adjusting the performance timing online while you are in a remote location. That is, each terminal used in each of the following embodiments displays the score distributed from the score providing server 101, and reproduces the voice or video of playing the provided score, which generates the pace of chorus or ensemble. Therefore, each performer can adjust the timing with the voice and the moving image simultaneously played on each terminal. In addition, since the currently played performance part is highlighted on the score, each performer can more accurately match the performance timing of each part by following the highlights on the score with their eyes. Can be done. By collectively outputting the audio and video played and distributed by each performer on the terminal 114 in this way, it is possible to experience an ensemble or chorus as if each performer were actually gathering and performing. Can be done.

In addition, it is not necessary for the user who received the score to post to the video site, and the video obtained by searching the performance video of the music of the score obtained by the user who received the score on the posting site, etc. May be. In other words, when the score of a certain song, including classical music, is published, the video that played that song is often shot and posted by one of the performers, and in fact, the same song has multiple videos. Is often present. Therefore, even if the user who received the electronic score does not post the video, the user can easily find such a video and obtain the URL of the video by searching the Internet or the like. By specifying such a URL, the server can access the performance video of the score provided to the user. The same applies to voice data.

Further, the musical score data of the electronic musical score used in this embodiment is MusicXML, and is basically stored in the MusicXML data format in the server. The score written in MusicXML can be sent to the terminal as it is and displayed on the browser, or it can be converted to another display format on the server, for example PDF, and sent to the terminal for display on the browser of the terminal. .. As described above, MusicXML is used for the score data of the present embodiment, but the present invention is not limited to this, and any format can be used as long as it can generate a score for display by performing some conversion or processing.

(System configuration)
The operation and processing of a specific system of the score providing method used in one embodiment of the present invention will be described below. FIG. 1 is an overall system configuration diagram of an embodiment of the present invention. This system includes a score providing server 101, a communication server 102, and an audio / video distribution server 104 as servers for controlling the entire system, such as distributing score data and displaying performance points. It is connected via the network 103. The terminal 111, which is a device for displaying music scores and moving images or playing music files, is basically wirelessly connected to the network 103, and can be, for example, a tablet terminal. In the present embodiment, the terminal 111 is connected to the network 103. The connection is made by a mobile phone line or a wireless network such as Wi-fi or BLUETOOTH (registered trademark). Here, the terminals 111 to 114 can be a tablet terminal, a smartphone, a mobile personal computer, or the like, but basically, the score data or the like is received from the server to display an image, play a video file or a music file, and the like. Any device such as a mobile or desktop personal computer or a dedicated terminal can be used as long as a certain input operation can be performed with a touch panel, a mouse or a keyboard. Further, the terminals 111 to 114 can be used not only for displaying the score and playing the moving image, but also for executing various processes related to access to the user's score providing server and inputting / outputting data.

Further, as the server of the present embodiment, three servers of a score providing server 101, a communication server 102, and an audio / video distribution server 104 are provided, and instructions such as displaying an image such as a score and starting playback are given from terminals 111 to 114. When transmitted to the score providing server 101, the score providing server 101 transmits score data, calculates the position of the specified playback position on the video, and instructs the communication server 102 to play the video, and as a result. , The audio / video distribution server 104 transmits and displays the video instructed to the terminals 111 to 114, but the present invention is not limited to this, and the image data of the image data from the reception of the instruction by the score providing server 101 to the audio / video distribution server 104. It is possible to give instructions for transmission, or it is possible to divide the functions among more servers.

Here, the audio / video distribution server 104 may be, for example, a video posting site or the like, or may simply be a server of a site that distributes video. Further, although described as a video distribution server in the present embodiment, it is sufficient that the music played based on the score can be listened to, so even if it is a content distribution server that does not accompany an image and can distribute only audio. good. In this case, it goes without saying that all the following description of the present embodiment also applies to the content distribution server that distributes music without moving images.

In the present embodiment, when a video playback instruction is transmitted from any of the terminals 111 to 114, video data for starting playback from the start position designated for the terminals 111 to 114 is transmitted, and the audio / video is played together with the score display. be able to. In the present embodiment, the image of the musical score and the time bar indicating the progress of the moving image or the sound are arranged at predetermined positions on the display screens of the terminals 111 to 114, but the present invention is not limited to this. It can also be displayed in combination by any of the above methods. Specifically, in the present embodiment, the images of the terminals 111 to 114 are divided into upper and lower parts and displayed, but the user can also display the images by dividing them into left and right parts or partially overlapping them. It can be displayed for ease of use. Of course, even in this case, the display of the score, the instruction for playing the moving image, and the like are processed by the server or the like of the present embodiment, and the moving image is distributed from the moving image distribution server. On the other hand, it is also possible to separately prepare a moving image playback terminal so that only the score is displayed on the terminals 111 to 114, and this can be realized by a plurality of different terminals.

(Providing sheet music and video playback processing)
FIG. 3 is a functional block diagram of a server according to an embodiment of the present invention, and FIG. 4 is a diagram showing an association between a bar on a musical score and a corresponding reproduction position in the present embodiment. The score providing server 101 receives score data from a server (not shown) or the like, stores the score data, manages the score data management module 301, performs necessary processing on the received score data, and transmits the received score data to the terminal. When the designation of is received, the position on the video is calculated, and the position of the currently played image in the entire video is calculated from the communication server 102 and the position of the score displayed on the terminal is calculated. It also includes a score data transmission module 302 that outputs highlights, and a score-related data reception module 303 that receives various electronic score-related data used in the present embodiment, such as a bar time conversion table. Further, the communication server 102 is a video management module 304 that manages data such as a URL of a video that plays a musical score displayed on the terminal 111, and a video / identification information acquisition that acquires video data when receiving video identification information. Upon receiving the designation of the video start position from the module 305 and the score providing server 101, the playback start position is specified to the audio / video distribution server 104 based on the audio / video data managed by the audio / video management module 304. Based on the start position designation module 306 that instructs playback, the playback time output module 307 that outputs the current playback time received from the audio / video distribution server 104 to the score providing server 101, and the correspondence between the acquired video and the electronic score. It is provided with a measure time conversion table creation module 308 that creates a measure time conversion table.

Hereinafter, the preparation process for the moving image reproduction service from the provision of the electronic musical score according to the embodiment of the present invention will be described with reference to FIGS. 3 to 5. In general, musical score data of electronic musical scores, for example, MusicXML format data, specify note data for each bar, and when the terminal receives the musical score data, a staff notation is displayed on the browser and the note data of each bar is displayed. Draw the score based on it. Therefore, when a certain measure is specified by touching it with a finger or clicking it with a mouse on the browser displaying the score data, it is possible to recognize which measure it is. In the present embodiment, the score is displayed on the browser of the terminal based on the score data in this way, and by designating a predetermined measure on the displayed score, the playback start position of the moving image is specified and the moving image is played. Is transmitted to the score providing server 101. The score providing server 101 calculates the playback position of the moving image from the designated measure, and in the present embodiment, calculates the playing time from the beginning of the moving image, and instructs the distribution server 102 to play the moving image.

Further, when the score data cannot be displayed as it is on the terminal because there is no plug-in or the like, the score data can be transmitted as image data, for example, PDF format data from the score providing server 101 and displayed on the browser (not shown). Any method known in the art can be used for generating PDF score data from the score data executed on the score providing server 101 and performing the moving image playback start processing based on the PDF data. can.

In order to provide the moving image reproduction service of the present embodiment as described above, the electronic musical score is first provided to the user, and finally the correspondence between the specified moving image and the musical score, specifically, the measure as shown in FIG. It is necessary to create a time conversion table and so on. This process will be described with reference to the flowchart shown in FIG.

(Creation of measure time conversion table)
FIG. 5 is a flowchart showing a process of acquiring an association between a bar on a musical score and a corresponding reproduction position according to the embodiment of the present invention. In the present embodiment, the process of providing the electronic sheet music is started, and the user is provided with a comprehensive service related to the sheet music. When instructed by specifying the score, the score providing server 101 transmits the score (step 501). After that, the user 203 displays the obtained score on a terminal (not shown) or prints it with a printer, performs a performance using the displayed or printed score, and takes a picture of the performance such as practice or production with a video camera or the like. Alternatively, the audio is recorded with a microphone (step 502), the captured audio / video is uploaded to the audio / video distribution server 104, and the audio / video ID is received (step 503). Here, the audio / video ID is identification information for identifying the video on the Internet, and is, for example, a URL in which the video is stored on a posting site or the like, but the identification information may be used without limitation. can.

When the user acquires the audio / video ID, the user performs some association with the provided electronic score to the communication server 102 by any method known in the present art and transmits the audio / video ID (step 504). The communication server 102 also acquires the information for identifying the provided electronic score by the moving image / identification information acquisition module 305, acquires the voice / video data by the voice / video ID, and electronically from the score providing server 101. Obtain the score (step 505). The measure time conversion table creation module 308 calculates the correspondence between the measures of the score and the playback time of the audio / video based on the acquired audio / video and electronic score (step 506).

In the present embodiment, the audio / video playback time 411 and the measure number 421 of the score are associated with each other by the measure time conversion table 401 as shown in FIG. 4, and the measure time conversion table 401 is referred to, for example. Since it can be determined that the number (4) (measure 4) corresponds to the audio / moving image reproduction time t4, the reproduction time t4 can be calculated from this. That is, the measure time conversion table creation module 308 creates the measure time conversion table 401 by associating the measures of the score with the playback time of the audio / moving image. Here, the bar time conversion table 401 can be automatically detected by any of the methods known in the present embodiment, or can be created by human support. The measure time conversion table 401 created in this way is transmitted to the score providing server 101 for the moving image reproduction service of the present embodiment (step 507).

As described above, according to the score providing method of the present embodiment, the electronic score desired by the user is provided, and after associating with the voice / moving image instructed by the user, the online ensemble support service is provided to the user. First, the video playback start process will be described, and then the process of highlighting the video playback position on the score will be described.

(Video playback start processing)
FIG. 6 is a diagram showing an operation of designating a position on the score of the present embodiment.

In the present embodiment, as shown in FIG. 6, the score data transmission module 302 transmits data in MusicXML format to the terminal 111, displays it on the browser to specify the playback position, and informs the server of the designated measure. As a result, the server calculates the playback position of the audio / moving image corresponding to the measure, and in the present embodiment, the playback time, and instructs the communication server 102 to play the moving image from that position. The communication server 102 identifies the performance video of the score displayed on the terminal, instructs the audio / video distribution server 104 to specify the instructed playback time and distributes the audio / video, and the terminal 111 outputs the audio / video. The video is played. Normally, MusicXML is handled on a terminal using a dedicated display application or playback application known in the present technology, so details are not described here, but for example, a score is displayed, and a touch with a finger or a click with a mouse, etc. Then, it is possible to embed a script or the like that sends the bar number corresponding to the position on the screen to the server so that the bar number is sent from the terminal to the server, or other present technology. By executing such a process by any method known in the field, the present embodiment can be implemented without particularly changing the software on the terminal side. On the contrary, the present embodiment is to be implemented by installing or downloading software for executing the necessary processing in the present embodiment such as sending the measure number to the server to the terminal in advance so that the software can be executed on the terminal. You can also.

Specifically, first, as shown in FIG. 6, when the user touches and specifies the fourth bar 602 at the top of the score 605 displayed on the terminal 111 with the finger 606, the terminal 111 starts playing the audio / video. Get the bar number to do. When the measure number is acquired, the terminal 111 transmits the acquired measure number, here, the fourth measure, to the score providing server 101 to notify the score (arrow 603). The score providing server 101 calculates the playback time of the corresponding audio / moving image from the acquired bar number.

In the present embodiment, as shown in FIG. 4 above, the audio / moving image reproduction time 411 and the measure number 421 of the score are associated with each other in advance, and the measure time conversion table 401 is created. The reproduction time can be calculated by referring to this measure time conversion table 401. The audio / video playback time is the length of the elapsed playback time based on the first image of the audio / video, and the playback time of the last audio / image of the audio / video is the length of the entire audio / video.

When the score providing server 101 acquires the reproduction time from the measure number, it notifies the communication server 102 of the reproduction time as shown in FIG. 6 and instructs the communication server 102 to reproduce the voice / moving image. In the communication server 102, the voice / video management module 304 manages the score data managed by the score providing server 101 in association with the voice / video of playing the music of the score, and the score providing server 101 manages the score. When playback is instructed by designating the playback start position (for example, playback time t4), the voice / video management module 304 identifies the voice / video distribution server 104 that distributes the corresponding voice / video, and the start position designation module 308. Directs the audio / video distribution to the audio / video distribution server 104 specified by the arrow 604 in FIG. 6 by designating the playback start position. The audio / video distribution server 104 instructed to distribute the audio / video distributes the designated audio / video from the designated playback position t4 to the terminal 111 as shown by the arrow 607 in FIG. Generally, the audio / video distribution server is provided with an interface for data acquisition and operation according to the type thereof, and although not described in detail in the present embodiment, the playback position is performed by using an interface known in the present technical field. To start playback.

(Sheet music highlight display processing)
Hereinafter, the score highlight display process of the present embodiment will be described with reference to FIG. 7. As described above, in the present embodiment, the MusicXML format data is managed by the score providing server 101, transmitted to the terminal 111, and displayed as a score on the browser. At this time, in the present embodiment, the position on the score corresponding to the audio / moving image reproduced on the terminal 111 is highlighted on the terminal 111. As a result, it is possible to know the part of the performance currently being performed by voice / video in units of measures of the score, which can be useful for checking the performance and practicing.

More specifically, the score highlight display process will be described. First, as shown by the arrow 703 in FIG. 7, the audio / video is played by streaming distribution from the audio / video distribution server 104. In this embodiment, the URL of the posting site and, if necessary, the playback time are set using the API of the posting site and played back. In such a state, the reproduction time output module 309 of the communication server 102 requests the reproduction time from the audio / video distribution server 104 as shown by the arrow 702 in FIG. 7, and acquires it as shown by the arrow 704 in FIG. Generally, the audio / video distribution server is provided with an interface for data acquisition and operation according to the type thereof, and although not described in detail in the present embodiment, the interface known in the present technical field is used at present. Get the playback time.

When the reproduction time output module 307 acquires the reproduction time, the reproduction time output module 307 notifies the score providing server 101 of the reproduction time. The score providing server 101 acquires the bar number as the position of the corresponding score by using the bar time conversion table 401 shown in FIG. 4 based on the notified playback time. Although not shown, if the reproduction time t acquired in this example is t2 or more and less than t3, it is determined by the measure time conversion table 401 that it corresponds to the second measure. In this way, the corresponding bar number is determined depending on which time interval range of the playback time in the bar time conversion table 401 the acquired playback time is included.

When the bar number is determined, the score providing server 101 instructs the terminal 111 to highlight the determined bar number, and the terminal 111 highlights the second bar. Regarding the specific processing, the highlight display can be controlled by any method known in the present technology. For example, the score providing server 101 changes the display of the score of the determined bar number to highlight. By sending to the terminal 111, the browser of the terminal 111 can interpret this and display the highlight, or when the server receives an instruction to switch to the highlight display, the specified part is highlighted. By embedding a script that changes to not.

By the above processing, the position on the score corresponding to the performance of the currently played voice / image is determined in bar units, and the determined bar is highlighted. Here, the communication server 102 performs the above-mentioned acquisition of the reproduction time at regular intervals and performs the highlight display processing after that, so that the corresponding measure is highlighted according to the reproduction of the audio / moving image. , Can be displayed as if moving on the score. By making the fixed interval sufficiently smaller than the time between each bar, the highlight can be displayed so as to move smoothly on the score. Alternatively, even if a fixed interval is not used, in the present embodiment, the reproduction time corresponding to each measure is stored in advance as shown in FIG. 4, so that the notification server 102 actually sets the reproduction time at each reproduction time. It is possible to confirm with the audio / video distribution server 104 whether or not there is, and execute a process of changing the bar to be highlighted.

In the present embodiment, the playback position of the moving image is highlighted, but the present invention is not limited to this, and it can be indicated by some marker or displayed by some method for distinguishing the measure from other measures.

(First Embodiment)
In the present embodiment, as described above, the voice / video distributed from the voice / video distribution server 104 is simultaneously played on a plurality of terminals, and each performer plays according to the voice / video, and the performance thereof. Is recorded with a microphone or a video camera, and the data is uploaded to the audio / video distribution server 104. The upload destination is not limited to the audio / video distribution server 104, but any server known in the present technical field. You can also. With reference to FIGS. 8 and 9, an online ensemble support process using audio and video when the communication delay of the present embodiment is small will be described. Here, the communication delay refers to a time delay (delay) from when the performer's voice / video is uploaded to the distribution server by streaming to when it is streamed to the performer's terminal. This is a process when there is no big difference in the communication delay of the terminals, and is a basic process of the present invention. That is, in the present embodiment, it is assumed that the time of each terminal is accurately matched by the Internet time setting without considering the communication delay, and the performance start time of each performer can be set to the same time. ..

Hereinafter, the present embodiment will be described with reference to FIGS. 8 to 10. FIG. 8 is a diagram showing a flow of a series of services when playing back a recorded voice of one embodiment of the present invention to support an online ensemble, and FIG. 9 is a diagram showing a recorded moving image of the present embodiment. It is a figure which shows the flow of a series of services at the time of playing and supporting an online ensemble. FIG. 10 is a flowchart showing a process of reproducing the recorded data of the present embodiment to support an online ensemble. First, the score is distributed from the score providing server 101 to each terminal 811 to 814 or 911 to 914, and the bar number and time for starting the performance are set from any of the performer's terminals 811 to 813 or 911 to 913 (step 1001). ). The measure numbers and times set in the performer's terminals 811 to 813 or 911 to 913 are the same. At the performance start time T1, the highlighting of the score starts, and each performer 1 to 3 starts playing in accordance with the highlight display (step 1002). The highlight display can be executed based on the tempo described in the score, the tempo of the performer posted on the video posting site, or the tempo when the same performer plays in advance.

When the performance is started, the voice or video is acquired by the microphones 801 to 803 or the cameras 901 to 903 for recording the performance of each performer, and the score providing server 101 notifies the voice / video distribution server 104 of the start of the performance. , The acquired performance data is upstreamed to the audio / video distribution server 104 (step 1003). The uploaded audio / video of the performance is streamed to the terminals 811 to 813 or 911 to 913 of each performer via the audio / video distribution server 104 (step 1004). In the case of voice data, the voices played by other performers are synthesized and output by a speaker or the like (not shown) of the terminals 811 to 813, but as shown in FIG. 8, a time bar or the like is displayed for each voice and other performances are performed. It is also possible to visually display the progress of the performer's performance. Generally, the voices of other performers to be delivered are more or less out of sync with their own performance due to communication delays, so only the performances of other performers are output, but not limited to this, their own performances. It is also possible to synthesize and output the performance.

Further, in the case of a moving image, a moving image of the performance of another performer is displayed as shown on the terminals 911 to 913, and if the communication delay is sufficiently small, the performance can be performed with the other performer at almost the same timing, as if the same place. You can get the feeling of playing together at. That is, the terminal 911, which is the terminal of the performer 1, displays the moving images of the performances of the performers 2 and 3, and the terminal 912, which is the terminal of the performer 2, displays the moving images of the performances of the performers 1 and 3. The terminal 913, which is the terminal of the performer 3, displays the moving images of the performances of the performers 1 and 2. Of course, you can also display your own performance video.

If the network delay is not small, it may occur as a time lag between the player on the terminal and the voice / video of another performer. In this case, mute the voice of the other performer. It is possible to do. Further, in the present embodiment, the audio / video of each performer is simultaneously distributed separately, but the present invention is not limited to this, and the audio / video distribution server 104 mixes the audio / video of each performer into one audio / video. It can also be delivered as. In addition to distribution to each performer's terminal, the audio / video of each performer shall be simultaneously distributed from the audio / video distribution server 104 to the non-performing viewer terminals 814 and 914 according to the performance start time. Can be done (step 1005). As described above, when the performances by each performer of the present embodiment are viewed on the terminals 813 and 914, the timings match, and even though each performer is playing remotely, the performances are performed at the same place. You can get the feeling of being there.

(Second Embodiment)
This embodiment is different from the first embodiment described above, and includes processing when the network environment is bad and the communication delay is different in each terminal, but the other processing is basically the same as that of the first embodiment. That is, when the communication delay is different between each terminal and the server, the delivery timing is different for each terminal, so it is necessary to adjust the communication delay. Therefore, by acquiring the delay time for each terminal in advance and providing a time difference at the start of the performance, it is possible to perform the performance in time so that the communication delay of each terminal is offset. Specifically, the audio / video uploaded to the audio / video distribution server 104 is delayed by a time determined based on the communication delay time acquired for each terminal, so that the communication delay is offset by each terminal. By shifting the performance start time with, it is possible to upload to the audio / video distribution server 104 almost at the same time. The time to start playing at each terminal can be delayed by the difference in communication delay with that terminal with reference to the terminal having the largest communication delay. Also, due to the characteristics of the system, it may be effective to start playing with a delay of the time obtained by multiplying the difference by a certain ratio, but in that case, the ratio is one of those known in the art. It can be determined by the method.

With reference to FIGS. 11 and 12, an online ensemble support process will be described using audio and moving images when the communication delay of the present embodiment differs depending on the terminal. FIG. 11 is a diagram showing a flow of a series of services when playing back a recorded voice of another embodiment of the present invention to support an online ensemble, and FIG. 12 is a diagram showing a recorded moving image of the present embodiment. It is a figure which shows the flow of a series of services when supporting an online ensemble by playing back. FIG. 13 is a flowchart showing a process of reproducing the recorded data of the present embodiment to support an online ensemble. First, the score is distributed from the score providing server 101 to each terminal 811 to 814 or 911 to 914, and the bar number and time of the performance start are set from any of the terminals 811 to 813 or 911 to 913 (step 1301). In the example shown in FIGS. 11 and 12, the difference between the communication delay of the terminal 811 or 911 and the terminal 813 or 913 is 6 seconds, and the difference between the communication delay of the terminal 812 or 912 and the terminal 813 or 913 is 3 seconds. Needless to say, it is possible to set a smaller delay or a larger delay. The communication delay can be measured in advance by any method known in the art and used, or it is automatically measured for each terminal and the terminals 812 to 813 or 912 in consideration of the communication delay. It is also possible to automatically set the performance start times T2 and T3 of ~ 913.

At the performance start time T1 of the terminal 811 or 911, the highlight display of the score starts, the performer 1 plays in accordance with the performance start time, and the performance voice / video is uploaded to the voice / video distribution server 104 (step 1302). ). When the performance is started, the voice or video is acquired by the microphone 801 or the camera 901 for recording the performance of each performer, and the music score providing server 101 notifies the voice / video distribution server 104 of the start of the performance, and the acquired performance is performed. The data is upstreamed to the audio / video distribution server 104.

Next, at the performance start time T2 of the terminal 812 or 912 (that is, 3 seconds after the start of the performance of the terminal 811 or 911), the highlighting of the score starts, and the performer 2 plays in accordance with the performance voice. -The video is upstreamed to the audio / video distribution server 104 (step 1303). When the performance is started, the voice or video is acquired by the microphone 802 or the camera 902 for recording the performance of each performer, and the music score providing server 101 notifies the voice / video distribution server 104 of the start of the performance, and the acquired performance is performed. The data is upstreamed to the audio / video distribution server 104. Here, since the audio / video of the performer 1 is synchronized with the highlight display of the terminals 812 and 912, it is possible to rely on it for the performance.

Further, at the performance start time T3 of the terminal 813 or 913 (that is, 6 seconds after the start of the performance of the terminal 811 or 911), the highlight display of the musical score starts, and the performer 3 plays in accordance with the performance voice. The video is upstreamed to the audio / video distribution server 104 (step 1304). When the performance is started, the voice or video is acquired by the microphone 803 or the camera 903 for recording the performance of each performer, and the music score providing server 101 notifies the voice / video distribution server 104 of the start of the performance, and the acquired performance is performed. The data is upstreamed to the audio / video distribution server 104. Here, since the audio / video of the performers 1 and 2 is synchronized with the highlight display of the terminals 813 and 913, the uploaded performance audio / video may be played by relying on it. The audio / video of each performer is simultaneously transmitted from the audio / video distribution server 104 to the viewer terminals 814 and 914 that do not perform via the server 104 according to the performance start time of the last performer (performer 3). Delivered (step 1305). In this way, even if the network environment is different for each terminal, when the performance by each performer of the present embodiment is viewed on the terminals 814 and 914, the timing is matched and each performer is playing remotely. Nevertheless, you can get the feeling of playing in the same place.

(Other aspects)
The present invention includes the following scope in addition to those described in the claims.

Aspect 1
A recorded data playback system that supports online ensembles via a network.
A musical score transmission means for transmitting musical scores to terminals via a network,
A reproduction means for reproducing the recorded data of the first performance related to the score from a designated position on the score displayed on the terminal, and a reproduction means.
A performance acquisition means for acquiring a second performance related to the score, which is performed on top of the first performance reproduced by the reproduction means, on a plurality of terminals and transmitting the second performance via a network.
A recording data reproduction system, which comprises an output means for outputting a plurality of second performances transmitted by the performance acquisition means together with the reproduced first performance.

Aspect 2
When the designation of the start position of the performance is received from the terminal, the playback time of the data recording the first performance corresponding to the designated position on the score is set, and the data recording the score and the first performance. The first aspect is characterized by further comprising a reproduction step of acquiring data according to the correspondence information generated from the above and transmitting the data to the terminal in order to reproduce the data recorded with the first performance from the acquired reproduction time. The recorded data reproduction method described.

Aspect 3
The reproduction step includes a moving image posting step of transmitting the first performance played in relation to the score from the terminal to a music providing server and receiving identification information of data recording the first performance. The recorded data reproduction method according to the second aspect.

Aspect 4
The music providing server is accessed from the terminal, the data recording the first performance played in connection with the score is selected, and the identification information of the data recording the selected first performance is used. The recorded data reproduction method according to aspect 3, further comprising a moving image selection step for receiving.

Aspect 5
A program that causes a computer to execute a recorded data reproduction method that supports an online ensemble via a network, and the recorded data reproduction method is
The score transmission step to send the score to the terminal via the network,
A reproduction step of reproducing the recorded data of the first performance related to the score from a designated position on the score displayed on the terminal and displaying the reproduction position on the score.
A performance acquisition step of acquiring a second performance related to the score on a plurality of terminals and transmitting it via a network, which is performed on top of the first performance reproduced in the reproduction step.
A program comprising: an output step of outputting a plurality of second performances transmitted in the performance acquisition step together with the reproduced first performance.

The invention according to claim 1 is a recorded data reproduction method that supports an online ensemble via a network, a score transmission step of transmitting a score to a plurality of terminals via the network, and a first score-related. The recorded data of the performance of is played back from the specified position on the score displayed on the terminal, and is superimposed on the playback step of displaying the playback position on the score and the first performance played in the playback step. The performance acquisition step of recording the second performance related to the score as data on each of the plurality of terminals and transmitting it via the network, and the plurality of second performances transmitted in the performance acquisition step are reproduced. It is equipped with an output step that outputs along with the first performance that has been performed, and when the performance start position and playback start time settings are received from one of multiple terminals, playback starts from the position corresponding to the specified position on the score. Further provided with a transmission step of acquiring the data in which the first performance is recorded so as to be played back at the time according to the correspondence information generated from the score and the data in which the first performance is recorded and transmitting it to a plurality of terminals. The reproduction step is characterized in that the reproduction starts from a designated position on the score at a set reproduction start time .

The invention according to claim 2 is characterized in that, in the recorded data reproduction method according to claim 1, the data in which the first performance is recorded is audio data obtained by recording the previous performance. ..

The invention according to claim 3 is the recorded data reproduction method according to claim 1 or 2, wherein the data recording the first performance is moving image data obtained by recording the first performance. It is characterized by.

The invention according to claim 4 is characterized in that, in the recorded data reproduction method according to any one of claims 1 to 3, the reproduction step starts reproduction at a different reproduction start time for each terminal.

The invention according to claim 5 is a program for causing a computer to execute a recorded data reproduction method for supporting an online ensemble via a network, and the recorded data reproduction method transfers a score to a plurality of terminals via the network. The score transmission step to be transmitted and the data recording the first performance related to the score are played back from the designated positions on the score displayed on a plurality of terminals, and the playback position is displayed on the score. A performance acquisition step of recording a second performance related to a score as data on each of a plurality of terminals and transmitting it via a network, which is performed on top of the first performance played in the step and the playback step. It includes an output step that outputs a plurality of second performances transmitted in the performance acquisition step together with the reproduced first performance, and receives a performance start position and a playback start time setting from one of the plurality of terminals. Then, the data in which the first performance is recorded so as to be played from the position corresponding to the designated position on the score at the playback start time is associated with the score and the data in which the first performance is recorded. It further comprises a transmission step of acquiring according to information and transmitting it to a plurality of terminals, and the reproduction step is characterized in that the reproduction is performed from a designated position on the score at a set reproduction start time.

According to the present invention, it is a recording data reproduction method that supports an online ensemble via a network, and records a score transmission step of transmitting a score to a plurality of terminals via the network and a first performance related to the score. The score is played back from the specified position on the score displayed on the terminal, and is superimposed on the playback step of displaying the playback position on the score and the first performance played in the playback step. A performance acquisition step in which the second performance related to the above is recorded as data on each of the plurality of terminals and transmitted via the network, and a plurality of second performances transmitted in the performance acquisition step are reproduced. It is equipped with an output step that outputs along with the performance of, and when the performance start position and playback start time settings are received from one of multiple terminals, the performance starts from the position corresponding to the specified position on the score. As described above, the reproduction step further includes a transmission step of acquiring the data recording the first performance according to the correspondence information generated from the score and the data recording the first performance, and transmitting the data to a plurality of terminals. Since the music is played from the specified position on the score at the set playback start time, the effect of network delay can be achieved by synthesizing the sound played along with the recorded playback or displaying the videos side by side. It is possible to support an online ensemble between performers in remote areas where you can get a realistic experience by reducing as much as possible.

Claims (5)

A recorded data playback method that supports online ensembles via a network.
The score transmission step to send the score to the terminal via the network,
A reproduction step of reproducing the recorded data of the first performance related to the score from a designated position on the score displayed on the terminal and displaying the reproduction position on the score.
A performance acquisition step of acquiring a second performance related to the score on a plurality of terminals and transmitting it via a network, which is performed on top of the first performance reproduced in the reproduction step.
A method for reproducing recorded data, comprising: an output step of outputting a plurality of second performances transmitted in the performance acquisition step together with the reproduced first performance. When the designation of the start position of the performance is received from the terminal, the playback time of the data recording the first performance corresponding to the designated position on the score is set, and the data recording the score and the first performance. The first aspect of the present invention is further comprising a reproduction step of acquiring data according to the correspondence information generated from the above and transmitting the acquired reproduction time to the terminal in order to reproduce the data recorded with the first performance. Recorded data reproduction method described in. The recorded data reproduction method according to claim 1 or 2, wherein the data obtained by recording the first performance is audio data obtained by recording the first performance. The recorded data reproduction method according to any one of claims 1 to 3, wherein the data obtained by recording the first performance is moving image data obtained by recording the first performance. The recorded data reproduction method according to any one of claims 1 to 4, wherein the reproduction step starts reproduction at a reproduction start time different for each terminal.

Images