KR102294752B1 - Remote sound sync system and method - Google Patents

Abstract

Disclosed is a remote sound synchronization system to provide the same effect as when musicians of each instrument play in an ensemble at the same time while playing with a time difference at different times. To this end, the remote sound synchronization system comprises: a network communication unit connected to a plurality of user terminals through a network and checking latency with each user terminal; and a time difference synchronization unit determining a time interval of performance data to be transmitted from the preceding terminal to the succeeding terminal with reference to a latency between the preceding terminal, which first generate the performance data among the user terminals, and the succeeding terminal, combining succeeding performance data generated in the succeeding terminal with preceding performance data to generate and provides ensemble data to the user terminal of the next performer, thereby allowing each user to perform synchronized performances in each user terminal.

Description

Remote sound sync system and method

The present invention relates to a remote sound synchronization system and method in which a plurality of performers sequentially receive performance data of a previous player to generate synchronized ensemble data.

As a form of performance in which several instruments are played together, an ensemble can realize a sound different from that of a solo performed with a single instrument because the unique tones of each instrument match. Unlike solo ensemble, performance results are not derived from the sum of individual skills. In an ensemble, it is very important for each performer to harmonize with the other performers, otherwise the notes of each performer will harmonize with each other and are easily disturbed. An ensemble is the art of timing, pitch, and mutual timing between performers.

For such an ensemble, many music lovers gather together on a set day with their respective instruments to practice and match the breath of the ensemble. One of the most important things for these music enthusiasts is that the members gather at a certain time regardless of whether their skills are good or bad. because it wins Performance skills are often referred to as secondary cases.

However, it is not easy to make an appointment and get together at the same time due to the personal schedules and work of several club members, and the social situation where people are restricted from gatherings due to infectious diseases such as Corona 19 makes it difficult to practice or perform ensembles. do. About this,

Republic of Korea Patent No. 10-1222711 proposes a "server providing an ensemble game" that receives performance data excluding only the part in charge of the performer among a plurality of performance parts, and obtains the effect of the ensemble by playing the part where the player is excluded. there is a bar However, in Patent Registration No. 10-1222711, since predetermined performance data exists, it is impossible to implement an ensemble in which several performers actually perform in sync with each other.

In addition, Republic of Korea Patent Registration No. 10-1133906 discloses that the ensemble participant can easily participate in the ensemble by extracting only what the ensemble participant wants out of the performance data of each instrument participating in the ensemble and providing it to the ensemble participant, and provides performance data for each instrument. A "method and system for providing an ensemble performance service" has been proposed to create an orchestra atmosphere by allowing each terminal to play.

However, Patent Registration No. 10-1133906 provides a MIDI file recorded in a database to each user terminal, but extracts and provides only the sound source of a specific instrument to the ensemble participant, so that the performers feel that they are actually performing an ensemble rather than a feeling recorded in the sheet music. There is a big aspect of playing along. It is overlooked that the actual ensemble is a field that requires teamwork with other performers, rather than determining the good or bad skills of an individual performer.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a remote sound synchronization system that enables a plurality of network-connected performers to perform in a form closest to an actual ensemble.

Another object of the present invention is to provide a remote sound synchronization system that minimizes the influence of network delay or traffic by allowing a plurality of performers to perform an asynchronous ensemble rather than an ensemble at the same time while performing an ensemble.

The above object is, according to the present invention, a network connection with a plurality of user terminals, a network communication unit for determining the latency with each user terminal, and a preceding terminal and a subsequent terminal for generating performance data first among the user terminals Determine the time interval of the performance data to be transmitted from the preceding terminal to the succeeding terminal with reference to latency, and combine the following performance data generated in the succeeding terminal with the preceding performance data to obtain the ensemble data This is achieved by a remote sound synchronization system including a time difference synchronizer that generates and provides to a user terminal of a next player, so that each user performs a synchronized performance in the respective user terminals.

According to the present invention, the musicians for each instrument can implement an effect similar to that of performing an ensemble at the same time period while performing the time difference at different times. Each performer has a sequential and asynchronous performance form that adds what he or she has performed to the performance data transmitted from the previous player, and the effect of performing an ensemble at the same time by listening to and listening to the progress or results of other players Implement time difference synchronization to obtain Through this, as mentioned in the prior art, the problem of mutual timing and breathing between the players pursued by the ensemble is solved.

1 shows a conceptual diagram between terminals having a remote sound synchronization system according to an embodiment of the present invention.
2 is a conceptual diagram of a remote sound synchronization system according to an embodiment of the present invention.
3 is a conceptual diagram illustrating a method of setting a performance timing for each player in a remote sound synchronization system according to an embodiment of the present invention.
4 is a conceptual diagram illustrating a method for synchronizing network latency and time difference for packets and a recording time setting method in a remote sound synchronization system according to an embodiment of the present invention.
FIG. 5 is a reference diagram for explaining a processing method according to an example of processing a delayed-delivered packet in a remote sound synchronization system according to an embodiment of the present invention.
FIG. 6 is a reference diagram for explaining a processing method according to another example of processing a delayed-delivered packet in a remote sound synchronization system according to an embodiment of the present invention.
FIG. 7 is a reference diagram for explaining a processing method according to another example of processing a delayed-delivered packet in a remote sound synchronization system according to an embodiment of the present invention.
FIG. 8 is a reference diagram for explaining a processing method according to another example of processing a delayed-delivered packet in a remote sound synchronization system according to an embodiment of the present invention.

The user terminal described in this specification may refer to any one of various terminals capable of communicating with a smart phone, a notebook computer, a personal computer (PC), and other networks, and having a speaker and a microphone.

As used herein, “latency” may refer to either a response delay between the remote sound synchronization system and the user terminal according to the present invention, or a response delay occurring between the user terminals. The latency between the user terminal and the remote sound synchronization system may refer to a value typically obtained by a ping test.

It should be understood that the preceding terminal and the following terminal described in this specification are distinguished according to the order of performance among the user terminals. The preceding terminal may refer to a terminal that starts playing before a subsequent terminal to generate performance data and transmits the performance data to the remote sound synchronization system according to the present invention.

In addition, there may be one, two, three or more of the following terminals.

Hereinafter, the present invention will be described in detail with reference to the drawings.

FIG. 1 shows a relationship between a plurality of terminals, and FIG. 2 shows a conceptual diagram of a remote sound synchronization system according to an embodiment of the present invention.

Referring to FIG. 2 , the remote sound synchronization system 100 according to the embodiment is network-connected to a plurality of user terminals 10a to 10n, and includes a network communication unit 110 , a time difference synchronization unit 120 and a buffer 130 . can be configured.

The network communication unit 110 measures latency by performing a ping test with the preceding user terminals 10a to 10(n-1). The measurement of latency is measured between the user terminals 10a to 10n.

The latency (1 to n-1) refers to the latency between the preceding user terminal (any one of 10a to 10(n-1)) and the following user terminal (the other one of 10(a+1) to 10n).

For example, the latency k corresponds to a measured value measured between the preceding terminal 10k and the succeeding terminal 10(k+1) at k where a<=k<n-1.

In addition, the network communication unit 110 may measure the latency between the preceding terminal and the following terminal to generate performance data following the preceding terminal under the control of the time difference synchronizer 120 .

For example, assuming that the preceding terminal is referenced "10a" and the following terminals are referenced "10b", "10c" and "10d",

3-1) latency between the preceding terminal 10a and the succeeding terminal 10b;

3-2) latency between the trailing terminal 10b and the trailing terminal 10c, and

3-3) Each of the latencies between the trailing terminal 10c and the trailing terminal 10d is measured.

The time difference control unit 120 calculates the latency (item 3-1, item 3-2, and item 3-3) between the preceding terminal and the succeeding terminal (this terminal) in the network communication unit 110 for the performance song to be performed with the ensemble. Then, with reference to this, it is possible to determine a time interval (Interval) of the performance data to be transmitted from the preceding terminal 10a to the succeeding terminal 10b.

The time interval may mean a time difference when performance data is transmitted from the preceding terminal 10a to the remote sound synchronization system 100 according to the embodiment, buffered in the buffer 130, and then transferred to the succeeding terminal 10b have. The time difference control unit 120 delays the performance data transferred from the preceding terminal 10a to the succeeding terminal 10b by the time interval between the preceding terminal 10a and the succeeding terminal 10b.

Similarly, when the performance data is transferred between the trailing terminal 10b and the trailing terminal 10c, the time difference control unit 120 is the performance data by the time interval of the performance data transmitted from the trailing terminal 10b to the trailing terminal 10c (or ensemble data) may be delayed.

Here, looking only at the relationship between the trailing terminal 10b and the trailing terminal 10c,

The following terminal (10b) -> corresponding to the preceding terminal compared to the following terminal (10c),

The following terminal 10c -> corresponds to the following terminal compared to the following terminal 10b.

The relationship between the preceding terminal and the succeeding terminal is that the side transmitting the performance data (or ensemble data, hereinafter omitted) corresponds to the preceding terminal, and the side receiving the transmitted performance data corresponds to the following terminal. However, in the present specification, for convenience of explanation and understanding, reference numeral "10a" is referred to as a preceding terminal, and the remaining user terminals 10b to 10n are referred to as subsequent terminals, and user terminals belonging to reference numerals 10b to 10n are also , it should be noted that, according to each data transfer order, it can be divided into a preceding terminal and a succeeding terminal. The relationship between the preceding terminal and the following terminal relates to the relationship between the transmitting side and the receiving side of the performance data, and this is the same throughout this specification, and the redundant description will be omitted below.

The time difference synchronizer 120 of the trailing terminal 10b receives the performance data of the performer transmitted from the preceding terminal 10a, fills it in the buffer 130, and reproduces the performance data after a delayed time according to the time interval, so that the performer make it audible The trailing terminal 10b reproduces the performance data, and the user of the trailing terminal 10b can perform the performance according to the part to be played while listening to the reproduced performance data. At this time, the following player recognizes the listened performance data in real time and performs his/her own performance in accordance with the timing and breathing, and the time difference synchronization unit of the trailing terminal 10b receives the following performance data of the following player.

The synthesizing unit 140 may generate ensemble data by combining the following performance data provided to the trailing terminal 10b with the performance data generated by the preceding terminal 10a.

The generated ensemble data is transmitted to the trailing terminal 10c according to the next sequence, and the time difference synchronizer 120 of the trailing terminal 10c likewise fills the buffer 130 and plays it, and the trailing performance generated by the trailing terminal 10c After receiving the ensemble data, the data is combined in the synthesizing unit 140 to generate secondary ensemble data. In this way, the time difference synchronization unit 120 obtains performance data and subsequent performance data from the user terminals 10a to 10n of all players related to one performance piece, and sequentially performs the subsequent performance data to the preceding performance data according to the performance order. , and the final ensemble data may be generated through the synthesizing unit 140 .

Here, since the ensemble data is a method of adding the preceding performance data transmitted from the succeeding terminal 10b to the preceding performance data generated by the preceding terminal 10a, the ensemble data provided to each user terminal as described below has a form of gradual accumulation.

- under -

4-1) User terminal 10a -> User terminal 10b: performance data of the preceding terminal 10a

4-2) User terminal 10b -> User terminal 10c: performance data of the preceding terminal 10a + performance data of the following terminal 10b

4-3) User terminal 10c -> User terminal 10d: performance data of the preceding terminal 10a + performance data of the succeeding terminal 10b + performance data of the succeeding terminal 10c

Here, referring to items 4-1) to 4-3), it can be seen that the preceding performance data or ensemble data transmitted to each user terminal is sequentially transmitted to each user terminal 10b to 10n at different times.

The time difference synchronizer 120 buffers the performance data (or ensemble data) received from the network communication unit 110 in the buffer 130 .

5-1) The playback time interval of the performance data (or ensemble data) buffered in the buffer 130 is proportional to the maximum value of the latency between the preceding terminal 10a and the following terminal 10b, and in the present invention, This is called a time interval. In the first embodiment of the present invention, 1.5 times the maximum latency value was set as the time interval.

5-2) In the second embodiment Latency at the maximum latency standard deviation plus 1.2 The ship was set as the time interval.

5-3) In the third embodiment Latency When the standard deviation exceeds the set value, the total playing time was used as the time interval.

The time difference synchronizer 120 of the succeeding terminal 10b may play the performance data transmitted from the preceding terminal 10a after buffering it in the buffer 130 with reference to the time interval. Accordingly, the succeeding terminal 10b does not cause a break in the performance data transmitted from the preceding terminal 10a, and the player on the trailing terminal 10b side can fully perform the ensemble while listening to the playback sound for the performance data.

The time interval between the trailing terminal 10b and the trailing terminal 10c is also calculated as described in item 5-1) or 5-2) or 5-3), provided that, in this case, the trailing terminal 10b is the preceding terminal. There is only a difference that is specified and calculated by designating the succeeding terminal 10c as the succeeding terminal.

3 is a conceptual diagram illustrating a method of setting a performance timing for each player in a remote sound synchronization system according to an embodiment of the present invention. The description of FIG. 2 will be described with reference to FIG. 1 .

Referring to FIG. 3 , the remote sound synchronization system 100 measures latency with the preceding terminal 10a. If it is assumed that the latency is 30 ms, the performance data generated from the point in time when the preceding player who owns the preceding terminal 10a starts recording is transmitted to the remote sound synchronization system 100, and after buffering the 30 ms latency, the succeeding terminal (10b). The succeeding terminal 10b reproduces the performance data transmitted from the preceding terminal 10a so that the following player can listen, and the succeeding player performs his/her performance according to the timing and breathing required for the ensemble while listening to the reproduced performance data. proceed The performance of the following player is recorded by a recording function provided in the trailing terminal 10b, and the recorded performance data is provided to the remote sound synchronization system 100 . Here, the performance data of the preceding player and the following player may be mixed to form one ensemble data. The formed ensemble data is transmitted to the next following terminal 10c, and the trailing terminal 10c generates its own performance data while listening to the reproduced sound for the ensemble data, similarly to the following terminal 10b, and uses it in the remote sound synchronization system 100. forward to

4 is a conceptual diagram illustrating a method for synchronizing network latency and time difference for packets and a recording time setting method in a remote sound synchronization system according to an embodiment of the present invention.

Referring to FIG. 4 , the remote sound synchronization system 100 according to an embodiment of the present invention periodically checks the latency with the preceding terminal 10a and provides it to the remote sound synchronization system 100, but according to the latency, the following terminal Instead of determining the time interval (Initial Latency) of (10b), an example of applying a time interval in which a buffering time is added to the time interval (Safe Record delay) is shown. The reason that the time interval (Safe Record delay) is set longer than the time interval (Initial Latency) according to the latency is the remote This is to secure time for the sound synchronization system 100 to cope with this.

5 to 8 are reference views for explaining a processing method according to an example of processing a delayed-delivered packet in a remote sound synchronization system according to an embodiment of the present invention.

Referring to FIG. 5 , packets 1, 2, and 3 of the packets constituting the performance data transmitted from the preceding terminal (P1) to the following terminal (P2) are delivered, but packets 4, 5, and 6 are delayed. An example is shown.

When the latency of packets 4, 5, and 6 exceeds the time interval set by more than the latency measured for the first leading terminal P1, the trailing terminal P2 records the performance data of the trailing player, and the remote sound synchronization system 100 ) is in the process of being transmitted. In this state, it is difficult to set a new time interval between the preceding terminal (P1) and the succeeding terminal (P2), and the remote sound synchronization system 100 solves this problem by repeatedly playing notes corresponding to the delayed-delivered packet. can be solved

The notes being repeatedly reproduced correspond to packets 4, 5, and 6, and FIG. 4 exemplifies the note corresponding to the “re” note. After packet 6, when a packet for new performance data arrives from the preceding terminal P1, it is transmitted to the following terminal P2, and if new performance data does not arrive, the remote sound synchronization system 100 regards it as a network error and You may also notify us of service interruption.

Referring to FIG. 6 , the remote sound synchronization system 100 according to an embodiment may acquire a sheet music for performing a concerto and respond to latency occurrence based thereon.

Among the packets transmitted from the preceding terminal P1 to the remote sound synchronization system 100, when the packet in the S4 area does not arrive within the time interval, the remote sound synchronization system 100 adjusts the pitch and score of the non-arrival packet belonging to the S4 area. It is possible to compare the registered pitches and reconstruct the packet of the S4 area with the pitch (B3) corresponding to the S4 area. In the drawing, the remote sound synchronization system 100 may remove the non-arrival packet of the S4 area, reconfigure it into a new packet including the pitch (B3) information in its place, and transmit it to the following terminal P2. In this case, the trailing terminal P2 does not recognize the pitch omission due to the non-arrival packet, and the entire ensemble may proceed naturally.

The embodiments of the present invention disclosed in the present specification and drawings are merely presented as specific examples to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. It will be apparent to those of ordinary skill in the art to which the present invention pertains that other modifications based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein.

10 (10a to 10n): user terminal
100: remote sound synchronization system
110: network communication unit
120: time difference synchronization unit
130: buffer
140: synthesis part

Claims (7)

The preceding terminal and the succeeding terminal are linearly included in a plurality of user terminals sequentially connected to the network,
a network communication unit for determining latency with the preceding terminal; and
Each user terminal determines the time interval of the performance data to be transmitted from the preceding terminal to the succeeding terminal with reference to the latency between the preceding terminal and the succeeding terminal according to the order of generating the performance data, a time difference synchronization unit for receiving the trailing performance data generated by the trailing terminal; and
and a synthesizing unit for generating ensemble data by combining the following performance data with the performance data and providing it to a user terminal of the next performer, each of which is synchronized in each user terminal by the network communication unit, time difference synchronization unit, and synthesizing unit to perform the performance,
and a database including sheet music data of a song to be played in the preceding terminal.
The remote sound synchronization system, characterized in that the time difference synchronization unit corresponds to the generation of latency based on the score data of the database.
According to claim 1,
After buffering the performance data and the subsequent performance data received from the network communication unit according to the time interval, a buffer for providing the performance data to the time difference synchronizer; Remote sound synchronization system further comprising a. 3. The method of claim 2,
The network communication unit,
Measures a latency value with each user terminal every predetermined time and notifies the time difference synchronizer,
The time difference synchronization unit,
The remote sound synchronization system, characterized in that adjusting the buffering time of the buffer according to the latency value measured at each predetermined time. According to claim 1,
The time difference synchronization unit,
A remote sound synchronization system, characterized in that sequentially supplying the performance data to each of the user terminals. delete According to claim 1,
The time difference synchronization unit,
When the latency occurring between the preceding terminal and the network communication unit exceeds a preset reference latency, the data of the packet exceeding the reference latency is extracted from the sheet music data and applied to the performance data and provided to the following terminal A remote sound synchronization system, characterized in that. According to claim 1,
The time difference synchronization unit,
When the latency occurring between the preceding terminal and the network communication unit exceeds a preset reference latency, the pitch of the non-arrival packet is compared with the pitch registered in the score, and corresponds to the area of the non-arrival packet among the packets of the performance data received within the latency A remote sound synchronization system, characterized in that packet reconstruction is performed by adding a packet in the non-arrival packet area at a pitch of .

Images