JP2021028649A - Information processor and program - Google Patents

Abstract

To allow a plurality of players to remotely play in concert in accordance with an appropriate tempo while listening to mutual musical performance.SOLUTION: A generation unit 23 stores sound data acquired from a reference terminal specified by a specification unit 22 from a point of time at which performance starts, and extracts a tempo from the stored sound data. The tempo is extracted by analyzing the sound data acquired from the reference terminal, detecting a peak of sound pressure and calculating time between timings, for example. When the extracted tempo continuously n-times falls within a prescribed range with the tempo which is set when performance starts as a center, the generation unit 23 generates reference timing data with a value obtained by correction processing such as quantization on the extracted n-pieces of tempos as reference timing. A reference timing data transmission unit 24 transmits newly generated reference timing data to respective user terminals 10.SELECTED DRAWING: Figure 4

Description

The present invention relates to a technique for a plurality of performers to perform an ensemble remotely.

A feature of the 5th generation mobile communication system (so-called 5G) is low communication delay, and services that take advantage of this feature are being studied. As one of the services, a service that enables a plurality of users in remote locations to perform an ensemble in a timely manner via a network is being studied. For example, in Patent Document 1, a performance start signal is transmitted from a network-connected mixer to another mixer at a timing corresponding to the delay time of communication between the mixers, and thereafter, each mixer follows its own radio clock. A mechanism for notifying each performer of the tempo of the performance by generating a click sound is disclosed.

Japanese Unexamined Patent Publication No. 2005-128296

The tempo of the performance is predetermined on the score, but it may change slightly depending on the action that is the basis of the tempo (for example, playing a drum or conducting a conductor). Therefore, an object of the present invention is to provide a mechanism for a plurality of performers to perform an ensemble at an appropriate tempo while listening to each other's performances remotely.

In order to solve the above problems, the present invention includes an acquisition unit that acquires the sound data or the image data from a plurality of terminals that generate sound data indicating the played sound or image data indicating the operation for the performance. , A specific unit that specifies one of the plurality of terminals as a reference terminal for generating a reference timing when a performance is performed on the plurality of terminals, and a specific unit that is acquired from the reference terminal by the acquisition unit. A generation unit that generates reference timing data for presenting the reference timing to the performer at each terminal based on the sound data or the image data, and the generated reference timing data are generated by the plurality of the reference timing data. The reference timing data transmission unit to be transmitted to the terminal and a plurality of the sound data acquired by the acquisition unit from a plurality of terminals other than the one terminal are mixed with one terminal of the plurality of terminals. Provided is an information processing apparatus including a sound data transmitting unit for transmitting data.

The specifying unit may specify the reference terminal based on an instruction input to at least one of the plurality of terminals.

The specific unit may specify the reference terminal based on the sound data or the image data acquired by the acquisition unit from at least one of the plurality of terminals.

The image data may include image data that captures the movement of the performer who performs the performance, or image data that shows an image that captures the movement of the conductor who directs the performance.

In the first period of the period in which the performance is performed, the generation unit corrects the time interval extracted from the sound data or the image data acquired from the reference terminal by the acquisition unit. The reference timing data for presenting the reference timing is generated at the time interval, and in the second period, the time interval extracted from the sound data or the image data acquired from the reference terminal by the acquisition unit is set. Corresponding, reference timing data for presenting the reference timing may be generated at non-constant time intervals.

In the first period, the generation unit first generates reference timing data for presenting the reference timing at predetermined fixed time intervals, and then the acquisition unit generates the reference timing data from the reference terminal. The reference timing data for presenting the reference timing may be generated at a fixed time interval corrected for the time interval extracted from the acquired sound data or the image data.

The generation unit generates reference timing data in the second period when the time interval extracted from the sound data or the image data acquired from the reference terminal by the acquisition unit does not fall within a predetermined range. You may try to do it.

The specific unit specifies two or more of the terminals as the reference terminal, and the generation unit uses the reference unit based on the sound data or the image data acquired from the two or more reference terminals by the acquisition unit. Reference timing data for presenting the timing at each terminal may be generated.

In addition, the present invention comprises an acquisition unit that acquires the sound data or the image data from a plurality of terminals that generate sound data indicating the played sound or image data indicating the operation for the performance of the computer. A specific unit that specifies one of the plurality of terminals as a reference terminal for generating a reference timing when a performance is performed on the plurality of terminals, and the said unit acquired from the reference terminal by the acquisition unit. A generation unit that generates reference timing data for presenting the reference timing to the performer at each of the terminals based on the sound data or the image data, and the generated reference timing data are transmitted to the plurality of terminals. A plurality of the sound data acquired by the acquisition unit from a plurality of terminals other than the one terminal are mixed and transmitted to the reference timing data transmission unit to be transmitted and one of the plurality of terminals. Provide a program for functioning as a sound data transmitter.

According to the present invention, it is possible for a plurality of performers to perform an ensemble at the same timing while listening to each other's performances remotely.

It is a figure which shows an example of the structure of the ensemble system which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the hardware composition of a user terminal 10. It is a block diagram which shows an example of the hardware composition of the server apparatus 20. It is a block diagram which shows an example of the functional structure of the ensemble system 1. It is a flowchart which shows an example of the operation of the server apparatus 20. It is a figure explaining the time-series relationship between the sound data in a server apparatus 20 and the sound data in a user terminal 10a. It is a figure which illustrated the method of presenting the reference timing in each user terminal 10.

[Constitution]
FIG. 1 is a diagram showing an example of the ensemble system 1 of the present embodiment. The ensemble system 1 includes a plurality of user terminals 10a, 10b, 10c (hereinafter collectively referred to as user terminals 10) that function as terminals according to the present invention, such as a smart phone, a tablet, a wearable terminal, or a personal computer. A server device 20 that functions as an information processing device according to the present invention is provided. The network 2 connects the user terminal 10 and the server device 20 in a communicable manner. The network 2 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), or a combination thereof, and includes a wired section or a wireless section. The network 2 is preferably, but is not necessarily limited to, a network compliant with the 5th generation mobile communication system characterized by low latency. Although FIG. 1 shows three user terminals 10a, 10b, and 10c as user terminals, these may be four or more.

In the ensemble system 1, a plurality of users, each of whom is a performer, can perform timed musical instruments (that is, ensemble) via a network 2 at locations separated from each other. Each user who belongs to one group performing an ensemble places the user terminal 10 near himself / herself when playing various musical instruments such as a guitar, drums, bass, and piano, and activates a program implemented in the user terminal 10. To do. In the example of FIG. 1, it is assumed that one group for performing an ensemble is composed of three users who use the user terminals 10a, 10b, and 10c, respectively. When the performance of each user is started, each user terminal 10 collects the sound related to the performance and transmits the sound data indicating the sound to the server device 20 via the network 2. The server device 20 mixes the sound data transmitted from each user terminal 10 belonging to the same group and transmits it to another user terminal 10 belonging to the same group. Each user terminal 10 outputs a sound corresponding to the mixed sound data. As a result, each user can perform his / her own performance while listening to each other's performance. However, in communication via the network 2, there is a considerable delay. Therefore, the ensemble system 1 according to the present embodiment has the features as described below.

FIG. 2 is a diagram showing an example of the hardware configuration of the user terminal 10. The user terminal 10 is physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus connecting them, and the like. In the following description, the word "device" can be read as a circuit, a device, a unit, or the like. The hardware configuration of the user terminal 10 may be configured to include one or more of the devices shown in the figure, or may be configured not to include some of the devices.

For each function in the user terminal 10, the processor 1001 performs an operation by loading predetermined software (program) on hardware such as the processor 1001 and the memory 1002, and controls the communication by the communication device 1004, or the memory 1002. And by controlling at least one of reading and writing of data in the storage 1003.

Processor 1001 operates, for example, an operating system to control the entire computer. The processor 1001 may be configured by a central processing unit (CPU) including an interface with a peripheral device, a control device, an arithmetic unit, a register, and the like. Further, for example, a baseband signal processing unit, a call processing unit, and the like may be realized by the processor 1001.

The processor 1001 reads a program (program code), a software module, data, and the like from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to these. As the program, a program that causes a computer to execute at least a part of the operations described later is used. The functional block of the user terminal 10 may be realized by a control program stored in the memory 1002 and operating in the processor 1001. Various processes may be executed by one processor 1001, but may be executed simultaneously or sequentially by two or more processors 1001. Processor 1001 may be implemented by one or more chips. The program may be transmitted from the network 2 to the user terminal 10 via a telecommunication line.

The memory 1002 is a computer-readable recording medium, and is composed of at least one such as a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). May be done. The memory 1002 may be referred to as a register, a cache, a main memory (main storage device), or the like. The memory 1002 can store a program (program code), a software module, or the like that can be executed to carry out the method according to the present embodiment.

The storage 1003 is a computer-readable recording medium, and is, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, an optical magnetic disk (for example, a compact disk, a digital versatile disk, or a Blu-ray). It may consist of at least one (registered trademark) disk), smart card, flash memory (eg, card, stick, key drive), floppy (registered trademark) disk, magnetic strip, and the like. The storage 1003 may be referred to as an auxiliary storage device. The storage 1003 stores a performance evaluation program and a musical score data group described later.

The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like. Communication device 1004 includes, for example, a high frequency switch, a duplexer, a filter, a frequency synthesizer, etc. in order to realize at least one of frequency division duplex (FDD) and time division duplex (TDD). It may be composed of. For example, the transmission / reception antenna, the amplifier unit, the transmission / reception unit, the transmission line interface, and the like may be realized by the communication device 1004. The transmission / reception unit may be physically or logically separated from each other in the transmission unit and the reception unit.

The input device 1005 is an input device (for example, a key, a microphone, a switch, a button, a sensor, etc.) that receives an input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may have an integrated configuration (for example, a touch screen).

Each device such as the processor 1001 and the memory 1002 is connected by a bus for communicating information. The bus may be configured by using a single bus, or may be configured by using a different bus for each device.

Further, the user terminal 10 includes hardware such as a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). It may be configured, and the hardware may realize a part or all of each functional block. For example, processor 1001 may be implemented using at least one of these hardware.

FIG. 3 is a diagram showing a hardware configuration of the server device 20. The hardware configuration of the server device 20 may be configured to include one or more of the devices shown in FIG. 2, or may be configured not to include some of the devices. Further, a plurality of devices having different housings may be connected by communication to form the server device 20.

The server device 20 is physically configured as a computer device including a processor 2001, a memory 2002, a storage 2003, a communication device 2004, an input device 2005, an output device 2006, a bus connecting them, and the like. For each function in the server device 20, by loading predetermined software (program) on hardware such as the processor 2001 and the memory 2002, the processor 2001 performs an operation and controls communication by the communication device 2004, or the memory 2002. And by controlling at least one of reading and writing of data in the storage 2003. The processor 2001, the memory 2002, the storage 2003, the communication device 2004, the input device 2005, the output device 2006, and the bus connecting them are the processor 1001, the memory 1002, the storage 1003, the communication device 1004, and the input device described for the user terminal 10. Since the hardware is the same as that of 1005, the output device 1006, and the bus connecting them, the description thereof will be omitted. However, the communication device 2004 does not have to be the hardware for performing communication according to the communication standard of wireless communication, and the hardware for performing communication between computers via the network 2 according to the communication standard of arbitrary wired communication (the hardware for performing communication between computers. It may be a transmission / reception device).

FIG. 4 is a block diagram showing an example of the functional configuration of the ensemble system 1. As shown in FIG. 3, in the user terminal 10, the functions of the input unit 11 and the output unit 12 are realized. Further, in the server device 20, the functions of the acquisition unit 21, the specific unit 22, the generation unit 23, the reference timing data transmission unit 24, and the sound data transmission unit 25 are realized.

In each user terminal 10, the input unit 11 collects a sound (referred to as a performance sound) when the user plays a musical instrument, generates sound data indicating the performance sound, and transmits the sound data to the server device 20 via the network 2. To do.

In the server device 20, the acquisition unit 21 acquires various data from each user terminal 10 via the network 2. For example, the acquisition unit 21 acquires the sound data generated by the input unit 11 of each user terminal 10 from each user terminal 10 via the network 2.

In the server device 20, the specific unit 22 uses any of a plurality of user terminals 10 belonging to the same group to generate a reference timing when a performance is performed on these user terminals 10. (Hereinafter referred to as a reference terminal). More specifically, the specifying unit 22 identifies the reference terminal based on the instruction input by the user to at least one of the plurality of user terminals 10 belonging to the same group. For example, the specific unit 22 can be used as a user terminal 10 or a reference terminal in which an instruction to collect the drum performance sound is input by the user on the condition that the reference timing is generated based on the drum performance sound. The user terminal 10 for which the instruction to function is input by the user is specified as a reference terminal. The reference timing corresponds to, for example, the tempo of a musical piece (also called a time signature or a rhythm). For example, if the tempo of the musical piece is constant, the reference timing is repeated at regular intervals in each user terminal 10 belonging to the same group. Will be presented to. Each user performs his / her own performance in accordance with the reference timing presented at each user terminal 10.

In the server device 20, the generation unit 23 generates reference timing data for presenting the reference timing in each user terminal 10 belonging to the same group.

In the server device 20, the reference timing data transmission unit 24 transmits the reference timing data generated by the generation unit 23 to each user terminal 10 belonging to the same group.

In the server device 20, the sound data transmission unit 25 receives from a plurality of user terminals 10 other than the one user terminal 10 for one user terminal 10 in the user terminal 10 group belonging to the same group, respectively. The acquired sound data is mixed and transmitted.

In each user terminal 10, the output unit 12 presents the reference timing to the user according to the reference timing data transmitted from the reference timing data transmission unit 24 of the server device 20. Further, the output unit 12 outputs a sound (performance sound) corresponding to the sound data transmitted from the sound data transmission unit 25 of the server device 20.

[motion]
The operation of the server device 20 will be described with reference to FIGS. 5 to 7. The procedure of each process shown in FIG. 5 is described in the program implemented in the server device 20. In FIG. 5, a plurality of users (here, three people) who play one musical piece ensemble each of them by operating their own user terminals 10a, 10b, and 10c, the date and time when the music starts to be played, the music, and the music. Enter setting information including information that specifies the instrument (also called a part) to be played by oneself. Here, "10:10" is input as the date and time when the performance of the music is started, "music a" is specified as the music to be played, and the user of the user terminal 10a plays the drum, and the user terminal 10b Is specified to play the guitar and the user of the user terminal 10c to play the bass. The acquisition unit 21 of the server device 20 acquires these setting information from each user terminal 10 via the network 2 (step S11).

In the server device 20, the specifying unit 22 specifies the reference terminal based on the above setting information (step S12). Here, on the condition that the reference timing is generated based on the playing sound of the drum, the specifying unit 22 specifies the user terminal 10a into which the information for designating the drum as the musical instrument to be played is input as the reference terminal.

In the server device 20, the generation unit 23 generates reference timing data based on the above setting information (step S13). Specifically, the generation unit 23 stores a database in which the music name of the music and the tempo on the score of the music are associated with each other, and the tempo corresponding to the music name "music a" included in the above setting information. (Eg 60 bpm) is specified. When the tempo is 60 bpm, the generation unit 23 will generate reference timing data for presenting the reference timing every second. For this reference timing data, for example, the presentation start time of the reference timing (for example, 10:10) and the time interval of the reference timing (for example, 1 second) are specified, and the reference is performed at the specified time interval from the specified presentation start time. It may be data instructing the user terminal 10 to present the timing to the user. Further, the reference timing data may be data for instructing the user terminal 10 to specify, for example, a time for presenting the reference timing each time and to present the reference timing to the user at that time. When instructing the user terminal 10 to present the reference timing at the specified time interval from the specified presentation start time as in the former case, the timers provided by each user terminal 10 and the server device 20 synchronize with each other. It is necessary to be there.

Next, in the server device 20, the reference timing data transmission unit 24 starts transmitting reference timing data to each user terminal 10a, 10b, 10c so that each user can present the reference timing (step S14). .. First, the reference timing data transmission unit 24 measures the amount of communication delay in the communication path between the server device 20 and each user terminal 10a, 10b, 10c. The delay amount is measured, for example, by transmitting a delay amount measurement packet from the reference timing data transmission unit 24 to each user terminal 10a, 10b, 10c, and calculating based on the time until the response is returned. The method, or conversely, each user terminal 10a, 10b, 10c transmits a delay amount measurement packet from each user terminal 10a, 10b, 10c to the server device 20, and each user terminal 10a, 10b, 10 is based on the time until the response is returned. There is a method such that the reference timing data transmission unit 24 acquires the calculated delay amount. Here, as a result of such measurement, the delay amount of the communication path between the server device 20 and the user terminal 10a is 10 ms (20 ms for the round trip of the same communication path), and between the server device 20 and the user terminal 10b. The delay amount of the communication path was 20 ms (40 ms for the round trip of the same communication path), and the delay amount of the communication path between the server device 20 and the user terminal 10c was 30 ms (60 ms for the round trip of the same communication path). To do.

Here, FIG. 6 shows the sound data in the server device 20 (top of the figure), the sound data in the user terminal 10a (second from the top), and the sound data in the user terminal 10b (third from the top). , Is a diagram for explaining the time-series relationship with the sound data (bottom) in the user terminal 10c. In FIG. 6, the horizontal axis is time. As shown in FIG. 6, in the server device 20, the drum sound data (hereinafter referred to as P1 sound source) input to the user terminal 10a and transmitted to the server device 20 and the drum sound data input to the user terminal 10b and transmitted to the server device 20 The guitar sound data (hereinafter referred to as P2 sound source) transmitted to the server device 20 and the base sound data (hereinafter referred to as P3 sound source) input to the user terminal 10c and transmitted to the server device 20 are the server device 20. At the same time, sound data (hereinafter referred to as an ensemble sound source) synchronized with these P1 sound source, P2 sound source, and P3 sound source is generated.

In this case, since the delay amount is 10 ms in the user terminal 10a as described above, for example, if the drum performance is started 10 ms before the start of the performance, the time when the sound data reaches the server device 20. However, it is just 10:10 when the performance starts. Similarly, since the delay amount of the user terminal 10b is 20 ms as described above, for example, when the guitar performance is started 20 ms before the start of the performance, the time when the sound data reaches the server device 20. However, it is just 10:10 when the performance starts. Similarly, since the delay amount of the user terminal 10c is 30 ms as described above, for example, if the bass performance is started 30 ms before the start of the performance, the time when the sound data reaches the server device 20. However, it is just 10:10 when the performance starts. Therefore, the reference timing data transmission unit 24 transmits reference timing data instructing each user terminal 10 to present the reference timing earlier by a period corresponding to the corresponding delay amount.

Here, FIG. 7 is a diagram illustrating a method of presenting a reference timing in each user terminal 10. In FIG. 7, the horizontal axis is time. The reference timing may be presented on each user terminal 10 as a sound such as a clicker sound emitted according to the tempo of the musical piece (top of FIG. 7), or the movement changes according to the tempo of the musical piece. It may be presented as a moving image such as a so-called level meter (second row from the top of FIG. 7), or it may be presented by displaying a number that counts the tempo of the music (3 from the top of FIG. 7). The stage) may be presented by changing the color so that the background color of the display area of each user terminal 10 is inverted according to the tempo of the music (bottom of FIG. 7). In FIG. 7, in the display area described as vocal image, guitar image, drum image, and bass image, for example, illustration images simulating the performances of vocal, guitar, drum, and bass are displayed. It has become.

Then, when the performance is started from the time when each user is instructed to start the performance, the input unit 11 of each user terminal 10 collects the performance sound of the musical instrument by the user and generates sound data indicating the performance sound. Is transmitted to the server device 20 via the network 2. The acquisition unit 21 of the server device 20 acquires these sound data, and the sound data transmission unit 25 starts a process of mixing these sound data and transmitting them to each user terminal 10 (step S15).

Here, as described above, in each user terminal 10, the performance is started earlier than 10:10, which is the period corresponding to each delay amount, but the sound data generated in each user terminal 10 is generated. The time when the player reaches the server device 20 is exactly 10:10 when the performance starts. The sound data (ensemble sound source) that has reached the server device 20 and is mixed is transmitted from the server device 20 to each user terminal 10. At this time, the time when the sound data reaches each user terminal 10 is After being transmitted from the server device 20 to each user terminal 10, the delay is increased by the amount of delay corresponding to each user terminal 10. Therefore, as shown in FIG. 6, the ensemble sound source (guitar and bass sound data input to the user terminals 10b and 10c) output from the user terminal 10a is 10 ms from the original start of performance at 10:10. Output starts at a later timing. Similarly, the ensemble sound source (drum and bass sound data input to the user terminals 10a and 10c) output from the user terminal 10b starts to be output at a timing later than 20 ms from the original performance start of 10:10. Will be done. Similarly, the ensemble sound source (drum and guitar sound data input to the user terminals 10a and 10b) output from the user terminal 10c starts to be output at a timing later than 30 ms from the original performance start of 10:10. Will be done.

Therefore, each user may feel a deviation corresponding to the above-mentioned delay amount between the performance performed by himself / herself and the ensemble sound source output from the user terminal 10. However, since the deviation is about several ms to several tens of ms, it is considered that there is no major hindrance to the user's playing behavior. Each user does not adjust his / her own performance to the ensemble sound source, but may perform his / her own performance according to the reference timing presented from the user terminal 10.

As described above, at the start of the performance, each user performs an ensemble according to the tempo determined on the score for each song. However, the tempo of the music may change slightly depending on, for example, the performance of the drum being played. Therefore, in the present embodiment, the generation unit 23 of the server device 20 changes from the reference timing according to the tempo on the score at the start of the performance to the reference timing according to the tempo according to the actual performance (performance on the reference terminal). (Fig. 5: Step S16).

Specifically, the generation unit 23 accumulates sound data acquired from the reference terminal specified by the specific unit 22 from the start of the performance, and extracts the tempo from the accumulated sound data. For this tempo extraction, for example, the sound data acquired from the reference terminal is analyzed to detect the peak of the sound pressure and the time between the peaks is calculated, or the sound data acquired from the reference terminal is analyzed to determine a predetermined frequency range. It is realized by a method such as detecting the timing of the sound belonging to the above and calculating the time between the timings. Then, the generation unit 23 continuously extracts the extracted tempo n (n is a predetermined threshold value) within a predetermined range centered on the tempo set at the start of the performance (for example, with respect to the tempo 100 bpm set at the start of the performance). When it belongs to (95 bpm to 105 bpm of ± 5 bpm), the reference timing data is generated by using the value of the extracted n tempos that have undergone correction processing such as quantizing as the reference timing. The generation of the reference timing data based on the performance on the reference terminal is always and repeatedly executed until the end of the performance of the music. When the reference timing data is changed, the reference timing data transmission unit 24 transmits the newly generated reference timing data to each user terminal 10. The process of step S16 is repeated until the performance is completed (step S17; NO).

According to the embodiment described above, among the user terminals 10 belonging to the same group, the user terminal 10 suitable for generating the reference timing is specified as the reference terminal, so that a plurality of users can remotely play each other. It is possible to perform ensemble at an appropriate tempo while listening.

[Modification example]
The present invention is not limited to the embodiments described above. The above-described embodiment may be modified as follows. Further, the following two or more modified examples may be combined and carried out.
[Modification 1]
The performance in the present invention is a concept that includes not only the performance of the musical instrument exemplified in the embodiment but also the performance by singing or vocal music.

[Modification 2]
In the above embodiment, the reference timing data is generated based on the sound data of the performance on the reference terminal, but the method of generating the reference timing data is not limited to this. For example, the generation unit 23 of the server device 20 may generate reference timing data based on image data indicating an operation for playing. The image data showing the movement for performance includes image data showing the performance movement of the user performing the performance or image data showing an image of the movement of the conductor who directs the performance. In this case, the acquisition unit 21 of the server device 20 acquires the image data in which the above operation is captured by the image pickup device included in the input unit 11 of the reference terminal via the network 2. Then, the generation unit 23 of the server device 20 generates reference timing data for presenting the reference timing to the user at each user terminal 10 based on the image data acquired from the reference terminal by the acquisition unit 21. Specifically, the generation unit 23 analyzes the image data acquired from the reference terminal and performs a characteristic operation (for example, a drum striking operation or a baton swinging up) performed at a timing corresponding to the tempo of the music. The reference timing is specified by extracting the motion of swinging down) by a technique such as image pattern matching and calculating the time between the motions.

[Modification 3]
In the above embodiment, the reference terminal is specified based on the instruction input by the user to the user terminal 10, but the method for specifying the reference terminal is not limited to this. For example, the specific unit 22 of the server device 20 may be
The reference terminal may be specified based on the sound data or the image data acquired by the acquisition unit 21 from the user terminal 10 of at least one of the plurality of user terminals 10 belonging to the same group. This sound data is sound data indicating a user's performance sound generated by the user terminal 10. Further, the image data is image data showing the performance operation of the user performing the performance as described in the second modification, or image data showing the image obtained by capturing the operation of the conductor who directs the performance. Specifically, the specific unit 22 frequency-analyzes the sound data acquired from each user terminal 10 on the condition that the reference timing is generated based on the playing sound of the drum, and identifies the playing sound of the drum. The user terminal 10 that collects the performance sound of the drum is specified as a reference terminal. Further, the specific unit 22 analyzes the image data acquired from each user terminal 10 on the condition that the reference timing is generated based on the performance sound of the drum, and images the operation estimated to be the performance of the drum. The user terminal 10 that is identified by a technique such as pattern matching and collects the performance sound of the drum is specified as a reference terminal. Further, the specific unit 22 analyzes the image data acquired from each user terminal 10 on the condition that the reference timing is generated based on the command of the conductor, and images the motion presumed to be the command of the performance. The user terminal 10 that is identified by a technique such as pattern matching and images the movement of the conductor is specified as a reference terminal.

Further, for example, when changing from the condition that the reference timing is generated based on the playing sound of the drum to the condition that the reference timing is generated based on the playing sound of the percussion during the performance, the above-mentioned sound is obtained. A method of identifying a reference terminal based on data or image data may be applied to change the reference terminal during performance. In which section of the song, the condition that the reference timing is generated based on the playing sound of the drum and the condition that the reference timing is generated based on the playing sound of the percussion are determined in advance for each song. It may be decided or specified by the user.

Further, when the sound data cannot be acquired from the reference terminal due to a reason such as temporary muffling of the musical instrument, the specific unit 22 follows the predetermined priority and refers to the user terminal 10 having the next priority. It may be specified as a terminal. Specifically, when the drum is set as the first priority and the bass is set as the second priority in advance as the priority, the user terminal 10 that has picked up the playing sound of the drum (that is, the reference at this point). When the sound data cannot be acquired from the terminal) due to muffling or the like, the specifying unit 22 may specify the user terminal 10 that has picked up the performance sound of the bass having the next priority as the reference terminal.

[Modification example 4]
In a musical piece, there may be a section in which the reference timing is desired to be constant (first period) and a section in which the reference timing is intentionally changed according to the performance or the like (second period). In the section where the reference timing is desired to be constant, the ensemble follows a regular tempo by correcting the tempo deviation created by the performer on the reference terminal. On the other hand, in the section where the reference timing is intentionally changed according to the performance or the like, the tempo generated by the performer on the reference terminal becomes the reference timing, so that the ensemble is full of emotion and ad lib. In this case, the generation unit 23 corrects the time interval extracted from the sound data or the image data acquired from the reference terminal by the acquisition unit 21 in the first period of the period in which the performance is performed. Generate reference timing data to output the reference timing at intervals. Specifically, as described in the embodiment, the generation unit 23 generates reference timing data for outputting the reference timing at a predetermined fixed time interval, and then the extracted tempo is n ( n is a predetermined threshold), and if it belongs to a predetermined range centered on the tempo set at the start of the performance (for example, 95 bpm to 105 bpm of ± 5 bpm with respect to the tempo 100 bpm set at the start of the performance), the extraction thereof. Reference timing data is generated using the values of the n tempos that have undergone correction processing such as quantizing as the reference timing. On the other hand, the generation unit 23 continuously extracts the extracted tempo n (n is a predetermined threshold value) within a predetermined range centered on the tempo set at the start of the performance (for example, with respect to the tempo 100 bpm set at the start of the performance). If it does not belong to ± 5 bpm (95 bpm to 105 bpm), the reference timing data is generated using the extracted tempo as the reference timing without performing the above correction processing. That is, the generation unit 23 uses the tempo extracted based on the sound data acquired from the reference terminal as the tempo of the reference timing subsequently presented by each user terminal 10. Therefore, the tempo produced by the performer on the reference terminal corresponds to the reference timing from a slightly delayed time. In this way, the generation unit 23 generates the reference timing data in the second period when the time interval extracted from the sound data or the image data acquired from the reference terminal by the acquisition unit 21 does not fall within the predetermined range. You may try to do it. The first period and the second period described above are not dynamically determined, and the first period and the second period may be predetermined over the entire music.

[Modification 5]
The identification unit 22 specifies two or more user terminals 10 as reference terminals, and the generation unit 23 generates reference timing data based on sound data or image data acquired from the two or more reference terminals by the acquisition unit 21. You may try to do it. For example, the specific unit 22 specifies two user terminals 10 that collect the performance sounds of the drum and the percussion as reference terminals, and the generation unit 23 specifies the sound data acquired from the two or more reference terminals by the acquisition unit 21. Alternatively, the value calculated by obtaining the reference timing based on the image data and further performing correction such as averaging the two reference timings may be used as the true reference timing.

[Modification 6]
As an example of the information processing device according to the present invention, a server device 20 different from the user terminal 10 has been illustrated, but the present invention can be applied to any information processing device that realizes the functional block illustrated in FIG. Further, any one of the user terminals 10 included in the same group performing the ensemble may function as the information processing device according to the present invention.

[Other variants]
The block diagram used in the description of the above embodiment shows a block of functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Further, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one physically or logically connected device, or directly or indirectly (for example, two or more physically or logically separated devices). , Wired, wireless, etc.) and may be realized using these plurality of devices. The functional block may be realized by combining the software with the one device or the plurality of devices.

Functions include judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, solution, selection, selection, establishment, comparison, assumption, expectation, and assumption. There are broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, assigning, etc., but only these. I can't. For example, a functional block (constituent unit) for functioning transmission is called a transmitting unit or a transmitter. As described above, the method of realizing each of them is not particularly limited.

For example, the user terminal 10 in one embodiment of the present disclosure may function as a computer that processes the wireless communication method of the present disclosure.

The notification of information is not limited to the embodiments / embodiments described in the present disclosure, and may be performed by other methods. For example, information notification includes physical layer signaling (for example, DCI (Downlink Control Information), UCI (Uplink Control Information)), upper layer signaling (for example, RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, etc. It may be carried out by notification information (MIB (Master Information Block), SIB (System Information Block)), other signals, or a combination thereof. Further, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

Each aspect / embodiment described in the present disclosure includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), and 5G (5th generation mobile communication). system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)) )), IEEE 802.16 (WiMAX®), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth®, and other systems that utilize and extend based on these. It may be applied to at least one of the next generation systems. Further, a plurality of systems may be applied in combination (for example, a combination of at least one of LTE and LTE-A and 5G).

The order of the processing procedures, sequences, flowcharts, and the like of each aspect / embodiment described in the present disclosure may be changed as long as there is no contradiction. For example, the methods described in the present disclosure present elements of various steps using exemplary order, and are not limited to the particular order presented.

Information and the like can be output from the upper layer (or lower layer) to the lower layer (or upper layer). Input / output may be performed via a plurality of network nodes.

The input / output information and the like may be stored in a specific location (for example, a memory), or may be managed using a management table. Input / output information and the like can be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

The determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (Boolean: true or false), or by comparing numerical values (for example, a predetermined value). It may be done by comparison with the value).

Each aspect / embodiment described in the present disclosure may be used alone, in combination, or switched with execution. Further, the notification of predetermined information (for example, the notification of "being X") is not limited to the explicit one, but is performed implicitly (for example, the notification of the predetermined information is not performed). May be good.
Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure may be implemented as an amendment or modification without departing from the purpose and scope of the present disclosure, which is determined by the description of the scope of claims. Therefore, the description of this disclosure is for purposes of illustration only and does not have any restrictive meaning to this disclosure.

Software is an instruction, instruction set, code, code segment, program code, program, subprogram, software module, whether called software, firmware, middleware, microcode, hardware description language, or another name. , Applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, features, etc. should be broadly interpreted to mean.
Further, software, instructions, information and the like may be transmitted and received via a transmission medium. For example, a website, where the software uses at least one of wired technology (coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), etc.) and wireless technology (infrared, microwave, etc.). When transmitted from a server, or other remote source, at least one of these wired and wireless technologies is included within the definition of transmission medium.

The information, signals, etc. described in the present disclosure may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.
The terms described in the present disclosure and the terms necessary for understanding the present disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). Also, the signal may be a message. Further, the component carrier (CC: Component Carrier) may be referred to as a carrier frequency, a cell, a frequency carrier, or the like.

The terms "system" and "network" used in this disclosure are used interchangeably.

In addition, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, relative values from predetermined values, or using other corresponding information. It may be represented. For example, the radio resource may be indexed.
The names used for the above parameters are not limited in any way. Further, mathematical formulas and the like using these parameters may differ from those explicitly disclosed in this disclosure. Since the various channels (eg, PUCCH, PDCCH, etc.) and information elements can be identified by any suitable name, the various names assigned to these various channels and information elements are in any respect limited names. is not it.

In the present disclosure, terms such as "mobile station (MS)", "user terminal 10 (user terminal)", "user equipment (UE)", and "terminal" are used interchangeably. obtain.
Mobile stations can be subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, mobile terminals, wireless, depending on the trader. It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.

The user terminal 10 may be called a transmitting device, a receiving device, a communication device, or the like. At least one of the user terminals 10 may be a device mounted on the mobile body, the mobile body itself, or the like. The moving body may be a vehicle (eg, car, airplane, etc.), an unmanned moving body (eg, drone, self-driving car, etc.), or a robot (manned or unmanned). ) May be. The user terminal 10 also includes a device that does not necessarily move during a communication operation. For example, the user terminal 10 may be an IoT (Internet of Things) device such as a sensor. The user terminal 10 in the present disclosure may be read as a base station. In this case, the base station may have the functions of the user terminal 10 described above.

The terms "determining" and "determining" may include a wide variety of actions. "Judgment" and "decision" are, for example, judgment (judging), calculation (calculating), calculation (computing), processing (processing), derivation (deriving), investigating (investigating), search (looking up, search, inquiry). It may include (eg, searching in a table, database or another data structure), ascertaining as "judgment" or "decision". Also, "judgment" and "decision" are receiving (for example, receiving information), transmitting (for example, transmitting information), input (input), output (output), and access. (Accessing) (for example, accessing data in memory) may be regarded as "judgment" or "decision". In addition, "judgment" and "decision" mean that the things such as solving, selecting, choosing, establishing, and comparing are regarded as "judgment" and "decision". Can include. That is, "judgment" and "decision" may include that some action is regarded as "judgment" and "decision". Further, "judgment (decision)" may be read as "assuming", "expecting", "considering" and the like.

The terms "connected", "coupled", or any variation thereof, mean any direct or indirect connection or connection between two or more elements, and each other. It can include the presence of one or more intermediate elements between two "connected" or "combined" elements. The connection or connection between the elements may be physical, logical, or a combination thereof. For example, "connection" may be read as "access". As used in the present disclosure, the two elements use at least one of one or more wires, cables and printed electrical connections, and, as some non-limiting and non-comprehensive examples, the radio frequency domain. Can be considered to be "connected" or "coupled" to each other using electromagnetic energies having wavelengths in the microwave and light (both visible and invisible) regions.

The phrase "based on" as used in this disclosure does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

Any reference to elements using designations such as "first", "second" as used in this disclosure does not generally limit the quantity or order of those elements. These designations can be used in the present disclosure as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted, or that the first element must somehow precede the second element.

The "means" in the configuration of each of the above devices may be replaced with a "part", a "circuit", a "device" and the like.

When "include", "including" and variations thereof are used in the present disclosure, these terms are as comprehensive as the term "comprising". Is intended. Furthermore, the term "or" used in the present disclosure is intended not to be an exclusive OR.

In the present disclosure, if articles are added by translation, for example a, an and the in English, the disclosure may include the nouns that follow these articles in the plural.

In the present disclosure, the term "A and B are different" may mean "A and B are different from each other". The term may mean that "A and B are different from C". Terms such as "separate" and "combined" may be interpreted in the same way as "different".

1 ... Ensemble system, 2 ... Network, 10 ... User terminal, 11 ... Input unit, 12 ... Output unit, 1001 ... Processor, 1002 ... Memory, 1003 ... Storage, 1004 ... Communication device, 1005 ... Input device, 1006 ... Output device , 20 ... server device, 21 ... acquisition unit, 22 ... specific unit, 23 ... generation unit, 24 ... reference timing data transmission unit, 25 ... sound data transmission unit, 2001 ... processor, 2002 ... memory, 2003 ... storage, 2004 ... Communication device, 2005 ... Input device, 2006 ... Output device

Claims (9)

An acquisition unit that acquires the sound data or the image data from a plurality of terminals that generate sound data indicating the played sound or image data indicating the operation for the performance.
A specific unit that specifies one of the plurality of terminals as a reference terminal for generating a reference timing when a performance is performed on the plurality of terminals.
A generation unit that generates reference timing data for presenting the reference timing to a performer at each of the terminals based on the sound data or the image data acquired from the reference terminal by the acquisition unit.
A reference timing data transmission unit that transmits the generated reference timing data to the plurality of terminals,
A sound data transmission unit that mixes and transmits a plurality of the sound data acquired by the acquisition unit from a plurality of terminals other than the one terminal is provided to one of the plurality of terminals. An information processing device characterized by. The information processing device according to claim 1, wherein the identification unit identifies the reference terminal based on an instruction input to at least one of the plurality of terminals. A claim, wherein the specific unit identifies the reference terminal based on the sound data or the image data acquired by the acquisition unit from at least one of the plurality of terminals. 1. The information processing apparatus according to 1. The image data according to any one of claims 1 to 3, wherein the image data includes image data that captures the movement of a performer who performs a performance, or image data that captures an image of the movement of a conductor who directs a performance. The information processing device according to item 1. The generator is used during the period during which the performance is performed.
In the first period, the reference timing data for presenting the reference timing at a fixed time interval corrected for the time interval extracted from the sound data or the image data acquired from the reference terminal by the acquisition unit. To generate
In the second period, a reference for presenting the reference timing at a non-constant time interval according to the time interval extracted from the sound data or the image data acquired from the reference terminal by the acquisition unit. The information processing apparatus according to any one of claims 1 to 4, wherein the information processing apparatus is characterized by generating timing data. In the first period, the generator is
First, the reference timing data for presenting the reference timing is generated at a predetermined fixed time interval, and then the reference timing data is generated.
Next, the acquisition unit generates reference timing data for presenting the reference timing at a fixed time interval corrected for the time interval extracted from the sound data or the image data acquired from the reference terminal. 5. The information processing apparatus according to claim 5. The generator
When the time interval extracted from the sound data or the image data acquired from the reference terminal by the acquisition unit does not fall within a predetermined range, the reference timing data in the second period is generated. The information processing apparatus according to claim 5 or 6. The specific unit identifies two or more of the terminals as the reference terminal,
The generation unit generates reference timing data for presenting the reference timing at each of the terminals based on the sound data or the image data acquired from the two or more reference terminals by the acquisition unit. The information processing apparatus according to any one of claims 1 to 7. Computer,
An acquisition unit that acquires the sound data or the image data from a plurality of terminals that generate sound data indicating the played sound or image data indicating the operation for the performance.
A specific unit that specifies one of the plurality of terminals as a reference terminal for generating a reference timing when a performance is performed on the plurality of terminals.
A generation unit that generates reference timing data for presenting the reference timing to a performer at each of the terminals based on the sound data or the image data acquired from the reference terminal by the acquisition unit.
A reference timing data transmission unit that transmits the generated reference timing data to the plurality of terminals,
It functions as a sound data transmission unit that mixes and transmits a plurality of the sound data acquired by the acquisition unit from a plurality of terminals other than the one terminal to one of the plurality of terminals. Program to let you.