WO2023195333A1

WO2023195333A1 - Control device

Info

Publication number: WO2023195333A1
Application number: PCT/JP2023/010952
Authority: WO
Inventors: 善政磯崎; 克典鈴木; 隆洋寺田; 隆志森; 潤石井; 伊吹半田; 琢哉藤島; 幸司谷高; 幸夫涌井; 宗一瀧川; 陽前澤; 陽貴大川; 吉勝松原; 保彦大場; 福太郎奥山; 智也佐々木
Original assignee: ヤマハ株式会社
Priority date: 2022-04-06
Filing date: 2023-03-20
Publication date: 2023-10-12
Also published as: JP2023154288A

Abstract

A control device according to one embodiment includes a first transmission unit, a first reception unit, and a first generation unit. The first transmission unit transmits, to a second communication base, first performance data including the contents of performance for a keyboard instrument at a first communication base. The first reception unit receives second performance data from the second communication base. The first generation unit generates a driving signal for producing sound corresponding to the second performance data, and outputs the driving signal to a sound production device at the first communication base. The first performance data and/or the second performance data includes key position signals indicating pressing amounts of keys in the keyboard instrument.

Description

Control device

The present invention relates to a control device.

Technology has been developed that allows multiple communication bases where musical instruments are played to be connected via a network, making it possible to perform in ensemble even when the instruments are placed in remote locations. For example, Patent Document 1 discloses a technique for reducing the influence of communication delay in order to realize a comfortable ensemble performance.

Japanese Patent Application Publication No. 2005-195982

When performing in an ensemble at multiple communication bases, it is difficult for multiple performers to feel a sense of unity from various viewpoints, compared to when performing in an ensemble at the same location.

One of the purposes of the present invention is to enable a plurality of musicians playing in an ensemble to feel a sense of unity.

A control device in one embodiment includes a first transmitter, a first receiver, and a first generator. The first transmitter transmits first performance data including performance details for a keyboard instrument at the first communication base to the second communication base. The first receiving section receives the second performance data from the second communication base. The first generation section generates a drive signal for producing sound according to the second performance data, and outputs it to the sound production device at the first communication base. At least one of the first performance data and the second performance data includes a key position signal indicating the amount of depression of a key on the keyboard instrument.

According to the present invention, it is possible for a plurality of performers playing in an ensemble to feel a sense of unity.

FIG. 1 is a diagram for explaining a communication system configuration in a first embodiment. It is a figure explaining the internal structure of the automatic performance piano in a 1st embodiment. FIG. 2 is a diagram illustrating the configuration of a control device in the first embodiment. It is a figure explaining the composition of an ensemble control function in a 1st embodiment. It is a figure explaining the positional relationship of the vibrator and the pick-up sensor in a 2nd embodiment. FIG. 7 is a diagram illustrating the configuration of a drive signal generation section in a second embodiment. FIG. 7 is a diagram illustrating the relationship between velocity and delay time in the third embodiment. FIG. 7 is a diagram illustrating the relationship between velocity and correction value in the third embodiment. It is a figure explaining the composition of the environment collecting device in communication base T1 in a 4th embodiment. It is a figure explaining the structure of the environment provision apparatus in communication base T2 in 4th Embodiment. It is a figure explaining the composition of the control signal generation part in a 5th embodiment. It is a figure explaining the display example of the screen in 6th Embodiment.

Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. The embodiments shown below are merely examples, and the present invention should not be construed as being limited to these embodiments. The configuration described in each embodiment can also be applied to other embodiments. In the drawings referred to in multiple embodiments described below, the same parts or parts having similar functions are denoted by the same or similar symbols (numerals followed by numbers such as A, B, etc.), The repeated explanation may be omitted. In order to clarify the explanation, the drawings may be explained schematically with some components omitted from the drawings.

<First embodiment> [Communication system]
FIG. 1 is a diagram illustrating the configuration of a communication system in one embodiment. The communication system includes a server 1000 connected to a network NW such as the Internet. Server 1000 includes a control unit such as a CPU, a storage unit, and a communication unit. The control unit provides a service for realizing an ensemble performance between communication bases by executing a predetermined program. The server 1000 controls communication between a plurality of communication bases connected to the network NW, and executes processing necessary for the automatic performance pianos 1 at each communication base to realize P2P type communication with each other. This processing may be realized by a known method. In FIG. 1, two communication bases T1 and T2 are illustrated, but the number is not limited to this, and even more communication bases may exist. In the following description, when the communication bases T1 and T2 are described without distinction, they are simply referred to as communication bases.

In this example, information related to the performance at each communication base is exchanged between the communication base T1 and the communication base T2 by P2P communication. Through this communication, an ensemble performance is realized between a plurality of communication bases. A self-playing piano 1 is arranged at each communication base. In this example, an environment collecting device 82 and an environment providing device 88 are connected to the automatic performance piano 1.

The environment collection device 82 includes a sensor for collecting information on the surrounding environment of the automatic piano 1, and outputs a collection signal indicating the measurement result of the sensor. The surrounding environment includes, for example, sound, light, vibration, temperature, air flow, and the like. Upon acquiring a control signal indicating the surrounding environment, the environment providing device 88 provides an environment based on the control signal. The environment collecting device 82 and the environment providing device 88 may be configured integrally. The environment providing device 88 may be provided depending on the number of other communication bases. For example, if there are three communication bases in addition to the communication base T1, three environment providing devices 88 may be provided in the communication base T1 corresponding to the respective communication bases. At least one of the environment collecting device 82 and the environment providing device 88 may be built into the automatic performance piano 1. Specific examples of the environment collecting device 82 and the environment providing device 88 will be described later.

The automatic performance piano 1 includes a keyboard instrument 10, a control device 20, a sensor 30, and a drive device 40.

[Automatic piano]
Next, the configuration of the automatic performance piano 1 will be explained.

FIG. 2 is a diagram illustrating the internal configuration of the automatic performance piano in the first embodiment. The keyboard instrument 10 of the automatic performance piano 1 corresponds to, for example, a grand piano. Keyboard instrument 10 includes a plurality of keys 12. The keyboard instrument 10 includes a hammer 14, a string 15, and a damper 18 provided corresponding to each key 12. In the automatic performance piano 1, the configurations provided corresponding to each key 12 are shown focusing on each configuration provided corresponding to one key 2 shown in FIG. Therefore, descriptions of each configuration provided corresponding to other keys 12 are omitted. Some components, such as the damper 18, may not be provided for some keys 12.

The keyboard instrument 10 includes a plurality of pedals 13. The plurality of pedals 13 are, for example, a damper pedal, a shift pedal, and a sostenuto pedal. In the automatic performance piano 1, the configurations provided corresponding to each pedal 13 are shown focusing on each configuration provided corresponding to one pedal 13 shown in FIG. Therefore, descriptions of each component provided corresponding to the other pedals 13 are omitted. The keyboard instrument 10 further includes a keyboard lid 11, a bridge 16, a soundboard 17, a straight post 19, and the like.

The sensor 30 includes a key sensor 32, a pedal sensor 33, and a hammer sensor 34. The key sensor 32 is provided corresponding to each key 12 and outputs a measurement signal according to the behavior of the key 12 to the control device 20. In this example, the key sensor 32 outputs a measurement signal according to the position (depression amount) of the key 12 to the control device 20. The position of the key 12 may be measured in continuous quantities (fine resolution) or by detecting that the key 12 passes a predetermined position. The key 12 may be detected at a plurality of positions within the pressing range of the key 12 (range from the rest position to the end position).

The hammer sensor 34 is provided corresponding to each hammer 14 and outputs a measurement signal according to the behavior of the hammer 14 to the control device 20. In this example, the hammer sensor 34 measures the position (rotation amount) of the hammer shank immediately before the hammer 14 hits the string 15, and outputs a measurement signal to the control device 20 in accordance with the measurement result. The position of the hammer shank may be measured in continuous quantities (fine resolution) or by detecting when the hammer shank passes a predetermined position. The position where the hammer shank is detected may be a plurality of positions within the range immediately before the hammer 14 hits the string 15.

The pedal sensor 33 is provided corresponding to each pedal 13 and outputs a measurement signal according to the behavior of the pedal 13 to the control device 20. In this example, the pedal sensor 33 outputs a measurement signal according to the position (depression amount) of the pedal 13 to the control device 20. The position of the pedal 13 may be detected as a continuous quantity (fine resolution), or may be detected when the pedal 13 passes a predetermined position. The position where the pedal 13 is detected may be any of a plurality of positions within the depression range of the pedal 13 (range from the rest position to the end position).

The drive device 40 includes a key drive device 42, a pedal drive device 43, a stopper 44, a vibrator 47, and a damper drive device 48. The key drive device 42 is provided corresponding to each key 12, and is driven to press down the key 12 under control using a drive signal from the control device 20. This mechanically reproduces the same situation as when the player presses the key 12. A pedal drive device 43 is provided corresponding to each pedal 13, and is driven to press down the pedal 13 under control using a drive signal from the control device 20. This mechanically reproduces the same situation as when the player depresses the pedal 13. The damper driving device 48 is provided corresponding to each damper 18, and is driven so as to separate the damper 18 from the string 15 under control using a drive signal by the control device 20. The damper drive device 48 may have a configuration that drives all dampers 18 simultaneously.

The stopper 44 is driven by control from the control device 20 so as to be in either a position where it collides with the hammer shank (blocking position) or a position where it does not collide with the hammer shank (retreat position). When the stopper 44 is in the blocking position, the movement of the hammer shank is restricted and the hammer 14 does not strike the string 15 even if the key 12 is pressed down. When the stopper 44 is in the retracted position, when the key 12 is pressed down, the hammer 14 interlocked with the key 12 strikes the string 15. When the strings 15 are struck, the keyboard instrument 10 generates sound.

In this example, the vibrator 47 is supported by a support section connected to the straight post 9 so as to be in contact with the surface of the soundboard 17 opposite to the part where the bridge piece 16 is arranged. The vibrator 47 vibrates the soundboard 17 under the control of the control device 20 using a drive signal. For example, when a drive signal containing a piano sound is supplied from the control device 20, the vibrator 47 applies vibrations to the soundboard 17 according to the drive signal. As a result, piano sound is emitted from the soundboard 17. A plurality of vibrators 47 may be arranged so as to be in contact with the soundboard 17. Instead of the vibrator 47 that vibrates the soundboard 17, a speaker that emits sound may be used.

Sound generation by the keyboard instrument 10 includes cases where it is realized by hitting the strings 15 with the hammer 14 and cases where it is realized by vibrating the soundboard 17 with the vibrator 47. Therefore, the keyboard instrument 10 can also be said to include a sounding device that generates a string hitting sound by driving the keys 12, and a sounding device that generates a sound from the soundboard 17 by driving the vibrator 47. . The driving of the key 12 and the driving of the vibrator 47 are realized by outputting a driving signal to the driving device 40 as described later.

The configuration of the control device 20 will be explained. In this example, the control device 20 is attached to the keyboard instrument 10. The control device 20 does not need to be a device attached to the keyboard instrument 10, and may be, for example, a personal computer, a tablet computer, a smartphone, or the like.

FIG. 3 is a diagram illustrating the configuration of the control device in the first embodiment. The control device 20 includes a control section 21 , a storage section 22 , an operation panel 23 , a communication section 24 , a sound source section 25 , and an interface 26 . Each of these components is connected via a bus 27.

The control unit 21 is an example of a computer including a processor such as a CPU and a storage device such as a RAM. The control unit 21 executes a program stored in the storage unit 22 using a CPU (processor), and causes the control device 20 to implement functions for executing various processes. The functions realized by the control device 20 include an ensemble control function to be described later. This ensemble control function controls each part of the control device 20 and each component connected to the interface 26. A sensor 30 and a drive device 40 are connected to the interface 26 . In this example, an external device 80 is further connected to the interface 26. The interface 26 transmits drive signals, control signals, etc. generated by the control unit 21 to the target configurations, and receives measurement signals, collection signals, etc. from each target configuration.

The storage unit 22 is a storage device such as a nonvolatile memory or a hard disk drive. The storage unit 22 stores a program executed by the control unit 21 and various data required when executing this program.

The operation panel 23 has operation buttons and the like that accept user operations. When a user's operation is accepted using this operation button, an operation signal corresponding to the operation is output to the control unit 21. The operation panel 23 may have a display screen. In this case, the operation panel 23 may be a touch panel in which a touch sensor is combined with a display screen.

The communication unit 24 is a communication module that communicates with other devices wirelessly, wired, etc. In this example, the other device with which the communication unit 24 communicates is the player piano 1 at the server 1000 or another communication base. In this example, performance data, environment data, etc. indicating the contents of a performance on the keyboard instrument 10 are communicated between the communication bases.

The sound source section 25 generates a sound signal under control from the control section 21. The sound signal is used as a drive signal for driving the vibrator 47 (vibration drive signal to be described later). The sound signal includes, in this example, a signal representing the sound of a piano. For example, the control unit 21 controls the sound source unit 25 to generate a sound signal representing a piano sound according to the performance content corresponding to the performance data. The performance data may be data generated based on a measurement signal generated by the sensor 30. The performance data may be, for example, MIDI format data including sound production control information such as note-on, note-off, note number, velocity, etc., or may be information directly indicated by a measurement signal.

The interface 26 is an interface that connects the control device 20 and each external component. Each component connected to interface 26 includes, in this example, sensor 30, drive device 40, and external device 80, as described above. The interface 26 outputs the measurement signal output from the sensor 30 to the control unit 21. The interface 26 outputs a drive signal for driving each device to the drive device 40. The drive signal is generated in an ensemble control function 100, which will be described later. The interface 26 may include a headphone terminal or the like to which a sound signal representing the piano sound generated by the sound source section 25 is supplied.

[Ensemble control function]
Next, the ensemble control function realized by the control unit 21 executing the program will be described. The structure for realizing the ensemble control function is not limited to being realized by executing a program, and at least a part of the structure may be realized by hardware. The configuration for realizing the ensemble control function may be realized not by the control device 20 but by a device connected to the interface 26 (for example, a computer in which this program is installed).

In this example, when the ensemble control function is realized, the control unit 21 controls the stopper 44 to be placed at the blocking position. In this case, when the user inputs a performance operation to the keys 12 and the pedal 13, the stopper 44 prevents the string from being struck, while the sound signal (for example, the sound of a piano performance) corresponding to the performance operation is It is generated in the sound source section 25. Using this sound signal, the vibrator 47 vibrates the soundboard 17, thereby emitting sound. A signal for driving the vibrator 47 is generated by a drive signal generation section 145 described below.

FIG. 4 is a diagram illustrating the configuration of the ensemble control function in the first embodiment. The ensemble control function 100 includes a performance data generation section 131, a performance data transmission section 133, a performance data reception section 143, and a drive signal generation section 145. In this example, the ensemble control function 100 further includes an environment data generation section 121, an environment data transmission section 123, and an environment data transmission section 123 as a function for sharing the surrounding environment of the automatic performance piano 1 between communication bases in conjunction with the ensemble performance. It includes a receiving section 183 and a control signal generating section 185.

The performance data generation unit 131 generates performance data indicating the content of the performance on the keyboard instrument 10 based on the measurement signal output from the sensor 30. In this example, the performance data includes a measurement signal output from the key sensor 32 (hereinafter referred to as a key position signal) and a measurement signal output from the pedal sensor 33 (hereinafter referred to as a pedal position signal). In this example, the key position signal includes the pitch of the pressed key 12 and the amount of pressing of the key 12. If the key sensor 32 is a sensor that measures the amount of depression of the key 12 at four locations, the information of the amount of depression of the key 12 included in the key position signal indicates the position of one of the four locations.

In this example, the pedal position signal includes the type of pedal 13 that was pressed and the amount of depression of the pedal 13. If the pedal sensor 33 is a sensor that measures the amount of pedal depression at three locations, the information on the amount of depression of the pedal 13 indicates the position of one of the three locations. The performance data may further include a measurement signal (hereinafter referred to as a hammer position signal) output from the hammer sensor 34. The hammer position signal includes, for example, the pitch of the key and the rotational position of the hammer 14.

It is assumed that the performance data generated by the performance data generation section 131 is data (for example, in MIDI format) that includes sound production control information generated based on the measurement results of the key sensor 32 and the pedal sensor 33. In this case, for example, in order to transmit a note-on, it is necessary that the amount of depression of the key 12 reaches a state where a note-on occurs.

On the other hand, according to the performance data generation unit 131 in this example, the amount of depression of the key 12 can be sequentially transmitted while the key 12 is being depressed. Therefore, it is possible to make the automatic performance piano 1 at another communication base recognize that the key 12 has started to be pressed even before the note-on is reached. For example, when the key 12 on the automatic performance piano 1 of the communication base T1 starts to be pressed, even before a note-on occurs, the key 12 on the automatic performance piano 1 of the communication base T2 is pressed down to the recognized pressing amount. You can start driving. By doing so, it is possible to drive the keys 12 at the communication base T2 so as to follow the playing operation on the keys 12 at the communication base T1 with a short delay time.

The performance data transmitter 133 transmits the performance data generated by the performance data generator 131 to other communication bases.

The performance data receiving unit 143 receives performance data transmitted from other communication bases.

The drive signal generation section 145 generates a drive signal used in the drive device 40 based on the performance data received by the performance data reception section 143. This drive signal includes a signal supplied to the key drive device 42 (key drive signal), a signal supplied to the pedal drive device 43 (pedal drive signal), and a signal supplied to the vibrator 47 (excitation drive signal). )including.

The key drive signal is generated based on the performance data, and more specifically, based on the key position signal included in the performance data. The key drive signal is a signal for controlling the key drive device 42 to drive the key 12 so as to reproduce the amount of depression corresponding to the key position signal. The key 12 to be driven is a key corresponding to the pitch specified by the key position signal. The pedal drive signal is generated based on the performance data, and more specifically, based on the pedal position signal. The pedal drive signal is a signal for controlling the pedal drive device 43 to move the pedal corresponding to the type specified by the pedal position signal to a position corresponding to the amount of depression.

The vibration driving signal is generated based on the performance data, and more specifically, is a signal generated by the sound source section 25 based on the key position signal and the pedal position signal. When the vibrator 47 vibrates the soundboard 17 in response to the vibration drive signal, the sound (piano sound in this example) corresponding to the signal generated in the sound source section 25 is transmitted around the keyboard instrument 10 via the soundboard 17. It spreads to

When generating a sound signal in the sound source section 25, the drive signal generation section 145 generates sound generation control information based on the key position signal and the pedal position signal, and generates a sound signal in the sound source section 25 based on the sound generation control information. You may let them. At this time, the drive signal generation unit 145 may generate the sound generation control information using a calculation that predicts the note-on timing and velocity from the change in the amount of depression of the key 12 indicated by the key position signal in the performance data. . Changes in the rotational position of the hammer 14 indicated by the hammer position signal in the performance data may be used for this predictive calculation. The predictive calculation may use a learned model obtained in advance by machine learning, or may use a fitting process that assumes a constant velocity trajectory, a constant acceleration trajectory, etc. based on changes in the amount of depression. This makes it possible to improve prediction accuracy even when the movements of the key 12 and the hammer 14 are not aligned.

It is assumed that the key 12 is driven based on performance data, and the hammer 14 that operates as a result hits the string 15. In this case, since it takes time to drive the key 12 after the sound generation instruction (for example, note-on), the timing at which the sound is produced is delayed. Therefore, the timing of sound generation is affected not only by the communication delay between communication points but also by the delay when the key 12 is driven.

On the other hand, according to the drive signal generation unit 145, the key 12 and the pedal 13 are driven by the key drive signal and the pedal drive signal, but since the stopper 44 prevents the hammer 14 from hitting the string 15, a string-striking sound is generated. do not. Instead, sound is generated from the soundboard 17 by driving the vibrator 47 with the vibration drive signal. Producing sound using the vibrator 47 does not require driving the key 12. Therefore, regarding the time from the sound generation instruction (for example, note-on) to the actual sound generation, the time for sound generation by the vibrator 47 is shorter than the time for sound generation by string striking.

At this time, since the sound generation by the vibrator 47 and the driving of the key 12 are controlled separately, a difference in timing occurs between them. On the other hand, since the amount of deviation is small, it has little effect on the user's senses.

The performance data and sound generation method to be transmitted are not limited to the above combinations. For example, instead of transmitting the amount by which the key 12 is pressed as the performance data, sound generation control information may be transmitted as the performance data. Even in this case, by using the sound generation by the vibrator 47 as described above, the time difference in sound generation between the communication bases can be made shorter than when the sound generation by string striking is used.

Each drive signal is generated based on the sound production control information in the performance data. At this time, regarding the key drive signal, the velocity value in the sound generation control information may be increased to a predetermined value or more. When the velocity value is small, that is, when the pressing speed of the key 12 is slow (when the rotating speed of the hammer 14 is small), the key driving device 42 slows down the driving speed of the key 12.

At this time, depending on the characteristics of the solenoid, the slower the driving speed is set, the more the key 12 may move with a delay from the scheduled timing. In order to compensate for the delay, it is conceivable to increase the velocity value. If the velocity value is increased, the delay time when the key 12 is driven becomes shorter, but the string striking sound becomes louder. On the other hand, in this example, the string striking of the hammer 14 is prevented by the stopper 44 and no string striking sound is produced, so that there is no effect on sound production. Therefore, the velocity value can be increased for keys 12 that do not contribute to sound production. Regarding the excitation drive signal, the velocity value is not changed so that the sound content remains unchanged. At this time, some of the keys 12 may not be driven so as not to affect the user's performance. The keys 12 that are not driven may be keys of pitches used for the performance music by setting the performance music in advance.

As another example, instead of using the vibrator to generate sound, the stopper 44 may be controlled to the retracted position to generate sound by driving the key 12 and striking the string. Even in this case, as described above, by transmitting the amount of depression of the key 12 as performance data, the time difference in sound production between communication bases can be made shorter than when the sound generation control information is transmitted as performance data. I can do it. In this case, both the sound generated by the user's performance operation (for example, the sound generated by the performance at the communication base T1) and the sound generated by the performance at another communication base (for example, the communication base T2) include string striking sounds.

At this time, in order to avoid affecting the user's performance, the pedal 13 may not be driven by the pedal drive device 43, while the damper 18 may be driven by the damper drive device 48.

As yet another example, sound generation by the vibrator 47 may be used while the stopper 44 is controlled to the retracted position. In this case, the drive signal generation unit 145 may prevent the key 12 and the pedal 13 from moving without generating the key drive signal and the pedal drive signal. In this way, while the sound generated by the user's performance operation includes the sound of string hitting, the sound generated by the performance at another communication base can be the sound generated by the vibrator 47 with less delay. Although a plurality of examples have been described regarding combinations of performance data and sound generation methods to be transmitted, which one to select may be set by operating the operation panel 23.

The environmental data generation unit 121 generates environmental data indicating the surrounding environment based on the collection signal output from the environment collection device 82. In this example, the ambient environment includes images and sounds around the device. Therefore, the environment collecting device 82 includes a device for collecting the surrounding environment, that is, a camera (imaging device) for obtaining images and a microphone (sound collecting device) for obtaining sound. In this example, the camera acquires an image of a range that includes the player of the keyboard instrument 10.

The information regarding the image included in the environmental data may be image information indicating the image (video) itself, but in this example, the information regarding the image is information obtained by capturing the movements of the performer using motion capture technology. include. The sensor that measures the movement of the performer is not limited to a camera, and may include an IMU (internal measurement unit), a pressure sensor, a displacement sensor, and the like. The motion information is, for example, information about a plurality of parts having predetermined features extracted from an image and indicated by the coordinates of each part. The environmental data may be transmitted in the form of audio data representing sound signals. In this case, the motion information can be synchronized with the sound signal included in the audio data by being transmitted as data on a predetermined channel in the audio data. Similarly, environmental data may be sent as part of existing data, such as in a format indicating pronunciation control information (for example, MIDI format) or in video data format, and then sent as part of other data. It may also be transmitted to a communication base.

The sounds collected by the environment collection device 82 may include sounds (piano sounds) generated by playing the keyboard instrument 10. The period during which the sound produced by playing the keyboard instrument 10 exists can be specified from the key position signal or the like. If the sound produced by the performance is produced by the vibrator 47, the sound can be identified by the sound source section 25. Therefore, when generating the environmental data, the environmental data generating section 121 may perform signal processing to cancel the sound component generated by the sound source section 25 from the sound included in the collected signal.

Even if the sound generated by the performance is a string striking sound, the environmental data generation unit 121 may perform signal processing to cancel the string striking sound component from the sound included in the collected signal. The string striking sound component may be generated by the sound source section 25 using a key position signal and a pedal drive signal. The environmental data generation unit 121 may generate environmental data for a period in which sounds generated by playing the keyboard instrument 10 exist without using the sounds included in the collected signals. At this time, the environment collecting device 82 may recognize the period during which the performance is being performed, and may not collect sounds during that period.

The environmental data transmitting unit 123 transmits the environmental data generated by the environmental data generating unit 121 to other communication bases.

The environmental data receiving unit 183 receives environmental data transmitted from other communication bases.

The control signal generating unit 185 generates a control signal used in the environment providing device 88 based on the environmental data received by the environmental data receiving unit 183. This control signal is a signal for reproducing information about the surrounding environment included in the environmental data. In this example, a signal for displaying an image on a display (display device) and a sound are sent from a speaker (sound emitting device). Contains signals for output. Therefore, the environment providing device 88 includes a display for displaying images and a speaker for outputting sound. The display may be placed at a position where the player can easily see the keyboard lid 11 of the keyboard instrument 10, the music stand, or the like. When displaying an image on the keyboard lid 11, a projector that projects an image onto the keyboard lid 11 may be used instead of the display. If a plurality of communication bases are to be communicated with, an environment providing device 88 may be provided corresponding to each communication base. In this case, the control signal supplied to the environment providing device 88 is generated based on the environmental data received from the communication base corresponding to the environment providing device 88.

The control signal generation unit 185 may generate an image imitating the performer using the motion information included in the environmental data, and generate a signal for displaying the generated image on a display. At this time, an image may be generated that emphasizes a specific part or action. The specific part may be, for example, the player's eyes, face, fingers, or the like. The specific motion may be, for example, a movement of the line of sight, a movement of the face, a movement of fingers during a musical performance, or the like. The control signal generation unit 185 generates an image such as a graph that numerically represents the movement of the performer using the motion information included in the environmental data, and generates a signal for displaying the generated image on a display. Good too. The performer can use the displayed information to tailor the performance.

The control signal generation unit 185 may generate a signal for displaying an image on the display based on the performance data received by the performance data reception unit 143. The image based on the performance data may include an image showing the performance content included in the performance data, for example, an image showing keys and pedals being operated.

In this way, according to the ensemble control function 100 in the first embodiment, the time difference in pronunciation between communication bases can be reduced, allowing users to feel closer to each other's surrounding environments. Therefore, a plurality of musicians playing together can feel a sense of unity.

<Second embodiment>
The performance content included in the performance data is not limited to indicating performance operations on the keys 12 and the like. In the second embodiment, an example will be described in which the performance data includes a signal indicating the vibration of the soundboard 17 to which the string-striking sound caused by the performance is transmitted. The vibration of the soundboard 17 is measured by a pickup sensor included in the sensor 30 in this example.

FIG. 5 is a diagram illustrating the positional relationship between the vibrator and the pickup sensor in the second embodiment. FIG. 5 is a diagram of the keyboard instrument 10 viewed from below. As shown in FIG. 5, the soundboard 17 is provided with two vibrators 47 (vibrators 47H and 47L). The vibrators 47H and 47L are connected between the plural sound bars 17a of the sound board 17. The vibrator 47H is provided at a position corresponding to the piece 16H among the two pieces 16 (pieces 16H (long piece) and 16L (short piece)). The vibrator 47L is provided at a position corresponding to the piece 16L. The bridge 16H is a bridge that supports the strings 15 on the treble side, and the bridge 16L is a bridge that supports the strings 15 on the bass side. The vibrator 47H is supported by a support portion 97H connected to the straight column 19. The vibrator 47L is supported by a support portion 97L connected to the straight column 19.

The vibrator 47 is not limited to being provided at a position corresponding to the piece 16 on the soundboard 17, but may be provided at a position away from the piece 16, or at a position corresponding to the sound bar 17a. may be provided. When provided at a position corresponding to the sound bar 17a, the vibrator 47 may be provided on the string 15 side of the sound board 17.

The pickup sensor 37H is attached to the soundboard 17 near the vibrator 47H, measures the vibration of the soundboard 17, and outputs a measurement signal indicating the measurement result. The pickup sensor 37L is attached to the soundboard 17 near the vibrator 47L, measures vibrations of the soundboard 17, and outputs a measurement signal indicating the measurement result. Therefore, the performance data that the performance data transmission section 133 transmits to other communication bases and the performance data that the performance data reception section 143 receives from the other communication bases are based on the measurement signal PU1 from the pickup sensor 37H and the performance data from the pickup sensor 37L. Contains measurement signal PU2.

FIG. 6 is a diagram illustrating the configuration of the drive signal generation section in the second embodiment. The drive signal generation section 145A generates vibration drive signals DS1 and DS2 from the measurement signals PU1 and PU2 included in the performance data received by the performance data reception section 143. The vibration drive signal DS1 is supplied to the vibrator 47H. The vibration drive signal DS2 is supplied to the vibrator 47L.

The drive signal generation section 145A includes a crosstalk processing section 1451, a sound imparting section 1453, and an amplification section 1455. The crosstalk processing unit 1451 performs a predetermined delay process and a predetermined filter process on the measurement signal PU1, and adds the processed signal to the measurement signal PU2. The crosstalk processing unit 1451 performs predetermined delay processing and predetermined filter processing on the measurement signal PU2, and adds the processed signal to the measurement signal PU1. This reduces the crosstalk components included in each of the measurement signals PU1 and PU2.

The sound imparting unit 1453 performs signal processing to impart acoustic effects such as a delay, compressor, expander, and equalizer to the measurement signals PU1 and PU2. The amplifying section 1455 amplifies the measurement signals PU1 and PU2 to output vibration drive signals DS1 and DS2 to be supplied to the vibrators 47H and 47L.

For example, if the size and shape of the soundboard 17 etc. are different between the keyboard instrument 10 at the communication base T1 and the keyboard instrument 10 at the communication base T2, a difference in the vibration mode of the soundboard 17 etc. will occur. Due to this, the sounds emitted from each keyboard instrument 10 are different. Parameters for signal processing in the crosstalk processing section 1451 and the sound imparting section 1453 are set corresponding to the different configurations of the keyboard instrument 10. As a result, even if the shapes of the keyboard instruments 10 at other communication bases are different, differences in pronunciation caused by the differences can be reduced.

When an ensemble is performed between multiple communication bases, the vibration of the soundboard 17 measured by the pickup sensors 37H and 37L is not only the vibration caused by the string striking sound, but also the vibration caused by the vibration caused by the vibration drive signals DS1 and DS2. Also includes vibrations caused by 47H and 47L. Therefore, before transmitting the performance data to another communication base, the performance data transmitter 133 performs signal processing to reduce the components of the vibration drive signals DS1 and DS2 for the measurement signals PU1 and PU2 included in the performance data. may be applied.

<Third embodiment>
As described above, when the velocity value is small, it takes time for the key driving device 42 to drive the key 12, and the movement of the key 12 is delayed compared to when the velocity value is large. As mentioned above, if the string strike sound does not occur, you can increase the velocity value, but if the string strike sound does occur, increasing the velocity value too much will significantly change the pronunciation content. Put it away. In the third embodiment, an example for reducing such changes in pronunciation content as much as possible will be described.

FIG. 7 is a diagram illustrating the relationship between velocity and delay time in the third embodiment. The horizontal axis is the velocity value, which is a value used as a calculation parameter in the drive signal generation section 145 to generate the key drive signal. The vertical axis corresponds to the delay time when the key driving device 42 drives the key 12 based on the key driving signal. The velocity value is obtained, for example, from information included in the performance data used by the drive signal generation section 145. More specifically, the velocity value may be obtained by calculation from a key position signal or a hammer position signal included in the performance data. If the performance data includes sound production control information, it may be acquired from the velocity value. In this example, the velocity takes values from "1" to "127".

As shown in FIG. 7, as the velocity decreases, the delay time increases. When the velocity becomes smaller than Vt, the delay time increases rapidly. Therefore, in this example, the drive signal generation unit 145 corrects the velocity to become larger when the velocity becomes smaller than Vt.

FIG. 8 is a diagram illustrating the relationship between velocity and correction value in the third embodiment. As shown in FIG. 8, when the velocity is a value smaller than Vt, a correction value obtained by increasing the value is used as a calculation parameter in the drive signal generation section 145. For example, when the velocity changes from "1" to "Vt", the correction value is set to change from "Va" to "Vt". In this way, the delay time can be reduced when the velocity is small. Since the correction value is larger than the value before correction, the sound will be louder than expected.On the other hand, compared to the effect of reducing the delay time, the change in sound volume has less impact on the listener. small. In order to strengthen the effect of reducing the delay time, the correction value may be set to "Vb" which is larger than "Va" when the velocity is "1".

When playing in ensemble, it is preferable to minimize the delay as much as possible. On the other hand, if the player is not performing in an ensemble but only listening to pronunciation based on performance data, it is preferable that the delay time does not change over the entire input value rather than reducing the delay. Therefore, in such a case, the drive signal generation unit 145 generates a key drive signal that is intentionally delayed to delay the timing at which the key 12 starts to be pressed in a range where the velocity value is large (a range equal to or greater than a predetermined value). A signal may also be generated. At this time, the time for delaying the timing may be gradually reduced as the input value decreases from "Vt" to "1".

<Fourth embodiment>
In the fourth embodiment, an example will be described in which different musical instruments are played at different communication bases. Here, the communication base T1 is a large hall where an orchestra can perform. The communication base T2 is a small studio like a soundproof room. In this example, an orchestra performs at the communication base T1, and a piano performs at the communication base T2. That is, there is no piano player in the orchestra at the communication base T1, but there is a piano player at the remote communication base T2.

The orchestra performance sound at the communication base T1 is transmitted to the communication base T2. The player at the communication base T2 plays the automatic performance piano 1 while listening to the performance sound received from the communication base T1. The content of the user's performance is transmitted as performance data from the automatic performance piano 1 to the automatic performance piano 1 at the communication base T2. Therefore, the automatic performance piano 1 at the communication base T2 generates sound so as to reproduce the performance at the communication base T1. That is, at the communication base T1, even if there is no piano player, the sound of the piano can be heard together with the sound of the orchestra. In the fourth embodiment, the sense of presence of the orchestral performance at the communication base T1 can be conveyed to the player of the automatic performance piano 1 at the communication base T2. The configuration for realizing this will be described in detail below.

FIG. 9 is a diagram illustrating the configuration of the environment collection device at the communication base T1 in the fourth embodiment. At the communication base T1, a conductor stand CS, a player piano 1, and a chair 50 are installed on a stage ST1 of a hall. An environment collecting device 82 including vibration measurement plates 821 and 822 and a microphone 823 is provided on the stage ST1. The vibration measurement plate 821 is placed at a location on the stage ST1 where the chair 50 is installed. Vibration measurement plate 821 measures vibrations transmitted via stage ST1 and outputs a collection signal indicating the measured vibrations. The vibration measurement plate 822 is placed at a location on the stage ST1 where the automatic performance piano 1 is installed. Vibration measurement plate 822 measures vibrations transmitted via stage ST1 and outputs a collection signal indicative of the measured vibrations. Microphone 823 is placed near chair 50 in this example. Microphone 823 collects the arriving sound and outputs a collected signal indicative of the sound.

FIG. 10 is a diagram illustrating the configuration of the environment providing device at the communication base T1 in the fourth embodiment. At the communication base T2, a player piano 1 and a chair 50 are installed on a stage ST2 of a studio. An environment providing device 88 including vibration generating plates 881 and 882 and a speaker 883 is provided on the stage ST2. The vibration generating plate 881 is placed at the location where the chair 50 is installed on the stage ST2. The vibration generating plate 881 vibrates based on the control signal. The vibration generating plate 882 is placed at the location where the automatic performance piano 1 is installed on the stage ST2. The vibration generating plate 882 vibrates based on the control signal. In this example, the speaker 883 is placed near the chair 50 (near the performer). Speaker 883 emits sound based on the control signal. When the control signals used in the vibration generating plates 881, 882 and the speaker 883 are generated, signal processing according to the vibration characteristics, signal processing for changing the transfer characteristics, etc. may be performed.

When the orchestra performs at the communication base T1, vibrations accompanying the performance are transmitted to the vibration measurement plates 821 and 822 via the stage ST1, and the sound of the performance is collected by the microphone 823. The collected signals output from each of the vibration measurement plates 821 and 822 and the microphone 823 are transmitted to the communication base T2 as environmental data. When the automatic performance piano 1 is played at the communication base T2, the content of the performance is transmitted as performance data to the communication base T1.

As a result, at the communication base T1, the automatic performance piano 1 is driven to produce sound based on the performance data from the communication base T2. That is, the automatic performance piano 1 at the communication base T1 is driven according to the performance on the automatic performance piano 1 at the communication base T2. At communication base T2, speaker 883 generates sound based on the environmental data from communication base T1. This sound is the sound collected by the microphone 823, and corresponds to the sound of an orchestra performance at the communication base T1.

Further, the vibration generating plate 881 and the vibration generating plate 882 are driven to vibrate based on the environmental data from the communication base T1. The vibration at the vibration generating plate 881 corresponds to the vibration measured by the vibration measuring plate 821 at the communication base T1. That is, vibrations that are transmitted to the chair 50 at the communication base T1 are also transmitted to the chair 50 at the communication base T2. The vibration at the vibration generating plate 882 corresponds to the vibration measured by the vibration measuring plate 822 at the communication base T1. The vibration at the vibration generating plate 882 corresponds to the vibration measured by the vibration measuring plate 822 at the communication base T1. That is, vibrations that are transmitted to the automatic performance piano 1 at the communication base T2 are also transmitted to the automatic performance piano 1 at the communication base T2. Therefore, the performer at the communication base T2 can feel as if he were playing at the communication base T1.

The vibration measurement plates 821, 822 and microphone 823 at the communication base T1 will also collect the components of the piano sound when the automatic performance piano 1 is driven. Therefore, signal processing for reducing the piano sound component is performed on the path up to the drive of the vibration generating plates 881, 882 and the speaker 883 at the communication base T2. The piano sound component can be generated from a signal for driving the automatic performance piano 1 at the communication base T1. Therefore, this signal processing may be executed by the environmental data generation unit 121, for example, when the environmental data transmitted from the communication base T1 is generated. By doing so, the influence of the performance at the communication base T2 can be reduced in the environment provided by the environment providing device 88 at the communication base T2.

<Fifth embodiment>
In the fifth embodiment, when the environment providing device 88 displays images of performers at other communication bases, it also displays its own image (the image of the performer transmitted to the other communication bases). I will explain about it.

FIG. 11 is a diagram illustrating the configuration of the control signal generation section in the fifth embodiment. The control signal generation section 185B in the fifth embodiment includes a self-portrait acquisition section 1851, a remote image acquisition section 1853, and an image composition section 1855.

The self-portrait acquisition unit 1851 acquires self-portrait information regarding images including the performer based on the collection signal output from the environment collection device 82. Based on the environmental data received by the environmental data receiving section 183, the remote image acquisition section 1853 acquires remote image information regarding images including the performer collected by the environment collection device 82 of another communication base. Both the self-portrait information and the remote image information are images including the player and the keyboard portion of the keyboard instrument 10.

The image composition unit 1855 generates a composite image based on self-portrait information and remote image information. The composite image is an image in which an image region of the performer included in the remote image information is extracted and superimposed on the image of the self-portrait information. At this time, the image synthesis unit 1855 identifies the keyboard part from the images in each of the self-portrait information and the remote image information, and determines the superimposition position of the performer's image in the remote image information so that the keyboard parts match each other. . The image synthesis unit 1855 superimposes, for example, an image obtained by applying a transformation matrix to the remote image information so as to maximize the cross-correlation between the keyboard parts, on the image of the self-portrait information. The image composition unit 1855 generates a control signal for displaying the composite image and outputs it to the environment providing device 88. The environment providing device 88 may be a display that displays a composite image on a display, or may be a projector that projects at least a portion of an image of remote image information onto a keyboard portion. When projecting using a projector, a predetermined transformation matrix depending on the position of the keyboard portion may be applied to the remote image information.

In a duet in which two people play one piano, each player plays in a different range, so the positions of the players relative to the keyboard are also different. Even when a duet is divided into two communication bases and played by one player at a time, the position of the player relative to the keyboard is almost the same as when playing one piano. Therefore, the generated composite image is obtained as an image of two players playing one piano.

In this example, the image compositing unit 1855 determines whether the images of the two performers have touched each other, and if they have touched each other, the area corresponding to the contact area can be made to emit light, etc., so that the area can be identified. Modify the composite image. At this time, the contact portion that emits light may be limited to a portion (for example, the arm or hand). The performers may be made to recognize that the images of the two performers have come into contact with each other using something other than the images. For example, if the images of two performers touch each other, the seat of the chair used by the performers may be vibrated. In this case, a configuration for vibrating the seat surface of the chair is included in the environment providing device 88, and is controlled by a control signal from the control signal generation unit 185B. In this way, even if two performers perform at different communication bases, they can experience the situation as if they were actually performing at the same location.

The remote image acquisition unit 1853 may acquire the above-mentioned operation information (remote operation information) instead of remote image information. In this case, the image synthesis unit 1855 may generate an image that resembles the performer using the motion information, and synthesize it with the image of the self-portrait information to generate a composite image. The image synthesis unit 1855 takes into account the time lag (communication delay, etc.) between self-portrait information and remote image information, and when generating a composite image, delays the image of self-portrait information and then images the remote image information. Alternatively, a future predicted image corresponding to the delay time may be generated from the image of the remote image information or the remote motion information and superimposed on the image of the self-portrait information.

<Sixth embodiment>
In the sixth embodiment, a case will be described in which two communication bases exist in the same room. In this case, two player pianos 1 that can communicate with each other exist in one room. In such a case, a screen may be placed between the two automatic performance pianos 1 on which images related to the content of each performance are displayed.

FIG. 12 is a diagram illustrating an example of screen display in the sixth embodiment. A player piano 1a corresponding to the communication base T1 and a player piano 1b corresponding to the communication base T2 are arranged in one room. A screen SC on which an image is projected by a projector PJ is arranged between the automatic performance piano 1a and the automatic performance piano 1b. In this example, an image corresponding to the performance data communicated between the automatic performance piano 1a and the automatic performance piano 1b is displayed on the screen SC.

The displayed image is an image related to the sound according to the content of the performance, and in this example, it is a band-shaped image displayed at a position determined by the pitch and timing of pronunciation and with a length corresponding to the length of the sound. In FIG. 12, the band-shaped image sba is an image showing a sound according to the performance content on the automatic performance piano 1a, and the band-shaped image sbb is an image showing the sound according to the performance content on the automatic performance piano 1a. The band images sba and sbb are displayed in a flowing manner depending on the direction of communication.

That is, when a key 12 is pressed on the automatic performance piano 1a, a band-shaped image sba corresponding to the sound corresponding to the key 12 is displayed on the screen so as to move toward the automatic performance piano 1b. Here, when the band-shaped image sba reaches the automatic performance piano 1b side, a sound corresponding to the image may be generated at the automatic performance piano 1b. In this case, after receiving the performance data corresponding to the band-shaped image sba, the automatic performance piano 1b may delay the timing until the timing is reached, and then drive the key 12. The relationship between the automatic performance piano 1a and the automatic performance piano 1b remains the same even if they are replaced.

Such a projector PJ and screen SC can also be said to be an example of the environment providing device 88. In this case, the environment providing device 88 is shared by the two automatic performance pianos 1a and 1b.

<Modified example>
The present invention is not limited to the embodiments described above, and includes various other modifications. For example, the embodiments described above have been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Some modified examples will be described below. Although the first embodiment will be described as a modified example, the present invention can also be applied as a modified example of other embodiments. It is also possible to combine a plurality of modifications and apply them to each embodiment.

(1) The drive signal generation unit 145 is not limited to predicting sound production control information such as note-on from the change in the amount of depression of the key 12 indicated by the key position signal in the performance data, but also uses other information to control sound production. Information may also be predicted. For example, the drive signal generation unit 145 extracts the movement of a finger toward the key 12 from the image of the performer included in the environmental data, and determines the subsequent movement when the key 12 is pressed based on the change in the finger movement. The pronunciation control information may be predicted by estimating. The information indicating the movement of the finger is also information regarding the depression of the key 12. Therefore, the finger image or finger movement may be obtained by the sensor 30. In this case, information indicating finger movements may be transmitted as performance data.

The sensor 30 may have a configuration that detects contact with or proximity to the key 12. In this case, the performance data generation unit 131 generates and transmits performance data based on this detection result, so that the key 12 starts to be pressed at another communication base before the key 12 actually starts being pressed. can be recognized. This may improve the prediction accuracy of the pronunciation control information.

In these predictions, even if a trained model is used that is machine-trained to learn the correlation between movement history information such as the movement of the key 12 or finger movement and pronunciation control information such as note-on timing and velocity value. good. A trained model may be generated to correspond to each performer.

Predicting the sound production control information in this way is not limited to being applied to the case of controlling the automatic performance piano 1 at another communication base, but may be used for various interlocking operations. For example, the present invention can be applied to a configuration in which a keyboard device and a sound source device are connected wirelessly. For example, if a note-on is generated by pressing a key on a keyboard device and then transmitted to a sound source device, the timing of sound generation will be delayed due to communication delay. On the other hand, by transmitting the key movement to the sound source device before a note-on occurs on the keyboard device, the influence of communication delay can be reduced by predictive calculation using the movement.

(2) The key position signal generated in response to the depression of a key 12 on the automatic performance piano 1 may be used to control other keys 12 on the same automatic performance piano 1. For example, control can be performed such that, in response to the depression of a key 12, a key 12 corresponding to a tone one octave higher than that key 12 is linked. The interlocking key 12 is not limited to a tone one octave higher, but may be any predetermined tone. The predetermined tone may be determined relative to the pitch of the depressed key 12, or may be determined absolutely regardless of the pitch. At this time, by using the key position signal instead of the sound generation control information, it is possible to reduce the time difference between pressing the key 12 to be played and driving the interlocking key 12.

(3) In the control device 20, the content of the performance on the automatic performance piano 1 may be recorded. The data to be recorded may be data based on the sound production control information, or may be data corresponding to signals output from the sensor 30, such as a key position signal, a hammer position signal, and a pedal position signal.

(4) The performance data transmitting unit 133 may include in the performance data information for setting whether or not to drive the keys 12 of the player piano 1 at another communication base for each key 12 and transmit the data. . The presence or absence of driving may be set by the performer at the transmission side communication base during the performance, or may be predetermined based on a specific key or range. At the communication base on the receiving side, the automatic performance piano 1 does not drive the key 12 but drives the vibrator 47 in response to a key position signal regarding the key 12 which is set not to be driven.

(5) The environment collection device 82 may include a sensor attached to the performer, for example, a sensor that measures the performer's breathing. The player piano 1 may transmit the measurement results of the player's breathing to other communication bases as environmental data, and display information according to changes in breathing on the display of the environment providing device 88 at the other communication base. good. A performer's breathing is closely related to the performance movement. For example, just before starting a performance, performers often take a deep breath. Accordingly, when a deep inhale is measured, information indicating this or the time until the performance is considered to begin may be displayed on the display. Assuming that the time from when a player takes a deep breath to the start of a performance differs depending on the player, the time may be set to vary depending on the player. In predicting this time, a learned model may be used that is machine-learned the correlation between the timing of a deep breath and the time until the start of the performance.

(6) The environment providing device 88 may be a small movable device capable of providing various environments, and may be, for example, in a humanoid shape imitating some kind of character. For example, the environment providing device 88 may be a humanoid robot that moves its arms and hands based on control signals. The environment providing device 88 may have a shape that can be attached to the performer (a wristwatch type, a shoulder type, a neck type, etc.). The configuration that provides various environments may be the above-mentioned display or speaker, or may be a heat source, cooling source, fan, etc. for controlling the temperature, or may be a configuration that provides various environments such as room brightness, color, etc. It may also be a light, a projector, etc. for controlling a pattern or the like. The environment providing device 88 may include, for example, a structure such as a robot arm for changing the position of a heat source or the like, or by arranging a plurality of heat sources and driving one of them, the position of the heat source can be substantially changed. may be changed. The heat source may be used, for example, to recreate the position of the performer at another communication location. The environment collecting device 82 only needs to include a sensor compatible with the environment providing device 88, and may include, for example, a temperature sensor, an air volume sensor, an illuminance sensor, and the like.

(7) The environment providing device 88 may be able to localize a sound image or reproduce a predetermined sound field by including a plurality of speakers. At this time, predetermined reverb processing or filter processing such as FIR may be added to the sound signal included in the environmental data. The environment collecting device 82 may collect information for reproducing the sound field characteristics of the room and transmit the collected information to other communication bases as environmental data. Thereby, the environment providing device 88 at the communication base on the receiving side may reproduce the sound field of the room at the communication base on the transmitting side based on the information included in the environmental data. At this time, the sound field of the room at the transmitting side communication base may be more accurately reproduced by including signal processing for canceling the sound field characteristics of the room at the receiving side communication base. Processing to reproduce such a sound field may be added to the excitation drive signal.

(8) A common metronome synchronized at each communication base may be realized using sound, light, vibration, etc. When synchronizing using absolute time, for example, time information used in a satellite positioning system such as a GPS signal may be used, or a time synchronization technique based on NTP (Network Time Protocol) may be used. In this case, the BPM value may be set and the beat start timing may be determined based on time information. The BPM value may be determined based on a preset performance piece, or may be set by the performer.

Instead of using absolute time as a reference, any one of a plurality of communication bases may be used as a metronome reference. In this case, the beat position may be analyzed from the performance content at the reference communication base and used as a metronome. When beat positions are detected from the performance contents at a plurality of communication bases, the beat position that can be matched and handled by the largest number of communication bases may be used as a metronome at other communication bases. Predetermined data (data including sound production control information, sound data, video data, etc.) may be played back according to these metronomes. The predetermined data may be obtained by recording the performance. For example, a drum rhythm pattern may be played by setting a metronome.

In the case where the metronome is realized by vibration, the player may be made to recognize the beat of the metronome by vibrating the movable components of the automatic performance piano 1. For example, the drive signal generation unit 145 may generate a drive signal to move the pedal 13 a little with each beat of the metronome. If the pedal 13 is a damper pedal, the amount by which the pedal 13 is moved is so small that the damper 18 does not separate from the strings 15. The structure of the metronome that moves with each beat is not limited to the pedal 13, but may be any key 12. In this case, the key 12 is pressed down to the extent that it does not produce a string strike or a note-on. It is preferable.

(9) The transmitted performance data may include time information when transmitting the performance data. In this way, by adjusting the performance data received from a plurality of communication bases on the time axis according to the time information, it is possible to correct the deviation on the time axis due to communication delay. For example, if player piano 1 is not played, or even if it is played, the performance data is not sent to other communication bases, the performance data from multiple other communication bases may differ depending on the communication delay. Even if the performance data is received at the same timing, by shifting the performance data on the time axis so that the time information is aligned, it is possible to drive the automatic performance piano 1 assuming that the delay amount is the same.

Using this time information, it is possible to recognize the delay time of performance data arriving from each communication base. The drive signal generation unit 145 may generate the drive signal such that the longer the delay time, the smaller the velocity value. The drive signal generation unit 145 may generate more reverberation as the delay time increases. In this way, the automatic performance piano 1 can realize sound production in which the length of the delay time is an effect of the distance. In other words, the long delay time can give the listener the feeling that the performance is being performed at a distant location. For each communication base, an image may be displayed on the display that visually indicates the magnitude of the delay time. Regarding each communication base, an image visually indicating the magnitude of the delay time may be presented using AR (Augmented Reality). For example, images related to each communication base may be presented by converting the delay time into a position/distance relationship in the AR space.

(10) The control device 20 may calculate the degree of correlation by comparing performance data between a plurality of communication bases, and display the degree of correlation on the display. The degree of correlation may be calculated using signal processing or DNN (Deep Neural Network), for example. At this time, as described above, the degree of correlation may be calculated using performance data that has been adjusted so that time information is consistent between a plurality of communication bases. The control device 20 may analyze the received performance data to identify chords or beat positions, and display the identified information on the display. At this time, the most likely chord and beat position among the performance data at the plurality of communication bases may be displayed on the display. The keyboard may be illuminated with light so that the player can recognize the keys 12 that correspond to the constituent notes of this chord.

(11) The control device 20 analyzes the chord from the received performance data, and if the likelihood of the chord is higher than a predetermined value, it identifies it as the current chord. When the vibrator 47 generates sound in response to a playing operation on the key 12, the control device 20 controls the vibrator 47 not to be driven by any playing operation on the key 12 other than the sound corresponding to the chord. .

(12) The control device 20 analyzes the beat position from the received performance data, and if the likelihood of the beat position is higher than a predetermined value, it identifies it as the current beat position. When the vibrator 47 generates a sound in response to a performance operation on the key 12, the control device 20 controls the control device 20 to generate a sound when the key 12 is pressed within a predetermined time range before reaching the next predicted beat position. , the sound generation by the vibrator 47 is realized by delaying the beat position to the predicted beat position. In this way, the performance sound may be matched to the beat position.

(13) The control device 20 specifies the volume from the received performance data, and also specifies the volume of the user's performance on the automatic performance piano 1. The volume is specified, for example, by the average value of velocity over a predetermined period of time in the past. The drive signal generation unit 145 generates a key drive signal or an excitation drive signal by adjusting the volume of the received performance data so that it approaches the volume of the player's own performance. When adjusting the volume, the volume may be changed gradually rather than suddenly. In this way, the volume balance in the ensemble can be adjusted. The volume balance may be set in advance so that one of the volumes is relatively loud.

(14) When the control device 20 causes the vibrator 47 to generate sound in response to a performance operation on the key 12, the control device 20 may delay the timing of the sound generation in response to the depression of the key 12. At this time, performance data transmitted to other communication bases and performance data transmitted from other communication bases are not delayed. Thereby, the performer plays by pressing down the key 12 early in consideration of the delay time, so that the influence of communication delay on the ensemble performance can be reduced.

(15) If the player piano 1 is not installed at any communication base, but is an acoustic piano in which the sensor 30 and the drive device 40 are not installed, the control device 20 controls the sensor 30 and the drive device 40. The device 40 may not include components related to the device 40, and may be composed of a desktop computer, a tablet computer, or the like.

In this case, the control device 20 may convert the performance sound into performance data and transmit it to another communication base. The performance sounds may be collected by a microphone, and the control device 20 may generate performance data by analyzing constituent sounds included in the collected performance sounds and converting them into sound production control information. Such processing can also be applied to musical instruments other than pianos.

(16) The environment collecting device 82 may include a sensor that detects the opening and closing of the keyboard lid 11, a sensor that detects the seating of the performer on the chair, and the like. In this case, the environment providing device 88 may have a display that displays the opening and closing of the keyboard lid 11 and the seating of the performer on the chair. The environment providing device 88 may have a structure that opens and closes the keyboard lid 11 in response to a control signal. In this case, in response to the opening and closing of the keyboard lid 11 at a specific communication base, the keyboard lids 11 at other communication bases may be linked.

(17) The keyboard instrument 10 in the automatic performance piano 1 is not limited to an acoustic piano such as a grand piano, but may be an electronic keyboard instrument. The electronic keyboard instrument may be a keyboard device having a structure corresponding to the keys 12, or may be a keyboard device in which the keys 12 have a sheet-like structure. In the case of a keyboard device having a sheet-like structure, it can be placed on the floor and played by stepping on it with feet, so it can be played even in situations where hands cannot be used. In the case of a keyboard device that is played with the feet, the playable range may be narrow. In such a case, a plurality of keyboard devices having different ranges set in advance may be used to allow a plurality of people to perform. In the case of a keyboard device having a sheet-like structure, it may be placed on the back side of a bedside table. In this case, a rotation mechanism may be provided in the support member that supports the side table so that either the front surface or the back surface of the side table can be switched to face the top surface.

(18) At least part of the functions of the control device 20 may be provided as a plug-in in software that implements the video conference system.

(19) The network NW that connects the communication bases may be a dedicated line realized by an optical cable or the like.

(20) The environment collecting device 82 and the environment providing device 88 may include a configuration for detachably attaching them to the automatic performance piano 1. The player piano 1 may also include a structure for attaching the environment collecting device 82 and the environment providing device 88. In this case, the environment collecting device 82 or the environment providing device 88 may be connected to the interface 26 by being attached to the automatic performance piano 1.

The above is the explanation regarding the modified example.

As described above, according to an embodiment of the present invention, the first transmitting unit transmits the first performance data including the performance content for the keyboard instrument at the first communication base to the second communication base; a first receiving section that receives second performance data; a first generating section that generates a drive signal for producing sound according to the second performance data and outputs it to the sound production device at the first communication base; There is provided a control device, wherein at least one of the first performance data and the second performance data includes a key position signal indicating a depression amount of a key on the keyboard instrument.

The sounding device may include a vibrator connected to a soundboard of the keyboard instrument. The sound generation according to the second performance data may be generated by vibration of the vibrator according to the drive signal.

The sounding device may include a key of the keyboard instrument, a hammer interlocked with the key, and a string struck by the hammer. The sound generation according to the second performance data may be generated by driving the key according to the drive signal. The drive signal may be a signal for driving the key so as to reproduce the depression amount according to the key position signal.

a second transmitting unit that acquires first environmental data according to information on the surrounding environment collected by the environment collecting device at the first communication base and transmits it to the second communication base; a second receiving unit that receives environmental data; and a second generating unit that generates a control signal for providing a surrounding environment according to the second environmental data and outputs it to the environment providing device at the first communication base. , may further include.

1, 1a, 1b: automatic piano, 10: keyboard instrument, 11: keyboard lid, 12: key, 13: pedal, 14: hammer, 15: strings, 16, 16H, 16L: bridge, 17: soundboard, 17a : Sound bar, 18: Damper, 19: Straight column, 20: Control device, 21: Control section, 22: Storage section, 23: Operation panel, 24: Communication section, 25: Sound source section, 26: Interface, 27: Bus , 30: sensor, 32: key sensor, 33: pedal sensor, 34: hammer sensor, 37H, 37L: pickup sensor, 40: drive device, 42: key drive device, 43: pedal drive device, 44: stopper, 47, 47H, 47L: Vibrator, 48: Damper drive device, 49H, 49L: Support section, 50: Chair, 82: Environment collection device, 88: Environment provision device, 100: Ensemble control function, 121: Environment data generation section, 123: Environmental data transmitting section, 131: Performance data generating section, 133: Performance data transmitting section, 143: Performance data receiving section, 145, 145A: Drive signal generating section, 183: Environmental data receiving section, 185, 185B: Control signal Generation unit, 821, 822: Vibration measurement plate, 823: Microphone, 881, 882: Vibration generation plate, 883: Speaker, 1000: Server, 1451: Crosstalk processing unit, 1453: Sound imparting unit, 1455: Amplification unit, 1851 : Self-portrait acquisition unit, 1853: Remote image acquisition unit, 1855: Image composition unit

Claims

a first transmitting unit that transmits first performance data including performance content for a keyboard instrument at the first communication base to a second communication base;
a first receiving unit that receives second performance data from the second communication base;
a first generation unit that generates a drive signal for producing sound according to the second performance data and outputs it to a sound production device in the first communication base;
including;
At least one of the first performance data and the second performance data includes a key position signal indicating a pressed amount of a key on the keyboard instrument.
Control device.
The sounding device includes an exciter connected to a soundboard of the keyboard instrument,
The sound generation according to the second performance data is generated by vibration of the vibrator according to the drive signal,
The control device according to claim 1.
The sounding device includes a key of the keyboard instrument, a hammer interlocked with the key, and a string struck by the hammer,
The sound generation according to the second performance data is generated by driving the key according to the drive signal,
The control device according to claim 1, wherein the drive signal is a signal for driving the key so as to reproduce the depression amount according to the key position signal.
a second transmission unit that acquires first environmental data according to information on the surrounding environment collected by an environment collection device at the first communication base and transmits it to the second communication base;
a second receiving unit that receives second environmental data from the second communication base;
a second generation unit that generates a control signal for providing a surrounding environment according to the second environment data and outputs the control signal to an environment providing device at the first communication base;
The control device according to any one of claims 1 to 3, further comprising the following.