JP2022063777A - Performance information prediction device, effective string vibration determination model training device, performance information generation system, performance information prediction method, and effective string vibration determination model training method - Google Patents

Abstract

To provide a technique capable of precisely converting the performance of an electronic stringed instrument into a piece of performance information.SOLUTION: A performance information prediction device in an embodiment includes: a pre-processing unit that generates a piece of volume-pitch featured data including volume information and pitch information of each string from string vibration waveform data representing the performance of a stringed instrument; and a performance information prediction unit that predicts the performance information of the performance of the stringed instrument from the volume-pitch featured data using a trained effective string vibration determination model.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a performance information prediction device, an effective string vibration determination model training device, a performance information generation system, a performance information prediction method, and an effective string vibration determination model training method.

A guitar controller (a guitar controller) that converts the string vibration waveform of an instrument such as a guitar into an electric signal by a magnetic pickup or a piezo pickup, and analyzes the pitch and volume to convert it into digital performance data such as a MIDI (Musical Instrumental Digital Interface) message. Or there is an electronic musical instrument called a guitar synthesizer). Unlike a dedicated guitar controller that only sounds a sound source, this type of controller retains the shape and functions of a normal guitar, but by mounting an independent pickup on each string for acquiring performance information, MIDI performance is performed. It can be said that it is the most common form because it has a great merit that it can be used.

Japanese Unexamined Patent Publication No. 9-6339 Japanese Unexamined Patent Publication No. 2000-105590

One of the major problems that has not been solved for many years in such non-keyboard instruments is that string vibrations unintentionally generated by the performer are also converted into performance information without distinction. For example, when playing while holding down the 3rd string, the palm of the hand touches the open 4th string and it vibrates slightly, or when the 1st string is picked, the momentum is 2 The fact that the strings also come into contact with each other and vibrate for a short period of time often occurs even in the performances of highly skilled performers. It would be ideal if all unnecessary string vibrations could be muted, leaving only the necessary string vibrations, but there are many cases where it is almost impossible depending on the performance.

There are also problems peculiar to the guitar controller, and these phenomena become more serious when used as a guitar controller for the following reasons, unlike the performance by the live sound or the electric sound of a normal guitar.

Case 1: Performance with a narrow dynamic range Even with an electric guitar, the volume may be intentionally adjusted to a different volume, but rather, like a distortion sound, it can be pronounced at the same volume regardless of the volume of the performance sound. Often preferred. However, in a normal electric guitar, the signals of all strings are mixed and distorted, so the sound of a small open string is drowned out by the sound of a string with a large amplitude, so this is not a problem. However, when playing with a guitar controller and playing a synthesizer, the problem becomes much bigger. It is not uncommon for synthesizer sounds to be unbalanced, and when playing a guitar synthesizer that has an independent sound source for each string, the sound of the open strings with slight string vibration and the sound of the vibration of the normally played strings are the same. Therefore, the musical sound of unnecessary string vibration is reproduced at the same volume as the music sound that was intentionally played, which greatly hinders the performance.

Case 2: Display and performance analysis The guitar controller not only plays the sound source, but also displays the performance on the score, discriminates the chords, inputs the phrase into the computer to create song data, and so on. May be used. At this time, if completely unnecessary note information is mixed in, the score may be very difficult to see, or it may be interpreted as a completely wrong chord, which may be a fatal problem. In order to solve this, it is necessary to individually remove unnecessary information from the recorded performance information, which may take a lot of time and effort.

Case 3: Monophonic mode There are multiple strings on a guitar and you can play chords, but there are cases where you want to play monophonically (single note) for solo or melody performance. In particular, he sometimes plays legato phrases that move smoothly to another pitch with a portamento effect. Many keyboard instruments are polyphonic keyboard instruments that can play chords for the same reason, and have a monophonic mode that dares to generate only a single note. Some guitar synthesizers have a built-in sound source that supports such a monophonic mode, and some guitar synthesizers drive an external monophonic sound source by transmitting a MIDI signal to the outside. Just as even if you press multiple keys, only one key is valid, even if multiple strings vibrate at the same time in the monophonic mode of the guitar, only one string vibration is valid. In the case of a keyboard, one note is generally determined by rules such as high-pitched tone priority, low-pitched tone priority, and late-arrival priority, but in the case of a guitar controller, there are many cases where late-arrival priority is given. However, in the case of a guitar, there are cases where notes with pitches that are far from each other are inadvertently generated compared to keyboard instruments, so if unnecessary notes are mixed in a single note phrase, the performance phrase will be greatly destroyed. .. In other words, another pronunciation is muted and replaced by the inadvertent pronunciation of the note. Even more influential is the performance when the portamento effect mentioned above is applied. For example, when playing using the high position of the 1st and 2nd strings, it is common for the sound of the open string of the 6th string, which is far away, to vibrate for a moment. The pitch inside suddenly moves in a completely unintended low direction, and a meaningless and large pitch change occurs between the next correct note and the return of the pitch.

Conventionally, in general, in a conventional guitar controller, a measure is taken to monitor the signal level of a string against these unnecessary string vibrations and simply invalidate a sound below a predetermined volume.

However, as a side effect of this measure, the musical tone of the playing style due to weak picking and low volume tapping is also ignored. The fact that what should be pronounced is not pronounced is a serious problem, and the current situation is that it is a major obstacle to high-speed phrase performance.

In view of the above problems, an object of the present disclosure is to provide a technique for converting the performance of an electronic stringed instrument into performance information with high accuracy.

In order to solve the above problems, one aspect of the present disclosure includes a preprocessing unit that generates volume pitch characterization data including volume information and pitch information of each string from string vibration waveform data representing stringed instrument performance, and a trained effective string. The present invention relates to a performance information prediction device including a performance information prediction unit that predicts performance information of the stringed instrument performance from the volume pitch characterization data using a vibration determination model.

According to the present disclosure, the performance of an electronic stringed instrument can be converted into performance information with high accuracy.

It is a schematic diagram which shows the guitar controller by one Example of this disclosure. It is a figure which shows the structure of the performance information by one Example of this disclosure. It is a figure which shows the tablature by one Example of this disclosure. It is a figure which shows the appearance of the guitar controller by one Example of this disclosure. It is a block diagram which shows the hardware composition of the guitar by one Example of this disclosure. It is a block diagram which shows the hardware composition of the control device by one Embodiment of this disclosure. It is a block diagram which shows the functional structure of the performance information prediction apparatus and the effective string vibration determination model training apparatus by one Embodiment of this disclosure. It is a schematic diagram which shows the volume pitch characteristic data frame by one Example of this disclosure. It is a schematic diagram which shows the volume detection and the pitch detection by one Example of this disclosure. It is a schematic diagram which shows the performance information generation processing by one Example of this disclosure. It is a figure which shows the architecture of the effective string vibration determination model by one Example of this disclosure. It is a figure which shows the architecture of the effective string vibration determination model by another Example of this disclosure. It is a flowchart which shows the performance information prediction processing by one Example of this disclosure. It is a schematic diagram which shows the operation of the effective string vibration determination model training apparatus by one Example of this disclosure. It is a flowchart which shows the effective string vibration determination model training process by one Example of this disclosure.

In the following examples, a guitar controller that generates performance information (for example, a MIDI message) from a string vibration waveform generated by playing a guitar is disclosed. The present disclosure is not limited to the guitar controller, and may be applied to any other performance information generation device that generates performance information from the performance of a stringed instrument having a string vibration waveform extraction function.
[Summary of this disclosure]
To outline the examples described below, as shown in FIG. 1, the guitar controller 10 according to the embodiment of the present disclosure includes a guitar 50 and a control device 100. The guitar controller 10 uses an effective string vibration determination model realized as a machine learning model such as a neural network to generate performance information from the string vibration waveform generated by the performance of the guitar 50.

As shown in FIG. 2, the performance information according to the embodiment of the present disclosure indicates the performance type of the pronunciation information, the muffling information, and the pitch change information.

The pronunciation information indicates the intentional string repellency determined by the effective string vibration determination model as a classification model and the strength of the string repellency by envelope detection. When the pronunciation is expressed by a MIDI message, the note number of the intentional string repellent and the strength of the string repellent may be expressed by Note On: 0x9n, kk, vv.

Further, the muffling information is detected by the envelope detection and represents 0) sound stop and 1) replacement. When the muffling is expressed by a MIDI message, it may be expressed by Note Off: 0x8n, kk, vv.

In addition, the pitch change information is detected by zero cross count. When the pitch change is expressed by a MIDI message, it may be expressed by Pitch Bend: 0xEn, ll, mm.

In the embodiment shown in FIG. 1, the guitar controller 10 has two operations, a training mode for training an effective string vibration determination model and a performance mode for predicting performance information using the trained effective string vibration determination model. The control device 100 has a mode, an effective string vibration determination model training device 200 used in the training mode, and a performance information prediction device 300 used in the performance mode.

First, in the training mode, the guitar controller 10 acquires training data from the training performance information database 80. The training data is composed of, for example, a pair of musical score data (for example, TAB score) and a MIDI file corresponding to the musical score data. The tablature may be, for example, written according to a well-known notation as shown in FIG. When the guitar 50 is played according to the score of the training score data acquired by the user, the effective string vibration determination model training device 200 uses the string vibration information generated by the guitar 50 based on the user's performance as the effective string vibration determination model to be trained. The MIDI message as performance information output from the effective string vibration determination model is compared with the training MIDI file, and the effective string vibration determination model is trained so that these errors are reduced. In the present disclosure, performance information is acquired from the string vibration determined to be valid by the effective string vibration determination model by using the volume pitch characteristic data characterized based on the volume and pitch extracted from the string vibration waveform data. .. When the training is completed, the effective string vibration determination model training device 200 provides the trained effective string vibration determination model to the performance information prediction device 300.

Next, in the performance mode, when the user plays the guitar 50, the performance information prediction device 300 inputs the string vibration information generated by the guitar 50 based on the user's performance into the trained effective string vibration determination model, and inputs the MIDI message. Acquire performance information such as. The acquired performance information is transmitted to, for example, an external playback device or a computer, and the user can play back the performance by the user via the playback device or use the performance information on the computer.

This makes it possible to eliminate unnecessary string vibration when converting the performance of an electronic stringed instrument into performance information, and to generate performance information based on effective string vibration.

The guitar controller 10 according to the embodiment described below has an effective string vibration determination model training device 200, but the present disclosure is not limited to this. For example, the effective string vibration determination model is based on an external computer or server. Trained and trained effective string vibration determination model and / or update information of the effective string vibration determination model may be provided to the performance information prediction device 300 from an external computer or server.
[Hardware configuration]
Next, the physical configuration of the guitar controller 10 will be described with reference to FIG. FIG. 4 is a diagram showing the appearance of the guitar controller 10 according to the embodiment of the present disclosure.

As shown in FIG. 4, the guitar controller 10 is a separate type performance information generation system including an interconnected guitar 50 and a control device 100.

The guitar 50 is a normal electric guitar, a hexadivided pickup for picking up the independent vibration of each of the six strings, a MIDI volume for controlling the volume of performance information, and a patch memory for the control device 100. It is equipped with an up / down switch for switching the number up and down. This information and the output of the normal pickup are transmitted to the control device 100 by the multi-cable. Further, the power supply is supplied from the control device 100 via the multi-cable. The hexadivided pickup of this embodiment is the same magnetic pickup as the normal pickup.

On the other hand, the control device 100 receives the input of the string vibration of the guitar and generates the performance information in the MIDI format. The destination of the performance information may be a sound source unit, a computer, or the like without limitation. As shown in FIG. 1, the control device 100 includes a foot switch for switching a bank number and a number of a patch memory storing various settings, a CONTROL switch capable of assigning and transmitting an arbitrary performance message, and a foot pedal. The currently selected patch memory number is displayed on the BANK / NUM screen. There is an LCD as the main display device, but a touch panel is mounted on the screen. In addition, a rotary encoder for inputting data is also installed on the panel. As terminals, the input terminal of the multi-cable from the guitar 50, GUITAR INPUT, the audio output terminal of the normal pickup, GUITAR OUT, the output terminal of the MIDI performance signal, MIDI OUT, the connection terminal with the host computer, USB to HOST, and the AC power input terminal AC POWER. Be prepared.

Next, with reference to FIG. 5, the hardware configuration of the guitar 50 according to the embodiment of the present disclosure will be described. FIG. 5 is a block diagram showing a hardware configuration of the guitar 50 according to an embodiment of the present disclosure.

As shown in FIG. 5, in the guitar 50, the signal through the buffer amplifier of the hexadivided pickup, the MIDI volume control, the up / down switch of the patch memory, and the signal of the normal pickup are transmitted to the control device 100 by a multi-cable. Will be done. The three normal pickups are selected by the pickup selector, and those that have passed through the buffer amplifier via the tone control circuit and the volume control circuit are transmitted to the control device 100.

Next, with reference to FIG. 6, the hardware configuration of the control device 100 according to the embodiment of the present disclosure will be described. FIG. 6 is a block diagram showing a hardware configuration of a control device according to an embodiment of the present disclosure.

As shown in FIG. 6, the control device 100 is composed of a CPU (Central Processing Unit) and a DSP (Digital Signal Processor), the CPU manages the functions and processing of the entire control device 100, and the DSP requires high-speed processing. Perform various waveform analysis processes. The RAM used by the CPU, the Flash ROM, the LCD controller that controls the LCD, the I / O interface connected to various I / O devices, the DSP, the USB interface, and the MIDI interface are connected to the bus of the CPU. Further, the I / O interface includes a foot switch, a rotary encoder, an LCD touch panel, a MIDI volume of the guitar 50, an A / D converter for detecting the position of the pedal of the control device 100, and an LED for displaying the number of the patch memory. Be connected. Although only one A / D converter is shown, the value is read by switching the input source in time division by a multiplexer. An independent A / D converter for converting the output of the six strings of the hex divided pickup into a high-speed digital signal is connected to the DSP to which the dedicated RAM and Flash ROM are connected, and high-speed analysis processing is performed. It can be performed.

However, the guitar 50 and the control device 100 are not limited to the hardware configuration described above, and may be realized by any other appropriate hardware configuration.
[Performance information prediction device]
Next, the performance information prediction device 300 according to the embodiment of the present disclosure will be described with reference to FIGS. 7 to 12. The performance information prediction device 300 predicts performance information (for example, a MIBI file) from the performance of the guitar 50 by the performer by using the effective string vibration determination model trained by the effective string vibration determination model training device 200. Specifically, when the performance information prediction device 300 acquires string vibration waveform data representing a guitar performance from the guitar 50, it executes volume detection and pitch detection for the acquired string vibration waveform data frame of each string and detects them. The volume pitch featured data including the volume information and pitch information of each string is acquired. Then, the performance information prediction device 300 inputs the acquired volume pitch characteristic data into the trained effective string vibration determination model, and determines the effectiveness of the string repellent of each string. The performance information prediction device 300 generates muffling information and pitch change information based on the volume and pitch of each string determined to be effective, and velocity information indicating the strength (speed) of the string repellent based on the volume. Is generated and added to the pronunciation information. In this way, the performance information prediction device 300 generates performance information (for example, a LIMITED message) at each time including sounding information, mute information and / or pitch change information from the volume pitch featured data at each time. Send to an external device (eg, playback device, computer, etc.). For example, these performance information generation processes may be realized by the CPU.

FIG. 7A is a block diagram showing a functional configuration of the performance information prediction device 300 according to the embodiment of the present disclosure.

As shown in FIG. 7A, the performance information prediction device 300 has a preprocessing unit 310 and a performance information prediction unit 320.

The preprocessing unit 210 generates volume pitch characteristic data including volume information and pitch information of each string from string vibration waveform data representing stringed instrument performance. Specifically, when the guitar 50 is played by the performer, the guitar 50 acquires string vibration waveform data indicating the time and the amplitude of each string and transmits it to the performance information prediction device 300. That is, since the guitar 50 is composed of 6 strings, 6 types of string vibration waveform data are generated. For example, string vibration waveform data can be acquired with a sampling rate of 20 KHz (50 μsec intervals) and a data word length of 16 bits, which is sufficient for string vibration analysis.

The preprocessing unit 310 executes volume detection and pitch detection on the string vibration waveform data of each string, and acquires the volume envelope and pitch envelope for each string as shown in FIG. For example, for volume detection, the preprocessing unit 310 converts the string vibration waveform data into an absolute value, performs low-pass filtering, and sets the amplitude level of the filtered waveform at the reference time, as shown in FIG. 9A. It may be determined as the volume. If the trained effective string vibration determination model does not detect a valid string repellent for the reference time, or if the detected volume is below a predetermined threshold value that is recognized as a mute state, the preprocessing unit 310 , Judges that there was no pronunciation, and outputs mute information as performance information. If not, the preprocessing unit 310 determines that there is a pronunciation, sets the detected volume as the velocity value of the pronunciation, and includes the velocity value together with the note number output from the effective string vibration determination model in the pronunciation information. ..

Further, regarding pitch detection, as shown in FIG. 9B, the preprocessing unit 320 executes high-pass filtering and low-pass filtering on the string vibration waveform data, counts zero cross points, and generates pitch information. .. Then, when there is a difference from the latest pitch information or pronunciation information, the preprocessing unit 320 determines that the pitch has been changed and outputs the pitch change information.

The preprocessing unit 310 executes the above-mentioned processing for all strings in parallel, detects string vibration in a predetermined processing cycle s as shown in FIG. 8, and overlaps string vibrations with respect to the time axis. Generate a waveform data frame. As shown in FIG. 8, the preprocessing unit 310 generates a volume pitch characteristic data frame including volume information and pitch information of each string.

The performance information prediction unit 320 predicts the performance information of the stringed instrument performance from the volume pitch characterization data by using the trained effective string vibration determination model. Specifically, the performance information prediction unit 320 inputs the volume pitch featured data frame generated by the preprocessing unit 310 into the trained effective string vibration determination model, and determines whether each string is valid or invalid. For example, the trained effective string vibration determination model inputs a volume pitch featured data frame at a reference time, p volume pitch featured data frames immediately before the reference time, and n volume pitch featured data frames immediately after the reference time. To determine the validity information indicating the validity or invalidity of each string at the reference time. Here, the predetermined numbers p and n may be the same or different predetermined values. For example, it is preferable that the predetermined numbers p and n are set to a value such that the time lag between the string repelling of the guitar 50 by the performer and the output of the performance information in the performance information prediction device 300 cannot be recognized by the performer. By using the volume pitch featured data frame in a certain time range in this way, it is possible to determine whether each string is the intended string repellent in consideration of the anteroposterior relationship between the frames, and it is possible to determine whether or not each string is the intended string repellent. It becomes possible to judge the time change.

When the effectiveness of each string is determined in this way, as shown in FIG. 10, the performance information prediction unit 320 determines that the string vibration at the reference time is effective in the performance information prediction unit 320. For, the volume detected for the string is determined as the volume of the string, and for the string determined that the string vibration at the reference time is invalid, the volume detected for the string is set to zero. Then, the performance information prediction unit 320 generates performance information by configuring sound information only for valid string vibrations and configuring pitch change information according to the detected pitch.

If there is no sound at the reference time, that is, if the reference time is in the mute state, the effective string vibration determination model may be trained to output invalidity for all strings.

In one embodiment, the effective string vibration determination model may be realized by a neural network. For example, the effective string vibration determination model may be a neural network having a network architecture as shown in FIG. In this case, the performance information prediction unit 320 inputs the volume pitch featured data frame of the reference time and the (p + n) volume pitch featured data frames before and after the reference time into the input layer of the neural network, and in the intermediate layer. The value indicating the validity or invalidity of each string is obtained from the output layer through the calculation.

Further, in another embodiment, the effective string vibration determination model may be realized by a recurrent neural network as shown in FIG. In this case, the performance information prediction unit 320 may use the volume pitch featured data frame having a time range wider than the time range of p and n described above. For example, the volume pitch featured data frame at the reference time t, the reference. The b (b> p) volume pitch featured data frames immediately before the time and the f (f> n) volume pitch featured data frames immediately after the reference time are input to the input layer _Xt-b of the recursive neural network. ... It may be input to X _t-1 , X _t , X _{t + 1} , ..., X _{t + f} , and a value indicating the validity or invalidity of each string may be obtained from the output layer through an operation in the intermediate layer. Recurrent neural networks are suitable for processing time-series data, and it is considered that the effectiveness of each string can be predicted with high accuracy from the volume pitch featured data frame in a certain time range.
[Performance information prediction processing]
Next, with reference to FIG. 13, the performance information prediction process according to the embodiment of the present disclosure will be described. The performance information prediction process is realized by the performance information prediction device 300 described above, and may be realized, for example, by the processor executing a program or an instruction. FIG. 13 is a flowchart showing a performance information prediction process according to an embodiment of the present disclosure.

As shown in FIG. 13, in step S101, the performance information prediction device 300 initializes the string number s to 0. Since the guitar 50 is composed of 6 strings, the string number s takes a value of 0 to 5.

In step S102, the performance information prediction device 300 detects the current volume l [s] of the strings s from the string vibration waveform data. For example, the performance information prediction device 300 may apply absolute value conversion and low-pass filtering to the string vibration waveform data to detect the volume l [s].

In step S103, the performance information prediction device 300 detects the current pitch p [s] of the strings s from the string vibration waveform. For example, the performance information prediction device 300 may apply high-pass filtering and low-pass filtering to the string vibration waveform data to detect the pitch p [s].

In step S104, the performance information prediction device 300 increments the string number s by 1.

In step S105, the performance information prediction device 300 determines whether or not the volume and pitch have been detected for all the strings.

In step S106, the performance information predictor 300 stores the current volumes l [0] to l [5] and pitches p [0] to p [5] of each of the six strings.

In step S107, the performance information prediction device 300 trains the volume pitch featured data frame at the reference time, p volume pitch featured data frames immediately before the reference time, and n volume pitch featured data frames immediately after the reference time. Input to the completed effective string vibration judgment model.

In step S108, the performance information prediction device 300 stores the determination results for each string in a [0] to a [5], respectively.

In step S109, the performance information prediction device 300 gives the digital signal processor (DSP) string vibration effectiveness information a [0] to a [5] indicating the determination result of each string, and controls the level signal. That is, the DSP may maintain the string vibration level for strings whose determination result is effective string vibration, and may set the string vibration level to silence for strings whose determination result is not effective string vibration.

In step S110, the performance information prediction device 300 resets the string number s to 0.

In step S111, the performance information prediction device 300 stores the volume information output by the DSP in l [s].

In step S112, the performance information prediction device 300 stores the pitch information output by the DSP in p [s].

In step S113, the performance information prediction device 300 determines whether the string s is being pronounced. When the string s is not being pronounced (S113: No), the performance information prediction device 300 proceeds to step S117. On the other hand, when the string s is being pronounced (S113: Yes), the performance information predictor 300 determines in step S114 whether the string vibration of the string s is invalid, that is, the string vibration effectiveness information of the string s. Is determined to be a [s] = 0.

When the string vibration of the string s is invalid (S114: Yes), the performance information prediction device 300 proceeds to step S116. On the other hand, when the string vibration of the string s is effective (S114: No), the performance information prediction device 300 determines in step S115 whether the volume l [s] is less than the predetermined minimum sounding volume.

In step S115, the performance information prediction device 300 determines whether the volume l [s] is higher than the predetermined pronunciation level. When the volume l [s] is larger than the predetermined pronunciation level (S115: Yes), the performance information prediction device 300 generates muffling information for the current pronunciation of the string s in step S116. On the other hand, when the volume l [s] is equal to or lower than the predetermined pronunciation level (S115: No), the performance information prediction device 300 proceeds to step S117.

In step S117, the performance information prediction device 300 determines whether l [s] is larger than the pronunciation level. When l [s] is larger than the pronunciation level (S117: Yes), the performance information prediction device 300 proceeds to step S118. On the other hand, when l [s] is equal to or lower than the pronunciation level (S117: No), the performance information prediction device 300 proceeds to step S122.

In step S118, the performance information prediction device 300 determines whether the string vibration effectiveness information of the string s is a [s] = 0. When the string vibration effectiveness information of the string s is a [s] = 0 (S118: Yes), the performance information prediction device 300 proceeds to step S122. On the other hand, when the string vibration effectiveness information of the string s is not a [s] = 0 (S118: No), the performance information prediction device 300 calculates the note number k based on p [s] in step S119.

In step S120, the performance information prediction device 300 calculates the velocity information v based on l [s].

In step S121, the performance information prediction device 300 generates pronunciation information of the note number k and the velocity information v.

In step S122, the performance information prediction device 300 determines whether the pitch p [s] matches the pitch P0 [s] of the previous string s. If they match (S122: Yes), the performance information prediction device 300 proceeds to step S125. On the other hand, if they do not match (S122: No), the performance information prediction device 300 obtains the difference and generates pitch bend information.

In step S124, the performance information prediction device 300 updates P0 [s] with p.

In step S125, the performance information prediction device 300 increments the string number s by 1.

In step S126, the performance information prediction device 300 determines whether all the strings have been processed, and if all the strings have not been processed (S126: Yes), returns to step S111 and all the strings have been processed. (S126: No), the process is terminated.
[Effective string vibration judgment model training device]
Next, the effective string vibration determination model training device 200 according to the embodiment of the present disclosure will be described with reference to FIGS. 14 and 7B. FIG. 14 is a schematic view showing the operation of the effective string vibration determination model training device 200 according to the embodiment of the present disclosure.

As shown in FIG. 14, the effective string vibration determination model training device 200 trains the effective string vibration determination model using the training data stored in the training performance information database 80 that stores the training data. Specifically, the training data is composed of a pair of the training score data and the training performance information corresponding to the score data, and plays the score displayed based on the training score data (for example, TAB score). The performer plays the guitar 50 under tempo control by the metronome. The string vibration waveform data representing the performance is provided to the effective string vibration determination model training device 200, and the effective string vibration determination model training device 200 performs volume detection and pitch detection on the acquired string vibration waveform data of each string. Executes to generate a volume pitch featured data frame consisting of the volume and pitch of each string. Then, the effective string vibration determination model training device 200 inputs the volume pitch characteristic data frames at the reference time and before and after the reference time into the effective string vibration determination model to be trained. Then, the effective string vibration determination model training device 200 compares the output from the effective string vibration determination model with the pronunciation information (for example, pronunciation information and muffling information) of the training performance information, and the effective string according to the error. Update the parameters of the vibration judgment model. The effective string vibration determination model training device 200 repeats the above-mentioned processing until a predetermined end condition is satisfied, and the effective string vibration determination model is such that the output from the effective string vibration determination model approaches the pronunciation information of the training performance information. Optimize.

FIG. 7B is a block diagram showing a functional configuration of an effective string vibration determination model training device 200 according to an embodiment of the present disclosure.

As shown in FIG. 7B, the effective string vibration determination model training device 200 has a preprocessing unit 210 and an effective string vibration determination model training unit 220.

The preprocessing unit 210 generates volume pitch characterization data composed of the volume and pitch of each string from the string vibration waveform data representing the performance of the stringed instrument played according to the training performance information. Specifically, when the guitar 50 is played by the performer, the guitar 50 acquires the string vibration waveform data indicating the time and the amplitude of each string and transmits it to the effective string vibration determination model training device 200. That is, since the guitar 50 is composed of 6 strings, 6 types of string vibration waveform data are generated.

The preprocessing unit 210 executes volume detection and pitch detection for the string vibration waveform data of each string, and acquires volume pitch characteristic data composed of the detected volume and pitch of each string. For example, the preprocessing unit 210 extracts a string vibration waveform frame having a window width w that overlaps with respect to the time axis from the string vibration waveform data, executes volume detection and pitch detection for each sampling, and sets each string vibration waveform frame to the volume pitch. It may be converted into a characterized data frame. The preprocessing unit 210 generates volume pitch characterization data and provides it to the effective string vibration determination model training unit 220.

The effective string vibration determination model training unit 220 uses the performance information for training to obtain a volume pitch featured data frame at a reference time and a volume pitch featured data frame before and after the volume pitch featured data frame at the reference time. To train an effective string vibration judgment model that judges the effectiveness of each string in playing a stringed instrument. Here, when predicting the performance information of the reference time of the prediction target, the effective string vibration determination model of the training target not only the volume pitch characterized data frame of the reference time but also the volume pitch of the time before and after the reference time. It takes a characterized data frame as an input and outputs the validity information of each string at the reference time. For example, the effective string vibration determination model training unit 220 has a volume pitch featured data frame at the reference time, p volume pitch featured data frames immediately before the reference time, and n volume pitch featured data immediately after the reference time. The frame may be input to the effective string vibration determination model. Here, the predetermined numbers p and n may be the same or different predetermined values. For example, it is preferable that the predetermined numbers p and n are set to a value such that the time lag between the string repelling of the guitar 50 by the performer and the output of the performance information in the performance information prediction device 300 cannot be recognized by the performer.

By using the volume pitch featured data frame in a certain time range in this way, it is possible to determine whether a new string repelling has occurred in consideration of the anteroposterior relationship between the frames, and to change the time of the string repelling. It becomes possible to judge.

Further, the effective string vibration determination model to be trained may be a pre-trained machine learning model, and the effective string vibration determination model training unit 220 is a pre-trained effective string vibration determination model by the above-mentioned training process. May be fine-tuned. This makes it possible to construct a highly accurate effective string vibration determination model with less training data than training an effective string vibration determination model from a machine learning model in the initial state.

When the effectiveness information of each string is acquired from the effective string vibration determination model, the effective string vibration determination model training unit 220 compares the pronunciation information derived based on the acquired effectiveness information with the pronunciation information of the training performance information. Then, update the parameters of the effective string vibration determination model so that they match. For example, when the effective string vibration determination model is realized by the neural network, the effective string vibration determination model training unit 220 may update the parameters of the neural network according to the comparison result according to a well-known error back propagation method. If the pronunciation information is output even though there is no pronunciation information of the training performance information, the pronunciation information is regarded as unnecessary string vibration.

The effective string vibration determination model training unit 220 repeats the above-mentioned processing until the predetermined end condition is satisfied, trains the effective string vibration determination model, and when the predetermined end condition is satisfied, the effective string at that time point. The vibration determination model is passed to the performance information prediction device 300 as a trained effective string vibration determination model. Here, the predetermined termination condition may be that all the prepared training data have been processed.
[Effective string vibration judgment model training process]
Next, with reference to FIG. 15, the effective string vibration determination model training process according to the embodiment of the present disclosure will be described. The effective string vibration determination model training process is realized by the above-mentioned effective string vibration determination model training device 200, and may be realized, for example, by executing a program or an instruction by a processor. FIG. 15 is a flowchart showing an effective string vibration determination model training process according to an embodiment of the present disclosure.

As shown in FIG. 15, in step S201, the effective string vibration determination model training device 200 selects training performance information from the training performance information database 80. Specifically, the effective string vibration determination model training device 200 may automatically select training performance information randomly, sequentially, or by user selection.

In step S202, the effective string vibration determination model training device 200 converts the performance information into the display information of the TAB staff.

In step S203, the effective string vibration determination model training device 200 displays the TAB score on the LCD of the control device 100 or the like.

In step S204, the effective string vibration determination model training device 200 starts the MIDI player according to the tempo of the performance information.

In step S205, the effective string vibration determination model training device 200 starts the metronome in accordance with the tempo.

In step S206, the MIDI player reproduces the performance information.

In step S207, the metronome reproduces the performance information. As a result, the performer is ready to acquire the performance, and the performer starts playing.

In step S208, the effective string vibration determination model training device 200 stores the pronunciation information or the muffling information of the s channel generated from the MIDI player in the memory p.

In step S209, the effective string vibration determination model training device 200 generates a volume pitch characteristic data frame from the string vibration waveform representing the performance by the performer, and trains the validity information indicating the validity or invalidity of each string at the reference time. It is input to the target effective string vibration determination model, and the sounding information or the muffling information is generated based on the determined effectiveness information and stored in the memory o.

In step S210, the effective string vibration determination model training device 200 compares the pronunciation information or muffling information of the memory p with the pronunciation information or muffling information of the effective string vibration determination model of the memory o.

In step S211th, the effective string vibration determination model training device 200 determines whether there is a difference between the pronunciation information or muffling information of the memory p and the pronunciation information or muffling information of the memory o.

When there is a significant difference (S211: Yes), the effective string vibration determination model training device 200 uses the optimization information for updating the effective string vibration determination model from the difference in the effective string vibration determination model in step S212. It is applied, and the process proceeds to step S213. On the other hand, when there is no significant difference (S211: No), the effective string vibration determination model training device 200 proceeds to step S213 without updating the effective string vibration determination model.

In step S213, the effective string vibration determination model training device 200 determines whether or not the performance has ended, and if not (S213: No), returns to step S206.

In step S214, the effective string vibration determination model training device 200 stops the metronome and the MIDI player.

In step S215, the effective string vibration determination model training device 200 determines whether or not there has been an end operation by a user or the like, and if there is no end operation (S215: No), returns to step S201, selects the next performance information, and ends the operation. If there is (S215: Yes), the process is terminated.

In the above-described embodiment, an effective string vibration determination model that predicts performance information from string vibration waveform data of a string instrument such as a guitar 50 is trained, and a performance that predicts performance information using the trained effective string vibration determination model. Although the information prediction system has been described, the present disclosure is not limited to this, and may be applied to wind instruments. That is, the present disclosure is applied to a performance information prediction system that trains an effective string vibration determination model that predicts performance information from air vibration waveform data of a wind instrument and predicts performance information using the trained effective string vibration determination model. You may.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-mentioned specific embodiments, and various modifications are made within the scope of the gist of the present invention described in the claims.・ Can be changed.

The inventions described in the original claims of the present application are described below.
[Additional Notes]
In one aspect of the disclosure,
A preprocessing unit that generates volume pitch characterization data including volume information and pitch information for each string from string vibration waveform data representing stringed instrument performance, and a preprocessing unit.
A performance information prediction unit that predicts the performance information of the stringed instrument performance from the volume pitch characterization data using the trained effective string vibration determination model.
A performance information prediction device having the above is provided.

In one embodiment, the performance information may be composed of pronunciation information, muffling information, and pitch change information.

In one embodiment, the trained effective string vibration determination model may output whether the vibration of each string is valid or invalid.

In one embodiment, the trained effective string vibration determination model may be realized by a neural network.

In one embodiment, the performance information may be described according to the MIDI protocol.

In another aspect of the present disclosure,
A preprocessing unit that generates volume pitch characterization data including volume information and pitch information for each string from string vibration waveform data representing string instrument performance performed according to training performance information, and a preprocessing unit.
An effective string vibration determination model training unit that trains an effective string vibration determination model that determines the effectiveness of vibration of each string from the volume pitch characteristic data using the training performance information.
An effective string vibration determination model training device having the above is provided.

In another aspect of the present disclosure,
With electronic stringed instruments
With the performance information prediction device described above,
With the above-mentioned effective string vibration judgment model training device,
A performance information generation system having the above is provided.

In another aspect of the present disclosure,
A step in which one or more processors generate volume pitch featured data, including volume information and pitch information for each string, from string vibration waveform data representing string instrument performance.
A step in which the one or more processors predict the performance information of the stringed instrument performance from the volume pitch characterization data using the trained effective string vibration determination model.
A performance information prediction method is provided.

In another aspect of the present disclosure,
A step in which one or more processors generate volume pitch characteristic data including volume information and pitch information of each string from string vibration waveform data representing a stringed instrument performance played according to training performance information.
A step in which the one or more processors use the training performance information to train an effective string vibration determination model that determines the effectiveness of vibration of each string from the volume pitch characteristic data.
An effective string vibration determination model training method having the above is provided.

10 Guitar controller 50 Guitar 100 Control device 200 Effective string vibration judgment model training device 210 Pre-processing unit 220 Effective string vibration judgment model training unit 300 Performance information prediction device 310 Pre-processing unit 320 Performance information prediction unit

Claims (9)

A preprocessing unit that generates volume pitch characterization data including volume information and pitch information for each string from string vibration waveform data representing stringed instrument performance, and a preprocessing unit.
A performance information prediction unit that predicts the performance information of the stringed instrument performance from the volume pitch characterization data using the trained effective string vibration determination model.
Performance information prediction device. The performance information prediction device according to claim 1, wherein the performance information is composed of pronunciation information, mute information, and pitch change information. The performance information prediction device according to claim 1 or 2, wherein the trained effective string vibration determination model outputs whether the vibration of each string is valid or invalid. The performance information prediction device according to any one of claims 1 to 3, wherein the trained effective string vibration determination model is realized by a neural network. The performance information prediction device according to any one of claims 1 to 4, wherein the performance information is described according to the MIDI protocol. A preprocessing unit that generates volume pitch characterization data including volume information and pitch information for each string from string vibration waveform data representing string instrument performance performed according to training performance information, and a preprocessing unit.
An effective string vibration determination model training unit that trains an effective string vibration determination model that determines the effectiveness of vibration of each string from the volume pitch characteristic data using the training performance information.
Effective string vibration determination model training device with. With electronic stringed instruments
The performance information prediction device according to any one of claims 1 to 5.
The effective string vibration determination model training device according to claim 6 and
Performance information generation system with. A step in which one or more processors generate volume pitch featured data, including volume information and pitch information for each string, from string vibration waveform data representing string instrument performance.
A step in which the one or more processors predict the performance information of the stringed instrument performance from the volume pitch characterization data using the trained effective string vibration determination model.
Performance information prediction method having. A step in which one or more processors generate volume pitch characteristic data including volume information and pitch information of each string from string vibration waveform data representing a stringed instrument performance played according to training performance information.
A step in which the one or more processors use the training performance information to train an effective string vibration determination model that determines the effectiveness of vibration of each string from the volume pitch characteristic data.
Effective string vibration determination model training method with.

Images