JP6760457B2 - Electronic musical instruments, control methods for electronic musical instruments, and programs - Google Patents

Description

The present invention relates to an electronic musical instrument that reproduces a singing voice in response to an operation of an operator such as a keyboard, a control method for the electronic musical instrument, and a program.

Conventionally, there is known a technique of an electronic musical instrument that reproduces a singing voice (vocal) in response to an operation of an operator such as a keyboard (for example, Patent Document 1). In this conventional technique, when there is a keyboard operator for instructing a pitch, a storage means for storing lyrics data, an instruction means for instructing that the lyrics data should be read from the storage means, and an instruction by the instruction means. It is provided with a reading means for sequentially reading lyrics data from the storage means, and a sound source for generating a singing voice of a pitch instructed by a keyboard operator with a tone color corresponding to the lyrics data read by the reading means.

Japanese Unexamined Patent Publication No. 6-332449

However, in the above-mentioned conventional technique, for example, when trying to output a singing voice according to the lyrics according to the progress of accompaniment data output by an electronic musical instrument, the key is specified regardless of which key is specified by the performer. If the singing voice corresponding to the lyrics is sequentially output each time, the output singing voice and the progress of the accompaniment data do not match depending on how the performer specifies the key. For example, if one bar contains four notes, and the performer specifies four or more pitches in one bar, the lyrics will advance ahead of the accompaniment data, and the performer will. If the pitch specified in the section of one bar is three or less, the lyrics will be delayed from the progress of the accompaniment data.

In this way, if the lyrics advance in sequence each time the performer specifies the pitch on the keyboard, for example, the lyrics will advance too much with respect to the accompaniment, or conversely, they will be too late. ..

An example of an electronic musical instrument of the embodiment is
When the first pitch to be specified according to the first timing is specified, the singing voice data of the first pitch corresponding to the first character corresponding to the first timing is output, and the singing voice data of the first pitch is output. When the second pitch to be specified according to the next second timing is specified before the second timing, the second character corresponding to the second timing without waiting for the arrival of the second timing. vocals output process of outputting the voice data of the second pitch corresponding to,
In the output singing data of the first pitch corresponding to the first character corresponding to the previous SL first timing, the first pitch and the third pitch other than the second pitch is higher than the second timing Also, when specified before, a pitch change process for changing the pitch of the singing voice data of the first pitch corresponding to the first character being output to the third pitch, and
To execute.

According to the present invention, it is possible to provide an electronic musical instrument that satisfactorily controls the progress of lyrics.

It is a figure which shows the appearance example of one Embodiment of an electronic keyboard instrument. It is a block diagram which shows the hardware configuration example of one Embodiment of the control system of an electronic keyboard instrument. It is a block diagram which shows the configuration example of the voice synthesis LSI. It is an operation explanatory diagram of the voice synthesis LSI. It is explanatory drawing of the lyrics control technique. It is a figure which shows the data structure example of this embodiment. It is a main flowchart which shows the control processing example of the electronic musical instrument in this embodiment. It is a flowchart which shows the detailed example of the initialization processing, the tempo change processing, and the song start processing. It is a flowchart which shows the detailed example of a switch process. It is a flowchart which shows the detailed example of the automatic performance interrupt processing. It is a flowchart which shows the detailed example of the 1st Embodiment of a song reproduction processing. It is a flowchart which shows the detailed example of the 2nd Embodiment of a song reproduction process. It is a flowchart which shows the detailed example of the next song event search process. It is a figure which shows the composition example of the lyrics control data in MusicXML format. It is a figure which shows the musical score display example by the lyrics control data in the MusicXML format.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an external example of an embodiment 100 of an electronic keyboard instrument. The electronic keyboard instrument 100 includes a keyboard 101 composed of a plurality of keys as a performance operator, and a first switch panel 102 for instructing various settings such as volume designation, song playback tempo setting, song playback start, and accompaniment playback. , A second switch panel 103 for selecting songs and accompaniments, selecting tones, and the like, and an LCD 104 (Liquid Crystal Display) for displaying lyrics, scores, and various setting information during song playback. Further, although not particularly shown, the electronic keyboard instrument 100 is provided with a speaker for emitting a musical sound generated by the performance on a back surface portion, a side surface portion, a back surface portion, or the like.

FIG. 2 is a diagram showing a hardware configuration example of an embodiment of the control system 200 of the electronic keyboard instrument 100 of FIG. In FIG. 2, the control system 200 includes a CPU (central processing unit) 201, a ROM (read-only memory) 202, a RAM (random access memory) 203, a sound source LSI (large-scale integrated circuit) 204, a voice synthesis LSI 205, and FIG. The key scanner 101 to which the keyboard 101, the first switch panel 102, and the second switch panel 103 are connected, and the LCD controller 208 to which the LCD 104 of FIG. 1 is connected are connected to the system bus 209, respectively. Further, a timer 210 for controlling the sequence of automatic performance is connected to the CPU 201. Further, the music sound output data 218 and the singing voice voice output data 217 output from the sound source LSI 204 and the voice synthesis LSI 205, respectively, are converted into an analog music sound output signal and an analog singing voice voice output signal by the D / A converters 211 and 212, respectively. The analog musical tone output signal and the analog singing voice output signal are mixed by the mixer 213, and after the mixed signal is amplified by the amplifier 214, they are output from a speaker or an output terminal (not shown).

The CPU 201 executes the control operation of the electronic keyboard instrument 100 of FIG. 1 by executing the control program stored in the ROM 202 while using the RAM 203 as the work memory. In addition to the control program and various fixed data, the ROM 202 stores song data including lyrics data and accompaniment data.

The timer 210 used in the present embodiment is mounted on the CPU 201, and counts the progress of the automatic performance of the electronic keyboard instrument 100, for example.

The sound source LSI 204 reads, for example, musical sound type data from a waveform ROM (not shown) in accordance with a sound control instruction from the CPU 201, and outputs the music to the D / A converter 211. The sound source LSI 204 has the ability to oscillate up to 256 voices at the same time.

When the voice synthesis LSI 205 is given the text data of the lyrics and the information on the pitch, the pitch and the start frame as the singing voice data 215 from the CPU 201, the voice synthesis LSI 205 synthesizes the corresponding singing voice voice data and outputs it to the D / A converter 212. To do.

The key scanner 206 constantly scans the key press / release state of the key 101 of FIG. 1, the switch operation state of the first switch panel 102, and the second switch panel 103, and interrupts the CPU 201. Communicate change.

The LCD controller 208 is an IC (integrated circuit) that controls the display state of the LCD 104.

FIG. 3 is a block diagram showing a configuration example of the speech synthesis LSI 205 of FIG. This speech synthesis LSI 205 is based on, for example, the technique of "statistical speech synthesis based on deep learning" described in the following document by inputting the singing voice data 215 instructed from the CPU 201 of FIG. 2 by the song reproduction process described later. , Singing voice voice output data 217 is synthesized and output.

(Reference)
Yoshi Hashimoto, Shinji Takagi, "Statistical Speech Synthesis Based on Deep Learning," Journal of the Acoustical Society of Japan, Vol. 73, No. 1 (2017), pp. 55-62

The speech synthesis LSI 205 includes a speech learning unit 301 and a speech synthesis unit 302. The voice learning unit 301 includes a learning text analysis unit 303, a learning acoustic feature extraction unit 304, and a model learning unit 305.

The learning text analysis unit 303 inputs the learning singing voice data 311 including the lyrics text, the pitch and the pitch, and analyzes the data. As a result, the learning text analysis unit 303 estimates and outputs the learning language feature sequence 313, which is a discrete numerical sequence expressing phonemes, parts of speech, words, pitches, etc. corresponding to the learning singing voice data 311.

The learning acoustic feature amount extraction unit 304 inputs and analyzes the learning singing voice voice data 312 recorded through a microphone or the like by a certain singer singing the lyrics text. As a result, the learning acoustic feature amount extraction unit 304 extracts and outputs the learning acoustic feature amount series 314 representing the voice features corresponding to the learning singing voice voice data 312.

The model learning unit 305 uses the following equation (1) to form a learning language feature sequence 313 (this is used).
And the acoustic model (put this)
From), the learning acoustic feature series 314 (this is
Probability of generating (put this)
And put) an acoustic model that maximizes
Is estimated by machine learning. That is, the relationship between the language feature series that is text and the acoustic feature series that is voice is expressed by a statistical model called an acoustic model.

The model learning unit 305 is an acoustic model calculated as a result of performing machine learning according to equation (1).
The model parameter expressing the above is output as the learning result 315 and set in the acoustic model unit 306 in the voice synthesis unit 302.

The speech synthesis unit 302 includes a text analysis unit 307, an acoustic model unit 306, and a vocalization model unit 308. The voice synthesis unit 302 synthesizes the singing voice output data 217 corresponding to the singing voice data 215 including the lyrics text by predicting it using a statistical model called an acoustic model set in the acoustic model unit 306. Execute the process.

The text analysis unit 307 inputs the text data of the lyrics designated by the CPU 201 of FIG. 2 and the singing voice data 215 including information on the pitch, the pitch, and the start frame as the result of the performer's performance in accordance with the automatic performance. , Analyze the data. As a result, the text analysis unit 307 analyzes and outputs the language feature sequence 316 expressing the phonemes, parts of speech, words, etc. corresponding to the singing voice data 215.

By inputting the language feature quantity sequence 316, the acoustic model unit 306 estimates and outputs the corresponding acoustic feature quantity sequence 317. That is, the acoustic model unit 306 uses the following equation (2) to input the language feature quantity series 316 (again) from the text analysis unit 307.
), And the acoustic model set as the learning result 315 by machine learning in the model learning unit 305.
Based on and, the acoustic feature series 317 (again)
Probability of generating (put this)
Estimated value of acoustic feature series 317 that maximizes
To estimate.

The vocalization model unit 308 generates the singing voice output data 217 corresponding to the singing voice data 215 including the lyrics text designated by the CPU 201 by inputting the acoustic feature quantity series 317. The singing voice output data 217 is output from the D / A converter 212 of FIG. 2 via the mixer 213 and the amplifier 214, and is particularly emitted from a speaker (not shown).

The acoustic features represented by the learning acoustic feature series 314 and the acoustic feature series 317 include spectral information that models the human vocal tract and sound source information that models the human vocal cords. As the spectrum parameter, for example, mer cepstrum, line spectrum pair (Line Spectral Pairs: LSP) and the like can be adopted. As the sound source information, a fundamental frequency (F0) indicating the pitch frequency of human voice can be adopted. The vocalization model unit 308 includes a sound source generation unit 309 and a synthetic filter unit 310. The sound source generation unit 309 sequentially inputs a sequence of sound source information 319 input from the acoustic model unit 306, so that the sound source information 319 is periodically repeated at the fundamental frequency (F0) included in the sound source information 319, and is included in the sound source information 319. A sound source signal consisting of a pulse train having a power value (in the case of a voiced phoneme) or white noise having a power value included in the sound source information 319 (in the case of an unvoiced phoneme) is generated. The synthetic filter unit 310 forms a digital filter that models the voice path based on the sequence of spectrum information 318 sequentially input from the acoustic model unit 306, and digitally uses the sound source signal input from the sound source generation unit 309 as an excitation source signal. The singing voice audio output data 217 of the signal is generated and output.

In the present embodiment, in order to predict the acoustic feature sequence 317 from the language feature sequence 316, the acoustic model unit 306 is implemented by a deep neural network (DNN). Correspondingly, the model learning unit 305 in the speech learning unit 301 learns the model parameters representing the nonlinear conversion function of each neuron in the DNN from the language feature quantity to the acoustic feature quantity, and learns the model parameters as the learning result. As 315, it is output to the DNN of the acoustic model unit 306 in the voice synthesis unit 302.

Usually, the acoustic feature amount is calculated in units of a frame having a width of, for example, 5.1 msec (milliseconds), and the language feature amount is calculated in units of phonemes. Therefore, the acoustic features and the language features have different time units. Since the acoustic model unit 306, which is a DNN, is a model representing a one-to-one correspondence between the input language feature sequence 316 and the output acoustic feature sequence 317, input / output data pairs having different time units are used. DNN cannot be trained. Therefore, in the present embodiment, the correspondence between the frame-based acoustic feature sequence and the phoneme-based language feature sequence is set in advance, and a pair of frame-based acoustic feature and language feature is generated.

FIG. 4 is an operation explanatory diagram of the speech synthesis LSI 205 showing the above-mentioned correspondence. For example, the singing phoneme string "/ k /" "/ i" which is a linguistic feature sequence corresponding to the lyrics character strings "ki", "ra", and "ki" (Fig. 4 (a)) of the singing of the children's song "Kirakira Hoshi" When / ”,“ / r / ”,“ / a / ”,“ / k / ”, and“ / i / ”(FIG. 4 (b)) are obtained, these language feature series are frame-based phoneme features. The quantity series (FIG. 4 (c)) is associated with a one-to-many relationship (relationship between (b) and (c) in FIG. 4). Since the language feature amount is used as an input to the DNN in the acoustic model unit 306, it needs to be expressed as numerical data. Therefore, as a language feature series, "Is the phoneme immediately before" / a / "? Binary data (0 or 1) to questions related to context such as "How many phonemes are contained in the current word?", Or numerical data obtained by concatenating consecutive answers are prepared. ..

The model learning unit 305 in the voice learning unit 301 of FIG. 3 is a phoneme sequence of the learning language feature series 313 corresponding to FIG. 4 (b) in frame units as shown by the broken line arrow group 401 of FIG. And the pair of the learning acoustic feature quantity series 314 corresponding to FIG. 4C are sequentially given to the DNN of the acoustic model unit 306 for learning. The DNN in the acoustic model unit 306 includes a neuron group consisting of an input layer, one or more intermediate layers, and an output layer, as shown as a group of gray circles in FIG.

On the other hand, at the time of speech synthesis, the phoneme sequence of the language feature quantity series 316 corresponding to FIG. 4B is input to the DNN of the acoustic model unit 306 in the frame unit. As a result, the DNN of the acoustic model unit 306 outputs the acoustic feature amount series 317 in the frame unit as shown by the thick solid line arrow group 402 in FIG. Therefore, also in the vocalization model unit 308, the sound source information 319 and the spectrum information 318 included in the acoustic feature quantity series 317 are given to the sound source generation unit 309 and the synthesis filter unit 310, respectively, in the above-mentioned frame unit, and the voice synthesis is executed. Will be done.

As a result, the vocalization model unit 308 outputs, for example, 225 samples of singing voice output data 217 for each frame, as shown by the thick solid line arrow group 403 in FIG. Since the frame has a time width of 5.1 msec, one sample is "5.1 msec ÷ 225 ≈ 0.0227 msec", and therefore the sampling frequency of the singing voice output data 217 is 1 / 0.0227 ≈ 44 kHz (kilohertz). Is.

The DNN learning is performed by the square error minimization standard calculated by the following equation (3) using the pair of the acoustic feature amount and the language feature amount in each frame.

here,
When
Are the acoustic features and language features in the t-th frame t, respectively.
Is the DNN model parameter of the acoustic model unit 306,
Is a non-linear transformation function represented by DNN. The DNN model parameters can be efficiently estimated by the backpropagation method. Considering the correspondence with the processing of the model learning unit 305 in the statistical speech synthesis represented by the above-mentioned equation (1), the learning of DNN can be expressed as the following equation (4).

Here, the following equation (5) holds.

As in the above equations (4) and (5), the relationship between the acoustic features and the language features is a normal distribution with the output of DNN as the average vector.
Can be represented by. In statistical speech synthesis processing using DNN, language features are usually used.
Covariance matrix independent of, i.e. the same covariance matrix in all frames
Is used. Also, the covariance matrix
When is an identity matrix, equation (4) shows a learning process equivalent to equation (3).

As described with reference to FIG. 4, the DNN of the acoustic model unit 306 estimates the acoustic feature sequence 317 independently for each frame. Therefore, the obtained acoustic feature sequence 317 includes discontinuities that deteriorate the quality of the synthesized speech. Therefore, in the present embodiment, for example, the quality of the synthesized speech can be improved by using a parameter generation algorithm using dynamic features.

The operation of the present embodiment having the configuration examples of FIGS. 1, 2 and 3 will be described in detail below. FIG. 5 is an explanatory diagram of the lyrics control technique. FIG. 5A is a diagram showing the relationship between the lyric text and the melody that progresses according to the automatic performance. For example, in the case of singing the above-mentioned children's song "Kirakira Hoshi", the song data includes "ki / Twin (first character)", "ra / kle (second character)", and "ki / twin (third character)". Each character (lyric information) of the lyrics of "ra / kle (4th character)", each timing information of t1, t2, t3, t4 that outputs each character of the lyrics, and the melody pitch of each character of the lyrics " Each pitch information such as E4 (first pitch), "E4 (second pitch)", "B4 (third pitch)", and "B4 (fourth pitch)" is included. At each timing of t5, t6, and t7 after t4, each character of the lyrics of "hi / lit (5th character)", "ka / tre (6th character)", and "ru / star (7th character)" Are associated with each other.

For example, the timings of t1, t2, t3, and t4 in FIG. 5 (b) correspond to the original vocalization timings t1, t2, t3, and t4 of FIG. 5 (a). Here, at the timings t1 and t2 corresponding to the original vocalization timing, the performer correctly presses the key of the same pitch E4 as the first pitch E4 included in the song data on the keyboard 101 of FIG. Suppose you did. In this case, the CPU 201 of FIG. 2 gives the performer the lyrics "ki / Twin (first character)" and "ra / kle (second character)" at the timings t1 and t2, and the performers at the respective timings t1 and t2. Singing voice data 215 including information indicating the specified pitch E4 and, for example, information indicating the time length of each quarter note length (obtained at least based on either the music data or the performance by the performer). , Output to the voice synthesis LSI 205 of FIG. As a result, the voice synthesis LSI 205 has quarter note length singing voice output data 217 corresponding to the lyrics "ki / Twin (first character)" and "ra / kle (second character)" at timings t1 and t2, respectively. Is output at the first pitch (= specified pitch) E4 and the second pitch (= specified pitch) E4, respectively. The determination "○" mark corresponding to the timings t1 and t2 indicates that the utterance was correctly performed according to the pitch and the lyrics information included in the song data.

Next, when the performer presses the keyboard 101 of FIG. 1 to specify the pitch at any of the original vocalization timings, the specified pitch is changed. If there is a discrepancy with the pitch that should be specified, the following control is executed. The CPU 201 of FIG. 2 controls the progress of lyrics and the progress of automatic accompaniment so that the singing voice corresponding to the character next to the character corresponding to the pitch to be specified is not output in accordance with any of the timings. For example, if the pitch to be specified according to a certain timing is the first pitch and the pitch to be specified according to the next timing of a certain timing is the second pitch, at a certain timing. If a pitch other than the first pitch is specified accordingly, it does not matter whether or not the singing voice corresponding to the first character corresponding to the first pitch is output, but the second The singing voice corresponding to the second character corresponding to the pitch is not output. After that, the CPU 201 restarts the progress of the lyrics and the progress of the automatic accompaniment when the pitch specified by the performer presses the key matches the pitch that should be originally specified.

For example, in FIG. 5B, the pitch G4 specified by the performer by pressing the key of the keyboard 101 of FIG. 1 at the timing t3 corresponding to the original vocalization timing is originally vocalized at the timing t3. It is assumed that the pitch does not match the third pitch B4. In this case, the CPU 201 of FIG. 2 may output "ki / twin (third character)" in accordance with the timing t3, but the fourth pitch (timing t4) to be specified next to the third pitch B4. The progress of the lyrics and the progress of the automatic accompaniment are controlled so that the singing voice corresponding to the characters after "ra / kle (fourth character)" corresponding to the fourth pitch to be specified according to is not output. The CPU 101 may be controlled so as not to pronounce "ki / twin (third character)" at the timing t3 of FIG. 5 (b). In this case, although the key is specified according to the timing to be specified, the key to be specified is not specified (the key is pressed by mistake), so that the singing voice is not output.

If the pitch specified by the performer by pressing the key matches the pitch that should be originally specified while the progress of the lyrics and the progress of the automatic accompaniment are controlled so that the singing voice is not output as described above, the CPU201 Resumes the progression of lyrics and the progression of automatic accompaniment. For example, after the progress of the lyrics and the progress of the automatic accompaniment are stopped at the timing t3 of FIG. 5B, the performer specifies the pitch B4 that matches the originally specified third pitch B4 at the timing t3'. , CPU201 produces a singing voice "i / in (third character')" corresponding to the lyrics information "ki / twin (third character)" corresponding to the third pitch B4 that should have been pronounced at the timing t3. Output and resume the progress of lyrics and the progress of automatic accompaniment.

When the progress of the lyrics and the progress of the automatic accompaniment are restarted as described above, for example, the singing voice "i / twin (third character)" corresponding to the lyrics information "ki / twin (third character)" restarted at the timing t3'in FIG. 5 (b). The utterance timing t4'of the lyrics information "ra / kle (fourth character)" that should be uttered after the utterance of "/ in (third character')" is the amount that the restarted utterance timing deviates from t3 to t3'. Only, it slides from the original utterance timing t4 to t4'.

Here, when the above-mentioned specified pitch does not match the pitch that should be originally specified, it can be said that it includes the case where the key is not pressed according to the timing to be specified. That is, although not shown in FIG. 5, even when the key of the keyboard 101 of FIG. 1 is not pressed and the pitch is not specified at any timing of the original vocalization timing, the pitch to be vocalized is supported. It can be said that the progress of the lyrics and the progress of the automatic accompaniment are controlled so that the singing voice corresponding to the next character (second character) of the character (first character) to be played is not output.

The timing t3 of FIG. 5C is a sound designated by the key press at the timing t3 corresponding to the original vocalization timing, assuming that the above-mentioned control operation according to the present embodiment is not performed. This is a description of the control operation when the high G4 does not match the third pitch B4 that should be originally specified at the timing t3. When the above-mentioned control operation according to the present embodiment is not performed, at the timing t3 of FIG. 5 (c), the "ra / kle (3)" following the "ki / twin (third character)" that should be originally uttered (3rd character) The fourth character) ”has been uttered, and unlike the present embodiment, the progress of the lyrics has become inappropriate. That is, the "ra / kle (fourth character)" that should be pronounced at the timing t4 is pronounced at the timing t3 earlier than the timing t4.

As described above, if the performer does not specify the correct pitch that matches the pitch that should be originally specified at the original vocalization timing, and if the control operation according to the present embodiment is not executed, the progress of the lyrics. And the progression of the automatic accompaniment did not match, and the performer had to correct the progression of the lyrics each time. On the other hand, in the present embodiment, the progress of the lyrics and the progress of the automatic accompaniment are stopped until the correct pitch that matches the pitch that should be originally specified by the performer is specified. It is possible to perform natural lyrics progression.

Next, the performer presses an arbitrary key (operator) on the keyboard 101 of FIG. 1 at a timing when none of the original vocalization timings have arrived, and the pitch specified by the key is next. If the pitch does not match the pitch to be specified in, the CPU 201 of FIG. 2 changes the pitch of the singing voice of the singing voice voice output data 217 output by the voice synthesis LSI 205 to the pitch specified by the performance operation. The singing voice data 215 instructing this is output to the voice synthesis LSI 205 of FIG. As a result, in the voice synthesis LSI 205 of FIG. 2 or 3, the pitch of the singing voice output data 217 being uttered is set to the sound specified by the CPU 201 at a timing when none of the original utterance timings has arrived. Change to high.

For example, in FIG. 5 (b), at the timing t1'in which none of the original vocalization timings t1, t2, t3, and t4 has arrived, the performer presses the key of pitch G4 on the keyboard 101 of FIG. Suppose you press a key. In this case, the CPU 201 determines that the designated pitch G4 does not match the second pitch E4 corresponding to the lyrics information "ra / kle (second character)" to be specified later (timing t2). As a result, the pitch of the singing voice voice output data 217 of the lyrics information "ki / Twin (first character)" output by the voice synthesis LSI 205 is the pitch specified by the performance operation from the first pitch E4 so far. The singing voice data 215 instructing to change to G4 and continue utterance is output to the voice synthesis LSI 205 of FIG. As a result, the voice synthesis LSI 205 of FIG. 2 or FIG. 3 has a singing voice of "i / in (first character')" corresponding to the phonation information "ki / Twin (first character)" at the timing t1'. The pitch of the voice output data 217 is changed from the first pitch E4 up to now to the pitch G4 specified by the CPU 201, and the utterance is continued.

The timing t1'in FIG. 5C is the timing t1'other than the original vocalization timing when the performer assumes that the above-mentioned control operation according to the present embodiment has not been performed. This is an explanation of the control operation when a key is pressed. When the above-mentioned control operation according to the present embodiment is not performed, at the timing t1'in FIG. 5C, the lyrics information "ra / kle (second character)" to be uttered at the next utterance timing t2. Is uttered.

As described above, when the control operation according to the present embodiment is not executed, even if the pitch specified by the performer at a timing other than the original vocalization timing does not match the pitch to be specified, the lyrics are steadily displayed. I went ahead and felt unnatural. On the other hand, in the present embodiment, the pitch uttered according to the singing voice output data 217 uttered from the original timing immediately before the timing is changed to the pitch specified by the performer. It will be possible to continue. In this case, for example, the pitch of the singing voice output data 217 corresponding to the lyrics information “ki / Twin (first character)” uttered at the original song playback timing t1 in FIG. 5 (b) is not interrupted. Sounds to continuously change to the pitch due to the new key press at the key press timing t1'. As a result, in the present embodiment, the lyrics can be progressed naturally.

Although not shown in FIG. 5, the performer specified the same pitch as the pitch according to the lyrics information to be uttered next at the timing when none of the original utterance timings arrived. And. In this case, the CPU 201 may control the progress of the lyrics and the progress of the automatic accompaniment to advance (jump) at once until the timing of the singing voice to be uttered next.

If the performer performs a performance operation at a timing other than the original vocalization timing and the pitch specified at that time does not match the pitch to be specified at the next timing, the already output singing voice The utterance according to the voice output data 217 may be repeatedly output (change the pitch). In this case, for example, following the singing voice voice output data 217 corresponding to the lyrics information "ki / Twin (first character)" uttered at the original song playback timing t1 of FIG. 5 (b), the key press timing t1 It sounds like the singing voice output data 217 corresponding to "ki / Twin (first character)" is uttered separately by the new key press in ′. Alternatively, it may be controlled so that the singing voice output data 217 is not uttered at a timing other than the utterance timing.

FIG. 6 is a diagram showing a data configuration example of song data read from ROM 202 to RAM 203 in FIG. 2 in the present embodiment. This data structure example conforms to the standard MIDI file format, which is one of the MIDI (Musical Instrument Digital Interface) file formats. This song data is composed of data blocks called chunks. Specifically, the song data consists of a header chunk at the beginning of the file, a track chunk 1 in which the lyrics data for the following lyrics part is stored, and a track chunk 2 in which the performance data for the accompaniment part is stored. It is composed.

The header chunk consists of four values: ChunkID, ChunkSize, FormatType, NumberOfTrack, and TimeDivision. The Chunk ID is a 4-byte ASCII code "4D 54 68 64" (numbers are hexadecimal numbers) corresponding to four single-byte characters "MThd" indicating that it is a header chunk. The ChunkSize is 4-byte data indicating the data length of the FormatType, NumberOfTrack, and TimeDivision parts excluding the ChunkID and the ChunkSize in the header chunk, and the data length is 6 bytes: "00 00 00 06" (numbers are hexadecimal numbers). It is fixed to. In the case of this embodiment, the Format Type is 2-byte data "00 01" (numbers are hexadecimal numbers), which means format 1 using a plurality of tracks. In the case of the present embodiment, the NumberOfTrack is 2-byte data "00 02" (numbers are hexadecimal numbers) indicating that two tracks corresponding to the lyrics part and the accompaniment part are used. The Time Division is data indicating a time base value indicating a resolution per quarter note, and in the case of the present embodiment, it is 2-byte data "01 E0" (number is a hexadecimal number) indicating 480 in decimal notation.

Track chunks 1 and 2 are ChunkID, ChunkSize, and DataTime_1 [i] and Event_1 [i] (in the case of track chunk 1 / lyrics part) or DeltaTime_2 [i] and Event_2 [i] (track chunk 2 / accompaniment part, respectively). Case) is composed of a performance data set (0 ≦ i ≦ L: track chunk 1 / lyrics part, 0 ≦ i ≦ M: track chunk 2 / accompaniment part). The Chunk ID is a 4-byte ASCII code "4D 54 72 6B" (numbers are hexadecimal numbers) corresponding to four single-byte characters "MTrk" indicating that it is a track chunk. The ChunkSize is 4-byte data indicating the data length of the portion of each track chunk excluding the ChunkID and the ChunkSize.

DeltaTime_1 [i] is variable length data of 1 to 4 bytes indicating the waiting time (relative time) from the execution time of Event_1 [i-1] immediately before that. Similarly, DeltaTime_2 [i] is 1 to 4 byte variable length data indicating the waiting time (relative time) from the execution time of Event_2 [i-1] immediately before that. Event_1 [i] is a meta-event that instructs the utterance timing and pitch of the lyrics in the track chunk 1 / lyrics part. Event_2 [i] is a MIDI event instructing note-on or note-off, or a meta-event instructing time signature in the track chunk 2 / accompaniment part. For each performance data set DeltaTime_1 [i] and Event_1 [i] for the track chunk 1 / lyrics part, after waiting for DeltaTime_1 [i] from the execution time of Event_1 [i-1] immediately before that, Event_1 [i] Is executed, the vocalization progress of the lyrics is realized. On the other hand, for the track chunk 2 / accompaniment part, in each performance data set DeltaTime_2 [i] and Event_2 [i], after waiting for the execution time of Event_2 [i-1] immediately before that, Event_2 [i] By executing i], the progress of automatic accompaniment is realized.

FIG. 7 is a main flowchart showing an example of control processing of an electronic musical instrument according to the present embodiment. This control process is, for example, an operation in which the CPU 201 of FIG. 2 executes a control process program loaded from the ROM 202 into the RAM 203.

The CPU 201 first executes the initialization process (step S701), and then repeatedly executes a series of processes from steps S702 to S708.

In this iterative process, the CPU 201 first executes the switch process (step S702). Here, the CPU 201 executes a process corresponding to the switch operation of the first switch panel 102 or the second switch panel 103 of FIG. 1 based on the interrupt from the key scanner 206 of FIG.

Next, the CPU 201 executes a keyboard process for determining and processing whether or not any key of the key 101 of FIG. 1 has been operated based on the interrupt from the key scanner 206 of FIG. 2 (step S703). Here, the CPU 201 outputs sound control data 216 instructing the sound source LSI 204 of FIG. 2 to start or stop sounding in response to an operation of pressing or releasing any key by the performer.

Next, the CPU 201 processes data to be displayed on the LCD 104 of FIG. 1, and executes a display process of displaying the data on the LCD 104 via the LCD controller 208 of FIG. 2 (step S704). The data displayed on the LCD 104 includes, for example, the lyrics corresponding to the singing voice output data 217 to be played, the score of the melody corresponding to the lyrics, and various setting information (see FIGS. 13 and 14 described later).

Next, the CPU 201 executes the song playback process (step S705). In this process, the CPU 201 executes the control process described with reference to FIG. 5 based on the performance of the performer, generates singing voice data 215, and outputs the singing voice data 215 to the speech synthesis LSI 205.

Subsequently, the CPU 201 executes sound source processing (step S706). In the sound source processing, the CPU 201 executes control processing such as envelope control of the musical sound being sounded in the sound source LSI 204.

Subsequently, the CPU 201 executes the speech synthesis process (step S707). In the speech synthesis process, the CPU 201 controls the execution of speech synthesis by the speech synthesis LSI 205.

Finally, the CPU 201 determines whether or not the performer has pressed a power-off switch (not shown) to power off (step S708). If the determination in step S708 is NO, the CPU 201 returns to the process in step S702. If the determination in step S708 is YES, the CPU 201 ends the control process shown in the flowchart of FIG. 7, and turns off the power of the electronic keyboard instrument 100.

8 (a), (b), and (c) are the initialization process of step S701 of FIG. 7, the tempo change process of step S902 of FIG. 9, which will be described later in the switch process of step S702 of FIG. 7, and the same. It is a flowchart which shows the detailed example of the song start processing of step S906 of FIG.

First, in FIG. 8A showing a detailed example of the initialization process of step S701 of FIG. 7, the CPU 201 executes the ticktime initialization process. In the present embodiment, the progress of the lyrics and the automatic accompaniment proceed in units of time called TickTime. The timebase value specified as the TimeDivision value in the header chunk of the song data of FIG. 6 indicates the resolution of the quarter note, and if this value is, for example, 480, the quarter note has a time length of 480TickTime. Further, the waiting time DeltaTime_1 [i] value and the DeltaTime_2 [i] value in the track chunk of the song data of FIG. 6 are also counted by the time unit of TickTime. Here, how many seconds 1 Tick Time actually becomes depends on the tempo specified for the song data. If the tempo value is Tempo [beat / minute] and the time base value is Time Division, the number of seconds of Tick Time is calculated by the following equation.

TickTime [seconds] = 60 / Tempo / TimeDivision (6)

Therefore, in the initialization process exemplified by the flowchart of FIG. 8A, the CPU 201 first calculates the TickTime [seconds] by the arithmetic processing corresponding to the above equation (6) (step S801). It is assumed that a predetermined value, for example, 60 [beats / second] is stored in the ROM 202 of FIG. 2 in the initial state of the tempo value Tempo. Alternatively, the tempo value at the time of the previous end may be stored in the non-volatile memory.

Next, the CPU 201 sets a timer interrupt according to the TickTime [seconds] calculated in step S801 with respect to the timer 210 of FIG. 2 (step S802). As a result, every time the TickTime [seconds] elapses in the timer 210, an interrupt for lyrics progression and automatic accompaniment (hereinafter referred to as “automatic performance interrupt”) is generated for the CPU 201. Therefore, in the automatic performance interrupt process (FIG. 10 described later) executed by the CPU 201 based on this automatic performance interrupt, the control process for advancing the lyrics progress and the automatic accompaniment is executed for each TickTime.

Subsequently, the CPU 201 executes other initialization processing such as initialization of the RAM 203 of FIG. 2 (step S803). After that, the CPU 201 ends the initialization process of step S701 of FIG. 7 illustrated by the flowchart of FIG. 8A.

The flowcharts of FIGS. 8B and 8C will be described later. FIG. 9 is a flowchart showing a detailed example of the switch processing in step S702 of FIG.

First, the CPU 201 determines whether or not the tempo of the lyrics progression and the automatic accompaniment has been changed by the tempo change switch in the first switch panel 102 of FIG. 1 (step S901). If the determination is YES, the CPU 201 executes the tempo change process (step S902). Details of this process will be described later with reference to FIG. 8 (b). If the determination in step S901 is NO, the CPU 201 skips the process in step S902.

Next, the CPU 201 determines whether or not any song song has been selected on the second switch panel 103 of FIG. 1 (step S903). If the determination is YES, the CPU 201 executes the song song reading process (step S904). This process is a process of reading the song data having the data structure described in FIG. 6 from the ROM 202 of FIG. 2 into the RAM 203. From then on, data access to track chunks 1 or 2 in the data structure illustrated in FIG. 6 is performed on the song data read into RAM 203. If the determination in step S903 is NO, the CPU 201 skips the process in step S904.

Subsequently, the CPU 201 determines whether or not the song start switch has been operated on the first switch panel 102 of FIG. 1 (step S905). If the determination is YES, the CPU 201 executes the song start process (step S906). Details of this process will be described later with reference to FIG. 8 (c). If the determination in step S905 is NO, the CPU 201 skips the process in step S906.

Finally, the CPU 201 determines whether or not other switches have been operated on the first switch panel 102 or the second switch panel 103 of FIG. 1, and executes a process corresponding to each switch operation (step S907). .. After that, the CPU 201 ends the switch process in step S702 of FIG. 7, which is illustrated in the flowchart of FIG.

FIG. 8B is a flowchart showing a detailed example of the tempo change process in step S902 of FIG. As mentioned above, when the tempo value is changed, the TickTime [seconds] is also changed. In the flowchart of FIG. 8B, the CPU 201 executes the control process related to the change of the TickTime [seconds].

First, the CPU 201 is subjected to the operation process corresponding to the above-described equation (6) in the same manner as in the case of step S801 of FIG. 8A, which is executed in the initialization process of step S701 of FIG. Is calculated (step S811). It is assumed that the tempo value Tempo is stored in the RAM 203 or the like after being changed by the tempo change switch in the first switch panel 102 of FIG.

Next, the CPU 201 performs the TickTime [calculated in step S811 with respect to the timer 210 of FIG. 2 in the same manner as in the case of step S802 of FIG. 8A executed in the initialization process of step S701 of FIG. Seconds] sets the timer interrupt (step S812). After that, the CPU 201 ends the tempo change process of step S902 of FIG. 9 exemplified by the flowchart of FIG. 8 (b).

FIG. 8C is a flowchart showing a detailed example of the song start process of step S906 of FIG.

First, in the progress of the automatic performance, the CPU 201 sets the values of the variables DeltaT_1 (track chunk 1) and DeltaT_2 (track chunk 2) on the RAM 203 for counting the relative time from the occurrence time of the immediately preceding event in units of TickTime. Is initially set to 0 for both. Next, the CPU 201 is a RAM 203 for designating each i of the performance data sets DeltaTime_1 [i] and Event_1 [i] (1 ≦ i ≦ L-1) in the track chunk 1 of the music data exemplified in FIG. Each value of the above variable AutoIndex_1 and the variable AutoIndex_1 on the RAM 203 for designating each i of the performance data sets DeltaTime_2 [i] and Event_2 [i] (1 ≦ i ≦ M-1) also in the track chunk 2. Both are initially set to 0 (above, step S821). As a result, in the example of FIG. 6, as an initial state, first, the first performance data set DeltaTime_1 [0] and Event_1 [0] in the track chunk 1, and the first performance data set DeltaTime_2 [0] in the track chunk 2 Event_1 [0] is referenced respectively.

Next, the CPU 201 initially sets the value of the variable SongIndex on the RAM 203 that indicates the current song position to 0 (step S822).

Further, the CPU 201 initializes the value of the variable SongStart on the RAM 203 indicating whether the lyrics and accompaniment progress (= 1) or not (= 0) to 1 (progress) (step S823).

After that, the CPU 201 determines whether or not the performer has set the first switch panel 102 of FIG. 1 to reproduce the accompaniment in accordance with the reproduction of the lyrics (step S824).

If the determination in step S824 is YES, the CPU 201 sets the value of the variable Bansou on the RAM 203 to 1 (with accompaniment) (step S825). On the contrary, if the determination in step S824 is NO, the CPU 201 sets the value of the variable Bansou to 0 (no accompaniment) (step S826). After the process of step S825 or S826, the CPU 201 ends the song start process of step S906 of FIG. 9 shown in the flowchart of FIG. 8 (c).

FIG. 10 shows an automatic performance interrupt process executed based on an interrupt generated every TickTime [seconds] in the timer 210 of FIG. 2 (see step S802 of FIG. 8A or step S812 of FIG. 8B). It is a flowchart which shows the detailed example of. The following processing is executed on the performance data sets of the track chunks 1 and 2 of the music data exemplified in FIG.

First, the CPU 201 executes a series of processes (steps S1001 to S1006) corresponding to the track chunk 1. First, the CPU 201 determines whether or not the SongStart value is 1, that is, whether or not the progress of the lyrics and accompaniment is instructed (step S1001).

When the CPU 201 determines that the progress of the lyrics and the accompaniment is not instructed (the determination in step S1001 is NO), the CPU 201 does not proceed with the lyrics and the accompaniment, and is illustrated in the flowchart of FIG. The automatic performance interrupt processing ends as it is.

When the CPU 201 determines that the progress of the lyrics and the accompaniment is instructed (the determination in step S1001 is YES), the DeltaT_1 value indicating the relative time from the occurrence time of the previous event regarding the track chunk 1 is set. It is determined whether or not the wait time DeltaTime_1 [AutoIndex_1] of the performance data set to be executed to be executed, which is indicated by the AutoIndex_1 value, is matched (step S1002).

If the determination in step S1002 is NO, the CPU 201 increments the DeltaT_1 value indicating the relative time from the occurrence time of the previous event by +1 with respect to the track chunk 1, and advances the time by 1 TickTime unit corresponding to the current interrupt. (Step S1003). After that, the CPU 201 shifts to step S1007, which will be described later.

When the determination in step S1002 becomes YES, the CPU 201 executes the event event [AutoIndex_1] of the performance data set indicated by the AutoIndex_1 value with respect to the track chunk 1 (step S1004). This event is a song event that includes lyrics data.

Subsequently, the CPU 201 stores the AutoIndex_1 value indicating the position of the next song event to be executed in the track chunk 1 in the variable SongIndex on the RAM 203 (step S1004).

Further, the CPU 201 increments the AutoIndex_1 value for referencing the performance data set in the track chunk 1 by +1 (step S1005).

Further, the CPU 201 resets the DeltaT_1 value indicating the relative time from the occurrence time of the song event referred to this time with respect to the track chunk 1 to 0 (step S1006). After that, the CPU 201 shifts to the process of step S1007.

Next, the CPU 201 executes a series of processes (steps S1007 to S1013) corresponding to the track chunk 2. First, the CPU 201 determines whether or not the DeltaT_2 value indicating the relative time from the occurrence time of the previous event regarding the track chunk 2 matches the waiting time DeltaTime_2 [AutoIndex_2] of the performance data set to be executed, which is indicated by the AutoIndex_2 value. Is determined (step S1007).

If the determination in step S1007 is NO, the CPU 201 increments the DeltaT_2 value indicating the relative time from the occurrence time of the previous event by +1 with respect to the track chunk 2, and advances the time by 1 TickTime unit corresponding to the current interrupt. (Step S1008). After that, the CPU 201 ends the automatic performance interrupt process shown in the flowchart of FIG.

If the determination in step S1007 is YES, the CPU 201 determines whether or not the value of the variable Bansou on the RAM 203 instructing accompaniment reproduction is 1 (with accompaniment) (step S1009) (step 8 (c)). See S824 to S826).

If the determination in step S1009 is YES, the CPU 201 executes the event Event_2 [AutoIndex_2] relating to the accompaniment related to the track chunk 2 indicated by the AutoIndex_2 value (step S1010). If the event Event_2 [AutoIndex_2] executed here is, for example, a note-on event, an accompaniment musical tone sounding command is issued to the sound source LSI 204 of FIG. 2 according to the key number and velocity specified by the note-on event. publish. On the other hand, if the event Event_2 [AutoIndex_2] is, for example, a note-off event, a mute command for the accompaniment musical tone being sounded to the sound source LSI 204 of FIG. 2 is issued according to the key number and velocity specified by the note-off event. publish.

On the other hand, if the determination in step S1009 is NO, the CPU 201 skips step S1010, so that the event Event_2 [AutoIndex_2] related to this accompaniment is not executed, and the progress is synchronized with the lyrics, so that the next step S1011 Proceed to the process of, and execute only the control process that advances the event.

After step S1010 or when the determination in step S1009 is NO, the CPU 201 increments the AutoIndex_2 value for referencing the performance data set for accompaniment data on track chunk 2 by +1 (step S1011).

Further, the CPU 201 resets the DeltaT_2 value indicating the relative time from the occurrence time of the event executed this time with respect to the track chunk 2 to 0 (step S1012).

Then, the CPU 201 is whether or not the waiting time DeltaTime_2 [AutoIndex_2] of the performance data set on the track chunk 2 to be executed next indicated by the AutoIndex_2 value is 0, that is, an event executed at the same time as this event. Whether or not it is determined (step S1013).

If the determination in step S1013 is NO, the CPU 201 ends the current automatic performance interrupt process shown in the flowchart of FIG.

If the determination in step S1013 is YES, the CPU 201 returns to step S1009 and repeats the control regarding the event Event_1 [AutoIndex_2] of the performance data set to be executed next on the track chunk 2 indicated by the AutoIndex_2 value. The CPU 201 repeatedly executes the processes of steps S1009 to S1013 as many times as the number of times it is executed simultaneously this time. The above processing sequence is executed when a plurality of note-on events are sounded at the same timing, such as a chord.

FIG. 11 is a flowchart showing a detailed example of the first embodiment of the song reproduction process of step S705 of FIG. This process executes the control process according to the present embodiment described with reference to FIG.

First, the CPU 201 determines in step S1004 in the automatic performance interrupt process of FIG. 10 whether or not a value is set in the variable SongIndex on the RAM 203 and is no longer a Null value (step S1101). This SongIndex value indicates whether or not the current timing is the reproduction timing of the singing voice.

When the determination in step S1101 becomes YES, that is, when the current time is the timing of song playback (t1, t2, t3, t4, t5, t6, t7, etc. in the example of FIG. 5), the CPU 201 performs the CPU 201 in step S703 of FIG. It is determined whether or not a new key press is detected on the keyboard 101 of FIG. 1 by the performer by the keyboard processing of (step S1102).

If the determination in step S1102 is YES, the CPU 201 reads the pitch from the song event Event_1 [SongIndex] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex on the RAM 203, and is designated by the key press by the performer. It is determined whether or not the designated pitch matches the read pitch (step S1103).

If the determination in step S1103 is YES, the CPU 201 sets the designated pitch designated by the key press by the performer in a register (not particularly shown) or a variable on the RAM 203 as the vocal pitch (step S1104).

Subsequently, the CPU 201 reads the lyrics character string from the song event Event_1 [SongIndex] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex on the RAM 203. The CPU 201 generates singing voice data 215 for uttering the singing voice output data 217 corresponding to the read lyrics character string at the voicing pitch set with the designated pitch based on the key press set in step S1104. Instruct the voice synthesis LSI 205 to perform vocalization processing (step S1105).

The processing of steps S1104 and S1105 described above corresponds to the control processing described above with respect to the song playback timings t1, t2, t3', and t4 of FIG. 5 (b).

After the process of step S1105, the CPU 201 stores the reproduced song position indicated by the variable SongIndex on the RAM 203 in the variable SongIndex_pre on the RAM 203 (step S1106).

Next, the CPU 201 clears the value of the variable SongIndex to the Null value, and sets the timing after that to a state other than the timing of song playback (step S1107).

Further, the CPU 201 sets a value 1 indicating the progress in the variable SongStart on the RAM 203 that controls the progress of the lyrics and the automatic accompaniment (step S1108). After that, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

As described with respect to the timing t3 of FIG. 5B, the pitch specified by the performance (key press) is the pitch read from the music data at the timing t3'with the progress of the lyrics and the automatic accompaniment stopped. If, the determination in step S1101 is YES → the determination in step S1102 is YES, and after the singing voice indicated by Event_1 [SongIndex] is uttered in step S1105, the value of the variable SongStart in step S1108 as described above. Is set to 1. As a result, in the automatic performance interrupt process of FIG. 10, the determination in step S1001 becomes YES, and the progress of the singing voice and the automatic accompaniment is restarted.

If the determination in step S1103 described above is NO, that is, if the specified pitch specified by the key press by the performer does not match the pitch read from the song data, the CPU 201 is on the RAM 203 that controls the progress of the lyrics and automatic accompaniment. The variable SongStart is set to the value 0 indicating the stop of progress (step S1109). After that, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

As described with respect to the timing t3 of FIG. 5B, when the designated pitch by the performance (key press) does not match the pitch read from the song data at the vocalization timing t3 of the singing voice, the determination in step S1101 is YES → The determination in step S1102 proceeds from YES to the determination in step S1103 to NO, and the value of the variable SongStart is set to 0 in step S1109 as described above. As a result, in the automatic performance interrupt process of FIG. 10, the determination in step S1001 becomes NO, and the progress of the singing voice and the automatic accompaniment is stopped.

When the determination in step S1101 described above is NO, that is, when the current time is not the timing of song playback, the CPU 201 detects a new key pressed by the performer on the keyboard 101 in FIG. 1 by the keyboard processing in step S703 in FIG. It is determined whether or not this is done (step S1110).

If the determination in step S1110 is NO, the CPU 201 as it is ends the song playback process in step S705 of FIG. 7 shown in the flowchart of FIG.

If the determination in step S1110 is YES, the CPU 201 determines the lyrics character string of the song event Event_1 [SongIndex_pre] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex_pre on the RAM 203, which is currently being uttered by the speech synthesis LSI 205. The singing voice data 215 instructing to change the pitch of the singing voice output data 217 corresponding to the above to the designated pitch based on the key press of the performer detected in step S1110 is generated and output to the voice synthesis LSI 205 ( Step S1111). At this time, in the singing voice data 215, the phonemes of the latter half of the phonemes of the lyrics that are already being uttered, for example, if the lyrics character string "ki", the phoneme strings "/ k /" and "/ i /" that compose it The frame in which the latter half of "/ i /" starts (see FIGS. 4 (b) and 4 (c)) is set at the start position of the change to the specified pitch.

By the process of step S1111 above, the pitch of the singing voice output data 217 uttered from t1 in FIG. 5B, for example, the original timing immediately before the current key pressing timing is specified by the performer. The pitch is changed to the specified pitch, and the utterance can be continued at the current key press timing t1'in FIG. 5B, for example.

After the process of step S1111, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

FIG. 12 is a flowchart showing a detailed example of the second embodiment of the song reproduction process of step S705 of FIG. This process executes another control process according to the present embodiment described with reference to FIG.

First, the CPU 201 determines whether or not a new key press is detected on the keyboard 101 of FIG. 1 by the performer by the keyboard processing of step S703 of FIG. 7 (step S1201).

If the determination in step S1201 is YES, the CPU 201 sets a value in the variable SongIndex on the RAM 203 indicating whether or not the current timing is the reproduction timing of the singing voice in step S1004 in the automatic performance interrupt process of FIG. It is determined whether or not the value is not the Null value (step S1102).

When the determination in step S1102 becomes YES, that is, when the current time is the timing of song playback (t1, t2, t3, t4, etc. in the example of FIG. 5), the CPU 201 is set to the designated sound designated by the key press by the performer. The pitch is set as the vocal pitch in a register (not shown) or a variable on the RAM 203 (step S1203).

Subsequently, the CPU 201 reads the lyrics character string from the song event Event_1 [SongIndex] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex on the RAM 203. The CPU 201 generates singing voice data 215 for uttering the singing voice output data 217 corresponding to the read lyrics character string at the voicing pitch set with the designated pitch based on the key press set in step S1104. Instruct the voice synthesis LSI 205 to perform vocalization processing (step S1204).

After that, the CPU 201 reads the pitch from the song event Event_1 [SongIndex] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex on the RAM 203, and the designated pitch specified by the key press by the performer is the song data. It is determined whether or not the pitch matches the pitch read from (step S1205).

If the determination in step S1205 is YES, the CPU 201 proceeds to step S1206. This process corresponds to the control process described above for the song playback timings t1, t2, t3', and t4 in FIG. 5 (b).

In step S1206, the CPU 201 stores the reproduced song position indicated by the variable SongIndex on the RAM 203 in the variable SongIndex_pre on the RAM 203.

Next, the CPU 201 clears the value of the variable SongIndex to the Null value, and sets the timing after that to a state other than the timing of song playback (step S1207).

Further, the CPU 201 sets a value 1 indicating the progress in the variable SongStart on the RAM 203 that controls the progress of the lyrics and the automatic accompaniment (step S1208). After that, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

If the determination in step S1205 described above is NO, the CPU 201, that is, if the specified pitch specified by the key press by the performer does not match the pitch read from the song data, the CPU 201 advances the lyrics and automatic accompaniment. A value 0 instructing the stop of progress is set in the variable SongStart on the RAM 203 to be controlled (step S1209). After that, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG. As a result, in the automatic performance interrupt process of FIG. 10, the determination in step S1001 becomes YES, and the progress of the singing voice and the automatic accompaniment is restarted, as in the case of step S1108 of FIG.

If the determination in step S1205 described above is NO, that is, if the specified pitch specified by the key press by the performer does not match the pitch read from the song data, the CPU 201 is on the RAM 203 that controls the progress of the lyrics and automatic accompaniment. The variable SongStart is set to a value 0 instructing the stop of progress (step S1210). After that, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG. As a result, in the automatic performance interrupt process of FIG. 10, the determination in step S1001 becomes NO, and the progress of the singing voice and the automatic accompaniment is stopped, which is the same as in the case of step S1109 of FIG.

This process corresponds to the control process described above with respect to the song reproduction timing t3 of FIG. 5 (b).

If the determination in step S1202 becomes NO after the determination in step S1201 described above becomes YES, that is, if the performance (key press) by the performer is uttered at a timing other than the timing at which the singing voice should be uttered, the following Control processing is executed.

First, the CPU 201 sets a value 0 in the variable SongStart on the RAM 203 to temporarily stop the progress of the singing voice and the automatic accompaniment (step S1211) (see step S1001 in FIG. 10).

Next, the CPU 201 causes the values of the variables DeltaT_1, DeltaT_2, AutoIndex_1, and AutoIndex_1 on the RAM 203 regarding the current singing voice and the progress position of the automatic accompaniment to be set to the variables DeltaT_1_now, DeltaT_2_now, and AutoIndex_1_12, respectively. ).

After that, the CPU 201 executes the next song event search process (step S1213). Here, the SongIndex value that indicates the event information regarding the next singing voice is searched. The details of this process will be described later.

After the search process of step S1213, the CPU 201 reads the pitch from the song event Event_1 [SongIndex] on the track chunk 1 of the song data on the RAM 203 indicated by the searched SongIndex value, and is designated by the key press by the performer. It is determined whether or not the designated pitch matches the read pitch (step S1214).

If the determination in step S1214 is YES, the CPU 201 advances the control process in the order of YES → S1206 → S1207 → S1208 in the determination of steps S1203 → S1204 → S1205.

By the above-mentioned series of control processing, when the performer presses a key having the same pitch as the pitch of the singing voice to be uttered next at the timing when the original utterance timing has not arrived, the CPU 201 , It is possible to control the progress of the lyrics and the progress of the automatic accompaniment at once (jump) until the timing of the singing voice to be uttered next.

If the determination in step S1214 described above is NO, the CPU 201 returns the values of the variables DeltaT_1_now, DeltaT_2_now, AutoIndex_1_now, and AutoIndex_2_now on the RAM 203 saved in step S1212 to the variables DeltaT_1 and DetAuto2_12, respectively. Then, the amount advanced by the search process in step S1213 is returned to the original progress position before the search (step S1215).

After that, the CPU 201 is singing voice output data corresponding to the lyrics character string of the song event Event_1 [SongIndex_pre] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex_pre on the RAM 203, which is currently being uttered by the voice synthesis LSI 205. Singing voice data 215 instructing to change the pitch of 217 to a designated pitch based on the key press of the performer detected in step S1201 is generated and output to the speech synthesis LSI 205 (step S1216).

The process of step S1216 is the same as the process of step S1111 of FIG. 11, and the singing voice output data 217 uttered from the original timing immediately before the current key press timing, for example, t1 of FIG. 5 (b). The utterance is changed from the pitch to the designated pitch played by the performer, and the utterance can be continued at the current key press timing t1'in FIG. 5B, for example.

After the process of step S1216, the CPU 201 resumes the progress of the lyrics and the automatic accompaniment temporarily stopped in step S1211 by setting the value 1 in the variable SongStart on the RAM 203 (step S1208). After that, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

When the determination in step S1201 described above is NO, that is, when the performer has not played (key pressed), the CPU 201 sets the current timing to the playback timing of the singing voice in step S1004 in the automatic performance interrupt process of FIG. It is determined whether or not a value is set in the variable SongIndex on the RAM 203 indicating whether or not the value is set and the value is not the Null value (step S1209).

If the determination in step S1209 is NO, the CPU 201 as it is ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

If the determination in step S1209 is YES, the CPU 201 sets the variable SongStart on the RAM 203 that controls the progress of the lyrics and the automatic accompaniment to a value 0 instructing the stop of the progress (step S1210). After that, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG. As a result, in the automatic performance interrupt process of FIG. 10, the determination in step S1001 becomes NO, and the progress of the singing voice and the automatic accompaniment is stopped, which is the same as in the case of step S1109 of FIG.

FIG. 13 is a flowchart showing a detailed example of the next song event search process in step S1213 of FIG. In the flowchart of FIG. 13, the portion with the same step number as in the case of the automatic performance interrupt process of FIG. 10 indicates the same process as in the case of FIG. In the process of FIG. 13, the control process group is basically the same as the series of control flows from steps S1002 to S1013 in the automatic performance interrupt process of FIG. 10, excluding only the process of executing the event. Will be executed.

That is, in FIG. 13, in step S1002, the CPU 201 waits for the performance data set related to the singing voice to be executed from now on, in which the DeltaT_1 value indicating the relative time from the occurrence time of the previous event relating to the track chunk 1 is indicated by the AutoIndex_1 value. In step S1003, the DeltaT_1 value is incremented to advance the progress of the singing voice until it is determined that the data matches the DeltaTime_1 [AutoIndex_1].

Further, in the CPU 201, the DeltaT_2 value indicating the relative time from the occurrence time of the previous event related to the track chunk 2 matches the waiting time DeltaTime_2 [AutoIndex_2] of the performance data set regarding the automatic accompaniment to be executed, which is indicated by the AutoIndex_2 value. Each time the determination in step S1007 becomes YES, the AutoIndex_2 value is advanced, and if the determination in step S1007 is NO, the DeltaT_2 value is incremented to advance the progress of automatic accompaniment, and then the process returns to the control process in step S1002. ..

When the determination in step S1002 becomes YES in the repetition of the above series of control processes, the CPU 201 stores the AutoIndex_1 value in the variable SongIndex on the RAM 203, and the next song of step S1213 in FIG. 12 shown in the flowchart of FIG. End the event search process.

FIG. 14 is a diagram showing a configuration example of song data when the song data illustrated as the data structure of FIG. 6 is implemented in the MusicXML format. With such a data structure, it is possible to have the lyrics character string and the score data of the melody. Then, the CPU 201 parses such song data in the display process of step S704 of FIG. 7, for example, on the key 101 of FIG. 1, a key corresponding to the melody corresponding to the lyrics character string currently playing the song. It is possible to have a function to guide the key press of the key corresponding to the lyrics character string by the performer by shining. At the same time, for example, the lyrics character string currently playing the song in the display example as shown in FIG. 15 and the corresponding score can be displayed on the LCD 104 of FIG.

In the embodiment described above, the acoustic model unit 306 is implemented by the DNN (deep neural network) in order to predict the acoustic feature sequence 317 from the language feature sequence 316. In addition, for the above prediction, the acoustic model unit 306 may be implemented by an HMM (Hidden Markov Model: Hidden Markov Model). In this case, the model learning unit 305 in the speech learning unit 301 learns a model in consideration of the context in order to accurately model the acoustic features of the speech. In order to model the acoustic features in detail, not only the phonemes immediately before and after, but also factors such as accent, part of speech, and sentence length are considered. However, since the number of context combinations is enormous, it is difficult to prepare speech data that can accurately learn the context-dependent model for all context combinations. In order to solve this problem, the model learning unit 305 can adopt a technique of context clustering based on a decision tree. In context clustering based on a decision tree, context-dependent models are classified using context-related questions such as "Is the previous phoneme / a /?", And model parameters of similar contexts are used as the learning result 315. It is set in the acoustic model unit 306. Since the context considered depends on the structure of the decision tree, a context-dependent model with high accuracy and high generalization performance can be estimated by selecting an appropriate decision tree structure. The acoustic model unit 306 in the speech synthesis unit 302 of FIG. 3 connects the context-dependent HMMs according to the language feature sequence 316 extracted from the singing voice data 215 by the text analysis unit 307, and the sound with the maximum output probability. The feature series 317 is predicted.

Although the embodiment described above is an embodiment of the present invention for an electronic keyboard instrument, the present invention can also be applied to other electronic musical instruments such as electronic stringed instruments.

In addition, the present invention is not limited to the above-described embodiment, and can be variously modified at the implementation stage without departing from the gist thereof. In addition, the functions executed in the above-described embodiment may be combined as appropriate as possible. The above-described embodiments include various steps, and various inventions can be extracted by an appropriate combination according to a plurality of disclosed constitutional requirements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the effect is obtained, the configuration in which the constituent requirements are deleted can be extracted as an invention.

Regarding the above embodiments, the following additional notes will be further disclosed.
(Appendix 1)
Judgment processing to determine whether the first pitch to be specified and the specified pitch match.
When a match with the first pitch is determined by the determination process, a singing voice corresponding to the first character corresponding to the first pitch is output, and the mismatch with the first pitch is determined by the determination process. Is determined, the singing voice corresponding to the second character corresponding to the second pitch to be specified next to the first pitch is sung until the determination process determines that the pitch matches the first pitch. Singing voice output processing that does not output,
An electronic musical instrument that runs.
(Appendix 2)
The electronic musical instrument according to Appendix 1, which executes an accompaniment data output process for outputting accompaniment data in accordance with the singing voice output process.
(Appendix 3)
After the singing voice output process outputs the singing voice corresponding to the first character corresponding to the first pitch, the pitch specified before the timing for designating the second pitch is the second pitch. The electronic musical instrument according to Appendix 1 or 2, which executes the pitch change process of changing the pitch of the singing voice corresponding to the output first character to the specified pitch and outputting the pitch in the case of inconsistency with the above. ..
(Appendix 4)
After the singing voice output process outputs the singing voice corresponding to the first character corresponding to the first pitch, the pitch specified before the timing for designating the second pitch is the second pitch. In the case of the same, before the timing for designating the second pitch arrives, the pitch of the singing voice corresponding to the second character is output at the second pitch according to the designation, and the pitch is output as described above. The electronic musical instrument according to Appendix 2 or 3, which executes a control process for advancing the progress of the accompaniment data output by the accompaniment data output process in accordance with the output of the singing voice corresponding to the second character.
(Appendix 5)
Based on the learned model generated by machine learning based on the singing voice data sung by a certain singer and the lyrics data, a voice synthesis process for synthesizing the singing voice corresponding to the certain singer is executed.
The singing voice output process is any one of Supplementary notes 1 to 4 that outputs the singing voice synthesized by the voice synthesis process at the pitch specified by the operated operator according to the timing when the operator is operated. The electronic musical instrument described in.
(Appendix 6)
The electronic musical instrument according to any one of Supplementary note 1 to 5, which executes a display process of displaying an identifier indicating a pitch to be specified in accordance with a timing at which the pitch should be specified by operating an operator.
(Appendix 7)
On the computer of electronic musical instruments
Judgment processing to determine whether the first pitch to be specified and the specified pitch match.
When a match with the first pitch is determined by the determination process, a singing voice corresponding to the first character corresponding to the first pitch is output, and the mismatch with the first pitch is determined by the determination process. Is determined, the singing voice corresponding to the second character corresponding to the second pitch to be specified next to the first pitch is sung until the determination process determines that the pitch matches the first pitch. Singing voice output processing that does not output,
How to run.
(Appendix 8)
On the computer of electronic musical instruments
Judgment processing to determine whether the first pitch to be specified and the specified pitch match.
When a match with the first pitch is determined by the determination process, a singing voice corresponding to the first character corresponding to the first pitch is output, and the mismatch with the first pitch is determined by the determination process. Is determined, the singing voice corresponding to the second character corresponding to the second pitch to be specified next to the first pitch is sung until the determination process determines that the pitch matches the first pitch. Singing voice output processing that does not output,
A program that executes.

100 Electronic keyboard instrument 101 Keyboard 102 First switch panel 103 Second switch panel 104 LCD
200 Control system 201 CPU
202 ROM
203 RAM
204 Sound source LSI
205 Speech synthesis LSI
206 Key scanner 208 LCD controller 209 System bus 210 Timer 211, 212 D / A converter 213 Mixer 214 Amplifier 301 Speech learning unit 302 Speech synthesis unit 303 Learning text analysis unit 304 Learning acoustic feature extraction 305 Model learning unit 306 Acoustic model Part 307 Text analysis part 308 Voice model part 309 Sound source generation part 310 Synthetic filter part 311 Learning singing voice data 312 Learning singing voice data 313 Learning language feature series 314 Learning acoustic feature series 315 Learning result 316 Language feature series 317 Acoustic feature series 318 Spectral information 319 Sound source information

Claims (5)

When the first pitch to be specified according to the first timing is specified, the singing voice data of the first pitch corresponding to the first character corresponding to the first timing is output, and the singing voice data of the first pitch is output. When the second pitch to be specified according to the next second timing is specified before the second timing, the second character corresponding to the second timing without waiting for the arrival of the second timing. vocals output process of outputting the voice data of the second pitch corresponding to,
In the output singing data of the first pitch corresponding to the first character corresponding to the previous SL first timing, the first pitch and the third pitch other than the second pitch is higher than the second timing Also, when specified before, a pitch change process for changing the pitch of the singing voice data of the first pitch corresponding to the first character being output to the third pitch, and
An electronic musical instrument that runs. Based on the trained model generated by machine learning using the singing voice data and lyrics data of a certain singer, a voice synthesis process for synthesizing the singing voice according to the certain singer is executed.
The electronic according to claim 1, wherein the singing voice output process outputs a singing voice synthesized by the voice synthesis processing according to the timing when the operator is operated at a pitch specified by the operated operator. Musical instrument. The electronic musical instrument according to claim 1 or 2, wherein a display process for displaying an identifier indicating a pitch to be specified is executed before the timing at which the pitch should be specified by operating the operator. The electronic musical instrument of the computer,
When the first pitch to be specified according to the first timing is specified, the singing voice data of the first pitch corresponding to the first character corresponding to the first timing is output, and the singing voice data of the first pitch is output. When the second pitch to be specified according to the next second timing is specified before the second timing, the second character corresponding to the second timing without waiting for the arrival of the second timing. vocals output process of outputting the voice data of the second pitch corresponding to,
In the output singing data of the first pitch corresponding to the first character corresponding to the previous SL first timing, the first pitch and the third pitch other than the second pitch is higher than the second timing Also, when specified before, a pitch change process for changing the pitch of the singing voice data of the first pitch corresponding to the first character being output to the third pitch, and
How to run. The electronic musical instrument of the computer,
When the first pitch to be specified according to the first timing is specified, the singing voice data of the first pitch corresponding to the first character corresponding to the first timing is output, and the singing voice data of the first pitch is output. When the second pitch to be specified according to the next second timing is specified before the second timing, the second character corresponding to the second timing without waiting for the arrival of the second timing. vocals output process of outputting the voice data of the second pitch corresponding to,
In the output singing data of the first pitch corresponding to the first character corresponding to the previous SL first timing, the first pitch and the third pitch other than the second pitch is higher than the second timing Also, when specified before, a pitch change process for changing the pitch of the singing voice data of the first pitch corresponding to the first character being output to the third pitch, and
A program that executes.