JP6766935B2 - Electronic musical instruments, control methods for electronic musical instruments, and programs - Google Patents

Description

The present invention relates to an electronic musical instrument that reproduces a singing voice in response to an operation of an operator such as a keyboard, a control method for the electronic musical instrument, and a program.

Conventionally, there is known a technique of an electronic musical instrument that reproduces a singing voice (vocal) in response to an operation of an operator such as a keyboard (for example, Patent Document 1). In this conventional technique, when there is a keyboard operator for instructing a pitch, a storage means for storing lyrics data, an instruction means for instructing that the lyrics data should be read from the storage means, and an instruction by the instruction means. It is provided with a reading means for sequentially reading lyrics data from the storage means, and a sound source for generating a singing voice of a pitch instructed by a keyboard operator with a tone color corresponding to the lyrics data read by the reading means.

JP-A-6-332449

However, in the above-mentioned conventional technique, for example, when trying to output a singing voice according to the lyrics according to the progress of the accompaniment data output by the electronic musical instrument, the key is specified regardless of which key is specified by the performer. If the singing voice corresponding to the lyrics is sequentially output each time, the output singing voice and the progress of the accompaniment data do not match depending on how the performer specifies the key. For example, if one bar contains four notes, and the performer specifies four or more pitches in one bar, the lyrics will advance ahead of the progress of the accompaniment data, and the performer will If the pitch specified in one bar is 3 or less, the lyrics will be delayed from the progress of the accompaniment data.

In this way, if the lyrics advance in sequence each time the performer specifies the pitch on the keyboard, for example, the lyrics will advance too much with respect to the accompaniment, or conversely, they will be too late. ..

An example of an electronic musical instrument of the embodiment is
Processing to start playback of song data with pitch data and lyrics data,
During playback of the song data, if the pitch is not specified by the user at the timing corresponding to the vocalization timing of the singing voice corresponding to the new lyrics in the lyrics data, the singing voice corresponding to the new lyrics is performed. When the pitch is output based on the pitch data included in the song data and the pitch is specified by the user at the timing corresponding to the timing of singing the singing voice according to the new lyrics, the new pitch is specified. Singing voice output control processing that controls the singing voice according to the lyrics to be output at the pitch specified by the user ,
To execute.

According to the present invention, it is possible to provide an electronic musical instrument that satisfactorily controls the progress of lyrics.

It is a figure which shows the appearance example of one Embodiment of an electronic keyboard instrument. It is a block diagram which shows the hardware configuration example of one Embodiment of the control system of an electronic keyboard instrument. It is a block diagram which shows the configuration example of the voice synthesis LSI. It is an operation explanatory diagram of the voice synthesis LSI. It is explanatory drawing of the lyrics control technique. It is a figure which shows the data structure example of this embodiment. It is a main flowchart which shows the control processing example of the electronic musical instrument in this embodiment. It is a flowchart which shows the detailed example of the initialization process, the tempo change process, and the song start process. It is a flowchart which shows the detailed example of a switch process. It is a flowchart which shows the detailed example of the automatic performance interrupt processing. It is a flowchart which shows the detailed example of the 1st Embodiment of a song reproduction processing. It is a flowchart which shows the detailed example of the 2nd Embodiment of a song reproduction process. It is a figure which shows the composition example of the lyrics control data by the MusicXML format. It is a figure which shows the musical score display example by the lyrics control data in the MusicXML format.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an external example of an embodiment 100 of an electronic keyboard instrument. The electronic keyboard instrument 100 includes a keyboard 101 composed of a plurality of keys as a performance operator, and a first switch panel 102 for instructing various settings such as volume designation, song playback tempo setting, song playback start, and accompaniment playback. , A second switch panel 103 for selecting songs and accompaniment songs, selecting tones, and the like, and an LCD 104 (Liquid Keyboard Display) for displaying lyrics, scores, and various setting information during song playback. Further, although not particularly shown, the electronic keyboard instrument 100 is provided with a speaker for emitting a musical sound generated by the performance on a back surface portion, a side surface portion, a back surface portion, or the like.

FIG. 2 is a diagram showing a hardware configuration example of an embodiment of the control system 200 of the electronic keyboard instrument 100 of FIG. In FIG. 2, the control system 200 includes a CPU (central processing unit) 201, a ROM (read-only memory) 202, a RAM (random access memory) 203, a sound source LSI (large-scale integrated circuit) 204, a voice synthesis LSI 205, and FIG. The key scanner 101 to which the keyboard 101, the first switch panel 102, and the second switch panel 103 are connected, and the LCD controller 208 to which the LCD 104 of FIG. 1 is connected are connected to the system bus 209, respectively. Further, a timer 210 for controlling the sequence of automatic performance is connected to the CPU 201. Further, the musical sound output data 218 and the singing voice output data 217 output from the sound source LSI 204 and the voice synthesis LSI 205, respectively, are converted into an analog musical sound output signal and an analog singing voice output signal by the D / A converters 211 and 212, respectively. The analog musical tone output signal and the analog singing voice output signal are mixed by the mixer 213, and after the mixed signal is amplified by the amplifier 214, they are output from a speaker or an output terminal (not shown).

The CPU 201 executes the control operation of the electronic keyboard instrument 100 of FIG. 1 by executing the control program stored in the ROM 202 while using the RAM 203 as the work memory. In addition to the control program and various fixed data, the ROM 202 stores song data including lyrics data and accompaniment data.

The timer 210 used in the present embodiment is mounted on the CPU 201, and counts the progress of the automatic performance of the electronic keyboard instrument 100, for example.

The sound source LSI 204 reads the musical tone type data from, for example, a waveform ROM (not shown) in accordance with the sound generation control instruction from the CPU 201, and outputs the music to the D / A converter 211. The sound source LSI 204 has the ability to oscillate up to 256 voices at the same time.

When the voice synthesis LSI 205 is given text data of lyrics, pitch, pitch, and start frame as singing voice data 215 from the CPU 201, the voice synthesis LSI 205 synthesizes the corresponding singing voice voice data and outputs it to the D / A converter 212. To do.

The key scanner 206 constantly scans the key press / release state of the key 101 of FIG. 1, the switch operation state of the first switch panel 102, and the second switch panel 103, and interrupts the CPU 201. Communicate change.

The LCD controller 208 is an IC (integrated circuit) that controls the display state of the LCD 104.

FIG. 3 is a block diagram showing a configuration example of the speech synthesis LSI 205 of FIG. This speech synthesis LSI 205 is based on, for example, the technique of "statistical speech synthesis based on deep learning" described in the following document by inputting the singing voice data 215 instructed from the CPU 201 of FIG. 2 by the song reproduction process described later. , Singing voice voice output data 217 is synthesized and output.

(Reference)
Yoshi Hashimoto, Shinji Takagi, "Statistical Speech Synthesis Based on Deep Learning," Journal of the Acoustical Society of Japan, Vol. 73, No. 1 (2017), pp. 55-62

The speech synthesis LSI 205 includes a speech learning unit 301 and a speech synthesis unit 302. The voice learning unit 301 includes a learning text analysis unit 303, a learning acoustic feature amount extraction unit 304, and a model learning unit 305.

The learning text analysis unit 303 inputs the learning singing voice data 311 including the lyrics text, the pitch and the pitch, and analyzes the data. As a result, the learning text analysis unit 303 estimates and outputs the learning language feature sequence 313, which is a discrete numerical sequence expressing phonemes, parts, words, pitches, etc. corresponding to the learning singing voice data 311.

The learning acoustic feature amount extraction unit 304 inputs and analyzes the learning singing voice voice data 312 recorded through a microphone or the like by a certain singer singing the lyrics text. As a result, the learning acoustic feature amount extraction unit 304 extracts and outputs the learning acoustic feature amount series 314 representing the voice features corresponding to the learning singing voice voice data 312.

The model learning unit 305 uses the following equation (1) to obtain a learning language feature sequence 313 (this is used).
And the acoustic model (put this)
From), the learning acoustic feature series 314 (this is
Probability of generating (put this)
And put) an acoustic model that maximizes
Is estimated by machine learning. That is, the relationship between the language feature series that is text and the acoustic feature series that is voice is expressed by a statistical model called an acoustic model.

The model learning unit 305 is an acoustic model calculated as a result of performing machine learning according to equation (1).
The model parameter expressing the above is output as the learning result 315 and set in the acoustic model unit 306 in the voice synthesis unit 302.

The speech synthesis unit 302 includes a text analysis unit 307, an acoustic model unit 306, and a vocal model unit 308. The voice synthesis unit 302 synthesizes the singing voice output data 217 corresponding to the singing voice data 215 including the lyrics text by predicting it using a statistical model called an acoustic model set in the acoustic model unit 306. Execute the process.

The text analysis unit 307 inputs the text data of the lyrics designated by the CPU 201 of FIG. 2 and the singing voice data 215 including information on the pitch, the pitch, and the start frame as the result of the performer's performance in accordance with the automatic performance. , Analyze the data. As a result, the text analysis unit 307 analyzes and outputs the language feature sequence 316 expressing the phonemes, part of speech, words, etc. corresponding to the singing voice data 215.

By inputting the language feature sequence 316, the acoustic model unit 306 estimates and outputs the corresponding acoustic feature sequence 317. That is, the acoustic model unit 306 uses the following equation (2) to input the language feature quantity series 316 from the text analysis unit 307 (this is repeated).
), And the acoustic model set as the learning result 315 by machine learning in the model learning unit 305.
Based on and, the acoustic feature series 317 (again)
Probability of generating (put this)
Estimated value of acoustic feature series 317 that maximizes
To estimate.

The vocalization model unit 308 generates the singing voice output data 217 corresponding to the singing voice data 215 including the lyrics text designated by the CPU 201 by inputting the acoustic feature amount series 317. The singing voice output data 217 is output from the D / A converter 212 of FIG. 2 via the mixer 213 and the amplifier 214, and is particularly emitted from a speaker (not shown).

The acoustic features represented by the learning acoustic feature series 314 and the acoustic feature series 317 include spectral information that models the human vocal tract and sound source information that models the human vocal cords. As the spectrum parameter, for example, a mer cepstrum, a line spectrum pair (Line Spectral Pairs: LSP), or the like can be adopted. As the sound source information, a fundamental frequency (F0) indicating the pitch frequency of human voice can be adopted. The vocalization model unit 308 includes a sound source generation unit 309 and a synthetic filter unit 310. The sound source generation unit 309 sequentially inputs a series of sound source information 319 input from the acoustic model unit 306, so that the sound source information 319 is periodically repeated at the fundamental frequency (F0) included in the sound source information 319, and is included in the sound source information 319. A sound source signal consisting of a pulse train having a power value (in the case of a voiced phoneme) or white noise having a power value included in the sound source information 319 (in the case of an unvoiced phoneme) is generated. The synthetic filter unit 310 forms a digital filter that models the voice path based on the sequence of spectrum information 318 sequentially input from the acoustic model unit 306, and digitally uses the sound source signal input from the sound source generation unit 309 as an excitation source signal. The singing voice audio output data 217 of the signal is generated and output.

In the present embodiment, in order to predict the acoustic feature sequence 317 from the language feature sequence 316, the acoustic model unit 306 is implemented by a deep neural network (DNN). Correspondingly, the model learning unit 305 in the speech learning unit 301 learns the model parameters representing the nonlinear conversion function of each neuron in the DNN from the language feature quantity to the acoustic feature quantity, and learns the model parameters. As 315, it is output to the DNN of the acoustic model unit 306 in the voice synthesis unit 302.

Usually, the acoustic feature amount is calculated in units of a frame having a width of, for example, 5.1 msec (milliseconds), and the language feature amount is calculated in units of phonemes. Therefore, the acoustic feature amount and the language feature amount have different time units. Since the acoustic model unit 306, which is a DNN, is a model representing a one-to-one correspondence between the input language feature sequence 316 and the output acoustic feature sequence 317, input / output data pairs having different time units are used. DNN cannot be trained. Therefore, in the present embodiment, the correspondence between the frame-based acoustic feature sequence and the phoneme-based language feature sequence is set in advance, and a pair of frame-based acoustic feature and language feature is generated.

FIG. 4 is an operation explanatory diagram of the speech synthesis LSI 205 showing the above-mentioned correspondence. For example, the singing phoneme sequence "/ k /" "/ i" which is a language feature sequence corresponding to the lyrics character strings "ki", "ra", and "ki" (Fig. 4 (a)) of the nursery rhyme "Kirakira Hoshi". When / ”,“ / r / ”,“ / a / ”,“ / k / ”, and“ / i / ”(FIG. 4 (b)) are obtained, these linguistic feature sequences are frame-based phoneme features. It is associated with a quantity series (FIG. 4 (c)) in a one-to-many relationship (relationship between (b) and (c) in FIG. 4). Since the language feature amount is used as an input to the DNN in the acoustic model unit 306, it needs to be expressed as numerical data. Therefore, as a language feature series, "Is the phoneme immediately before" / a / "? Binary data (0 or 1) for contextual questions such as "How many phonemes are contained in the current word?", Or numerical data obtained by concatenating consecutive answers are prepared. ..

The model learning unit 305 in the voice learning unit 301 of FIG. 3 is a phoneme sequence of the learning language feature series 313 corresponding to FIG. 4 (b) in frame units as shown by the broken line arrow group 401 of FIG. And the pair of the learning acoustic feature quantity series 314 corresponding to FIG. 4C are sequentially given to the DNN of the acoustic model unit 306 for learning. The DNN in the acoustic model unit 306 includes a neuron group consisting of an input layer, one or more intermediate layers, and an output layer, as shown as a group of gray circles in FIG.

On the other hand, at the time of speech synthesis, the phoneme sequence of the language feature sequence 316 corresponding to FIG. 4B is input to the DNN of the acoustic model unit 306 in the frame unit. As a result, the DNN of the acoustic model unit 306 outputs the acoustic feature amount series 317 in the frame unit as shown by the thick solid line arrow group 402 in FIG. Therefore, also in the vocalization model unit 308, the sound source information 319 and the spectrum information 318 included in the acoustic feature quantity series 317 are given to the sound source generation unit 309 and the synthesis filter unit 310, respectively, in the above-mentioned frame unit, and the voice synthesis is executed. Will be done.

As a result, the vocalization model unit 308 outputs, for example, 225 samples of singing voice output data 217 for each frame, as shown by the thick solid line arrow group 403 in FIG. Since the frame has a time width of 5.1 msec, one sample is "5.1 msec ÷ 225 ≈ 0.0227 msec", and therefore the sampling frequency of the singing voice output data 217 is 1 / 0.0227 ≈ 44 kHz (kilohertz). Is.

The DNN learning is performed by the square error minimization standard calculated by the following equation (3) using the pair of the acoustic feature amount and the language feature amount in each frame.

here,
When
Are the acoustic features and language features in the t-th frame t, respectively.
Is the DNN model parameter of the acoustic model unit 306,
Is a non-linear transformation function represented by DNN. The DNN model parameters can be efficiently estimated by the error back propagation method. Considering the correspondence with the processing of the model learning unit 305 in the statistical speech synthesis represented by the above-mentioned equation (1), the learning of DNN can be expressed as the following equation (4).

Here, the following equation (5) holds.

As in Eqs. (4) and (5) above, the relationship between acoustic features and linguistic features is a normal distribution with the output of DNN as the average vector.
Can be represented by. In statistical speech synthesis processing using DNN, language features are usually used.
Covariance matrix independent of, i.e. the same covariance matrix in all frames
Is used. Also, the covariance matrix
When is an identity matrix, Eq. (4) shows the learning process equivalent to Eq. (3).

As described with reference to FIG. 4, the DNN of the acoustic model unit 306 estimates the acoustic feature sequence 317 independently for each frame. Therefore, the obtained acoustic feature sequence 317 includes discontinuities that deteriorate the quality of the synthesized speech. Therefore, in the present embodiment, for example, the quality of the synthesized speech can be improved by using a parameter generation algorithm using dynamic features.

The operation of the present embodiment having the configuration examples of FIGS. 1, 2 and 3 will be described in detail below. FIG. 5 is an explanatory diagram of the lyrics control technique. FIG. 5A is a diagram showing the relationship between the lyric text and the melody that progresses according to the automatic performance. For example, in the case of singing the above-mentioned children's song "Kirakira Hoshi", the song data includes "ki / Twin (first character)", "ra / kle (second character)", and "ki / twin (third character)". Each character (lyric information) of the lyrics of "ra / kle (4th character)", each timing information of t1, t2, t3, t4 that outputs each character of the lyrics, and the melody pitch of each character of the lyrics " Each pitch information such as E4 (first pitch), "E4 (second pitch)", "B4 (third pitch)", and "B4 (fourth pitch)" is included. At each timing of t5, t6, and t7 after t4, each character of the lyrics of "hi / lit (5th character)", "ka / tre (6th character)", and "ru / star (7th character)" Are associated with each other.

For example, the timings of t1, t2, t3, and t4 in FIG. 5B correspond to the original vocalization timings t1, t2, t3, and t4 in FIG. 5A. Here, at the timings t1 and t2 corresponding to the original vocalization timing, the performer presses the key of the same pitch E4 as the first pitch E4 indicated by the pitch information included in the song data on the keyboard 101 of FIG. Suppose you press the key correctly twice. In this case, the CPU 201 of FIG. 2 is designated by the performer as the lyrics "ki / Twin (first character)" and "ra / kle (second character)" at timings corresponding to the timings t1 and t2, respectively. FIG. 2 shows singing voice data 215 including information indicating the pitch E4 and, for example, information indicating the time length of each quarter note length (obtained at least based on either the song data or the performance by the performer). Is output to the voice synthesis LSI 205 of. As a result, the voice synthesis LSI 205 has quarter note length singing voice output data 217 corresponding to the lyrics "ki / Twin (first character)" and "ra / kle (second character)" at timings t1 and t2, respectively. Is output at the first pitch (= specified pitch) E4 and the second pitch (= specified pitch) E4, respectively. The determination "○" mark corresponding to the timings t1 and t2 indicates that the utterance was correctly performed according to the pitch and lyrics information indicated by the pitch information included in the song data.

Further, it is assumed that the performer presses a key having a pitch G4 different from the original fourth pitch B4 on the keyboard 101 of FIG. 1 at the timing t4 corresponding to the original vocalization timing. In this case, the CPU 201 specifies the lyrics "ra / kle (fourth character)" at the timing t4, and the pitch G4 corresponding to the key played at the timing t4, for example, the time length of the eighth note length is The designated singing voice data 215 is output to the voice synthesis LSI 205 of FIG. As a result, the voice synthesis LSI 205 outputs the singing voice output data 217 of the eighth note length corresponding to the lyrics "ra / kle (fourth character)" at the timing t4 at the played (keyed) pitch G4. ..

According to this embodiment, even when the performer performs the performance (key press) operation at the timing corresponding to the original vocalization timing, the pitch specified by the performer's operation is converted into the singing voice audio output data 217. By reflecting it, it is possible to better reflect the intention of the performer in the singing voice uttered.

Next, at the original utterance timing, if the performer does not press any of the keys on the keyboard 101 of FIG. 1 at that timing and the pitch is not specified, the following control is executed. To. The CPU 201 of FIG. 2 controls to output a singing voice corresponding to the character (lyric information) corresponding to the timing so as to output the singing voice at the pitch indicated by the pitch information included in the song data. As a result, the voice synthesis LSI 205 of FIG. 2 or FIG. 3 outputs the singing voice voice output data 217 corresponding to the characters corresponding to the timing at the pitch indicated by the pitch information included in the song data. To do.

For example, in FIG. 5B, it is assumed that the performer does not play (press) the key of the keyboard 101 of FIG. 1 at the timing t3 corresponding to the original vocalization timing. In this case, the CPU 201 of FIG. 2 outputs the singing voice corresponding to the lyrics information "ki / twin (third character)" corresponding to the timing t3 at the third pitch B4 indicated by the pitch information included in the song data. The singing voice data 215 specified as described above is output to the voice synthesis LSI 205 of FIG. As a result, the voice synthesis LSI 205 of FIG. 2 or FIG. 3 corresponds to the singing voice voice output data 217 corresponding to the lyrics information "ki / twin (third character)" corresponding to the timing t3 in accordance with the timing t3. Output at the third pitch B4.

The timing t3 of FIG. 5C is based on the assumption that the above-mentioned control operation according to the present embodiment is not performed, and the performer responds to the timing t3 corresponding to the original vocalization timing of the keyboard 101 of FIG. This is an explanation of the control operation when the key of is not pressed. When the above-mentioned control operation according to the present embodiment is not performed, the lyrics character string “ki / twin (third character)” that should be originally uttered is uttered at the timing t3 in FIG. 5 (c). Absent.

As described above, when the performer does not perform the performance operation at the original utterance timing, if the control operation according to the present embodiment is not executed, the lyric character string to be uttered is not uttered, which is not possible. It had become a natural feeling. For example, when the melody is played along with the automatic accompaniment, the output by the automatic accompaniment goes ahead of the output of the singing voice according to the lyrics. On the other hand, in the present embodiment, when the performer does not perform the performance operation at the original vocal timing, the song is sung according to the lyrics information (characters) included in the song data corresponding to the timing. It is possible to output at the pitch corresponding to the lyrics information (characters) included in the data. Thereby, in the present embodiment, the lyric progression can be performed naturally.

Next, when the performer plays (presses) an arbitrary key (operator) of the keyboard 101 of FIG. 1 at a timing when none of the original vocalization timings have arrived, the CPU 201 of FIG. The singing voice data 215 instructing to change the pitch of the singing voice corresponding to the singing voice voice output data 217 output in the voice synthesis LSI 205 to the pitch specified by the performance operation is output to the voice synthesis LSI 205 of FIG. To do. As a result, in the voice synthesis LSI 205 of FIG. 2 or 3, the pitch of the singing voice output data 217 being uttered is set to the sound specified by the CPU 201 at a timing when none of the original utterance timings has arrived. Change to high.

For example, in FIG. 5B, at the timings t1', t3', and t4'in which none of the original vocalization timings t1, t2, t3, and t4 has arrived, the performer performs the key on the keyboard 101 of FIG. It is assumed that the keys of pitches G4, A4, and E4 are pressed, respectively. In this case, the CPU 201 has the lyrics character strings “ki / twin (first character)”, “ki / twin (third character)”, and “ra / kle (fourth character)” output from the speech synthesis LSI 205, respectively. Singing voice data 215 instructing to change each pitch E4, B4, and G4 of the singing voice output data 217 to each pitch G4, A4, and E4 specified by the performance operation and continue utterance. Is output to the voice synthesis LSI 205 of FIG. As a result, the speech synthesis LSI 205 of FIG. 2 or 3 has "i / in (1st character)" corresponding to the utterance of the lyrics character string "ki / Twin (first character)" at the timings t1', t3', and t4'. "I / in (3rd character')" according to "1st character')", "ki (3rd character)", and "a / le (4th character)" according to "ra (4th character)" Each pitch of the singing voice voice output data 217 of "')" is changed to each pitch G4, A4, and E4 designated by the CPU 201, and the utterance is continued.

The timings t1', t3', and t4'in FIG. 5C show the timings t1', other than the original vocalization timing, assuming that the above-mentioned control operation according to the present embodiment is not performed. The control operation when the key of the keyboard 101 of FIG. 1 is played (pressed) is described in t3'and t4'. When the above-mentioned control operation according to the present embodiment is not performed, at the timings t1', t3', and t4' in FIG. 5C, singing voices corresponding to the following lyrics, which are not the original vocalization timings, are produced, respectively. It is output and the lyrics progress.

As described above, when the performer performs the performance operation at a timing other than the original vocalization timing, if the control operation according to the present embodiment is not executed, the progress of the lyrics will proceed, which is not possible. It had become a natural feeling. On the other hand, in the present embodiment, the pitch of the singing voice output data 217 uttered at that timing is changed to the pitch played by the performer and continued. In this case, for example, "ki / Twin (first character)" and "ki / twin (third character)" uttered at the original song playback timings t1, t3, and t4 in FIG. 5 (b). And the singing voice output data 217 corresponding to "ra / kle (fourth character)" is specified by a new key press at each key press timing t1', t3', and t4' without interruption. It sounds like it changes continuously in pitch. Thereby, in the present embodiment, the lyric progression can be performed naturally.

When the performer performs a performance operation at a timing other than the original utterance timing, the pitch of the utterance based on the singing voice output data 217 uttered at that timing is changed to the pitch specified by the performer. Then, it may be controlled to repeat newly. In this case, for example, "ki / Twin (first character)", "ki / twin (third character)" uttered at the original song playback timings t1, t3, and t4 in FIG. 5 (b). And, following the singing voice output data 217 corresponding to "ra / kle (fourth character)", at the respective pitches specified by the new key presses at the key press timings t1', t3', and t4', It sounds like the singing voice output data 217 corresponding to "ki / twin (first character)", "ki / twin (third character)", and "ra / kle (fourth character)" is uttered separately. Alternatively, it may be controlled so that the singing voice output data 217 is not uttered at a timing other than the utterance timing.

Furthermore, when the performer performs a performance operation at a timing other than the original utterance timing, the utterance of the singing voice output data 217 that should be uttered not immediately before the timing but immediately after the timing is specified by the performer. It may be controlled so that it is uttered in advance at a different pitch. In this case, for example, "ra / kle (second character)" and "ra / kle (fourth character)" to be uttered at the original song playback timings t2, t4, and t5 in FIG. 5 (b). , And the singing voice output data 217 corresponding to "hi / lit (fifth character)", and the respective pitches specified by the new key presses at the key press timings t1', t3', and t4'. The singing voice output data 217 corresponding to "ra / kle (second character)", "ra / kle (fourth character)", and "hi / lit (fifth character)" may be uttered.

FIG. 6 is a diagram showing a data configuration example of song data read from ROM 202 to RAM 203 in FIG. 2 in the present embodiment. This data structure example conforms to the standard MIDI file format, which is one of the MIDI (Musical Instrument Digital Interface) file formats. This song data is composed of data blocks called chunks. Specifically, the song data consists of a header chunk at the beginning of the file, a track chunk 1 in which the lyrics data for the following lyrics part is stored, and a track chunk 2 in which the performance data for the accompaniment part is stored. It is composed.

The header chunk consists of four values: ChunkID, ChunkSize, FormatType, NumberOfTrack, and TimeDivision. The Chunk ID is a 4-byte ASCII code "4D 54 68 64" (numbers are hexadecimal numbers) corresponding to four single-byte characters "MThd" indicating that it is a header chunk. The ChunkSize is 4-byte data indicating the data length of the FormatType, NumberOfTrack, and TimeDivision parts excluding the ChunkID and the ChunkSize in the header chunk, and the data length is 6 bytes: "00 00 00 06" (numbers are hexadecimal numbers). It is fixed to. In the case of this embodiment, the Format Type is 2-byte data "00 01" (numbers are hexadecimal numbers), which means format 1 using a plurality of tracks. In the case of the present embodiment, the NumberOfTrack is 2-byte data "00 02" (numbers are hexadecimal numbers) indicating that two tracks corresponding to the lyrics part and the accompaniment part are used. The Time Division is data indicating a time base value indicating a resolution per quarter note, and in the case of the present embodiment, it is 2-byte data "01 E0" (number is a hexadecimal number) indicating 480 in decimal notation.

Track chunks 1 and 2 are ChunkID, ChunkSize, and DataTime_1 [i] and Event_1 [i] (in the case of track chunk 1 / lyrics part) or DeltaTime_2 [i] and Event_2 [i] (track chunk 2 / accompaniment part, respectively). Case) is composed of a performance data set (0 ≦ i ≦ L: track chunk 1 / lyrics part, 0 ≦ i ≦ M: track chunk 2 / accompaniment part). The Chunk ID is a 4-byte ASCII code "4D 54 72 6B" (numbers are hexadecimal numbers) corresponding to four single-byte characters "MTrk" indicating that it is a track chunk. The ChunkSize is 4-byte data indicating the data length of the portion of each track chunk excluding the ChunkID and the ChunkSize.

DeltaTime_1 [i] is variable length data of 1 to 4 bytes indicating the waiting time (relative time) from the execution time of Event_1 [i-1] immediately before that. Similarly, DeltaTime_2 [i] is variable length data of 1 to 4 bytes indicating the waiting time (relative time) from the execution time of Event_2 [i-1] immediately before that. Event_1 [i] is a meta-event that instructs the utterance timing and pitch of the lyrics in the track chunk 1 / lyrics part. Event_2 [i] is a MIDI event instructing note-on or note-off, or a meta-event instructing time signature in the track chunk 2 / accompaniment part. For each performance data set DeltaTime_1 [i] and Event_1 [i] for the track chunk 1 / lyrics part, after waiting for DeltaTime_1 [i] from the execution time of Event_1 [i-1] immediately before that, Event_1 [i] Is executed, the vocalization progress of the lyrics is realized. On the other hand, for the track chunk 2 / accompaniment part, in each performance data set DeltaTime_2 [i] and Event_2 [i], after waiting for the execution time of Event_2 [i-1] immediately before that, Event_2 [i] By executing i], the progress of automatic accompaniment is realized.

FIG. 7 is a main flowchart showing an example of control processing of an electronic musical instrument according to the present embodiment. This control process is, for example, an operation in which the CPU 201 of FIG. 2 executes a control process program loaded from the ROM 202 into the RAM 203.

The CPU 201 first executes the initialization process (step S701), and then repeatedly executes a series of processes from steps S702 to S708.

In this iterative process, the CPU 201 first executes the switch process (step S702). Here, the CPU 201 executes a process corresponding to the switch operation of the first switch panel 102 or the second switch panel 103 of FIG. 1 based on the interrupt from the key scanner 206 of FIG.

Next, the CPU 201 executes a keyboard process for determining and processing whether or not any key of the key 101 of FIG. 1 has been operated based on the interrupt from the key scanner 206 of FIG. 2 (step S703). Here, the CPU 201 outputs sound control data 216 instructing the sound source LSI 204 of FIG. 2 to start or stop sounding in response to an operation of pressing or releasing any key by the performer.

Next, the CPU 201 processes data to be displayed on the LCD 104 of FIG. 1, and executes a display process of displaying the data on the LCD 104 via the LCD controller 208 of FIG. 2 (step S704). The data displayed on the LCD 104 includes, for example, the lyrics corresponding to the singing voice output data 217 to be played, the score of the melody corresponding to the lyrics, and various setting information (see FIGS. 13 and 14 described later).

Next, the CPU 201 executes the song playback process (step S705). In this process, the CPU 201 executes the control process described with reference to FIG. 5 based on the performance of the performer, generates singing voice data 215, and outputs the singing voice data 215 to the speech synthesis LSI 205.

Subsequently, the CPU 201 executes sound source processing (step S706). In the sound source processing, the CPU 201 executes control processing such as envelope control of the musical sound being sounded in the sound source LSI 204.

Subsequently, the CPU 201 executes the speech synthesis process (step S707). In the speech synthesis process, the CPU 201 controls the execution of speech synthesis by the speech synthesis LSI 205.

Finally, the CPU 201 determines whether or not the performer has pressed a power-off switch (not shown) to power off (step S708). If the determination in step S708 is NO, the CPU 201 returns to the process in step S702. If the determination in step S708 is YES, the CPU 201 ends the control process shown in the flowchart of FIG. 7, and turns off the power of the electronic keyboard instrument 100.

8 (a), (b), and (c) are the initialization process of step S701 of FIG. 7, the tempo change process of step S902 of FIG. 9, which will be described later in the switch process of step S702 of FIG. 7, and the same. It is a flowchart which shows the detailed example of the song start processing of step S906 of FIG.

First, in FIG. 8A showing a detailed example of the initialization process in step S701 of FIG. 7, the CPU 201 executes the ticktime initialization process. In the present embodiment, the progress of the lyrics and the automatic accompaniment proceed in units of time called TickTime. The timebase value specified as the TimeDivision value in the header chunk of the song data of FIG. 6 indicates the resolution of the quarter note, and if this value is, for example, 480, the quarter note has a time length of 480TickTime. Further, the waiting time DeltaTime_1 [i] value and the DeltaTime_2 [i] value in the track chunk of the song data of FIG. 6 are also counted by the time unit of TickTime. Here, how many seconds 1 Tick Time actually becomes depends on the tempo specified for the song data. If the tempo value is Tempo [beat / minute] and the time base value is Time Division, the number of seconds of Tick Time is calculated by the following equation.

TickTime [seconds] = 60 / Tempo / TimeDivision (6)

Therefore, in the initialization process exemplified by the flowchart of FIG. 8A, the CPU 201 first calculates the TickTime [seconds] by the arithmetic processing corresponding to the above equation (6) (step S801). It is assumed that a predetermined value, for example, 60 [beats / second] is stored in the ROM 202 of FIG. 2 in the initial state of the tempo value Tempo. Alternatively, the tempo value at the time of the previous end may be stored in the non-volatile memory.

Next, the CPU 201 sets a timer interrupt according to the TickTime [seconds] calculated in step S801 with respect to the timer 210 of FIG. 2 (step S802). As a result, every time the TickTime [seconds] elapses in the timer 210, an interrupt for lyrics progression and automatic accompaniment (hereinafter referred to as “automatic performance interrupt”) is generated for the CPU 201. Therefore, in the automatic performance interrupt process (FIG. 10 described later) executed by the CPU 201 based on this automatic performance interrupt, the control process for advancing the lyrics progress and the automatic accompaniment is executed for each TickTime.

Subsequently, the CPU 201 executes other initialization processing such as initialization of the RAM 203 of FIG. 2 (step S803). After that, the CPU 201 ends the initialization process of step S701 of FIG. 7 exemplified by the flowchart of FIG. 8A.

The flowcharts of FIGS. 8B and 8C will be described later. FIG. 9 is a flowchart showing a detailed example of the switch processing in step S702 of FIG.

First, the CPU 201 determines whether or not the tempo of the lyrics progression and the automatic accompaniment has been changed by the tempo change switch in the first switch panel 102 of FIG. 1 (step S901). If the determination is YES, the CPU 201 executes the tempo change process (step S902). Details of this process will be described later with reference to FIG. 8 (b). If the determination in step S901 is NO, the CPU 201 skips the process in step S902.

Next, the CPU 201 determines whether or not any song song has been selected on the second switch panel 103 of FIG. 1 (step S903). If the determination is YES, the CPU 201 executes the song song reading process (step S904). This process is a process of reading the song data having the data structure described in FIG. 6 from the ROM 202 of FIG. 2 into the RAM 203. From then on, data access to track chunks 1 or 2 in the data structure illustrated in FIG. 6 is performed on the song data read into RAM 203. If the determination in step S903 is NO, the CPU 201 skips the process in step S904.

Subsequently, the CPU 201 determines whether or not the song start switch has been operated on the first switch panel 102 of FIG. 1 (step S905). If the determination is YES, the CPU 201 executes the song start process (step S906). Details of this process will be described later with reference to FIG. 8 (c). If the determination in step S905 is NO, the CPU 201 skips the process in step S906.

Finally, the CPU 201 determines whether or not other switches have been operated on the first switch panel 102 or the second switch panel 103 of FIG. 1, and executes a process corresponding to each switch operation (step S907). .. After that, the CPU 201 ends the switch process in step S702 of FIG. 7, which is illustrated in the flowchart of FIG.

FIG. 8B is a flowchart showing a detailed example of the tempo change process in step S902 of FIG. As mentioned above, when the tempo value is changed, the TickTime [seconds] is also changed. In the flowchart of FIG. 8B, the CPU 201 executes the control process related to the change of the TickTime [seconds].

First, the CPU 201 is subjected to the operation process corresponding to the above-described equation (6) in the same manner as in the case of step S801 of FIG. 8A, which is executed in the initialization process of step S701 of FIG. Is calculated (step S811). It is assumed that the tempo value Tempo is stored in the RAM 203 or the like after being changed by the tempo change switch in the first switch panel 102 of FIG.

Next, the CPU 201 performs the TickTime [calculated in step S811 with respect to the timer 210 of FIG. 2 in the same manner as in the case of step S802 of FIG. 8A executed in the initialization process of step S701 of FIG. Seconds] sets the timer interrupt (step S812). After that, the CPU 201 ends the tempo change process of step S902 of FIG. 9 exemplified by the flowchart of FIG. 8 (b).

FIG. 8C is a flowchart showing a detailed example of the song start process of step S906 of FIG.

First, in the progress of the automatic performance, the CPU 201 sets the values of the variables DeltaT_1 (track chunk 1) and DeltaT_2 (track chunk 2) on the RAM 203 for counting the relative time from the occurrence time of the immediately preceding event in units of TickTime. Is initially set to 0 for both. Next, the CPU 201 is a RAM 203 for designating each i of the performance data sets DeltaTime_1 [i] and Event_1 [i] (1 ≦ i ≦ L-1) in the track chunk 1 of the song data exemplified in FIG. Each value of the above variable AutoIndex_1 and the variable AutoIndex_1 on the RAM 203 for designating each i of the performance data sets DeltaTime_2 [i] and Event_2 [i] (1 ≦ i ≦ M-1) also in the track chunk 2. Both are initially set to 0 (above, step S821). As a result, in the example of FIG. 6, as an initial state, first, the first performance data set DeltaTime_1 [0] and Event_1 [0] in the track chunk 1, and the first performance data set DeltaTime_2 [0] in the track chunk 2 Event_1 [0] is referenced respectively.

Next, the CPU 201 initially sets the value of the variable SongIndex on the RAM 203 that indicates the current song position to 0 (step S822).

Further, the CPU 201 initializes the value of the variable SongStart on the RAM 203 indicating whether the lyrics and accompaniment progress (= 1) or not (= 0) to 1 (progress) (step S823).

After that, the CPU 201 determines whether or not the performer has set the first switch panel 102 of FIG. 1 to reproduce the accompaniment in accordance with the reproduction of the lyrics (step S824).

If the determination in step S824 is YES, the CPU 201 sets the value of the variable Bansou on the RAM 203 to 1 (with accompaniment) (step S825). On the contrary, if the determination in step S824 is NO, the CPU 201 sets the value of the variable Bansou to 0 (no accompaniment) (step S826). After the process of step S825 or S826, the CPU 201 ends the song start process of step S906 of FIG. 9 shown in the flowchart of FIG. 8 (c).

FIG. 10 shows an automatic performance interrupt process executed based on an interrupt generated every TickTime [seconds] in the timer 210 of FIG. 2 (see step S802 of FIG. 8A or step S812 of FIG. 8B). It is a flowchart which shows the detailed example of. The following processing is executed on the performance data sets of the track chunks 1 and 2 of the song data exemplified in FIG.

First, the CPU 201 executes a series of processes (steps S1001 to S1006) corresponding to the track chunk 1. First, the CPU 201 determines whether or not the SongStart value is 1, that is, whether or not the progress of the lyrics and accompaniment is instructed (step S1001).

When the CPU 201 determines that the progress of the lyrics and the accompaniment is not instructed (the determination in step S1001 is NO), the CPU 201 does not proceed with the lyrics and the accompaniment, and is illustrated in the flowchart of FIG. The automatic performance interrupt processing ends as it is.

When the CPU 201 determines that the progress of the lyrics and the accompaniment is instructed (the determination in step S1001 is YES), the DeltaT_1 value indicating the relative time from the occurrence time of the previous event regarding the track chunk 1 is set. It is determined whether or not the waiting time DeltaTime_1 [AutoIndex_1] of the performance data set to be executed to be executed, which is indicated by the AutoIndex_1 value, is matched (step S1002).

If the determination in step S1002 is NO, the CPU 201 increments the DeltaT_1 value indicating the relative time from the occurrence time of the previous event by +1 with respect to the track chuck 1, and advances the time by 1 TickTime unit corresponding to the current interrupt. (Step S1003). After that, the CPU 201 shifts to step S1007, which will be described later.

When the determination in step S1002 becomes YES, the CPU 201 executes the event event [AutoIndex_1] of the performance data set indicated by the AutoIndex_1 value with respect to the track chuck 1 (step S1004). This event is a song event that includes lyrics data.

Subsequently, the CPU 201 stores the AutoIndex_1 value indicating the position of the next song event to be executed in the track chunk 1 in the variable SongIndex on the RAM 203 (step S1004).

Further, the CPU 201 increments the AutoIndex_1 value for referencing the performance data set in the track chunk 1 by +1 (step S1005).

Further, the CPU 201 resets the DeltaT_1 value indicating the relative time from the occurrence time of the song event referred to this time with respect to the track chunk 1 to 0 (step S1006). After that, the CPU 201 shifts to the process of step S1007.

Next, the CPU 201 executes a series of processes (steps S1007 to S1013) corresponding to the track chunk 2. First, the CPU 201 determines whether or not the DeltaT_2 value indicating the relative time from the occurrence time of the previous event regarding the track chunk 2 matches the waiting time DeltaTime_2 [AutoIndex_2] of the performance data set to be executed, which is indicated by the AutoIndex_2 value. Is determined (step S1007).

If the determination in step S1007 is NO, the CPU 201 increments the DeltaT_2 value indicating the relative time from the occurrence time of the previous event by +1 with respect to the track chuck 2, and advances the time by 1 TickTime unit corresponding to the current interrupt. (Step S1008). After that, the CPU 201 ends the automatic performance interrupt process shown in the flowchart of FIG.

If the determination in step S1007 is YES, the CPU 201 determines whether or not the value of the variable Bansou on the RAM 203 instructing accompaniment reproduction is 1 (with accompaniment) (step S1009) (step 8 (c)). See S824 to S826).

If the determination in step S1009 is YES, the CPU 201 executes the event Event_2 [AutoIndex_2] relating to the accompaniment related to the track chuck 2 indicated by the AutoIndex_2 value (step S1010). If the event Event_2 [AutoIndex_2] executed here is, for example, a note-on event, an accompaniment musical tone sounding command is issued to the sound source LSI 204 of FIG. 2 according to the key number and velocity specified by the note-on event. publish. On the other hand, if the event Event_2 [AutoIndex_2] is, for example, a note-off event, a mute command for the accompaniment musical tone being sounded to the sound source LSI 204 of FIG. 2 is issued according to the key number and velocity specified by the note-off event. publish.

On the other hand, if the determination in step S1009 is NO, the CPU 201 skips step S1010, so that the event Event_2 [AutoIndex_2] related to this accompaniment is not executed, and the progress is synchronized with the lyrics, so that the next step S1011 Proceed to the process of, and execute only the control process that advances the event.

After step S1010 or when the determination in step S1009 is NO, the CPU 201 increments the AutoIndex_2 value for referencing the performance data set for accompaniment data on track chunk 2 by +1 (step S1011).

Further, the CPU 201 resets the DeltaT_2 value indicating the relative time from the occurrence time of the event executed this time with respect to the track chunk 2 to 0 (step S1012).

Then, the CPU 201 is whether or not the waiting time DeltaTime_2 [AutoIndex_2] of the performance data set on the next executed track chunk 2 indicated by the AutoIndex_2 value is 0, that is, an event executed at the same time as this event. Whether or not it is determined (step S1013).

If the determination in step S1013 is NO, the CPU 201 ends the current automatic performance interrupt process shown in the flowchart of FIG.

If the determination in step S1013 is YES, the CPU 201 returns to step S1009 and repeats the control process relating to the event Event_2 [AutoIndex_2] of the performance data set to be executed next on the track chunk 2 indicated by the AutoIndex_2 value. The CPU 201 repeatedly executes the processes of steps S1009 to S1013 as many times as the number of times it is executed simultaneously this time. The above processing sequence is executed when a plurality of note-on events are sounded at the same timing, such as a chord.

FIG. 11 is a flowchart showing a detailed example of the first embodiment of the song reproduction process of step S705 of FIG. This process executes the control process according to the present embodiment described with reference to FIG.

First, the CPU 201 determines in step S1004 in the automatic performance interrupt process of FIG. 10 whether or not a value is set in the variable SongIndex on the RAM 203 and is no longer a Null value (step S1101). This SongIndex value indicates whether or not the current timing is the reproduction timing of the singing voice.

When the determination in step S1101 becomes YES, that is, when the current timing of song playback (t1, t2, t3, t4, etc. in the example of FIG. 5) is reached, the CPU 201 performs the performer by the keyboard processing in step S703 of FIG. It is determined whether or not a new key press is detected on the keyboard 101 of FIG. 1 according to the above (step S1102).

If the determination in step S1102 is YES, the CPU 201 sets the designated pitch specified by the key press by the performer in a register (not particularly shown) or a variable on the RAM 203 as the vocal pitch (step S1103).

Subsequently, the CPU 201 reads the lyrics character string from the song event Event_1 [SongIndex] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex on the RAM 203. The CPU 201 generates singing voice data 215 for uttering the singing voice output data 217 corresponding to the read lyrics character string at the singing voice pitch set based on the key press set in step S1103. Instruct the voice synthesis LSI 205 to perform vocalization processing (step S1105).

The processing of steps S1103 and S1105 described above corresponds to the control processing described above with respect to the song playback timings t1, t2, or t4 of FIG. 5 (b).

On the other hand, the determination in step S1101 determines that the current time is the timing for playing the song (t1, t2, t3, t4, etc. in the example of FIG. 5), and the determination in step S1102 is NO, that is, the current key press If it is determined that the sound has not been detected, the CPU 201 reads the pitch data from the song event Event_1 [SongIndex] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex on the RAM 203, and this pitch. Is set as the vocal pitch in a register (not shown) or a variable on the RAM 203 (step S1104).

After that, the CPU 201 utters the singing voice output data 217 corresponding to the lyrics character string read from the song event Event_1 [SongIndex] at the utterance pitch set in step S1104 by executing the process of step S1105 described above. The singing voice data 215 for making the voice synthesis LSI 205 is generated, and the voice synthesis LSI 205 is instructed to perform the vocalization process (step S1105).

The processing of steps S1104 and S1105 described above corresponds to the control processing described above with respect to the song reproduction timing t3 of FIG. 5 (b).

After the process of step S1105, the CPU 201 stores the reproduced song position indicated by the variable SongIndex on the RAM 203 in the variable SongIndex_pre on the RAM 203 (step S1106).

Further, the CPU 201 clears the value of the variable SongIndex to the Null value, and sets the timing after that to a state other than the timing of song playback (step S1107). After that, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

When the determination in step S1101 described above is NO, that is, when the current time is not the timing of song playback, the CPU 201 detects a new key pressed by the performer on the keyboard 101 in FIG. 1 by the keyboard processing in step S703 in FIG. It is determined whether or not this is done (step S1108).

If the determination in step S1108 is NO, the CPU 201 as it is ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

If the determination in step S1108 is YES, the CPU 201 determines the lyrics character string of the song event Event_1 [SongIndex_pre] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex_pre on the RAM 203, which is currently being uttered by the speech synthesis LSI 205. The singing voice data 215 instructing to change the pitch of the singing voice output data 217 corresponding to the above to the designated pitch based on the key press of the performer detected in step S1108 is generated and output to the voice synthesis LSI 205 ( Step S1109). At this time, in the singing voice data 215, the phonemes of the latter half of the phonemes of the lyrics that are already being uttered, for example, if the lyrics character string "ki", the phoneme strings "/ k /" and "/ i /" that compose it The frame in which the latter half of "/ i /" starts (see FIGS. 4 (b) and 4 (c)) is set at the start position of the change to the specified pitch.

By the process of step S1109 above, the pitch of the singing voice output data 217 that is uttered from the original timing immediately before the current key press timing, for example, t1, t3, and t4 in FIG. 5 (b), respectively. Is changed to the designated pitch played by the performer, and each utterance can be continued at the current key press timings t1', t3', and t4' in FIG. 5 (b), for example.

After the process of step S1109, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

FIG. 12 is a flowchart showing a detailed example of the second embodiment of the song reproduction process of step S705 of FIG. This process executes another control process according to the present embodiment described with reference to FIG. In FIG. 12, the steps with the same step numbers as in the case of the first embodiment of FIG. 11 shall execute the same processing as in the case of the first embodiment. The part where the control process of the second embodiment of FIG. 12 is different from the control process of the first embodiment of FIG. 11 is that the determination of step S1101 described above in the description of the first embodiment is NO, that is, at the present time. This is the control process of steps S1201 and S1202 when it is not the timing of song playback and the determination in step S1108 is YES, that is, when a new key press by the performer is detected.

In FIG. 12, when the determination in step S1108 is YES, the CPU 201 first sets the designated pitch designated by the key press by the performer as the vocal pitch in a register (not particularly shown) or a variable on the RAM 203 ( Step S1201).

After that, the CPU 201 reads the lyrics character string from the song event Event_1 [SongIndex] on the track chunk 1 of the song data on the RAM 203 indicated by the variable SongIndex on the RAM 203. The CPU 201 generates singing voice data 215 for newly uttering the singing voice output data 217 corresponding to the read lyrics character string at the utterance sound pitch set with the specified pitch based on the key press set in step S1103. Then, the voice synthesis LSI 205 is instructed to perform vocalization processing (step S1202).

After the process of step S1202, the CPU 201 ends the song reproduction process of step S705 of FIG. 7 shown in the flowchart of FIG.

As described above, by the control process of the second embodiment described above, for example, "ki / Twin (first character)" uttered at the original song playback timings t1, t3, and t4 of FIG. 5 (b), respectively. , "Ki / twin (third character)", and singing voice output data 217 corresponding to "ra / kle (fourth character)", followed by new key press timings t1', t3', and t4'. Corresponds to "ki / twin (1st character)", "ki / twin (3rd character)", and "ra / kle (4th character)" at each specified pitch specified by the key press. There is an effect that the singing voice voice output data 217 sounds as if it is uttered separately.

FIG. 13 is a diagram showing a configuration example of song data when the song data illustrated as the data structure of FIG. 6 is implemented in the MusicXML format. With such a data structure, it is possible to have the score data of the lyrics character string (character) and the melody (note). Then, the CPU 201 parses such song data in the display process of step S704 of FIG. 7, for example, on the key 101 of FIG. 1, a key corresponding to the melody corresponding to the lyrics character string currently playing the song. It is possible to have a function to guide the key press of the key corresponding to the lyrics character string by the performer by shining. At the same time, for example, the lyrics character string currently playing the song in the display example as shown in FIG. 14 and the corresponding score can be displayed on the LCD 104 of FIG.

In the embodiment described above, the acoustic model unit 306 is implemented by the DNN (deep neural network) in order to predict the acoustic feature sequence 317 from the language feature sequence 316. In addition, for the above prediction, the acoustic model unit 306 may be implemented by an HMM (Hidden Markov Model: Hidden Markov Model). In this case, the model learning unit 305 in the speech learning unit 301 learns a model in consideration of the context in order to accurately model the acoustic features of the speech. In order to model the acoustic features in detail, not only the phonemes immediately before and after, but also factors such as accent, part of speech, and sentence length are considered. However, since the number of context combinations is enormous, it is difficult to prepare speech data that can accurately learn the context-dependent model for all context combinations. In order to solve this problem, the model learning unit 305 can adopt a technique of context clustering based on a decision tree. In context clustering based on a decision tree, context-dependent models are classified using context-related questions such as "Is the previous phoneme / a /?", And model parameters of similar contexts are used as the learning result 315. It is set in the acoustic model unit 306. Since the context considered depends on the structure of the decision tree, it is possible to estimate a model that depends on the context with high accuracy and high generalization performance by selecting an appropriate decision tree structure. The acoustic model unit 306 in the speech synthesis unit 302 of FIG. 3 connects context-dependent HMMs according to the language feature sequence 316 extracted from the singing voice data 215 by the text analysis unit 307, and the sound with the maximum output probability. The feature series 317 is predicted.

Although the embodiment described above is an embodiment of the present invention for an electronic keyboard instrument, the present invention can also be applied to other electronic musical instruments such as electronic stringed instruments.

In addition, the present invention is not limited to the above-described embodiment, and can be variously modified at the implementation stage without departing from the gist thereof. In addition, the functions executed in the above-described embodiment may be combined as appropriate as possible. The above-described embodiments include various steps, and various inventions can be extracted by an appropriate combination according to a plurality of disclosed constitutional requirements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the effect is obtained, the configuration in which the constituent requirements are deleted can be extracted as an invention.

Regarding the above embodiments, the following additional notes will be further disclosed.
(Appendix 1)
A plurality of controls according to the timing at which the first pitch of the song data having at least the information indicating the first pitch and the lyrics information including the first character corresponding to the first pitch should be specified. When the pitch is specified by operating one of the controls, the singing voice output process that outputs the singing voice corresponding to the first character at the specified pitch according to the operation, and
When the pitch is not specified because none of the plurality of controls is operated according to the timing at which the first pitch should be specified, the singing voice corresponding to the first character is produced. A singing voice output control process that controls output at the first pitch according to the timing at which one pitch should be specified, and
An electronic musical instrument that runs.
(Appendix 2)
The electronic musical instrument according to Appendix 1, which executes the singing voice output processing and the accompaniment data output processing for outputting the accompaniment data in accordance with the singing voice output control processing.
(Appendix 3)
The song data includes information indicating a second pitch to be designated next to the first pitch, and a second character corresponding to the second pitch as the lyrics information.
After the singing voice output process or the singing voice output control process outputs the singing voice corresponding to the first character, the sound is produced by operating any of the controls before the timing for designating the second pitch. The electronic according to Appendix 1 or 2, which executes a pitch change process for changing the pitch of the singing voice corresponding to the first character to the specified pitch according to the operation when the pitch is specified. Musical instrument.
(Appendix 4)
The voice synthesis process, which generates the singing voice by voice synthesis, is executed.
In the singing voice output processing, when the pitch is specified according to the timing at which the pitch should be specified, the singing voice synthesized by the voice synthesis processing is output at the specified pitch according to the specified timing. In addition, when the pitch is not specified according to the timing at which the pitch should be specified, the singing voice synthesized by the voice synthesis process is output at the pitch to be specified according to the timing to be specified. The electronic musical instrument according to any one of 3.
(Appendix 5)
The electronic musical instrument according to Appendix 4, wherein the voice synthesis process synthesizes a singing voice according to a certain singer based on a learned model generated by machine learning using singing voice data and lyrics data sung by a certain singer. ..
(Appendix 6)
The electronic musical instrument according to any one of Supplementary note 1 to 5, which executes a display process of displaying an identifier indicating a pitch to be specified in accordance with a timing at which the pitch should be specified by operating the operator.
(Appendix 7)
On the computer of electronic musical instruments
A plurality of controls according to the timing at which the first pitch of the song data having at least the information indicating the first pitch and the lyrics information including the first character corresponding to the first pitch should be specified. When the pitch is specified by operating one of the controls, the singing voice output process that outputs the singing voice corresponding to the first character at the specified pitch according to the operation, and
When the pitch is not specified because none of the plurality of controls is operated according to the timing at which the first pitch should be specified, the singing voice corresponding to the first character is produced. A singing voice output control process that controls output at the first pitch according to the timing at which one pitch should be specified, and
How to run.
(Appendix 8)
On the computer of electronic musical instruments
A plurality of controls according to the timing at which the first pitch of the song data having at least the information indicating the first pitch and the lyrics information including the first character corresponding to the first pitch should be specified. When the pitch is specified by operating one of the controls, the singing voice output process that outputs the singing voice corresponding to the first character at the specified pitch according to the operation, and
When the pitch is not specified because none of the plurality of controls is operated according to the timing at which the first pitch should be specified, the singing voice corresponding to the first character is produced. A singing voice output control process that controls output at the first pitch according to the timing at which one pitch should be specified, and
A program that executes.

100 Electronic keyboard instrument 101 Keyboard 102 First switch panel 103 Second switch panel 104 LCD
200 Control system 201 CPU
202 ROM
203 RAM
204 Sound source LSI
205 Speech synthesis LSI
206 Key scanner 208 LCD controller 209 System bus 210 Timer 211, 212 D / A converter 213 Mixer 214 Amplifier 301 Speech learning unit 302 Speech synthesis unit 303 Learning text analysis unit 304 Learning acoustic feature extraction 305 Model learning unit 306 Acoustic model Part 307 Text analysis part 308 Speaking model part 309 Sound source generation part 310 Synthetic filter part 311 Learning singing voice data 312 Learning singing voice data 313 Learning language feature series 314 Learning acoustic feature series 315 Learning result 316 Language feature series 317 Acoustic feature series 318 Spectral information 319 Sound source information

Claims (8)

Processing to start playback of song data with pitch data and lyrics data,
During playback of the song data, if the pitch is not specified by the user at the timing corresponding to the vocalization timing of the singing voice corresponding to the new lyrics in the lyrics data, the singing voice corresponding to the new lyrics is performed. When the pitch is output based on the pitch data included in the song data and the pitch is specified by the user at the timing corresponding to the timing of singing the singing voice according to the new lyrics, the new pitch is specified. Singing voice output control processing that controls the singing voice according to the lyrics to be output at the pitch specified by the user ,
An electronic musical instrument that runs. The singing voice output control process changes the pitch of the singing voice being uttered to the pitch specified by the user when the pitch is specified by the user during the utterance of the singing voice corresponding to a certain lyrics. Item 1. The electronic musical instrument according to Item 1. The song data according to claim 1 or 2, wherein the song data has a lyric part whose pitch is changed by being designated by a user, and an accompaniment part that progresses in accordance with the progress of the lyric part. Electronic musical instrument. The electronic musical instrument according to any one of claims 1 to 3 , wherein a display process for displaying an identifier indicating a pitch to be specified by the user is executed before the timing at which the user should specify the pitch . Equipped with a keyboard with multiple keys
The electronic musical instrument according to claim 4, wherein in the display process, the identifier is displayed on the key by illuminating the key according to the pitch to be specified by the user . It has a memory that stores a trained model generated by machine learning the singing voice data of a singer.
The electronic musical instrument according to any one of claims 1 to 5, wherein the singing voice is generated by voice synthesis based on the learned model . On the computer of electronic musical instruments
Processing to start playback of song data with pitch data and lyrics data,
During playback of the song data, if the pitch is not specified by the user at the timing corresponding to the vocalization timing of the singing voice corresponding to the new lyrics in the lyrics data, the singing voice corresponding to the new lyrics is performed. When the pitch is output based on the pitch data included in the song data and the pitch is specified by the user at the timing corresponding to the timing of singing the singing voice according to the new lyrics, the new pitch is specified. Singing voice output control processing that controls the singing voice according to the lyrics to be output at the pitch specified by the user ,
How to run. On the computer of electronic musical instruments
Processing to start playback of song data with pitch data and lyrics data,
During playback of the song data, if the pitch is not specified by the user at the timing corresponding to the vocalization timing of the singing voice corresponding to the new lyrics in the lyrics data, the singing voice corresponding to the new lyrics is performed. When the pitch is output based on the pitch data included in the song data and the pitch is specified by the user at the timing corresponding to the timing of singing the singing voice according to the new lyrics, the new pitch is specified. Singing voice output control processing that controls the singing voice according to the lyrics to be output at the pitch specified by the user ,
A program that executes.