JP5549651B2 - Lyric output data correction device and program - Google Patents

Description

  The present invention relates to a lyrics output data correction device and a program for correcting the output timing of lyrics defined in association with musical score data so as to synchronize with music data in time.

  2. Description of the Related Art Conventionally, a system is known that associates lyrics prepared separately from a music acoustic signal along a time axis with a music acoustic signal of a song including a singing voice and an accompaniment sound (see, for example, Patent Document 1).

  In this type of system, a discriminator (a so-called discriminating model) generated in advance by machine learning by extracting a pre-defined feature quantity from a signal (hereinafter referred to as a vocal signal) corresponding to a vocal voice included in a music acoustic signal. ) To identify each phoneme of the vocal speech. Further, in this type of system, each of the specified phonemes and each character constituting the lyrics are sequentially associated with each other along the time axis.

JP 2008-134606 A

  By the way, in the system described in Patent Document 1, speech recognition processing is used to specify phonemes in vocal speech, and the amount of processing until each phoneme is specified in the recognition phase becomes enormous. In particular, in the system described in Patent Document 1, since the identification model used for speech recognition is generated by machine learning, the amount of processing required for the learning phase becomes enormous.

  That is, the system described in Patent Document 1 has a problem that the total amount of processing required to specify the time in the music acoustic signal to which the characters constituting the lyrics should be associated becomes enormous.

  Therefore, an object of the present invention is to associate characters constituting the lyrics with the time in the music acoustic signal corresponding to the characters by a simpler method.

  In the lyrics output data correction device of the present invention made to achieve the above object, the musical sound transition acquisition means acquires a musical sound transition waveform in which the sound pressure of the musical sound constituting the target music has changed along the time axis, and outputs it. The sound transition acquisition means represents the musical score of the musical piece simulating the target musical piece, and for each output sound output from the sound source module, the sound of the output sound is based on musical score data in which at least the pitch and the performance start timing are defined. An output sound transition waveform in which the pressure has shifted along the time axis is acquired.

  The lyrics output data acquisition means is data defining the lyrics output timing, which is the output timing of the lyrics constituent characters constituting the lyrics of the target song, and the lyrics output timing for at least one of the lyrics constituent characters is the score data The lyrics output data associated with the specific start timing which is at least one timing defined for the above is acquired.

  Further, the time deviation amount derivation means is extracted from the musical sound information indicating the characteristics of the musical sound transition waveform extracted from the musical sound transition waveform acquired by the musical sound transition acquisition means and the output sound transition waveform acquired by the output sound transition acquisition means. Based on the result of comparison with the output sound information representing the characteristics of the output sound transition waveform, the amount of time deviation indicating the amount of deviation of the performance start timing of the musical sound corresponding to each output sound with respect to the performance start timing of each output sound Is derived.

Then, the timing correction means corrects the lyrics output timing in the lyrics output data acquired by the lyrics output data acquisition means according to the time deviation amount derived by the time deviation amount deriving means so as to coincide with the musical performance start timing. Specify the timing of the revised lyrics output.
In the time shift amount deriving means of the present invention, the tone change deriving means extracts a tone non-harmonic that is a non-harmonic component of the tone transition waveform from the tone transition waveform, and the tone non-harmonic along the time axis. The musical sound change representing the change is derived as musical sound information, and the output sound change deriving means extracts the output sound non-harmonic, which is a non-harmonic component of the output sound transition waveform, from the output sound transition waveform, An output sound change representing a change in the output sound inharmonic along the line is derived as output sound information.
Then, the time correlation deriving means sets the time correlation value indicating the correlation value between the musical sound change and the output sound change to the reference position defined on the musical sound change time axis on the time base of the output sound change. The time derived by the time correlation deriving means is derived every time the output sound change is expanded or contracted along the time axis by matching the set position, and the set position is sequentially changed along the time axis within the specified range. Among the correlation values, the time correction amount deriving means derives the expansion / contraction rate and set position of the output sound change corresponding to the time correlation value having the maximum value as the time correction amount (time shift amount).

  According to such a lyrics output data correction device, the lyrics output timing in the lyrics output data can be corrected to match the performance start timing of the musical sound constituting the target music along the time axis. .

  According to the lyrics output data correction device of the present invention, it is not necessary to execute any speech recognition processing when correcting the lyrics output timing (that is, defining the corrected lyrics output timing). Therefore, according to the lyrics output data correction device of the present invention, the total processing required to specify the musical performance start timing (that is, the time in the musical sound transition waveform (musical sound signal)) to which the lyrics constituent characters should be associated. The amount can be reduced as compared with the apparatus described in Patent Document 1.

  In other words, according to the lyrics output data correction apparatus of the present invention, the music composition corresponding to the lyrics constituent characters is played back in a simpler manner (i.e., the musical sound transition waveform (music acoustic signal)). Time).

  Furthermore, a system is assumed in which the lyrics output data corrected by the lyrics output data correction device of the present invention is delivered to an apparatus not equipped with a sound module along with lyrics telop data representing individual lyrics constituent characters and music acoustic data. To do. At this time, if the device outputs the lyrics constituent characters in time synchronization with the musical sound in the music sound data in conjunction with the reproduction of the music sound data, the device can also enjoy karaoke.

  Note that the musical tone transition waveform referred to here includes, for example, a sampled (sampled) analog waveform in which the sound pressures of all musical tones constituting the target music have shifted along the time axis. In addition, the output sound transition waveform here includes an audio signal generated by rendering data representing a musical piece simulating the target musical piece in the MIDI format.

Furthermore, the lyric structure character referred to here, it may be a respective characters constituting the lyrics, but it may also be a phrase or phrases and a group according to specific rules each of the character.

In general, inharmonic components included in musical tone transitions and output sound transitions are often instrumental sounds of musical instruments (for example, drums and basses) that rhythm.
The instrumental sound of a musical instrument that engraves this rhythm can be detected more reliably than other musical instrument sounds. For this reason, the amount of time deviation derived by the lyrics output data correction device of the present invention can more reliably match the performance start timing of each output sound in the score data with the performance start timing of the musical sound. It will be a thing.

Therefore, according to the lyrics output data correction device of the present invention, the corrected lyrics output timing can be reliably matched with the musical performance start timing.
Furthermore, in the lyrics output data correction device according to the present invention, the pitch correction amount deriving means converts the pitch of the output sound to the output sound based on the result of comparing one piece of musical sound information with one piece of output sound information. The pitch correction amount is derived so as to match the pitch of the corresponding musical sound, and the score data correction means derives the pitch of each output sound specified in the score data by the pitch correction amount derivation means. Modified score data in which the score data is corrected may be generated by shifting according to the high correction amount.

In this case, the time shift amount deriving means, correct the score modified sound transition waveform which is the output sound transition waveform based on the data, yet good as the output sound transition waveform obtained in output sound transition acquisition unit.

  According to such a lyrics output data correction device, since the output sound transition waveform acquired by the output sound transition acquisition means becomes the correction sound transition waveform, the pitch deviation between the musical sound transition waveform is suppressed to a minimum, The accuracy of deriving the amount of time deviation can be improved. As a result, according to the lyrics output data correction device of the present invention, the corrected lyrics output timing can be surely matched with the tone output timing.

In the pitch correction amount deriving unit in the lyrics output data correcting device of the present invention, the tone distribution deriving unit represents the frequency included in the tone transition waveform and the strength of each frequency, and normalized the strength of the frequency. The tone pitch distribution is derived as one piece of tone information, and the output tone distribution deriving means represents the frequency included in the output tone transition waveform and the strength of each frequency, and the output tone normalized with respect to the strength of the frequency. The pitch distribution is derived as one of the output sound information, and the pitch correlation deriving means determines the pitch correlation value representing the correlation value between the output pitch distribution and the musical tone pitch distribution as the musical pitch distribution. The output pitch distribution may be derived from the specified position every time the output pitch distribution is shifted along the frequency axis. In this case, the pitch correction amount deriving unit is arranged along the frequency axis from the specified position corresponding to the pitch correlation value having the maximum value among the pitch correlation values derived by the pitch correlation deriving unit. the shift amount, be derived as pitch correction amount not good.

  If the musical score data is corrected according to the pitch correction amount derived in this way, the ratio of the frequency and the strength of each frequency included in the corrected output sound transition waveform is changed to the frequency and each frequency included in the musical sound transition waveform. The strength ratio can be more closely approximated.

  In particular, the musical tone pitch distribution and the output pitch distribution derived by the lyrics output data correction device of the present invention are the frequency included in the tone transition waveform and the output tone transition waveform, and the strength of the frequency. Has been normalized. Therefore, according to the lyrics output data correction device of the present invention, even if the amplitude of the tone transition waveform and the amplitude of the output tone transition waveform are greatly different from each other, the output transition waveform based on the corrected score data is converted into the tone transition waveform. Can be approached.

  In the present invention, the lyrics output data may have the lyrics output timing defined by the elapsed time from the specific start timing for at least some of the lyrics constituent characters.

In this case, in the lyrics output data correcting device of the present invention, associating means, at least for the lyrics output timing of the lyrics structure characters lyric output timing defined by the elapsed time, but it may also define a modified lyrics output timing .

  According to such a lyrics output data correction device, the corrected lyrics output timing can be defined even for the lyrics output data in which the lyrics output timing is defined by the elapsed time from one lyrics constituent character.

  The definition of the lyrics output timing of the lyrics constituent characters whose lyrics output timing is specified by the elapsed time is that the lyrics output timing of each lyrics constituent character is associated with the performance start timing of the output sound corresponding to the lyrics constituent character. In addition, it may be performed without performing the association. As a specific method of this association, a section where the tempo is constant in the target music is specified, and the performance start timing of the output sound included in the score data is included in the lyrics output data for the same section where the tempo is constant It may be associated with the lyrics output timing of the lyrics constituent characters. Further, for example, when the musical score data is generated in advance as data of the MIDI (Music Instrument Digital Interface) standard, it is conceivable to newly add a MIDI track representing the lyrics output timing in the lyrics output data. That is, in the newly added track, the lyrics output timing of each of the lyrics constituent characters may be expressed in association with the performance start timing of the output sound corresponding to each lyrics constituent character.

  By the way, in the present invention, the musical score data has the performance start timing of at least some of the output sounds defined as the specific start timing, and the lyrics output data has the lyrics composition timing at the lyrics output timing of each of the lyrics constituent characters. The performance start timing of the output sound corresponding to the character may be associated.

In this case, the timing correcting means in the present invention, for each of the lyric structure characters, but it may also define a modified lyrics output timing.
According to such a lyric output data correction device, it is possible to generate lyric output data in which the lyric output timing of each lyric constituent character is associated with the performance start timing of the output sound corresponding to the lyric constituent character.

Furthermore, in the present invention, the performance start timing correction means derives a modified performance start timing obtained by shifting the performance start timing of the output sound by a time shift amount, and the timing correction means corrects the performance start timing correction means derived by the performance start timing correction means. the performance start timing, is also not good as a modified lyrics output timing.

According to such a lyrics output data correction device, the corrected lyrics output timing can be replaced with the corrected performance start timing.
Further, in the present invention, the performance start timing correction means derives a corrected performance start timing obtained by shifting the performance start timing of the output sound by a time shift amount, and the timing correction means performs the correction performance start timing and the performance of the output sound. the difference between the start timing, by shifting the lyrics output timing, but it may also be prescribed a modified lyrics output timing.

  According to such a lyrics output data correction device, the corrected lyrics output timing can be defined by shifting the difference between the corrected performance start timing and the performance start timing of the output sound, and the lyrics output timing.

The present invention may be a program for causing a computer to function as a lyrics output data correction device.
When the present invention is implemented as a program, the program acquires a musical sound transition waveform in which the sound pressure of the musical sound constituting the target music has changed along the time axis in the musical sound transition acquisition procedure, and obtains an output sound transition acquisition procedure. Represents the musical score of the musical piece simulating the target musical piece, and for each output sound output from the sound source module, the sound pressure of the output sound is time based on the musical score data in which at least the pitch and the performance start timing are defined. The output sound transition waveform that has shifted along the axis is acquired. Furthermore, in the lyrics output data acquisition procedure, the data is for defining the lyrics output timing, which is the output timing of the lyrics constituent characters constituting the lyrics of the target music, and the lyrics output timing for at least one of the lyrics constituent characters is a score. The lyrics output data associated with the specific start timing which is at least one timing defined for the data is acquired.

Then, in the time deviation amount derivation procedure, the musical sound information representing the characteristics of the musical sound transition waveform extracted from the musical sound transition waveform and the output sound information representing the characteristics of the output sound transition waveform extracted from the output sound transition waveform are compared. Based on the result, a time deviation representing the deviation of the performance start timing of the musical sound corresponding to each output sound with respect to the performance start timing of each output sound is derived, and the musical performance start timing is determined by the timing correction procedure. as match, the lyrics output timing of the lyrics output data acquired by the lyric output data acquisition step, define a modified lyrics output timing corrected according to the time shift amount derived in the time shift amount derivation procedure.
The time deviation amount derivation procedure is based on extracting the tone non-harmonic that is a non-harmonic component of the tone transition waveform from the tone transition waveform acquired in the tone transition acquisition procedure, and changing the tone non-harmonic along the time axis. From the tone change derivation procedure for deriving the tone change representing the tone information as the tone information and the output tone transition waveform acquired in the output tone transition acquisition procedure, the output tone non-harmonic that is the non-harmonic component of the output tone transition waveform is obtained. The output sound change derivation procedure for extracting the output sound change representing the change of the output sound non-harmonic along the time axis as output sound information, the musical sound change derived by the musical sound change derivation procedure, and the output sound The time correlation value representing the correlation value with the output sound change derived in the change derivation procedure is set to the reference position specified on the time axis of the musical sound change and the set position set on the time axis of the output sound change. Derived every time the output sound change is expanded and contracted along the time axis In addition, the time correlation derivation procedure for sequentially changing the set position along the time axis within the specified range and the time correlation value derived from the time correlation derivation procedure corresponding to the maximum time correlation value The computer executes a time correction amount derivation procedure for deriving the expansion / contraction rate of the output sound change and the set position as a time correction amount, and the time correction amount derived in the time correction amount derivation procedure is set as a time shift amount.

  If the program of the present invention is made in this way, for example, it can be recorded on a computer-readable recording medium such as a DVD-ROM, CD-ROM, hard disk, etc. If necessary, it can be used by being acquired and activated by a computer via a communication line. And by making a computer perform each procedure, the computer can be functioned as a lyrics output data correction apparatus described in Claim 1.

It is a block diagram which shows schematic structure of the music data delivery system comprised centering on the information processing apparatus to which this invention was applied. It is a flowchart which shows the process sequence of the data correction process in 1st embodiment. It is a flowchart which shows the process sequence of a pitch correction process. It is explanatory drawing explaining the outline | summary of a pitch correction process. It is a flowchart which shows the process sequence of a time gap | deviation amount derivation | leading-out process. It is explanatory drawing explaining the outline | summary of a time gap | deviation amount derivation | leading-out process. It is drawing which shows the outline | summary of the lyric output data in 2nd embodiment. It is a flowchart which shows the process sequence of the data correction process in 2nd embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
[First embodiment]
<About music data distribution system>
Here, FIG. 1 is a block diagram showing a schematic configuration of a music data distribution system including a lyrics output data correction device to which the present invention is applied.

The music data distribution system 1 includes a music data storage server 3 that stores music data MD including lyrics output data DO, and an information processing device 20 that performs processing on the music data MD stored in the music data storage server 3. And at least one mobile terminal 5A to 5n (n is a natural number of 1 or more representing the number of mobile terminals) to which the music data MD processed by the information processing apparatus 20 is distributed.
<About the music data storage server>
The music data storage server 3 is a device that functions as a database that stores music data MD1 to MDm (m is a natural number of 1 or more representing the number of music data). The music data MD in the present embodiment includes music acoustic data DW, music MIDI data DM, and lyrics data DL.

  Among these, the music acoustic data DW is a sample of an analog waveform (that is, a music transition waveform) in which the sound pressures of all the musical sounds constituting one music (hereinafter referred to as a specific music) change along the time axis ( Sampled data, for example, audio files in WAV or MP3 format prepared in advance for each song.

  The music MIDI data DM is data representing the music score of a music simulating a specific music according to the well-known MIDI (Musical Instrument Digital Interface) standard (that is, corresponding to the music data of the present invention), and is prepared in advance for each music piece. Has been. Each of the music MIDI data DM includes identification data that is data for distinguishing music, a score track that represents a score for each instrument used in the music, and a section in which the music is divided (for example, A melody or rust). ) At least tempo data representing the tempo.

  Of these, at least the pitch (so-called note number) and the period during which the sound module outputs the output sound (hereinafter referred to as the note length) are defined for each output sound output from the MIDI sound source in the score track. ing. However, the note length of the musical score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the output sound is started, and the output of the output sound is ended. It is defined by the performance end timing (so-called note-off timing) that represents the time from the start of performance of the music.

  Note that the score track includes, for example, keyboard instruments (eg, piano and pipe organ), stringed instruments (eg, violin, viola, guitar, koto), percussion instruments (eg, drums, cymbals, timpani, xylophone, etc.), and wind instruments. (For example, clarinet, trumpet, flute, shakuhachi, etc.)

  The lyric data DL is data relating to lyrics displayed on a display device that constitutes a well-known karaoke device, and includes lyrics telop data DT representing characters (hereinafter, lyric constituent characters) constituting the lyrics of the specific music, and lyrics constituent characters. Lyric output data DO, in which a timing correspondence relationship for associating the lyric output timing with the performance of the music MIDI data DM is defined.

  Specifically, in the timing correspondence relationship in the present embodiment, the timing for starting the output of the lyrics telop data DT is associated with the timing for starting the performance of the music MIDI data DM (an example of the specific start timing in the present invention). Above, the lyrics output timing of each lyrics constituent character along the time axis of the target song is defined by the elapsed time from the start of the performance of the song MIDI data DM. The elapsed time referred to here is, for example, a time representing the timing of executing color change of the displayed lyrics constituent characters, and is defined by the color change speed. Further, the lyric constituent characters here may be each of the characters constituting the lyric, or may be a phrase or a phrase in which each of the characters is grouped according to a specific rule along the time axis.

  As the timing correspondence in the present embodiment, the timing of ending the output of each lyric constituent character specified from the color change speed (hereinafter referred to as the lyric output end timing) is from the start of the performance of the music MIDI data DM. It may be defined by the elapsed time.

The music acoustic data DW, the music MIDI data DM, and the lyrics data DL are stored in the music data storage server 3 in association with each corresponding music.
<About mobile devices>
Among these, the mobile terminal 5 is a terminal (for example, a well-known mobile phone) that can reproduce the music acoustic data DW acquired from the information processing apparatus 20, and includes an information receiving unit 6, a display unit 7, and a sound output unit 8. A communication unit 9, a storage unit 10, and a control unit 11.

  Among these, the information reception part 6 receives the information input via the input device (not shown). The display unit 7 displays an image including at least information indicated by a character code based on a command from the control unit 11. The sound output unit 8 reproduces and outputs at least the music sound data DW, and includes, for example, a PCM sound source and a speaker.

The communication unit 9 is for the mobile terminal 5 to perform information communication with the outside via a communication network (for example, a public wireless communication network or a network line). The storage unit 10 stores various processing programs and various data. The control unit 11 controls the units 6, 7, 8, 9, and 10 that constitute the mobile terminal 5 according to a processing program stored in the storage unit 10 and the like.
<Information processing device>
Next, the information processing apparatus 20 will be described.

  The information processing apparatus 20 includes a communication unit 21, an input receiving unit 22, a display unit 23, a voice input unit 24, a voice output unit 25, a sound source module 26, a storage unit 27, and a control unit 30. I have.

  Among these, the communication unit 21 performs communication between the information processing apparatus 20 and the outside via a communication network (for example, a public wireless communication network or a network line). The input receiving unit 22 is an input device (for example, a keyboard or a pointing device) that receives input of information and commands in accordance with an external operation. The display unit 23 is a display device (for example, a liquid crystal display or a CRT) that displays an image including at least information indicated by a character code. The voice input unit 24 is a device (so-called microphone) that converts voice into an electrical signal and inputs the electrical signal to the control unit 30. The sound output unit 25 is a device (so-called speaker) that converts an electrical signal from the control unit 30 into sound and outputs the sound.

  Furthermore, the sound module 26 is a device (for example, a MIDI sound source) that outputs a sound (that is, an output sound) that simulates a sound from a sound source based on the music MIDI data DM. The storage unit 27 is a non-volatile storage device (for example, a hard disk device) configured to be able to read and write stored contents.

  Further, the control unit 30 is stored in the ROM 31 that stores processing programs and data that need to retain stored contents even when the power is turned off, the RAM 32 that temporarily stores processing programs and data, and the ROM 31 and RAM 32. It is mainly composed of a known computer having at least a CPU 33 that executes each process (various operations) according to the processing program.

In the ROM 31, the CPU 33 performs a data correction process for correcting the lyrics output timing in the lyrics output data DO corresponding to the target music so as to coincide with the musical performance start timing in the music acoustic data DW corresponding to the target music. The processing program for executing is stored. That is, by executing the data correction process, the information processing apparatus 20 functions as the lyrics output data correction apparatus of the present invention.
<About data correction processing>
Next, data correction processing executed by the CPU 33 will be described.

Here, FIG. 2 is a flowchart showing a processing procedure of data correction processing in the present embodiment.
The data correction process is started when an activation command for starting the data correction process is input via the input receiving unit 22.

  As shown in FIG. 2, when the data correction process is started, the music MIDI data DM corresponding to the music (hereinafter referred to as the target music) specified by the information input via the input receiving unit 22 is used. Is acquired from the music data storage server 3 (S110 (S means step)).

  Subsequently, music acoustic data DW corresponding to the target music is acquired from the music data storage server 3 (S130). A musical sound transition waveform in the music acoustic data DW is acquired from the acquired music acoustic data DW (S140).

  Then, based on the music MIDI data DM acquired in S110 and the musical sound transition waveform acquired in S140, the musical composition is set so that the pitch of the output sound matches the pitch of the musical sound constituting the target musical composition. A pitch correction process for correcting the MIDI data DM is executed (S150). Hereinafter, the music MIDI data DM in which the output sound has been corrected is referred to as corrected music MIDI data DM.

In addition, the pitch correction process is performed between the performance start timing of the output sound (hereinafter referred to as the corrected output sound) modified so that the pitch of the output sound matches the pitch of the musical sound, and the performance start timing of the musical sound. A time for deriving a deviation amount (hereinafter referred to as a time deviation amount) and correcting the corrected music MIDI data DM so that the performance start timing of each output sound matches the performance start timing of the musical sound constituting the target music. A deviation amount derivation process is executed (S170).
<Pitch correction processing details>
Here, the pitch correction process started in S150 of the data correction process will be described.

  When the pitch correction process is started, as shown in FIG. 3, all output sounds are set along the time axis based on all score tracks included in the music MIDI data DM acquired in the previous S110. An output sound transition waveform that is a waveform that has been shifted is acquired (S310). Specifically, the acquisition of the output sound transition waveform in the present embodiment is executed by well-known rendering that generates an audio signal (waveform) from MIDI standard data.

  Subsequently, the obtained output sound transition waveform is subjected to frequency analysis (in this embodiment, discrete Fourier transform) for each unit time set along the time axis, and is included in the output sound transition waveform of the unit time. A power spectrum representing the frequency and the intensity at each frequency is derived (S320). Based on the derived power spectrum, an average output sound spectrum obtained by arithmetically averaging the intensity at each frequency for each frequency along the time axis is derived (S330). The intensity at the frequency of the derived average output sound spectrum is averaged for each frequency range (for example, a semitone unit, hereinafter, a specified pitch range) so that the boundaries are adjacent to each other to obtain a representative value (S340). ). Furthermore, a normalized output sound spectrum (see FIG. 4A) is derived by normalizing the intensity at the frequency in the average output sound spectrum averaged in S340 so that the variance is “1” and the average is “0”. (S350).

  Subsequently, the musical sound transition waveform acquired in S140 is subjected to frequency analysis for each unit time set along the time axis, and a power spectrum in the unit time is derived (S360). Based on the derived power spectrum, an average musical sound spectrum obtained by arithmetically averaging the intensity at each frequency for each frequency along the time axis is derived (S370). The intensity at the frequency of the derived average tone spectrum is averaged for each specified pitch range to obtain a representative value (S380), and the intensity at the frequency of the average tone spectrum averaged at S380 is expressed as variance “1”, average “ A normalized musical tone spectrum (see FIG. 4B) normalized to be “0” is derived (S390).

  Note that the representative value obtained in S340 and S380 of the present embodiment may be the intensity at a frequency corresponding to the center value in the specified pitch range. In this case, specifically, for each 20 Cent (every fifth of a semitone), a process of extracting a frequency value (power) closest to the 20 Cent grid is performed.

  In detail, as described later, a correlation value between the normalized output sound spectrum and the normalized musical sound spectrum (hereinafter referred to as a pitch correlation value) is derived (S400). Then, it is determined whether or not the shift amount of the normalized output sound spectrum with respect to the normalized musical sound spectrum is equal to or greater than a predetermined upper limit value (S410). As a result of the determination, if the shift amount is less than the upper limit value (S410: NO), the normalized output sound spectrum is shifted by a predetermined amount along the frequency axis (S420), and the process returns to S400. The pitch correlation value is derived again.

  That is, in S400 to S420 of this embodiment, as shown in FIG. 4C, the normalized output sound spectrum is shifted from the lower limit value to the upper limit value along the frequency axis with respect to the normalized musical sound spectrum. However, every time the normalized output sound spectrum is shifted, a pitch correlation value is derived.

  When the shift amount of the normalized output sound becomes equal to or greater than the upper limit value (S410: YES), a correction amount (hereinafter referred to as pitch correction) for matching the pitch of the output sound to the pitch of the musical sound constituting the target music. Is determined (S430). In S430 of the present embodiment, specifically, among the pitch correlation values derived in the previous S400, the shift amount of the normalized output sound spectrum corresponding to the pitch correlation value having the maximum value is calculated. Derived as a pitch correction amount.

  Subsequently, according to the derived pitch correction amount, the corrected music MIDI data is generated by correcting the pitches of the individual output sounds defined for all the score tracks in the music MIDI data DM (S440). That is, the corrected music MIDI data generated in S440 of the present embodiment is obtained by shifting the pitch of the output sound from the pitch of the output sound prepared in advance by a pitch correction amount.

After that, the pitch correction process is terminated and the process returns to the data correction process.
<About the processing content of the time deviation derivation process>
Next, the time lag amount derivation process activated in S170 of the data correction process will be described.

  When this time shift amount derivation process is started, as shown in FIG. 5, all the corrected output sounds are timed based on all the score tracks included in the corrected music MIDI data generated in the previous S440. A modified sound transition waveform that is a waveform that has shifted along the axis is acquired (S510). The acquisition of the modified sound transition waveform in the present embodiment may be executed by the same method as in S310.

  Subsequently, an output sound non-harmonic that is a non-harmonic component of the acquired modified sound transition waveform is derived from the modified sound transition waveform (S520), and further, the non-harmonic of the musical sound transition waveform acquired in the previous S140 is obtained. A musical tone non-harmonic wave component is derived from the musical tone transition waveform (S530). The derivation of these non-harmonic components may be performed by passing the corrected sound transition waveform or the musical sound transition waveform through a filter prepared in advance.

  Furthermore, the output sound non-harmonic and the musical sound non-harmonic are each divided into specific blocks each having a time length defined along the time axis (S540). The specific block to be divided is a constant tempo section indicating that the tempo of the target music is constant. This fixed tempo section is determined by specifying the time at which the tempo is changed according to the tempo specified in the tempo data of the music MIDI data DM as the start time and end time of each fixed tempo section. Regarding the specific block of musical tone non-harmonic, after the specific block of output non-harmonic is determined, the start time and the end time of each specific block of output sound non-harmonic are determined from the start of the performance of the target music. The time is determined by specifying the start time and the end time of each specific block of the musical tone non-harmonic.

  Then, a set of specific blocks is selected from the specific blocks divided in S540 (S550), and both the musical tone non-harmonic and output sound non-harmonic are set along the time axis for the set of specific blocks. Unit data representing the change is generated (S560). As shown in FIGS. 6 (A) and 6 (B), the unit data in the present embodiment includes the amplitude value of the non-harmonic component in the specified section for each specified section that is shorter than the specific block. After the addition, it is generated by normalizing the value added for each specified section. In the following, the unit data for output sound non-harmonic is referred to as output sound unit data (corresponding to the output sound change in the present invention), and the unit data for musical sound non-harmonic is referred to as the musical unit data (musical sound change in the present invention) Equivalent).

  Matching the output sound setting position specified on the time axis of the output sound unit data with the music sound setting position specified on the time axis of the music sound unit data, the correlation value between the output sound unit data and the sound unit data (Hereinafter referred to as a time correlation value) is derived (S570). Then, it is determined whether or not the expansion / contraction rate of the output sound unit data with respect to the musical sound unit data is equal to or higher than a predetermined upper limit value (expansion rate upper limit value) (S580). As a result of the determination, if the expansion / contraction rate of the musical sound unit data is less than the upper limit value of the expansion / contraction rate (S580: NO), the output sound unit data is expanded by a predetermined rate along the time axis (S590). ), The process returns to S570.

  Further, if the expansion / contraction rate of the musical sound unit data has reached the upper limit value of the expansion / contraction rate (S580: YES), the shift amount along the time axis of the output sound unit data with respect to the musical sound unit data is set to a predetermined upper limit value. It is determined whether or not it is equal to or greater than (the upper limit value of the shift amount) (S600). As a result of the determination, if the shift amount of the musical sound unit data is less than the upper limit value of the shift amount (S600: NO), the set position of the output sound unit data is shifted by a predetermined time (S610) and output. After setting the expansion / contraction rate of the sound unit data as the lower limit value, the process returns to S570.

  That is, in S570 to S610 of this embodiment, as shown in FIG. 6 (C), the time correlation value is set each time the musical sound unit data is expanded until the expansion / contraction rate of the output sound unit data reaches the upper limit value. To derive. Then, the derivation of the time correlation value is executed while shifting the output sound unit data along the time axis until the upper limit value of the shift amount is reached with respect to the musical sound unit data.

  On the other hand, if the result of determination in S600 is that the shift amount of the output sound unit data is greater than or equal to the upper limit value of the shift amount (S600: YES), the performance of the modified output sound is performed at the performance start timing of the musical sound constituting the target music. A correction amount for matching the start timing, that is, a time shift amount is derived (S620). In S620 of this embodiment, specifically, the expansion / contraction of the output sound unit data corresponding to the time correlation value having the maximum value among all the time correlation values derived in S570 for a set of specific blocks. The rate and the shift amount are derived as a time shift amount with respect to the specific block selected in S550.

  According to the derived amount of time deviation, modified score data in which the performance start timing of each output sound is modified is generated (S630). In S630 of the present embodiment, the pitch of the output sound is corrected based on the shift amount of the output sound unit data and the expansion / contraction rate of the output sound unit data derived as the amount of time shift with respect to the specific block selected in S550. The start time and end time of the specific block in the corrected musical score data are corrected. Then, the performance start timing of the output sound is expanded or contracted according to the period specified by the start time and the end time after the correction so that the interval ratio of the performance start timing of the output sound before the correction is maintained. Then, modified score data in which the performance start timing of each output sound for the specific block is modified is generated. In S630 of this embodiment, the performance end timing of the output sound is also corrected. The method for correcting the performance end timing of the output sound may be the same method as the performance start timing of the output sound.

  Subsequently, it is determined whether or not the amount of time deviation is derived for all the specific blocks divided in S540 (S640). As a result of the determination, the amount of time deviation is derived for all the specific blocks. If not (S640: NO), the process returns to S550. In S550, a new specific block is selected, and the steps up to S620 are executed. In S550, specific blocks are acquired in order from the longest time length, and the amount of time shift is derived. However, in a specific block adjacent to a specific block for which the amount of time deviation has already been derived, the corrected start time or end time of the specific block that has already been derived is derived as a value in the own specific block.

  On the other hand, if the result of determination in S640 is that the amount of time deviation has been derived for all the specific blocks (S640: YES), then this time correction process is terminated and the process returns to the data correction process.

  When the process proceeds to S190 in the data correction process (see FIG. 2), the lyrics output timing of each lyrics constituent character of the lyrics output track defined in S120 is acquired (S190). Subsequently, according to the time deviation amount derived in the time deviation amount derivation process, the lyrics output timing of each of the lyrics constituent characters acquired in S190 is corrected so as to coincide with the musical performance start timing in the music acoustic data DW. (S200).

  Specifically, the method of correcting the lyrics output timing in S200 of the present embodiment may be the same method as the correction of the performance start timing and performance end timing of the output sound in S630.

  Then, lyrics output data (that is, corrected lyrics output data) DO that defines the lyrics output timing (that is, corrected lyrics output timing) and the lyrics output end timing that are corrected in S200 are generated (S210).

Thereafter, the data correction process is terminated.
[Effect of the first embodiment]
As described above, according to such data correction processing, the lyrics output timing in the lyrics output data DO is made to coincide with the performance start timing of the musical sound constituting the target music along the time axis. It can be corrected.

  Moreover, in the data correction process of the present embodiment, when the lyrics output timing is corrected (that is, the corrected lyrics output timing is defined), it is not necessary to execute any speech recognition process, so the lyrics output timing of the lyrics constituent characters The total amount of processing required to specify the musical performance start timing should be reduced as compared with the apparatus described in Patent Document 1.

  In other words, according to the lyrics output data correction device of the present invention, the lyrics output timing of the lyrics constituent characters can be associated with the musical performance start timing corresponding to the lyrics constituent characters by a simpler method.

  In particular, in the data correction process according to the present embodiment, the pitch correction process is executed before the time deviation amount derivation process, and the correction has the pitch of the output sound corrected to match the pitch of the musical sound. After the music MIDI data DM is generated, an output sound transition waveform is acquired from the corrected music MIDI data DM. As a result, according to the data correction process, the pitch shift between the output sound transition waveform and the musical sound transition waveform is suppressed to the minimum, and the accuracy of deriving the time shift amount can be improved.

By the way, the corrected lyrics output data DO generated in the data correction processing of the present embodiment is data for making the lyrics output timing for each lyric constituent character coincide with the musical performance start timing in the music sound data DW. . Therefore, in the music data distribution system 1 of the present embodiment, the lyrics telop data DT and the modified lyrics output data DO are distributed to the portable terminal 5 that does not have the sound source module, together with the music acoustic data DW, and the portable If the terminal 5 outputs the lyrics constituent characters in time synchronization with the musical sound in the music acoustic data in conjunction with the reproduction of the music acoustic data DW, the mobile terminal 5 can also enjoy karaoke.
[Second Embodiment]
Next, a second embodiment of the present invention will be described.

The music data distribution system in the second embodiment is different from the music data distribution system 1 in the first embodiment only in the structure of the lyrics output data DO and the processing content of the data correction processing executed by the information processing apparatus 20. . For this reason, in 2nd embodiment, the same code | symbol is attached | subjected to the structure similar to the music data delivery system 1 of 1st embodiment, description is abbreviate | omitted, the structure of the lyrics output data DO, and the information processing apparatus 20 The contents of the data correction process executed by will be mainly described.
<About lyrics output data DO>
In the lyrics output data DO of this embodiment, the timing correspondence relationship for associating the lyrics output timing of the lyrics constituent characters with the performance of the music MIDI data DM is defined in the same way as the lyrics output data DO of the first embodiment.

Specifically, in the timing correspondence relationship in the present embodiment, as shown in FIG. 7, the lyrics output timing for each of the lyrics constituent characters is associated with the performance start timing of the output sound corresponding to the lyrics constituent characters. . Furthermore, in the timing correspondence relationship in the present embodiment, as shown in FIG. 7, the lyrics output end timing for each of the lyrics constituent characters is associated with the performance end timing of the output sound corresponding to the lyrics constituent characters. .
<About data correction processing>
Next, data correction processing in the present embodiment will be described.

Here, FIG. 8 is a flowchart showing a processing procedure of data correction processing in the present embodiment.
The data correction process is started when an activation command for starting the data correction process is input via the input receiving unit 22.

As shown in FIG. 8, when the data correction process is started, the music MIDI data DM corresponding to the target music is acquired from the music data storage server 3 (S710).
Subsequently, music acoustic data DW corresponding to the target music is acquired from the music data storage server 3 (S730). A musical sound transition waveform in the music acoustic data DW is acquired from the acquired music acoustic data DW (S740).

Then, a pitch correction process is executed (S750). Since the pitch correction process is the same as the pitch correction process (S150) in the first embodiment, a detailed description thereof will be omitted.
Further, a time deviation amount derivation process is executed (S770). Since this time deviation amount derivation process is the same as the time deviation amount derivation process (S170) in the first embodiment, a detailed description thereof will be omitted.

  Subsequently, a difference between the performance start timing of the output sound in the music MIDI data DM corrected in S770 and the performance start timing of the output sound in the music MIDI data DM before correction acquired in S710 is derived (S790). . In S790 of the present embodiment, specifically, a difference (hereinafter, an onset difference time dOnset, an offset difference time dOffset) is derived for each output sound based on the following equation (1).

  In Equation (1), aOnset is the performance start timing of the output sound in the modified music MIDI data DM, and bOnset is the performance start timing of the output sound in the music MIDI data DM before correction. In addition, aOffset in the equation (1) is the performance end timing of the output sound in the modified music MIDI data DM, and bOffset is the performance end timing of the output sound in the music MIDI data DM before correction.

That is, in S790, an onset difference time dOnset and an offset difference time dOffset are derived for each output sound.
Subsequently, the lyrics output data DO for the target music is acquired from the music data storage server 3, and the lyrics output timing of each lyrics constituent character in the lyrics output data DO is acquired (S800).

  Further, according to the onset difference time dOnset and the offset difference time dOffset derived in S790, each of the lyrics constituent characters in the lyrics output data DO acquired in S800 so as to coincide with the musical performance start timing in the music acoustic data DW. The lyrics output timing is corrected (S810).

  Specifically, in S810 of the present embodiment, the corrected lyrics output timing mOnset and the corrected lyrics output end timing mOffset are derived for each of the lyrics constituent characters based on (2) below.

  In Equation (2), lOnset is the lyrics output timing of the lyrics constituent characters in the lyrics output data DO, and lOffset is the lyrics output end timing of the lyrics constituent characters in the lyrics output data DO.

  That is, in S810, the onset difference time dOnset and the offset difference time dOffset are shifted so that the lyrics output timing and the lyrics output end timing of each of the lyrics constituent characters coincide with the performance start timing of each musical sound in the music acoustic data DW. Will be fixed.

  Then, lyrics output data (that is, corrected lyrics output data) in which the lyrics output timing (that is, corrected lyrics output timing) and the lyrics output end timing corrected in S810 are defined is generated (S820).

Thereafter, the data correction process is terminated.
[Effects of Second Embodiment]
Also in the data correction process of this embodiment, the same effect as the data correction process of 1st embodiment can be acquired.

In particular, when correcting the lyric output data DO configured as in the present embodiment, it is not necessary to associate the lyric output timing of each lyric constituent character with the performance start timing of the output sound. Can be generated.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, in S810 of the data correction process of the second embodiment, the lyrics output timing and the lyrics output end timing of each of the lyrics constituent characters are corrected by shifting the onset difference time dOnset and the offset difference time dOffset. The correction method of the lyrics output timing and the lyrics output end timing is not limited to this, and the lyrics output data correction method in the data correction processing of the first embodiment may be applied.

  In the data correction process of the first embodiment, the lyrics output data DO corresponding to the target music is acquired from the music data storage server 3, and the performance start timing of the output sound defined in the music MIDI data DM is obtained. After associating the lyrics output timing of the lyrics constituent characters defined in the lyrics output data DO, the lyrics output timing of the lyrics constituent characters associated with the performance start timing of the output sound may be corrected. At this time, the lyrics output end timing may also be associated with the performance end timing of the output sound estimated to correspond to each lyrics constituent character.

  As this association method, the music MIDI data DM in which the lyrics output timing of each lyrics constituent character specified from the speed of color change is set as the performance start timing of the output sound estimated to correspond to each lyrics constituent character. A new track (hereinafter referred to as a lyrics output track) is generated.

  Note that the method of correcting the lyrics output timing (or lyrics output end timing) of the lyrics constituent characters in the data correction processing of the first embodiment is the same as that of the lyrics constituent characters included in the same tempo intervals. The lyrics output timing (or lyrics output end timing) may be shifted by the onset difference time dOnset or the offset difference time dOffset.

In the data correction process of the above embodiment (here, both the first embodiment and the second embodiment), both the pitch correction process and the time deviation amount derivation process are executed. The process executed in the data correction process may be only the time deviation amount derivation process.
[Correspondence between Embodiment and Claims]
Finally, the relationship between the description of the above embodiment and the description of the scope of claims will be described.

  S140 and S740 in the data correction process of the above embodiment correspond to the musical sound transition acquisition means in the description of the claims, and S510 of the time shift amount derivation process corresponds to the output sound transition acquisition means. S190 and S800 correspond to the lyrics output data acquisition means. Further, S520 to S620 of the time shift amount derivation process correspond to time shift amount derivation means, and S200 and S810 in the data correction process correspond to timing correction means.

  Further, S520, S540 to S560 in the time deviation amount derivation process correspond to output sound change derivation means, S530 to S560 correspond to musical sound change derivation means, and S570 to S610 correspond to time correlation derivation means. S620 corresponds to time correction amount deriving means. Further, S320 to S430 in the pitch correction process correspond to a pitch correction amount deriving unit, and S440 corresponds to a score data correcting unit. Of these, S360 to S390 correspond to the musical sound distribution deriving means, S320 to S350 correspond to the output sound distribution deriving means, and S400 to S420 correspond to the pitch correlation deriving means.

  Further, S630 in the time deviation amount derivation process corresponds to performance start timing correction means.

  DESCRIPTION OF SYMBOLS 1 ... Music data delivery system 3 ... Music data storage server 5 ... Portable terminal 6 ... Information reception part 7 ... Display part 8 ... Sound output part 9 ... Communication part 10 ... Memory | storage part 11 ... Control part 20 ... Information processing apparatus 21 ... Communication Unit 22: Input receiving unit 23 ... Display unit 24 ... Audio input unit 25 ... Audio output unit 26 ... Sound source module 27 ... Storage unit 30 ... Control unit 31 ... ROM 32 ... RAM 33 ... CPU

Claims (8)

A musical sound transition acquisition means for acquiring a musical sound transition waveform in which the sound pressure of the musical sound constituting the target music has changed along the time axis;
The musical score of the musical composition that simulates the target musical piece, and for each output sound output from the sound source module, the sound pressure of the output sound is time axis based on the musical score data in which the pitch and the performance start timing are defined at least. Output sound transition acquisition means for acquiring an output sound transition waveform that has shifted along
It is data that defines the lyrics output timing, which is the output timing of the lyrics constituent characters constituting the lyrics of the target music, and the lyrics output timing for at least one of the lyrics constituent characters is specified for the score data Lyrics output data acquisition means for acquiring lyrics output data associated with a specific start timing that is at least one timing;
The musical sound information representing the characteristics of the musical sound transition waveform extracted from the musical sound transition waveform acquired by the musical sound transition acquisition means, and the output sound transition waveform extracted from the output sound transition waveform acquired by the output sound transition acquisition means. Based on the result of comparison with the output sound information representing the characteristics, a time shift for deriving a time shift amount representing a shift amount of the performance start timing of the musical sound corresponding to each output sound with respect to the performance start timing of each of the output sounds. A quantity derivation means;
Modified lyrics output in which the lyrics output timing in the lyrics output data acquired by the lyrics output data acquisition means is corrected according to the time deviation amount derived by the time deviation amount deriving means so as to coincide with the performance start timing of the musical sound for example Bei and timing correction means for defining the timing,
The time shift amount derivation means includes:
From the musical tone transition waveform acquired by the musical tone transition acquisition means, a musical tone non-harmonic that is a non-harmonic component of the musical tone transition waveform is extracted, and a musical tone change that represents a musical non-harmonic change along the time axis is obtained. A musical sound change deriving means derived as the musical sound information;
From the output sound transition waveform acquired by the output sound transition acquisition means, the output sound non-harmonic that is a non-harmonic component of the output sound transition waveform is extracted, and the change of the output sound non-harmonic along the time axis is extracted. Output sound change deriving means for deriving an output sound change representing as the output sound information;
A time correlation value representing a correlation value between the musical sound change derived by the musical sound change deriving means and the output sound change derived by the output sound change deriving means is defined on the time axis of the musical sound change. Each time the output sound change is expanded or contracted along the time axis by matching the set position set on the time axis of the output sound change with a reference position, the set position is derived within the specified range on the time axis. A time correlation deriving means that sequentially changes along
Of the time correlation values derived by the time correlation deriving means, the time correction amount for deriving the expansion / contraction rate of the output sound change and the set position corresponding to the time correlation value having the maximum value as the time correction amount Derivation means and
With
The lyrics output data correction apparatus characterized in that the time correction amount derived by the time correction amount deriving means is set as the time shift amount . Based on the result of comparing one of the musical sound information and one of the output sound information, the pitch correction amount is adjusted so that the pitch of the output sound matches the pitch of the musical sound corresponding to the output sound. A pitch correction amount deriving means for deriving;
A musical score that generates corrected musical score data in which the musical score data is corrected by shifting the pitch of each of the output sounds defined in the musical score data according to the pitch correction amount derived by the pitch correction amount deriving means. Data correction means;
With
The time shift amount derivation means includes:
The corrected sound transition waveform, which is the output sound transition waveform based on the modified score data generated by the score data correction means, is the output sound transition waveform acquired by the output sound transition acquisition means. Item 2. The lyrics output data correction device according to Item 1 . The pitch correction amount derivation means includes:
A musical sound distribution deriving unit that represents the frequency included in the musical sound transition waveform and the strength of each frequency, and derives a musical sound pitch distribution normalized with respect to the strength of the frequency as one of the musical sound information;
An output sound distribution deriving unit that represents the frequency and the strength of each frequency included in the output sound transition waveform, and derives an output pitch distribution normalized with respect to the strength of the frequency as one of the output sound information;
A pitch correlation value representing a correlation value between the output pitch distribution derived by the output sound distribution deriving means and the musical tone pitch distribution derived by the musical sound distribution deriving means is determined in advance of the musical tone pitch distribution. A pitch correlation deriving means for deriving each time the output pitch distribution is shifted along the frequency axis from a defined position,
Of the pitch correlation values derived by the pitch correlation deriving means, the shift amount along the frequency axis from the specified position corresponding to the pitch correlation value having the maximum value is the pitch correction amount. The lyrics output data correction device according to claim 2 , wherein the lyrics output data correction device is derived as follows. The lyrics output data is
For at least some of the lyrics constituent characters, the lyrics output timing is defined by the elapsed time from the specific start timing,
The timing correction means includes
For the lyrics output timing of the lyric structure characters the lyric output timing defined by the elapsed time, according to any one of claims 1 to 3, characterized in that defining the modified lyrics output timing Lyrics output data correction device. The performance start timing of at least some of the output sounds is defined as the specific start timing,
The lyrics output data is
The specific start timing corresponding to the lyrics constituent characters is associated with the lyrics output timing of each of the lyrics constituent characters,
The timing correction means includes
The lyrics output data correction device according to any one of claims 1 to 3 , wherein the corrected lyrics output timing is defined for each of the lyrics constituent characters. A performance start timing correction means for deriving a modified performance start timing obtained by shifting the performance start timing of the output sound by the time offset amount,
The timing correction means includes
6. The lyrics output data correction device according to claim 5 , wherein the corrected performance start timing derived by the performance start timing correction means is the corrected lyrics output timing. Performance start timing correction means for deriving a modified performance start timing obtained by shifting the performance start timing of the output sound by the time offset amount;
The timing correction means includes
The modified lyrics output timing is defined by shifting the difference between the corrected performance start timing derived by the performance start timing correcting means and the performance start timing of the output sound, and the lyrics output timing. The lyrics output data correction device according to claim 4 or 5 . A musical sound transition acquisition procedure for acquiring a musical sound transition waveform in which the sound pressure of the musical sound constituting the target music has changed along the time axis,
The musical score of the musical composition that simulates the target musical piece, and for each output sound output from the sound source module, the sound pressure of the output sound is time axis based on the musical score data in which the pitch and the performance start timing are defined at least. Output sound transition acquisition procedure for acquiring the output sound transition waveform that has shifted along
It is data that defines the lyrics output timing, which is the output timing of the lyrics constituent characters constituting the lyrics of the target music, and the lyrics output timing for at least one of the lyrics constituent characters is specified for the score data A lyrics output data acquisition procedure for acquiring lyrics output data associated with a specific start timing that is at least one timing;
The musical sound information representing the characteristics of the musical sound transition waveform extracted from the musical sound transition waveform acquired in the musical sound transition acquisition procedure, and the output sound transition waveform extracted from the output musical transition waveform acquired in the output sound transition acquisition procedure. Based on the result of comparison with the output sound information representing the characteristics, a time shift for deriving a time shift amount representing a shift amount of the performance start timing of the musical sound corresponding to each output sound with respect to the performance start timing of each of the output sounds. A quantity derivation procedure;
Modified lyrics output in which the lyrics output timing in the lyrics output data acquired in the lyrics output data acquisition procedure is corrected according to the amount of time deviation derived in the time deviation amount derivation procedure so as to coincide with the performance start timing of the musical sound Let the computer execute the timing correction procedure that defines the timing ,
The time deviation amount derivation procedure is as follows:
From the musical tone transition waveform acquired in the musical tone transition acquisition procedure, a musical tone non-harmonic that is a non-harmonic component of the musical tone transition waveform is extracted, and a musical tone change that represents a musical non-harmonic change along the time axis is obtained. A musical sound change derivation procedure derived as the musical sound information;
The output sound non-harmonic, which is a non-harmonic component of the output sound transition waveform, is extracted from the output sound transition waveform acquired in the output sound transition acquisition procedure, and the change of the output sound non-harmonic along the time axis is extracted. An output sound change derivation procedure for deriving an output sound change representing the output sound information;
A time correlation value representing a correlation value between the musical sound change derived in the musical sound change derivation procedure and the output sound change derived in the output sound change derivation procedure is defined on the time axis of the musical sound change. Each time the output sound change is expanded or contracted along the time axis by matching the set position set on the time axis of the output sound change with a reference position, the set position is derived within the specified range on the time axis. A time correlation derivation procedure that sequentially changes along
Of the time correlation values derived in the time correlation deriving procedure, the time correction amount for deriving the expansion / contraction rate of the output sound change and the set position corresponding to the time correlation value having the maximum value as the time correction amount Derivation procedure and
Let the computer run
A program characterized in that the time correction amount derived in the time correction amount derivation procedure is set as the time deviation amount .