JP5782972B2 - Information processing system, program - Google Patents

Description

  The present invention relates to an information processing system and program for performing information processing on music data.

2. Description of the Related Art Conventionally, an information distribution system including a music playback device that plays music and an information processing device that performs data communication between the music playback devices is known (see, for example, Patent Document 1).
In the information distribution system described in Patent Document 1, an information processing apparatus acquires music data, which is a plurality of types of data groups related to music, and transmits the music data to a music playback device. Yes. “Music data” as used herein refers to the musical score of a musical piece, along with performance speed information indicating the progression of the musical piece, and a musical piece in which at least the pitch and performance start timing are specified for each output sound output from the sound module. Musical score data (so-called MIDI data), lyrics telop data representing the lyrics constituting the lyrics of the music, and data defining the lyrics output timing which is the output timing of the lyrics constituting characters, and at least one lyrics constituting character The lyric output timing with respect to the lyric output data associated with at least one timing in the music score data is associated with each tune.

  By the way, portable devices that have become popular in recent years play and play music waveform data (so-called audio file format data such as MP3 and WAV) that represents a musical sound waveform in which the sound pressure of the musical sound constituting the music has changed along the time axis. By doing so, it is configured to be able to function as a music playback device. As a method of using such a portable device, there is a demand not only to listen to music played back but also to enjoy karaoke when playing back the music.

JP 2008-026674 A

  However, in order to realize karaoke by effectively using music waveform data in a mobile device, it is necessary to acquire lyrics data corresponding to the music in the mobile device. Here, the lyric data that needs to be acquired by the portable device is data relating to the lyrics of the music, and includes lyric telop data and lyric output data.

  Generally, the lyric output data is generated so that the lyric output timing of each lyric constituent character corresponds to the performance of the tune based on the music score data. It is not generated to match.

  In particular, music score data is generally generated by a person who listens to a performance based on music waveform data specifying and defining the pitch of each output sound and the performance start timing. Therefore, in the music score data, the pitch of the output sound and the performance start timing do not coincide with the pitch and performance timing of each musical sound in the music waveform data, and there is a deviation.

  For this reason, when the portable device outputs (displays) the lyrics constituent characters based on the lyrics output data together with the playback performance of the music based on the music waveform data, the output timing of the lyrics constituent characters is set to the individual musical tone output timing. There arises a problem that it does not coincide with the performance timing.

  Accordingly, an object of the present invention is to reduce a deviation in the output of lyrics constituent characters with respect to a musical performance timing based on music waveform data in a portable device.

  The present invention made to achieve the above object is a portable device in which music waveform data representing a music transition waveform in which the sound pressure of a musical sound constituting a music has shifted along the time axis is stored in a storage unit, and The present invention relates to an information processing system including an information processing apparatus that performs data communication with a portable device.

  The information processing apparatus in the information processing system of the present invention is an apparatus that acquires music score data, lyrics data, and lyrics output data, and executes information processing on at least the lyrics output data.

  This musical score data represents the musical score of the music, and is data in which at least the pitch and the performance start timing are specified for each output sound output from the sound source module together with performance speed information indicating the progress of the music. The lyric data is data representing lyric constituent characters constituting the lyrics of the music. The lyric output data is data defining the lyric output timing which is the output timing of the lyric constituent characters, and the lyric output timing for at least one lyric constituent character is associated with at least one timing in the music score data. It is data.

  And in the portable device in the information processing system of the present invention, the identification information acquisition means acquires the music identification information for identifying the target music that is the designated music, and the identification information transmission means obtains the acquired music identification information. Transmit to the information processing device. The speed information transmission means acquires music speed information representing the progress of the target music from the music waveform data of the target music corresponding to the music identification information acquired by the identification information acquisition means, and transmits it to the information processing apparatus.

  Further, in the portable device in the information processing system of the present invention, the lyric information acquisition means is the lyric data of the target music, and the corrected output data that is the lyric output data processed by the information processing device, and the target music The corrected output data is acquired from the information processing apparatus, and the performance output means plays the target music based on the music waveform data of the target music, and the lyrics defined in the corrected output data acquired by the lyrics information acquisition means The lyrics constituent characters represented by the lyrics data are sequentially displayed according to the output timing.

  On the other hand, in the information processing apparatus in the information processing system of the present invention, the data acquisition unit acquires the music score data and the lyrics output data of the target music corresponding to the music identification information transmitted by the identification information transmission unit, and the acquisition Performance speed acquisition means acquires performance speed information from the music score data. Then, based on the result of comparing the music speed information transmitted by the speed information transmitting means and the performance speed information acquired by the performance speed acquiring means, the time lag amount deriving means is configured for each performance start timing of the output sound. Then, a time shift amount representing a shift amount of the start timing of the musical sound corresponding to each output sound is derived.

  Furthermore, in the information processing apparatus in the information processing system according to the present invention, the data correction unit converts the lyric output timing in the lyric output data acquired by the data acquisition unit according to the amount of time deviation derived by the time deviation amount derivation unit. When the lyric output data corrected to coincide with the start timing is generated as corrected output data, the lyric transmitting means sends the generated corrected output data and the lyric data of the target music corresponding to the music identification information to the portable device. Send.

  In the corrected output data generated by such an information processing system, the lyrics output timing coincides with the tone output start timing in the music waveform data. Therefore, when the target music is played on the portable device based on the music waveform data, the lyrics constituent characters can be sequentially displayed in accordance with the output timing of the musical sound, and the display of each lyrics constituent character is the music waveform. It is possible to reduce the occurrence of deviation with respect to the musical performance timing based on the data.

  In particular, according to the information processing apparatus of the present invention, the data transmitted from the information processing apparatus to the portable device can be lyric data and lyric output data, so the amount of communication from the information processing apparatus to the portable device is small. can do.

  The “music score data, lyrics data, and lyrics output data” acquired by the information processing apparatus may be stored in an information storage unit provided in the information processing apparatus, or separately from the information processing apparatus. You may store in the provided apparatus. Further, the “lyric constituent characters” referred to here may be each of characters constituting the lyrics, or may be a phrase or a phrase in which each of the characters is grouped according to a specific rule.

Further, “progress of music” here includes information representing the tempo of the music and a bass waveform representing the timing at which the bass instrument is played.
By the way, in the present invention, the speed information transmission means of the portable device specifies the tempo information indicating the tempo of the target music and the base waveform information indicating the timing when the specific musical instrument is played, which is specified based on the music waveform data of the target music. May be acquired as music speed information.

In this case, the music score data defines the music tempo information, which is information indicating the tempo of the music corresponding to the music score data, and the output music for each type of instrument played in the corresponding music. It is preferable that the pitch and the performance start timing are defined, and further, the performance speed acquisition means of the information processing apparatus has the music tempo information and the performance start timing of the output sound of a specific instrument changed along the time axis. the base instrument waveform, have preferred to be acquired as performance speed information.

  In the information processing system of the present invention, based on the music waveform data on the mobile terminal side, the tempo of the target music is acquired as music speed information, and the base waveform information using the timing at which the base waveform appears is used as music speed information. get.

Therefore, the time lag amount deriving means can calculate the tempo shift in the target music and the timing shift of the musical instrument included in the target music.
Various methods are conceivable as methods for deriving the amount of time shift using such music speed information and performance speed information.

  As an example, the tempo ratio between tempo information and music tempo information is derived, and the music tempo information is multiplied by the derived tempo ratio to obtain the bass instrument waveform after correcting the overall tempo of the music score data. To do.

  Then, every time the acquired bass instrument waveform is shifted along the time axis with respect to the base waveform information, the correlation value between the base instrument waveform and the base waveform information is derived and the correlation value is maximized. It is conceivable to derive the amount as a time shift amount.

  As another example, the first peak along the time axis of the bass instrument waveform obtained after correcting the tempo of the entire musical score data matches the first peak along the time axis of the bass waveform information. It is also conceivable to derive the shift amount along the time axis of the bass instrument waveform to be generated as the time shift amount.

  And according to the information processing system of the present invention, in order to generate the corrected output data using such a time deviation amount, the deviation of the output timing of the lyrics constituent characters with respect to the musical performance timing based on the music waveform data, It can be reduced more reliably.

  However, the “specific instrument” referred to here may include a bass guitar, a contrabass, or the like, and may also include a percussion instrument (such as a drum or a cymbal). The base waveform information may be acquired by passing the music waveform data through a filter that passes a signal in the frequency band of the sound of a specific instrument.

By the way, in the present invention, the base waveform information is a sampling waveform. In this case, the base waveform information portable device speed information transmitting means transmits, it is not preferable in which sampled at a sampling frequency lower than the sampling frequency of the tone transition waveform.

According to such an information processing system, the amount of music speed information transmitted from the portable device to the information processing apparatus can be suppressed.
In the information processing system of the present invention, in the portable device, the spectrum transmission means acquires the music waveform data of the target music, and derives the musical sound spectrum distribution that is the frequency spectrum of the musical sound transition waveform represented by the music waveform data. Then, it may be transmitted to the information processing apparatus.

  In this case, in the information processing apparatus, the output sound spectrum deriving means acquires the music score data of the target music, and based on the music score data, the output sound transition waveform is a waveform in which the output sound has shifted along the time axis. The output sound spectrum distribution which is the frequency spectrum of is derived. Then, the pitch correlation deriving means calculates a pitch correlation value representing a correlation value between the musical sound spectrum distribution transmitted by the spectrum transmitting means and the output sound spectrum distribution derived by the output sound spectrum deriving means, as a musical sound spectrum distribution. Is derived each time the output sound spectrum distribution is shifted along the frequency axis from the prescribed position prescribed in advance. Then, the score data correcting means calculates the shift amount along the frequency axis from the specified position corresponding to the pitch correlation value having the maximum value among the pitch correlation values derived by the pitch correlation deriving means. It is preferable to generate new music score data by shifting the pitch of each output sound defined in the music score data of the target music as the pitch correction amount.

The performance speed acquisition means preferably acquires performance speed information from the new music score data generated by the score data correction means.
According to such an information processing system, it is possible to derive performance speed information after correcting the music score data so as to match the pitch of the musical sound. As a result, according to the information processing system of the present invention, the lyrics output timing defined in the corrected output data can be more reliably matched with the tone output start timing in the music waveform data.

  Note that the present invention may be a program executed by a computer provided in a portable device. However, the portable device acquires music score data, lyric data, and lyric output data, performs data communication with an information processing apparatus that executes information processing on at least lyric output data, Data is stored in the storage unit.

The program of the present invention includes an identification information acquisition procedure for acquiring music identification information for identifying a target music that is a designated music, an identification information transmission procedure for transmitting the acquired music identification information to an information processing device, From the music waveform data of the target music corresponding to the music identification information, the music speed information indicating the progress of the target music is acquired and transmitted to the information processing apparatus, the speed information transmission procedure, the corrected output data, and the music identification information The lyric data of the corresponding target music is defined in the lyric information acquisition procedure acquired from the information processing device, and the target music is played based on the music waveform data of the target tune, and the corrected output data acquired in the lyric information acquisition procedure The computer to execute a performance output procedure for sequentially displaying the lyrics constituting characters represented by the lyrics data in accordance with the lyrics output timing A.
The correction output data acquired in the lyrics information acquisition procedure of the present invention is a data acquisition procedure for acquiring music score data of the target music corresponding to the music identification information transmitted in the identification information transmission procedure, and lyrics output data; The performance speed acquisition procedure for acquiring the performance speed information from the music score data acquired in the data acquisition procedure, the performance speed information acquired in the performance speed acquisition procedure, and the music speed information transmitted in the speed information transmission procedure And a time shift amount derivation procedure for deriving a time shift amount representing a shift amount of the start timing of the musical sound corresponding to each output sound with respect to the performance start timing of each of the output sounds, and the time shift In accordance with the amount of time deviation derived in the amount derivation procedure, the lyric output timing in the acquired lyric output data is expressed as the start timing of the musical sound. The result of the data correction procedure is performed as the processing for generating the lyric output data modified to match the ring as the corrected output data.
Further, in the program of the present invention, in the speed information transmission procedure, the tempo information indicating the tempo of the target music specified based on the music waveform data of the target music, and the base indicating the timing when the specific musical instrument is played. Waveform information is acquired as the music speed information. The music score data defines music tempo information, which is information representing the tempo of the music corresponding to the music score data, and the output sound for each type of musical instrument played in the corresponding music. The pitch and performance start timing are specified.
Then, the performance speed information acquired by the performance speed acquiring procedure, the music tempo information, and Ru-based instrument waveform der the performance start timing remained along the time axis of the output sound of a particular instrument.

  However, the corrected output data acquired in the lyric information acquisition procedure is a result of information processing performed by the information processing apparatus, and acquires music score data and lyrics output data of the target music corresponding to the music identification information. The musical sound corresponding to each output sound with respect to each performance start timing of the output sound based on the result of comparison between the performance speed information acquired from the acquired music score data and the music speed information transmitted in the speed information transmission procedure Is a lyric output data obtained by deriving a time deviation amount representing a deviation amount of the start timing of the voice and correcting the lyrics output timing in the lyrics output data so as to coincide with the start timing of the musical sound according to the derived time deviation amount .

  If the program of the present invention is made in this way, it can be recorded on a computer-readable recording medium (for example, a memory card) and loaded into the computer as necessary to start up or communicate as necessary. It can be used by being acquired and activated by a computer via a line. Then, by causing the computer of the mobile device to execute each procedure, the computer of the mobile device can function as a mobile device (music playback device) that constitutes the information processing system according to claim 1.

1 is a block diagram showing a schematic configuration of an information processing system to which the present invention is applied. It is a flowchart which shows the process sequence of the process performed with an information processing system. It is a figure which shows the estimation method of music BPM. It is a flowchart which shows the process sequence of the pitch correction | amendment process performed in an information processing server. It is a figure which shows the process outline | summary of a pitch correction process. It is a flowchart which shows the process sequence of the timing correction process performed in an information processing server. It is a figure which shows the outline | summary of a timing correction process.

Embodiments of the present invention will be described below with reference to the drawings.
<Information processing system>
Here, FIG. 1 is a block diagram showing a schematic configuration of an information processing system to which the present invention is applied.

  The information processing system 1 of this embodiment is an information distribution system that distributes data that has been processed according to a processing program to a portable terminal 5 that functions as a portable audio player.

  In order to realize this, the information processing system 1 can reproduce at least one portable terminal 5A to 5n (n is one or more representing the number of portable terminals) that can reproduce the music acoustic data WD stored in the storage unit 10. A natural number) and an information processing server 20 that executes various processes on the music data MD including the lyrics data group DL related to the lyrics of the music. In the information processing system 1, the portable terminal 5 and the information processing server 20 are connected to each other so as to be capable of data communication.

That is, in the information processing system 1 of the present embodiment, the stored music sound data WD is reproduced on the portable terminal 5 in which the music sound data WD is stored, and based on the data distributed from the information processing server 20. This is a system for outputting lyrics along with the reproduction of the music acoustic data WD.
<About mobile devices>
Here, the mobile terminal will be described.

  As shown in FIG. 1, the portable terminal 5 includes an information receiving unit 6, a display unit 7, a sound output unit 8, a communication unit 9, a storage unit 10, and a control unit 11, and music acoustic data WD. (For example, a well-known portable audio player or a well-known mobile phone that functions as an audio player).

  Among these, the information reception part 6 receives the information input via the input device (not shown). The display unit 7 displays an image including at least information indicated by a character code based on a command from the control unit 11. The sound output unit 8 is a known device that reproduces music acoustic data WD and outputs sound, and includes, for example, a PCM sound source and a speaker.

  The communication unit 9 is for the mobile terminal 5 to perform information communication with the outside via a communication network (for example, a public wireless communication network or a network line). The storage unit 10 is a non-volatile storage device (for example, a hard disk device or a flash memory) configured to be able to read and write stored contents, and stores various processing programs and various data.

  The music acoustic data WD stored in the storage unit 10 is an analog waveform (that is, a music transition waveform) in which the sound pressures of all the musical sounds constituting one music (hereinafter referred to as a specific music) change along the time axis. ) Sampled (sampled) music waveform data DW and music tag data DD which is information (for example, an identification number) for identifying a specific music.

  That is, one piece of music acoustic data WD is data in which song tag data DD and song waveform data DW are associated with each specific song, for example, an audio file in WAV or MP3 format. In the present embodiment, at least one piece of music acoustic data WD is stored in the storage unit 10, and the symbol “m” in FIG. 1 indicates the number of music waveform data stored in the storage unit 10 (1 The above natural number).

  The control unit 11 is stored in the ROM 12 that stores processing programs and data that need to retain stored contents even when the power is turned off, the RAM 13 that temporarily stores processing programs and data, and the ROM 12 and RAM 13. It is mainly composed of a known computer having at least a CPU 14 that executes each process (various operations) according to the processing program.

The ROM 12 reproduces the music waveform data DW corresponding to the music specified through the information receiving unit 6 (hereinafter referred to as the target music), and the lyrics data corresponding to the target music from the information processing server 20. Stored is a program for the control unit 11 to execute a karaoke playback process of acquiring the group DL and outputting the lyrics along with the playback of the music waveform data DW.
<Information processing server>
Next, the information processing server 20 will be described.

  The information processing server 20 includes a communication unit 21, an input reception unit 22, a display unit 23, a sound input unit 24, a sound output unit 25, a sound source module 26, a storage unit 27, and a control unit 30. I have.

  Among these, in the communication unit 21, the information processing server 20 communicates with the outside via a communication network (for example, a public wireless communication network or a network line). The input receiving unit 22 is an input device (for example, a keyboard or a pointing device) that receives input of information and commands in accordance with an external operation.

  The display unit 23 is a display device (for example, a liquid crystal display or a CRT) that displays an image including at least information indicated by a character code. The sound input unit 24 is a device (so-called microphone) that converts sound into an electric signal and inputs the electric signal to the control unit 30. The sound output unit 25 is a device (so-called speaker) that converts an electrical signal from the control unit 30 into sound and outputs the sound. Furthermore, the sound source module 26 is a device (for example, a MIDI sound source) that outputs a sound (that is, an output sound) that simulates a sound from a sound source, based on data defined by the MIDI (Musical Instrument Digital Interface) standard. .

The storage unit 27 is a non-volatile storage device (for example, a hard disk device or a flash memory) configured to be able to read and write stored contents. The storage unit 27 stores music data MD1 to MDl (l is a natural number of 1 or more representing the number of music data).
<About music data>
The music data MD in the present embodiment includes music MIDI data DM (corresponding to music score data in the claims) and a lyrics data group DL.

  Of these, the music MIDI data DM is data representing the musical score of a music simulating a specific music according to the well-known MIDI (Musical Instrument Digital Interface) standard, and is prepared in advance for each music. Each of the music MIDI data DM includes identification data that is data for distinguishing music, a music track representing a music score for each musical instrument used in the music, and a section in which the music is divided (for example, A melody or rust). ) At least tempo data (corresponding to music tempo information in the claims) representing tempo (so-called BPM (Beats Per Minute)).

  The musical score track defines at least a pitch (so-called note number) and a period during which the sound module outputs the output sound (hereinafter, note length) for each output sound output from the MIDI sound source. Yes. However, the note length of the musical score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the output sound is started, and the output of the output sound is ended. It is defined by the performance end timing (so-called note-off timing) that represents the time from the start of performance of the music.

  Note that the musical score track is, for example, a keyboard instrument (eg, piano or pipe organ), a stringed instrument (eg, violin, viola, guitar, bass guitar, koto), or a percussion instrument (eg, drum, cymbal, timpani, xylophone, etc.). , And wind instruments (for example, clarinet, trumpet, flute, shakuhachi, etc.).

  On the other hand, the lyric data group DL is data relating to lyrics displayed on a display device constituting a well-known karaoke device, and lyrics telop data DT representing characters constituting the lyrics of a specific musical piece (hereinafter referred to as lyric constituent characters). And lyrics output data DO in which a timing correspondence relationship for associating the lyrics output timing, which is the output timing of the lyrics constituent characters, with the performance of the music MIDI data DM is provided.

  Specifically, the timing correspondence relationship in the present embodiment is based on the timing of starting the output of the lyrics telop data DT to the timing of starting the performance of the music MIDI data DM, and along the time axis of the specific music. The lyrics output timing of each lyrics constituent character is defined by the elapsed time from the start of the performance of the music MIDI data DM. The elapsed time referred to here is, for example, a time representing the timing of executing color change of the displayed lyrics constituent characters, and is defined by the color change speed. Further, the lyric constituent characters here may be each of the characters constituting the lyric, or may be a phrase or a phrase in which each of the characters is grouped according to a specific rule along the time axis.

In the present embodiment, the music MIDI data DM and the lyrics data group DL are stored in the storage unit 27 in association with each corresponding specific music.
Further, the control unit 30 is stored in the ROM 31 that stores processing programs and data that need to retain stored contents even when the power is turned off, the RAM 32 that temporarily stores processing programs and data, and the ROM 31 and RAM 32. It is mainly composed of a known computer having at least a CPU 33 that executes each process (various operations) according to the processing program.

The ROM 31 generates corrected output data DO in which the lyrics output timing of the lyrics output data DO corresponding to the target music is corrected to coincide with the musical performance start timing in the music waveform data DW corresponding to the target music. A processing program for the control unit 30 to execute the data correction process to be output is stored.
<Overview of Karaoke playback processing and data correction processing>
Next, karaoke reproduction processing executed by the control unit 11 of the portable terminal 5 and data correction processing executed by the control unit 30 of the information processing server 20 will be described.

Here, FIG. 2 is a flowchart showing the processing procedure of the karaoke reproduction processing and the processing procedure of the data correction processing.
This karaoke reproduction process is activated when an activation command is input via the information receiving unit 6.

  As shown in FIG. 2, in the karaoke playback process executed by the control unit 11 of the mobile terminal 5, music tag data DD corresponding to the target music is acquired from the storage unit 10 of the mobile terminal 5 and stored in the information processing server 20. Transmit (SS10). Furthermore, in the karaoke playback process, the result of executing various processes on the music waveform data DW corresponding to the target music acquired from the storage unit 10 of the mobile terminal 5 (hereinafter referred to as terminal transmission information) is the information processing server 20. (SS20, SS30, SS40).

  On the other hand, in the data correction process executed by the control unit 30 of the information processing server 20, when the music tag data DD from the mobile terminal 5 is acquired, the music data of the specific music corresponding to the music tag data DD and the mobile terminal 5 The modified output data DO is generated based on the terminal transmission information. Further, in the data correction process, the generated corrected output data DO and the lyrics telop data DT of the target music are transmitted to the portable terminal 5 (SS50).

In the portable terminal 5 that has received the corrected output data DO and the lyrics telop data DT of the target music from the information processing server 20, the music waveform data DW corresponding to the target music is played back and the music waveform data in the karaoke playback process. The lyrics composing characters are output along with the playback of DW.
<About karaoke playback processing>
Next, the karaoke reproduction process executed by the control unit 11 of the mobile terminal 5 will be described in detail.

This karaoke reproduction process is activated when an activation command is input via the information receiving unit 6.
When the karaoke playback process is started, first, the music tag data DD corresponding to the music (that is, the target music) specified by the information input via the information receiving unit 6 is stored in the portable terminal 5. Obtained from the unit 10 (S110 (S means step)). Subsequently, the music tag data DD acquired in S110 is transmitted to the information processing server 20 (SS10).

Further, music waveform data DW corresponding to the target music is acquired (S120), and a music spectrum is derived from the acquired music waveform data DW (S130).
Specifically, in S130 of the present embodiment, a waveform segment obtained by cutting out the musical sound transition waveform represented by the music waveform data DW along the time axis for each unit time is analyzed by frequency analysis (for example, FFT (Fast Fourier transform)). Discrete Fourier transform). The unit time is a time length defined in advance as a so-called time window, and is set redundantly along the time axis in the music waveform data DW.

  And about the frequency spectrum of all the waveform segments derived | led-out by this frequency analysis, the result of averaging the amplitude (absolute value) for every frequency component is derived | led-out as a music spectrum.

Then, the music spectrum derived | led-out by S130 is transmitted to the information processing server 20 as one of terminal transmission information (SS20).
Furthermore, based on the music waveform data DW acquired in S120, the tempo of the target music (so-called BPM (Beats Per Minute), hereinafter referred to as music BPM) is estimated (S140).

Here, FIG. 3 is a diagram illustrating a method for estimating the music BPM in the present embodiment.
Specifically, in S140 of the present embodiment, a waveform segment is cut out for each unit time along the time axis from the musical sound transition waveform represented by the music waveform data DW. Then, the frequency spectrum of the percussion instrument (for example, drum) musical tone is simulated with an envelope, and a reference spectrum (see FIG. 3A) prepared in advance is cut out for each waveform segment for each unit time. The results are compared with each other (that is, the envelope of the frequency spectrum), and the similarity of the reference spectrum (hereinafter referred to as spectrum similarity) with respect to the frequency spectrum (envelope) of each waveform segment is derived.

  Furthermore, as shown in FIG. 3B, a similarity transition representing a transition of the spectrum similarity along the time axis of the target music is derived, and in the similarity transition, the maximum value of the spectrum similarity is obtained, A time interval (maximum in FIG. 3B: TI) between the maximum values of the spectral similarity adjacent to each other is obtained. Then, a time interval distribution representing a histogram with the time interval TI as a class as shown in FIG. 3C is generated, and the time interval corresponding to the largest frequency in the time interval distribution (hereinafter, the mode time interval). The reciprocal of TI is estimated as the music BPM.

  The method for estimating the music BPM will be described in more detail. In the present embodiment, if the unit of the most frequent time interval TI is seconds [s: second], the music BPM is estimated at 60 / the most frequent time interval TI. If the unit of the most frequent time interval TI is minutes [m: minute], it is preferable to estimate with the music BPM = 1 / the most frequent time interval TI.

  The estimation of the music BPM in S140 may be executed for a specific section such as a singing section (for example, rust) in which lyrics are defined in the target music, or for all sections of the target music. May be executed.

  In the karaoke playback process, the music BPM estimated in S140 is transmitted to the information processing server 20 as one of the terminal transmission information (SS30). Thereafter, bass waveform information, which is a waveform in which the tone of the bass guitar has shifted along the time axis, is extracted from the tone transition waveform represented by the music waveform data DW (S150). Specifically, in S150 of the present embodiment, a band limiting filter that allows a signal in the frequency band of the musical sound of the bass guitar to pass is prepared in advance, and a musical sound transition waveform represented by the music waveform data DW is stored in the band limiting filter. Pass through. Then, a signal waveform obtained by re-sampling the signal waveform after passing through the band limiting filter at a sampling frequency lower than the sampling frequency of the musical tone transition waveform (that is, a down-sampled signal waveform) is extracted as base waveform information.

In the karaoke playback process, the base waveform information is transmitted to the information processing server 20 as one of the terminal transmission information (SS40).
In the karaoke playback process, when the corrected output data DO and the lyrics telop data DT of the target music are received from the information processing server 20, the music waveform data DW corresponding to the target music is played and the music waveform data DW is played back. In addition, the lyrics constituent characters are displayed on the display unit 7 (S160). The display of the lyrics constituent characters in S160 includes changing the color of the lyrics constituent characters displayed on the display unit 7 together with the reproduction of the music waveform data DW.

Thereafter, the karaoke playback process is terminated.
<About data correction processing>
Next, data correction processing executed by the control unit 30 of the information processing server 20 will be described in detail.

  As shown in FIG. 2, when the data correction process is started, it is first determined whether or not the music tag data DD is received from the portable terminal 5 (S310). If the music tag data DD is not received as a result of the determination in S310 (S310: NO), the process waits until the music tag data DD is received, and receives the music tag data DD from the portable terminal 5 (S310: YES). ), The process proceeds to S320.

In S320, the music MIDI data DM and the lyrics data group DL corresponding to the target music represented by the music tag data DD received in S310 are acquired. Then, based on the music MIDI data DM acquired in S320 and the music spectrum received from the portable terminal 5, the music MIDI is set so that the pitch of the output sound matches the pitch of the musical sound constituting the target music. A pitch correction process for correcting the data DM is executed (S330). Hereinafter, the music MIDI data DM in which the output sound has been corrected is referred to as corrected music MIDI data DM.
<Pitch correction processing details>
Here, the pitch correction process started in S150 of the data correction process will be described.

  When this pitch correction process is started, as shown in FIG. 4, all output sounds are set along the time axis based on all score tracks included in the music MIDI data DM acquired in the previous S320. An output sound transition waveform that is a waveform that has been shifted is acquired (S510). Specifically, the acquisition of the output sound transition waveform in the present embodiment is executed by well-known rendering that generates an audio signal (waveform) from MIDI standard data.

  Subsequently, the obtained output sound transition waveform is subjected to frequency analysis for each unit time set along the time axis, and a frequency spectrum for the output sound transition waveform of each unit time is derived (S520). Based on the frequency spectrum derived in S520, an average output sound spectrum obtained by arithmetically averaging the intensity of each frequency component for each frequency along the time axis is derived (S530). The intensity of the frequency component in the derived average output sound spectrum is averaged for each frequency range (for example, semitone unit, hereinafter, the specified pitch range) so that the boundaries are adjacent to each other, and used as a representative value ( S540). Further, a normalized output sound spectrum (see FIG. 5A) is derived by normalizing the intensity of the frequency component in the average output sound spectrum averaged in S540 so that the variance is “1” and the average is “0”. (S550).

  Subsequently, with respect to the music spectrum acquired from the mobile terminal 5, the intensity of the frequency component is averaged for each specified pitch range to obtain a representative value, and the intensity of the frequency component becomes variance “1” and average “0”. A normalized musical tone spectrum (see FIG. 5B) normalized as described above is derived (S560).

  Note that the representative value obtained in S540 and S560 of the present embodiment may be the intensity at a frequency corresponding to the center value in the specified pitch range. In this case, specifically, for each 20 Cent (every fifth of a semitone), a process of extracting a frequency value (power) closest to the 20 Cent grid is performed.

  Then, a correlation value (hereinafter referred to as a pitch correlation value) between the normalized output sound spectrum and the normalized musical sound spectrum is derived (S570). Then, it is determined whether or not the shift amount of the normalized output sound spectrum with respect to the normalized musical sound spectrum is equal to or greater than a predetermined upper limit value (S580). As a result of the determination, if the shift amount is less than the upper limit value (S580: NO), the normalized output sound spectrum is shifted by a predetermined amount along the frequency axis (S590), and the process returns to S570. The pitch correlation value is derived again.

  That is, in S570 to S590 of the present embodiment, as shown in FIG. 5C, the normalized output sound spectrum is shifted from the lower limit value to the upper limit value along the frequency axis with respect to the normalized musical sound spectrum. On the other hand, a pitch correlation value is derived each time the normalized output sound spectrum is shifted.

  Then, when the shift amount of the normalized output sound is equal to or greater than the upper limit (S580: YES), a correction amount (hereinafter referred to as pitch correction) for matching the pitch of the output sound to the pitch of the musical sound constituting the target music. (S600) is determined. In S600 of the present embodiment, specifically, among the pitch correlation values derived in the previous S570, the shift amount of the normalized output sound spectrum corresponding to the pitch correlation value having the maximum value is calculated. Determined as pitch correction amount.

  Subsequently, the corrected music MIDI data DM is generated by correcting the pitches of the individual output sounds defined for all the score tracks in the music MIDI data DM according to the derived pitch correction amount (S610). . That is, the modified music MIDI data DM generated in S610 of the present embodiment is obtained by shifting the pitch correction amount of the pitch of all output sounds from the pitch of the output sound prepared in advance.

After that, the pitch correction process is terminated and the process returns to the data correction process.
Subsequently, in the data correction process shown in FIG. 2, the music MIDI is matched with the music BPM based on the tempo data of the target music included in the music MIDI data DM and the music BPM from the mobile terminal 5. BPM correction for correcting the tempo data included in the data DM is executed (S340).

  Specifically, in S340 of this embodiment, tempo data is extracted from the music MIDI data DM, and a ratio α between the extracted tempo data (ie, BPM) and the music BPM from the portable terminal 5 is derived. Then, multiplying the derived ratio α by the tempo data of all sections along the time axis of the music MIDI data is executed as BPM correction.

  Furthermore, based on the base waveform information from the portable terminal 5, the output sound (hereinafter referred to as a corrected output sound) that has been corrected by the pitch correction process so that the pitch of the output sound matches the pitch of the musical tone. The amount of deviation between the performance start timing and the performance start timing of the musical sound (hereinafter referred to as the time deviation amount) is derived, and the performance of each output sound is matched so as to coincide with the performance start timing of the musical sound constituting the target music. Timing correction processing for generating corrected music MIDI data DM with the start timing corrected is executed (S350).

<Processing details of timing correction processing>
Next, the timing correction process started in S350 of the data correction process will be described.

  When this timing correction process is started, as shown in FIG. 6, an output simulating a bass guitar based on a score track corresponding to the bass guitar among the score tracks included in the modified music MIDI data DM, as shown in FIG. A bass instrument waveform, which is a waveform in which the sound has shifted along the time axis, is acquired (S710).

  Then, the base instrument waveform is made to match the base setting position defined on the time axis of the base instrument waveform acquired in S710 with the waveform reference position defined on the time axis of the base waveform information from the portable terminal 5. And a correlation value between the base waveform information (hereinafter referred to as a time correlation value) are derived (S720).

  Subsequently, it is determined whether or not the shift amount along the time axis of the bass instrument waveform with respect to the bass waveform information is equal to or greater than a predetermined upper limit value (upper limit value of the shift amount) (S730). As a result of the determination, if the shift amount of the bass instrument waveform is less than the upper limit value of the shift amount (S730: NO), the waveform setting position of the bass instrument waveform is shifted by a predetermined shift amount (ie, time). (S740), the process returns to S720.

  That is, in S720 to S740 of this embodiment, as shown in FIG. 7, the bass instrument waveform is shifted with respect to the bass waveform information from the portable terminal 5 until it reaches the upper limit value, and each time the bass instrument waveform is shifted. Deriving a time correlation value.

  On the other hand, if the result of determination in S730 is that the shift amount is greater than or equal to the upper limit value of the shift amount (S730: YES), the performance start timing of the modified output sound is made to coincide with the performance start timing of the musical sound constituting the target music. Correction amount, that is, a time shift amount is determined (S750). Specifically, in S750 of the present embodiment, the shift amount of the bass instrument waveform with respect to the bass waveform information corresponding to the time correlation value having the maximum value among all the time correlation values derived in S720 is calculated. Determine as quantity.

  Subsequently, the performance start of each output sound specified in the music MIDI data DM so as to coincide with the performance start timing of each music sound in the music waveform data DW according to the amount of time deviation derived in the timing correction processing. The timing is corrected (S760). Specifically, in S760 of the present embodiment, the performance start timing of each output sound defined in the music MIDI data DM is shifted by a time shift amount along the time axis.

Thereafter, the process returns to the data correction process shown in FIG.
In the data correction process, the performance start timing of each output sound of the guide melody included in the corrected music MIDI data DM generated in the timing correction process, and each of the guide melody included in the music MIDI data DM before correction A time difference (hereinafter referred to as a timing difference) from the performance start timing of the output sound is derived (S360).

  Subsequently, the lyric output data DO of the target music is acquired, and the lyric output timing of each lyric constituent character defined in the lyric output data DO matches the musical performance start timing in the music waveform data DW. Corrected output data DO with the lyrics output timing corrected is generated (S370). Specifically, in S370 of the present embodiment, the corrected output data DO is generated by shifting the lyrics output timing of each of the lyrics constituent characters along the time axis of the target music by the timing difference derived in S360.

  Further, the lyrics telop data DT of the target music and the corrected output data DO (in FIG. 2: corrected lyrics data) generated in S370 are transmitted to the mobile terminal 5 that has transmitted the music tag data DD received in S310. (S380).

Thereafter, the process returns to S310 and waits until new music tag data DD is received.
[Effect of the embodiment]
As described above, in the corrected output data DO generated by the information processing server 20 constituting the information processing system 1 of the present embodiment, the lyrics output timing coincides with the tone output start timing based on the music waveform data DW. Will be. Therefore, when the target music is played on the portable terminal 5 constituting the information processing system 1 based on the music waveform data DW, the lyrics constituent characters are sequentially displayed in accordance with the musical sound output timing of the target music. It is possible to reduce the deviation of the display of each lyric constituent character from the musical performance timing.

  Further, in the information processing system 1 of the present embodiment, the difference between the BPM based on the music waveform data DW and the BPM based on the music MIDI data DM, the timing at which the bass guitar is played in the music waveform data DW, and the music MIDI data DM The timing deviation of the bass guitar output is derived. Since the information processing server 20 generates the corrected output data DO using these deviations, the generated corrected output data DO is the lyric constituent characters with respect to the musical performance timing based on the music waveform data DW. The deviation of the output timing is more reliably reduced.

  Moreover, in the above-described embodiment, the data transmitted from the information processing server 20 to the portable terminal 5 is two data of the lyrics telop data DT and the corrected output data DO. Therefore, according to the information processing system 1, the mobile terminal 5 transmits the lyrics constituent characters from the information processing server 20 to the mobile terminal 5 in order to sequentially display the lyrics constituent characters in accordance with the output timing of the musical tone of the target music. The amount of data communication can be reduced.

Further, in the above embodiment, the base waveform information transmitted from the mobile terminal 5 to the information processing server 20 is a signal waveform obtained by down-sampling the tone transition waveform after passing through the band limiting filter. Therefore, according to the information processing system 1, the amount of communication from the portable terminal 5 to the information processing server 20 can be reduced more reliably.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, in the karaoke playback process of the above embodiment, the music tag data DD of the target music is transmitted to the information processing server 20 in S110. However, in S110, the data transmitted from the portable terminal 5 is the music tag data DD. It is not limited. That is, in S110, the data transmitted from the portable terminal 5 may be music waveform data DW for a predetermined period (for example, a period corresponding to a so-called intro) from the start of performance of the target music. In this case, the information processing server 20 may recognize the music by analyzing the music waveform data DW by a known method. At this time, the music tag data DD may not be stored in the storage unit 10 of the mobile terminal 5.

In other words, as long as the information processing server 20 can identify the target music piece, the data transmitted from the portable terminal 5 in S110 may be any data.
Further, in the timing correction process of the above embodiment, the base instrument waveform is shifted until the upper limit is reached with respect to the base waveform information from the portable terminal 5, and the time correlation value is derived each time the base instrument waveform is shifted. In the results, the shift amount that maximizes the time correlation value is determined as the time shift amount, but the method for determining the time shift amount is not limited to this. For example, along the time axis of the base instrument waveform with respect to the base waveform information derived so that the first peak along the time axis of the base instrument waveform matches the first peak along the time axis of the base waveform information. The shift amount may be determined as a time shift amount.

In the karaoke playback process of the above embodiment, the base waveform information is transmitted to the information processing server 20 as one of terminal transmission information at SS40. However, the data transmitted at SS40 is limited to this. is not. For example, percussion instrument waveform information that is a waveform in which the musical tone of a percussion instrument has changed may be transmitted to the information processing server 20 as one of the terminal transmission information. In this case, the percussion instrument waveform information is preferably terminal transmission information in addition to the base waveform information.
[Correspondence between Embodiment and Claims]
Finally, the relationship between the description of the above embodiment and the description of the scope of claims will be described.

  S110 (SS10) in the karaoke playback process of the above embodiment corresponds to the identification information acquisition unit and the identification information transmission unit in the description of the claims, and S140 (SS30) and S150 (SS40) correspond to the claims. This corresponds to the speed information transmitting means in the description. Furthermore, the function of acquiring the data transmitted at SS50 of the data correction process in the portable terminal 5 corresponds to the lyrics information acquisition means in the description of the claims, and S160 in the karaoke playback process is claimed. This corresponds to the performance output means in the description of the range.

  Further, S320 in the data correction process of the above embodiment corresponds to the data acquisition means in the description of the claims, and S340 in the data correction process and S710 in the timing correction process acquire the performance speed in the description of the claims. Corresponds to means. S720 to S750 in the timing correction process correspond to the time deviation amount deriving means in the description of the claims, and S370 in the data correction process corresponds to the data correction means in the description of the claims, and the data correction S380 in the processing corresponds to the lyrics transmitting means in the description of the claims.

  Further, S130 (SS20) in the karaoke playback process of the above embodiment corresponds to the spectrum transmitting means in the description of the claims, and S510 to S550 in the pitch correction process are the output sound spectrum in the description of the claims. S570 to S590 in the pitch correction process corresponds to the derivation means, and the pitch correlation derivation means in the description of the claims corresponds to the pitch correlation, and S600 and S610 in the pitch correction processing correspond to the musical score in the description of the claims. It corresponds to data correction means.

  DESCRIPTION OF SYMBOLS 1 ... Information processing system 5 ... Portable terminal 6 ... Information reception part 7 ... Display part 8 ... Sound output part 9 ... Communication part 10 ... Storage part 11 ... Control part 12 ... ROM 13 ... RAM 14 ... CPU 20 ... Information processing server 21 ... Communication unit 22 ... Input reception unit 23 ... Display unit 24 ... Sound input unit 25 ... Sound output unit 26 ... Sound source module 27 ... Storage unit 30 ... Control unit 31 ... ROM 32 ... RAM 33 ... CPU

Claims (4)

Information processing for performing data communication between a portable device in which music waveform data representing a musical tone transition waveform in which the sound pressure of the musical tone constituting the musical piece has shifted along the time axis is stored in the storage unit, and the portable device An information processing system comprising a device,
The information processing apparatus includes:
A musical score data that represents the musical score of the music, along with performance speed information that represents the progress of the music, and at least the pitch and performance start timing for each output sound that is output from the sound module, and the lyrics that constitute the lyrics of the music Lyric data representing constituent characters, and data defining the lyrics output timing which is the output timing of the lyrics constituent characters, and the lyrics output timing for at least one of the lyrics constituent characters is at least one in the musical score data Is a device that acquires lyric output data associated with one timing and performs information processing on at least the lyric output data;
The portable device is
Identification information acquisition means for acquiring music identification information for identifying the target music that is the designated music;
Identification information transmission means for transmitting the music identification information acquired by the identification information acquisition means to the information processing apparatus;
Speed information transmission means for acquiring music speed information representing the progress of the target music from the music waveform data of the target music corresponding to the music identification information acquired by the identification information acquisition means, and transmitting the music speed information to the information processing apparatus. When,
Lyrics for acquiring from the information processing apparatus the lyrics data of the target music and the correction output data that is the lyrics output data that has been processed by the information processing apparatus, and the correction output data of the target music Information acquisition means;
The lyric constituent characters represented by the lyric data according to the lyric output timing defined in the corrected output data acquired by the lyric information acquisition means while playing the target tune based on the tune waveform data of the tune Performance output means for sequentially displaying
The information processing apparatus includes:
Data acquisition means for acquiring music score data of the target music corresponding to the music identification information transmitted by the identification information transmission means, and lyrics output data;
Performance speed acquisition means for acquiring the performance speed information from the music score data acquired by the data acquisition means,
Corresponding to each output sound with respect to each performance start timing of the output sound based on the result of comparing the music speed information transmitted by the speed information transmitting means and the performance speed information acquired by the performance speed acquiring means A time shift amount deriving means for deriving a time shift amount representing a shift amount of the start timing of the musical sound,
The lyrics output data in which the lyrics output timing in the lyrics output data acquired by the data acquisition means is corrected so as to coincide with the start timing of the music according to the amount of time deviation derived by the time deviation amount deriving means. Data correction means for generating as the corrected output data,
Lyrics transmitting means for transmitting the corrected output data generated by the data correcting means and the lyrics data of the target music corresponding to the music identification information to the portable device,
The mobile device speed information transmitting means includes:
The tempo information indicating the tempo of the target music specified based on the music waveform data of the target music, and the base waveform information indicating the timing when the specific musical instrument is played are acquired as the music speed information,
The music score data is
The music tempo information, which is information representing the tempo of the music corresponding to the music score data, is specified, and the pitch and performance start timing for the output sound are specified for each type of musical instrument played in the corresponding music. Has been
The performance speed acquisition means of the information processing apparatus includes:
An information processing system, wherein the musical instrument tempo information and a bass instrument waveform in which a performance start timing of an output sound of the specific instrument has changed along a time axis are acquired as the performance speed information. The base waveform information is a sampling waveform,
The base waveform information transmitted by the speed information transmitting means of the portable device is:
The information processing system according to claim 1, wherein the information is sampled at a sampling frequency lower than a sampling frequency of the tone transition waveform. The portable device is
Obtaining a music waveform data of the target music, deriving a music spectrum distribution which is a frequency spectrum of a music transition waveform represented by the music waveform data, and comprising a spectrum transmission means for transmitting to the information processing apparatus,
The information processing apparatus includes:
Output sound that obtains music score data of the target music and derives an output sound spectrum distribution that is a frequency spectrum of an output sound transition waveform that is a waveform in which the output sound has shifted along the time axis based on the music score data Spectrum derivation means;
A pitch correlation value representing a correlation value between the musical sound spectrum distribution transmitted by the spectrum transmitting means and the output sound spectrum distribution derived by the output sound spectrum deriving means is defined in advance in the musical spectrum distribution. A pitch correlation deriving means for deriving each time the output sound spectrum distribution is shifted from the position along the frequency axis;
Among the pitch correlation values derived by the pitch correlation deriving means, a shift amount along the frequency axis from the specified position corresponding to the pitch correlation value having the maximum value is used as a pitch correction amount. Musical score data correcting means for generating new musical score data by shifting the pitch of each of the output sounds defined in the musical score data of the target music,
The performance speed acquisition means includes
The information processing system according to claim 1 or 2, wherein the performance speed information is acquired from new musical score data generated by the musical score data correcting means. A musical score data that represents the musical score of the music, along with performance speed information that represents the progress of the music, and at least the pitch and performance start timing for each output sound that is output from the sound module, and the lyrics that constitute the lyrics of the music Lyric data representing constituent characters, and data defining the lyrics output timing which is the output timing of the lyrics constituent characters, and the lyrics output timing for at least one of the lyrics constituent characters is at least one in the musical score data Lyric output data associated with a specific start timing that is one timing, and at least data communication with an information processing device that performs information processing on the lyric output data, and composes a song Music waveform data that represents the sound transition waveform in which the sound pressure of the music to be played changes along the time axis. Data is a program which is executed by a provided computer to a portable device which is stored in the storage unit,
An identification information acquisition procedure for acquiring music identification information for identifying a target music that is a designated music;
An identification information transmission procedure for transmitting the music identification information acquired in the identification information acquisition procedure to the information processing apparatus;
A speed information transmission procedure for acquiring music speed information representing the progress of the target music from the music waveform data of the target music corresponding to the music identification information acquired in the identification information acquisition procedure, and transmitting it to the information processing apparatus. When,
The music score data of the target music corresponding to the music identification information transmitted in the identification information transmission procedure, the data acquisition procedure for acquiring the lyrics output data, and the performance speed information from the music score data acquired in the data acquisition procedure The performance start timing of each of the output sounds based on the performance speed acquisition procedure to be acquired and the result of comparison between the performance speed information acquired in the performance speed acquisition procedure and the music speed information transmitted in the speed information transmission procedure In accordance with a time deviation amount derivation procedure for deriving a time deviation amount representing the deviation amount of the start timing of the musical sound corresponding to each output sound, and the acquired lyrics according to the time deviation amount derived in the time deviation amount derivation procedure Correct the lyrics output data in which the lyrics output timing in the output data is corrected to match the start timing of the musical sound It is a result in which the data correction step of generating a force data is performed as the processing, word information which the modified output data, and the lyrics data of the target musical piece corresponding to the music identification information, acquires from the information processing apparatus Acquisition procedure;
The lyric constituent characters represented by the lyric data according to the lyric output timing defined in the modified output data acquired in the lyric information acquisition procedure while playing the target tune based on the tune waveform data of the tune And a performance output procedure for sequentially displaying
In the speed information transmission procedure,
The tempo information indicating the tempo of the target music specified based on the music waveform data of the target music, and the base waveform information indicating the timing when the specific musical instrument is played are acquired as the music speed information,
The music score data is
The music tempo information, which is information representing the tempo of the music corresponding to the music score data, is specified, and the pitch and performance start timing for the output sound are specified for each type of musical instrument played in the corresponding music. Has been
The performance speed information acquired by the performance speed acquisition procedure is:
The music tempo information, and program characterized based instruments waveform der Rukoto the performance start timing of the output sound of a particular instrument remained along the time axis.

Images