JP5983670B2 - Program, information processing apparatus, and data generation method - Google Patents

Description

  The present invention relates to a program for generating data, an information processing apparatus, and a data generation method.

Conventionally, a song evaluation technique for evaluating the skill of a song that sang the song melody has been known (see Patent Document 1).
In the technique described in Patent Document 1, skill is evaluated by comparing the voice sung by the user with the model voice prepared for each piece of music, and a comment corresponding to the evaluation is presented after singing.

JP 2007-233303 A

  By the way, in the music sung by a professional singer, in many cases, a unique characteristic appears for each singer of the music. When a user sings using a karaoke device, an information processing device, or the like, if the singing-specific characteristics of the singer can be reproduced, it sounds like singing better.

For this reason, in a karaoke apparatus, an information processing apparatus, etc., about the characteristic of the way of singing peculiar to a singer, before the user actually sings, the characteristic of the way of singing is required to be presented.
However, in the conventional technology, the difference between the voice sung by the user and the model voice is only presented after the user sings. Therefore, users such as karaoke devices and information processing devices have a problem that they cannot recognize how to sing before singing about the singing-specific characteristics of the singer.

That is, in a karaoke apparatus, an information processing apparatus, or the like, there is a problem that the user does not present any characteristics of the singing-specific singing method before the user actually sings.
Therefore, an object of the present invention is to provide a technique capable of generating data necessary for presenting the characteristics of a singer-specific singing method.

  The present invention made to achieve the above object relates to a program to be executed by a computer. The program of the present invention causes a computer to execute a first acquisition step, a second acquisition step, an extraction step, a timing specifying step, a first generation step, a time calculation step, and a second generation step. .

  Among these, in the first acquisition step, musical score data representing a musical score of a music composed of a plurality of notes, and at least the musical score data in which the pitch and performance start timing are defined for each of the plurality of notes. , Acquired from the first storage unit. In the second acquisition step, music data including the vocal sound that sang the music is acquired from the second storage unit.

  Furthermore, in the extraction step, vocal data representing vocal sound is extracted from the music data acquired in the second acquisition step. In the timing specifying step, based on the vocal data extracted in the extracting step, each utterance timing that can be regarded as a timing at which utterance is started in the vocal data is specified. In the first generation step, based on each of the utterance timings specified in the timing specification step and the performance start timing of the notes in the score data acquired in the first acquisition step, To generate timing pair data.

  Then, in the time calculation step, the time is calculated because it is the time difference between the performance start timing and the utterance timing in the timing pair data based on each of the timing pair data generated in the pair generation step. Then, in the time calculation step, the time is calculated because it is the time difference between the performance start timing and the utterance timing based on each of the timing pair data generated in the first generation step. Then, in the second generation step, the “time” calculated in the time calculation step is associated with the musical score data notes corresponding to the performance start timing used for the calculation of the time, so that the singer's way of singing the notes Singing feature data representing the feature is generated.

  The singing feature data generated by such a program is obtained by associating “time” with the notes of the score data. In particular, in the program of the present invention, the difference between the performance start timing of the notes in the score data and the utterance timing detected from the vocal sound sung by a professional singer included in the music data is calculated as “time”. Therefore, “time” represents a characteristic of “for” as a singing technique used by a professional singer when singing music.

In addition, in the singing feature data, the “time” is associated with the musical score data corresponding to the performance start timing used to calculate the time.
For this reason, according to the singing characteristic data, “time” in the note can be presented on the display unit connected to the karaoke apparatus or the information processing apparatus. In particular, according to the singing feature data, the user of the karaoke apparatus or the information processing apparatus can present “time” before actually singing.

  And a user of a karaoke apparatus or an information processing apparatus in which “the time” is presented on the display device, how a professional singer uses “for” as a singing technique in the song being sung Can be recognized. As a result, the user of the karaoke apparatus or the information processing apparatus can bring the user's own way of singing closer to that of a professional singer.

  In the timing specifying step in the present invention, each utterance start timing at which the sound pressure in the vocal data is equal to or higher than a predetermined threshold may be specified as the utterance timing.

  Furthermore, the first generation step in the present invention may generate timing pair data that combines the performance start timing of the notes in the score data and the utterance start timing with the shortest time length from the performance start timing. Therefore, the time calculation step may calculate the time for sound generation as the time difference between the performance start timing and the utterance start timing in the generated timing pair data.

  According to such a program, “time for sound generation” can be calculated as a kind of “time for sound”. This “pronunciation time” is a time difference between the note performance start timing in the score data and the utterance start timing at which the singer actually started speaking in the music data, and is a characteristic of the singing method peculiar to the singer.

For this reason, according to the program of the present invention, singing characteristic data including “time for pronunciation” can be generated as a characteristic of a singing method peculiar to a singer.
As a result, in the karaoke apparatus or the information processing apparatus, it is possible to present how much the timing at which a professional singer starts speaking deviates from the performance start timing in a note.

  In the time calculation step according to the present invention, the representative value for pronunciation, which is a representative value in the musical composition of time for sound generation, is calculated, and based on the time for sound generation and the representative value for sound generation, the specified time from the representative value for sound generation The time for special pronunciation, which is the time for sound that deviates more than a long time, may be calculated as time.

  The “special pronunciation time” specified in such a program is a time for notes that clearly show “for” as a singing skill in the music, and is a prominent feature in the way of singing the singer who sang the music. is there.

Therefore, according to the program of the present invention, singing feature data including “time for special pronunciation” can be generated as a prominent feature in the way of singing a singer.
As a result, the location where the professional singer prominently uses “for” in the music can be presented via the karaoke apparatus or the information processing apparatus. And the user of a karaoke apparatus or an information processing apparatus can make the user's own way of singing close to the way of singing a professional singer only at the place where the professional singer uses “for” prominently.

  In the timing specifying step of the present invention, the pitch change timing, which is the timing at which the vocal frequency transition representing the pitch transition in the vocal data falls within the pitch range of the musical score data, is specified as the utterance timing. To do. In the first generation step, a musical note performance start timing in the score data and the performance start timing are within a specified time range defined in advance along the time axis, and the time length from the performance start timing is the shortest. Timing pair data in which the pitch change timing is associated is generated.

  Further, in this case, the time calculation step calculates the time for the pitch, which is the time difference between the performance start timing and the pitch change timing in the timing pair data generated in the first generation step, as the time.

  According to such a program, “time for pitch” can be calculated as a kind of “for time”. This “pitch time” is the time difference between the pitch of the note in the score data and the pitch of the singing voice sung by the singer and the timing at which the note starts to be played. Appears as a feature of

Therefore, according to the program of the present invention, singing feature data including “time for pitch”, which is a characteristic of a singing method peculiar to each singer, can be generated.
As a result, in the karaoke apparatus or the information processing apparatus, it is possible to present how much the professional singer has shifted the timing for matching the pitch of the sung voice to the pitch of the note in the score data.

  Further, in the time calculation step according to the present invention, in the time calculation step, a representative value is calculated for the pitch, which is a representative value in the music of the time for the pitch. The time may be specified because of a special pitch that is a time because the pitch deviates from the representative value by a specified time length or more.

  The “special pitch time” specified in such a program is a note in which the pitch of the note in the score data matches the pitch in the singing voice sung by a professional singer more prominently. This is a remarkable feature of how to sing a singer who sang music.

Therefore, according to the program of the present invention, singing feature data including “time for special pitch”, which is a remarkable feature in the way of singing a singer, can be generated.
As a result, it is possible to present, via the karaoke apparatus or the information processing apparatus, a position where the difference in timing for matching the pitches of the singing voices sung by the professional singer with the pitches of the musical notes in the musical score data is significant. And the user of the karaoke apparatus or the information processing apparatus can use the user himself / herself only in a portion where there is a significant difference in the timing of matching the pitch of the singing voice sung by a professional singer with the pitch of the notes in the musical score data. Can be brought close to how professional singers sing.

  Furthermore, in the program of the present invention, the computer may execute the third acquisition step, the fourth acquisition step, and the display control step. In the third acquisition step, the score data of the designated music is acquired. In the fourth acquisition step, the singing feature data generated in the data generation step is acquired corresponding to the score data acquired in the third acquisition step. Further, in the display control step, each of the notes constituting the score data acquired in the third acquisition step is displayed on the display unit, and each time is counted because it is included in the singing feature data acquired in the fourth acquisition step. It is displayed on the display unit in association with the musical score data note corresponding to the performance start timing used for calculating the time.

  According to such a program, it is possible to present “time” in the notes constituting the music based on the score data on the display unit connected to the karaoke apparatus or the information processing apparatus. Furthermore, in the program of the present invention, if each of the notes constituting the musical score data is sequentially displayed on the display unit along the time axis, before the user of the karaoke apparatus or the information processing apparatus actually sings, “Time” can be displayed on the display.

  As a result, users of karaoke devices and information processing devices whose “time” is presented on the display unit, how the professional singer uses “for” as a singing technique in the song being sung You can recognize if you are.

By the way, this invention may be made | formed as information processing apparatus.
An information processing apparatus according to the present invention includes a first acquisition unit, a second acquisition unit, an extraction unit, a timing specifying unit, a first generation unit, a time calculation unit, and a second generation unit.

  The first acquisition unit acquires the musical score data from the first storage unit, and the second acquisition unit acquires the music piece data from the second storage unit. The extracting means extracts vocal data from the music data, and the timing specifying means specifies each utterance timing based on the extracted vocal data. The first generation means generates timing pair data based on each of the specified utterance timings and the performance start timing of the notes in the musical score data, and the time calculation means therefore adds the timing pair data to each of the generated timing pair data. Based on this, the time is calculated. Then, the second generation means generates singing feature data by associating the calculated “time” with notes of the score data corresponding to the performance start timing used for calculating the time.

According to such an information processing apparatus, the same effect as the program according to the present invention can be obtained.
The present invention may be implemented as a data generation method executed by the information processing apparatus.

The data generation method according to the present invention includes a first acquisition procedure, a second acquisition procedure, an extraction procedure, a timing specifying procedure, a first generation procedure, a time calculation procedure, and a second generation procedure.
In the first acquisition procedure, the information processing apparatus acquires score data from the first storage unit, and in the second acquisition procedure, the information processing apparatus acquires music data from the second storage unit. In the extraction procedure, the information processing device extracts vocal data from the acquired music data, and in the timing specifying procedure, the information processing device specifies each utterance timing based on the extracted vocal data. In the first generation procedure, the information processing device generates timing pair data based on each of the specified utterance timings and the performance start timing of the notes in the score data. Therefore, in the time calculation procedure, the generated timing pair Based on each data, the information processing apparatus calculates time. Further, in the second generation procedure, the information processing apparatus generates singing feature data by associating the calculated “time” with notes of musical score data corresponding to the performance start timing used for calculating the time. .

According to such a data generation method, the same effect as the program according to the present invention can be obtained.

It is a block diagram which shows schematic structure of the system provided with the information processing apparatus with which this invention was applied. It is a flowchart which shows the process sequence of the data generation process which information processing apparatus performs. It is a flowchart which shows the process sequence of the calculation process for pronunciation. It is a figure which illustrates the outline | summary of the calculation method of time for pronunciation. It is a flowchart which shows the process sequence of a calculation process for a pitch. It is a figure which illustrates the outline | summary of the calculation method of time for a pitch. It is a figure which illustrates the outline | summary of singing characteristic data. It is a flowchart which shows the process sequence of the display process which a karaoke apparatus performs. It is a figure which illustrates the aspect of the display displayed on a display part by performing a display process.

Embodiments of the present invention will be described below with reference to the drawings.
<System configuration>
The karaoke device 30 shown in FIG. 1 is a device that displays on the display unit 64 the characteristics of how to sing a professional singer expressed in the music while playing the music specified by the user.

  The system 1 constructed in order for the karaoke device 30 to display such characteristics of how to sing a professional singer includes the information processing device 2, the information processing server 10, and the karaoke device 30.

  The information processing device 2 calculates the singing feature data SF based on the music data WD and the MIDI music MD prepared for each music. The singing feature data SF referred to here is data representing features in how to sing a professional singer who sings music.

In the information processing server 10, at least the singing feature data SF and the MIDI music MD calculated by the information processing device 2 are stored in the storage unit 14 in association with each corresponding music.
<Music data>
Next, the music data WD is prepared in advance for each specific music, and includes music management information in which information related to the music is described, and master waveform data representing the performance sound of the music. The music management information includes music identification information (hereinafter referred to as music ID) for identifying music.

  The master waveform data of this embodiment is audio data including performance sounds of a plurality of musical instruments and vocal sounds sung by a professional singer. The audio data may be data constituted by an audio file in an uncompressed audio file format, or data constituted by an audio file in an audio compression format.

  In the following, voice waveform data representing the performance sound of the musical instrument included in the master waveform data is referred to as accompaniment data, and voice waveform data representing the vocal sound included in the master waveform data is referred to as vocal data.

Musical instrument performance sounds included in the accompaniment data of the present embodiment include percussion instrument (eg, drum, drum, cymbal, etc.) performance sounds, stringed instrument (eg, guitar, bass, etc.) performance sounds, percussion instrument (eg, piano) ) And wind instruments (eg, trumpet, clarinet, etc.).
<MIDI music>
The MIDI music MD is prepared in advance for each music and has performance data and lyrics data.

  Of these, the performance data is data representing the score of one piece of music according to the well-known MIDI (Musical Instrument Digital Interface) standard. This performance data has at least a music ID and a music score track representing a music score for each musical instrument used in the music.

  The musical score track defines at least the pitch (so-called note number) and the period during which the MIDI sound source outputs the performance sound (hereinafter referred to as the note length) for each performance sound output from the MIDI sound source. Has been. The note length in the score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the performance sound and the music until the output of the performance sound ends. Performance end timing (so-called note-off timing) representing the time from the start of the performance.

  That is, in the score track, one note NO is defined by the note number and the note length represented by the note-on timing and note-off timing. The musical score track functions as one musical score by arranging note NO in the order of performance. Note that the musical score track is prepared for each instrument such as a keyboard instrument, a stringed instrument, a percussion instrument, and a wind instrument, for example. Among these, in this embodiment, a specific musical instrument (for example, vibraphone) is defined as a musical instrument responsible for singing melody in music.

  On the other hand, the lyrics data is data relating to the lyrics of the music, and includes lyrics telop data and lyrics output data. The lyrics telop data represents characters that constitute the lyrics of the music (hereinafter referred to as lyrics component characters). The lyrics output data is data in which a timing correspondence relationship that associates the lyrics output timing, which is the output timing of the lyrics constituent characters, with the performance of the performance data is defined.

Specifically, in the timing correspondence relationship in the present embodiment, the timing for starting the output of the lyrics telop data is associated with the timing for starting the performance of the performance data. Furthermore, in the timing correspondence relationship, the lyrics output timing of each lyrics constituent character along the time axis of the music is defined by the elapsed time from the performance start of the performance data. As a result, the note NO of each performance sound defined in the score track is associated with each of the lyrics constituent characters.
<Information processing device>
The information processing apparatus 2 is a known information processing apparatus (for example, a personal computer) including an input receiving unit 3, an information output unit 4, a storage unit 5, and a control unit 6.

  The input receiving unit 3 is an input device that receives input of information and commands from the outside. The input device here is, for example, a key or switch, a reading drive for reading data stored in a portable storage medium (for example, CD, DVD, flash memory), or a communication port for acquiring information via a communication network. Etc. The information output unit 4 is an output device that outputs information to the outside. Here, the output device is a writing drive that writes data to a portable storage medium, a communication port that outputs information to a communication network, or the like.

  The storage unit 5 is a known storage device configured to be able to read and write stored contents. The storage unit 5 stores at least one piece of music data WD and at least one MIDI piece of music MD in association with each common piece of music.

  The control unit 6 is a known control device that is configured around a known microcomputer including a ROM 7, a RAM 8, and a CPU 9. The ROM 7 stores processing programs and data that need to retain stored contents even when the power is turned off. The RAM 8 temporarily stores processing programs and data. The CPU 9 executes each process according to a processing program stored in the ROM 7 or RAM 8.

The ROM 7 of the present embodiment has a processing program for the control unit 6 to execute a data generation process for calculating the song feature data SF based on the music data WD and the MIDI music MD stored in the storage unit 5. It is remembered.
<Information processing server>
The information processing server 10 includes a communication unit 12, a storage unit 14, and a control unit 16.

  Among these, the communication unit 12 performs communication between the information processing server 10 and the outside via a communication network. That is, the information processing server 10 is connected to the karaoke apparatus 30 via a communication network. The communication network referred to here may be a wired communication network or a wireless communication network.

  The storage unit 14 is a known storage device configured to be able to read and write stored contents. The storage unit 14 stores at least a plurality of MIDI music pieces MD. 1 is an identifier for identifying the MIDI music piece MD stored in the storage unit 14 of the information processing server 10, and is a natural number of 1 or more. Furthermore, the storage unit 14 stores singing feature data SF generated by the information processing apparatus 2 executing data generation processing.

The control unit 16 is a known control device that is configured around a known microcomputer including a ROM 18, a RAM 20, and a CPU 22. The ROM 18, RAM 20, and CPU 22 are configured similarly to the ROM 7, RAM 8, and CPU 9, respectively.
<Karaoke equipment>
The karaoke apparatus 30 includes a communication unit 32, an input reception unit 34, a music playback unit 36, a storage unit 38, an audio control unit 40, a video control unit 46, and a control unit 50.

  In the communication unit 32, the karaoke apparatus 30 communicates with the outside via a communication network. The input receiving unit 34 is an input device that receives input of information and commands in accordance with external operations. Here, the input device is, for example, a key, a switch, a reception unit of a remote controller, or the like.

  The music playback unit 36 performs a music performance based on the MIDI music MD downloaded from the information processing server 10. The music reproducing unit 36 is, for example, a MIDI sound source. The voice control unit 40 is a device that controls voice input / output, and includes an output unit 42 and a microphone input unit 44.

  A microphone 62 is connected to the microphone input unit 44. Thereby, the microphone input part 44 acquires a user's song sound. A speaker 60 is connected to the output unit 42. The output unit 42 outputs the sound source signal of the music reproduced by the music reproducing unit 36 and the sound source signal of the singing sound from the microphone input unit 44 to the speaker 60. The speaker 60 outputs the sound source signal output from the output unit 42 instead of sound.

  The video control unit 46 outputs a video or an image based on the video data sent from the control unit 50. The video control unit 46 is connected to a display unit 64 that displays video or images.

  The control unit 50 is configured around a known computer having at least a ROM 52, a RAM 54, and a CPU 56. The ROM 52, RAM 54, and CPU 56 are configured similarly to the ROM 7, RAM 8, and CPU 9, respectively.

The ROM 52 stores a processing program for the control unit 50 to execute display processing. The display process is a process of playing the music designated by the user and displaying on the display unit 64 the characteristics of how to sing a professional singer expressed in the music, the lyrics of the music, and the singing melody.
<Data generation processing>
Next, data generation processing executed by the control unit 6 of the information processing apparatus 2 will be described.

This data generation process is started at the timing when a start command for starting the processing program is input via the input receiving unit 3 of the information processing apparatus 2.
In the data generation process, as shown in FIG. 2, when activated, the control unit 6 first selects one piece of music data WD stored in the storage unit 5 of the information processing device 2 from one piece of music data WD. The music data WD is acquired (S110). In S110 of the present embodiment, the control unit 6 acquires the music data WD from the storage unit 5. However, the control unit 6 via the input reception unit 3 via a portable storage medium or communication network. The music data WD may be acquired from a server or the like.

  In the data generation process, the control unit 6 subsequently acquires master waveform data included in the music data WD acquired in S110 (hereinafter referred to as “acquired music data”) (S120). Further, the control unit 6 separates and extracts vocal data and accompaniment data from the master disk waveform data acquired in S120 (S130). As a method of separating accompaniment data and vocal data, which is executed by the control unit 6 in S130, the pitch and pitch estimated using a well-known method (for example, “PreFEst” described in Japanese Patent Laid-Open No. 2008-134606). A method using wave components can be considered. Note that PreFEst is a technique for estimating the pitch of a vocal (that is, the fundamental frequency) and the magnitude of a harmonic component by regarding the most prevalent voice waveform in the master waveform data as vocal data.

  In the data generation process, subsequently, the control unit 6 specifies a vocal sound pressure transition representing a transition of the sound pressure level in the vocal data (S140). Further, the control unit 6 specifies a vocal frequency transition representing a transition of the fundamental frequency f0 in the vocal data (S150).

  Specifically, in S140 and S150 of the present embodiment, the control unit 6 first sets the specified time window AW (j) as vocal data. The specified time window AW (j) is an analysis window having a predetermined unit time (for example, 10 [ms]). In the present embodiment, the specified time window AW is set to be adjacent to and continuous with each other along the time axis. The symbol j is an identifier for identifying the specified time window AW.

Subsequently, the control unit 6 calculates the sound pressure level Lp in each prescribed time window AW (j) in the vocal data by a known method. The sound pressure level Lp is obtained by dividing a root mean square p of sound pressure in a prescribed time window AW (j) of vocal data by a common logarithm obtained by dividing by a sound pressure p ₀ as a reference (usually, “ 20 ") (ie Lp = 20 × log ₁₀ (p / p ₀ )).

Furthermore, the control unit 6 specifies the vocal sound pressure transition by arranging the sound pressure level Lp in each specified time window AW (j) along the time axis in the vocal data.
In addition, in order to identify the vocal frequency transition, the control unit 6 derives the fundamental frequency f0 in each prescribed time window AW (j) in the vocal data. Various known methods can be considered as a method for deriving the fundamental frequency f0. As an example, the control unit 6 performs frequency analysis (for example, DFT) for each specified time window AW (j) set in vocal data, and sets the strongest frequency component as the fundamental frequency f0 as a result of autocorrelation. It is possible.

  And the control part 6 specifies vocal frequency transition by arrange | positioning the fundamental frequency f0 derived | led-out for every these regulation | regulation time window AW (j) along the time axis in vocal data.

  In the data generation process, the control unit 6 subsequently acquires one MIDI music MD associated with the same music ID as the music data WD acquired in S110 (S160). Furthermore, the control unit 6 matches the performance timing of each note constituting the MIDI song MD acquired in S160 (hereinafter referred to as “acquired MIDI”) with the playback time of each sound corresponding to each note of the acquired song data. Then, the acquired MIDI is adjusted (S170). As a method for adjusting the acquired MIDI, it is conceivable to use a known method (for example, the method described in Japanese Patent No. 5310777). In the method described in Japanese Patent No. 5310679, the control unit 6 renders the acquired MIDI, and converts both the rendering result of the acquired MIDI and the master waveform data of the acquired music data into spectral data in a predetermined time unit. And the performance start timing and performance end timing of each performance sound are corrected so that the time on both spectrum data may synchronize. Note that DP matching may be used when adjusting the time on the spectrum data so as to be synchronized.

  Further, in the data generation process, the control unit 6 acquires a melody track representing the singing melody from the MIDI music MD whose time has been adjusted in S170 (S180). In the melody track acquired in S180, each note (hereinafter referred to as "melody note") NO (i) constituting the singing melody is defined. Each melody note NO (i) includes a note number (pitch), performance start timing nnt (i), performance end timing nft (i), various time control information (eg, tempo, resolution, etc.), etc. Note properties are specified. Note that the symbol i is an identifier for identifying a melody note, and is defined so as to increase along the time axis of the singing melody.

  In the data generation process, the control unit 6 generates time-series note data (S190). This time-series note data is obtained by associating note properties relating to each melody note NO (i) with note transitions in which the melody note NO (i) is arranged along the time axis.

  Specifically, in S190 of the present embodiment, first, the control unit 6 generates a note transition in which the melody note NO (i) is arranged along the time axis. The controller 6 sets a specified time window AW (j) for the note transition. The specified time window AW (j) set for the note transition is the same as the specified time window AW (j) set for vocal data. That is, the specified time window AW (j) set for the note transition and vocal data means that the same timing is used if the code j is common.

  Subsequently, the control unit 6 compares the number of specified time windows AW set for note transition with the number of specified time windows AW set for vocal data. As a result of the comparison, the control unit 6 adopts the one having a large number as the number of the specified time windows AW (that is, “jmax”) in both the note transition data and the vocal data. Note that the control unit 6 may perform complementation by adding “0 value” to the smaller number so that the number matches the specified time window AW.

  Further, the control unit 6 corresponds to the performance start timing nnt (i) and performance end timing nft (i) of each melody note NO (i) within the specified time window AW (j) set to note transition. A specified time window AW (j) is specified. Then, the control unit 6 associates a note start flag indicating that it corresponds to the performance start timing nnt (i) with the specified time window AW (j) corresponding to the performance start timing nnt (i). Furthermore, the control unit 6 associates a note end flag indicating that it corresponds to the performance end timing nft (i) with the specified time window AW (j) corresponding to the performance end timing nft (i). At the same time, the control unit 6 adds melody notes corresponding to all the specified time windows AW from the specified time window AW associated with the note start flag to the specified time window AW associated with the note end flag. Time series note data is generated by associating the pitch of NO (i).

  In the data generation process, the control unit 6 displays the vocal sound pressure transition specified in S140, the vocal frequency transition specified in S150, and the time-series note data generated in S190 for each specified time window AW (j). (S200). Below, the data which matched vocal sound pressure transition, vocal frequency transition, and time series note data are also called synchronous data.

  In the data generation process, the control unit 6 executes a sound generation calculation process for calculating “time for sound generation” based on the synchronization data generated in S200 (S210). This “pronunciation time” is one of the characteristics of professional singers. The “pronunciation time” calculated in S210 of this embodiment is the performance start timing nnt (i) of the melody note NO (i) and the professional singer starts to pronounce the melody note NO (i). It is the time difference from the timing to perform.

  Further, in the data generation process, the control unit 6 executes the pitch calculation process to calculate “time for pitch” based on the synchronization data (S220). This “pitch time” is one of the features of professional singers. The “pitch time” calculated in S220 of the present embodiment is the performance start timing nnt (i) of the melody note NO (i) and the pitch uttered by a professional singer. ) Is the time difference from the timing that matches the pitch.

  Furthermore, in the data generation process, the control unit 6 generates singing feature data SF (S230). Specifically, in S230 of the present embodiment, the control unit 6 associates the time for sound generation calculated in S210 with the performance start timing nnt (i) used to calculate the time for sound generation. Further, the control unit 6 associates the pitch time calculated in S220 with the performance start timing nnt (i) used for calculating the pitch time.

In the data generation process, the control unit 6 stores the singing feature data SF generated in S230 in the storage unit 5 (S240).
Thereafter, the control unit 6 ends the data generation process.
<Calculation process for pronunciation>
In the sound generation calculation process executed in S210 of the data generation process, as shown in FIG. 3, the control unit 6 acquires the vocal sound pressure transition specified in the previous S140 (S310). Subsequently, the control unit 6 sets the specified time window AW (j) in the vocal sound pressure transition to an initial value (j = 1) (S320).

  Further, in the calculation process for sound generation, the control unit 6 determines whether or not the sound pressure level Lp (j) of the vocal sound pressure transition in the set specified time window AW (j) is larger than the threshold value Tha. (S330). This threshold value Tha is defined in advance as a sound pressure level that can be regarded as being uttered by a person.

  As a result of the determination in S330, if the sound pressure level Lp (j) of the vocal sound pressure transition in the set specified time window AW (j) is equal to or less than the threshold value Tha (S330: NO), the control unit 6 Shifts the calculation process for sound generation to S350, which will be described in detail later. On the other hand, if the sound pressure level Lp (j) of the vocal sound pressure transition in the specified time window AW (j) is larger than the threshold value Tha (S330: YES), the control unit 6 determines the specified time window set at the present time. A sound generation start flag is set in AW (j) (S340). This sounding start flag represents the timing at which sounding is started in vocal data. The specified time window AW associated with the pronunciation start flag is an example of the utterance timing in the claims.

  Subsequently, the control unit 6 increments the identifier of the set specified time window AW (j) by one (S350). Further, the control unit 6 determines whether or not the identifier of the specified time window AW (j) set at the present time is equal to a value obtained by adding “1” to the number of specified time windows AW (ie, jmax + 1). Is determined (S360). As a result of the determination in S360, if the identifier of the set specified time window AW (j) is smaller than the value obtained by adding “1” to the number of specified time windows AW (S360: NO), the controller 6 Returns the calculation process to S330 for sound generation. And the control part 6 repeats the step from S330 to S360.

  On the other hand, as a result of the determination in S360, if the identifier of the specified time window AW (j) set at the present time matches the value obtained by adding “1” to the number of specified time windows AW (S360: YES) ), The control unit 6 shifts the calculation process for sounding to S370. In S370, the control unit 6 acquires one specified time window AW (j) in the specified time window AW (j) associated with the note start flag. In the present embodiment, when the process proceeds to S370 for the first time, the control unit 6 determines the earliest specified time along the time axis in the specified time window AW (j) associated with the note start flag. A window AW (j) is acquired. Hereinafter, the specified time window AW (j) acquired in S370 is referred to as a note start timing NS.

  Subsequently, the control unit 6 identifies a specified time window AW (j) that satisfies a specified condition among the specified time windows AW (j) in which the sound generation start flag is set (S380). The specified condition here is that the time length from the note start timing NS is within a specified range and the time length from the note start timing NS is the shortest. Note that the specified range in the present embodiment is specified both before and after the time axis with the note start timing NS as a starting point. In this case, the time length after the note start timing NS along the time axis may be defined to be longer than the previous time length along the time axis from the note start timing NS. Further, the time length after the note start timing NS along the time axis may be the same as the time length after the note start timing NS along the time axis. Hereinafter, the specified time window AW (j) specified in S380 is referred to as a sound generation start timing PS.

At the same time, the control unit 6 generates timing pair data of the specified sound generation start timing PS and the note start timing NS (S380).
In the sound generation calculation process, the control unit 6 calculates the time difference between the note start timing NS and the sound generation start timing PS, which constitutes the timing pair data generated in S380, as time for sound generation (S390). In this S390, the number of time windows from the specified time window AW as the note start timing NS to the specified time window AW as the sound generation start timing PS is multiplied by the unit time of the time window, thereby generating the time for sound generation. What is necessary is just to calculate.

  Note that if the sound generation start timing PS cannot be specified in S380, the control unit 6 may set the sound generation start timing PS to be undetermined. In this case, the control unit 6 may set a predetermined undetermined value (for example, “−9.99999” [ms]) as a time for sound generation with respect to the note start timing NS in S390.

  Subsequently, the control unit 6 determines whether or not S370 to S390 are completed for all the specified time windows AW (j) associated with the note start flag (S400). As a result of the determination in S400, if S370 to S390 are not completed for all the specified time windows AW (j) associated with the note start flag (S400: NO), the control unit 6 calculates for sound generation. The process returns to S370. In S370, the control unit 6 determines the next specified rule along the time axis from the previously acquired specified time window AW (j) in the specified time window AW (j) associated with the note start flag. A time window AW (j) is acquired. Thereafter, the control unit 6 executes up to S400.

  On the other hand, as a result of the determination in S400, if S370 to S390 have been completed for all the specified time windows AW (j) associated with the note start flag (S400: YES), the control unit 6 generates a sound. The calculation process proceeds to S410. In S410, the control unit 6 calculates a representative value in the musical piece of time for sound generation (hereinafter referred to as “representative value for sound generation”).

  Specifically, in S410, the control unit 6 calculates, as a representative value for sound generation, an arithmetic average of all sound generation times except for the undetermined value calculated in S390. Note that the method for calculating the representative value for pronunciation is not limited to this. For example, the median value in time for all pronunciations except for the undetermined value may be used as the representative value for pronunciation, or all the pronunciations except for the undetermined value. Therefore, the mode value in time may be used as a representative value for pronunciation.

  Further, in the calculation process for sound generation, the control unit 6 specifies time for special sound generation (S420). Specifically, in S420, the control unit 6 uses the special pronunciation for each of the pronunciation times that deviate from the representative value for the pronunciation by a specified time length or more in all the pronunciation times except for the undetermined value calculated in S390. Therefore, specify as time. In other words, the time for special pronunciation is the time for sound generation with the note NO in which “for” as a singing technique is prominent in the music.

  And the control part 6 sets the flag showing that it is the start timing of time for special pronunciation in the regulation time window AW (j) with which the note start flag in time for the specified special pronunciation was matched.

Thereafter, the control unit 6 ends the calculation process for sound generation and shifts the process to S220 of the data generation process.
In other words, in the calculation process for sound generation, as shown in FIG. 4, the control unit 6 specifies each timing when the sound pressure level Lp (j) of the vocal sound pressure transition becomes larger than the threshold value Tha. From the identified timing, the control unit 6 identifies the sound generation start timing PS that satisfies the specified condition. The specified condition is that the time length from the note start timing (ie, performance start timing nnt) NS is within a specified range and the time length from the note start timing NS is the shortest.

The control unit 6 further specifies timing pair data of the specified sound generation start timing PS and note start timing NS. Then, the control unit 6 calculates a time difference between the note start timing NS and the sound generation start timing PS constituting the timing pair as “time for sound generation”. When “time for pronunciation” is calculated for all the note start timings NS in one piece of music, the control unit 6 remarkably indicates “for” as a singing technique in the music from the calculated “time for pronunciation”. Specify the time for the special pronunciation to appear.
<Calculation processing for pitch>
In the pitch calculation process executed in S220 of the data generation process, as shown in FIG. 5, the control unit 6 acquires the vocal frequency transition specified in the previous S140 (S510). Subsequently, the control unit 6 converts the vocal frequency transition into a vocal pitch transition (S520). In S520, the control unit 6 changes the fundamental frequency f0 of each prescribed time window AW (j) in the vocal frequency transition to the scale pitch (that is, the pitch corresponding to the note number) MS (j). , Convert to vocal pitch transition. Note that the change to the scale pitch MS (j) may be realized by setting the scale pitch closest to the fundamental frequency f0.

  In the calculation process for pitch, the control unit 6 subsequently sets the specified time window AW (j) in the vocal pitch transition to an initial value (j = 1) (S530). Further, the control unit 6 determines whether or not the specified time window AW (j) set at the present time satisfies the setting condition (S540).

  The setting condition in S540 is to satisfy both of the following two conditions. That is, one of the setting conditions is that the sound pressure level Lp (j) in the set specified time window AW (j) is larger than the threshold value Tha. Another setting condition is that in the vocal pitch transition, the scale pitch MS (j) in the specified time window AW (j) and the scale pitch MS (j in the specified time window AW (j-1). The absolute value of the difference from -1) is equal to or greater than a preset threshold value Thb. In other words, in S540, the control unit 6 determines whether or not the singer is uttering in the specified time window AW (j) and the pitch of the uttered singing voice has changed by the set threshold Thb or more.

  As a result of the determination in S540, if the set specified time window AW (j) does not satisfy the setting condition (S540: NO), the control unit 6 performs calculation processing for pitching to S560 described later. Transition. On the other hand, as a result of the determination in S540, if the set specified time window AW (j) satisfies the setting condition (S540: YES), the control unit 6 indicates that the pitch has changed in the vocal data. The pitch change flag is set in the set specified time window AW (j) (S550).

  Subsequently, the control unit 6 increments the identifier of the set specified time window AW (j) by one (S560). Further, the control unit 6 determines whether or not the identifier of the specified time window AW (j) set at the present time is equal to a value obtained by adding “1” to the number of specified time windows AW (ie, jmax + 1). Is determined (S570). As a result of the determination in S570, if the identifier of the set specified time window AW (j) is smaller than the value obtained by adding “1” to the number of specified time windows AW (S570: NO), the controller 6 Returns the calculation process to S540 for the pitch. Then, the control unit 6 repeats S540 to S570.

  On the other hand, as a result of the determination in S570, if the identifier of the set specified time window AW (j) matches the value obtained by adding “1” to the number of specified time windows AW (S570: YES), The controller 6 shifts the calculation process to S580 due to the pitch. In S580, the control unit 6 acquires one specified time window AW (j) in the specified time window AW (j) associated with the note start flag. In the present embodiment, when first shifting to S580, the control unit 6 sets the earliest specified time along the time axis in the specified time window AW (j) associated with the note start flag. A window AW (j) is acquired. Hereinafter, the specified time window AW (j) acquired in S580 is also referred to as a note start timing NS.

  Subsequently, the control unit 6 specifies a specified time window AW (j) that satisfies the second specified condition among the specified time windows AW (j) in which the pitch change flag is set (S590). The second specified condition here refers to a specified time window AW (j) in which the scale pitch MS (j) corresponding to the pitch of the melody note NO (i) corresponding to the note start timing NS is associated. It is to be. Further, the second specified condition here is that the time length from the note start timing NS is within a specified range and the time length from the note start timing NS is the shortest. Hereinafter, the specified time window AW (j) specified in S590 is referred to as pitch change timing CT.

At the same time, the control unit 6 generates timing pair data of the specified pitch change timing CT and the note start timing NS (S590).
In the calculation process for pitch, the control unit 6 calculates the time difference between the note start timing NS and the pitch change timing CT, which constitutes the timing pair data generated in S590, as time for pitch (S600). The calculation of the time for pitch in S600 is performed by adding the unit of the specified time window AW to the number of specified time windows AW from the specified time window AW as the note start timing NS to the specified time window AW as the pitch change timing CT. It can be realized by multiplying time.

  Note that if the pitch change timing CT cannot be specified in S590, the control unit 6 may set the pitch change timing CT to be undetermined. In S600, the control unit 6 may set a predetermined undetermined value (for example, “−9.999” [ms]) as the time for pitch with respect to the note start timing NS in S600.

  Subsequently, the control unit 6 determines whether or not S580 to S600 have been completed for all the specified time windows AW (j) associated with the note start flag (S610). As a result of the determination in S610, if S580 to S600 are not completed for all the specified time windows AW (j) associated with the note start flag (S610: NO), the control unit 6 has a pitch. The calculation process is returned to S580. In S580, the control unit 6 determines the next specified rule along the time axis from the previously acquired specified time window AW (j) in the specified time window AW (j) associated with the note start flag. A time window AW (j) is acquired. Thereafter, the control unit 6 executes up to S610.

  On the other hand, as a result of the determination in S610, if S580 to S600 have been completed for all the specified time windows AW (j) associated with the note start flag (S610: YES), the control unit 6 Therefore, the calculation process is shifted to S620. In S620, the control unit 6 calculates a representative value in the musical piece of time due to pitch (hereinafter referred to as “representative value because of pitch”).

  Specifically, in S620, the control unit 6 calculates, as a representative value for the pitch, the result of arithmetically averaging the time for all the pitches except the undetermined value calculated in S600. Note that the method for calculating the representative value for the pitch is not limited to this. For example, the median value in time for all the pitches except the undetermined value may be used as the representative value for the pitch, or the undetermined value is excluded. A mode value in time for all pitches may be used as a representative value for pitches.

  Further, in the calculation process for pitch, the control unit 6 specifies the time for the special pitch (S630). Specifically, in S630, the control unit 6 determines the pitch times that are deviated from the representative value by a specified time length or more because of the pitch, in all the pitch values except the undetermined value calculated in S600. It is specified as time because it is a special pitch. And the control part 6 sets the flag showing that it is the start timing of time for special pitch to the regulation time window AW (j) with which the note start flag in time for the specified special pitch was matched. .

Then, the control part 6 complete | finishes a calculation process for a pitch, and transfers a process to S230 of a data generation process.
That is, in the calculation process for pitch, as shown in FIG. 6, the control unit 6 specifies the timing satisfying the setting condition as the pitch change timing CT in the vocal pitch transition. The control unit 6 further specifies a timing pair of the specified pitch change timing CT and the note start timing NS. Then, the control unit 6 calculates a time difference between the note start timing NS and the pitch change timing CT constituting the timing pair as “time for pitch”. When “time for pitch” is calculated for all the note start timings NS in one musical piece, the control unit 6 determines the time difference from the calculated “time for pitch” until the pitches match. The time is specified because of the special pitch that appears prominently.

  In S230 of the data generation process that is executed after the calculation process for pitch is completed, as shown in FIG. 7, the control unit 6 uses the time for sound generation and the performance start timing nnt used to calculate the time for sound generation as shown in FIG. (I) As a result, it is associated with the melody note NO (i). The control unit 6 associates the time for the pitch with the performance start timing nnt (i) used for calculating the time for the pitch, and consequently the melody note NO (i).

  However, when the time for sound generation and the time for pitch calculated in the data generation process are undetermined values, the control unit 6 sets the undetermined value as the time for sound generation and the time for pitches as a melody note NO. It may be associated with (i).

The singing feature data SF generated by the control unit 6 of the information processing device 2 executing the data generation process may be stored in the storage unit 14 of the information processing server 10 using a portable storage medium. When the information processing device 2 and the information processing server 10 are connected via a communication network, the singing feature data SF stored in the storage unit 5 of the information processing device 2 is transferred via the communication network. Thus, the information may be stored in the storage unit 14 of the information processing server 10. However, in any case, the singing feature data SF is stored in the storage unit 14 of the information processing server 10 in association with the MIDI music MD of the corresponding music.
<Display processing>
The display process executed by the control unit 50 of the karaoke apparatus 30 is started when a command for starting a processing program for executing the display process is input.

  In the display process, when activated, as shown in FIG. 8, the control unit 50 stores the MIDI music MD corresponding to the music specified via the input receiving unit 34 into the storage unit 14 of the information processing server 10. (S910). Subsequently, the control unit 50 acquires lyrics data corresponding to the MIDI music MD acquired in S910 (S920). Furthermore, the control unit 50 acquires singing feature data SF corresponding to the MIDI music MD acquired in S910 (S930).

  In the display process, the control unit 50 subsequently plays the MIDI musical piece MD acquired in S910 (S940). Specifically, in S940, the control unit 50 outputs the MIDI music piece MD to the music reproduction unit 36. The music reproducing unit 36 that has acquired the MIDI music MD performs the music. Then, the sound source signal of the music played by the music playback unit 36 is output to the speaker 60 via the output unit 42. Then, the speaker 60 outputs the sound source signal instead of sound.

  Further, in the display process, the control unit 50 performs display based on the MIDI music MD acquired in S910, the lyrics data acquired in S920, and the singing feature data SF acquired in S930 (S950). In S950, specifically, the control unit 50 outputs the melody track, the lyric data, and the singing feature data SF in the MIDI music piece MD to the video control unit 46. The video control unit 46 sequentially outputs the lyrics data and the melody note NO (i) constituting the melody track to the display unit 64 along the time axis. Then, the display unit 64 sequentially displays the lyrics data and the melody note NO (i) constituting the melody track along the time axis as shown in FIG. Further, the video control unit 46 sequentially outputs the time for sound generation and the time for pitch included in the singing feature data SF to the display unit 64 along the time axis. Then, the display unit 64 sequentially displays the time for sound generation and the time for sound pitch in association with the performance start timing nnt (i) of the corresponding melody note NO (i) along the time axis.

  In S950, the control unit 50 may output the transition of the pitch of the singing voice sung by the user of the karaoke apparatus 30 to the video control unit 46 in accordance with the performance of the music. Then, the video control unit 46 changes the pitch of the singing voice sung by the user of the karaoke apparatus 30 (for example, the solid line waveform shown in FIG. 9) from the deviation from the melody note NO (i). May be displayed in a recognizable manner. Furthermore, in S950, the control unit 50 may output the pitch transition of vocal data sung by a professional singer to the video control unit 46. And the video control part 46 may display the pitch transition (namely, vocal frequency transition, the waveform of the broken line shown in FIG. 9) of the vocal data which the professional singer sang.

  Further, in S950, the control unit 50 may output to the video control unit 46 the result of classifying the time for sound generation and the time for pitches into predetermined classification ranges. That is, for example, if it is time for sound generation, the control unit 50 classifies it as “small for sound generation” indicating that the time for sounding is short until the first time (for example, 200 [ms]). Then, the control unit 50 classifies the sound for sound generation from the first time to the second time (for example, 400 [ms]), and classifies it as “medium for sound generation”. Classify as “Large for Pronunciation” Then, the control unit 50 may output the classification result to the video control unit 46 as a time for pronunciation. In this case, the video control unit 46 causes the display unit 64 to display the classification result.

  Further, if the time is due to the pitch, the control unit 50 classifies, for example, “small due to the pitch” indicating that the time is short until the first time. Then, the control unit 50 classifies as “medium for pitch” if the time for pitch is from the first time to the second time, and sets it as “large for pitch” if it is over the second time. Classify. Then, the control unit 50 may output the classification result to the video control unit 46 as a time for pitch. In this case, the video control unit 46 causes the display unit 64 to display the classification result.

  Furthermore, the control unit 50 may output only the time for sound generation and the time for pitch to the video control unit 46 for time for special sounding and time for special pitch. That is, in S950, only the time for special pronunciation and the time for special pitch among the time for pronunciation and the time for pitch may be presented on the display unit 64 in association with the melody note NO.

Subsequently, in the display process, the control unit 50 determines whether or not the music performance has ended (S960). If the result of this determination is that the music performance has not ended (S960: NO), the control unit 50 returns the display processing to S940. On the other hand, if the result of determination in S960 is that the music performance has ended (S960: YES), the control unit 50 ends the display process.
[Effect of the embodiment]
As described above, the singing feature data SF of the present embodiment associates “time for sound generation” and “time for pitch” with the melody note NO (i).

Among these, “time for pronunciation” and “time for pitch” represent the characteristics of “for” as a singing technique used by professional singers when singing music.
The time for sounding is the time difference between the performance start timing nnt (i) of the melody note NO (i) and the utterance start timing at which the singer actually started uttering in the music data WD. Also, the time for pitch is the time between the pitch of the melody note NO (i) and the pitch of the singing voice sung by the singer, and the performance start timing nnt (i) of the melody note NO (i). It is a time difference.

  In the karaoke apparatus 30, “time for pronunciation” and “time for pitch” are displayed on the display unit 64 in association with the performance start timing nnt of the melody note NO sequentially displayed along the time axis.

  Therefore, according to the karaoke apparatus 30, the location where the professional singer uses “for” in the music can be presented via the display unit 64. Further, according to the karaoke apparatus 30, the timing at which a professional singer matches the pitch of the melody note NO (i) with the pitch of the singing voice from the performance start timing nnt (i) of the melody note NO (i). It can be shown how much it is shifted.

  Moreover, according to the karaoke apparatus 30, “time for sound generation” and “time for pitch” in the melody note NO (i) are sequentially displayed on the display unit 64 along the time axis. Therefore, according to the karaoke apparatus 30, the “time for sound generation” and the “time for pitch” in the melody note NO (i) can be presented before the user of the karaoke apparatus 30 actually sings.

  For these reasons, the user of the karaoke apparatus 30 whose time is presented on the display unit 64 recognizes how the professional singer uses “for” as a singing technique in the song being sung. it can. As a result, the user of the karaoke apparatus 30 can bring the user's own way of singing closer to that of a professional singer.

  Further, the “time for special pronunciation” specified in the data generation process is “time for sound generation” in the melody note NO in which “for” as a singing technique appears remarkably in music. In addition, the “time for special pitch time” is a time difference in the melody note NO in which the pitch of the pitch in the singing voice sung by a professional singer is more pronounced than the pitch of the melody note NO (i). It is.

The singing feature data SF of the present embodiment includes “time for special pronunciation” and “time for special pitch”.
For this reason, according to the karaoke apparatus 30, the location which a professional singer uses "for" notably in a music can be displayed via the display part 64. FIG. In addition, according to the karaoke apparatus 30, it is possible to display, via the display unit 64, a portion where a shift in timing for matching the pitches of musical notes with the pitches of singing voices sung by a professional singer is significant.

Therefore, the user of the karaoke apparatus 30 has a noticeable shift in the pitch at which the professional singer sings “for” and the pitch of the singing voice sung by the professional singer to the pitch of the melody note NO (i). Even a simple part can bring the user's own way of singing closer to that of a professional singer.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, the data generation process in the above embodiment is executed by the information processing apparatus 2, but the apparatus that executes the data generation process in the present invention is not limited to the information processing apparatus 2. That is, the apparatus that executes the data generation process may be the information processing server 10 or the karaoke apparatus 30. In this case, the information processing apparatus 2 may be omitted from the system 1.

  The display process in the above embodiment is executed by the karaoke apparatus 30, but the apparatus that executes the display process in the present invention is not limited to the karaoke apparatus 30, and may be the information processing apparatus 2. In this case, the information processing device 2 needs to be configured similarly to the karaoke device 30.

  In the above embodiment, the data generation process and the display process are configured as separate processes. However, in the present invention, the data generation process and the display process may be configured as a single process. In this case, one process including the data generation process and the display process may be executed by the information processing apparatus 2 or may be executed by the karaoke apparatus 30.

  In the data generation process of the above-described embodiment, two processes of the calculation process for sound generation and the calculation process for pitch are executed. However, in the present invention, the calculation process for sound generation and the calculation process for pitch are performed. As long as at least one of them is executed.

  In addition, the aspect which abbreviate | omitted a part of structure of the said embodiment as long as the subject could be solved is also embodiment of this invention. Further, an aspect configured by appropriately combining the above embodiment and the modification is also an embodiment of the present invention. Moreover, all the aspects which can be considered in the limit which does not deviate from the essence of the invention specified by the wording described in the claims are the embodiments of the present invention.

  DESCRIPTION OF SYMBOLS 1 ... System 2 ... Information processing apparatus 3 ... Input reception part 4 ... Information output part 5 ... Memory | storage part 6, 16, 50 ... Control part 7, 18, 52 ... ROM 8, 20, 54 ... RAM 9, 22, 56 ... CPU 10 ... Information processing server 12 ... Communication unit 14 ... Storage unit 30 ... Karaoke device 32 ... Communication unit 34 ... Input reception unit 36 ... Music playback unit 38 ... Storage unit 40 ... Voice control unit 42 ... Output unit 44 ... Microphone input unit 46 ... Video control unit 60 ... Speaker 62 ... Microphone 64 ... Display unit

Claims (8)

1st acquisition which acquires the score data which represents the score of the music comprised by several notes, Comprising: The musical score data with which the pitch and the performance period were matched with each of these several notes from the 1st memory | storage part Steps,
A second acquisition step of acquiring music data including a vocal sound of singing the music from the second storage unit;
An extraction step of extracting vocal data representing the vocal sound from the music data;
A timing specifying step of specifying each utterance start timing at which the sound pressure in the vocal data is equal to or higher than a predetermined threshold as an utterance timing that can be regarded as a timing at which utterance is started in the vocal data;
Based on each of the utterance timings identified in the timing identification step and the performance start timing of the notes in the score data acquired in the first acquisition step, the performance start timing of the notes in the score data and the performance start timing A first generation step of generating timing pair data combining the utterance start timing with the shortest time length ;
Calculating a representative value for pronunciation, which is a representative value in the music for the time of pronunciation, which is a time difference between the performance start timing and the utterance start timing in the timing pair data generated in the first generation step, If, on the basis of the representative values for sound, and time calculating step for calculating the sound for the representative value specified time or length deviating the pronunciation for special sound because time is time, since the time,
Therefore, singing feature data representing the characteristics of the singer's singing method is generated by associating the time thus calculated in the time calculating step with the note of the score data corresponding to the performance start timing used for calculating the time. A program for causing a computer to execute the second generation step. 1st acquisition which acquires the score data which represents the score of the music comprised by several notes, Comprising: The musical score data with which the pitch and the performance period were matched with each of these several notes from the 1st memory | storage part Steps,
A second acquisition step of acquiring music data including a vocal sound of singing the music from the second storage unit;
An extraction step of extracting vocal data representing the vocal sound from the music data;
The pitch change timing, which is the timing at which the vocal frequency transition representing the transition of the pitch in the vocal data falls within the pitch range of the note in the score data, can be regarded as the timing at which the vocal data is started. A timing specifying step for specifying the voice timing;
Based on each of the utterance timings identified in the timing identification step and the performance start timing of the notes in the score data acquired in the first acquisition step, the performance start timing of the notes in the score data and the performance start timing A first generation step of generating timing pair data in which the pitch change timing is associated with the pitch change timing that is within a specified time range that is specified in advance along the time axis and that has the shortest time length from the performance start timing. When,
Calculating a representative value for a pitch, which is a representative value in the music for a pitch, which is a time difference between the performance start timing and the pitch change timing in the timing pair data generated in the first generation step, To calculate the time for the special pitch, which is the time for the pitch that is deviated from the representative value for the pitch by a specified time length or more, based on the time for the pitch and the representative value for the pitch. A time calculation step;
Therefore, singing feature data representing the characteristics of the singer's singing method is generated by associating the time thus calculated in the time calculating step with the note of the score data corresponding to the performance start timing used for calculating the time. A second generation step and
A program that causes a computer to execute. The timing specifying step specifies, as the utterance timing, each utterance start timing at which a sound pressure in the vocal data is equal to or higher than a predetermined threshold value,
The first generation step generates the timing pair data combining the performance start timing of the notes in the musical score data and the utterance start timing with the shortest time length from the performance start timing,
Therefore, the time calculating step calculates the time for sound generation as the time for the sound generation, which is a time difference between the performance start timing and the utterance start timing in the timing pair data generated in the first generation step. The program according to claim 2. Therefore, the time calculation step includes
Calculating a representative value for pronunciation, which is a representative value in the song for the pronunciation time,
  Based on the time for sound generation and the representative value for sound generation, the time for special sound generation, which is the time for sound generation that deviates from the representative value for sound generation by a specified time length or more, is calculated as the time for the purpose. The program according to claim 3. 1st acquisition which acquires the score data which represents the score of the music comprised by several notes, Comprising: The musical score data with which the pitch and the performance period were matched with each of these several notes from the 1st memory | storage part Means,
Second acquisition means for acquiring music data including the vocal sound of singing the music from the second storage unit;
Extraction means for extracting vocal data representing the vocal sound from the music data;
Timing specifying means for specifying each utterance start timing at which the sound pressure in the vocal data is equal to or greater than a predetermined threshold value as an utterance timing that can be regarded as a timing at which utterance is started in the vocal data;
Based on each of the utterance timings specified by the timing specifying means and the performance start timing of the notes in the score data acquired by the first acquisition means, from the performance start timing of the notes in the score data and the performance start timing First generation means for generating timing pair data combining the utterance start timing with the shortest time length;
Calculating a representative value for pronunciation, which is a representative value in the music for the time of pronunciation, which is a time difference between the performance start timing and the utterance start timing in the timing pair data generated by the first generation means, And, based on the representative value for sound generation, a time calculating means for calculating a time for special sound generation, which is a time for the sound generation that deviates from the representative value for sound generation by a specified time length or more, and as a time,
Therefore, singing feature data representing the characteristics of the singer's singing method is generated by associating the time thus calculated by the time calculating means with the note of the score data corresponding to the performance start timing used for calculating the time. With second generation means
An information processing apparatus comprising: 1st acquisition which acquires the score data which represents the score of the music comprised by several notes, Comprising: The musical score data with which the pitch and the performance period were matched with each of these several notes from the 1st memory | storage part Means,
Second acquisition means for acquiring music data including the vocal sound of singing the music from the second storage unit;
Extraction means for extracting vocal data representing the vocal sound from the music data;
A timing specifying means for specifying a pitch change timing, which is a timing at which a vocal frequency transition representing a transition of a pitch in the vocal data is within a range of the pitch of the note in the score data, as an utterance timing;
Based on each of the utterance timings specified by the timing specifying means and the performance start timing of the notes in the score data acquired by the first acquisition means, from the performance start timing of the notes in the score data and the performance start timing First generation means for generating timing pair data that is associated with the pitch change timing that is within a specified time range that is specified in advance along the time axis and that has the shortest time length from the performance start timing. When,
Calculating a representative value for a pitch, which is a representative value in the music for a pitch, which is a time difference between the performance start timing and the pitch change timing in the timing pair data generated by the first generator, To calculate the time for the special pitch, which is the time for the pitch that is deviated from the representative value for the pitch by a specified time length or more, based on the time for the pitch and the representative value for the pitch. Time calculation means;
Therefore, singing feature data representing the characteristics of the singer's singing method is generated by associating the time thus calculated by the time calculating means with the note of the score data corresponding to the performance start timing used for calculating the time. With second generation means
An information processing apparatus comprising: The information processing apparatus acquires, from the first storage unit, musical score data representing a musical score of a music composed of a plurality of notes, each of which is associated with a pitch and a performance period. A first acquisition procedure,
A second acquisition procedure in which the information processing apparatus acquires from the second storage unit music data including a vocal sound that sang the music;
An extraction procedure by which the information processing apparatus extracts vocal data representing the vocal sound from the music data;
A timing specifying procedure for the information processing apparatus to specify each utterance start timing at which the sound pressure in the vocal data is equal to or higher than a predetermined threshold value as utterance timing that can be regarded as a timing at which utterance is started in the vocal data;
Based on each of the utterance timings specified in the timing specification procedure and the performance start timing of the notes in the score data acquired in the first acquisition procedure, from the performance start timing of the notes in the score data and the performance start timing A first generation procedure in which the information processing apparatus generates timing pair data that combines the utterance start timing with the shortest time length of
Calculating a representative value for pronunciation, which is a representative value of the musical composition for the time of pronunciation, which is a time difference between the performance start timing and the utterance start timing in the timing pair data generated in the first generation procedure; And a time calculation procedure for the information processing device to calculate the time for special pronunciation, which is the time for sound generation, which is deviated from the representative value for sound generation by a specified time length or more based on the representative value for sound generation. When,
Therefore, the singing feature data representing the singing feature of the singer by associating with the notes of the score data corresponding to the performance start timing used for the calculation of the time for the time calculated in the time calculation procedure, A second generation procedure generated by the information processing apparatus;
A data generation method comprising: The information processing apparatus acquires, from the first storage unit, musical score data representing a musical score of a music composed of a plurality of notes, each of which is associated with a pitch and a performance period. A first acquisition procedure,
A second acquisition procedure in which the information processing apparatus acquires from the second storage unit music data including a vocal sound that sang the music;
An extraction procedure by which the information processing apparatus extracts vocal data representing the vocal sound from the music data;
The pitch change timing, which is the timing at which the vocal frequency transition representing the transition of the pitch in the vocal data falls within the pitch range of the note in the score data, can be regarded as the timing at which the vocal data is started. A timing specifying procedure specified by the information processing apparatus as the utterance timing;
Based on each of the utterance timings identified in the timing identification procedure and the performance start timing of the notes in the score data acquired in the first acquisition procedure, the performance start timing of the notes in the score data , and the performance start timing, The information processing apparatus generates timing pair data that is associated with the pitch change timing that is within a specified time range that is specified in advance along the time axis and that has the shortest time length from the performance start timing. A first generation procedure to:
Calculating a representative value for a pitch, which is a representative value in the music for a pitch, which is a time difference between the performance start timing and the pitch change timing in the timing pair data generated in the first generation procedure, Based on the time for pitch and the representative value for pitch, the information processing as the time for the special pitch, which is the time for the pitch that is deviated from the representative value for the pitch by a specified time length or more, as the time. A time calculation procedure for the device to calculate,
Therefore, the singing feature data representing the singing feature of the singer by associating with the notes of the score data corresponding to the performance start timing used for the calculation of the time for the time calculated in the time calculation procedure, A data generation method comprising: a second generation procedure generated by the information processing apparatus.