JP2015191177A - Program, information processing device, and data generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of generating data necessary for indicating features of a way of singing specific to a singer.SOLUTION: The information processing device executes: first and second acquisition steps for acquiring musical score data and music data respectively; a pitch transition derivation step for deriving a vocal pitch transition; a specification step for specifying a vibrato waveform on the basis of the vocal pitch transition; a reference calculation step for calculating a high tone reference pitch and a low tone reference pitch on the basis of a pitch at the top of the vibrato waveform; a determination step for determining at least either of a center pitch difference based on the musical score data, the high tone reference pitch, and the low tone reference pitch, or a vibrato depth based on the high tone reference pitch and the low tone reference pitch; and a data generation step for generating singing feature data by associating at least either of the center pitch difference or the vibrato depth with a performance period in the musical score data corresponding to the vibrato waveform.

Description

  The present invention relates to a program for generating data, an information processing apparatus, and a data generation method.

Conventionally, a song evaluation technique for evaluating the skill of a song that sang the song melody has been known (see Patent Document 1).
In the technique described in Patent Document 1, skill is evaluated by comparing the voice sung by the user with the model voice prepared for each piece of music, and a comment corresponding to the evaluation is presented after singing.

JP 2007-233303 A

  By the way, in the music sung by a professional singer, in many cases, a unique characteristic appears for each singer of the music. When a user sings using a karaoke device, an information processing device, or the like, if the singing-specific characteristics of the singer can be reproduced, it sounds like singing better.

For this reason, in a karaoke apparatus, an information processing apparatus, etc., about the characteristic of the way of singing peculiar to a singer, it is calculated | required before a user actually sings the characteristic of the way of singing.
However, in the conventional technology, the difference between the model voice and the voice sung by the user is only presented after the user sings. Therefore, users such as karaoke apparatuses and information processing apparatuses have a problem that it is difficult to recognize before singing at what timing and how to sing about the singing-specific characteristics of the singer.

That is, in a karaoke apparatus, an information processing apparatus, etc., there was a problem that the user did not present anything about the characteristics of the singing method specific to the singer before the user actually sang.
Therefore, an object of the present invention is to provide a technique capable of generating data necessary for presenting the characteristics of a singer-specific singing method.

The present invention made to achieve the above object relates to a program to be executed by a computer.
In the program according to the present invention, the computer executes the first acquisition step, the second acquisition step, the pitch transition determination step, the specifying step, the reference calculation step, the determination step, and the data generation step.

  Among these, in the first acquisition step, the score data representing the score of a music composed of a plurality of notes, the score data in which the pitch and the performance period are associated with each of the plurality of notes, Obtained from the storage unit. In the second acquisition step, music data including the vocal sound that sang the music is acquired from the second storage unit. In the extraction step, vocal data representing vocal sound is extracted from the music data.

  Further, in the pitch transition determining step, a vocal pitch transition representing transition of pitch in the vocal data is determined based on the vocal data. In the specifying step, based on the vocal pitch transition, a vibrato waveform that is a singing section sung using vibrato in the vocal pitch transition is specified. In the reference calculation step, based on the peak pitch of the vibrato waveform, a treble reference pitch that represents a high pitch in the vibrato waveform and a low reference pitch that represents a low pitch in the vibrato waveform are calculated. .

  In the determination step, at least one of the central pitch difference and the vibrato depth is determined based on the high pitch reference pitch and the low pitch reference pitch. The central pitch difference here is the difference between the central pitch in the vibrato waveform and the pitch of the note corresponding to the singing section in the score data. The vibrato depth referred to here represents a fluctuation width along the frequency axis in the vibrato waveform.

  In the data generation step, singing feature data is generated by associating at least one of the central pitch difference and the vibrato depth with the section of the score data corresponding to the singing section.

  The singing feature data generated by such a program includes at least one of “center pitch difference” and “vibrato depth”, which are features related to vibrato used by the singer who sang the music, in the score data. It is associated with a note. Here, the “center pitch difference” and the “vibrato depth” are information representing the characteristics relating to the vibrato used by the singer, and are unique to each singer.

  According to such singing characteristic data, when displaying the notes constituting the music based on the score data, that is, the singing melody, the vibrato used by the singer is displayed on the display unit connected to the karaoke device or the information processing device. “Center pitch difference” and “Vibrato depth” can be presented. In addition, the “center pitch difference” and “vibrato depth” can be presented before the user of the karaoke apparatus or the information processing apparatus actually sings.

  And the user of the karaoke device or information processing device in which the “center pitch difference” and “vibrato depth” of the vibrato are presented on the display device, the professional singer is the singer Recognize how "vibrato" is used. As a result, the user of the karaoke apparatus or the information processing apparatus can bring the user's own way of singing closer to that of a professional singer.

  In the determination step according to the present invention, an average value of the high pitch reference pitch and the low pitch reference pitch is determined as the central pitch in the vibrato waveform, and the determined central pitch and the note corresponding to the singing section in the score data are determined. The difference from the pitch may be determined as the central pitch difference.

  According to such a program, the average value of the high pitch reference pitch and the low pitch reference pitch can be calculated as the central pitch in the vibrato waveform. According to the program of the present invention, the difference between the central pitch and the pitch of the note in the vibrato waveform can be calculated as the central pitch difference.

Furthermore, in the determination step according to the present invention, the difference between the high pitch reference pitch and the low pitch reference pitch may be calculated as the vibrato depth.
According to such a program, the difference between the high pitch reference pitch and the low pitch reference pitch can be calculated as the vibrato depth.

  In the reference calculation step according to the present invention, the vertex detection step, the treble reference calculation step, and the bass reference calculation step may be executed by a computer. In the vertex detection step, an inflection point in the vibrato waveform identified in the identification step is detected. In the treble reference calculation step, a representative value of the pitch at the maximum inflection point among the inflection points of the vibrato waveform detected in the vertex detection step is calculated as the treble reference pitch. Further, in the bass reference calculation step, the representative value of the pitch at the minimum inflection point among the inflection points of the vibrato waveform detected in the vertex detection step is calculated as the bass reference pitch.

According to such a program, the treble reference pitch and the bass reference pitch can be calculated more reliably.
Further, in the reference calculation step of the present invention, the treble reference pitch and the bass reference pitch are calculated by using the same number of inflection points as the minimum in the vibrato waveform and the maximum inflection points in the vibrato waveform. You may do it.

According to such a program, the singular point at the inflection point of the vibrato waveform can be excluded, and the calculation accuracy of the high pitch reference pitch and the low pitch reference pitch can be improved.
In the program of the present invention, the display control step may be executed by a computer. In this case, in the display control step, a plurality of notes constituting the musical score data acquired in the first acquisition step are displayed on the display unit, and at least of the central pitch difference included in the singing feature data and the vibrato depth Either one is displayed on the display unit in association with the section in the musical score data corresponding to the singing section.

According to such a program, the “center pitch difference” and “vibrato depth” of the vibrato used by the singer can be presented on the display unit connected to the karaoke apparatus or the information processing apparatus.
As a result, users of karaoke devices and information processing devices in which the “center pitch difference” and “vibrato depth” of vibrato are presented on the display device can be used by professional singers as singing skills in singing songs. It is possible to recognize how “Vibrato” is used.

The present invention may be implemented as an information processing apparatus.
The information processing apparatus of the present invention includes a first acquisition unit, a second acquisition unit, an extraction unit, a pitch transition determination unit, a specification unit, a reference calculation unit, a determination unit, and a data generation unit. .

  The first acquisition unit acquires the musical score data from the first storage unit, and the second acquisition unit acquires the music piece data from the second storage unit. The extracting means extracts vocal data, and the pitch transition determining means determines a vocal pitch transition representing a transition of pitch in the vocal data. The specifying means specifies a vibrato waveform that is a singing section sung using vibrato in the vocal pitch transition. The reference calculation means calculates a high pitch reference pitch and a low pitch reference pitch based on the peak pitch of the vibrato waveform.

  Further, the determining means determines at least one of the central pitch difference and the vibrato depth based on the high pitch reference pitch and the low pitch reference pitch. Further, the data generating means generates singing feature data by associating at least one of a central pitch difference and a vibrato depth with a section of the musical score data corresponding to the singing section.

According to such an information processing apparatus, an effect similar to that of the program according to claim 1 can be obtained.
The present invention may be implemented as a data generation method executed by the information processing apparatus.

The data generation method of the present invention includes a first acquisition procedure, a second acquisition procedure, an extraction procedure, a pitch transition determination procedure, a specific procedure, a reference calculation procedure, a determination procedure, and a data generation procedure. .
In the first acquisition procedure, the information processing apparatus acquires score data from the first storage unit, and in the second acquisition procedure, the information processing apparatus acquires song data from the second storage unit. In the extraction procedure, the information processing device extracts vocal data, and in the pitch transition determination procedure, the information processing device determines the vocal pitch transition based on the extracted vocal data. In the specifying procedure, the information processing apparatus specifies a vibrato waveform based on the determined vocal pitch transition.

  In the reference calculation procedure, the information processing apparatus calculates the high pitch reference pitch and the low pitch reference pitch based on the pitch of the apex in the identified vibrato waveform. In the determination procedure, the information processing apparatus determines at least one of the central pitch difference and the vibrato depth. In the data generation procedure, the information processing apparatus generates singing feature data by associating at least one of the central pitch difference and the vibrato depth with the section of the musical score data corresponding to the singing section.

  According to such a data generation method, an effect similar to that of the program according to claim 1 can be obtained.

It is a block diagram which shows schematic structure of the system provided with the information processing apparatus with which this invention was applied. It is a flowchart which shows the process sequence of the data generation process which information processing apparatus performs. It is a flowchart which shows the process sequence of a vibrato feature calculation process. It is a figure which illustrates the outline | summary of the calculation method of a vibrato feature. It is a figure which illustrates the outline | summary of singing characteristic data. It is a flowchart which shows the process sequence of the display process which a karaoke apparatus performs. It is a figure which illustrates the aspect of the display displayed on a display part by performing a display process.

Embodiments of the present invention will be described below with reference to the drawings.
<System configuration>
The karaoke device 30 shown in FIG. 1 is a device that displays on the display unit 64 the characteristics of how to sing a professional singer expressed in the music while playing the music specified by the user.

  The system 1 constructed in order for the karaoke device 30 to display such characteristics of how to sing a professional singer includes the information processing device 2, the information processing server 10, and the karaoke device 30.

  The information processing device 2 calculates the singing feature data SF based on the music data WD and the MIDI music MD prepared for each music. The singing feature data SF referred to here is data representing features in how to sing a professional singer who sings music.

In the information processing server 10, at least the singing feature data SF and the MIDI music MD calculated by the information processing device 2 are stored in the storage unit 14 in association with each corresponding music.
<Music data>
Next, the music data WD is prepared in advance for each specific music, and includes music management information in which information related to the music is described, and master waveform data representing the performance sound of the music. The music management information includes music identification information (hereinafter referred to as music ID) for identifying music.

  The master waveform data of this embodiment is audio data including performance sounds of a plurality of musical instruments and vocal sounds sung by a professional singer. The audio data may be data constituted by an audio file in an uncompressed audio file format, or data constituted by an audio file in an audio compression format.

  In the following, voice waveform data representing the performance sound of the musical instrument included in the master waveform data is referred to as accompaniment data, and voice waveform data representing the vocal sound included in the master waveform data is referred to as vocal data.

Musical instrument performance sounds included in the accompaniment data of the present embodiment include percussion instrument (eg, drum, drum, cymbal, etc.) performance sounds, stringed instrument (eg, guitar, bass, etc.) performance sounds, percussion instrument (eg, piano) ) And wind instruments (eg, trumpet, clarinet, etc.).
<MIDI music>
The MIDI music MD is prepared in advance for each music and has performance data and lyrics data.

  Of these, the performance data is data representing the score of one piece of music according to the well-known MIDI (Musical Instrument Digital Interface) standard. This performance data has at least a music ID and a music score track representing a music score for each musical instrument used in the music.

  The musical score track defines at least the pitch (so-called note number) and the period during which the MIDI sound source outputs the performance sound (hereinafter referred to as the note length) for each performance sound output from the MIDI sound source. Has been. The note length in the score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the performance sound and the music until the output of the performance sound ends. Performance end timing (so-called note-off timing) representing the time from the start of the performance.

  That is, in the score track, one note NO is defined by the note number and the note length represented by the note-on timing and note-off timing. The musical score track functions as one musical score by arranging note NO in the order of performance. Note that the musical score track is prepared for each instrument such as a keyboard instrument, a stringed instrument, a percussion instrument, and a wind instrument, for example. Among these, in this embodiment, a specific musical instrument (for example, vibraphone) is defined as a musical instrument responsible for singing melody in music.

  On the other hand, the lyrics data is data relating to the lyrics of the music, and includes lyrics telop data and lyrics output data. The lyrics telop data represents characters that constitute the lyrics of the music (hereinafter referred to as lyrics component characters). The lyrics output data is data in which a timing correspondence relationship that associates the lyrics output timing, which is the output timing of the lyrics constituent characters, with the performance of the performance data is defined.

Specifically, in the timing correspondence relationship in the present embodiment, the timing for starting the output of the lyrics telop data is associated with the timing for starting the performance of the performance data. Furthermore, in the timing correspondence relationship, the lyrics output timing of each lyrics constituent character along the time axis of the music is defined by the elapsed time from the performance start of the performance data. Thereby, each performance sound (namely, note NO) prescribed | regulated to the score track | truck and each lyric component character are matched.
<Information processing device>
The information processing apparatus 2 is a known information processing apparatus (for example, a personal computer) including an input receiving unit 3, an information output unit 4, a storage unit 5, and a control unit 6.

  The input receiving unit 3 is an input device that receives input of information and commands from the outside. The input device here is, for example, a key or switch, a reading drive for reading data stored in a portable storage medium (for example, CD, DVD, flash memory), or a communication port for acquiring information via a communication network. Etc. The information output unit 4 is an output device that outputs information to the outside. Here, the output device is a writing drive that writes data to a portable storage medium, a communication port that outputs information to a communication network, or the like.

  The storage unit 5 is a known storage device configured to be able to read and write stored contents. The storage unit 5 stores at least one piece of music data WD and at least one MIDI piece of music MD in association with each common piece of music.

  The control unit 6 is a known control device that is configured around a known microcomputer including a ROM 7, a RAM 8, and a CPU 9. The ROM 7 stores processing programs and data that need to retain stored contents even when the power is turned off. The RAM 8 temporarily stores processing programs and data. The CPU 9 executes each process according to a processing program stored in the ROM 7 or RAM 8.

The ROM 7 of the present embodiment has a processing program for the control unit 6 to execute a data generation process for calculating the song feature data SF based on the music data WD and the MIDI music MD stored in the storage unit 5. It is remembered.
<Information processing server>
The information processing server 10 includes a communication unit 12, a storage unit 14, and a control unit 16.

  Among these, the communication unit 12 performs communication between the information processing server 10 and the outside via a communication network. That is, the information processing server 10 is connected to the karaoke apparatus 30 via a communication network. The communication network referred to here may be a wired communication network or a wireless communication network.

  The storage unit 14 is a known storage device configured to be able to read and write stored contents. The storage unit 14 stores at least a plurality of MIDI music pieces MD. 1 is an identifier for identifying the MIDI music piece MD stored in the storage unit 14 of the information processing server 10, and is a natural number of 1 or more. Furthermore, the storage unit 14 stores singing feature data SF generated by the information processing apparatus 2 executing data generation processing.

The control unit 16 is a known control device that is configured around a known microcomputer including a ROM 18, a RAM 20, and a CPU 22. The ROM 18, RAM 20, and CPU 22 are configured similarly to the ROM 7, RAM 8, and CPU 9, respectively.
<Karaoke equipment>
The karaoke apparatus 30 includes a communication unit 32, an input reception unit 34, a music playback unit 36, a storage unit 38, an audio control unit 40, a video control unit 46, and a control unit 50.

  In the communication unit 32, the karaoke apparatus 30 communicates with the outside via a communication network. The input receiving unit 34 is an input device that receives input of information and commands in accordance with external operations. Here, the input device is, for example, a key, a switch, a reception unit of a remote controller, or the like.

  The music playback unit 36 performs a music performance based on the MIDI music MD downloaded from the information processing server 10. The music reproducing unit 36 is, for example, a MIDI sound source. The voice control unit 40 is a device that controls voice input / output, and includes an output unit 42 and a microphone input unit 44.

  A microphone 62 is connected to the microphone input unit 44. Thereby, the microphone input part 44 acquires a user's song sound. A speaker 60 is connected to the output unit 42. The output unit 42 outputs the sound source signal of the music reproduced by the music reproducing unit 36 and the sound source signal of the singing sound from the microphone input unit 44 to the speaker 60. The speaker 60 outputs the sound source signal output from the output unit 42 instead of sound.

  The video control unit 46 outputs a video or an image based on the video data sent from the control unit 50. The video control unit 46 is connected to a display unit 64 that displays video or images.

  The control unit 50 is configured around a known computer having at least a ROM 52, a RAM 54, and a CPU 56. The ROM 52, RAM 54, and CPU 56 are configured similarly to the ROM 7, RAM 8, and CPU 9, respectively.

The ROM 52 stores a processing program for the control unit 50 to execute display processing. The display process is a process of playing the music designated by the user and displaying on the display unit 64 the characteristics of how to sing a professional singer expressed in the music, the lyrics of the music, and the singing melody.
<Data generation processing>
Next, data generation processing executed by the control unit 6 of the information processing apparatus 2 will be described.

This data generation process is started at the timing when a start command for starting the processing program is input via the input receiving unit 3 of the information processing apparatus 2.
In the data generation process, as shown in FIG. 2, when activated, the control unit 6 first selects one piece of music data WD stored in the storage unit 5 of the information processing device 2 from one piece of music data WD. The music data WD is acquired (S110). In S110 of the present embodiment, the control unit 6 acquires the music data WD from the storage unit 5. However, the control unit 6 via the input reception unit 3 via a portable storage medium or communication network. The music data WD may be acquired from a server or the like.

  In the data generation process, the control unit 6 subsequently acquires master waveform data included in the music data WD acquired in S110 (hereinafter referred to as “acquired music data”) (S120). Further, the control unit 6 separates and extracts vocal data and accompaniment data from the master disk waveform data acquired in S120 (S130). As a method of separating accompaniment data and vocal data, which is executed by the control unit 6 in S130, the pitch and pitch estimated using a well-known method (for example, “PreFEst” described in Japanese Patent Laid-Open No. 2008-134606). A method using wave components can be considered. Note that PreFEst is a technique for estimating the pitch of a vocal (that is, the fundamental frequency) and the magnitude of a harmonic component by regarding the most prevalent voice waveform in the master waveform data as vocal data.

  In the data generation process, subsequently, the control unit 6 specifies a vocal sound pressure transition representing a transition of the sound pressure level in the vocal data (S140). Further, the control unit 6 specifies a vocal frequency transition representing a transition of the fundamental frequency f0 in the vocal data (S150).

  Specifically, in S140 and S150 of the present embodiment, the control unit 6 first sets the specified time window AW (j) as vocal data. The specified time window AW (j) is an analysis window having a predetermined unit time (for example, 10 [ms]). In the present embodiment, the specified time window AW is set to be adjacent to and continuous with each other along the time axis. The symbol j is an identifier for identifying the specified time window AW.

  Subsequently, the control unit 6 calculates the sound pressure level Lp in each prescribed time window AW (j) in the vocal data by a known method. The sound pressure level Lp is a predetermined coefficient (usually “20”, which is the common logarithm of the root mean square p of the sound pressure in the prescribed time window AW (j) of the vocal data divided by the reference sound pressure p0. ]) (That is, Lp = 20 × log10 (p / p0)).

Furthermore, the control unit 6 specifies the vocal sound pressure transition by arranging the sound pressure level Lp in each specified time window AW (j) along the time axis in the vocal data.
In addition, in order to identify the vocal frequency transition, the control unit 6 derives the fundamental frequency f0 in each prescribed time window AW (j) in the vocal data. Various known methods can be considered as a method for deriving the fundamental frequency f0. As an example, the control unit 6 performs frequency analysis (for example, DFT) for each specified time window AW (j) set in vocal data, and sets the strongest frequency component as the fundamental frequency f0 as a result of autocorrelation. It is possible.

  And the control part 6 specifies vocal frequency transition by arrange | positioning the fundamental frequency f0 derived | led-out for every these regulation | regulation time window AW (j) along the time axis in vocal data.

  In the data generation process, the control unit 6 subsequently acquires one MIDI music MD associated with the same music ID as the music data WD acquired in S110 (S160). Furthermore, the control unit 6 matches the performance timing of each note constituting the MIDI song MD acquired in S160 (hereinafter referred to as “acquired MIDI”) with the playback time of each sound corresponding to each note of the acquired song data. Then, the acquired MIDI is adjusted (S170). As a method for adjusting the acquired MIDI, it is conceivable to use a known method (for example, the method described in Japanese Patent No. 5310777). In the method described in Japanese Patent No. 5310679, the control unit 6 renders the acquired MIDI, and converts both the rendering result of the acquired MIDI and the master waveform data of the acquired music data into spectral data in a predetermined time unit. And the performance start timing and performance end timing of each performance sound are corrected so that the time on both spectrum data may synchronize. Note that DP matching may be used when adjusting the time on the spectrum data so as to be synchronized.

  Further, in the data generation process, the control unit 6 acquires a melody track representing the singing melody from the MIDI music MD whose time has been adjusted in S170 (S180). In the melody track acquired in S180, each note (hereinafter referred to as "melody note") NO (i) constituting the singing melody is defined. Each melody note NO (i) includes a note number (pitch), performance start timing nnt (i), performance end timing nft (i), various time control information (eg, tempo, resolution, etc.), etc. Note properties are specified. Note that the symbol i is an identifier for identifying a melody note, and is defined so as to increase along the time axis of the singing melody.

  In the data generation process, the control unit 6 generates time-series note data (S190). This time-series note data is obtained by associating note properties relating to each melody note NO (i) with note transitions in which the melody note NO (i) is arranged along the time axis.

  Specifically, in S190 of the present embodiment, first, the control unit 6 generates a note transition in which the melody note NO (i) is arranged along the time axis. The controller 6 sets a specified time window AW (j) for the note transition. The specified time window AW (j) set for the note transition is the same as the specified time window AW (j) set for vocal data. That is, the specified time window AW (j) set for the note transition and vocal data means that the same timing is used if the code j is common.

  Subsequently, the control unit 6 compares the number of specified time windows AW set for note transition with the number of specified time windows AW set for vocal data. As a result of the comparison, the control unit 6 adopts the one having a large number as the number of the specified time windows AW (that is, “jmax”) in both the note transition data and the vocal data. Note that the control unit 6 may perform complementation by adding “0 value” to the smaller number so that the number matches the specified time window AW.

  Further, the control unit 6 corresponds to the performance start timing nnt (i) and performance end timing nft (i) of each melody note NO (i) within the specified time window AW (j) set to note transition. A specified time window AW (j) is specified. Then, the control unit 6 associates a note start flag indicating that it corresponds to the performance start timing nnt (i) with the specified time window AW (j) corresponding to the performance start timing nnt (i). Furthermore, the control unit 6 associates a note end flag indicating that it corresponds to the performance end timing nft (i) with the specified time window AW (j) corresponding to the performance end timing nft (i). At the same time, the control unit 6 adds melody notes corresponding to all the specified time windows AW from the specified time window AW associated with the note start flag to the specified time window AW associated with the note end flag. Time series note data is generated by associating the pitch of NO (i).

  In the data generation process, the control unit 6 displays the vocal sound pressure transition specified in S140, the vocal frequency transition specified in S150, and the time-series note data generated in S190 for each specified time window AW (j). (S200). Below, the data which matched vocal sound pressure transition, vocal frequency transition, and time series note data are also called synchronous data.

  In the data generation process, the control unit 6 executes a vibrato feature calculation process for identifying a section sung using vibrato as a singing technique in the vocal data based on the synchronization data and calculating a vibrato feature (S210). ).

  This vibrato feature is one of the features of a professional singer's singing, and is information on the vibrato used by the professional singer in the music. The vibrato feature calculated in S210 of the present embodiment includes a central pitch difference and a vibrato depth. The central pitch difference is the melody note NO in which the central pitch of the vibrato in the vocal data section sung using vibrato and the performance timing corresponding to the vocal data section sung using vibrato are specified. It is a difference from the pitch of (i). Further, the vibrato depth represents a fluctuation width along the frequency axis of the fundamental frequency f0 in a section of vocal data sung using vibrato.

  Furthermore, in the data generation process, the control unit 6 generates singing feature data SF (S220). Specifically, in S220 of the present embodiment, singing feature data SF is generated by adding identification information (hereinafter referred to as “vibrato ID”) to the vibrato feature amount calculated in S210.

In the data generation process, the control unit 6 stores the singing feature data SF generated in S220 in the storage unit 5 (S230).
Thereafter, the control unit 6 ends the data generation process.
<Vibrato feature calculation processing>
In the vibrato feature calculation process executed in S210 of the data generation process, as shown in FIG. 3, the control unit 6 acquires the vocal frequency transition specified in the previous S140 (S710).

  Subsequently, the control unit 6 performs smoothing differentiation on the vocal frequency transition acquired in S710 for each specified time window AW for the specified number (S720). Furthermore, the control unit 6 specifies a specified time window AW (j) where the result of smoothing differentiation of the vocal frequency transition and a specified axis (so-called X axis) intersects as an intersection point PI (j) (S720). . This intersection point PI (j) is a vertex in the vocal frequency transition and an inflection point. In S720, the control unit 6 associates each intersection point PI with whether the intersection point PI itself is a maximum inflection point or a minimum inflection point.

  In the vibrato feature calculation process, the control unit 6 extracts one intersection point PI (j) from the intersection point PI (j) (S730). In S730, the control unit 6 first acquires the earliest intersection PI (j) along the time axis. Hereinafter, the intersection PI (j) acquired in S730 is referred to as a determination target intersection PI (j).

  Furthermore, the control unit 6 determines whether the number of intersection points PI existing within a specified time length range from the determination target intersection point PI (j) is greater than a predetermined threshold value Thc (for example, “4”). It is determined whether or not (S740). As a result of the determination in S740, if the number of intersection points PI is equal to or less than the threshold Thc (S740: NO), the control unit 6 shifts the vibrato feature calculation process to S770 described later. On the other hand, as a result of the determination in S740, if the number of intersection points PI is greater than the threshold Thc (S740: YES), the control unit 6 shifts the vibrato feature calculation process to S750.

  In S750, the control unit 6 determines that the pitch difference between the highest pitch and the lowest pitch of the vocal frequency transition within the range of the time length specified from the determination target intersection PI (j) is a predetermined rule. It is determined whether it is smaller than the value. As a result of the determination in S750, if the pitch difference within the specified time length is greater than or equal to the specified value (S750: NO), the control unit 6 moves the vibrato feature calculation process to S770. On the other hand, as a result of the determination in S750, if the pitch difference within the specified time length is smaller than the specified value (S750: YES), the control unit 6 moves the vibrato feature calculation process to S760. .

  In S760, the control unit 6 assigns a vibrato flag to all the specified time windows AW (j) existing within the range of the time length specified from the determination target intersection PI (j). This vibrato flag represents a section that may have been sung by vibrato. Hereinafter, one area including the specified time window AW (j) to which the vibrato flag is assigned is referred to as a vibrato candidate area CS (k). Here, the symbol k is an identifier for identifying the vibrato candidate area CS and is a vibrato ID.

  In subsequent S770, the control unit 6 determines whether or not the processing from S730 to S760 has been executed for all the intersection points PI. As a result of the determination in S770, if the processing from S730 to S760 has not been executed for all the intersection points PI (S770: NO), the control unit 6 returns the vibrato feature calculation processing to S730. In S730, one of the earliest intersections PI along the time axis is extracted from the intersections PI corresponding to the specified time window AW to which no vibrato flag is assigned. Thereafter, the control unit 6 executes steps up to S770.

  On the other hand, as a result of the determination in S770, if the processes from S730 to S760 have been executed for all the intersection points PI (S770: YES), the control unit 6 moves the vibrato feature calculation process to S780. In S780, the control unit 6 determines whether or not the pitch of the melody note NO changes in each of the vibrato candidate areas CS. If the pitch of the melody note NO has not changed as a result of the determination in S780 (S780: NO), the control unit 6 sets each vibrato candidate area CS (k) as the vibrato area VT (k). Then, the vibrato feature calculation process is shifted to S800 described later. The vibrato area VT referred to here is a section of a specified time window AW indicating that the vocal data is sung by vibrato.

  Note that if the pitch of the melody note NO has changed as a result of the determination in S780 (S780: YES), the control unit 6 displays the specified time window AW (j) corresponding to the timing at which the pitch has changed. Then, the vibrato candidate area CS is divided (S790). Thereafter, the control unit 6 sets all the vibrato candidate areas CS including the divided vibrato candidate area CS to the vibrato area VT, and shifts the vibrato feature calculation process to S800. In S800, the control unit 6 acquires a vocal frequency transition corresponding to one of the vibrato regions VT as a vibrato waveform.

  Subsequently, the control unit 6 calculates the high pitch reference pitch and the low pitch reference pitch based on the vibrato waveform acquired in S800 (hereinafter referred to as “acquired vibrato waveform”) (S810). The high pitch reference pitch represents the pitch of the high range in the acquired vibrato waveform. Specifically, in S810, the control unit 6 determines the fundamental frequency at the inflection point that is the maximum among the inflection points in the acquired vibrato waveform (that is, the specified time window AW in the vocal frequency transition corresponding to the intersection PI). The representative value of f0 is calculated as the high pitch reference pitch. In S810, the control unit 6 calculates the representative value of the fundamental frequency f0 at the inflection point that is the minimum among the inflection points of the acquired vibrato waveform as the bass reference pitch. The representative value referred to here may be an arithmetic average of the fundamental frequency f0 at the inflection point at which the maximum or minimum is reached. In addition, the representative value may be an arithmetic average of the fundamental frequency f0 excluding the highest fundamental frequency f0 and the lowest fundamental frequency f0 among the fundamental frequencies f0 at the inflection point where the maximum or the minimum is the inflection point. The representative value may be the median value of the fundamental frequency f0 at the inflection point at which the maximum value or the minimum value, or may be the mode value.

  In the present embodiment, the control unit 6 uses the same number of inflection points as the minimum in the acquired vibrato waveform and the number of inflection points as the maximum in the acquired vibrato waveform. A reference pitch is calculated.

  In the vibrato feature calculation process, the control unit 6 subsequently calculates the vibrato center pitch (S820). The vibrato center pitch is the center pitch in the acquired vibrato waveform. Specifically, in S820, the control unit 6 calculates the arithmetic average result of the treble reference pitch and the bass reference pitch calculated in S810 as the vibrato center pitch.

  Further, the control unit 6 calculates the difference between the vibrato central pitch calculated in S820 and the pitch of the melody note NO (i) in which the performance period corresponding to the acquired vibrato waveform is defined as the central pitch difference. (S830). Subsequently, the control unit 6 calculates the difference between the treble reference pitch and the bass reference pitch as the vibrato depth (S840). In the present embodiment, the control unit 6 calculates the central pitch difference and the vibrato depth as cent values.

  In the vibrato feature calculation process, the control unit 6 determines whether or not the steps from S800 to S840 have been executed for all the vibrato regions VT (S850). As a result of the determination in S850, if the steps from S800 to S840 have not been executed for all the vibrato regions VT (S850: NO), the control unit 6 returns the vibrato feature calculation processing to S830. In S830, the control unit 6 acquires a vocal frequency transition corresponding to one of the vibrato regions VT where the steps from S800 to S840 have not been performed as a vibrato waveform. Thereafter, the controller 6 executes steps up to S850.

  On the other hand, as a result of the determination in S850, if the steps from S800 to S840 have been executed for all the vibrato regions VT (S850: YES), the control unit 6 ends the vibrato feature calculation process, and the data generation process The process is shifted to S220.

  That is, in the vibrato feature calculation process, as shown in FIG. 4, the control unit 6 identifies the section sung using vibrato in the vocal data, that is, the vibrato region VT, based on the vocal frequency transition. And the control part 6 calculates a high pitch reference pitch and a low pitch reference pitch based on the pitch of the vertex in the vocal frequency transition corresponding to the specified vibrato area | region VT. Further, the control unit 6 calculates the difference between the central pitch in the vibrato waveform and the pitch of the melody note NO (i) in which the performance period corresponding to the vibrato waveform is defined as the central pitch difference. At the same time, the control unit 6 calculates the difference between the high pitch reference pitch and the low pitch reference pitch as the vibrato depth.

  In S220 of the data generation process executed after the vibrato feature calculation process is completed, the control unit 6 assigns a vibrato ID to the vibrato feature amount as shown in FIG. That is, the control unit 6 generates the singing feature data SF by associating the vibrato feature amount with the vibrato region VT.

The singing feature data SF generated by the control unit 6 of the information processing device 2 executing the data generation process may be stored in the storage unit 14 of the information processing server 10 using a portable storage medium. When the information processing device 2 and the information processing server 10 are connected via a communication network, the singing feature data SF stored in the storage unit 5 of the information processing device 2 is transferred via the communication network. Thus, the information may be stored in the storage unit 14 of the information processing server 10. However, in any case, the singing feature data SF is stored in the storage unit 14 of the information processing server 10 in association with the MIDI music MD of the corresponding music.
<Display processing>
The display process executed by the control unit 50 of the karaoke apparatus 30 is started when a command for starting a processing program for executing the display process is input.

  In the display process, when activated, as shown in FIG. 6, the control unit 50 stores the MIDI music MD corresponding to the music specified via the input receiving unit 34 as the storage unit 14 of the information processing server 10. (S910). Subsequently, the control unit 50 acquires lyrics data corresponding to the MIDI music MD acquired in S910 (S920). Furthermore, the control unit 50 acquires singing feature data SF corresponding to the MIDI music MD acquired in S910 (S930).

  In the display process, the control unit 50 subsequently plays the MIDI musical piece MD acquired in S910 (S940). Specifically, in S940, the control unit 50 outputs the MIDI music piece MD to the music reproduction unit 36. The music reproducing unit 36 that has acquired the MIDI music MD performs the music. Then, the sound source signal of the music played by the music playback unit 36 is output to the speaker 60 via the output unit 42. Then, the speaker 60 outputs the sound source signal instead of sound.

  Further, in the display process, the control unit 50 performs display based on the MIDI music MD acquired in S910, the lyrics data acquired in S920, and the singing feature data SF acquired in S930 (S950). In S950, specifically, the control unit 50 outputs the melody track, the lyric data, and the singing feature data SF in the MIDI music piece MD to the video control unit 46. The video control unit 46 sequentially outputs the lyrics data and the melody note NO (i) constituting the melody track to the display unit 64 along the time axis. Then, the display unit 64 sequentially displays the lyrics data along the time axis, and sequentially displays the melody note NO (i) constituting the melody track along the time axis as shown in FIG.

  Furthermore, the video control unit 46 sequentially outputs the vibrato feature amount included in the singing feature data SF to the display unit 64 along the time axis at the timing of the corresponding vibrato region VT. Then, the display unit 64 sequentially displays the vibrato feature amount along the time axis at the timing of the corresponding vibrato region VT.

  In S950, the control unit 50 may output the transition of the pitch of the singing voice sung by the user of the karaoke apparatus 30 to the video control unit 46 in accordance with the performance of the music. Then, the video control unit 46 changes the pitch of the singing voice sung by the user of the karaoke device 30 (for example, the solid line waveform shown in FIG. 7) from the deviation from the melody note NO (i). May be displayed in a recognizable manner. Furthermore, in S950, the control unit 50 may output the pitch transition of vocal data sung by a professional singer to the video control unit 46. Then, the video control unit 46 may display the pitch transition of vocal data sung by a professional singer (that is, vocal frequency transition, broken line waveform shown in FIG. 7).

  In FIG. 7, the vibrato region VT is illustrated so that the vibrato feature amount and the vibrato region VT are easily associated with each other. However, the display unit 64 according to the present embodiment includes the vibrato feature amount. Only the vibrato area VT need not be displayed.

  Further, in S950, the control unit 50 may output the result of classifying the vibrato feature amount for each predefined classification range to the video control unit 46. That is, for example, the control unit 50 classifies the vibrato feature, and the control unit 50 classifies the central pitch as “central pitch” if the central pitch difference is higher than a specified pitch difference, for example. Then, if the central pitch difference is lower than the specified pitch difference, the control unit 50 classifies it as “central pitch low”. Furthermore, for example, if the vibrato depth is less than the specified pitch difference, the control unit 50 classifies the “vibrato shallow”. Then, if the vibrato depth is equal to or greater than the specified pitch difference, the control unit 50 classifies it as “deep vibrato”. Then, the control unit 50 may output the classification result to the video control unit 46 as a vibrato feature. In this case, the video control unit 46 causes the display unit 64 to display the classification result.

Subsequently, in the display process, the control unit 50 determines whether or not the music performance has ended (S960). If the result of this determination is that the music performance has not ended (S960: NO), the control unit 50 returns the display processing to S940. On the other hand, if the result of determination in S960 is that the music performance has ended (S960: YES), the control unit 50 ends the display process.
[Effect of the embodiment]
As described above, the singing feature data SF of the present embodiment specifies the vibrato region VT and associates the specified vibrato region VT with the “center pitch difference” and the “vibrato depth”. The “center pitch difference” and “vibrato depth” are information representing characteristics relating to the vibrato used by the singer, and are unique to each singer.

And in the karaoke apparatus 30 displayed on the display part 64 based on singing characteristic data SF, "central pitch difference" and "vibrato depth" are displayed sequentially along a time axis.
For this reason, the user of the karaoke apparatus 30 can recognize how a professional singer uses “vibrato” as a singing technique in the song being sung. As a result, the user of the karaoke apparatus 30 can bring the user's own way of singing closer to that of a professional singer.

  In the data generation processing of the present embodiment, the same number of inflection points as the maximum and minimum inflection points in the vibrato waveform are used to calculate the treble reference pitch and the bass reference pitch. It is said.

For this reason, according to the data generation processing, the singular point can be removed from the high pitch reference pitch and the low pitch reference pitch, and the calculation accuracy of the high pitch reference pitch and the low pitch reference pitch can be improved.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, the data generation process in the above embodiment is executed by the information processing apparatus 2, but the apparatus that executes the data generation process in the present invention is not limited to the information processing apparatus 2. That is, the apparatus that executes the data generation process may be the information processing server 10 or the karaoke apparatus 30. In this case, the information processing apparatus 2 may be omitted from the system 1.

  The display process in the above embodiment is executed by the karaoke apparatus 30, but the apparatus that executes the display process in the present invention is not limited to the karaoke apparatus 30, and may be the information processing apparatus 2. In this case, the information processing apparatus 2 is provided with a video control unit 46, and a display unit 64 needs to be connected to the video control unit 46.

  In the above embodiment, the data generation process and the display process are configured as separate processes. However, in the present invention, the data generation process and the display process may be configured as a single process. In this case, one process including the data generation process and the display process may be executed by the information processing apparatus 2 or may be executed by the karaoke apparatus 30.

  By the way, in the vibrato feature calculation process of the above embodiment, both the central pitch difference and the vibrato depth are calculated as the vibrato feature. However, in the present invention, of the central pitch difference and the vibrato depth, Only one of these may be calculated as the vibrato feature. In this case, in the vibrato feature calculation process, S820 and S830 may be omitted, or S840 may be omitted.

  In the data generation process of the above embodiment, the singing feature data SF is generated by associating the “central pitch difference” and the “vibrato depth” with the vibrato region VT. The object to be associated with the “vibrato depth” is not limited to the vibrato region VT. For example, the object associated with “center pitch difference” and “vibrato depth” may be a melody note NO including the vibrato region VT.

In this case, the object associated with the melody note NO may be at least one of “center pitch difference” and “vibrato depth”.
In addition, the aspect which abbreviate | omitted a part of structure of the said embodiment as long as the subject could be solved is also embodiment of this invention. Further, an aspect configured by appropriately combining the above embodiment and the modification is also an embodiment of the present invention. Moreover, all the aspects which can be considered in the limit which does not deviate from the essence of the invention specified by the wording described in the claims are the embodiments of the present invention.

  DESCRIPTION OF SYMBOLS 1 ... System 2 ... Information processing apparatus 3 ... Input reception part 4 ... Information output part 5 ... Memory | storage part 6, 16, 50 ... Control part 7, 18, 52 ... ROM 8, 20, 54 ... RAM 9, 22, 56 ... CPU 10 ... Information processing server 12 ... Communication unit 14 ... Storage unit 30 ... Karaoke device 32 ... Communication unit 34 ... Input reception unit 36 ... Music playback unit 38 ... Storage unit 40 ... Voice control unit 42 ... Output unit 44 ... Microphone input unit 46 ... Video control unit 60 ... Speaker 62 ... Microphone 64 ... Display unit

Claims (8)

1st acquisition which acquires the score data which represents the score of the music comprised by several notes, Comprising: The musical score data with which the pitch and the performance period were matched with each of these several notes from the 1st memory | storage part Steps,
A second acquisition step of acquiring music data including a vocal sound of singing the music from the second storage unit;
An extraction step of extracting vocal data representing the vocal sound from the music data;
Based on the vocal data, a pitch transition determination step for determining a vocal pitch transition representing a transition of a pitch in the vocal data;
Based on the vocal pitch transition, a specific step of identifying a vibrato waveform that is a singing section sung using vibrato in the vocal pitch transition;
Based on the peak pitch in the vibrato waveform, a reference calculation for calculating a high pitch reference pitch that represents a high pitch in the vibrato waveform and a low pitch reference pitch that represents a low pitch in the vibrato waveform Steps,
Based on the high pitch reference pitch and the low pitch reference pitch, a central pitch difference that is a difference between a central pitch in the vibrato waveform and a pitch of a note corresponding to the singing section in the score data, or A determination step of determining at least one of vibrato depths representing a fluctuation width along a frequency axis in the vibrato waveform;
A data generation step of generating singing feature data is executed on a computer by associating at least one of the central pitch difference and the vibrato depth with an interval in the score data corresponding to the singing interval. A program characterized by letting The determining step includes
An average value of the high pitch reference pitch and the low pitch reference pitch is determined as a central pitch in the vibrato waveform,
The program according to claim 1, wherein a difference between the central pitch and a pitch of a note corresponding to the singing section in the score data is determined as the central pitch difference. The determining step includes
The program according to claim 1 or 2, wherein a difference between the high pitch reference pitch and the low pitch reference pitch is calculated as the vibrato depth. The reference calculating step includes
A vertex detection step of detecting an inflection point in the vibrato waveform;
Of the inflection points of the vibrato waveform, a treble reference calculation step of calculating a representative value of the pitch at the inflection point that is a maximum as the treble reference pitch;
The bass reference calculation step of calculating a representative value of a pitch at an inflection point that is a minimum among the inflection points of the vibrato waveform as the bass reference pitch. The program according to any one of claims 1 to 3. The reference calculating step includes
The treble reference pitch and the bass reference pitch are calculated using the same number of inflection points as the minimum in the vibrato waveform and the number of inflection points as the maximum in the vibrato waveform. The program according to claim 4. A plurality of notes constituting the musical score data acquired in the first acquisition step are displayed on a display unit, and at least one of the central pitch difference and the vibrato depth included in the singing feature data 6. The display control step of causing the computer to execute a display control step of associating with a section in the score data corresponding to the singing section and displaying on the display unit. A program according to any one of the above. 1st acquisition which acquires the score data which represents the score of the music comprised by several notes, Comprising: The musical score data with which the pitch and the performance period were matched with each of these several notes from the 1st memory | storage part Means,
Second acquisition means for acquiring music data including the vocal sound of singing the music from the second storage unit;
Extraction means for extracting vocal data representing the vocal sound from the music data;
Based on the vocal data, a pitch transition determining means for determining a vocal pitch transition representing a transition of a pitch in the vocal data;
Based on the vocal pitch transition, a specifying means for identifying a vibrato waveform that is a singing section sung using vibrato in the vocal pitch transition;
Based on the peak pitch in the vibrato waveform, a reference calculation for calculating a high pitch reference pitch that represents a high pitch in the vibrato waveform and a low pitch reference pitch that represents a low pitch in the vibrato waveform Means,
Based on the high pitch reference pitch and the low pitch reference pitch, a central pitch difference that is a difference between a central pitch in the vibrato waveform and a pitch of a note corresponding to the singing section in the score data, or Determining means for determining at least one of vibrato depths representing a fluctuation width along a frequency axis in the vibrato waveform;
Data generation means for generating singing feature data by associating at least one of the central pitch difference and the vibrato depth with the section of the score data corresponding to the singing section. A characteristic information processing apparatus. The information processing apparatus acquires, from the first storage unit, musical score data representing a musical score of a music composed of a plurality of notes, each of which is associated with a pitch and a performance period. A first acquisition procedure,
A second acquisition procedure in which the information processing apparatus acquires from the second storage unit music data including a vocal sound that sang the music;
An extraction procedure by which the information processing apparatus extracts vocal data representing the vocal sound from the music data;
Based on the vocal data, a pitch transition determination procedure in which the information processing apparatus determines a vocal pitch transition representing a transition of a pitch in the vocal data;
Based on the vocal pitch transition, a specific procedure in which the information processing device identifies a vibrato waveform that is a singing section sung using vibrato in the vocal pitch transition;
Based on the peak pitch in the vibrato waveform, the high pitch reference pitch representing the high pitch in the vibrato waveform and the low pitch reference pitch representing the low pitch in the vibrato waveform are the information processing. A reference calculation procedure calculated by the device;
Based on the high pitch reference pitch and the low pitch reference pitch, a central pitch difference that is a difference between a central pitch in the vibrato waveform and a pitch of a note corresponding to the singing section in the score data, or A determination procedure in which the information processing apparatus determines at least one of vibrato depths representing a fluctuation width along a frequency axis in the vibrato waveform;
A data generation procedure in which the information processing apparatus generates singing feature data by associating at least one of the central pitch difference and the vibrato depth with a section in the score data corresponding to the singing section. A data generation method comprising: and.