JP6380305B2 - Data generation apparatus, karaoke system, and program - Google Patents

Description

  The present invention relates to a technique for generating data representing characteristics of a singing method used by a specific person in music and a karaoke system using the data.

2. Description of the Related Art Conventionally, there has been known a karaoke apparatus that outputs a singing voice input via a microphone during performance of a designated music piece and evaluates the singing voice (see Patent Document 1).
In the karaoke apparatus described in Patent Document 1, the guide melody data included in the song data of the music is used as a reference, and the pitch transition of the singing melody represented by the guide melody data and the transition of the fundamental frequency of the singing voice are obtained. A basic score is calculated based on the comparison result.

JP 2007-334364 A

  However, in the prior art, the object to be compared with the singing voice to determine the singing voice of the song by the user is the pitch transition of the singing melody represented by the guide melody data. Thus, when the pitch transition of the singing melody represented by the guide melody is set as a comparison target, it is not possible to properly evaluate the singing including the characteristics of how to sing when the user sings the music.

That is, there is a problem that the pitch transition of the singing melody represented by the guide melody data is not suitable as a comparison target for the singing voice.
Therefore, an object of the present invention is to automatically generate data that can accurately detect the characteristics of a singing method used by a specific person who sings music.

  In order to achieve the above object, the present invention provides a data generation apparatus comprising a synthetic sound acquisition means, a synthesis transition derivation means, an original acquisition means, a song transition derivation means, a difference calculation means, and a registration means. About.

  The synthesized sound acquisition means acquires singing synthesized sound data. The singing synthesized sound data referred to here is synthesized speech that is generated based on the score data and the lyrics data, and is synthesized by voice synthesis according to the score represented by the score data. The musical score data represents a musical score of a designated musical piece that is a designated musical piece having a plurality of notes composed of combinations of pitches and note values. The lyrics data represents lyrics assigned to at least a part of a plurality of notes constituting the designated music.

  The synthesis transition deriving unit derives the singing synthesis f0 transition representing the transition of the fundamental frequency of the synthesized speech from the singing synthesized sound data acquired by the synthesized sound acquiring unit. Furthermore, the original acquisition means acquires singing voice data that is a singing voice of a specific person singing the designated music piece.

  Then, the singing transition derivation means derives an original f0 transition representing transition of the fundamental frequency of the singing voice from the singing voice data acquired by the original acquisition means. The difference calculating means calculates difference data representing a difference along the time axis in the designated music between the singing composition f0 transition derived by the composition transition deriving means and the original f0 transition derived by the singing transition deriving means.

Further, the registration unit stores the difference data calculated by the difference calculation unit in the storage unit.
The score data has notes composed of combinations of pitches and note values, and the singing characteristics such as the singing technique used in the designated music are not indicated in the score data. For this reason, the synthesized voice is a transition of the sound of a singing that does not show a singing feature such as a singing technique, and the singing synthesized f0 transition is a transition of the fundamental frequency of the synthesized voice.

On the other hand, the singing voice data is a voice when a specific person sings a designated music piece, and expresses characteristics of how to sing the specific person in the designated music piece.
Therefore, the difference data calculated by the data generation device represents the characteristics of the singing method used by the specific person, and represents the characteristics of the singing method by the specific person.

In other words, according to the data generation device, data representing characteristics of how to sing by a specific person can be automatically generated.
As a result, in the karaoke apparatus, the difference data can be used to evaluate the user's singing voice or to notify the characteristics of the way of singing to be used when singing.

The specific person mentioned here may be a professional singer (so-called artist) who sings the designated music piece, or may be a general person rather than a professional singer.
The data generation apparatus may further include a technique determination unit and a technique data generation unit.

  The technique determination means compares the singing composition f0 transition derived by the composition transition deriving means with the original f0 transition derived by the singing transition derivation means, and as a result of comparison, the original f0 transition indicates the singing composition f0 transition. It is determined whether or not the singing voice data satisfies technical conditions defined in advance as conditions for various singing techniques used in singing.

  Further, if the result of the determination by the technique determination means satisfies the technique condition, the technique data generation means specifies the type of singing technique that matches the technique condition and the timing using the singing technique, and the specified singing Technique data representing the timing of using the technique type and the singing technique is generated.

Then, the registration unit stores song data, which is data in which the technique data generated by the technique data generation unit and the difference data are associated with each other along the time axis of the designated music piece, in the storage unit.
According to such a data generation device, it is possible to generate technique data representing the type of singing technique used by a specific person in the designated music piece and the timing using the singing technique. And according to a data generation device, song data which is data which matched the technique data and difference data can be memorized in a storage part.

As a result, according to the data generation device, the content of data stored in the storage unit can be enhanced.
Moreover, even if the technique data generation means does not satisfy the technique conditions as a result of the comparison by the technique determination means, there is a difference of a predetermined value or more between the singing synthesis f0 transition and the original f0 transition. For example, the contents of the difference and the timing at which the difference exists are generated as technique data.

  According to such a data generation device, even if the difference does not satisfy the technical condition, the content of the difference can be converted into data as characteristics of the singing method used by a specific person who sang the designated music. it can.

As a result, according to the data generation device, more detailed contents can be included in the singing data as the characteristics of the singing method used by a specific person.
The present invention is a karaoke system comprising synthetic sound acquisition means, synthesis transition derivation means, performance execution means, original acquisition means, singing transition derivation means, difference calculation means, registration means, and processing execution means. May have been made.

The synthesized sound acquisition means acquires singing synthesized sound data. The composition transition deriving unit derives the singing composition f0 transition. The performance executing means plays the designated music according to the score data.
The original acquisition means acquires singing voice data which is a singing voice of a specific person singing the designated musical piece played by the performance executing means. The singing transition derivation means derives the original f0 transition. The difference calculating means calculates difference data in the designated music between the song composition f0 transition and the original f0 transition.

Further, the registration unit stores the difference data in the storage unit. The process execution means executes a specified process, which is a process specified for the performance of the designated music, based on the difference data.
The difference data in such a karaoke system is data representing the characteristics of the singing method used by a specific person in the designated music.

And according to a karaoke apparatus, a prescription | regulation process can be performed based on difference data, and it can be made into a thing with high preference regarding the performance of a designated music.
When the designated music is played by the performance executing means, the process executing means performs notification processing for notifying the characteristics of the singing method used by a specific person who sang the designated music based on the difference data stored in the storage unit. It is executed as a specified process.

According to such a karaoke system, the characteristic of the way of singing used by a specific person can be notified.
Further, the karaoke system may include singing voice acquisition means and singing voice transition derivation means.

  The singing voice acquisition means acquires singing voice data that is voice input through a microphone during the performance of the designated music by the performance execution means. The singing voice transition deriving unit derives the singing voice f0 transition representing the transition of the fundamental frequency of the singing voice based on the singing voice data acquired by the singing voice acquiring unit.

Further, the process execution means executes the process based on the singing voice f0 transition derived by the singing voice transition deriving means as the regulation process.
According to such a karaoke system, the process based on the singing voice f0 transition can be executed as the prescribed process.

  Moreover, the process execution means in the karaoke system may include a difference generation means. In the difference generation means, singing difference data representing a difference along the time axis in the designated music between the singing voice f0 transition derived by the singing voice transition deriving means and the singing synthesis f0 transition derived by the synthesis transition deriving means is generated.

  Furthermore, the process execution means compares the singing difference data generated by the difference generation means with the difference data stored in the storage unit, and as a result of the comparison, the evaluation process is defined as a higher evaluation as the matching degree is higher. It may be executed as a process.

  According to such a karaoke system, the higher the degree of coincidence between the singing difference data and the difference data, the higher the evaluation. As a result, according to the karaoke system, the higher the reproducibility, the higher the evaluation can be made for a specific person who sang the designated music.

  The process execution means in the karaoke system changes the singing voice f0 transition derived by the singing voice transition deriving means so that a specific person approaches the singing voice singing the designated music based on the difference data stored in the storage unit. And the change process which outputs the audio | voice based on the changed singing voice f0 transition is performed as a prescription | regulation process.

According to such a karaoke system, the singing voice of the user can be changed and output so that a specific person approaches the singing voice of singing the designated music piece.
The present invention may be a program that causes a computer to execute a synthetic sound acquisition procedure, a synthesis transition derivation procedure, an original acquisition procedure, a singing transition derivation procedure, a difference calculation procedure, and a registration procedure.

  As described above, if the present invention is implemented as a program, it is used by loading the computer from a recording medium as necessary and starting it, or by acquiring it and starting it through a communication line as necessary. be able to. And by making a computer perform each procedure, the computer can be functioned as a data generation apparatus, and the data showing the characteristic of how to sing used by a specific person can be generated.

  The recording medium referred to here includes, for example, a computer-readable electronic medium such as a DVD-ROM, a CD-ROM, and a hard disk.

It is a block diagram which shows schematic structure of a karaoke system. It is a flowchart which shows the process sequence of a music analysis process. It is explanatory drawing explaining the outline | summary of difference data. It is a flowchart which shows the process sequence of a song technique determination process. It is a flowchart which shows the process sequence of a performance process. It is a figure which shows an example of the display of a singing technique. It is a figure which shows an example of a change audio | voice. It is a figure which shows an example of the evaluation method. It is a figure explaining the effect of an embodiment. It is a figure explaining the effect of an embodiment. It is a figure explaining the effect of an embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
<System configuration>
A karaoke system 1 shown in FIG. 1 includes an information processing device 2, an information processing server 10, and a karaoke device 30. In the karaoke system 1, the song data EM generated by the information processing device 2 is stored in the information processing server 10, and the karaoke device 30 plays the designated music and the song times stored in the information processing server 10. A process based on the data EM is executed.

  Note that the music referred to here is a music in which lyrics are assigned to at least a part of a plurality of notes constituting the music, out of music having a plurality of notes made up of combinations of pitches and note values.

  The song data EM referred to here is data representing characteristics of a specific person who sings a song in how to sing. The song data EM is generated by the information processing device 2 based on the music data WD stored in the information processing device 2 and the MIDI music MD.

As shown in FIG. 1, the karaoke system 30 included in the karaoke system 1 may be plural or singular.
<Music data>
The music data WD is prepared in advance for each music. The music data WD includes music management information and master waveform data. The music management information is information for identifying a music, and has a music ID that is unique identification information assigned to each music.

  The master waveform data is audio data representing the transition of the musical performance sound. The performance sound of the music mentioned here includes a performance sound of a plurality of musical instruments and a singing voice sung by a professional singer. The audio data may be data configured by an audio file in an uncompressed audio file format, or may be data configured by an audio file in an audio compression format.

In the following, the speech waveform data representing the transition of the singing speech included in the master waveform data is referred to as singing speech data.
<MIDI music>
The MIDI musical piece MD is prepared in advance for each musical piece, and has score data and lyrics data.

  The musical score data is data representing the musical score of one piece of music according to the well-known MIDI (Musical Instrument Digital Interface) standard. The score data includes a song ID and a score track that represents the score for each instrument used in the song.

  The musical score track defines at least the pitch (so-called note number) and the period during which the MIDI sound source outputs the performance sound (hereinafter referred to as tone value) for each performance sound output from the MIDI sound source. Has been. The note value in the score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the performance sound, and the music until the output of the performance sound ends. Performance end timing (so-called note-off timing) representing the time from the start of the performance.

  That is, in the musical score track, one note NO is defined by the note number and the note value represented by the note-on timing and the note-off timing. The musical score track functions as one musical score by arranging note NO in the order of performance.

  As a score track in the present embodiment, at least a score track of a specific instrument in charge of a melody line representing a singing melody is prepared. As an example of this specific musical instrument, vibraphone can be considered.

  The lyrics data is data relating to the lyrics of the music. The lyric data includes lyric telop data and lyric output data. The lyrics telop data represents characters that constitute the lyrics of the music (hereinafter referred to as lyrics component characters). The lyric output data is data in which a timing correspondence relationship for associating the lyric output timing, which is the output timing of the lyrics constituent characters, with the performance of the score data is defined.

In the timing correspondence relationship in the present embodiment, the timing for starting the output of the lyrics telop data is associated with the timing for starting the performance of the musical score data. Furthermore, in the timing correspondence relationship, the lyrics output timing of each lyrics constituent character along the time axis of the music is defined by the elapsed time from the start of performance of the score data. As a result, the notes of the individual performance sounds defined in the score track are associated with the lyrics constituent characters.
<Information processing device>
The information processing apparatus 2 is a known information processing apparatus including an input receiving unit 3, an information output unit 4, a storage unit 5, and a control unit 6. As an example of the information processing apparatus 2, a known personal computer can be considered.

  The input receiving unit 3 is an input device that receives input of information and commands from the outside. The input device here is, for example, a key or switch, a reading drive for reading data stored in a portable storage medium (for example, CD, DVD, flash memory), or a communication port for acquiring information via a communication network. Etc. The information output unit 4 is an output device that outputs information to the outside. Here, the output device is a writing drive that writes data to a portable storage medium, a communication port that outputs information to a communication network, or the like.

  The storage unit 5 is a known storage device configured to be able to read and write stored contents. In the storage unit 5, music data WD and MIDI music MD are stored in association with each common music.

  The control unit 6 is a known control device that is configured around a known microcomputer including a ROM 7, a RAM 8, and a CPU 9. The ROM 7 stores processing programs and data that need to retain stored contents even when the power is turned off. The RAM 8 temporarily stores processing programs and data. The CPU 9 executes each process according to a processing program stored in the ROM 7 or RAM 8.

In the ROM 7 of the present embodiment, a processing program for the control unit 6 to execute music analysis processing for generating song data EM based on the music data WD and the MIDI music MD stored in the storage unit 5. Is remembered.
<Information processing server>
The information processing server 10 includes a communication unit 12, a storage unit 14, and a control unit 16.

  Among these, the communication unit 12 performs communication between the information processing server 10 and the outside via a communication network. That is, the information processing server 10 is connected to the karaoke apparatus 30 via a communication network. The communication network referred to here may be a wired communication network or a wireless communication network.

  The storage unit 14 is a known storage device configured to be able to read and write stored contents. The storage unit 14 stores a plurality of MIDI music pieces MD. Further, the storage unit 14 stores song data EM generated by the information processing apparatus 2 executing the music analysis process. 1 is an identifier for identifying the MIDI music piece MD and the song data EM stored in the storage unit 14 of the information processing server 10, and is a natural number of 1 or more.

The control unit 16 is a known control device that is configured around a known microcomputer including a ROM 18, a RAM 20, and a CPU 22. The ROM 18, RAM 20, and CPU 22 are configured similarly to the ROM 7, RAM 8, and CPU 9, respectively.
<Karaoke equipment>
The karaoke apparatus 30 includes a communication unit 32, an input reception unit 34, a music playback unit 36, a storage unit 38, an audio control unit 40, a video control unit 46, and a control unit 50.

  In the communication unit 32, the karaoke apparatus 30 communicates with the outside via a communication network. The input receiving unit 34 is an input device that receives input of information and commands in accordance with external operations. Here, the input device is, for example, a key, a switch, a reception unit of a remote controller, or the like.

The music playback unit 36 performs a music performance based on the MIDI music MD downloaded from the information processing server 10. The music reproducing unit 36 is, for example, a MIDI sound source.
The voice control unit 40 is a device that controls voice input / output. The voice control unit 40 includes an output unit 42 and a microphone input unit 44. A microphone 62 is connected to the microphone input unit 44. Thereby, the microphone input part 44 acquires a user's song sound. A speaker 60 is connected to the output unit 42. The output unit 42 outputs the sound source signal of the music reproduced by the music reproducing unit 36 and the sound source signal from the microphone input unit 44 to the speaker 60. The speaker 60 outputs the sound source signal output from the output unit 42 instead of sound.

  The video control unit 46 outputs a video or an image based on the video data sent from the control unit 50. The video control unit 46 is connected to a display unit 64 that displays video or images.

  The control unit 50 is configured around a known computer having at least a ROM 52, a RAM 54, and a CPU 56. The ROM 52, RAM 54, and CPU 56 are configured similarly to the ROM 7, RAM 8, and CPU 9, respectively.

The ROM 52 stores a processing program for the controller 50 to execute performance processing. The performance process is a process of performing a specified process based on the song data EM corresponding to the specified music while playing the specified music. The prescribed process is a process prescribed for the performance of the designated music piece, and is a process executed based on the song data EM.
<Music analysis processing>
The music analysis process executed by the information processing apparatus 2 will be described with reference to FIG. This music analysis process is activated when a predefined activation command is input. The start command is input, for example, when master waveform data including a singing voice sung by a professional singer is read. Alternatively, it is input when the user's singing sound is input to the microphone.

  When the music analysis process is activated, the control unit 6 first acquires a music ID as shown in FIG. 2 (S110). The song ID acquired in S110 is a song ID representing a song to be generated for the song data EM, and is a song ID corresponding to a song input via the input receiving unit 3. Note that the music corresponding to the music ID acquired in S110 is the designated music.

  In the music analysis process, the control unit 6 acquires the MIDI music MD including the music ID acquired in S110 (S120). Further, the control unit 6 acquires lyrics data included in the MIDI music piece MD acquired in S120 (S130).

  Subsequently, in the music analysis process, the control unit 6 generates and acquires song synthesis sound data based on the score data included in the MIDI music MD acquired in S120 and the lyrics data acquired in S130 (S140). ). The singing synthesized sound data referred to here is a synthesized speech obtained by synthesizing the lyrics represented by the lyric data according to the musical score represented by the musical score data (that is, the singing generated by the information processing device 2) in the designated music. (Hereinafter referred to as “singing synthesized sound”). Since the method for generating this singing synthesized sound is well known, detailed description thereof is omitted here. For example, it is conceivable to use formant synthesis as a method for generating the singing synthesized sound.

  Further, the control unit 6 derives the singing synthesis f0 transition based on the singing synthesized sound data acquired in S140 (S150). The song synthesis f0 transition is a transition of the fundamental frequency f0 of the song synthesis sound. In the present embodiment, the following method may be used as a method for specifying the transition of the fundamental frequency f0. The control unit 6 sets the analysis windows representing the predetermined unit time in the singing synthesized sound data so as to be adjacent to each other. Frequency analysis (for example, DFT) is performed on the singing synthesized sound data in each of the set analysis windows. Further, the control unit 6 calculates one fundamental frequency f0 for one analysis window by setting the strongest frequency component as the fundamental frequency f0 as a result of autocorrelation. Then, the singing synthesis f0 transition is derived by arranging the fundamental frequency f0 along the time axis.

  Subsequently, in the music analysis process, the control unit 6 acquires singing voice data of the designated music (S160). Further, the control unit 6 derives the original f0 transition based on the singing voice data acquired in S160 (S170). The original f0 transition referred to here is a transition of the fundamental frequency f0 of the singing voice represented by the singing voice data of the designated music. As a method for specifying the transition of the fundamental frequency f0, a method similar to S150 may be used except that “singing synthesized sound data” is replaced with “singing voice data”.

  In the music analysis process, the control unit 6 calculates difference data according to the singing synthesis f0 transition calculated in S150 and the original f0 transition calculated in S170 (S180). As shown in FIG. 3, the difference data referred to here represents a difference along the time axis in the designated music between the song composition f0 transition and the original f0 transition. The difference data in the present embodiment may be obtained by subtracting the song synthesis f0 transition from the original f0 transition.

  Subsequently, in the music analysis process, the control unit 6 executes a singing technique determination process (S190). In this singing technique determination process, as will be described in detail later, as a result of comparing the singing synthesis f0 transition and the original f0 transition, if the technique condition is satisfied, the professional singer uses the singing technique in the specific music. Identify. In the singing technique determination process, technique data representing the type of the specified singing technique and the timing using the singing technique is generated. In the singing technique determination process, for example, the singing composition f0 transition is compared with the original f0 transition, and as a result, the singing composition f0 transition is a transition that satisfies the technique conditions of various singing techniques with respect to the original f0 transition. When it is determined that various singing techniques are used. In the singing technique determination process, various singing techniques may be determined using the difference data calculated in step S180 as a result of comparing the singing synthesis f0 transition and the original f0 transition.

The technique condition referred to here is a condition defined in advance as a condition of various singing techniques used in singing a specific music piece.
Further, in the music analysis process, the control unit 6 generates song data EM and stores it in the storage unit 5 (S200). The song data EM in the present embodiment is data representing the singing of a professional singer in the designated song, and includes difference data, technique data, song composition f0 transition, and song ID. Specifically, in S200 of the present embodiment, the song data EM is generated by associating the difference data, the technique data, and the singing composition f0 transition along the time axis in the designated song and by associating the song ID of the designated song. To do. Alternatively, the song data EM may not be created, and the difference data, technique data, singing synthesis f0 transition, and music ID may be associated with each other and stored in the storage unit 5.

Thereafter, the control unit 6 ends the music analysis process.
Note that the song data EM generated when the control unit 6 of the information processing device 2 executes the music analysis process may be stored in the storage unit 14 of the information processing server 10 using a portable storage medium. . When the information processing device 2 and the information processing server 10 are connected via a communication network, the song data EM stored in the storage unit 5 of the information processing device 2 is transferred via the communication network. Thus, the information may be stored in the storage unit 14 of the information processing server 10.

That is, the information processing apparatus 2 that executes the music analysis process functions as a data generation apparatus.
<Singing technique judgment processing>
Next, the singing technique determination process executed in S190 of the music analysis process will be described with reference to FIG.

  When the singing technique determination process is activated, the control unit 6 first determines whether or not “shearing”, which is one of the singing techniques, is used as shown in FIG. 4 (S310). In this embodiment, when the original f0 transition satisfies the following technique condition with respect to the singing synthesis f0 transition, it is determined that “shakuri” is used.

  The technical condition indicating that “shakuri” is used is that the original f0 transition is lower along the frequency axis than the singing synthesis f0 transition at the performance start timing of the note to which the lyrics are assigned. Furthermore, the original f0 transition after the performance start timing of the note rises at a rate higher than a prescribed rate during unit time and reaches the singing synthesis f0 transition.

  Subsequently, in the singing technique determination process, the control unit 6 determines whether or not “vibrato” which is one of the singing techniques is used (S320). In this embodiment, when the original f0 transition satisfies the following technique condition with respect to the singing synthesis f0 transition, it is determined that “vibrato” is used.

  The technique condition indicating that “vibrato” is used is within the frequency range in which the original f0 transition is defined with the singing composition f0 transition as the reference frequency in the section of one note to which the lyrics are assigned. It is that it fluctuates along the frequency direction.

  Furthermore, in the singing technique determination process, the control unit 6 determines whether or not “fist” which is one of the singing techniques is used (S330). In this embodiment, when the original f0 transition satisfies the following technique condition with respect to the singing synthesis f0 transition, it is determined that “fist” is used.

  The technique condition indicating that “fist” is used is that the original f0 transition is in the frequency direction during the unit time with respect to the singing synthesis f0 transition in the section of one note to which the lyrics are assigned. That is, the original f0 transition has returned to the frequency before the increase after rising above the specified value.

  In the singing technique determination process, the control unit 6 determines whether “Tame”, which is one of the singing techniques, is used (S340). In this embodiment, when the original f0 transition satisfies the following technique condition with respect to the singing synthesis f0 transition, it is determined that “tame” is used.

  The technical condition indicating that “Tame” is used is that the original note pitch is changed to the pitch of the previous note along the time axis from the current note at the performance start timing of the note to which the lyrics are assigned. The transition of f0 is maintained for a predetermined time specified in advance, and then transitions to the pitch of the current note.

  Subsequently, in the singing technique determination process, the control unit 6 determines whether or not “facial expression” which is one of the singing techniques is used (S350). In this embodiment, when the original f0 transition satisfies the following technique condition with respect to the singing synthesis f0 transition, it is determined that “facial expression” is used. In addition, "facial expression" here is a singing technique that expresses the strength of singing.

  The technique condition indicating that “facial expression” is used is that the singing composition f0 transition is continuous in one musical note interval to which the lyrics are assigned, whereas the original f0 transition is interrupted. It is that you are.

  Further, in the singing technique determination process, the control unit 6 determines whether or not the “extended technique” that is one of the singing techniques is used (S360). In this embodiment, even if the conditions from S310 to S350 described above are not satisfied, if there is a difference greater than a predetermined value between the singing synthesis f0 transition and the original f0 transition, the difference The timing at which the content and the difference exist is determined as a section in which the “extended technique” is used. That is, the “extended technique” is a technique condition such as a difference that does not conform to the singing technique defined by the technique condition or a difference that does not satisfy the technique condition between the singing synthesis f0 transition and the original f0 transition. Is the content of the difference when there is an incompatible difference.

  And in a singing technique determination process, the control part 6 produces | generates technique data based on the singing technique determined by each step from S310 to S360 (S370). The technique data referred to here is data representing the type of singing technique and the timing using the singing technique.

Then, this singing technique determination process is complete | finished, and it returns to S200 of a music analysis process.
That is, in the singing technique determination process, the singing synthesis f0 transition and the original f0 transition are compared. As a result of the comparison, it is determined whether or not technical conditions defined in advance as conditions for various singing techniques used in singing are satisfied. And as a result of the determination, if the technique condition is satisfied, the type of the singing technique and the timing using the singing technique that match the technique condition are specified, and the type of the specified singing technique and the timing using the singing technique are determined. Generate technique data to represent.
<Performance processing>
Next, the performance process which the karaoke apparatus 30 performs is demonstrated.

  When the performance process shown in FIG. 5 is started, the control unit 50 first acquires the song ID of the song (ie, the designated song) designated via the input receiving unit 34 (S410). And the control part 50 acquires the MIDI music MD corresponding to music ID acquired by S410 from the memory | storage part 14 of the information processing server 10 (S420).

  Subsequently, in the performance process, the control unit 50 acquires song data EM including the song ID acquired in S410 from all song data EM stored in the storage unit 14 of the information processing server 10. (S430). Further, in the performance process, the control unit 50 acquires lyric data from the MIDI music MD corresponding to the music ID acquired in S420 (S440).

  In the performance process, the control unit 50 starts playing the designated music based on the MIDI music MD acquired in S420 (S450). In the performance of the designated music based on the MIDI music MD in S450, the control unit 50 sequentially outputs the MIDI music MD to the music playback unit 36 along the time axis. The music reproducing unit 36 that has acquired the MIDI music MD performs the music. Then, the sound source signal of the music played by the music playback unit 36 is output to the speaker 60 via the output unit 42. Then, the speaker 60 outputs the sound source signal instead of sound.

  Further, in the performance process, the control unit 50 outputs the lyrics data acquired in S430 to the video control unit 46 (S460). The video control unit 46 that has acquired the lyrics data displays the lyrics on the display unit 64 in conjunction with the performance of the designated music by the music playback unit 36.

  In the performance process, the control unit 50 outputs the singing technique to be used for the designated music to the video control unit 46 based on the technique data included in the song data EM (S470). The video control unit 46 that has acquired the singing technique displays the singing technique on the display unit 64 in conjunction with the performance of the designated music piece by the music reproducing unit 36. Note that the singing technique displayed on the display unit 64 includes an icon representing the singing technique as shown in FIG. The icons representing this singing technique are the types of “shrimp”, “tame”, “vibrato”, “fist”, “facial expression”, “expansion technique”, and the timing of performing each singing technique. Is shown.

  That is, in S460, when the designated music is played, based on the song data EM, the notification process for notifying the characteristics of the singing method used by the professional singer who sang the designated music is executed as one of the regulation processes. To do.

  Subsequently, in the performance process, the control unit 50 acquires the voice input through the microphone 62 and the microphone input unit 44 as singing voice data (S480). Then, the control unit 50 calculates the singing voice f0 transition based on the singing voice data acquired in S530 (S490). The singing voice f0 transition here represents the transition of the fundamental frequency of the voice represented by the singing voice data. In addition, the calculation method of singing voice f0 transition should just use the method similar to S150 of a music analysis process except the data made into the execution object of a process being "singing voice data". For this reason, detailed description of the calculation method of the singing voice f0 transition is omitted.

  Moreover, the control part 50 changes the singing voice f0 transition calculated by S490 based on the difference data contained in the song data EM acquired in S430 (S500). In S500, the singing voice f0 transition is changed so as to approach the singing voice when the professional singer sings the designated music piece. Specifically, as shown in FIG. 7, difference data is added to the transition of the singing voice f0 along the time axis of the designated music. Thereby, the characteristic of the way of singing which the professional singer used with the designated music is reflected in the change sound mentioned later.

  Further, in the performance process, the control unit 50 outputs a changed sound that is a sound based on the transition of the singing voice f0 changed in S500 (S510). In S <b> 510, the control unit 50 outputs a sound source signal representing the changed sound to the output unit 42. The sound source signal is output to the speaker 60 via the output unit 42, returned to the changed sound by the speaker 60, and output.

  Subsequently, the control unit 50 calculates singing difference data (S520). The singing difference data referred to here is data representing a difference along the time axis in the designated music between the singing synthesis f0 transition and the singing voice f0 transition included in the singing time data EM acquired in S430.

  Further, the control unit 50 compares the singing difference data calculated in S520 with the difference data (S530). And as a result of the comparison in S530, if the correlation (namely, the degree of coincidence) between the singing difference data and the difference data is equal to or greater than a predetermined value, the control unit 50 plays the specified musical piece. It is specified that the characteristics of the singing method used by the professional singer who has sang the song can be reproduced (S540).

  Subsequently, the control unit 50 determines whether or not the performance of the designated music has been completed (S550). As a result of the determination in S550, if the performance of the designated music has not ended (S550: NO), the control unit 50 returns the performance processing to S460.

  On the other hand, if the result of determination in S550 is that the performance of the designated music has been completed (S550: YES), the control unit 50 evaluates the singing voice of the designated music by the user (S560). In the evaluation in S560, as shown in FIG. 8, the degree of coincidence between the singing difference data and the difference data is calculated for each section defined in the designated music, and as the calculated result of the degree of coincidence increases, What is necessary is just high evaluation. Note that the degree of coincidence may be calculated by comparing the singing difference data and the difference data and calculating the correlation coefficient in S530. Further, the calculation result of the degree of coincidence referred to here may be the sum of the degrees of coincidence calculated for each specified section, the result of weighted addition, a representative value, or other It may be a calculation result.

  That is, in S560, if the user's singing voice is able to reproduce the characteristics of the singing method used by the professional singer who sang the designated music piece, an evaluation process that is highly evaluated is executed as one of the defining processes.

Thereafter, the control unit 50 ends the performance process.
[Effect of the embodiment]
The score data in this embodiment has notes composed of combinations of pitches and note values, and the score data does not indicate singing characteristics such as the singing technique used in the designated music. For this reason, the singing synthesized sound does not show the characteristics of the singing technique such as the singing technique, and the singing synthesis f0 transition is the transition of the standard frequency f0 of the standard singing voice in the designated music.

On the other hand, the singing voice data is a voice when a professional singer sings a designated music piece, and expresses the characteristics of how the professional singer sings the designated music piece.
Therefore, the difference data calculated by the music analysis process represents the characteristics of the singing method used by the professional singer, and the difference data represents the characteristics of the singing method by the professional singer. .

In other words, according to the music analysis process, data representing characteristics of how to sing by a professional singer can be automatically generated.
Also, in the music analysis process, technique data representing the type of singing technique used by a professional singer and timing using the singing technique in the designated music is generated, and the data in which the technique data and the difference data are associated with each other It is stored in the storage unit as data EM.

As a result, according to the music analysis process, the contents of the song data EM can be enriched, and data representing in more detail how to sing a professional singer can be generated as the song data EM.
Also, in the music analysis process, even if the difference between the singing composition f0 transition and the original f0 transition is not enough to satisfy the technical conditions, the content of the difference is sung by the professional singer who sang the designated music Data (ie, “extended technique”).

As a result, according to the data generation device, more detailed contents can be included in the song data EM as the characteristics of the singing method used by a specific person.
In the performance process, the singing technique used by the professional singer in the designated music can be notified. For this reason, the user of the karaoke apparatus 30 can recognize the singing technique used by the professional singer who sang the designated music while singing the designated music.

  In the performance process, the singing voice of the user is changed and output so as to approach the way of singing a professional singer in the designated music piece. For this reason, according to performance processing, a user's singing voice can be brought close to how to sing a professional singer.

  Furthermore, in performance processing, the higher the degree of coincidence between the singing difference data and the difference data, the higher the evaluation. For this reason, according to the performance processing, the higher the reproducibility, the higher the evaluation can be made by a professional singer who sang the designated music.

  By the way, in the karaoke system 1 of the said embodiment, the object compared with the original f0 transition in a music analysis process and the object compared with the singing voice f0 transition in a performance process are made into singing synthetic | combination f0 transition.

  In this way, according to the performance processing, the singing voice of the user of the karaoke apparatus 30 can be evaluated regardless of the transition mode (hereinafter referred to as f0 transition mode) of the fundamental frequency f0 characteristic of the singing synthesis f0 transition. .

  As an example of the “f0 transition mode” mentioned here, as shown in FIG. 9, when singing the lyrics of “ka line”, “sa line”, “ta line” or its muddy sound, the fundamental frequency f0 For example, the transition is interrupted before the performance start timing of the note to which the lyrics are assigned. As an example of the “f0 transition mode”, as shown in FIG. 10, when the lyrics of “A line”, “N line”, and “La line” are sung, the time to the note to which the lyrics are assigned is shown. It is mentioned that the transition of the fundamental frequency is smoothly connected from the immediately preceding note along the axis (continuous pitch change in FIG. 10).

Furthermore, as an example of the “f0 transition mode”, as shown in FIG. 11, “na rows”, in which notes having the same pitch continue along the time axis and are assigned to subsequent notes along the time axis, Or, when the lyrics of “ma line” are sung, the transition of the fundamental frequency may shift from a pitch lower than the pitch of the note to which the lyrics are assigned to the pitch (in FIG. 11). Pitch fluctuation).
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  (1) The song data EM in the above embodiment is data representing the singing of a professional singer in the designated song, but this song data EM is data representing the singing of a specific person in the designated song. If so, that particular person may not be a professional singer. That is, the specific person may be a general person who sang the designated music piece. In this case, the song data EM includes difference data representing the difference along the time axis in the designated music between the original f0 transition indicating the transition of the fundamental frequency of the singing voice sung by the specific person and the singing synthesis f0 transition. Any data may be used as long as the data is included.

  (2) In the above embodiment, the execution subject of the music analysis process is the information processing device. However, the execution subject of the music analysis process may be the information processing server 10 or the karaoke device 30. .

  (3) The aspect which abbreviate | omitted a part of structure of the said embodiment is also embodiment of this invention. Further, an aspect configured by appropriately combining the above embodiment and the modification is also an embodiment of the present invention. Moreover, all the aspects which can be considered in the limit which does not deviate from the essence of the invention specified by the wording described in the claims are the embodiments of the present invention.

(4) The present invention can be realized in various forms such as the data generation apparatus and karaoke system 1 described above, a program for causing a computer to function as the data generation apparatus, and a medium on which the program is recorded. it can.
<Example of correspondence>
The function obtained by executing S140 in the music analysis process is an example of a synthetic sound acquisition unit. The function obtained by executing S150 is an example of a composite transition deriving unit. The function obtained by executing S160 is an example of the original acquisition unit. The function obtained by executing S170 is an example of singing transition deriving means. The function obtained by executing S180 is an example of the difference calculation means. The function obtained by executing S200 is an example of a registration unit.

  Moreover, the function obtained by performing S310-S360 in a singing technique determination process is an example of a technique determination means. The function obtained by executing S370 is an example of the technique data generation means.

  The function obtained by executing S470, S500, S510, S530, and S560 in the performance process is an example of the process execution means. The function obtained by executing S480 is an example of a singing voice acquisition unit. The function obtained by executing S490 is an example of singing voice transition deriving means. The function obtained by executing S520 is an example of the difference generation unit.

  DESCRIPTION OF SYMBOLS 1 ... Karaoke system 2 ... Information processing apparatus 3 ... Input reception part 4 ... Information output part 5, 14, 38 ... Memory | storage part 6, 16, 50 ... Control part 7, 18, 52 ... ROM 8, 20, 54 ... RAM 9 22, 22 ... CPU 10 ... Information processing server 12 ... Communication unit 30 ... Karaoke device 32 ... Communication unit 34 ... Input reception unit 36 ... Music playback unit 40 ... Audio control unit 42 ... Output unit 44 ... Microphone input unit 46 ... Video Control unit 60 ... Speaker 62 ... Microphone 64 ... Display unit

Claims (9)

Score data representing the score of a designated song that is a designated song having a plurality of notes composed of combinations of pitches and note values, and lyrics assigned to at least a part of the plurality of notes constituting the designated song Synthesized sound acquisition means for acquiring singing synthesized sound data that is generated based on the lyric data that is expressed and synthesized by voice synthesis of the lyrics represented by the lyric data according to the score represented by the score data;
Synthesis transition deriving means for deriving a singing synthesis f0 transition representing a transition of the fundamental frequency of the synthesized speech from the singing synthesized sound data acquired by the synthesized sound acquiring means;
Original acquisition means for acquiring singing voice data which is a singing voice of a specific person singing the designated music;
Singing transition deriving means for deriving an original f0 transition representing the transition of the fundamental frequency of the singing voice from the singing voice data acquired by the original acquiring means;
Difference calculating means for calculating difference data representing a difference along the time axis in the designated music between the singing composition f0 transition derived by the composition transition deriving means and the original f0 transition derived by the singing transition deriving means;
A data generation apparatus comprising: registration means for storing difference data calculated by the difference calculation means in a storage unit. Singing voice data indicated by the original f0 transition with respect to the singing synthesis f0 transition as a result of comparing the singing synthesis f0 transition derived by the synthetic transition derivation means with the original f0 transition derived by the singing transition derivation means. Is a technique judging means for judging whether or not a technique condition defined in advance as a condition of various singing techniques used in singing is satisfied,
As a result of determination by the technique determination means, when the technique condition is satisfied, the type of singing technique and timing using the singing technique that meet the technique condition are specified, and the specified singing technique type and singing technique are specified. Technical data generation means for generating technical data representing timing using
The registration means includes
The data according to claim 1, wherein song data, which is data in which the technique data generated by the technique data generating means and the difference data are associated with each other along a time axis in the designated music, is stored in the storage unit. Generator. The technique data generation means includes
Even if the technique condition is not satisfied as a result of the comparison by the technique determination means, if there is a difference greater than a predetermined value between the singing synthesis f0 transition and the original f0 transition, the difference The data generation apparatus according to claim 2, wherein the technique data and the timing at which the difference exists are generated as the technique data. Score data representing the score of a designated song that is a designated song having a plurality of notes composed of combinations of pitches and note values, and lyrics assigned to at least a part of the plurality of notes constituting the designated song Synthesized sound acquisition means for acquiring singing synthesized sound data that is generated based on the lyric data that is expressed and synthesized by voice synthesis of the lyrics represented by the lyric data according to the score represented by the score data;
Synthesis transition deriving means for deriving a singing synthesis f0 transition representing a transition of the fundamental frequency of the synthesized speech from the singing synthesized sound data acquired by the synthesized sound acquiring means;
Performance execution means for playing the designated music according to the score data;
Original acquisition means for acquiring singing voice data, which is a singing voice of a specific person singing the designated music played by the performance executing means;
Singing transition deriving means for deriving an original f0 transition representing the transition of the fundamental frequency of the singing voice from the singing voice data acquired by the original acquiring means;
Difference calculating means for calculating difference data representing a difference along the time axis in the designated music between the singing composition f0 transition derived by the composition transition deriving means and the original f0 transition derived by the singing transition deriving means;
Registration means for storing the difference data calculated by the difference calculation means in a storage unit;
A karaoke system comprising: process execution means for executing a specified process, which is a process specified for the performance of the designated music, based on the difference data stored in the storage unit. The process execution means includes
When the designated music is played by the performance executing means, based on the difference data stored in the storage unit, the type of singing technique used by the specific person who sang the designated music and the timing using the singing technique The karaoke system according to claim 4, wherein a notification process for notifying is performed as the defining process. Singing voice acquisition means for acquiring singing voice data which is voice input via a microphone during the performance of the designated music piece by the performance execution means;
Singing voice transition deriving means for deriving a singing voice f0 transition representing a transition of the fundamental frequency of the voice based on the singing voice data acquired by the singing voice acquisition means,
The process execution means includes
The karaoke system according to claim 4 or 5, wherein a process based on the singing voice f0 transition derived by the singing voice transition deriving unit is executed as the defining process. The process execution means includes
Difference generating means for generating singing difference data representing a difference along the time axis in the designated music between the singing voice f0 transition derived by the singing voice transition deriving means and the singing synthesis f0 transition derived by the synthetic transition deriving means. ,
The singing difference data generated by the difference generating means and the difference data stored in the storage unit are compared, and as a result of the comparison, an evaluation process with a higher evaluation as the degree of coincidence is defined as the prescribed process. The karaoke system according to claim 6, which is executed. The process execution means includes
Based on the difference data stored in the storage unit, the singing voice f0 transition derived by the singing voice transition deriving means is changed, and a changing process for outputting a sound based on the changed singing voice f0 transition is executed as the prescribed process. The karaoke system according to claim 6 or 7. Score data representing the score of a designated song that is a designated song having a plurality of notes composed of combinations of pitches and note values, and lyrics assigned to at least a part of the plurality of notes constituting the designated song A synthetic sound acquisition procedure for acquiring singing synthetic sound data that is generated based on the lyric data that is expressed and synthesized by synthesizing the lyrics represented by the lyric data according to the musical score represented by the musical score data;
A synthesis transition derivation procedure for deriving a singing synthesis f0 transition representing a transition of the fundamental frequency of the synthesized speech based on the singing synthesized sound data acquired in the synthetic sound acquisition procedure;
An original acquisition procedure for acquiring singing voice data which is a singing voice of a specific person singing the designated music;
A singing transition derivation procedure for deriving an original f0 transition representing a transition of the fundamental frequency of the singing voice based on the singing voice data acquired in the original acquisition procedure;
A difference calculation procedure for calculating difference data representing a difference along the time axis in the designated music between the singing composition f0 transition derived in the synthesis transition derivation procedure and the original f0 transition derived in the singing transition derivation procedure;
A registration procedure for storing the difference data calculated in the difference calculation procedure in a storage unit;
A program to be executed by a computer.