JP2019090936A - Singing support device and karaoke device - Google Patents

Abstract

To provide a technique capable of supporting singing without interrupting singing of a singer.SOLUTION: A singing support device for supporting singing of a singer comprises: a guide melody synthesis section for generating two or more types of guide melodies which correspond to music to be sung and guide singing of the singer by voice synthesis; and a guide melody output section for switching two or more types of guide melodies and outputting the melodies by adjusting them to the music.SELECTED DRAWING: Figure 3

Description

  The present invention relates to the technology of a singing support device.

  As a conventional communication karaoke apparatus, there is known a singing support apparatus that supports a singer by a guide melody. The guide melody is the main melody played to induce singing when the singer can not sing along with the performance of the karaoke song, and a model vocal voice (see, for example, Patent Document 1 and Patent Document 2). .

JP, 2005-148627, A JP 2005-049410 A

  However, if a singer gives a normal guide vocal to a portion where he can sing, he or she will be covered with the singer's singing, so the singer may be confused and difficult to sing or the audience may be difficult to hear. Therefore, there has been a demand for a technology that can support singing without interfering with the singing of the singer.

  The present invention has been made to solve the above-described problems, and can be realized as the following modes.

(1) According to one aspect of the present invention, there is provided a singing support device for supporting singing by a singer. The singing support device generates a guide melody corresponding to the song to be sung and guides the singing of the singer by two or more types by speech synthesis; and a guide melody synthesizing unit; and the two or more types of guide melody And a guide melody output unit configured to switch and output according to the music. According to the singing support device of this aspect, since it is possible to switch and output a plurality of types of guide melody, it is possible to support melody sound without having a form suitable for singing of the singer, for example, singing and singing of the singer . Therefore, singing support can be performed without interfering with the singing of the singer.

(2) In the song support device of the above aspect, the guide melody synthesis unit may perform speech synthesis using an acoustic model in which acoustic parameters are learned for the guide melody by a statistical method. According to the singing support device of this aspect, it is possible to generate a guide melody by speech synthesis with a small amount of data.

(3) In the song support apparatus of the above aspect, the guide melody synthesis unit is configured to generate at least a melody sound without lyrics and a vocal sound with lyrics by voice synthesis using the acoustic model. It may be generated as a melody. According to the singing support device of this aspect, it is possible to support the melody sound without the singer singing and being covered by the melody sound without the lyrics. Therefore, singing support can be performed without interfering with the singing of the singer.

(4) In the song support apparatus of the above aspect, the synthesis of the melody sound is performed based on a first acoustic model obtained by learning a song sound without lyrics such as humming and scat of the music, The synthesis of the vocal sound may be performed based on a second acoustic model obtained by learning a singing sound using the lyrics of the music. According to the singing support device of this aspect, it is possible to easily generate two or more types of guide melodies by speech synthesis.

(5) The singing support apparatus according to the above aspect, further comprising: a singing situation determination unit that determines the degree of the singing of the singer; the guide melody output unit determines the guide melody according to the determined degree of the singing You may switch. According to the singing support device of this aspect, since the guide melody can be switched and output according to the singing situation, the melody sound can be supported more effectively without being covered by the singing person and singing.

  The present invention can be realized in various aspects. For example, the present invention can be realized in the form of a karaoke apparatus, a karaoke system, a singing support method, a computer program or the like provided with the singing support device of this form. The computer program may be recorded on a non-transitory tangible recording medium readable by a computer.

It is explanatory drawing which shows the outline | summary of the karaoke apparatus containing a song assistance apparatus. It is a figure which shows the various acoustic parameters contained in an acoustic model. It is a flowchart showing guide melody output processing. It is explanatory drawing which shows the outline | summary of the karaoke apparatus containing the song assistance apparatus in 2nd Embodiment. It is the figure which defined the relationship between an average volume, singing likelihood, an average pitch difference, and a guide melody.

A. First embodiment:
FIG. 1 is an explanatory view showing an outline of a karaoke apparatus 200 including a singing support apparatus 100 according to an embodiment of the present invention. The karaoke apparatus 200 includes a singing support apparatus 100, a music playing apparatus 105, and a mixing amplifier 50. The singing support device 100 is a device that supports singing of a singer by a guide melody in the karaoke device 200. In the present embodiment, the singing support device 100 includes a guide melody synthesis unit 10, a singing situation determination unit 20, and a guide melody output unit 30.

  The guide melody synthesizing unit 10 generates two or more types of guide melodies for one music by speech synthesis. The “guide melody” is a main melody for guiding a singer of the singer or a vocal vocal model. In the present embodiment, the guide melody synthesis unit 10 performs speech synthesis using an acoustic model in which acoustic parameters are learned by a statistical method. More specifically, the guide melody is generated using a Hidden Markov Model (hereinafter also described as HMM (Hidden Markov Model)) or a deep neural network (hereinafter also described as DNN (Deep Neural Network)). Details of the acoustic parameters used to learn the acoustic model will be described later.

  The singing situation determination unit 20 determines the degree of singing (song voice) of a singing person input from the voice input unit 40 (for example, a microphone). The degree of singing means the degree of whether the singer can sing the song. In the present embodiment, the singing situation determination unit 20 determines, for example, based on the volume, the pitch, and the lyrics of the input singing.

  The guide melody output unit 30 switches the plurality of guide melodies generated by the guide melody synthesis unit 10 according to the degree of singing determined in the singing situation determination unit 20, and outputs the same to the speaker 60 via the mixing amplifier 50. .

  The music playing device 105 is a device that plays music in the karaoke device 200. The music playing device 105 includes a storage unit 15, a control unit 25, and a music output unit 35. The music playing device 105 is operated by the operation unit 70 (for example, a remote control), and a music to be played is selected.

  The storage unit 15 acquires and stores data of music to be played from the network NW, for example, performance data (hereinafter referred to as "MIDI data") of an accompaniment musical tone created in accordance with the MIDI standard. The control unit 25 controls the music output unit 35 in accordance with the operation instruction input from the operation unit 70. For example, control of the order in which the music is played, and control of fast-forwarding and pause of the music are performed. The storage unit 15 may store a part or all of the MIDI data in advance without acquiring the MIDI data from the network NW.

  The music output unit 35 acquires MIDI data from the storage unit 15, decodes the MIDI data to generate a karaoke music to be played, and outputs the karaoke music to the speaker 60 via the mixing amplifier 50. The mixing amplifier 50 outputs the guide melody, the singing of the singer input from the voice input unit 40, and the karaoke music, from the speaker 60 that is synthesized.

  FIG. 2 is a view showing various acoustic parameters modeled by the acoustic model. The fundamental frequency is generally treated as a logarithmic fundamental frequency pt, and as the related parameter, the voiced / unvoiced distinction, the first derivative (Δpt) and the second derivative (Δ2pt) of the logarithmic fundamental frequency can be considered. These are sometimes referred to as sound source information. The unvoiced portion has no value of the logarithmic fundamental frequency pt. For this reason, voiced / voiceless distinction is made by a method such as putting a predetermined constant in the unvoiced part. Further, as the spectrum parameter, there are Mel Cepstrum ct, its first derivative (Δct), second derivative (Δ2ct) and the like. These are sometimes referred to as spectral information. Furthermore, in addition to such sound source information and spectrum information, the present embodiment deals with singing expression information.

  The song expression information includes a vibrato period V1ft and an amplitude V1at of the pitch, and a vibrato period V2ft and an amplitude V2at of the sound magnitude, which are modeled in phoneme units. The period of the pitch vibrato, the amplitude of the pitch vibrato, the period of the vibrato of the loudness, and the amplitude of the vibrato of loudness are also corresponding to the temporal derivative (Δ) and the second derivative (Δ2) However, for convenience of illustration, in FIG. 2, illustration of the first derivative and the second derivative of these periods and amplitudes is omitted. Among the above-mentioned parameters, the first derivative and the second derivative of each parameter including the mel cepstrum ct are used to consider time variation. By considering the dynamic feature, the connection between the sound and the sound when synthesizing the singing voice becomes smooth. The description of the speech synthesis method using dynamic features is omitted.

  When DNN is used as an acoustic model, a spectrum may be modeled instead of mer cepstrum ct, or a speech waveform may be modeled as an acoustic parameter instead of the acoustic parameter.

  In the present embodiment, the acoustic model is a first acoustic model obtained by learning a singing sound without lyrics such as humming or scat, and a second acoustic learning model obtained by learning a singing sound using the lyrics. And the acoustic model of

  FIG. 3 is a flowchart showing guide melody output processing in the present embodiment. The guide melody output process is a process that is started at the same time as playing a music in karaoke, and is a process for switching the guide melody according to the degree of singing. For convenience of illustration, the guide melody is described as a guide. The guide melody output unit 30 first sets the guide melody to be output to "melody sound" (step S100). The "melody sound" is a melody-only sound without lyrics. As the melody sound, for example, an instrumental sound or a sound by humming is used.

  Next, the singing situation determination unit 20 determines whether the reproduction portion of the current music is a singing section (step S110). When the reproduction portion of the music is not in the singing section (step S110: NO), the singing situation judging unit 20 judges whether the music is ended because the singing voice is not switched according to the degree of the singing (step S210).

  When the reproduction portion of the music is a singing section (step S110: YES), the singing situation determination unit 20 acquires the singing of the singer from the voice input unit 40 (step S120). The singing situation determination unit 20 calculates the average volume of the acquired singing (step S130). The average volume can be determined, for example, as an average of the volume of singing in a section of a predetermined length.

  Next, the singing situation determination unit 20 determines whether the calculated average volume is equal to or more than a predetermined threshold (step S140). The threshold can be set arbitrarily. If the average volume is smaller than the threshold (step S140: NO), in other words, if it is determined that singing is not possible, the guide melody output unit 30 sets the guide melody to be output to "vocal sound" (step S165), It transfers to step S210. "Vocal sound" is a singing voice with lyrics. In the present embodiment, the guide melody of the vocal sound is generated by speech synthesis using the second acoustic model described above.

  If the average volume is equal to or higher than the threshold (YES in step S140), in other words, if it is determined that singing is possible, the singing situation determination unit 20 calculates the singing likelihood of the singing obtained in step S120 (step S150). The singing likelihood indicates the likelihood of singing in a section of a predetermined length, more specifically, the degree of correctness of the lyrics and the score split (the pronunciation timing of the lyrics). The likelihood of singing becomes higher when singing according to the score.

  Since the lyrics are known in advance, the correctness / incorrectness judgment of the lyrics in the calculation of the singing likelihood may use, for example, voice recognition or may be judged by the correctness of the phonemes. Further, the correctness determination of the score may be made, for example, by obtaining a phoneme alignment indicating which timing each phoneme of the singing voice occupies and which timing, and the degree of matching with the note boundary of the score.

  Next, the singing situation determination unit 20 determines whether the calculated singing likelihood is equal to or more than a predetermined threshold (step S160). The threshold can be set arbitrarily. If the singing likelihood is smaller than the threshold (step S160: NO), in other words, if it is determined that singing is not possible, the guide melody output unit 30 sets the guide melody to be output to "vocal sound" (step S165) , Shift to step S210. The “vocal sound” set in step S165 immediately after step S140 and the “vocal sound” set in step S165 immediately after step S160 may not be the same guide melody, for example, the volume etc. are different. It may be

  If the singing likelihood is equal to or higher than the threshold (step S160: YES), in other words, if it is determined that singing is possible, the singing situation determination unit 20 calculates the average pitch difference of the singing obtained in step S120 (step S170). ). The average pitch difference can be obtained, for example, as an average of absolute values of differences between the pitch of a song in a section of a predetermined length and the pitch of a score. The pitch can be determined, for example, by extracting the fundamental frequency. Note that, instead of the average pitch difference, for example, the number of times a semitone or more is removed in singing in a section having a predetermined length may be used.

  Next, the singing situation determination unit 20 determines whether the calculated average pitch difference is equal to or less than a predetermined threshold (step S180). The threshold can be set arbitrarily. If the average pitch difference is larger than the threshold (step S180: NO), in other words, if it is determined that the correct pitch can not be used for singing, the guide melody output unit 30 sets the output guide melody to "melody sound". Then, the process proceeds to step S210 (step S195). In the present embodiment, the guide melody of the melody sound is generated by speech synthesis using the first acoustic model described above.

  If the average pitch difference is equal to or less than the threshold (YES in step S180), in other words, if it is determined that singing at the correct pitch is possible based on the average volume, the singing likelihood and the average pitch difference, the guide melody output unit 30 determines whether the guide is set to "melody sound" (step S190). When the guide is not the "melody sound" (step S190: NO), the guide melody output unit 30 sets the guide melody to be output to the "melody sound" (step S195), and proceeds to step S210.

  If the guide is "melody sound" (step S190: YES), the guide melody output unit 30 determines whether a predetermined time has elapsed since the guide was changed to "melody sound" (step S200). The predetermined time can be set arbitrarily. If the predetermined time has not elapsed (step S200: NO), the guide melody output unit 30 sets the guide melody to be output to "absent" (step S205), and the process proceeds to step S210. Even if the guide melody output unit 30 determines that the singing situation determination unit 20 can sing, the guide melody to be output remains "melody sound" until the predetermined time elapses, and after the predetermined time elapses By setting the guide melody to be output as "none", the singer can support singing until the degree of singing becomes stable.

  If the predetermined time has elapsed (step S200: YES), the singing situation determination unit 20 determines whether the music is finished (step S210).

  If the music has not ended (step S210: NO), the process is repeated from step S110. If the music has ended (step S210: YES), the guide melody output process ends.

  According to the song support device 100 of the present embodiment described above, the guide melody output unit 30 switches the guide melody according to the degree of singing. More specifically, when the average volume is less than the threshold and when the singing likelihood is less than the threshold, in other words, when not singing, the guide melody output unit 30 sets the guide melody to the vocal sound. In addition, when the average volume and the singing likelihood are equal to or higher than the threshold and the average pitch difference is smaller than the threshold, in other words, if the lyrics are being sung, the guide melody output unit 30 sets the guide melody to the melody sound. When the average volume, the singing likelihood, and the average pitch difference are equal to or more than the threshold, in other words, when singing, the guide melody output unit 30 sets the guide melody to none. Therefore, the singing support device 100 can be performed without disturbing the singing of the singer.

  Further, in the present embodiment, the singing situation determination unit 20 sequentially determines the average volume, the singing likelihood, and the average pitch difference. For example, when the average volume is less than a threshold, the singing likelihood calculation / determination The guide melody output unit 30 sets the guide melody to the vocal sound without performing the calculation / determination of the average pitch difference. Therefore, the processing of the singing situation judging unit 20 can be minimized.

  Further, in the present embodiment, the guide melody synthesis unit 10 performs speech synthesis using an acoustic model in which acoustic parameters have been learned by a statistical method. Therefore, it is possible to generate a guide melody by speech synthesis without recording speech for each song.

  Further, in the present embodiment, the guide melody synthesis unit generates the melody sound without the lyrics using the first acoustic model obtained by learning the singing sounds without the lyrics, and the singing sound using the lyrics Using the second acoustic model obtained by learning and generating vocal sounds with lyrics. Therefore, the guide melody synthesizing unit 10 can easily generate two or more types of guide melodies. Moreover, the melody sound without the lyrics can support the melody sound without being sung by the singer. Therefore, singing support can be performed without interfering with the singing of the singer.

B. Second embodiment:
FIG. 4 is an explanatory view showing an outline of the karaoke apparatus 200 including the singing support device 100 in the second embodiment. The song support device of the second embodiment differs from the song support device 100 of the first embodiment in that the operation instruction is received from the operation unit 70A, but the other configuration is the same as the song support device 100 of the first embodiment. Therefore, the operation unit 70A will be described, and the description of the other configurations will be omitted.

  The operation unit 70A controls switching of the guide melody. More specifically, when the singer operates the operation unit 70A, the guide melody output by the guide melody output unit 30 can be switched. The operation unit 70A includes the buttons 71-73. The button 71 sets the guide melody to "vocal sound", the button 72 sets the guide melody to "melody sound", and the button 73 sets the guide melody to "none".

  For example, when the singer performs an operation to switch the guide melody from the vocal sound to the melody sound, more specifically, when the singer selects the button 72, the guide melody output unit 30 guides the user through the operation unit 70A. In response to the melody switching instruction, the guide melody is switched from the vocal sound to the melody sound.

  According to the song support apparatus 100 of the present embodiment described above, the guide melody output unit 30 follows the request of the singer in order to switch the guide melody output by the guide melody output unit 30 by the operation of the singer. You can switch the guide melody. Therefore, singing support can be performed without interfering with the singing of the singer.

C. Other embodiments:
In the above-described embodiment, the guide melody synthesis unit 10 performs speech synthesis using an acoustic model in which acoustic parameters are learned by a statistical method. Instead of this, the guide melody synthesizing unit 10 may perform speech synthesis using a waveform connection method. Further, a guide melody generated by speech synthesis using an acoustic model in which acoustic parameters are learned by a statistical method and a guide melody generated by speech synthesis performed using a waveform connection method may be switched and output.

  In the above embodiment, the guide melody output unit 30 switches and outputs two or more types of guide melodies generated by the guide melody combining unit 10, but the guide melody combining unit 10 generates the guide melody in real time. May be generated. For example, the guide melody synthesis unit 10 may generate a guide melody by switching an acoustic model in which acoustic parameters used for speech synthesis are learned according to the degree of singing.

  Further, in the above embodiment, the guide melody synthesis unit 10 generates a melody sound without lyrics using the first acoustic model obtained by learning a singing sound without lyrics, and sings using the lyrics. Although the vocal sound having the lyrics is generated using the second acoustic model obtained by learning the sound, the type of guide melody and the acoustic model are not limited to this. Instead of this, for example, the guide melody synthesis unit 10 uses the acoustic model obtained by sampling the vocal sound of the male voice using the acoustic model obtained by sampling the singing voice by the male voice and the acoustic sound obtained by sampling the singing voice by the female voice. And the vocal sound of the female voice may be generated.

  In the above embodiment, the guide melody output unit 30 switches the guide melody between “vocal sound”, “melody sound”, and “none” according to the degree of singing, but the present invention is not limited to this. For example, in FIG. 3, steps S200 and S205 may be omitted to always output a guide melody.

  Further, in the above embodiment, the singing situation judging unit 20 judges the degree of singing according to the volume, the lyrics, and the pitch of the input song, more specifically, the average volume, the singing likelihood and the average volume difference. However, the determination of the degree of singing is not limited to these factors. For example, elements such as intonation, vibrato, and crouching may be used.

  In the above embodiment, when there are a plurality of singers, the degree of singing may be determined for each singer and the guide state may be adjusted to the person who can sing correctly. According to this aspect, a singer who is not singing can guide the singing voice of the singing singer who is singing, so the guidance of singing by the guide melody can be minimized. Therefore, singing support can be performed without interfering with the singing of the singer.

  Further, in the above embodiment, the range of the length of the song used to determine the degree of singing may be a range divided based on a predetermined rule other than time. As the rule, for example, a measure or a phrase can be used.

  In the above embodiment, the length of the song used to determine the degree of singing may be longer as the numerical value used for the determination is closer to the threshold and shorter as it is farther from the threshold. For example, one phrase may be used if it is long, and one measure may be used if it is short. According to this aspect, the guide melody output unit 30 can immediately switch to an appropriate guide melody according to the degree of singing. In addition, when it is difficult to determine whether you are singing or not singing, more careful and accurate determination can be performed.

  Further, in the above embodiment, the singing used for the determination may be weighted by the time going back from the present to calculate the average. For example, the weight may be increased as the song used for determination is closer to the current state, and the weight may be smaller as it goes back to the past. According to this aspect, it is possible to take into consideration the degree of the past singing while reacting promptly to the degree of the latest singing, so that the determination result of the degree of the singing becomes stable.

  In the above embodiment, the average volume and the average pitch difference are used to determine the degree of singing, but instead, other statistical criteria such as median or mode may be used. .

  Further, in the above-described embodiment, the singing situation determination unit 20 sequentially determines the average volume, the singing likelihood, and the average pitch difference with respect to the degree of singing. On the other hand, the singing situation judging unit 20 may judge the degree of singing based on a map or a function in which the relationship between the average sound volume and the singing likelihood and the average pitch difference is defined. FIG. 5 is a diagram defining the relationship between the average volume, the singing likelihood, the average pitch difference, and the guide melody. According to FIG. 5, when the average sound volume is less than the threshold and the singing likelihood and the average pitch difference are equal to or more than the threshold, the guide melody is set to no. The first vocal sound, the second vocal sound, and the third vocal sound have different volumes. In addition, FIG. 5 is only an example about the kind of guide melody set according to the degree of singing.

  In the first embodiment, the singing situation determination unit 20 determines the degree of singing. Alternatively, the degree of singing may be determined using a scoring function provided in the karaoke apparatus 200. For example, it is possible to use a criterion as to whether the score score in real time by the scoring function exceeds a threshold.

  The present invention is not limited to the above-described embodiment, and can be realized in various configurations without departing from the scope of the invention. For example, the technical features in the embodiments corresponding to the technical features in the respective forms described in the section of the summary of the invention are for solving the problems described above or for achieving some or all of the effects described above It is possible to replace or combine as appropriate. Also, if the technical features are not described as essential in the present specification, they can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Guide melody synthetic | combination part 15 ... Memory | storage part 20 ... Song condition determination part 25 ... Control part 30 ... Guide melody output part 35 ... Music output part 40 ... Speech input part 50 ... Mixing amplifier 60 ... Speaker 70, 70A ... Operation part 71 , 72, 73 ... button 100 ... singing support device 105 ... music playing device 200 ... karaoke device NW ... network

The present invention has been made to solve the above-described problems, and can be realized as the following modes. The present invention can be implemented in the following forms. According to one aspect of the present invention, there is provided a singing support device that supports singing by a singer. The song support apparatus generates a guide melody corresponding to the song to be sung and which guides the singer of the song by two or more types for the same melody by voice synthesis; And a guide melody output unit configured to switch the guide melody of the type or more and output the selected melody in accordance with the music. According to the singing support device of this aspect, since it is possible to switch and output a plurality of types of guide melody, it is possible to support melody sound without having a form suitable for singing of the singer, for example, singing and singing of the singer . Therefore, singing support can be performed without interfering with the singing of the singer.

Claims (6)

A singing support device for supporting singing by a singer,
A guide melody combining unit that generates two or more types of guide melodies that correspond to the song to be sung and guide the singer of the song by voice synthesis;
A song support device comprising: a guide melody output unit configured to switch between the two or more types of guide melodies and output according to the music. The singing support device according to claim 1, wherein
The said guide melody synthetic | combination part is a song assistance apparatus which performs a voice synthesis using the acoustic model which learned the acoustic parameter by the statistical method to the said guide melody. The singing support device according to claim 1 or 2, wherein
The song support device, wherein the guide melody synthesis unit generates at least a melody sound without lyrics and a vocal sound with lyrics as the two or more types of guide melody by the voice synthesis. The singing support device according to claim 3, wherein
The synthesis of the melody sound is performed based on a first acoustic model obtained by learning a singing sound without lyrics such as humming or scat of the music, and the synthesis of the vocal sound is performed on the lyrics of the music. The singing support device performed based on the 2nd acoustic model obtained by learning the singing voice using. A singing support device according to any one of claims 1 to 4, wherein
A singing situation judging unit that judges the degree of the singing by the singing person;
The song support device according to claim 1, wherein the guide melody output unit switches the guide melody according to the determined degree of the singing.   The karaoke apparatus provided with the song assistance apparatus as described in any one of Claim 1- Claim 5.

Images