JP6150276B2 - Speech evaluation apparatus, speech evaluation method, and program - Google Patents

Description

  The present invention relates to an utterance evaluation apparatus, an utterance evaluation method, and a program, and more particularly, to an improvement for evaluating an utterance method in which a voice is mixed with a back voice.

  There is a falsetto as a singing technique in vocal music, which corresponds to a back voice in Japanese music. In recent years, more and more singers (artists) of popular music use this back voice to sing the treble part, and this is desired for those who want to be good singers or those who want to improve karaoke. Therefore, a technique for discriminating voices and back voices in speech has been proposed (see, for example, Non-Patent Document 1).

Satoshi Kojima, 4 others, “Examination of acoustic features for discriminating back and local voices in singing”, IEICE Technical Report, The Institute of Electronics, Information and Communication Engineers, October 27, 2012, 112 (266 ), P. 67-72

  By the way, there is a way of singing that doesn't make you feel the joint with the part that sings with the local voice by uttering by mixing the back voice with the voice instead of just making the back voice. Therefore, after the voice and back voice can be freely spoken, it is desired to learn a singing method in which the voice is mixed with the back voice as the next step. In the technology for clearly identifying the voice, the voice based on the singing method in which the voice is mixed with the back voice is determined as either the back voice or the voice. It has been impossible with the above-mentioned conventional technique to perform utterance for switching between the back voice and the voice while changing the pitch (pitch) and to evaluate whether or not the switching is successful.

  The present invention has been made against the background of the above circumstances, and the object of the present invention is to provide an utterance evaluation apparatus, an utterance evaluation method, and a program for objectively evaluating an utterance method in which a voice is mixed with a back voice There is to do.

In order to achieve such an object, the gist of the first aspect of the present invention is an utterance evaluation apparatus that evaluates utterance based on voice information input from the voice input apparatus, and is input from the voice input apparatus. An extraction unit that extracts a fundamental wave and a plurality of harmonics in the voice information to be performed, and a first voice information corresponding to at least one of the voices of the subject and a speaker other than the subject and a second voice corresponding to the back voice Learning data based on the fundamental wave and a plurality of harmonics extracted by the extraction unit in each piece of speech information is input to a learning model, and the first fusion rate output from the learning model is set to a predetermined value. and a evaluation unit for learning to modify the learning model is performed, the evaluation unit, the learning line evaluation data based on the fundamental wave is extracted and a plurality of harmonics by the extraction unit Based on that input to the learning model, it is characterized in that calculates a first fusion rate as an index according to the switching table voice and falsetto in the audio information.

As described above, according to the first aspect of the present invention, the extraction unit that extracts the fundamental wave and the plurality of harmonics in the voice information input from the voice input device , and at least the subject and the speaker other than the subject acquired in advance. Learning data based on the fundamental wave and a plurality of harmonics extracted by the extraction unit in each of the first voice information corresponding to one voice and the second voice information corresponding to the back voice is input to the learning model, and the learning model and a evaluation unit for the learning to modify the learning model is performed such that the value first fusion rate predetermined output from the evaluation unit, the fundamental wave and a plurality extracted by the extraction unit based on the evaluation data based on the harmonic to be input to the learning model in which the learning is performed, the first as an index according to the switching table voice and falsetto in the voice information Since it is intended to calculate a slip ratio, it can be easily grasped by practical processing whether the voice it sent and the front voice and falsetto are mixed at a ratio of how. That is, it is possible to provide an utterance evaluation apparatus that objectively evaluates a utterance method in which a voice is mixed with a back voice.

The gist of the second invention subordinate to the first invention is that the learning model is a neural network or a support vector machine.
The gist of the third invention, which is dependent on the first invention or the second invention, is that the evaluation unit extracts the frequency of the fundamental wave extracted by the extraction unit in the voice information input from the voice input device. In accordance with the change in the voice information, the switching of the voice and the back voice in the voice information is evaluated based on the smoothness of the change in the first fusion rate calculated corresponding to the voice information. In this way, it is possible to suitably evaluate whether or not the switching is successfully performed by performing the utterance for switching the back voice and the voice while changing the pitch of the singing voice.

The gist of the fourth invention, which is dependent on the third invention, is that the first singing method for singing the voice and the back voice is fundamental and a plurality of harmonics in the voice information input from the voice input device. A second evaluation unit that evaluates the singing of the voice and the back voice in the voice information based on a second fusion rate determined based on the wave, the evaluation unit being evaluated by the second evaluation unit; Learning is performed by the learning data including learning data based on a fundamental wave and a plurality of harmonics in each of the first voice information corresponding to the voice in the first singing method and the second voice information corresponding to the back voice. The evaluation unit extracts the second singing method continuously changed between the voice and the back voice by the extraction unit in the voice information input from the voice input device. Evaluating the switching of voice and back voice in the voice information based on the smoothness of the change in the first fusion rate calculated corresponding to the voice information according to the frequency change of the fundamental wave It is. In this way, it is possible to more suitably evaluate how much the voice and back voice are mixed in the voice produced by using the spoken voice and back voice of the model in the learning data of the model. can do.

The gist of the fifth invention according to any one of the first to fourth inventions is that the learning data is the intensity of the plurality of harmonics in each of the first voice information and the second voice information. Is divided by the intensity of the fundamental wave, and the evaluation data is the intensity of the plurality of harmonics extracted by the extraction unit in the voice information input from the voice input device by the intensity of the fundamental wave. It is the value divided. In this way, it is possible to easily grasp how much the voice and the back voice are mixed in the voice that is produced by using a practical model.

In order to achieve the above object, the gist of the sixth invention is an utterance evaluation method for evaluating utterance based on voice information input from a voice input device, which is input from the voice input device. An extraction process for extracting a fundamental wave and a plurality of harmonics in the voice information, and first voice information corresponding to at least one voice of a subject and a speaker other than the subject acquired in advance and a second voice corresponding to a back voice The learning data based on the fundamental wave and the plurality of harmonics extracted by the extraction process in each information is input to a learning model, and the first fusion rate output from the learning model is set to a predetermined value. and evaluation process that learning is performed to correct the learning model, said extracting said training evaluation data based on the fundamental wave and a plurality of harmonics are extracted in the process has been performed the learning mode Based on that input to Le, and a calculation step of calculating the first fusion rate as an index according to the switching table voice and falsetto in the audio information, characterized in that it contains. In this way, it is possible to easily grasp by a practical process how much the voice and the back voice are mixed in the voice that they have made. That is, it is possible to provide an utterance evaluation method that objectively evaluates an utterance method in which a voice is mixed with a back voice.

In order to achieve the above object, the gist of the seventh aspect of the present invention is that the speech input is provided to a control device provided in a speech evaluation device that evaluates speech based on speech information input from a speech input device. An extraction step for extracting a fundamental wave and a plurality of harmonics in voice information inputted from the apparatus, and correspondence to first voice information and back voice corresponding to at least one voice of a subject and a speaker other than the subject acquired in advance Learning data based on the fundamental wave and the plurality of harmonics extracted in the extraction step in each of the second audio information to be input to the learning model, and the first fusion rate output from the learning model is a predetermined value an evaluation step of the learning to modify the learning model is performed such that, evaluation de based on the fundamental wave and a plurality of harmonics are extracted in the extraction step Based on to input data to the learning model in which the learning is performed, and a calculation step of calculating the first fusion rate as an index according to the switching table voice and falsetto in the voice information, that is performed It is a featured program. In this way, it is possible to easily grasp by a practical process how much the voice and the back voice are mixed in the voice that they have made. That is, it is possible to provide a program that objectively evaluates the utterance method in which the voice is mixed with the back voice.

It is a block diagram which illustrates the composition of the utterance evaluation device which is one example of the present invention. It is a figure explaining the speech training method (procedure) used as the premise of speech evaluation of a present Example. It is a functional block diagram explaining the principal part of the control function with which CPU of the utterance evaluation apparatus of FIG. 1 was equipped. It is the figure which plotted the pitch at the time of uttering the ascending and descending scales with the vowel “A”. It is the figure which plotted the value which remove | divided the spectrum gravity center of the several harmonic component with the fundamental frequency with respect to the same audio | voice as shown in FIG. 4 for every time. It is a figure which shows an example of the relationship between the frequency of the fundamental wave which concerns on the rising scale of the utterance of "e" by a man, and a 1st fusion rate which are the evaluation results by the utterance evaluation apparatus of FIG. It is a figure which shows an example of the relationship between the frequency of the fundamental wave which concerns on the rising scale of the utterance of "e" by a man, and a 1st fusion rate which are the evaluation results by the utterance evaluation apparatus of FIG. It is a figure which shows an example of the relationship between the frequency of the fundamental wave which concerns on the rising scale of the utterance of "e" by a man, and a 1st fusion rate which are the evaluation results by the utterance evaluation apparatus of FIG. It is a figure which shows an example of the relationship between the frequency of the fundamental wave which concerns on the rising scale of the utterance of "e" by a woman, and a 1st fusion rate which are the evaluation results by the utterance evaluation apparatus of FIG. It is a figure which shows an example of the relationship between the frequency of the fundamental wave which concerns on the rising scale of the utterance of "e" by a woman, and a 1st fusion rate which are the evaluation results by the utterance evaluation apparatus of FIG. It is a figure which shows an example of the relationship between the frequency of the fundamental wave which concerns on the rising scale of the utterance of "e" by a woman, and a 1st fusion rate which are the evaluation results by the utterance evaluation apparatus of FIG. It is a figure which shows the relationship between the frequency of the fundamental wave corresponding to the audio | voice shown in FIG. 6, and a 2nd fusion rate. It is a figure which shows the relationship between the frequency of the fundamental wave corresponding to the audio | voice shown in FIG. 7, and a 2nd fusion rate. It is a figure which shows the relationship between the frequency of the fundamental wave corresponding to the audio | voice shown in FIG. 8, and a 2nd fusion rate. It is a figure which shows the relationship between the frequency of the fundamental wave corresponding to the audio | voice shown in FIG. 9, and a 2nd fusion rate. It is a figure which shows the relationship between the frequency of the fundamental wave corresponding to the audio | voice shown in FIG. 10, and a 2nd fusion rate. It is a figure which shows the relationship between the frequency of the fundamental wave corresponding to the audio | voice shown in FIG. 11, and a 2nd fusion rate. It is a figure which shows the evaluation result screen which is an example of the evaluation result displayed on the touchscreen display of the utterance evaluation apparatus of FIG. It is a flowchart explaining the principal part of an example of the utterance evaluation control by CPU of the utterance evaluation apparatus of FIG.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating the configuration of an utterance evaluation apparatus 10 that is an embodiment of the present invention. As shown in FIG. 1, the utterance evaluation apparatus 10 according to the present embodiment includes a CPU 12, a ROM 14, and a RAM 16. The CPU 12 is a central processing unit in the utterance evaluation device 10. The ROM 14 is a read-only memory. The RAM 16 is a write / read memory as needed. The utterance evaluation device 10 includes a storage unit 18 such as a well-known memory card, for example. The utterance evaluation apparatus 10 includes a communication unit 20 that communicates with a communication line such as the Internet using radio waves. The utterance evaluation device 10 includes a speaker 22, a microphone 24, and an A / D converter 26. The microphone 24 is a voice input device. The A / D converter 26 is a device for converting the sound input from the microphone 24 into a digital signal. The utterance evaluation device 10 includes a touch panel display 28. The touch panel display 28 includes a display device 30 that displays an image (video), and a touch panel 32 that performs input to the utterance evaluation device 10 in response to contact with a user's finger, an attached pen, or the like.

  The utterance evaluation apparatus 10 according to the present embodiment preferably includes a touch panel display 28 and a mobile phone having various functions including an access function to the Internet according to an operation through the touch panel display 28, that is, a so-called smartphone. In addition, a program (application software: mobile application) according to an embodiment of the present invention is installed. A program according to an embodiment of the present invention is installed in a general personal computer, a tablet terminal, a portable music player, a consumer game machine or the like having a configuration equivalent to that of the CPU 12, ROM 14, RAM 16, and the like. Also good. The present invention is a karaoke apparatus having a configuration such as a MIDI (Musical Instrument Digital Interface) sound source and an amplifier mixer, which outputs a performance music selected from a large number of performance music and amplifies and outputs a sound input by a microphone. It may be applied. The communication function such as the communication unit 18 is not necessarily provided. In the utterance evaluation apparatus that performs utterance evaluation but does not output sound such as recorded user song data, the speaker 22 is not necessarily provided.

  The microphone 24 is a device (electroacoustic transducer) that converts sound (vibration) such as voice uttered by a person into an electrical signal. The microphone 24 is preferably a general microphone that is provided in a mobile phone or the like and picks up air conduction sound, for example. Preferably, for example, a microphone that picks up a solid conduction sound by a patch-like vibration pickup brought into contact with a human throat may be used. The A / D converter 26 converts audio information as an analog signal input from the microphone 24 into a digital signal and supplies the digital signal to the CPU 12 or the like.

  The utterance evaluation apparatus 10 performs signal processing according to a program stored in advance in the ROM 14 while the CPU 12 uses the temporary storage function of the RAM 16. That is, the utterance evaluation device 10 includes a so-called microcomputer (electronic control device). With such a configuration, the CPU 12 executes various controls related to the utterance evaluation control of the present embodiment, which will be described later.

  FIG. 2 is a diagram for explaining an utterance training method (procedure) that is a premise of the utterance evaluation of the present embodiment. As shown in FIG. 2, the present embodiment is premised on an utterance training method for training different voice production methods in each of a plurality of stages from the first stage “Stage.1” to the sixth stage “Stage.6”. . In the first stage “Stage.1”, training is performed in which the back voice and the voice are clearly separated. The pitch (scale) is fixed at a predetermined scale. In the second stage “Stage.2”, training is performed to utter different scales by clearly separating the back voice and the voice. In the third stage "Stage.3", the back voice and the voice are clearly separated, and each is trained to sing a simple melody song. In the fourth stage "Stage.4", training is performed to switch back and forth between both voices and voices. In the 5th stage “Stage.5”, there is a training to make voices mixed with back voices and voices, and to make inconspicuous the change of voice zone that occurs at the turning point between voice and back voices at the time of vocalization. Done. In the sixth stage "Stage.6", training is performed to utter a voice that mixes back voice and voice and strengthens both voices to produce an integrated voice.

  As shown in FIG. 2, the fourth stage “Stage.4” is further divided into, for example, “Stage.4a”, “Stage.4b”, and “Stage.4c” as a plurality of stages. “Stage.4a” corresponds to an utterance in which the switching between the back voice and the voice is clear, but the switching is clear and the back voice and the voice are clearly switched. “Stage.4b” corresponds to an utterance in which the back voice and the voice are mixed slightly when switching between the back voice and the voice. “Stage.4c” corresponds to a utterance that can be mixed back and voices when switching between back and voices. In the present embodiment, the evaluation unit 52 shown in FIG. 3 described in detail below evaluates the switching of voice and back voice in the fourth stage “Stage.4”. The second evaluation unit 54 evaluates whether the voice and the voice are sung in “Stage.3” or the like, that is, whether the voice and the voice are clearly separated.

  FIG. 3 is a functional block diagram for explaining a main part of the control function provided in the CPU 12 of the utterance evaluation apparatus 10. The recording unit 50, the evaluation unit 52, the second evaluation unit 54, and the display control unit 56 illustrated in FIG. 3 are preferably all functionally provided in the CPU 12. That is, the CPU 12 is caused to function as the recording unit 50, the evaluation unit 52, the second evaluation unit 54, and the display control unit 56 shown in FIG. 3 by executing the program of this embodiment stored in the ROM 14 or the like. The recording unit 50, the evaluation unit 52, the second evaluation unit 54, and the display control unit 56 are configured as individual control units and perform various functions described in detail below by communicating information with each other. It may be a thing. The extraction unit 60 included in the evaluation unit 52 may be configured as a control unit different from the evaluation unit 52 and perform various functions described in detail below by communicating information with each other. . The second evaluation unit 54 is not necessarily provided in the utterance evaluation apparatus 10 of the present embodiment.

  The recording unit 50 records voice information input from the microphone 24. For example, audio information input by the microphone 24 and converted into a digital signal via the A / D converter 26 is stored (accumulated) in an audio database 38 formed in the storage unit 18 or the like. For example, each file is stored with specific identification information as an audio file of a predetermined format such as an AVI (Audio-Video Interleaved) format file or a WAVE sound file. Recording by the recording unit 50 or storage in the voice database 38 is performed as a new voice file every time the voice input from the microphone 24 is interrupted (every period of time during which no voice is input for a predetermined time or longer). Alternatively, the sound information corresponding to the sound input from the microphone 24 at a predetermined time may be stored as one unit sound file. Preferably, the voice information input by the microphone 24 is associated with the identification information of each user (user to be evaluated by the utterance evaluation apparatus 10) in the voice database 38 for each user who is the utterance subject. Remember.

  In the utterance evaluation by the utterance evaluation device 10, the voice of the user (speaking subject) to be recorded by the recording unit 50 is preferably switched at least once between the voice and the back voice (preferably 1 each. (Repeated more than once). In this switching between voice and back voice, not only the voice and voice can be switched clearly, but, for example, in the stage “Stage.4c” shown in FIG. Including. The user's voice to be recorded by the recording unit 50 corresponds to the user's voice input from the microphone 24. That is, it corresponds to the utterance corresponding to the audio information to be extracted by the extracting unit 60, and further, to be evaluated by the evaluating unit 52 to the second evaluating unit 54. The utterances to be recorded by the recording unit 50 are preferably voices and back voices in utterances corresponding to a single vowel, such as extending the utterance of “A” or “E” or uttering it as an upper or lower scale. It is preferable that they are separated. Preferably, it may include consonants such as singing a predetermined task song (simple melody).

  The evaluation unit 52 of this embodiment includes an extraction unit 60. The extraction unit 60 extracts a fundamental wave and a plurality of harmonics in the audio information input from the microphone 24. Preferably, data of a predetermined section (for example, a section corresponding to a predetermined time) is read from the sound information stored in the sound database 38. Then, by performing speech analysis on the read data, a fundamental wave (a component corresponding to the lowest frequency obtained as a result of Fourier transform) and a plurality of harmonics (a component corresponding to an integer multiple of the fundamental wave frequency) ). In order to perform such control, the extraction unit 60 includes a voice analysis unit 62, a vocal cord signal spectrum estimation unit 64, and a feature amount extraction unit 66. Hereinafter, processing by each control unit will be described. The extraction unit 60 may be provided as a control unit different from the evaluation unit 52.

  The voice analysis unit 62 performs voice analysis based on a predetermined algorithm for the voice information input from the microphone 24. For example, speech analysis of target speech information is performed by LPC (Linear Predictive Coding) analysis or cepstrum analysis that is generally used as a speech analysis technique. Preferably, data of a predetermined section is read from the voice information stored in the voice database 38, and voice analysis such as LPC analysis or cepstrum analysis is performed on the data, so that the frequency of the fundamental wave (hereinafter referred to as target frequency) is obtained. , Called fundamental frequency) and vocal tract characteristics. This vocal tract characteristic takes a value specific to each person who is the utterance subject, and is called formant (a harmonic group corresponding to air in the vocal tract or resonance frequency). That is, the voice analysis unit 62 preferably estimates various information including the fundamental frequency of the target voice information. For example, the formant frequency and the fundamental frequency (pitch information) corresponding to the vocal tract characteristics are estimated. , Estimation (extraction) of signal intensity corresponding to volume (voice volume) and information related to discrimination of voiced or unvoiced sound.

  In addition, by removing vocal tract characteristics (formant) specific to each speaker from the voice and analyzing the vocal cord sound source wave from which the vocal tract characteristics have been removed, it is possible to appropriately discriminate between voice and back voice in the target voice. Is possible.

  The vocal cord signal spectrum estimation unit 64 estimates the spectrum of a signal obtained by removing the vocal tract characteristics estimated by the voice analysis unit 62 from the voice information recorded by the recording unit 50. Preferably, the spectrum of a signal obtained by removing the vocal tract characteristic estimated by the voice analysis unit 62 from the data of the predetermined section read from the voice database 38 is estimated. For example, in the LPC analysis, an inverse filter of the vocal tract characteristic (frequency transfer characteristic) estimated by the voice analysis unit 62 is configured. Then, after passing the target audio information through such an inverse filter to obtain a residual signal, the spectrum is estimated by Fourier transform of the residual signal. In the cepstrum analysis, after obtaining the residual signal by passing the target speech information through a low-pass lifter, that is, by removing the low quefrency part of the speech information with a lifter, the residual signal The spectrum is estimated by Fourier transform.

  The feature quantity extraction unit 66 extracts a fundamental wave and a plurality of harmonics in the voice information input from the microphone 24 based on the fundamental frequency obtained by the voice analysis unit 62. For example, from the spectrum estimated by the vocal cord signal spectrum estimation unit 64, a plurality of remaining harmonics excluding the fundamental wave are extracted based on the fundamental frequency estimated by the speech analysis unit 62. Preferably, the relative level of the fundamental wave and each harmonic is extracted. That is, a relative ratio between the sound pressure level corresponding to the fundamental wave and the sound pressure level corresponding to each harmonic in the target audio information is extracted (calculated).

  The second evaluation unit 54 evaluates the singing of the voice and the back voice in the voice information (the voice and the back voice are divided). For example, in the “Stage.3” shown in FIG. 2, such evaluation is performed for the first singing method in which the voice and the back voice are separately sung. Preferably, the second fusion rate described in detail below is evaluated as an index relating to the singing of the voice and the back voice in the target audio information. Here, the voice is also referred to as a local voice and refers to a series of voices (voice zones) in which the harmonic energy of the fundamental wave in the vocal cord sound source wave tends to be relatively larger than the back voice. This vocalization is considered to be caused by the vocal cords that the closed muscle group surrounding the vocal cords works preferentially on the ring-shaped thyroid muscle and the entire vocal cords vibrate. The back voice is also called a falset, and is a voice series (voice zone) in which the harmonic energy of the fundamental wave in the vocal cord sound source wave tends to be relatively smaller than the voice. It is considered that this back voice is caused by the ring-shaped thyroid muscles surrounding the vocal cords acting preferentially on the obturator muscle group in terms of vocalization mechanism, and the marginal portion of the vocal cords vibrating. That is, in this embodiment, the voice and the back voice are considered to belong to different vocal tracts, and the voices of the respective vocal tracts are considered to be uttered by one mechanism (for example, use of vocal cords). In order to evaluate the fusion rate of voice and back voice in the target speech information, the second evaluation unit 54 includes a second fusion rate calculation part 74 and a voice / back voice vocalization ability evaluation part 76. Hereinafter, processing by each control unit will be described.

  The second fusion rate calculation unit 74 calculates the fusion state (fusion rate) of the voice and back voice, which are the two voice zones in the target voice information. Preferably, a second fusion rate is calculated as one scale value obtained by indexing the fusion state. Preferably, a value (= fg / f0) obtained by dividing the spectral centroid fg of a plurality of harmonics in the target audio information by the fundamental wave frequency (fundamental frequency) f0 is calculated as the second fusion rate. In other words, the spectrum centroid is calculated from the relative frequency (relative value of the sound pressure level) of the fundamental frequency extracted in the target audio information and the frequency corresponding to each harmonic. And the ratio of the calculated spectrum gravity center and a fundamental frequency (pitch frequency) is calculated | required, and the ratio is calculated as a 2nd fusion rate.

  In the present embodiment, preferably, the voice and back voice are evaluated by the second evaluation unit 54 based on as many harmonic components as possible. That is, the spectrum centroid of as many harmonic components as possible is obtained, and the spectrum centroid is used for the evaluation of the voice and the back voice. Such a spectrum centroid also depends on the frequency (pitch frequency) of the fundamental wave. Therefore, in this embodiment, in order to eliminate the influence of the fundamental frequency, that is, the influence of the fundamental frequency, the harmonic structure of the vocal cord sound source is indexed (standard) by dividing the spectrum centroid by the fundamental frequency as described above. Process).

  The voice / back voice utterance ability evaluation unit 76 evaluates the fusion rate of the voice and back voice based on the fundamental wave and the plurality of harmonics in the target voice information. Specifically, such evaluation is performed based on the second fusion rate fg / f0 calculated by the second fusion rate calculation unit 74. Preferably, it is evaluated that the higher the second fusion rate fg / f0, the higher the voice ratio and the lower the back voice ratio. That is, the relationship that is a criterion for evaluation by the voice / back voice utterance ability evaluation unit 76 is that the higher the second fusion rate fg / f0, the higher the voice ratio in the target voice information and the lower the voice ratio. As such, it is predetermined.

  FIG. 4 is a diagram in which the pitch (fundamental frequency) when the vowel “A” is uttered ascending and descending scales is plotted for each time. FIG. 5 is a diagram in which values (= fg / f0) obtained by dividing the spectral centroids fg of a plurality of harmonic components by the fundamental frequency f0 for the same sound are plotted for each time. In FIG. 5, a section where a person who is proficient in the method of uttering a back sound actually hears the utterance and the utterance is determined to be a back sound is indicated by a hatched area. In other words, the remaining section not indicated by the hatched area in FIG. 5 corresponds to a section in which the utterance is determined to be a voice. In the example shown in FIG. 5, fg / f0 continuously changes in both voice and back voice sections, and the fg / f0 is relatively low in the section corresponding to the back voice, and in the section corresponding to the voice. It can be seen that the value is relatively high. Therefore, it is understood that fg / f0, which is a value obtained by dividing the spectral centroid fg of a plurality of harmonic components by the fundamental frequency f0, is an index value that can evaluate the fusion rate related to the singing of the voice and the back voice. . For example, if the index value is equal to or greater than the reference value (fg / f0 = about 2.8) indicated by the broken line in FIG. It is possible to evaluate the voice and the back voice objectively and uniquely.

  The evaluation unit 52 evaluates switching between voice and back in the voice information based on the fundamental wave and the plurality of harmonics extracted by the extraction unit 60 from a predetermined relationship. For example, in the “Stage.4” shown in FIG. 2, such evaluation is performed for the second singing method that is continuously changed between the voice and the back voice. Preferably, the first fusion rate, which will be described in detail below, is evaluated as an index related to the switching of voice and back voice in the target voice information. In order to evaluate the switching of voice and back voice in the target audio information, the evaluation unit 52 includes a calculation formula 70 and calculation constant information 72. Hereinafter, the control of the evaluation unit 52 will be described in detail.

  The calculation formula 70 is preferably a learning model (mathematical model). More preferably, it is a non-linear learning model that performs non-linear gain learning. For example, with respect to the input voice information, a value between “−1” and “1” as the back voice, that is, a value between −1 and 1 (continuous value) is output. As the calculation formula 70, a support vector machine (SVM) is preferably used. Alternatively, a neural network is preferably used. The calculation formula 70 is preferably based on a fundamental wave and a plurality of harmonics in the first voice information corresponding to the pre-acquired voice and the second voice information corresponding to the back voice, respectively, as will be described in detail below. Learning is performed using the learning data (learning sample).

  In the learning of the evaluation unit 52, learning speech information is input to the extraction unit 60, and learning data based on the fundamental wave and a plurality of harmonics extracted by the extraction unit 60 is input to the calculation formula 70. At this time, the constant set in the calculation formula 70 is set based on the value stored in the calculation constant information 72. However, when such a constant is set in the calculation formula 70, the ratio (first fusion rate described later) output from the calculation formula 70 with respect to the input of predetermined voice information is the significance of the voice information for learning. Therefore, if it is inappropriate, it is necessary to correct (learn) the constant set in the calculation formula 70. In this correction, first, the calculation formula 70 is corrected so that there is no contradiction between the ratio output from the calculation formula 70 with respect to the input of predetermined voice information and the learning voice information. For example, the correction of the calculation formula 70 is performed so that “−1” is output from the calculation formula 70 when the voice information corresponding to the correct back voice is input as the learning voice information. The calculation formula 70 is corrected so that “1” is output from the calculation formula 70 when the voice information corresponding to the correct voice is input as the voice information for learning. Next, the calculation formula 70 is corrected so that the correspondence between the learning voice information is changed and the correspondence between the ratio output from the calculation formula 70 with respect to the input and the learning voice information is consistent. Done. Next, the extraction object is changed about the some harmonic extracted by the extraction part 60. FIG. The constants in the calculation formula 70 corrected as described above are stored in the calculation constant information 72, whereby the learning of the calculation formula 70 is completed.

  The learning data used for the learning of the evaluation unit 52 preferably includes the intensity of the plurality of harmonics in each of the first voice information corresponding to the voice obtained in advance and the second voice information corresponding to the back voice. The value divided by the intensity. For example, a value obtained by dividing the intensity of the first harmonic by the intensity of the fundamental wave with respect to the nth harmonic from the second harmonic extracted corresponding to each of the first voice information and the second voice information (relative level). ) L02, a value L03 obtained by dividing the intensity of the second harmonic by the intensity of the fundamental wave,..., And a value L0n obtained by dividing the intensity of the nth harmonic by the intensity of the fundamental wave. As will be described later, the first voice information and the second voice information for learning used for learning by the evaluation unit 52 correspond to voices and back voices uttered by a user to be evaluated by the evaluation unit 52, that is, a subject. However, it may correspond to speech and back speech produced by a speaker other than the subject, for example, a trainer who is proficient in speech and back speech. Learning may be performed by the first voice information and the second voice information corresponding to the utterance of a single user, or by learning data corresponding to the voice and back voice uttered by a plurality of users, respectively. Learning may be performed. The plurality of users may or may not include users to be evaluated by the evaluation unit 52. For example, voice information corresponding to voices and back voices uttered by a plurality of trainers or the like and voice information corresponding to voices and back voices uttered by a subject may be used as learning data. That is, the first voice information and the second voice information for learning used for learning by the evaluation unit 52 correspond to at least one voice and back voice of a subject and a speaker other than the subject acquired in advance. The calculation formula 70 may be stored in the storage unit 18 or the like that has been learned (determined) in advance by a computer or the like different from the utterance evaluation apparatus 10. That is, the learning of the calculation formula 70 is not necessarily performed by the utterance evaluation apparatus 10 of the present embodiment.

  The learning in the evaluation unit 52 is preferably performed based on the first voice information corresponding to the voice and the second voice information corresponding to the back voice in the first singing method evaluated by the second evaluation unit 54. . Specifically, the intensity of a plurality of harmonics in each of the first voice information evaluated as voice by the second evaluation unit 54 and the second voice information evaluated as back voice is divided by the intensity of the fundamental wave. Learning is performed by learning data including learning data that is the obtained value. In addition to such learning data, learning is performed by learning data based on the fundamental wave and a plurality of harmonics in the first voice information corresponding to the voice and the second voice information corresponding to the back voice that are not related to the evaluation by the second evaluation unit 54. May be performed. When the voicers of the speech information to be evaluated by the evaluation unit 52 and the second evaluation unit 54 are the same, the evaluation unit 52 learns the calculation formula 70 based on the voice and back voice of the user who is the evaluation target. Will be done.

  The evaluation unit 52 inputs the evaluation data based on the fundamental wave and the plurality of harmonics extracted by the extraction unit 60 to the calculation formula 70, and uses the value output from the calculation formula 70 as the voice in the target speech information and Calculated as the first fusion rate of the back voice. The first fusion rate is, for example, a value between “−1” for the back voice and “1” for the voice, that is, a value between −1 and 1 (continuous value). The evaluation unit 52 preferably calculates evaluation data based on the fundamental wave and the plurality of harmonics extracted by the extraction unit 60. This evaluation data is preferably a value obtained by dividing the intensity of a plurality of harmonics extracted by the extraction unit 60 in the audio information input from the microphone 24 by the intensity of the fundamental wave. For example, regarding the n-th harmonic from the second harmonic extracted by the extraction unit 60, a value (relative level) L02 obtained by dividing the intensity of the first harmonic by the intensity of the fundamental wave, and the intensity of the second harmonic A value L03 divided by the intensity of the fundamental wave,..., And a value L0n obtained by dividing the intensity of the nth harmonic wave by the intensity of the fundamental wave are calculated. Then, the values L02, L03,..., L0n are input to the calculation formula 70, and the value output from the calculation formula 70 is calculated as the first fusion rate.

  The evaluation unit 52 preferably evaluates switching between voice and back voice (transition between voice and back voice) in the target voice information. That is, the voice conversion point shock at the portion where the voice and the back voice are switched is evaluated. This voice point shock refers to a sudden change of vocal tract that occurs at the transition between voice and back voice during utterance, and refers to fluctuations that occur in the voice volume and pitch accompanying such a change of voice. This voice conversion point shock is considered to be a phenomenon that occurs because the force relationship between the ring-shaped thyroid muscle that dominates when producing a reverse voice and the closing muscle group that dominates when producing a voice is abruptly changed. . It is thought that the complicated movement of the joint part between the cricoid cartilage and the arytenoid cartilage on the upper rear left and right is also caused. For example, in the example shown in FIG. 5, the reference value indicated by the broken line (fg / f0 = about 2.8) corresponds to the voice point, and the place where the transition of the second fusion rate crosses the reference value is voiced. It corresponds to a point.

  The evaluation unit 52 preferably uses the target voice based on the smoothness of the change in the first fusion rate output from the calculation formula 70 in accordance with the change in the frequency of the fundamental wave extracted by the extraction unit 60. Evaluate the change of voice and back in information. In the present embodiment, that the change in the first fusion rate is smooth means that when a function of the frequency of the fundamental wave extracted by the extraction unit 60 and the first fusion rate is taken into account, a derivative ( Having a derivative). When the speech information to be evaluated is, for example, uttered as an up / down scale corresponding to a single vowel, the change in the frequency of the fundamental wave extracted by the extraction unit 60 is in time series. In this case, the evaluation unit 52 preferably evaluates the switching of the voice and the back voice in the target voice information based on the smoothness of the change along the time series of the first fusion rate. When the voice information to be evaluated sings a predetermined task song (simple melody), the evaluation unit 52 preferably uses the fundamental wave extracted by the extraction unit 60 for the voice information to be evaluated. Rearrange as frequency changes. Then, based on the smoothness of the change in the first fusion rate output from the calculation formula 70 in accordance with the change in the frequency of the rearranged fundamental wave, the switching of the voice and the back voice in the target voice information is evaluated. To do. Hereinafter, specific examples of such evaluation will be described with reference to FIGS.

  6 to 8 show the relationship between the frequency (pitch) of the fundamental wave extracted by the extraction unit 60 and the first fusion rate corresponding to the rising scale of the utterance of “e (e)” by a male (male voice). It is a figure which shows an example. The frequency of the fundamental wave corresponds to the horizontal axis, and the first fusion rate corresponds to the vertical axis (the same applies in the following description). FIG. 6 corresponds to the utterance in which the switching between the back voice and the voice is clear, but the switching is clear and the back voice and the voice are clearly switched, and the stage “Stage. Corresponds to 4a ". FIG. 7 corresponds to the utterance in which the back voice and the voice are mixed slightly when switching between the back voice and the voice, and corresponds to the stage “Stage.4b” in FIG. 2. FIG. 8 corresponds to the utterance in which the back voice and the voice can be mixed considerably when switching between the back voice and the voice, and corresponds to the stage “Stage.4c” in FIG. 2.

  6 to 8, in FIG. 6, the first fusion rate from the vicinity of the frequency of 100 (Hz) to the vicinity of 250 (Hz) is maintained at approximately “1”, and from that to the vicinity of 300 (Hz). The first fusion rate rapidly changes to approximately “−1”. That is, the function of the frequency of the fundamental wave extracted by the extraction unit 60 and the first fusion rate is not smooth, and there is a gap where the change is discontinuous between about 250 (Hz) and about 300 (Hz). To do. In FIG. 7, although there is a gap where the change is discontinuous between about 250 (Hz) and about 300 (Hz), the first fusion is performed between the frequency of about 100 (Hz) and about 250 (Hz). The rate gradually decreases, and it can be said that the function of the frequency of the fundamental wave extracted by the extraction unit 60 and the first fusion rate is smoother than in the example shown in FIG. In FIG. 8, the first fusion rate is gradually decreased from “1” to “−1” between the frequency near 100 (Hz) and the vicinity of 500 (Hz), and a gap where the change becomes discontinuous is not confirmed. Therefore, it can be said that the relationship shown in FIG. 8 is the smoothest.

  9 to 11 show the relationship between the first fusion rate and the frequency (pitch) of the fundamental wave extracted by the extraction unit 60 corresponding to the rising scale of the utterance of “e (e)” by a woman (woman voice). It is a figure which shows an example. FIG. 9 corresponds to the utterance in which the switching between the back voice and the voice is clear but the switching is clear and the back voice and the voice are clearly switched, and the stage “Stage. Corresponds to 4a ". FIG. 10 corresponds to the utterance in which the back voice and the voice are mixed slightly when switching the back voice and the voice, and corresponds to the stage “Stage.4b” in FIG. FIG. 11 corresponds to the utterance in which the back voice and the voice can be mixed considerably when switching between the back voice and the voice, and corresponds to the stage “Stage.4c” in FIG. 2.

  9 to FIG. 11, in FIG. 9, the first fusion rate from the vicinity of the frequency of 100 (Hz) to the vicinity of 220 (Hz) is maintained at approximately “1”, and changes so as to drop substantially vertically therefrom. The first fusion rate rapidly changes to approximately “−1” until the vicinity of 300 (Hz). Furthermore, a relatively large gap where the function becomes discontinuous exists in the vicinity of the frequency of about 250 (Hz). That is, the function of the fundamental wave frequency extracted by the extraction unit 60 and the first fusion rate is not smooth. In FIG. 10, there is a gap where the function is discontinuous in the vicinity of a frequency of about 250 (Hz), but the size of the gap is smaller than the example shown in FIG. Therefore, in the example shown in FIG. 10, it can be said that the function of the frequency of the fundamental wave extracted by the extraction unit 60 and the first fusion rate is smoother than in the example shown in FIG. In FIG. 11, the first fusion rate is gradually decreased from “1” to “−1” between the vicinity of the frequency of 100 (Hz) and the vicinity of 500 (Hz), and the gap in which the change is discontinuous is not confirmed. Therefore, it can be said that the relationship shown in FIG. 11 is the smoothest.

  As described above, as shown in FIGS. 6 to 11, the smoothness of the relationship between the frequency of the fundamental wave extracted by the extraction unit 60 and the first fusion rate is as follows. Whether or not the back voice can be mixed well is reflected in an easy-to-understand manner. Accordingly, it is possible to objectively evaluate the transition between the voice and the back voice in the voice information, using the smoothness as an index of the function of the fundamental frequency extracted by the extraction unit 60 and the first fusion rate. . As an aspect of the evaluation by the evaluation unit 52, a graph showing the relationship shown in FIGS. 6 to 11, that is, a graph showing the smoothness of the change in the first fusion rate in accordance with the change in the frequency of the fundamental wave may be displayed. It is also possible to perform further detailed evaluation such as displaying the analysis result of the graph together. For example, in the relationship between the frequency of the fundamental wave and the first fusion rate, a differential coefficient (derivative function) of a function in a section where the first fusion rate is between “−1” and “1” is calculated, and the continuity thereof is calculated. It is conceivable to evaluate the skill of switching the voice and the back voice in the voice information. Alternatively, based on a predetermined relationship, based on the smoothness of the function, the target audio information is any of the stages “Stage.4a”, “Stage.4b”, and “Stage.4c” shown in FIG. It is also possible to evaluate whether the utterance corresponds to.

  FIGS. 12 to 14 show the frequency (pitch) of the fundamental wave extracted by the extraction unit 60 and the second fusion rate calculation unit corresponding to the same audio information as the utterance that is the evaluation target in the examples shown in FIGS. 7 is a diagram illustrating an example of a relationship with a second fusion rate fg / f0 calculated by 74. FIG. The frequency of the fundamental wave corresponds to the horizontal axis, and the second fusion rate corresponds to the vertical axis (the same applies in the following description). That is, the example of FIG. 12 is the male utterance of the stage “Stage.4a” in FIG. 2, the example of FIG. 13 is the male utterance of the stage “Stage.4b” in FIG. 2, and the example of FIG. Corresponds to male utterances at stage “Stage.4c”. 15 to 17 show the frequency of the fundamental wave extracted by the extraction unit 60 and the second fusion rate fg / f0 corresponding to the same voice information as the utterance that is the evaluation target in the examples shown in FIGS. It is a figure which shows an example of a relationship. That is, the example in FIG. 15 is for the female utterance at the stage “Stage.4a” in FIG. 2, the example in FIG. 16 is for the female utterance at the stage “Stage.4b” in FIG. 2, and the example in FIG. Corresponds to the female utterances at stage "Stage.4c". As shown in FIGS. 12 to 17, in the relationship between the frequency of the fundamental wave extracted by the extraction unit 60 and the second fusion rate fg / f0, when switching between voice and back voice in voice information, the voice and back voice are switched. There is no noticeable difference depending on whether or not can be mixed well.

  As described above, the second fusion rate fg / f0 calculated by the second fusion rate calculation unit 74 is preferably used for evaluating whether or not the voice and the back voice can be clearly distinguished. It can be said that it is not effective as an index for evaluating the skill of transition at the transition between voice and back voice. Accordingly, the first singing method for singing the voice and the back voice in “Stage.3” shown in FIG. 2 is evaluated by the second evaluation unit 54 based on the second fusion rate fg / f0, and is shown in FIG. By performing evaluation based on the first fusion rate by the evaluation unit 52 for the second singing method that is continuously changed between the voice and the back voice in “Stage.4”, skillful utterance in each singing method Can be suitably evaluated.

  The display control unit 56 displays the evaluation result by the evaluation unit 52 on the display device 30 or the like in the touch panel display 28. FIG. 18 is a diagram illustrating an evaluation result screen 80 that is an example of an evaluation result by the evaluation unit 52 displayed on the touch panel display 28. As shown in FIG. 18, the display control unit 56 preferably uses the first fusion rate according to the relationship shown in FIGS. 6 to 11, that is, the change in the frequency of the fundamental wave, as the evaluation result by the evaluation unit 52. A graph showing the smoothness of changes is displayed. Preferably, the analysis result of the graph is also displayed. For example, in the relationship between the frequency of the fundamental wave and the first fusion rate, the continuity related to the derivative of the function in the section where the first fusion rate is between “−1” and “1” is a scale corresponding to the evaluation result. (Comment) is displayed. Preferably, as an evaluation result by the evaluation unit 52, it is displayed which of the stages “Stage.4a”, “Stage.4b”, and “Stage.4c” shown in FIG. In this way, by displaying the evaluation result by the evaluation unit 52 in a visually easy-to-understand manner, the user can easily and objectively confirm his skill in singing.

  FIG. 19 is a flowchart for explaining a main part of the utterance evaluation control by the CPU 12 of the utterance evaluation apparatus 10, and is repeatedly executed at a predetermined cycle.

  First, in S1, a practice stage and a model are selected via the touch panel 32 or the like according to the screen displayed on the display device 30. Next, in S2, it is determined which of the third stage “Stage.3” and the fourth stage “Stage.4” has been selected. In S2, when the fourth stage “Stage.4” is selected, the processing from S13 is executed. In S2, when the third stage “Stage.3” is selected, S3. The practice menu screen of “Stage.3” is displayed on the display device 30 and a guide sound is output from the speaker 22. Next, in S4, voice information (singing voice) to be evaluated is acquired. In other words, voice information is input from the microphone 24 via the A / D converter 26, and the voice database formed in the storage unit 18 or the like is associated with the identification information of the user who is the main utterance. 38 is stored (accumulated).

  Next, in S5, the 1st audio | voice information (phonetic singing voice | voice) corresponding to a voice is extracted from the audio | voice information acquired in S4 corresponding to all the vowels. In parallel with the process of S5, in S6, the second voice information (back vocal singing voice) corresponding to the back voice corresponding to all the vowels is extracted from the voice information acquired in S4. Next, in S7, spectrum analysis is performed on the singing voice extracted in S5 and S6 by a technique such as cepstrum analysis. Next, in S8, the primary data, that is, the frequency (pitch) and volume of the fundamental wave are extracted as a result of the spectrum analysis in S7. Next, in S9, the secondary data, that is, the fundamental wave and the relative level of each harmonic (harmonic level pattern) are extracted. Next, in S10, from the primary data extracted in S8 and the secondary data extracted in S9, a second fusion rate (voice zone fusion ratio), that is, a spectrum of a plurality of harmonics in the target speech information. A value obtained by dividing the center of gravity fg by the frequency f0 of the fundamental wave (= fg / f0) is calculated. In parallel with the process of S10, in S11, the intensity of the plurality of harmonics in each of the first voice information and the second voice information extracted in S5 and S6 is used as learning data of the calculation formula 70 in the evaluation unit 52. A value divided by the intensity of the fundamental wave is calculated. Next, in S12, based on the second fusion rate calculated in S10, an evaluation of the first singing method for singing the voice and the back voice (evaluation according to the third stage “Stage.3”) is performed. After the evaluation result is displayed on the display device 30, this routine is terminated.

  In S <b> 13, the practice menu screen of “Stage.4” is displayed on the display device 30 and a guide sound is output from the speaker 22. Next, in S14, voice information (singing voice) to be evaluated is acquired. In other words, voice information is input from the microphone 24 via the A / D converter 26, and the voice database formed in the storage unit 18 or the like is associated with the identification information of the user who is the main utterance. 38 is stored (accumulated). Next, in S15, spectrum analysis is performed on the singing voice acquired in S14 by a technique such as cepstrum analysis. Next, in S16, primary data, that is, the frequency (pitch) and volume of the fundamental wave are extracted as a result of the spectrum analysis in S15. Next, in S17, secondary data, that is, the fundamental wave and the relative level of each harmonic (harmonic level pattern) are extracted. Next, in S18, the fundamental wave and the relative level of each harmonic extracted in S17 are input to the calculation formula 70, and the value output from the calculation formula 70 is the voice and back voice in the target speech information. Calculated as the first fusion rate. Next, in S19, based on the first fusion rate calculated in S18, the evaluation of the second singing method continuously changed between the voice and the back voice (fourth stage “Stage.4 ) Is performed, and the evaluation result is displayed on the display device 30, and then this routine is terminated.

  In the above control, S4 and S14 are processing of the recording unit 50, S15 to S18 are processing of the evaluation unit 52, calculation process and calculation step, S5 to S11 are processing of the second evaluation unit 54, and S16 and S17 are processing. S12 and S19 correspond to the processing of the display control unit 56, respectively, to the processing of the extraction unit 60, the extraction process, and the extraction step. The process corresponding to the third stage “Stage.3”, that is, the processes of S3 to S12 are not necessarily executed.

  As described above, according to the present embodiment, it is possible to easily grasp how much the voice and the back voice are mixed in the voice produced by the practical process. That is, it is possible to provide the utterance evaluation apparatus 10 that objectively evaluates the utterance method in which the voice is mixed with the back voice.

  In addition, it is possible to suitably evaluate whether or not the switching is successfully performed by changing the pitch of the singing voice while performing the utterance for switching the back voice and the voice.

  In addition, by using the voice and back voice produced by the model for the learning data of the model, it is possible to more suitably evaluate how much the voice and back voice are mixed in the voice produced by the self. .

  In addition, it is possible to easily grasp how much the voice and back voice are mixed with each other by using a practical model.

  The preferred embodiments of the present invention have been described in detail with reference to the drawings. However, the present invention is not limited to these embodiments, and may be implemented in other modes.

  For example, in the above-described embodiment, the voice information recorded by the recording unit 50 is stored in the voice database 38, and the voice information stored in the voice database 38 is read out by the extraction unit 60 to perform processing such as voice analysis. Although the aspect has been described, the audio information recorded by the recording unit 50 may be processed in real time by the extraction unit 60, the evaluation unit 52, and the like. In such an aspect, the user can confirm in real time the fusion rate of the voice and back voice related to his / her current utterance, and can realize more efficient training.

  The speech evaluation device (speech evaluation method) of the present invention can evaluate the switching between voice and back voice regardless of whether it is male (male voice) or female (female voice). It may be possible to increase flexibility according to the gender and age of the speaker. The speech evaluation according to the present invention includes various forms such as a communication speech method course via the Internet, a speech training mode (speech training game) as an application in a smartphone, and speech training software as software installed and used in a personal computer. It is widely used for training menus.

  In addition, although not illustrated one by one, the present invention is implemented with various modifications within a range not departing from the gist thereof.

  10: utterance evaluation device, 52: evaluation unit, 54: second evaluation unit, 60: extraction unit, 70: calculation formula

Claims (7)

An utterance evaluation device that evaluates utterance based on voice information input from a voice input device,
An extraction unit for extracting a fundamental wave and a plurality of harmonics in the voice information input from the voice input device;
The fundamental wave and the plurality of harmonics extracted by the extraction unit in each of the first voice information corresponding to the voice of at least one of the subject and the speaker other than the subject acquired in advance and the second voice information corresponding to the back voice An evaluation unit that inputs learning data based on the learning model, and performs learning for correcting the learning model so that the first fusion rate output from the learning model becomes a predetermined value ,
The evaluation unit is based on the fact that to input evaluation data based on the fundamental wave and a plurality of harmonics extracted by the extraction unit to the learning is performed learning model, switching table voice and falsetto in the voice information An utterance evaluation apparatus for calculating the first fusion rate as an index related to the above . The utterance evaluation apparatus according to claim 1, wherein the learning model is a neural network or a support vector machine. The evaluation unit is configured to calculate the first fusion rate calculated corresponding to the voice information according to a change in the frequency of the fundamental wave extracted by the extraction unit in the voice information input from the voice input device. based on the smoothness of the change, utterance evaluation apparatus according to claim 1 or 2 is to evaluate the switching table voice and falsetto in the audio information. About the 1st singing method which sings each voice and back voice, based on the 2nd fusion rate determined based on the fundamental wave in a voice information inputted from the voice input device, and a plurality of harmonics, the voice information A second evaluation unit that evaluates the singing of voice and back voice in
The evaluation unit includes a fundamental wave and a plurality of harmonics in each of the first voice information corresponding to the voice in the first singing method and the second voice information corresponding to the back voice evaluated by the second evaluation unit. Learning is performed with the learning data including learning data based on
The evaluation unit has a frequency of the fundamental wave extracted by the extraction unit in the voice information input from the voice input device for the second singing method continuously changed between the voice and the back voice. depending on the change, on the basis of the smoothness of the change in the first fusion rate calculated in response to the audio information, according to claim 3 is intended to evaluate the switching table voice and falsetto in the voice information Utterance evaluation device. The learning data is a value obtained by dividing the intensity of the plurality of harmonics in each of the first audio information and the second audio information by the intensity of the fundamental wave,
The evaluation data, according to claim 4 the intensity of the plurality of harmonics extracted by the extraction unit from claim 1 which is a value obtained by dividing the intensity of the fundamental wave in the audio information input from the voice input device The utterance evaluation apparatus according to any one of the above. A speech evaluation method for evaluating speech based on speech information input from a speech input device,
An extraction process for extracting a fundamental wave and a plurality of harmonics in the voice information input from the voice input device;
The fundamental wave and the plurality of harmonics extracted by the extraction process in the first voice information corresponding to the voice of at least one of the subject and the speaker other than the subject acquired in advance and the second voice information corresponding to the back voice respectively. An evaluation process in which learning is performed to input learning data based on the learning model, and to correct the learning model so that the first fusion rate output from the learning model becomes a predetermined value ;
Based on the input of the evaluation data based on the fundamental wave and the plurality of harmonics extracted in the extraction process to the learning model in which the learning is performed , utterance evaluation method characterized by a calculation step of calculating the first fusion rate, including. In the control device provided in the utterance evaluation device that evaluates the utterance based on the voice information input from the voice input device,
An extraction step of extracting a fundamental wave and a plurality of harmonics in the voice information input from the voice input device;
The fundamental wave and the plurality of harmonics extracted by the extraction step in the first voice information corresponding to the voice of at least one of the subject and the speaker other than the subject acquired in advance and the second voice information corresponding to the back voice respectively. the based learning data input to the training model, the evaluation step in which the first fusion rate output from the learning model is Ru performed learning to correct the learning model such that the predetermined value,
Based on inputting the evaluation data based on the fundamental wave and the plurality of harmonics extracted in the extraction step to the learning model in which the learning is performed , And a calculation step of calculating the first fusion rate.