JP2007233077A - Evaluation device, control method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To more exactly point out a gap in vocal timing with a model singing or performance and a vocal error for singing or performance of a musical instrument by a trainee. <P>SOLUTION: A correspondent part detecting section 112 matches time axes of model voice data and trainee voice data by DP matching and associates sounds having the same position on the time axes with each other. A vocal content comparing section 113 and a vocal timing comparing section 114 compare vocal contents and vocal timings of the model voice data and the trainee voice data, and indicate a part in disagreement. Thus, the trainee can clearly be aware of the existence in a gap in the vocal timing or the vocal error in the own singing, and visually grasp the part in disagreement and the content of the disagreement. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for indicating to a practitioner a shift in pronunciation timing between a singing (or performance) as an example and a singing (or performance) of a practitioner and a pronunciation error.

  The karaoke apparatus has a function of displaying lyrics telop on the screen and changing the color of the telop in order according to the accompaniment. With such a function, the karaoke apparatus can guide the singer to pronounce the correct lyrics at the correct timing.

However, depending on the singer, although there is the guidance as described above, it may sound at a timing later than the accompaniment, conversely, the timing of pronunciation may be too early, or the content of the lyrics itself may be wrong. is there. For such a singer, it is desirable to promptly indicate a mistake in pronunciation timing and lyrics, but a technique effective for this is disclosed in Patent Document 1. That is, the note-on timing included in the MIDI data representing the accompaniment is compared with the timing at which the voice of the singer is picked up by the microphone, and the time lag between the two is detected.
JP 2005-173256 A

  By applying the technique described in Patent Document 1, it is considered possible to point out to the singer the difference in pronunciation timing. However, this technique simply compares the note-on timing with the sound collection timing, and does not collate the words (sounds) constituting the lyrics with the voice of the singer. Therefore, as a matter of course, there is a problem that it is impossible to point out mistakes in the lyrics that have been sung, and it is impossible to accurately grasp the deviation in pronunciation timing. The latter problem occurs when, for example, the pronunciation timing itself matches the accompaniment, but each word constituting the lyrics is pronounced one tempo earlier or one tempo later.

  Specifically, as illustrated in FIG. 7A, according to the accompaniment, the word “ri” is to be pronounced within the period in which the word “ri” of “the dream of the crack is ...” In this case, the next word “shi” is pronounced, and the next word “hi” is pronounced at the timing when “shi” should be pronounced. That is, in the technique described in Patent Document 1, any sound may be used, so as long as any sound is pronounced at the correct timing of the “shi”, it is regarded as a correct pronunciation. . In this respect, there is a limit when the technique of Patent Document 1 is applied. Furthermore, these problems occur not only when practicing singing, but also when practicing playing musical instruments by imitating model performances.

  The present invention has been made in view of the above-described background, and its purpose is to more accurately detect a deviation in pronunciation timing and a mistake in pronunciation with the exemplar singing or playing with respect to the singing or playing of the instrument of the practitioner. It is to show.

  In order to solve the above-mentioned problem, the present invention provides a first storage means for storing first sound data representing a plurality of phonemes whose sound generation timings are arranged in time series in association with the sound generation timings of the respective phonemes, The second storage means for storing the second sound data supplied from the sound collection means for collecting the sound, and the first sound data and the second sound data are associated with each other in a frame unit of a predetermined time length. Corresponding location detecting means for generating corresponding location data representing the associated frame, and the pronunciation timing of the phoneme represented by the second sound data, the pronunciation timing of the phoneme represented by the first sound data, and the corresponding location data Comparing means for determining whether or not a difference between a sound generation timing of a phoneme represented by the first sound data and a sound generation timing of a phoneme represented by the second sound data exceeds a threshold value, Comparison It provides an assessment device characterized by an output means for outputting information specifying the phoneme determining that the difference exceeds the threshold value by.

  According to the present invention, by associating the first sound data and the second sound data in units of frames having a predetermined time length, the phonemes represented by the first sound data and the phonemes represented by the second sound data can be accurately determined. Can be associated. Therefore, it is possible to more accurately indicate a difference in sound generation timing.

  In addition, the present invention includes: first storage means for storing sound data representing a plurality of phonemes whose sound generation timings are arranged in time series in association with sound generation timings of the respective phonemes; and sound collection means for collecting sounds. A second storage means for storing the supplied second sound data, and the first sound data and the second sound data are associated with each other in a frame unit of a predetermined time length, and a correspondence representing the associated frame A corresponding location detecting means for generating location data, a phoneme represented by the first sound data and a phoneme represented by the second sound data are compared in units of frames represented by the corresponding location data, and the first A comparing means for determining whether or not a difference between a phoneme represented by the sound data and a phoneme represented by the second sound data exceeds a threshold; and a phoneme for which the difference is determined to exceed the threshold by the comparing means Output information Provides an assessment apparatus, characterized in that it comprises a power means.

  According to the present invention, by associating the first sound data and the second sound data in units of frames having a predetermined time length, the phonemes represented by the first sound data and the phonemes represented by the second sound data can be accurately determined. Can be associated. Therefore, it is possible to more accurately indicate the difference between these sounds.

  According to the present invention, there is provided a first storage means for storing first sound data representing a plurality of phonemes whose sound generation timings are arranged in time series in association with the sound generation timing of each phoneme, and a sound collecting device for collecting sounds. An evaluation apparatus control method comprising: a second storage unit that stores second sound data supplied from a sound unit; and a control unit, wherein the control unit includes the first sound data and the second sound data. The corresponding sound data in units of frames of a predetermined time length, generating corresponding location data representing the correlated frames, and the control means, the sound generation timing of the phoneme represented by the second sound data, The sound generation timing of the phoneme represented by the first sound data is specified based on the sounding timing of the phoneme represented by the first sound data and the corresponding portion data, and the sounding timing of the phoneme represented by the first sound data and the sounding timing of the phoneme represented by the second sound data Difference And determining whether more than a threshold value, said control means provides a control method characterized by comprising the step of outputting information specifying the sound the difference is determined to exceed the threshold value.

  According to the present invention, there is provided a first storage means for storing first sound data representing a plurality of phonemes whose sound generation timings are arranged in time series in association with the sound generation timing of each phoneme, and a sound collecting device for collecting sounds. An evaluation apparatus control method comprising: a second storage unit that stores second sound data supplied from a sound unit; and a control unit, wherein the control unit includes the first sound data and the second sound data. The corresponding sound data in units of frames of a predetermined time length, generating corresponding location data representing the correlated frames, and the control means comprising the phoneme represented by the first sound data and the second sound The phonemes represented by the data are compared in units of frames represented by the corresponding location data, and whether or not the difference between the phonemes represented by the first sound data and the phonemes represented by the second sound data exceeds a threshold value. A step of judging, and It means, provides a control method characterized by comprising the step of notifying the information specifying the sound the difference is determined to exceed the threshold value.

  Furthermore, the present invention may also take the form of a program that causes a computer to realize functions. Note that in the present invention, the term “pronunciation” includes not only a sound produced when a person sings but also a performance sound produced by playing an instrument. Further, in the present invention, each “phoneme” is consciously pronounced as a group of sounds, and may be any sound that is meaningful for pointing out a pronunciation timing or a mistake in pronunciation.

  ADVANTAGE OF THE INVENTION According to this invention, with respect to a practitioner's song and musical instrument performance, the difference in pronunciation timing and the mistake of pronunciation with the example song and performance can be shown more correctly.

Next, the best mode for carrying out the present invention will be described.
1. Configuration FIG. 1 is a block diagram illustrating a hardware configuration of a karaoke apparatus 1 as an evaluation apparatus according to an embodiment of the invention. A CPU (Central Processing Unit) 11 reads a computer program stored in a ROM (Read Only Memory) 12 or a storage unit 14, loads it into a RAM (Random Access Memory) 13, and executes it to execute a karaoke device. 1 part is controlled. The storage unit 14 is a large-capacity storage unit such as a hard disk, and includes an accompaniment data storage area 14a, an exemplary voice data storage area 14b, a lyrics data storage area 14c, and a trainer voice data storage area 14d. Yes. The display unit 15 is, for example, a liquid crystal display, and displays various screens such as a menu screen for operating the karaoke apparatus 1 and a karaoke screen in which lyrics telop is superimposed on a background image under the control of the CPU 11. The operation unit 16 includes various keys and outputs a signal corresponding to the pressed key to the CPU 11. The microphone 17 is a sound collecting unit that picks up the sound produced by the singer. The sound processing unit 18 converts sound (analog data) collected by the microphone 17 into digital data and supplies it to the CPU 11. The speaker 19 is connected to the sound processing unit 18 and emits sound output from the sound processing unit 18.

  In the accompaniment data storage area 14a of the storage unit 14, for example, accompaniment data in the MIDI (Musical Instruments Digital Interface) format, etc., and information indicating the pitch (pitch) of various musical instruments that accompany each song is in progress of the song. Accompaniment data written with it is stored. In the exemplary audio data storage area 14b, for example, audio data in the WAVE format or MP3 (MPEG Audio Layer-3) format, etc., which is voiced by the singer along the accompaniment represented by the accompaniment data (hereinafter, exemplary audio) Voice data (hereinafter referred to as model voice data) is stored. The lyrics data storage area 14c stores lyrics data indicating lyrics corresponding to the model voice data.

Here, FIG. 2 is a diagram for explaining the correspondence between the model voice data and the lyrics data. As shown in the figure, the lyric data includes each word (phoneme) that constitutes the lyric, and a pronunciation timing that represents a time during which these phonemes should be pronounced. The model voice data, each word (phoneme) constituting the lyrics, and the pronunciation timing of each phoneme are associated with each other. In the example shown in FIG. 2, “Su” in “Sustained Dream of Dreams” starts sounding at the sounding timing T ₁ , “Gi” starts sounding at the sounding timing T ₂ , and “S” It indicates that ... starts the pronunciation sounding timing T ₃ (the same applies hereinafter). This sounding timing is represented by an elapsed time from the start of the accompaniment based on the accompaniment data.

Returning to the description of FIG.
In the practitioner audio data storage area 14d, audio data A / D converted from the microphone 17 via the audio processing unit 18 is stored in time series, for example, in WAVE format or MP3 (MPEG Audio Layer-3) format. Since this voice data is voice data representing the voice of the practitioner (hereinafter referred to as “practice voice”), it is hereinafter referred to as “practice voice data”. The CPU 11 compares the practitioner voice data with the above-described model voice data, thereby detecting a difference in pronunciation timing and pronunciation content and displaying the difference on the display unit 15 to notify the practitioner. To do. The practitioner can recognize a location where his / her pronunciation timing is shifted or a location where the pronunciation is wrong by referring to the notification content. In the following description, for convenience of explanation, when it is not necessary to distinguish between “exemplary voice data” and “trainer voice data”, these are collectively referred to as “voice data”.

  Next, the software configuration of the karaoke apparatus 1 will be described with reference to the block diagram shown in FIG. The basic analysis unit 111, the corresponding part detection unit 112, the pronunciation content comparison unit 113, the pronunciation timing comparison unit 114, and the notification unit 115 illustrated in FIG. 3 execute the computer program stored in the ROM 12 or the storage unit 14 by the CPU 11. It is realized by. The arrows in the figure schematically show the flow of data. In FIG. 3, the basic analysis unit 111 converts the model voice data read from the model voice data storage area 14b and the trainer voice data read from the trainer voice data storage area 14d to a predetermined time length. Each frame is separated and subjected to FFT (Fast Fourier Transform) to calculate the spectrum of the respective audio data.

  Based on the spectrum of each speech data calculated by the basic analysis unit 111, the corresponding location detection unit 112 correlates the phoneme (word) included in the model speech data and the phoneme (word) included in the trainer speech data. (Corresponding location) is obtained. Corresponding portions of the trainer voice and the model voice are supplied from the corresponding portion detection unit 112 to the pronunciation content comparison unit 113 and the pronunciation timing comparison unit 114. The pronunciation content comparison unit 113 compares the pronunciation content of the model voice and the pronunciation content of the practitioner voice corresponding to each other, and performs a process of detecting a difference between the two. The sound generation timing comparison unit 114 compares the sound generation timings of the model voices corresponding to each other and the sound generation timings of the practitioner voices, and performs a process of detecting a difference between the two. The notification unit 115 generates information and various messages that specify the differences detected by the pronunciation content comparison unit 113 and the pronunciation timing comparison unit 114, and displays them on the display unit 15 to notify the practitioner. .

  By the way, the model voice and the practice person voice may be shifted in time as shown in FIG. Therefore, the corresponding location detection unit 112 needs to perform time normalization (DTW; Dynamic Time Warping) by expanding and contracting the time axes of both audio data. In the present embodiment, DP (Dynamic Programming) matching is used as a technique for performing this DTW. Specifically, the processing is as follows.

  The corresponding location detection unit 112 forms a coordinate plane (hereinafter referred to as a DP plane) as shown in FIG. The vertical axis of this DP plane corresponds to the parameter obtained by applying the inverse Fourier transform to the logarithm of the absolute value of the spectrum of each frame of the model voice data, and the horizontal axis is obtained from each frame of the trainer voice data. It corresponds to a parameter (cepstrum) obtained by applying inverse Fourier transform to the logarithm of the absolute value of the spectrum. In FIG. 4, a1, a2, a3... An are obtained by arranging the frames of the model voice data according to the time axis, and b1, b2, b3. They are arranged according to the time axis. The intervals of a1, a2, a3... An on the vertical axis and the intervals of b1, b2, b3... Bn on the horizontal axis all correspond to the time length of the frame. Each lattice point in the DP plane corresponds to a DP matching score which is a value indicating the Euclidean distance of each parameter of a1, a2, a3... And each parameter of b1, b2, b3. It is attached. For example, the lattice points positioned by a1 and b1 include the parameters obtained from the first frame of the series of exemplary voice data and the parameters obtained from the first frame of the series of trainer voice data. A value indicating the Euclidean distance is associated. After forming the DP plane having such a structure, the corresponding part detection unit 112 performs all the processes from the lattice point (starting end) positioned by a1 and b1 to the lattice point (ending point) positioned by an and bn. A route is searched, and for each searched route, the DP matching score of each lattice point traced from the beginning to the end is accumulated, and the minimum accumulated value is obtained. The path with the smallest accumulated value of the DP matching score is considered as a scale of expansion / contraction when the time axis of each frame of the trainer voice data is expanded / contracted in accordance with the time axis of the model voice data.

And the corresponding location detection part 112 is a process which specifies the path | route where the accumulated value of DP matching score becomes the minimum from a DP plane, and expands / contracts the time axis | shaft of the trainer voice data according to the content of the specified path | route. Perform alignment processing. Specifically, the trainer's voice is such that the DP matching score of each lattice point on the path specified from the DP plane represents the Euclidean distance of the parameter obtained from the frame having the same position on the time axis. After rewriting the contents of the time stamp of each frame of data, each frame having the same position on the time axis is sequentially associated as a set. For example, in the path marked on the DP plane shown in FIG. 4, it can be seen that the starting point positioned by a1 and b1 advances to the lattice point positioned by upper right a2 and b2. In this case, since the positions on the time axis of the frames a2 and b2 are the same from the beginning, it is not necessary to rewrite the contents of the time stamp of the frame b2. Furthermore, in this route, it can be seen that the grid point positioned by a2 and b2 advances from the grid point positioned by a2 and b3 on the right. In this case, not only the frame b2 but also the frame b3 need to have the same position on the time axis as the frame a2, so that the time stamp paired with the frame b3 is one frame earlier. Replace with As a result, the frame a2 and the frames b2 and b3 are associated as a set of frames having the same position on the time axis. Such time stamp replacement and frame association are performed for all frame sections from b1 to bn. As a result, for example, as shown in FIG. 5A, even if the pronunciation timing of the practitioner voice is delayed from the pronunciation timing of the model voice, as shown in FIG. Frames (phonemes) having the same position on the time axis adjusted by the expansion and contraction can be associated with the other sound data.
The above is the mechanism of DP matching.

2. Operation Next, the operation of the karaoke apparatus 1 will be described with reference to the flowchart shown in FIG.
The practitioner operates the operation unit 16 of the karaoke apparatus 1 to select a song to be sung and instructs the accompaniment data to be reproduced. The CPU 11 starts the process shown in FIG. 6 in response to this instruction. First, the CPU 11 reads the accompaniment data of the designated song from the accompaniment data storage area 14a and supplies it to the audio processing unit 18 (step S1). The sound processing unit 18 converts the supplied accompaniment data into an analog sound signal and supplies the analog sound signal to the speaker 19 for sound emission. At this time, the CPU 11 controls the display unit 15 to display the lyrics read from the lyrics data storage area 14c, and displays a message prompting a song such as “Please sing along with the accompaniment”, You may make it change the color of the lyrics in order as the accompaniment progresses. The practitioner sings along with the accompaniment emitted from the speaker 19. At this time, the voice of the practitioner is picked up by the microphone 17 and converted into a voice signal, which is supplied to the voice processing unit 18. The trainer speech data A / D converted by the speech processing unit 18 is stored in the trainer speech data storage area 14d of the storage unit 14 in time series together with information indicating the elapsed time from the start of performance to the sounding timing. (Step S2).

  When the reproduction of the accompaniment data is completed (step S3; YES), the CPU 11 reads out the processing of the basic analysis unit 111 described above, that is, the model voice data read from the model voice data storage area 14b and the trainer voice data storage area 14d. The trainee voice data is separated into frame units each having a predetermined time length, and FFT is applied to each frame to calculate the spectrum of each voice data (step S4). Next, the CPU 11 matches the time axes of both audio data by the processing of the corresponding part detection unit 112 described above, that is, DP matching, and associates and associates the frames having the same position on the time axis. Corresponding portion data representing a frame is generated (step S5).

  Subsequently, the CPU 11 performs the process of the sound generation timing comparison unit 114 described above, that is, the process of comparing the sound generation timings of the model voices corresponding to each other and the sound generation timings of the practitioner voices, and detecting portions where the sound generation timings are different. (Step S6). Specifically, the CPU 11 specifies the pronunciation timing of the practitioner voice based on the practitioner voice data and the corresponding portion data obtained by the process of step S5. For this reason, the CPU 11 first identifies the corresponding part of the reference voice and the practicer voice on the same time axis as shown in FIG. 5B. When the correspondence between the two is specified, the CPU 11 on the time axis of the trainee voice as shown in FIG. 5A, a phoneme (for example, “su”) and a phoneme that is pronounced next (for example, “ ”) And the boundary (cut) can be specified. As described above, the trainer voice data is stored together with the information indicating the elapsed time from the start of the performance to the sounding timing, so the CPU 11 can specify the elapsed time corresponding to the boundary between the sounds. This elapsed time becomes the pronunciation timing of each sound included in the trainee's voice.

  Next, the CPU 11 reads out the sound generation timing of the model voice data associated with the trainee voice from the lyrics data storage area 14c. Next, the CPU 11 calculates a difference between the pronunciation timing of the practitioner voice and the pronunciation timing of the model voice corresponding to the practitioner voice. Then, the CPU 11 determines whether or not the difference exceeds a predetermined threshold value. The threshold value at this time is a minimum time interval at which the timing of sound generation is determined to be shifted, and is stored in the storage unit 14 in advance. The sound (phoneme) at the location where the difference is determined to exceed the threshold value is the difference between the sound generation timing of the model sound and the sound generation timing of the practitioner's sound.

  Next, the CPU 11 performs the process of the pronunciation content comparison unit 113 described above, that is, the process of comparing the model voice and the practitioner voice corresponding to each other, and detecting a difference between these voices (step S7). Specifically, first, the CPU 11 compares the voice represented by the trainer voice data with the voice represented by the model voice data associated with the voice, and calculates the difference between the two spectra. When different people pronounce the same word, their speech waveforms are roughly similar, although there are some individual differences. Therefore, it can be determined whether the model voice and the practitioner voice are different depending on whether the spectra are similar. As a more accurate method, there is a method of detecting a difference between the model voice and the trainer voice by performing a formant analysis on the spectrum of the voice data obtained in step S4. This formant is a peak portion of a spectrum biased to a specific frequency. Even if the voice quality is different, a formant specific to the word (speech) appears when the same word is pronounced. Therefore, by analyzing the appearance state of this formant, the content of pronunciation can be identified. In this way, the CPU 11 takes the spectrum or formant difference between the model voice and the trainer voice, and if the difference exceeds the threshold, the difference between the pronunciation content of the model voice and the pronunciation content of the trainer voice. Judged as a place As a threshold used at this time, an upper limit value of a difference that allows a person to pronounce the same word may be determined in advance and stored in the storage unit 14 as a threshold.

  Next, the CPU 11 determines whether or not a difference is detected by the comparison in steps S6 and S7 (step S8). If a difference is not detected (step S8; NO), the CPU 11 displays a message such as “You are singing well” on the display unit 15 and ends the process. On the other hand, when a different part is detected (step S8; YES), the CPU 11 generates information for specifying the trainer voice (phoneme) or the model voice (phoneme) corresponding to the different part and displays it. Notification processing such as display by the unit 15 is performed (step S9). Specifically, the CPU 11 displays the lyrics corresponding to the detected different portions with an underline, or displays the lyrics (characters) in different colors and thicknesses from the display modes of other characters. Further, the CPU 11 also displays a message such as how the sound generation timing is shifted and how the sound generation is wrong.

Here, there are three aspects as shown in (a) to (c) of FIG.
First, as shown in FIG. 7 (a), a series of lyrics “Shibino Yume” out of the lyrics “Yume of Susashiri Hibiki ...” is from the model voice. This is the case when it is pronounced at an early timing. In this way, when the pronunciation timing of the practitioner's voice is more than a predetermined number (here, two words or more) continuously deviating from the pronunciation timing of the model voice, the practitioner's singing is “lyric deviation”. It is judged that. In this case, as shown in FIG. 8 (a), the CPU 11 displays the underlined “Shibino Yume” in the lyrics of “The Dream of Susashiri Hibiki ...” and displays the model voice. In addition, a message for notifying the practitioner of the occurrence of a lyric deviation that is pronounced at an earlier timing than the accompaniment is displayed.

Next, as shown in FIG. 7 (b), the second mode is a timing in which only “no” of the practitioner's voice “is a dream of the crack” is earlier than the model voice. Is pronounced in When the sound generation timing of the practitioner voice is deviated from the sound generation timing of the model voice of less than a predetermined number (here, less than two words) in this way, the practitioner's singing is “timing misalignment”. Judge. In this case, as shown in FIG. 8 (b), the CPU 11 displays an underline for only “no” in the lyrics of “the dream of the crack,” and an exemplary voice or accompaniment. A message is displayed to inform the practitioner that the pronunciation is made at an earlier timing.
As described above, when the sound generation timing is deviated, the CPU 11 determines whether or not the deviated phoneme (word) continues for a predetermined number or more, and determines that the predetermined number or more continues. Different messages are displayed on the display unit 15 when it is determined that they are not continuous.

  And, as shown in FIG. 7 (c), the third mode is mistaken as the lyrics “Sugisarisashi” in “A dream of a crack is too much”. This is the case when singing. When the pronunciation content is different as described above, the CPU 11 determines that the practitioner's singing is “Lyrics mistake”. In this case, as shown in FIG. 8 (c), the CPU 11 makes a mistake in the correct lyrics of “Let ’s dream of a crack that has passed through ...” and the correct lyrics of “Let ’s dream of a crack that has passed through”. In addition to displaying the pronunciation in parallel, the wrong pronunciation of the practitioner “T” is displayed with an underline, and a message for notifying the practitioner that the lyrics are incorrect is displayed on the display unit 15. indicate.

  By the way, as shown in FIGS. 8A to 8C, the CPU 11 also displays a message “Do you want to sing again? Yes / No” on the display unit 15. Here, when the practitioner operates the operation unit 16 and selects “Yes”, the CPU 11 determines that the re-practice of singing has been instructed (step S10; Yes). Then, the CPU 11 obtains lyric data (in this case, lyrics such as “Yoshinari Hibino Yume”) and the accompaniment data corresponding to the lyric data centering on the difference in pronunciation timing or pronunciation content. The lyrics data storage area 14c and the accompaniment data storage area 14a are read out and supplied to the audio processing unit 18 for reproduction (step S11). At this time, the CPU 11 controls the display unit 15 to display the lyrics read from the lyrics data storage area 14c, and further changes the color of the lyrics in order as the accompaniment progresses. The practitioner sings the lyrics displayed on the display unit 15 in accordance with the accompaniment.

  Thereafter, the processing of the CPU 11 returns to step S2 described above. That is, the practitioner's voice is picked up by the microphone 17, converted into a voice signal, and supplied to the voice processing unit 18. The trainer speech data A / D converted by the speech processing unit 18 is stored in the trainer speech data storage area 14d of the storage unit 14 in time series (step S2). Thereafter, the processes of steps S3 to S11 described above are repeated for the stored trainer voice data. Thereby, the practitioner can practice the lyrics in the same place repeatedly until he / she is satisfied. Then, when the practitioner selects “No” on the screens shown in FIGS. 8A to 8C (step S10; No), the processing of the CPU 11 ends.

  As described above, in the present embodiment, after matching the time axes of the model voice data and the trainee voice data, the sounds having the same position on the time axis are associated with each other and compared, and the sound generation timing and the sound generation are compared. Display differences in content. Therefore, the practitioner can clearly recognize that his / her singing has a difference in pronunciation timing and a mistake in pronunciation, and can visually grasp the difference portion and the content of the difference.

3. Modifications The embodiment described above may be modified as follows.
(1) In the above-described embodiment, the case where a practitioner's singing is evaluated has been described as an example. However, the present invention is not limited thereto, and the practitioner's musical instrument performance may be evaluated. In this case, the accompaniment data storage area 14a stores performance data of an instrument (for example, bass or drum) other than the instrument (for example, guitar) to be practiced, and the model audio data storage area 14b stores an exemplary model performance. In the lyrics data storage area 14c, the pitch of the performance sound and the sounding timing are stored in association with each other, and the performance data of the practitioner is stored in the practitioner voice data storage area 14d. Has been. Based on these data, the CPU 11 detects the difference between the model performance and the practice performance through the same processing as described above, and notifies the information specifying the difference. As described above, since the present invention can include singing and performing, the term “pronunciation” in the present invention includes not only a sound uttered when a person sings but also a performance sound uttered by playing an instrument. . Further, in the present invention, “phoneme” is consciously pronounced as a group of sounds, whether in singing or playing, and points out a pronunciation timing or a mistake in pronunciation. It only has to be meaningful.

(2) In the lyric data shown in FIG. 2, the timing at which each sound is to be pronounced is considered as the “sounding timing”. This is because the difference in sound generation timing is largely affected by the timing at which sound generation should start. However, the present invention is not limited to this, and the timing of ending sound generation may be included in the concept of “sound generation timing”. For example, in FIG. 2, the timing T _{1 at} which the pronunciation of the voice “su” at the beginning is started, and the timing at which the pronunciation of the voice “su” is ended, although not shown (timing T ₂ later than timing T _1). May be compared between the model voice and the practitioner voice. In this way, it is possible to evaluate even a slight deviation from the start to the end of pronunciation.

(3) The number of times that the CPU 11 determines that the difference in sound generation timing exceeds the threshold may be accumulated, and a message corresponding to the accumulation result may be displayed on the display unit 15. The fact that the number of times that the difference in pronunciation timing exceeds the threshold value means that the difference in pronunciation timing has occurred frequently, so the CPU 11 says, for example, “There are many lyrics deviations. If you display a message such as “Let's practice more firmly” or “Your lyrics shift is much less.
This applies not only to the pronunciation timing but also to the pronunciation content, and the CPU 11 accumulates the number of times it is determined that the difference in the pronunciation content exceeds the threshold, and sends a message according to the accumulated result. You may make it display on the display part 15. FIG.

(4) In the embodiment, the evaluation results as shown in FIG. 8 are displayed after the practitioner's singing is over. Instead, for example, the pronunciation timing and the difference in pronunciation contents in the past singing of the practitioner are stored as a history, and the karaoke singing (accompaniment data) before the practicing karaoke singing (reproduction of accompaniment data) In synchronism with the reproduction of the sound, a portion where the pronunciation timing and the content of the pronunciation are likely to differ may be displayed. Specifically, the CPU 11 stores the phonemes for which the difference is determined to exceed the threshold value in steps S6 and S7 in association with the lyrics data stored in the lyrics data storage area 14c. In the accompaniment data storage area 14a and the lyrics data storage area 14c, the lyrics data and the accompaniment data are stored in association with each other. Therefore, when the trainee is instructed to reproduce accompaniment data (karaoke singing), the CPU 11 is stored in association with the lyrics data corresponding to the accompaniment data to be reproduced prior to or in synchronization with the reproduction. Information that identifies a phoneme that has been determined (a phoneme for which the difference has been determined to exceed the threshold value in steps S6 and S7 in the past). In the case of informing prior to playback, the CPU 11 tends to delay the pronunciation timing of “Shihibino Yume”, for example, “You are a dream of Sugisari Hibiki ...” at the beginning. Let's watch out. In the case of displaying a message such as “” or informing in synchronization with the reproduction, the CPU 11 highlights, for example, the “Shibino Yume” part of the lyrics of “Let ’s dream of the crack”. do it. This applies not only to the pronunciation timing but also to the pronunciation content. In this way, the practitioner can visually grasp a portion where the pronunciation timing and the content of the pronunciation are easily mistaken before singing (or during singing).

(5) The form of notification by the notification unit 116 is not limited to display, but may be a form in which a voice message specifying a phoneme is output. Moreover, the form which transmits the information which specifies a phoneme to an e-mail format of a practitioner's mail terminal may be sufficient. In addition, information specifying phonemes may be output to a storage medium and stored. In this case, a practitioner can refer to the information by reading the information from the storage medium using a computer. it can. In short, any information may be used as long as it outputs information for identifying a phoneme so that a message or information can be transmitted to the practitioner by some means.

(6) In the embodiment, the trainer voice data is stored in the storage unit 14 such as a hard disk. However, when the trainer voice is immediately discarded after the evaluation of the singing, the trainer voice data is stored. May be stored in the RAM 13.

(7) In the embodiment, when practicing voice data is stored, so-called karaoke singing is performed in which lyrics are displayed and the practitioner sings while reproducing accompaniment data. Not necessary. That is, the practitioner may sing without displaying the lyrics or reproducing the accompaniment data, and record it and compare it with the model voice. Even a practitioner with a very high singing ability cannot sing without mistakes in pronunciation timing and lyrics without displaying lyrics or accompaniment. It becomes possible.

(8) In the embodiment, every time the CPU 11 executes the process shown in FIG. 6, the frequency analysis is performed on the model voice data (step S <b> 4). The result of the frequency analysis may be stored in the storage unit 14, or if the frequency analysis has been performed once in the past, the result may be stored in the storage unit 14. The model voice data and the practice person voice data are data in the WAVE format or the MP3 format. However, the data format is not limited to this, and any format may be used as long as the data indicates voice. May be.

(9) Furthermore, in the embodiment, the model voice data is stored in the storage unit 14 and the CPU 11 of the karaoke apparatus 1 reads the model voice data from the storage unit 14. Audio data may be received via the network.

(10) In the embodiment, the karaoke apparatus 1 realizes all of the functions shown in FIG. On the other hand, two or more devices connected via a communication network may share the above functions, and a system including the plurality of devices may realize the karaoke device 1 of the embodiment. For example, a computer device that includes a microphone, a speaker, a display device, an input device, and the like and realizes the notification unit 115, a basic analysis unit 111, a corresponding location detection unit 112, a pronunciation content comparison unit 113, and a pronunciation timing comparison unit 114 are realized. The server device may be configured as a system connected by a communication network. In this case, the computer device converts the voice input from the microphone into voice data and transmits it to the server device, and the server device performs a comparison process between the received voice data and the model voice data and the lyrics data, The comparison result may be transmitted to the computer device.

(11) Programs executed by the CPU 11 of the karaoke apparatus 1 as the evaluation apparatus in the above-described embodiment are a magnetic tape, a magnetic disk, a floppy (registered trademark) disk, an optical recording medium, a magneto-optical recording medium, and a CD (Compact Disk). )-It can be provided in a state stored in a recording medium such as a ROM, a DVD (Digital Versatile Disk), or a RAM. It is also possible to download to the karaoke apparatus 1 via a network such as the Internet.

It is a block diagram which shows an example of the hardware constitutions of a karaoke apparatus. It is a figure which shows an example of the content of model audio | voice data and lyrics data. It is a block diagram which shows an example of the software structure of a karaoke apparatus. It is a figure which shows DP matching. It is a figure which shows the expansion-contraction of the time axis in DP matching. It is a flowchart which shows the flow of the process which CPU of a karaoke apparatus performs. It is a figure explaining the various aspects at the time of pronunciation timing and the content of pronunciation differing. It is a figure which shows an example of the screen displayed in a karaoke apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Karaoke apparatus, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Memory | storage part, 15 ... Display part, 16 ... Operation part, 17 ... Microphone, 18 ... Audio | voice processing part, 19 ... Speaker, 111 ... Basics Analysis unit 112... Corresponding location detection unit 113. Sound generation content comparison unit 114. Sound generation timing comparison unit 115 115 notification unit.

Claims (12)

First storage means for storing first sound data representing a plurality of phonemes whose pronunciation timings are arranged in time series in association with the pronunciation timing of each phoneme;
Second storage means for storing second sound data supplied from the sound collection means for collecting sound;
Corresponding location detection means for associating the first sound data and the second sound data in units of frames of a predetermined time length, and generating corresponding location data representing the associated frames;
The sound generation timing of the phoneme represented by the second sound data is specified based on the sound generation timing of the phoneme represented by the first sound data and the corresponding portion data, and the sound generation timing of the phoneme represented by the first sound data Comparing means for determining whether or not a difference between the phoneme generation timing represented by the second sound data exceeds a threshold value;
And an output means for outputting information for identifying a phoneme for which the difference is determined to exceed a threshold value by the comparison means. Determining means for determining whether or not a predetermined number of phonemes determined by the comparing means that the difference exceeds a threshold value;
The output means, in addition to the information for identifying the phoneme for which the difference is determined to exceed the threshold, and the case where it is determined that the predetermined number of phonemes determined by the determination means to exceed the threshold are consecutive. The evaluation apparatus according to claim 1, wherein different messages are output when it is determined that the predetermined number or more does not continue. Accumulating means for accumulating the number of times that the difference is determined by the comparing means to exceed a threshold;
The evaluation device according to claim 1, wherein the output unit outputs a message corresponding to a result of accumulation by the accumulation unit in addition to information specifying a phoneme for which the difference is determined to exceed a threshold value. History storage means for storing the phonemes determined by the comparison means to have the difference exceeding a threshold value, in association with the first sound data;
Third storage means for storing accompaniment data in association with the first sound data;
Reproduction means for reproducing the accompaniment data stored in the third storage means;
Prior to or in synchronization with the reproduction of the accompaniment data by the reproduction means, the history is associated with the first sound data stored in association with the accompaniment data by the third storage means. The evaluation apparatus according to claim 1, further comprising: a pre-output unit that outputs information for specifying a phoneme stored in the storage unit. First storage means for storing sound data representing a plurality of phonemes whose pronunciation timings are arranged in time series in association with the pronunciation timing of each phoneme;
Second storage means for storing second sound data supplied from the sound collection means for collecting sound;
Corresponding location detection means for associating the first sound data and the second sound data in units of frames of a predetermined time length, and generating corresponding location data representing the associated frames;
The phonemes represented by the first sound data and the phonemes represented by the second sound data are compared in units of frames represented by the corresponding location data, and the phonemes represented by the first sound data and the second sounds are compared. A comparing means for determining whether or not a difference between the phonemes represented by the data exceeds a threshold;
And an output means for outputting information for identifying a phoneme for which the difference is determined to exceed a threshold value by the comparison means.   The evaluation device according to claim 1, wherein the output unit notifies the practitioner of information specifying a phoneme for which the difference is determined to exceed a threshold by the comparison unit. Accumulating means for accumulating the number of times that the difference is determined by the comparing means to exceed a threshold;
6. The evaluation apparatus according to claim 5, wherein the output means outputs a message corresponding to an accumulation result by the accumulation means in addition to information specifying a phoneme for which the difference is determined to exceed a threshold value. History storage means for storing the phonemes determined by the comparison means to have the difference exceeding a threshold value in association with the first sound data;
Third storage means for storing accompaniment data in association with the first sound data;
Reproduction means for reproducing the accompaniment data stored in the third storage means;
Prior to or in synchronization with the reproduction of the accompaniment data by the reproduction means, the history is associated with the first sound data stored in association with the accompaniment data by the third storage means. The evaluation apparatus according to claim 5, further comprising: a pre-output unit that outputs information for specifying a phoneme stored in the storage unit. Supplied from a first storage means for storing first sound data representing a plurality of phonemes whose sound generation timings are arranged in time series in association with sound generation timings of the respective phonemes, and a sound collection means for collecting sounds. A method for controlling an evaluation apparatus, comprising: a second storage unit that stores second sound data; and a control unit.
The control means associating the first sound data and the second sound data in units of frames of a predetermined time length, and generating corresponding location data representing the associated frames;
The control means specifies the sound generation timing of the phoneme represented by the second sound data based on the sound generation timing of the phoneme represented by the first sound data and the corresponding portion data, and the first sound data is Determining whether the difference between the pronunciation timing of the phoneme represented and the pronunciation timing of the phoneme represented by the second sound data exceeds a threshold;
The control means comprises a step of outputting information for identifying a sound for which it is determined that the difference exceeds a threshold value. Supplied from a first storage means for storing first sound data representing a plurality of phonemes whose sound generation timings are arranged in time series in association with sound generation timings of the respective phonemes, and a sound collection means for collecting sounds. A method for controlling an evaluation apparatus, comprising: a second storage unit that stores second sound data; and a control unit.
The control means associating the first sound data and the second sound data in units of frames of a predetermined time length, and generating corresponding location data representing the associated frames;
The control means compares the phoneme represented by the first sound data with the phoneme represented by the second sound data in units of frames represented by the corresponding location data, and the phoneme represented by the first sound data Determining whether the difference from the phoneme represented by the second sound data exceeds a threshold;
The control means comprises a step of outputting information for identifying a sound for which it is determined that the difference exceeds a threshold value. Supplied from a first storage means for storing first sound data representing a plurality of phonemes whose sound generation timings are arranged in time series in association with sound generation timings of the respective phonemes, and a sound collection means for collecting sounds. A computer comprising second storage means for storing second sound data;
A corresponding location detection function for associating the first sound data and the second sound data in units of frames of a predetermined time length, and generating corresponding location data representing the associated frames;
The sound generation timing of the phoneme represented by the second sound data is specified based on the sound generation timing of the phoneme represented by the first sound data and the corresponding portion data, and the sound generation timing of the phoneme represented by the first sound data And a comparison function for determining whether or not the difference between the pronunciation timing of the phoneme represented by the second sound data exceeds a threshold value;
A program for realizing an output function for outputting information for identifying a sound for which the difference is determined to exceed a threshold value by the comparison means. First sound data representing a plurality of phonemes whose sound generation timing is continuous in time series is supplied from first storage means for storing the sound data in association with sound generation timing of each phoneme, and sound collection means for collecting sounds. A computer comprising second storage means for storing second sound data;
A corresponding location detection function for associating the first sound data and the second sound data in units of frames of a predetermined time length, and generating corresponding location data representing the associated frames;
The phonemes represented by the first sound data and the phonemes represented by the second sound data are compared in units of frames represented by the corresponding location data, and the phonemes represented by the first sound data and the second sounds are compared. A comparison function for determining whether the difference from the phoneme represented by the data exceeds a threshold;
A program for realizing an output function for outputting information for identifying a sound for which the difference is determined to exceed a threshold value by the comparison means.

Images