JP4983958B2 - Singing scoring device and singing scoring program - Google Patents

Description

  The present invention relates to a singing scoring device and a singing scoring program suitable for use in a karaoke apparatus.

  A karaoke equipped with a singing scoring device that scores singing by comparing the pitch extracted from the singing voice of the singer against the scoring standard, with the pitch and pronunciation timing of each note constituting the main melody part of the karaoke accompaniment as the scoring standard Various devices have been developed. This type of apparatus is disclosed in, for example, Patent Document 1.

JP-A-11-194782

  By the way, as in the technique disclosed in Patent Document 1 above, in the method of scoring by comparing the scoring standard and the singing pitch, the scoring standard is a karaoke song of a style called “rap” that does not have a clear melody. There is a problem that singing cannot be scored because there is no.

  In addition, even if the scoring standard is compared with the singing pitch, if only a part of the karaoke song is sung carefully and the karaoke accompaniment is stopped, the singing score is only given to that part. There is also an adverse effect that can be scored. In order to avoid such harmful effects, if the singing is not continued for a certain period of time, the score may be invalidated. However, in this case, all the very short songs will not be scored even if they are sung. Problems arise.

  The present invention has been made in view of such circumstances, and provides a singing scoring device and a singing scoring program capable of scoring even a song without a melody or a song with a very short performance time. It is an object.

According to the first aspect of the present invention, the data is generated in synchronization with the reproduction of the karaoke song, and is matched with the sample singing sound data representing the example singing sound sung as a model and the reproduced karaoke song. From the singing sound data corresponding to the predetermined number of data framed by the framing means, and the framing means for dividing the user singing data obtained from the user singing sound sung by the user into predetermined frames. A first feature extracting unit that extracts a voice feature amount of the model singing sound; and a second feature extracting unit that extracts the voice feature amount of the user singing sound from the user singing data of a predetermined number of data framed by the framing unit. Similarity calculation for calculating the similarity between the voice feature amount of the model singing sound and the voice feature amount of the user singing sound respectively extracted by the feature extraction means and the first and second feature extraction means And stage, in response to said degree of similarity calculated by the similarity calculation means, determination means for determining suitability of the user singing sound for each frame for model singing sound, the number of frames of the user singing data and model singing sound Only when the ratio with the number of frames of data is equal to or greater than a certain value, there is provided scoring means for scoring the user's song based on the result of the determination by the determination means for each frame.

According to the second aspect of the present invention, the data is generated in synchronization with the reproduction of the karaoke song, and is matched with the sample singing sound data representing the example singing sound sung as a model and the reproduced karaoke song. From the singing sound data for the predetermined number of data framed by the framing process, the framing process for dividing the user singing data obtained from the user singing sound sung by the user into predetermined frames A first feature extraction process for extracting a voice feature value of a model singing sound, and a second feature for extracting a voice feature value of a user singing sound from user song data for a predetermined number of data framed by the framing process Similarity calculation for calculating the similarity between the voice feature amount of the model singing sound and the voice feature amount of the user singing sound extracted by the feature extraction process and the first and second feature extraction processes, respectively. And a determination process for determining, for each frame, the suitability of the user singing sound with respect to the model singing sound according to the similarity calculated by the similarity calculating process, and the number of frames of the user singing data and the model singing sound. Only when the ratio with the number of frames of data is a certain value or more, the computer executes a scoring process for scoring the user's song based on the result of the determination process determining whether each frame is appropriate.

  In the present invention, the user singing for the model singing sound according to the similarity between the voice feature extracted from the model singing sound sung as a model and the voice feature extracted from the user singing sound sung by the user. Since the sound is judged to be appropriate and graded, it can be scored whether the lyrics of the song are sung correctly. Therefore, even a karaoke song of a style called “rap” without a clear melody can be scored.

  Moreover, in this invention, only when the ratio of the number of frames of user song data and the number of frames of model song sound data is a certain value or more, the user's song is scored based on the result of determining the suitability for each frame. Therefore, even a song with a very short performance time can be scored.

It is a block diagram which shows the structure of one Embodiment by this invention. It is a flowchart which shows operation | movement of a karaoke process. It is a flowchart which shows the operation | movement of a partial scoring process. It is a flowchart which shows the operation | movement of a MFCC calculation process. It is a figure which shows an example of a filter bank.

Embodiments of the present invention will be described below with reference to the drawings.
A. Constitution
FIG. 1 is a block diagram showing a configuration of a karaoke apparatus provided with a singing scoring apparatus according to an embodiment of the present invention. In this figure, the CPU 10 controls each part of the apparatus by executing a predetermined program stored in the program ROM 11 in accordance with a switch event supplied from the switch part 14. Characteristic processing operations (karaoke processing, partial scoring processing, and MFCC calculation processing) of the CPU 10 according to the gist of the present invention will be described later.

  The program ROM 11 stores various programs executed by the CPU 10 and control data. The various programs stored in the program ROM 11 include “karaoke processing”, “partial scoring processing”, and “MFCC calculation processing” described later. The RAM 12 includes a work area and a buffer area. In the work area of the RAM 12, various register / flag data used for the processing of the CPU 10 are temporarily stored. In the buffer area of the RAM 12, model song data and user song data, which will be described later, are temporarily stored.

  The karaoke data memory 13 is composed of an electrically rewritable nonvolatile memory such as a flash memory, for example, and stores karaoke data of a plurality of songs. For example, in addition to the power switch, the switch unit 14 includes various switches such as a song selection switch for selecting a song to accompany karaoke and a start / stop switch for instructing start / stop of karaoke. An event is generated and supplied to the CPU 10. When karaoke is started by operating the start / stop switch of the switch unit 14, the CPU 10 reads out karaoke data from the karaoke data memory 13 for a song selected in advance by operating the song selection switch.

  The karaoke data of one song stored in the karaoke data memory 13 is composed of lyrics data and voice data. The lyrics data is information for displaying subtitles of the lyrics of the song in synchronization with the karaoke accompaniment. The audio data includes accompaniment data and sample singing data compressed and encoded in MP3 format in a dual monaural mode having a karaoke track and a vocal track.

  That is, accompaniment data obtained by sampling the karaoke accompaniment sound is compressed and stored in the karaoke track, and the vocal track samples the singing sound sung by the singer as a model in synchronization with the karaoke accompaniment sound, for example. The model singing data is stored after being compressed and encoded.

  The microphone 15 converts the input user's singing sound into a singing voice signal and outputs it. The codec 16 stores user singing data obtained by A / D converting the singing voice signal supplied from the microphone 15 in the buffer area of the RAM 12 under the control of the CPU 10. Also, the codec 16 decodes (decompresses) the MP3 format model song data read from the karaoke data memory 13 under the control of the CPU 10 and stores it in the buffer area of the RAM 12. Note that the user song data and model song data stored in the buffer area of the RAM 12 during karaoke execution are updated every 256 msec corresponding to a frame of 1024 sampling points.

  Further, the codec 16 decodes (decompresses) the MP3 format accompaniment data read from the karaoke data memory 13 under the control of the CPU 10, and performs karaoke accompaniment obtained by D / A converting the decoded accompaniment data. The sound signal and the singing voice signal supplied from the microphone 15 are mixed to generate an audio output. The audio output is supplied to, for example, an external audio input terminal of a television receiver (not shown) to reproduce the audio. Under the control of the CPU 10, the video encoder 17 converts the lyrics data read from the karaoke data memory 13 into a video output for subtitle display. The video output is supplied to, for example, a video input terminal of a television receiver (not shown) and displayed on the screen as lyrics subtitles.

B. Action
Next, with reference to FIGS. 2-5, operation | movement of the karaoke apparatus by the said structure is demonstrated. Below, each operation | movement of the karaoke process, partial scoring process, and MFCC calculation process which CPU10 performs is described.

(1) Karaoke processing operation
FIG. 2 is a flowchart showing the operation of karaoke processing. When the apparatus power is turned on, the CPU 10 advances the process to step SA1 shown in FIG. 2 and waits for a karaoke start instruction. Here, when a karaoke start instruction is generated in response to an operation of a start / stop switch provided in the switch unit 14, the determination result in step SA1 is “YES”, and the process proceeds to the next step SA2.

  In step SA2, karaoke data (lyric data and audio data) of a song preselected by the operation of the song selection switch is read from the karaoke data memory 13, and the lyrics data in the read karaoke data is supplied to the video encoder 17 to input lyrics. Convert to video output for subtitle display. In step SA2, the voice data in the read karaoke data, that is, the accompaniment data of the karaoke track compressed and encoded in the MP3 format and the sample song data of the vocal track are supplied to the codec 16 to be decoded (decompressed). .

  Next, in step SA3, the karaoke accompaniment sound signal obtained by D / A converting the accompaniment data decoded in step SA2 and the singing voice signal supplied from the microphone 15 are mixed to generate an audio output. The codec 16 is instructed. Thus, for example, if audio output is supplied to an external audio input terminal of a television receiver (not shown) and video output is supplied to a video input terminal, lyrics subtitles are displayed on the screen and karaoke accompaniment sounds are reproduced. .

  When the karaoke accompaniment starts, the CPU 10 advances the process to step SA4, stores the model song data decoded by the codec 16 in the above step SA2 in the buffer area of the RAM 12, and in the subsequent step SA5, the user who generates the codec 16 The song data is stored in the buffer area of the RAM 12.

  In step SA6, based on both voice feature values MFCC extracted from the sample song data and user song data for 1024 sampling points buffered in the RAM 12, the user with respect to the sample song sound (example song data). The degree of similarity of the singing sound (user singing data) is calculated, and a partial scoring process (described later) for singing a song based on the result of determining the suitability according to the calculated degree of similarity is executed. The partial scoring process is executed every 256 msec because data for 1024 sampling points buffered in the RAM 12 is used.

  Next, in step SA7, it is determined whether or not there is a karaoke stop instruction. If there is no karaoke stop instruction, the determination result is “NO”, and the process returns to step SA2 described above. Thereafter, until the karaoke accompaniment reaches the end of the song or the karaoke stop instruction is generated by operating the start / stop switch of the switch unit 14, the above-described steps SA2 to SA6 are repeated to advance the karaoke accompaniment. A score is given every 256 msec. For example, when the karaoke accompaniment reaches the end of the song and a karaoke stop instruction is issued, the determination result in step SA7 is “YES”, and the process proceeds to step SA8 to execute the scoring process.

  In the scoring process, it is determined whether or not the ratio between the number of frame counters and the number of frames of the entire music is a certain value or more. As will be described later, the frame counter is a counter that counts frames of model singing data that are not silent. A frame refers to a segment of data buffered every 1024 sampling points (every 256 msec). The number of frames of the entire music corresponds to the number obtained by dividing the model song data by frames.

  Therefore, in the scoring process, it is determined whether or not the karaoke accompaniment has been sung over a certain ratio. If the singing is not performed over a certain ratio, the partial scoring obtained in the partial scoring process in step SA6 is invalidated and the singing evaluation is performed. Is scored as zero, and the process proceeds to the next step SA9.

  On the other hand, if the karaoke accompaniment is sung at a certain ratio or more, the partial score obtained by the partial scoring process in step SA6 is divided by the number of frame counters, and the percentage of the value is calculated as score data. Thereafter, the process proceeds to step SA9, where the calculated score data is converted into a video output by the video encoder 17, whereby the user's singing score is displayed on the screen and the present process is terminated.

(2) Partial scoring operation
Next, the operation of the partial scoring process will be described with reference to FIG. When this processing is executed through the above-described karaoke processing step SA6 (see FIG. 2), the CPU 10 proceeds to step SB1 shown in FIG. 3 and samples 1024 sampling points stored in the buffer area of the RAM 12. Check if the singing data is silent.

  Subsequently, in step SB2, based on the check result in step SB1, it is determined whether the sample singing data is silent. If the sample singing data is in the silent state, it is assumed that it is not a singing part, and the determination result here is “YES”, and this processing is once completed. In this case, the frame including the model song data in the silent state is discarded, and the process waits until the next frame.

  On the other hand, if it is model singing data that is not silent, the determination result of step SB2 is “NO”, and the process proceeds to step SB3. In step SB3, the frame counter is incremented and incremented. The frame counter is a counter that counts the frames of the model song data that is not silent, and the value represents the progress position of the song. Next, in step SB4, a model song data MFCC calculation process is executed.

  Here, the operation of the MFCC calculation process will be described with reference to FIG. When the MFCC calculation process is executed via the above step SB4, the CPU 10 advances the process to step SC1 shown in FIG. 4, and singing sample data for 1024 sampling points (non-silent state) stored in the buffer area of the RAM 12 ( Hereinafter, the input signal is subjected to low-order high-pass filtering to remove a DC component (bias noise). Subsequently, in step SC2, the input signal is converted into a spectral domain by applying a fast Fourier transform FFT to the input signal from which the bias has been removed by applying a Hanning window.

  Next, in step SC3, a filter bank process is performed on the input signal converted into the spectral domain to generate a 20-dimensional spectral sequence used as a feature quantity. That is, in this filter bank processing, as shown in FIG. 5, a filter bank having 20 triangular windows having a logarithmic scale with respect to the frequency axis is used. Subsequently, in step SC4, a logarithmic process for converting a 20-dimensional spectrum sequence in the linear region into a logarithmic spectrum sequence is performed. In step SC4, a DCT process is performed in which a discrete cosine transform DCT is performed on the logarithmic spectrum sequence to convert it into a cepstrum region.

Next, in step SC6, the 12 coefficients from low order excluding the lowest-order coefficient C ₀ is the spectral DC component from among the DCT coefficients obtained by the DCT process in the step SC5, the audio characteristics of the cepstrum domain After executing a coefficient extraction process for extracting as a quantity MFCC (Mel Frequency Cepstrum Coefficient), the present process is completed and the process returns to the partial scoring process shown in FIG.

  As described above, in the example song data MFCC calculation process in step SB4, the speech feature value MFCC of the cepstrum area is calculated from the sample song data for 1024 sampling points stored in the buffer area of the RAM 12 and not in silence. It has become.

  Thereafter, the process proceeds to step SB5 shown in FIG. 3 to check whether the user singing data for 1024 sampling points stored in the buffer area of the RAM 12 is in a silent state. In step SB6, based on the check result in step SB5, it is determined whether the user singing data is silent. If the user singing data is in a silent state, the determination result is “YES” assuming that it is not a singing portion, and this processing is once completed. In this case, the frame including the user singing data in the silent state is discarded, and the process waits until the next frame.

  On the other hand, if the user song data is not silent, the determination result in step SB6 is “NO”, and the process proceeds to step SB7. In step SB7, user song data MFCC calculation processing is executed. In the user song data MFCC calculation process, the speech feature value MFCC is calculated from the user song data for 1024 sampling points that are not silenced and stored in the buffer area of the RAM 12, as in step SB4 described above.

Subsequently, in step SB8, the model song is used as a measure for measuring the similarity between the voice feature value MFCC of the model song data calculated in step SB4 and the voice feature value MFCC of the user song data calculated in step SB7. A vector a = (a ₁ , a ₂ ,..., A ₁₂ ) representing the voice feature value MFCC of the data, and a vector b = (b ₁ , b ₂ ,..., B ₁₂ ) representing the voice feature value MFCC of the user song data. Euclidean distance d (a, b) between is calculated.

  Next, in step SB9, whether or not the Euclidean distance d (a, b) calculated in step SB8 is equal to or smaller than a preset threshold value, that is, whether the model singing sound is similar to the user singing sound. Judge whether. When the Euclidean distance d (a, b) calculated in step SB8 is equal to or greater than the threshold value and the similarity between the model singing sound and the user singing sound is low, the determination result is “NO”, and this processing is performed. Finish.

  On the other hand, when the Euclidean distance d (a, b) calculated in step SB8 is less than the threshold value and the similarity between the singing sound of the model and the singing sound of the user is high, the determination result is “YES”. Thus, the process proceeds to step SB10. In step SB10, the scoring result of the frame that is the scoring target is accepted, the partial score is incremented, and the process is completed.

  As described above, in the present embodiment, accompaniment data obtained by sampling a karaoke accompaniment sound and model singing data obtained by sampling a singing sound sung by a singer as a model are stored in the karaoke data memory 13, When the accompaniment data is read from the karaoke data memory 13 and the karaoke accompaniment sound is reproduced in response to the start instruction, the user singing data obtained by sampling the singing sound sung by the user in accordance with the reproduced karaoke accompaniment sound, and the above The sample singing data read from the karaoke data memory 13 in synchronization with the accompaniment data is divided into frames for a predetermined number of data, and the voice feature quantity MFCC of the sample singing sound is calculated from the sample singing data in the divided frames. The voice feature amount MFCC of the user singing sound is extracted from the user singing data.

  Then, based on the extracted voice feature value MFCC of the model singing sound and the voice feature value MFCC of the user singing sound, the similarity of the user singing sound with respect to the model singing sound is calculated to determine suitability, and singing is scored based on the result. So, you can score whether the song lyrics are sung correctly. As a result, even a karaoke song of a style called “rap” without a clear melody can be scored.

  In this embodiment, the ratio between the number of frames sung by the user and the number of frames in which singing singing data exists in the karaoke song is taken, and the singing score is given only when the ratio is a certain value or more. Therefore, even a song with a very short performance time can be scored.

  In the embodiment described above, the singing score is calculated by calculating the similarity of the user singing sound with respect to the model singing sound based on the voice feature value MFCC, but in addition to this, the singing scoring method based on the conventional pitch extraction is used in combination. It is good also as an aspect. For example, when the accompaniment data of a karaoke song includes a melody part and a rap part, an identification flag for distinguishing the melody part from the rap part is provided in the accompaniment data, and the melody is referenced with reference to this identification flag. When the accompaniment data of the part is reproduced, the pitch is extracted and the singing is performed. On the other hand, when the accompaniment data of the rap part is reproduced, the voice feature amount MFCC is extracted and the singing is performed. If it does in this way, the appropriateness of the pitch of a user singing sound and the appropriateness of the sung lyrics can be determined simultaneously.

  In the above-described embodiment, the MFCC (Mel Frequency Cepstrum Coefficient), which is a feature amount of the cepstrum region, is extracted as a parameter representing the feature amount of the voice. However, instead of this, other parameters such as an LPC cepstrum are extracted. A feature parameter may be extracted.

In addition, in the present embodiment, as a measure for measuring the similarity between the voice feature value MFCC of the sample song data and the voice feature value MFCC of the user song data, a vector a = representing the voice feature value MFCC of the sample song data Euclidean distance d (a, b) between (a ₁ , a ₂ ,..., A ₁₂ ) and a vector b = (b ₁ , b ₂ ,..., B ₁₂ ) representing the voice feature value MFCC of the user song data. However, the present invention is not limited to this, and the degree of similarity may be calculated using another scale such as the Itakura distance.

  In the above-described embodiment, the sample singing data read from the karaoke data memory 13 in synchronization with the reproduction of the karaoke accompaniment sound is divided into frames for a predetermined number of data, and the audio feature value MFCC for each divided frame is obtained. However, instead of this, the voice feature quantity MFCC for each frame can be calculated from the model song data in advance and stored in the karaoke data memory 13 instead of the model song data. . In this way, the above-described example song data MFCC calculation process in step SB4 (see FIG. 3) is not required, and the processing load on the CPU 10 can be reduced.

10 CPU
11 Program ROM
12 RAM
13 Karaoke data memory 14 Switch unit 15 Microphone 16 Codec 17 Video encoder

Claims (2)

Data generated in synchronization with the reproduction of the karaoke song, from the model singing sound data representing the model singing sound sung as a model, and the user singing sound sung by the user in accordance with the reproduced karaoke song Framing means for dividing the obtained user singing data into predetermined frames for each predetermined number of data;
First feature extraction means for extracting a voice feature amount of model singing sound from model singing sound data for a predetermined number of data framed by the framing means;
Second feature extraction means for extracting voice feature values of user singing sound from user singing data for a predetermined number of data framed by the framing means;
Similarity calculating means for calculating the similarity between the voice feature quantity of the model singing sound and the voice feature quantity of the user singing sound respectively extracted by the first and second feature extracting means;
A determination unit that determines the suitability of the user singing sound with respect to the model singing sound for each frame according to the similarity calculated by the similarity calculating unit;
Scoring means for scoring the user's singing based on the result of determination by the determination means for each frame only when the ratio of the number of frames of the user singing data and the number of frames of the sample singing sound data is equal to or greater than a certain value A singing scoring device comprising: Data generated in synchronization with the reproduction of the karaoke song, from the model singing sound data representing the model singing sound sung as a model, and the user singing sound sung by the user in accordance with the reproduced karaoke song A framing process for dividing the obtained user singing data into predetermined frames for each predetermined number of data;
A first feature extraction process for extracting a voice feature amount of a model singing sound from model singing sound data for a predetermined number of data framed by the framing process;
A second feature extraction process for extracting voice feature quantities of user singing sound from user singing data for a predetermined number of data framed by the framing process;
A similarity calculation process for calculating a similarity between the voice feature quantity of the model singing sound and the voice feature quantity of the user singing sound, respectively extracted by the first and second feature extraction processes;
A determination process for determining the suitability of the user singing sound for the model singing sound for each frame according to the similarity calculated by the similarity calculating process;
A scoring process for scoring the user's song based on the result of determining whether each frame is appropriate or not only when the ratio between the number of frames of the user song data and the number of frames of the sample song sound data is a certain value or more. When
A singing scoring program, which is executed by a computer.

Images