JP4135615B2 - Musical sound comparison device and musical sound comparison program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To exactly judge matching and mismatching of inputted speech and the pitch of musical sound to be actually produced. <P>SOLUTION: A singing guide device is equipped with: a pitch extraction section 32 which extracts the pitch by each of prescribed frames from the speech data based on a speech signal of user input; a reproduction control section 30 which generates the musical sound of the pitch based on the musical sound data at the sound production timing of the musical piece data to be reproduced and silences the musical piece at a silencing timing; and a comparative decision section 34 which compares the pitch extracted by each of the frames until the sound production timing reaches the silencing timing, and the pitch based on the musical sound data, accumulates the results of the comparison by each of the frames by associating the results with the musical sound data, and decides the exactness of the pitch extracted from the speech data based on the results of the comparison after lapse of the silencing timing. <P>COPYRIGHT: (C)2005,JPO&amp;NCIPI

Description

  The present invention relates to an apparatus that compares an audio signal applied to a user input with a pitch of a musical sound that should be emitted.

  There has been proposed an apparatus for extracting the pitch of a user's singing by the pitch extraction technique and comparing the difference with a musical sound to be actually pronounced.

For example, Japanese Patent Laid-Open No. 2004-228561 compares a musical data based on a singing voice signal with performance data, and displays a musical piece reproduction device that can display out-of-pitch on a lyrics or a score displayed on the screen of the display device. Is disclosed.
JP 2000-29473 A

  For example, in the apparatus described in the above-mentioned Patent Document 1, the position of a location where a mismatch such as out of pitch or tempo has occurred is obtained. As a result, there is a problem in that accurate judgment cannot be made. For example, even if it deviates from the pitch that should be originally spoken even in a short time, if it is determined that the pitch is inconsistent, even if there is a slight mistake in the user's singing, that part is wrong. It will be judged.

  It is an object of the present invention to provide an apparatus that can accurately determine the coincidence and disagreement between an input voice and the pitch of a musical sound that should actually be emitted.

An object of the present invention is a musical sound comparison device that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and outputs a comparison result, and the voice applied to the user input Based on the sound data obtained by digitizing the signal, a pitch extracting means for extracting a pitch for each predetermined frame, and generating a musical tone having a pitch based on the musical sound data at the sounding timing of the music data to be reproduced, Comparing the reproduction means for muting the music to the mute timing of the music data, the pitch extracted for each frame from the sound generation timing to the mute timing, and the pitch based on the musical sound data, the comparison result for each frame, a comparison means for accumulating in association with the tone data for each said frame, the extracted A first counter indicating that the pitch matches the pitch based on the musical tone data, a second counter indicating that the pitch is higher than the pitch, a third counter indicating that the pitch is lower than the pitch, and the audio signal If the level does not reach a predetermined value, the comparison means for counting up the count value of any of the fourth counters indicating that there has been no input , and the first to th Referencing the counter value of the fourth counter , based on the comparison result, determination means for determining the accuracy of the pitch extracted from the audio data, and display control for displaying the determination result by the determination means on a display device And a musical tone comparison device characterized by comprising:

An object of the present invention is a musical sound comparison device that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and outputs a comparison result, and the voice applied to the user input Based on the sound data obtained by digitizing the signal, a pitch extracting means for extracting a pitch for each predetermined frame, and generating a musical tone having a pitch based on the musical sound data at the sounding timing of the music data to be reproduced, Comparing the reproduction means for muting the music to the mute timing of the music data, the pitch extracted for each frame from the sound generation timing to the mute timing, and the pitch based on the musical sound data, Comparing means for accumulating the comparison results for each frame in association with the musical sound data, extracted for each frame A first counter indicating that the pitch matches the pitch based on the musical tone data, a second counter indicating that the pitch is higher than the pitch, a third counter indicating that the pitch is lower than the pitch, and the audio signal If the level does not reach the predetermined value, the comparison means for counting up the count value of any one of the fourth counters indicating that there has been no input, and the first to thirst times after the mute timing has elapsed. Referencing the counter value of the fourth counter, based on the comparison result, determination means for determining the accuracy of the pitch extracted from the audio data, and display control for displaying the determination result by the determination means on a display device And the reproducing means is activated in response to muting the musical sound at the muting timing, and the determination means is activated.

In still another preferred embodiment, the comparison means includes a fifth counter that counts the number of times the pitch and pitch are compared for each frame, and the determination means determines that the counter value of the first counter is When the counter value of the other counter is larger than the counter value of the first counter / the counter value of the fifth counter, the ratio of the pitch and the pitch is determined, and the ratio is When it is larger than the predetermined threshold, it is determined that the pitch is accurate. The threshold value may be changed according to the set difficulty level.

An object of the present invention is a musical sound comparison device that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and outputs a comparison result, and the voice applied to the user input Based on the sound data obtained by digitizing the signal, a pitch extracting means for extracting a pitch for each predetermined frame, and generating a musical tone having a pitch based on the musical sound data at the sounding timing of the music data to be reproduced, Comparing the reproduction means for muting the music to the mute timing of the music data, the pitch extracted for each frame from the sound generation timing to the mute timing, and the pitch based on the musical sound data, Comparing means for accumulating the comparison results for each frame in association with the musical sound data, extracted for each frame A first counter indicating that the pitch matches the pitch based on the musical tone data, a second counter indicating that the pitch is higher than the pitch, and a third counter indicating that the pitch is lower than the pitch, for each frame One of a fourth counter that counts the number of times the pitch and the pitch are compared, and a comparison unit that counts up the count value, and after the mute timing has elapsed, is extracted from the audio data based on the comparison result Determining means for determining the accuracy of the pitch, when the counter value of the first counter is larger than the counter value of the other counter, the counter value of the first counter / the value of the fourth counter Based on the counter value, a ratio of the pitch and the pitch is obtained, and when the ratio is larger than a predetermined threshold, it is determined that the pitch is accurate. And judging means is achieved by the determination means according to the determination result tone comparison device, characterized in that it comprises a display control means for displaying on a display device. The threshold may be changed according to the set difficulty level.

An object of the present invention is a musical sound comparison device that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and outputs a comparison result, and the voice applied to the user input Based on the sound data obtained by digitizing the signal, a pitch extracting means for extracting a pitch for each predetermined frame, and generating a musical tone having a pitch based on the musical sound data at the sounding timing of the music data to be reproduced, Comparing the reproduction means for muting the music to the mute timing of the music data, the pitch extracted for each frame from the sound generation timing to the mute timing, and the pitch based on the musical sound data, Comparing means for accumulating the comparison results for each frame in association with the musical sound data, extracted for each frame A first counter indicating that the pitch matches the pitch based on the musical tone data, a second counter indicating that the pitch is higher than the pitch, and a third counter indicating that the pitch is lower than the pitch, for each frame One of a fourth counter that counts the number of times the pitch and the pitch are compared, and a comparison unit that counts up the count value, and after the mute timing has elapsed, is extracted from the audio data based on the comparison result Determining means for determining the accuracy of the pitch, when the counter value of the first counter is larger than the counter value of the other counter, the counter value of the first counter / the value of the fourth counter Based on the counter value, a ratio of the pitch and the pitch is obtained, and when the ratio is larger than a predetermined threshold, it is determined that the pitch is accurate. Determining means, and display control means for displaying a determination result by the determining means on a display device, wherein the reproducing means is activated in response to the sound being silenced at the mute timing. This is achieved by the featured tone comparison device. The threshold may be changed according to the set difficulty level.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the apparatus which can judge correctly whether the input audio | voice and the pitch of the musical sound which should be emitted actually correspond, and mismatch.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. FIG. 1 is a block diagram showing a hardware configuration of the singing guide device according to the first embodiment of the present invention. As shown in FIG. 1, the singing guide device 10 temporarily stores a CPU 12, a ROM 14 that stores various programs for guiding the singing and music data of the music to be played, and data according to the singing of the singer. In addition, the RAM 16 used as a work area when executing the program, the input device 16 such as a keyboard and a mouse, the display device 18 including a CRT display device or an LCD, the music data from the external device, or the external device It has a MIDI (Musical Instrument Digital Interface) interface (I / F) 20 for outputting music data, and an AD converter (ADC) 24 for converting an audio signal transmitted through a microphone 22 into a digital signal.

  FIG. 2 is a functional block diagram of the CPU 12 and its peripheral members. As shown in FIG. 2, the CPU 12 receives the music data, and reproduces the music based on the received music data, and the pitch of the singer's voice based on the digital signal from the ADC 26. A pitch extraction unit 32 to extract, a comparison / determination unit 34 that compares the pitch of the music to be reproduced and the extracted pitch, a score display control unit 36 that controls the score display of the music to be reproduced, and to be displayed And a scroll display control unit 38 for controlling the scrolling of the musical score.

  The music data is stored in the ROM 14 of the singing guide device 10 or is given to the reproduction control unit 30 from an external device via the MIDI I / F 22. FIG. 3 is a diagram showing a data structure of music data. As shown in FIG. 3, the music data has a data group as shown in Note [0], Note [1],. In a data group corresponding to each musical tone (for example, Note [0]), the sounding start tick of the musical tone, that is, the ITime indicating the number of clocks until the musical sound starts to be generated, corresponds to the time for extending the musical tone to be generated. IGate indicating the gate time to be played, Pit indicating the pitch (note number) of the musical sound to be generated, Vel indicating the velocity of the musical sound to be generated, and the comparison result between the musical sound and the pitch extracted based on the digital signal of the sound IMatchCnt, iUpCnt, iDownCnt, and sMatch, respectively, and a note pointer iNp indicating the note number of the next musical sound are included.

  The first pitch comparison result iMatchCnt indicates the number of pitch matches, and the second pitch comparison result iUpCnt indicates the number of times that the pitch of the singing is higher than the pitch of the musical tone, that is, the number of times that the pitch has increased. The pitch comparison result iDownCnt of 3 indicates the number of times that the pitch of the singing is lower than the pitch of the musical tone, that is, the so-called flat frequency. The fourth comparison result Smatch indicates the final comparison result of the pitch. As the value of Smatch, for example, a value corresponding to “◯ (maru)” indicating that the pitches match or a value corresponding to “x (X)” indicating that the pitches do not match can be taken. .

  In the present embodiment, ITTime, IGate, Pit, Vel, and iNP are prepared in advance among the above-described data group for each musical tone. Further, the pitch comparison result stores a necessary value by a comparison determination process described later.

  FIG. 4 is a main flowchart of processing executed by the singing guide device according to the present embodiment. As shown in FIG. 4, when the power is turned on, the singing guide device 10 first initializes variables used for processing (step 401) and displays a score based on music data (step). 402).

  FIG. 5 is a diagram showing variables used for processing. In this embodiment, there are parameters used for reproduction control (see FIG. 5A) and parameters used for pitch extraction (see FIG. 5B). The parameters for playback control include iResolution indicating the resolution of a quarter note, lBeat indicating the numerator part, lBeatDenomi indicating the denominator part, lBPM indicating the tempo value (BPM), doTimeOfMeas indicating the number of seconds in one measure, When displaying a score, IOffsetX indicating the number of frames of the score to be displayed in advance of the timing to be sounded, INowTime indicating the current time during playback, IBaseTime indicating the playback start time, and the current playback position during playback in ticks IowTick indicating, IOffset indicating the preceding time when displaying in advance, Reproducing note pointer np indicating which note is currently being reproduced, and up to which note among the notes that have already been pronounced is to be displayed Display note pointer nd indicating It includes prior display note pointer na showing the to be displayed notes. Further, the playback control parameters include music data Note [N] to be sounded, OnBuf [N] which is a data buffer for the music to be sounded, and iPlay and iWave which are state variables for playback and audio input. It is. These parameters will be described in more detail in connection with the playback process.

  As shown in FIG. 5B, the pitch extraction parameters include cmData [N] that is a variable (complex number) for executing FFT, doData [N] that indicates the spectrum power, doFS that indicates the sampling frequency, IFrameSize indicating the frame size, iLowerNN indicating the note number of the lowest pitch of the extracted pitch, iUpperNN indicating the note number of the highest pitch of the extracted pitch, iBNoteN indicating the integer part of the extracted pitch, and the fractional part of the extracted pitch IBNoteD indicating i, and iJudge indicating scoring results are included.

  After step 401 and step 402, the singing guide device 10 waits for input from the input device 18 by the user (step 403). More specifically, here, input by a key or a mouse for instructing to start or stop playback is waited. When the input of the input device 18 is a start instruction (Yes in Step 404), the singing guide device 10 starts a playback routine (Step 405) and also starts a voice input routine (Step 405). 406). On the other hand, if it is an end instruction (Yes in Step 407), the reproduction routine and the voice input routine are stopped (Steps 408 and 409).

  FIG. 6 is a flowchart showing a reproduction routine executed by the reproduction control unit 30 of the singing guide device 10. This reproduction routine is started in step 405 of the main flow and operates in parallel with the main routine.

  In the reproduction routine, first, note pointers np, nd and na in the reproduction control parameters are initialized (step 601). Next, initialization for scroll display is performed (step 602), and the current time is given to the reproduction start time IBaseTime (step 603).

  Next, it is determined whether or not the state variable iPlay is “0 (zero)” (step 604). Since this state variable iPlay is set to “1” during playback and “0” at the end of playback, if it is determined yes in step 604, a scoring process is executed (step 605). .

On the other hand, if it is determined NO in step 604, the current time is given to the current time INowTime at the time of reproduction (step 606). Next, the playback control unit 30 updates the time variable (step 607). Here, INowTick indicating the tick of the current playback position is updated using the following equation.
INowTick = (INowTime-IBaseTime)
* IResolution * IBPM / 60

  That is, the position (tick) corresponding to the elapsed time from the start of reproduction is calculated from the tempo variable (IBPM) and the definition of the quarter note tick (iResolution). Note that IBPM is divided by 60 because it is a beat per minute.

  Next, the playback control unit 30 refers to the calculated INowTick, and if it is time to key off, the musical tone of the music data of a predetermined note (Note [i]) in a note-off process (step 608) described later. Is turned off and at the timing when the key should be turned on, the music data of a predetermined note is generated in a note-on process (step 609) described later. Further, the playback control unit 30 compares the INowTick corresponding to the current time, the music data, and the sounding start timing of each note to determine a note to be displayed (scroll display processing: step 610). The score is scroll-displayed (score display processing: step 611). Details of the scroll display process will also be described later. The reproduction control unit 30 repeatedly executes the processing of steps 606 to 611 until a stop instruction is issued.

  Next, the voice input routine will be described with reference to FIG. This reproduction routine is started in step 406 of the main flow, and operates in parallel with the main routine at a predetermined interval, for example, with a digital signal set in the processing area of the RAM 16 via the ADC 26. As shown in FIG. 7, the CPU 12 copies the input audio data to a complex FFT calculation variable and temporarily stores it in the RAM 16 (step 701). The variable is then multiplied by a window function such as a Hanning window (step 702). Next, FFT is executed (step 703). Since the result appears in the real part and the imaginary part, the power, that is, the square root of each square sum is obtained (step 704). When voice is input through a microphone, low-frequency noise or wide-area noise may be mixed, so that filtering is performed on data in the frequency domain (step 705). This can be realized by simply performing multiplication. Next, a fundamental frequency identification process (step 706), a comparison determination process (step 707), and a pitch display process (step 708) described later are executed.

  FIG. 8 is a flowchart showing the basic frequency identification process in more detail. In this process, the fundamental frequency is identified by the power spectrum. In general, in a singing voice, the fundamental frequency is identified by matching with the spectrum structure of a voice including a harmonic overtone by utilizing the property that a harmonic overtone exists with respect to the fundamental frequency. Further, by limiting from a general range exceeding the matching pitch range, frequency identification can be realized at higher speed. First, the matching evaluation value variable dpVMax is initialized, and the note number variable iBNoteN corresponding to the identified pitch is initialized (step 801). Next, the note number variable i is set to the note number iLowerNN corresponding to the lowest pitch sound to be extracted (step 802). The processing after step 804 is repeated until the note number “i” reaches the highest pitch sound “iUpperNN” to be extracted (see step 803).

  The CPU 12 initializes a variable k for a pitch smaller than the note number (step 804). The following processing is repeated until the variable k reaches a predetermined maximum number kMAX (see step 805).

The CPU 12 initializes the overtone counter j and initializes the evaluation value variable dPitVal at a certain pitch (step 806). Further, until reaching the upper limit value JMAX of the overtone (step 807), the CPU 12 obtains the frequency to be inspected (step 808), specifies which power spectrum is the target frequency, and calculates its index (step) 809), the power spectrum is added as an evaluation value (step 810), and the overtone counter j is incremented (step 811). More specifically, since the frequency is obtained from the note number in step 808, the reference pitch is set to A4 = 440 Hz, and the note number of A4 is set to 69. In particular,
dFr = 440 * power (2, (i−69) + k / KMAX) / 12) * j
Is required.

In step 809, the index iN is
iN = (dFr * iFrameSize / doFs)
Can be obtained.

Furthermore, in step 810, the evaluation value dPitVal is
dPitCal + doData [iN] * (J-MAX-J) / JMAX
It becomes.

  When the overtone counter j reaches the upper limit value JMAX of the overtone (Yes in Step 807), the evaluation value dPitVal for the target frequency is compared with the maximum value dPVMax so far, and the larger one is the maximum value dPVMax. (Steps 812 and 813). In step 813, note number i is set as note number variable IBNoteN, and k * 100 / KMAX is set in variable iBNoteD representing a step having a pitch smaller than the note number. That is, the pitch representing the maximum value is specified by the integer part IBMNoteN and the decimal part IBMNoteD. After such processing, the variable k is incremented.

Next, the comparison determination process (step 707 in FIG. 7) will be described in more detail with reference to FIG. In this process, the pitch extracted by the fundamental frequency identification process is compared with the note number currently being sounded to check for a match or mismatch (large or small).
First, the pointer p of the note that is sounding is initialized (step 901). The CPU 12 executes the following processing until the pointer p exceeds a predetermined number (step 902).

  If the extracted pitch iBNoteN is equal to the pitch Pitch of the sounding buffer OnBuf [p] (Yes in step 904) for the data being sounded (Yes in step 903), the CPU 12 In the data, Note [OnBuf [p]. iMatchCnt indicating the number of matches in the pitch comparison result of iNp] is incremented (step 905).

  If the pitch Pitch of the sound generation buffer OnBuf [p] is larger than the extracted pitch iBNoteN (Yes in Step 906), the CPU 12 determines Note [OnBuf [p]. In step 907, iUpCnt indicating the number of times iNp] is pitched is incremented.

  Alternatively, when the pitch Pitch of the sound generation buffer OnBuf [p] is smaller than the extracted pitch iBNoteN (Yes in Step 908), the CPU 12 determines Note [OnBuf [p]. iDownCnt indicating the number of times of flattening in the pitch comparison result of iNp] is incremented (step 909). Thereafter, the note pointer p is incremented (step 910), and the process returns to step 902. In this way, a pitch comparison result can be obtained for each musical sound (Note [i]) of the music data (see FIG. 3).

  Next, steps 605, 608, 609, 610, etc. in the main flow (FIG. 6) will be described below. FIG. 10 is a flowchart showing the note-on process. Note-on generates a musical tone having a pitch when there is a musical tone to be generated at the current time. Also, data is set in the sounding buffer, and preparations are made so that processing can be easily executed during note-off processing. The CPU 12 initializes the note pointer p (step 1001), and executes the processing after step 1003 for all the notes (musical sounds). In the musical sound data, if the pronunciation start tick ITime of Note [p] has passed the current time INowTick (Yes in step 1003), the CPU 12 MIDI OUTs the note-on event (step 1004). Next, the CPU 12 initializes the pointer q of the buffer OnBuf for storing data during sound generation (step 1005), and if OnBuf [q] is empty (Yes in step 1006), the note-off is MIDI OUT. The data for this is created (step 1007). More specifically, in step 1007, in OnBuf [q], the sum of ITime and IGate of the musical sound data Note [p] is set as ITime, “1” is set in IGate, and Note [p in Pit. ] Pit, Note [p] Stat is set in Stat, and “0” is set in Vel.

  Further, the CPU 12 initializes the counter of the pitch comparison result to “0” in OnBuf “q”, stores the invalid value “−1” in the pitch comparison result SMatch, and sets “p” in the note pointer iNP. (Step 1008). Thereafter, the note pointer p is incremented (step 1010), and the process returns to step 1002. On the other hand, if no in step 1006, the parameter q is incremented to search for an empty OnBuf [q] (step 1009).

  Next, the note-off process will be described in more detail with reference to FIG. In the note-off process, the musical sound to be muted is muted along with the elapsed time, and whether or not the song has been sung correctly is determined according to the counter value acquired by the comparison determination process (FIG. 9). The CPU 12 initializes the note pointer p (step 1101), and executes the following processing for all sound generating buffers OnBuf [p].

  The CPU 12 determines whether the data is sounding and the current time has exceeded the mute time (step 1103). If yes in step 1103, the note-off event is MIDI OUT (step 1104). Next, the musical sound data Note [OnBuf [p]. iNP] based on iMatchCnt, iUpCnt, and iDownCnt, if the pitch comparison result sMatch is correct, “CORRECT” (steps 1105, 1106), if it is higher, “UPPIT” (steps 1107, 1108), it is lowered. (If flat) “DOWNNPIT” (steps 1109 and 1110) is set. After such processing, the note pointer p is incremented (step 1111).

  FIG. 12 is a flowchart showing the scoring process (step 605 in FIG. 6) in more detail. The scoring process counts the number of musical sounds that are determined to be correct in the pitch comparison result, that is, “CORRECT” in the reproduced music. More specifically, the CPU 12 initializes the note pointer p, initializes the counter iOKCnt (step 1201), and executes the following processing until the pointer p points to the end of the music (step 1201).

  If the pitch comparison result SMatch of the music data Note [p] is “CORRECT”, the CPU 12 repeats the process of incrementing the counter iOKCnt (steps 1203 to 1205). Finally, the score iJudge is calculated as 100 * iOKCnt / p (step 1206). This is the percentage that you sang correctly.

  Next, processing related to image display will be described. FIG. 13 is a flowchart showing the scroll display initialization process (step 602 in FIG. 6) in more detail. In the present embodiment, in the scroll display of the musical score based on the music data, the display is started prior to the performance timing of the music, and the notes and the extracted pitch can be displayed in an overlapping manner. In addition, the score based on the music data is scrolled to the left, and the position of the image on the screen at the currently played location is constant. In addition, the reproduction of the music is displayed based on a tick that is a time unit of the music, but the pitch of the input voice is the cut frame unit. Therefore, it is necessary to calculate how many frames one bar corresponds to and calculate the distance on the X axis for the preceding display.

As shown in FIG. 13, the CPU 12 calculates the number of ticks corresponding to one measure (step 1301). More specifically,
IOoffset = iResolution * IBeat * 4
/ IBeatDenomi
It becomes.

Next, the CPU 12 calculates the actual time of one measure (step 1302). More specifically,
doTimeOfMeas = (60 * IBeat * 4
/ IBeatDenomi) / IBPM
It becomes. Here, 1 BPM (Beat Per Minutes) is a tempo value (that is, the number of quarter notes per minute). Therefore, by multiplying by “60”, the unit is set to second.

Further, the number of frames corresponding to one bar is calculated next (step 1303). More specifically,
IOffsetX = doTimeOfMeas * doFs
/ IFrameSize
Can be obtained from

  Next, the scroll display process (step 610 in FIG. 6) will be described in more detail. In the scroll display, whether the display is right or wrong is determined in consideration of the INowTick that is a tick corresponding to the current time, the music data, and the sound generation start timing of each musical sound (note). Steps 1401 to 1403 are loops for displaying in advance, and steps 1404 to 1406 are loops for displaying at the timing to be played.

  In the first half of the loop, the CPU 12 uses the ITime of the musical tone data Note [na] played after the current performance timing and the Ioffset calculated in the scroll display initialization (FIG. 13) for the note pointer na for the preceding display. As long as the sum is smaller than INowTick (Yes in Step 1401), the musical sound is displayed (Steps 1402, 1403). In the latter half of the loop, the CPU 12 displays the musical sound for the note pointer nd for display as long as the ITime of Note [nd] is smaller than INowTick (No in step 1404) (steps 1405, 1406). ). In this way, the musical sound to be displayed can be displayed. The display position in the Y direction (vertical direction) is determined based on the Pit in the Note [na or nd].

Next, the pitch display process (step 708 in FIG. 7) will be described in more detail with reference to FIG. In the pitch display process, the entire screen is scrolled at the timing of displaying the pitch, and the position of the musical note indicating the musical tone based on the pitch extracted in the fundamental frequency identification process (see FIG. 8), or its Y direction The mark which shows the pitch concerning a user's voice is added up and down (vertical direction).
First, the CPU 12 copies the rectangular area of the screen to scroll the entire screen to the left (step 1501). Next, the CPU 12 sets the width of the display screen as w and the height as h (step 1502).

  The CPU 12 calculates the x coordinate where the mark is to be displayed (step 1503). Here, a position of a predetermined ratio of the width (a position of 2/3 in the present embodiment) is determined as the x coordinate. Further, the CPU 12 calculates the y coordinate where the mark is to be displayed (step 1504). Here, the uppermost part of the height of the display screen corresponds to the maximum pitch value (iUpperNN), and the lowermost part corresponds to the minimum value of pitch (iLowerNN). Thereafter, a predetermined mark is drawn at a position specified by the calculated x coordinate and y coordinate (step 1505).

FIG. 16 is a diagram illustrating an example of a display image. In this example, a pitch display area (reference numeral 1601) is provided on the display screen, and a pitch (for example, reference numeral 1602) corresponding to the original pitch of the score and a pitch extracted from the user's voice are shown there. A mark (see reference numeral 1603) is displayed. Here, the uppermost portion of the pitch display area 1601 is the highest pitch sound to be extracted (pitch = 75; E ♭ 5), and the lowermost portion is the lowest pitch sound to be extracted (pitch 56: A ♭ 3). ing. Further, the position in the X direction 2/3 of the screen is the performance timing, and further, the preceding display timing is provided in front of it (left side in the X direction).

  As described above, in this embodiment, when a user sings a song through a microphone while playing a song, the display screen shows a mark indicating the pitch of the voice sung along with the mark of the pitch of the musical sound. Can do. According to the present embodiment, the pitch based on the music data and the change in the pitch of the singing song are graphically displayed. Training can be effectively promoted. In addition, even if the user does not have a musical score, the user can proceed with singing training with a sense of play by referring to the screen.

The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the invention described in the claims, and these are also included in the scope of the present invention. Needless to say.
For example, in the above embodiment, as shown in FIG. 15, the mark indicating the pitch of the music is displayed at equal intervals without reflecting the length of the sound. However, the present invention is not limited to this. A mark indicating the pitch may be displayed at an interval proportional to the length of the note sound.

Moreover, the mark which shows the extracted pitch was represented as a discontinuous point (refer the code | symbol 1603 of FIG. 15). However, the present invention is not limited to this, and a continuous change may be expressed by connecting adjacent marks.
In the present embodiment, the coordinates of the current position in the music data are fixed and displayed. However, the present invention is not limited to this, and the current position may be moved to the right in the X direction. In this case, the entire screen may be rewritten when the pointer representing the X coordinate of the current position advances to the right end in the display area.

  Moreover, you may process combining a note-off process (FIG. 11) and a pitch display process (refer FIG. 15). FIG. 17 is a flowchart in which note-off processing and pitch display processing are combined. Here, steps 1701 to 1710 are the same as the processing of FIG. Thereafter, the actual time of the gate time IGate of the musical tone to be determined is obtained (step 1711). The determined real time frame is then identified (step 1712). Thereafter, the x coordinate to be displayed in the display area is calculated (step 1713), and a mark indicating the extracted pitch is added to the position of the calculated y coordinate and y coordinate corresponding to the pitch (step 1714). ). Thereafter, the note pointer p is incremented (step 1715), and the process returns to step 1702.

  Furthermore, in addition to the display of the above embodiment (see FIG. 15), the pitch comparison result may be displayed. For example, in the example of FIG. 18, a unique code is used for each of the case where the pitch of the user's song is higher (reference numeral 1801), lower (reference numeral 1802), or correct (reference numeral 1803). It may be given. Of course, this code is not limited to an arrow or the like, and may be a circle or cross mark, or a more cartoonized mark (for example, a picture of a balloon / a picture of a broken balloon, a smile / crying face). May be.

  Next, a second embodiment of the present invention will be described. In the second embodiment, more accurate pitch extraction can be performed by setting the range variable based on the designation of the target person, that is, the user's singing attribute (child, adult female, adult male). Yes. In addition, it is possible to perform the training of the song more effectively by making the pitch that the user should emit as a guide sound, or by not making a guide sound, that is, by making a mu, Training is also possible.

  Furthermore, the difficulty level is designated and the determination according to the difficulty level is realized according to the user who sings. For example, it is relatively easy for beginners and children, and strict judgments can be presented to skilled people and adults.

  Since the hardware configuration, functional block diagram, main flow, etc. in the second embodiment are the same as those in the first embodiment described above, only the added processing flow will be described here. .

  FIG. 19 is a flowchart showing the target person specifying process. The user operates the input device in advance and designates “child”, “adult female”, or “adult male” as the user attribute. In response to this, the CPU 12 stores the user attribute iVRange in the RAM 16. In the subject designation process, the CPU 12 sets initial values of the lowest sound iLowerNN and the highest sound iUpperNN of the pitch to be extracted (step 1901).

The CPU 12 refers to the designated user attribute and sets the lowest sound and the highest sound as follows (steps 1902 to 1906).
Child: lowest note = 58, highest note = 79
Adult female: lowest note = 55, highest note = 79
Adult male: lowest note = 43, highest note = 67
When the user attribute is not designated by the user, the initial value is maintained.

  Thus, by determining the range of the lowest sound and the highest sound according to the user attribute, more accurate and high-speed pitch extraction can be performed.

  Next, the pronunciation of the guide sound according to the second embodiment will be described. FIG. 20 is a flowchart showing note-on processing according to the second embodiment. In this process, processing steps are partially added to the note-on process in FIG. 10, steps 2001 to 2003 correspond to steps 1001 to 1003 in FIG. 10, and steps 2007 to 2013 are in FIG. Corresponds to Steps 1004 to 1010. Therefore, only the processing steps added below will be described. In addition, the user operates the input device in advance to designate whether or not guide sound is output. In response to this, the CPU 12 stores the value of the variable iGuideNote (“1”: with guide sound, “0”: no guide sound) indicating whether or not the guide sound is output in the RAM 16.

  When it is time to sound Note [p] in the musical sound data (Yes in Step 2003), the CPU 12 sets the velocity Vel of the Note [p] to the minimum value “1” (Step 1). 2004). Next, the value of iGuideNote is checked. If the value is “1”, that is, if a guide sound is to be output (Yes in step 2005), the velocity Vel of Note [p] is set to a predetermined value GVEL. (Step 2006). Thereby, in the case of designation without guide sound, the sound that the user should sing is not output (that is, muted), while in the case of designation with guide sound, the sound that the user should sing is output at a predetermined volume. .

  Next, comparison determination processing according to the second embodiment will be described with reference to FIG. This comparison determination process is configured to count when the level of the input waveform of the voice input by the user does not reach a predetermined value in the comparison determination process (FIG. 9) according to the first embodiment. Therefore, processing steps therefor are added. 21, steps 2101 to 2103 correspond to steps 901 to 903 in FIG. 9, and steps 2107 to 2113 correspond to steps 904 to 910 in FIG. 9. Therefore, only the added processing steps will be described.

  The CPU 12 provides counters iAllCnt and iNoInputCnt as pitch comparison results in the musical sound data Note []. When the data is sounding (Yes in Step 2103), the CPU 12 determines that Note [OnBuf [p]. iNp] counter iAllCnt is incremented (step 2105). If there is no input, that is, if the level of the waveform of the voice applied to the user input has not reached the predetermined value (Yes in step 2106), Note [OnBuf [p]. iNp] counter iNoInputCnt is incremented. Thereby, it is possible to know the total number of inputs (final counter value in iAllCnt) and the total number when there is no voice input (final counter value of iNoInputCnt) for each musical tone.

  Next, note-off processing according to the second embodiment will be described with reference to FIG. Here, using the threshold value corresponding to the degree of difficulty iDff set by user input or set according to the above-described user attribute, it is possible to determine the correct rate according to the degree of difficulty.

  The CPU 12 finds the maximum one of the pitch comparison results iMatchCnt, iUpCnt, iDownCnt, and iNoInputCnt in the sounding buffer OnBuf [p] (step 2203). If the value other than iMatchCnt is the maximum (No in Step 2204), the musical sound data Note [OnBuf [p]. As a determination result sMatch of iNp], a result (any one of UPPIT, DOWNNPIT, and NG) corresponding to the highest value among iUpCnt, iDownCnt, and iNoInputCnt is stored (step 2205).

  If it is determined yes in step 2206, the ratio r of the number of times iMatchCnt that has been sung correctly is obtained for all the determination times iAllCnt (step 2206). Next, the CPU 12 compares the threshold value iCrate [iDiff] set in accordance with the difficulty level iDiff with the above r, and if r is large (Yes in Step 2207), the correct result (CORRECT) as the determination result sMatch. ) Is stored (step 2208), otherwise NG is stored as sMatch (step 2209). This makes it possible to change the degree of correct answer (CORRECT) according to the degree of difficulty. Therefore, by setting the degree of difficulty, it is possible to realize various levels of training from beginners to advanced users.

  Further, in the second embodiment, as shown in FIG. 23, the result of the pitch comparison determination may be displayed on the score. Here, the notes that could not be sung are marked with a “cross”, the upward notes are marked with an upward arrow, and the down (flat) notes are marked with a downward arrow. In addition, the notes that were sung correctly are marked with a “circle”. By displaying in this way, the user's weakness or where they are good at can be quickly identified on the score, so it is also suitable for self-study, and appropriate instruction is also provided when students and teachers take lessons together. Can be used as a reference material.

  In the first embodiment and the second embodiment, the present invention is applied to a device that guides the user's singing. However, the input voice is not limited to a voice, but is emitted from an instrument and a microphone. It may be acquired by. Further, it goes without saying that the present invention can be applied even when an electric signal is output from the beginning as in an electronic guitar.

FIG. 1 is a block diagram showing a hardware configuration of the singing guide device according to the first embodiment of the present invention. FIG. 2 is a functional block diagram of the CPU and its peripheral members. FIG. 3 is a diagram for explaining music data according to the present embodiment. FIG. 4 is a flowchart showing a main flow of the singing guide device according to the present embodiment. FIG. 5 is a diagram for explaining control variables used in the present embodiment. FIG. 6 is a flowchart showing a reproduction routine according to the present embodiment. FIG. 7 is a flowchart showing a voice input routine according to the present embodiment. FIG. 8 is a flowchart showing frequency identification processing according to the present embodiment. FIG. 9 is a flowchart showing the comparison determination process according to the present embodiment. FIG. 10 is a flowchart showing the note-on process according to this embodiment. FIG. 11 is a flowchart showing note-off processing according to the present embodiment. FIG. 12 is a flowchart showing the scoring process according to the present embodiment. FIG. 13 is a flowchart showing scroll display initialization processing according to the present embodiment. FIG. 14 is a flowchart showing scroll display processing according to the present embodiment. FIG. 15 is a flowchart showing the pitch display process according to the present embodiment. FIG. 16 is a diagram illustrating an example of an image displayed on the screen of the display device in the present embodiment. FIG. 17 is a flowchart showing another example of the note-off process according to the present embodiment. FIG. 18 is a diagram illustrating an example of an image displayed on the screen of the display device in the present embodiment. FIG. 19 is a flowchart illustrating a target person specifying process according to the second embodiment. FIG. 20 is a flowchart illustrating note-on processing according to the second embodiment. FIG. 21 is a flowchart illustrating the comparison determination process according to the second embodiment. FIG. 12 is a flowchart illustrating note-off processing according to the second embodiment. FIG. 23 is a diagram illustrating an example of an image displayed on the screen of the display device in the present embodiment.

Explanation of symbols

10 Singing Guide Device 12 CPU
14 ROM
16 RAM
18 Input device 20 Display device 22 MIDI I / F
24 ADC
30 Playback Control Unit 32 Pitch Extraction Unit 34 Comparison Judgment Unit 36 Score Display Control Unit 38 Scroll Display Control Unit

Claims (12)

A musical sound comparison device that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and outputs a comparison result,
Pitch extraction means for extracting a pitch for each predetermined frame based on voice data obtained by digitizing the voice signal applied to the user input;
Reproduction means for generating a musical tone having a pitch based on the musical tone data at the sounding timing of the musical piece data to be reproduced, and for silencing the musical piece at the silencing timing of the musical piece data;
From the sounding timing to the mute timing, the pitch extracted for each frame is compared with the pitch based on the musical sound data, and the comparison result for each frame is accumulated in association with the musical sound data. Comparing means , for each frame, a first counter indicating that the extracted pitch matches the pitch based on the musical tone data, a second counter indicating that the pitch is higher than the pitch, and lower than the pitch Comparison means for counting up a count value of any one of a third counter indicating that the input is not performed and a fourth counter indicating that there is no input when the level of the audio signal does not reach a predetermined value When,
Determination means for determining the accuracy of the pitch extracted from the audio data based on the comparison result with reference to the counter values of the first to fourth counters after the mute timing has elapsed;
A musical sound comparison apparatus comprising: display control means for displaying a determination result by the determination means on a display device. A musical sound comparison device that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and outputs a comparison result,
Pitch extraction means for extracting a pitch for each predetermined frame based on voice data obtained by digitizing the voice signal applied to the user input;
Reproduction means for generating a musical tone having a pitch based on the musical tone data at the sounding timing of the musical piece data to be reproduced, and for silencing the musical piece at the silencing timing of the musical piece data;
From the sounding timing to the mute timing, the pitch extracted for each frame is compared with the pitch based on the musical sound data, and the comparison result for each frame is accumulated in association with the musical sound data. Comparing means , for each frame, a first counter indicating that the extracted pitch matches the pitch based on the musical tone data, a second counter indicating that the pitch is higher than the pitch, and lower than the pitch Comparison means for counting up a count value of any one of a third counter indicating that the input is not performed and a fourth counter indicating that there is no input when the level of the audio signal does not reach a predetermined value When,
Determination means for determining the accuracy of the pitch extracted from the audio data based on the comparison result with reference to the counter values of the first to fourth counters after the mute timing has elapsed;
Display control means for displaying a determination result by the determination means on a display device,
2. A musical sound comparison apparatus according to claim 1, wherein the determination means is activated in response to the reproduction means muting the musical sound at the muting timing . The comparison means comprises a fifth counter for counting the number of times the pitch and pitch are compared for each frame;
When the counter value of the first counter is larger than the counter values of the other counters, the determining means determines the pitch and the pitch based on the counter value of the first counter / the counter value of the fifth counter. The musical tone comparison apparatus according to claim 1 or 2 , wherein a ratio of matching the pitch is obtained, and when the ratio is larger than a predetermined threshold, it is determined that the pitch is accurate. A musical sound comparison device that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and outputs a comparison result,
Pitch extraction means for extracting a pitch for each predetermined frame based on voice data obtained by digitizing the voice signal applied to the user input;
Reproduction means for generating a musical tone having a pitch based on the musical tone data at the sounding timing of the musical piece data to be reproduced, and for silencing the musical piece at the silencing timing of the musical piece data;
From the sounding timing to the mute timing, the pitch extracted for each frame is compared with the pitch based on the musical sound data, and the comparison result for each frame is accumulated in association with the musical sound data. Comparing means , for each frame, a first counter indicating that the extracted pitch matches the pitch based on the musical tone data, a second counter indicating that the pitch is higher than the pitch, and a pitch A third counter indicating lower, a fourth counter for counting the number of times the pitch and pitch are compared for each frame, a comparison means for counting up the count value;
A determination means for determining the accuracy of the pitch extracted from the audio data based on the comparison result after the mute timing has elapsed, wherein a counter value of the first counter is higher than a counter value of another counter; When the ratio is larger, the ratio of the pitch and the pitch is obtained based on the counter value of the first counter / counter value of the fourth counter, and when the ratio is larger than a predetermined threshold, A determination means for determining that the pitch is accurate;
A musical sound comparison apparatus comprising: display control means for displaying a determination result by the determination means on a display device. A musical sound comparison device that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and outputs a comparison result,
Pitch extraction means for extracting a pitch for each predetermined frame based on voice data obtained by digitizing the voice signal applied to the user input;
Reproduction means for generating a musical tone having a pitch based on the musical tone data at the sounding timing of the musical piece data to be reproduced, and for silencing the musical piece at the silencing timing of the musical piece data;
From the sounding timing to the mute timing, the pitch extracted for each frame is compared with the pitch based on the musical sound data, and the comparison result for each frame is accumulated in association with the musical sound data. Comparing means , for each frame, a first counter indicating that the extracted pitch matches the pitch based on the musical tone data, a second counter indicating that the pitch is higher than the pitch, and a pitch A third counter indicating lower, a fourth counter for counting the number of times the pitch and pitch are compared for each frame, a comparison means for counting up the count value;
A determination means for determining the accuracy of the pitch extracted from the audio data based on the comparison result after the mute timing has elapsed, wherein a counter value of the first counter is higher than a counter value of another counter; When the ratio is larger, the ratio of the pitch and the pitch is obtained based on the counter value of the first counter / counter value of the fourth counter, and when the ratio is larger than a predetermined threshold, A determination means for determining that the pitch is accurate;
Display control means for displaying a determination result by the determination means on a display device,
2. A musical sound comparison apparatus according to claim 1, wherein the determination means is activated in response to the reproduction means muting the musical sound at the muting timing . The musical sound comparison device according to any one of claims 3 to 5, wherein the threshold value changes according to a set difficulty level. A computer-readable musical sound comparison program that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and operates the computer to output a comparison result. ,
A pitch extraction step of extracting a pitch for each predetermined frame based on voice data obtained by digitizing the voice signal applied to the user input;
A reproduction step of generating a musical tone having a pitch based on the musical sound data at the sounding timing of the musical piece data to be reproduced, and silencing the musical piece at the silencing timing of the musical piece data;
From the sounding timing to the mute timing, the pitch extracted for each frame is compared with the pitch based on the musical sound data, and the comparison result for each frame is accumulated in association with the musical sound data. A first counter indicating that the extracted pitch matches the pitch based on the musical tone data, and a second counter indicating that the pitch is higher than the pitch for each frame. A count value of any one of a third counter indicating that the pitch is lower than the pitch and a fourth counter indicating that there is no input when the level of the audio signal does not reach a predetermined value. A comparison step to count up,
A determination step of determining the accuracy of the pitch extracted from the audio data based on the comparison result with reference to the counter values of the first to fourth counters after the mute timing has elapsed;
A musical sound comparison program that causes the computer to execute a display control step of displaying a determination result of the determination step on a display device. A computer-readable musical sound comparison program that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and operates the computer to output a comparison result. ,
A pitch extraction step of extracting a pitch for each predetermined frame based on voice data obtained by digitizing the voice signal applied to the user input;
A reproduction step of generating a musical tone having a pitch based on the musical sound data at the sounding timing of the musical piece data to be reproduced, and silencing the musical piece at the silencing timing of the musical piece data;
From the sounding timing to the mute timing, the pitch extracted for each frame is compared with the pitch based on the musical sound data, and the comparison result for each frame is accumulated in association with the musical sound data. A first counter indicating that the extracted pitch matches the pitch based on the musical tone data, and a second counter indicating that the pitch is higher than the pitch for each frame. A count value of any one of a third counter indicating that the pitch is lower than the pitch and a fourth counter indicating that there is no input when the level of the audio signal does not reach a predetermined value. A comparison step to count up,
A determination step of determining the accuracy of the pitch extracted from the audio data based on the comparison result with reference to the counter values of the first to fourth counters after the mute timing has elapsed;
Causing the computer to execute a display control step of displaying a determination result of the determination step on a display device;
Wherein in response to said musical tone to mute timing in the playback step to cause silencing, tone comparison program characterized that you start the determination step. In the comparison step, causing the computer to execute a step of counting up the number of times the pitch and pitch are compared for each frame by a fifth counter; and
In the determination step, when the counter value of the first counter is larger than the counter value of the other counter, the pitch and the pitch value are determined based on the counter value of the first counter / the counter value of the fifth counter. The computer is caused to execute a step of obtaining a ratio with which the pitch matches, and a step of determining that the pitch is accurate when the ratio is larger than a predetermined threshold. Item 9. The musical tone comparison program according to item 7 or 8 . A computer-readable musical sound comparison program that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and operates the computer to output a comparison result. ,
A pitch extraction step of extracting a pitch for each predetermined frame based on voice data obtained by digitizing the voice signal applied to the user input;
A reproduction step of generating a musical tone having a pitch based on the musical sound data at the sounding timing of the musical piece data to be reproduced, and silencing the musical piece at the silencing timing of the musical piece data;
From the sounding timing to the mute timing, the pitch extracted for each frame is compared with the pitch based on the musical sound data, and the comparison result for each frame is accumulated in association with the musical sound data. A first counter indicating that the extracted pitch matches the pitch based on the musical tone data, and a second counter indicating that the pitch is higher than the pitch for each frame. And a third counter indicating that the pitch is lower than the pitch, and a fourth counter that counts the number of times the pitch and pitch are compared for each frame, a comparison step for counting up the count value,
A determination step for determining the accuracy of the pitch extracted from the audio data based on the comparison result after the mute timing has elapsed, in which the counter value of the first counter is set to another counter When the counter value is larger than the counter value of the first counter / the counter value of the fourth counter, the ratio of the pitch and the pitch is determined, and the ratio is a predetermined value. A determination step that determines that the pitch is accurate when the pitch is greater than
A musical sound comparison program that causes the computer to execute a display control step of displaying a determination result of the determination step on a display device. A computer-readable musical sound comparison program that compares a pitch extracted from a voice signal applied to a user input with a pitch of a musical sound that should be emitted and operates the computer to output a comparison result. ,
A pitch extraction step of extracting a pitch for each predetermined frame based on voice data obtained by digitizing the voice signal applied to the user input;
A reproduction step of generating a musical tone having a pitch based on the musical sound data at the sounding timing of the musical piece data to be reproduced, and silencing the musical piece at the silencing timing of the musical piece data;
From the sounding timing to the mute timing, the pitch extracted for each frame is compared with the pitch based on the musical sound data, and the comparison result for each frame is accumulated in association with the musical sound data. A first counter indicating that the extracted pitch matches the pitch based on the musical tone data, and a second counter indicating that the pitch is higher than the pitch for each frame. And a third counter indicating that the pitch is lower than the pitch, and a fourth counter that counts the number of times the pitch and pitch are compared for each frame, a comparison step for counting up the count value,
A determination step of determining the accuracy of the pitch extracted from the audio data based on the comparison result after the mute timing has elapsed, in which the counter value of the first counter is set to another counter When the counter value is larger than the counter value of the first counter / the counter value of the fourth counter, a ratio at which the pitch and the pitch match is obtained, and the ratio is a predetermined threshold value. A determination step for determining that the pitch is accurate when the pitch is greater;
Causing the computer to execute a display control step of displaying the determination result of the determination step on a display device;
The reproduction in response to cause silencing the musical sound mute timing in step, tone comparison program said determining step, characterized that you start. The musical tone comparison program according to any one of claims 9 to 11, wherein the threshold value changes in accordance with a set difficulty level.

Images