JP2008039833A - Voice evaluation apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide voice evaluation technology for evaluating sound volume change in each syllable in a karaoke device for scoring singing. <P>SOLUTION: A comparison evaluation section 7 reads non-sound region reference data stored in a prescribed area in a storage section 14 and singer voice non-sound region data, and compares both of the non-sound regions for each syllable, and judges a mismatching amount, and thereby, this is reflected to scoring of singer's voice. Thus, a time axis of a singer voice data is expanded or contracted, and matched to a time axis of a guide vocal data, and by comparing the sound volume change for each syllable which is separated with syllable separation data, even delicate change in each syllable is evaluated. Consequently, an accurate scoring result can be output, and regarding a point to be corrected further, training is carried out by specifying a correction point for each syllable. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for evaluating a volume change in each syllable in a karaoke apparatus for scoring a song.

Some karaoke apparatuses have a scoring function for displaying the skill of a singer's singing in points. Of these scoring functions, data such as pitch data and volume data extracted from the singer's singing voice signal, and singing melody of karaoke songs, so that the actual singing skill and scoring result correspond as much as possible. And the corresponding data (guide melody). (For example, Patent Document 1)
Japanese Patent Laid-Open No. 10-69216

The karaoke apparatus having such a scoring function makes it possible to compare and change the volume change of each note in units of one sound. This scoring function is based on MIDI (Musical Instruments Digital Interface: registered trademark). ) Since it was compared with the singing voice of the singer based on the guide melody that was converted into data, the scoring was based on the notes on the score. However, actual singing varies in volume within a single note. For example, there are techniques such as crescendo that gradually increases the sound within one note, and staccato that cuts the sound short, and the volume changes variously within the note.
Therefore, when scoring based on the guide melody, the singer who sings close to the sample singing (hereinafter referred to as “guide vocal”) and the singer who does not sing, may not have an actual skillful scoring result. there were.

  This invention is made in view of the above-mentioned situation, and it aims at providing the audio | voice evaluation apparatus which compares and evaluates a guide vocal and a song person's singing voice | voice about the volume change in the notebook of a karaoke song. .

  In order to solve the above-mentioned problems, the present invention provides a storage means for storing first voice data indicating a singing voice of music and voice separation data indicating a voice break of the singing voice, and the storage according to the progress of the music. Reading means for reading out the first voice data and the voice separation data from the means, voice input means for inputting the voice of the singer, converting the input voice into second voice data, and outputting the voice data; The second voice data is analyzed with reference to the first voice data and the voice break data read by the reading means, and the syllable portion corresponding to the syllable of the voice indicated by the first voice data is determined as the first voice data. The syllable association means cut out from the two audio data and the first and second audio data are compared in terms of the volume change mode for each syllable part associated by the syllable association means. To provide a speech evaluation apparatus characterized by comprising an evaluation means for evaluating that corresponds to the result.

  Moreover, in another preferable aspect, the syllable association means sets the time width of the second audio data for the first and second audio data corresponding to the associated syllable portion, as the first audio data. It may be expanded and contracted to be the same as the time width.

  In another preferred aspect, the evaluation means divides the sound volume at a predetermined level for each syllable part associated by the syllable association means into sections using a threshold as a threshold, and determines that the volume is lower than the threshold. The evaluation may be performed based on the length of the determined section or the length of the section determined to be larger than the threshold value. Further, the evaluation means may automatically set the threshold value for each syllable part associated by the syllable association means based on the volume of the syllable part. Further, a period until the volume of each syllable part exceeds the threshold value may be excluded from the evaluation.

  In another preferable aspect, the evaluation unit may extract and compare the volume change rate of the syllable part for each syllable part associated by the syllable association unit.

  Evaluate even subtle changes in each syllable by expanding and contracting the time axis of the singer's voice data, aligning it with the time axis of the guide vocal data, and comparing the volume change for each syllable divided by the syllable break data. Can do.

  Hereinafter, an embodiment of the present invention will be described.

<Embodiment>
FIG. 1 is a block diagram illustrating a hardware configuration of a karaoke apparatus 1 as a voice evaluation apparatus according to an embodiment of the invention. A CPU (Central Processing Unit) 11 reads a computer program stored in a ROM (Read Only Memory) 12 or a storage unit 14, loads it into a RAM (Random Access Memory) 13, and executes it to execute a karaoke apparatus. 1 part is controlled. The storage unit 14 is a large-capacity storage unit such as a hard disk, and has a music data storage area 14a and a singer voice data storage area 14b. The display unit 15 is, for example, a liquid crystal display, and displays various screens such as a menu screen for operating the karaoke apparatus 1 and a karaoke screen in which lyrics telop is superimposed on a background image under the control of the CPU 11. The operation unit 16 includes various keys and outputs a signal corresponding to the pressed key to the CPU 11. The microphone 17 picks up sound produced by the singer. The sound processing unit 18 performs A / D conversion on the sound collected by the microphone 17 and supplies it to the CPU 11. The speaker 19 is connected to the audio processing unit 18 and emits sound based on the audio signal output from the audio processing unit 18.

  A plurality of song data of karaoke songs are stored in the song data storage area 14a, and each song data has a guide melody track, an accompaniment data track, a lyrics data track, a guide vocal track, and a syllable break data track. .

  The guide melody track is data indicating the melody of the vocal part of the music, and event data such as note-on (pronunciation command), velocity (sound intensity), note-off (silence command) for each note, and the following Delta time data indicating the time until the event data is read and executed. The accompaniment data track is composed of a plurality of tracks of each accompaniment instrument, and each instrument track has the same data structure as the above-described guide melody track. In the case of this embodiment, data in the MIDI (registered trademark) format is stored.

  The lyrics data track has text data indicating the lyrics of the music, line feed data indicating the line break of the lyrics, and wipe start time data indicating the wipe start time for each character of the lyrics. Then, when it is played back by the karaoke device and the lyrics telop is displayed on the screen, one character of the lyrics starts to change color from the left side of the character at the corresponding wipe start time, and at the next character wipe start time. When it reaches, the color change is controlled so that the color change is completed for the entire character. In this case, the wipe start time data is also provided in the line feed data, and for the character displayed at the end of the line, the time interval between the wipe start time data of the character and the wipe start time data of the line feed data is It becomes the color change time of characters. The speed at which the color of each character changes is determined from the number of dots in the horizontal direction of the character and the color change time (the time difference between the wipe start time of the character and the wipe start time of the next character).

The guide vocal track is audio data (hereinafter referred to as guide vocal data) in which a singer's voice as a sample is recorded, and is audio data in the WAVE format or MP3 (MPEG Audio Layer-3) format, for example. As shown in FIG. 2, the syllable separation data track is syllable separation data indicating the time at which the guide vocal is divided into syllables, and indicates the time when each syllable is uttered as t ₁ , t ₂ , t _3. ing. Here, the vertical axis in the figure represents the volume of the guide vocal, the horizontal axis represents the time axis indicating the progress of the guide vocal, and the lyrics corresponding to each syllable are represented in the upper part of the figure. When information such as a frame number and a sampling number is attached to the guide vocal data, these may be used as syllable delimiter data instead of the time information.

  In the singer voice data storage area 14b, voice data A / D converted from the microphone 17 via the voice processing unit 18 (hereinafter referred to as singer voice data) is stored in time series, for example, in WAVE format or MP3 format. Is done.

  Next, functions realized by the CPU 11 executing a computer program stored in the ROM 12 or the storage unit 14 will be described. FIG. 3 is a block diagram illustrating functions realized by the CPU 11.

In the figure, a guide vocal volume extraction unit 2 reads guide vocal data and syllable separation data from the music data storage area 14a, extracts the volume of the guide vocal, and creates guide vocal volume data, and syllable separation data. Has the function of dividing the guide vocal volume data for each syllable according to the syllable break time, and generating the guide vocal syllable unit volume data for all syllables. For example, as shown in FIG. 2, the guide vocal syllable unit volume data corresponding to the syllable “a” is volume data indicating a change in volume over time between t ₁ and t ₂ .

The guide vocal silent region extraction unit 3 extracts the silent region time for each syllable as reference data for comparison with the singer's voice based on the guide vocal syllable unit volume data, and associates the silent region time with each syllable. It has a function to create silent area reference data. For example, when the guide vocal syllable unit volume data of the nth syllable indicates a volume change as shown in FIG. 4A, from the time t _nth when the volume falls below the threshold volume V _nth. The silent region time t _noff = t _{n + 1} -t _nth until the next syllable separation time t _{n + 1} is created as the silent region reference data of the _nth syllable. Here, the threshold volume V _nth is automatically set as follows, for example. When the volume change in the nth syllable as shown in FIG. 4 (a) is divided into frame units divided by a predetermined time unit, and the frequency of occurrence of each volume is expressed by the number of frames, as shown in FIG. 4 (b). It can be represented by a histogram. Here, the volume minimum value of n-th syllable and _{V nmin,} when the median of the volume and _{V NMED,} threshold volume _{V nth} is automatically set as _{(V nmin + V nmed) /} 2. When the threshold sound volume is determined in this way, it becomes difficult to be affected by ambient noise entering the microphone 17 and accurate detection can be performed. When the volume of ambient noise is kept constant, the threshold volume may be determined as a certain level.

  The alignment unit 4 has a function of adjusting a temporal shift between the syllables of the guide vocal and the singer's voice. As shown in FIG. 5, when there is a deviation between the guide vocal (FIG. 5 (a)) and the singer's voice (FIG. 5 (b)), in order to accurately compare the two, the guide vocal and the time axis Therefore, it is necessary to perform DTW (Dynamic Time Warping: time normalization) by expanding and contracting the time axis of the singer's voice. In the present embodiment, DP (Dynamic Programming) matching is used as a technique for performing this DTW. Specifically, the processing is as follows.

  The alignment unit 4 forms a coordinate plane (hereinafter referred to as a DP plane) as shown in FIG. The vertical axis of the DP plane is the absolute value of the spectrum of each frame for the spectrum obtained by separating guide vocal data into frames each having a predetermined time length and applying FFT (Fast Fourier Transform) to each. The horizontal axis represents the spectrum of the singing voice data obtained in the same manner, and the logarithm of the absolute value of the spectrum obtained from each frame is the inverse Fourier. It corresponds to the parameter obtained by conversion. In FIG. 6, a1, a2, a3... An are obtained by arranging the frames of the guide vocal data according to the time axis, and b1, b2, b3... Bn are the frames of the singer voice data. They are arranged according to the time axis. The intervals of a1, a2, a3... An on the vertical axis and the intervals of b1, b2, b3... Bn on the horizontal axis all correspond to the time length of the frame. Each lattice point in the DP plane corresponds to a DP matching score which is a value indicating the Euclidean distance of each parameter of a1, a2, a3... And each parameter of b1, b2, b3. It is attached. For example, the lattice points positioned by a1 and b1 include parameters obtained from the first frame of a series of frames of guide vocal data and parameters obtained from the first frame of a series of frames of singing voice data. A value indicating the Euclidean distance is associated. After forming the DP plane having such a structure, the alignment unit 4 performs the entire path from the lattice point (starting end) positioned by a1 and b1 to the lattice point (end) positioned by an and bn. For each route searched, the DP matching score of each lattice point traced from the start end to the end is accumulated to obtain the minimum accumulated value. The path with the smallest accumulated value of the DP matching score is considered as a scale of expansion / contraction when the time axis of each frame of the singer's voice data is expanded / contracted in accordance with the time axis of the guide vocal data.

  And the alignment part 4 is the process which specifies the path | route where the accumulated value of DP matching score becomes the minimum from a DP plane, and is the process which expands / contracts the time axis | shaft of singer voice data according to the content of the specified path | route. I do. Specifically, the singer voice so that the DP matching score of each lattice point on the path specified from the DP plane represents the Euclidean distance of the parameter obtained from the frame having the same position on the time axis. After rewriting the contents of the time stamp of each frame of data, each frame having the same position on the time axis is sequentially associated as a set. For example, in the path marked on the DP plane shown in FIG. 6, it can be seen that the path from the starting point positioned by a1 and b1 progresses to the lattice point positioned by upper right a2 and b2. In this case, since the positions on the time axis of the frames a2 and b2 are the same from the beginning, it is not necessary to rewrite the contents of the time stamp of the frame b2. Furthermore, in this route, it can be seen that the grid point positioned by a2 and b2 advances from the grid point positioned by a2 and b3 on the right. In this case, not only the frame b2 but also the frame b3 need to have the same position on the time axis as the frame a2, so that the time stamp paired with the frame b3 is one frame earlier. Replace with As a result, the frame a2 and the frames b2 and b3 are associated as a set of frames having the same position on the time axis. Such time stamp replacement and frame association are performed for all frame sections from b1 to bn. As a result, for example, as shown in FIG. 5B, even if there is a portion where the sounding time of the singer's voice is deviated from the sounding time of the guide vocal, the time axis of the singer's voice data is set to the time of the guide vocal data. The time axis can be adjusted as shown in FIG. The above is the mechanism of DP matching.

  The singer voice volume extracting unit 5 extracts the volume of the singer's voice from the singer voice data obtained by the alignment unit 4 and subjected to the time expansion / contraction, as in the case of the guide vocal volume extraction unit 2. It has a function of creating voice volume data and a function of generating singer voice syllable unit volume data for each syllable.

The singer voice silence area extraction unit 6, like the guide vocal silence area extraction unit 3, sets a silence area time (for example, _t′noff for the nth syllable) for each syllable based on the singer voice syllable unit volume data. ) Is extracted, and a singer's voice silence area data in which a silence area time is associated with each syllable is created.

The comparative evaluation unit 7 acquires the silent region reference data from the guide vocal silent region extraction unit 3 and the singer voice silent region data from the singer voice silent region extraction unit 6, and each of the guide vocal and the singer voice is syllable. It has a function of comparing the silent region time for each and evaluating the sound length of the syllable. For example, the silent region time t _3off corresponding to the third syllable of the guide vocal shown in FIG. _5A and the silent region corresponding to the third syllable of the singer voice _subjected to the time expansion and contraction shown in FIG. The time t ′ _3off is compared, and if t _3off > t ′ _{3off, the} third syllable is evaluated as having a short silent region time of the singer's voice, that is, a long sounding time.

  Next, the operation of the karaoke apparatus 1 will be described. The practitioner operates the operation unit 16 of the karaoke apparatus 1 to select a song to be sung and instructs the accompaniment to be reproduced. The CPU 11 starts processing in response to this instruction. The CPU 11 first reads the accompaniment data track of the designated song from the song data storage area 14 a and supplies it to the audio processing unit 18. The sound processing unit 18 converts the supplied accompaniment data into an analog sound signal and supplies the analog sound signal to the speaker 19 for sound emission. At this time, the CPU 11 controls the display unit 15 to read out the lyrics data track from the song data storage area 14a, displays the read lyrics, and changes the characters of the lyrics in accordance with the progress of the song. The singer sings along with the accompaniment emitted from the speaker 19. At this time, the voice of the singer is picked up by the microphone 17, converted into a voice signal, and supplied to the voice processing unit 18. The singer voice data A / D converted by the voice processing unit 18 is stored in time series in the singer voice data storage area 14b of the storage unit 14.

  When the reproduction of the accompaniment data is completed, the CPU 11 performs processing of the alignment unit 4. That is, the guide vocal data is read from the music data storage area 14a, and the singer voice data read from the singer voice data storage area 14b is read. Then, by DP matching, the time axis of the singer voice data is expanded and contracted so as to match the time axis of the guide vocal data, the time stamp of the singer voice data is rewritten, and stored in the singer voice data storage area 14b of the storage unit 14 To do.

  Next, the CPU 11 performs processing of the guide vocal volume extraction unit 2 and the singer voice volume extraction unit 5. That is, the guide vocal volume extraction unit 2 guides the vocal vocal syllable unit volume data in association with each syllable for each syllable of the guide vocal based on the guide vocal data and the syllable separation data read from the music data storage area 14a. Is generated and stored in a predetermined area of the storage unit 14. Similarly, the singer voice volume extraction unit 5 generates singer voice syllable unit volume data in association with the syllable for each syllable of the singer voice data with the rewritten time stamp, and the storage unit 14. Stored in a predetermined area.

  Next, the CPU 11 performs processing of the guide vocal silent area extracting unit 3 and the singer voice silent area extracting unit 6. The guide vocal silent region extraction unit 3 reads guide vocal syllable unit volume data for all syllables stored in a predetermined area of the storage unit 14, calculates threshold volume for each syllable of the guide vocal, Silence area time for all syllables is calculated and stored in a predetermined area of the storage unit 14 as silence area reference data. Similarly, the singer voice silence area extraction unit 6 reads out singer voice syllable unit data for all syllables stored in a predetermined area of the storage unit 14, and for all syllables of the singer voice. Then, the threshold volume is calculated, the silent area time for all syllables is calculated, and is stored in a predetermined area of the storage unit 14 as singer voice silent area data.

Next, the CPU 11 performs processing of the comparative evaluation unit 7. The comparative evaluation unit 7 reads out the silent region reference data and the singer's voice silent region data stored in a predetermined area of the storage unit 14 and compares the silent region times of both for each syllable. Then, each syllable of the singer's voice is compared with the silent area time of each syllable of the guide melody, and the amount of deviation is judged, so that it is reflected in the singer's voice scoring. In order to instruct the singer, which syllable is shifted how may be displayed on the display unit 15. In such a case, for example, as shown in FIG. 7, display may be performed so that the silent region time of the syllable can be understood for each syllable of the lyrics. Here, the horizontal axis represents each syllable of the lyrics. In addition, the vertical axis is the silence region time of each syllable normalized by the total time of the syllable divided by the syllable separation data. The larger the numerical value, the longer the silent region time of the syllable, that is, the shorter the pronunciation time. It means that, n-th silence area time syllable _{as t noff / (t n + 1} -t n), are displayed. The guide vocal is treated as the teacher's voice, and the singer's voice is treated as the student's voice.

  In this way, the time axis of the singer's voice data is expanded and contracted to match the time axis of the guide vocal data, and the volume change for each syllable divided by the syllable break data is compared, so that subtle changes within each syllable can also be detected. Evaluation can be made. Therefore, a highly accurate scoring result can be obtained, and points that should be further corrected can be instructed by clearly specifying correction points for each syllable.

  As mentioned above, although embodiment of this invention was described, for example, you may implement this invention, changing embodiment mentioned above as follows.

<Modification 1>
In the embodiment, the syllable sound lengths of the guide vocal and the singer's voice are compared. However, as the sound volume changes, the crescendo that gradually increases and the decrescendo that gradually decreases are evaluated. You may go. In this case, instead of extracting the silent region of the embodiment, as shown in FIG. 8, the volume change rate α in each syllable is extracted by extracting the volume change of each syllable by primary approximation or the like, What is necessary is just to compare a guide vocal and a singer's voice. Here, for example, as shown in FIG. 8, if the time occupied by the syllable is set to 100%, the section to which the volume change rate is compared is added with a time of 30% from the time when the syllable is pronounced. the time t _ns 70% of the time from may be such a properly set time until the time t _ne plus a. In this way, the inflection in each syllable can be evaluated, and a more accurate scoring result can be obtained.

<Modification 2>
In the embodiment, the syllable separation data is created in advance for each music piece. However, the syllable separation data may be automatically created from the spectrum obtained from the guide vocal data and the detected / non-detected state of the pitch. . In this way, it is possible to save the trouble of creating syllable break data for a large number of music pieces.

<Modification 3>
In the embodiment, the silent region times of the guide vocal and the singer's voice are extracted and compared with each other, but only the region where the volume is higher than the threshold volume is detected and the syllable is pronounced. You may extract and compare the sound area time to consider. Moreover, you may consider a part other than a silence area | region as a sound area.

<Modification 4>
In the embodiment, by DP matching, the time axis of the singer's voice data is expanded and contracted to match the time axis of the guide vocal data, the time axis is adjusted, and the singer's voice is divided into syllables by the syllable separation data. The singing voice may be divided into syllables by comparing the spectrum and pitch of the guide vocal data and the singing voice data and detecting the syllable of the singing voice corresponding to each syllable of the guide vocal. In this case, since the time axes of the syllabary of the guide vocal and the singer's voice are not aligned, it may be compared as the ratio of the silent area time to the time occupied by the entire syllable.

It is a block diagram which shows the structure of the hardware of the karaoke apparatus which is the audio | voice evaluation apparatus which concerns on embodiment of this invention. It is explanatory drawing which shows the time which delimits the syllable which syllable delimiter data has. It is a block diagram which shows the structure of the software of the karaoke apparatus which is the audio | voice evaluation apparatus which concerns on embodiment of this invention. It is explanatory drawing which shows the detection method of silence area | region time. It is explanatory drawing which shows expansion / contraction of the time of singer audio | voice data. It is explanatory drawing which shows DP plane at the time of performing DP matching. It is explanatory drawing which shows the example of the screen on which the evaluation result of a singer voice is displayed. It is explanatory drawing which shows the volume change rate which is the audio | voice evaluation method which concerns on the modification 1. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Karaoke apparatus, 2 ... Guide vocal volume extraction part, 3 ... Guide vocal silence area extraction part, 4 ... Alignment part, 5 ... Singer voice volume extraction part, 6 ... Singer voice silence area extraction part, 7 ... Comparison evaluation , 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... storage unit, 14a ... song data storage area, 14b ... singer voice data storage area, 15 ... display unit, 16 ... operation unit, 17 ... microphone, 18 ... Sound processor, 19 ... Speaker

Claims (6)

Storage means for storing first voice data indicating a singing voice of music and voice break data indicating a break of voice of the singing voice;
Reading means for reading out the first audio data and the audio delimiter data from the storage means according to the progress of the music;
Voice input means for inputting the voice of the singer, converting the input voice into second voice data, and outputting the second voice data;
The second voice data is analyzed with reference to the first voice data and the voice break data read by the reading means, and the syllable portion corresponding to the syllable of the voice indicated by the first voice data is Syllable association means cut out from the second audio data;
An evaluation unit that compares the first and second audio data with respect to a change in volume for each syllable portion associated by the syllable association unit, and performs an evaluation corresponding to the comparison result; A voice evaluation apparatus characterized by the above.   The syllable association means makes the time width of the second sound data the same as the time width of the first sound data for the first and second sound data corresponding to the associated syllable portions. The voice evaluation device according to claim 1, wherein the voice evaluation device is expanded and contracted.   The evaluation means divides the sound volume of a predetermined level for each syllable part associated by the syllable association means into sections using a threshold as a threshold, and determines the length of the section determined to be less than the threshold or the The voice evaluation apparatus according to claim 1, wherein the evaluation is performed based on a length of a section determined to have a volume larger than a threshold value.   4. The speech evaluation apparatus according to claim 3, wherein the evaluation unit automatically sets the threshold value for each syllable part associated by the syllable association unit based on a volume of the syllable part. .   5. The speech evaluation apparatus according to claim 3, wherein the evaluation unit excludes a period until the volume of each syllable part exceeds the threshold value from the evaluation.   The speech evaluation according to claim 1 or 2, wherein the evaluation unit extracts and compares a volume change rate of the syllable part for each syllable part associated by the syllable association unit. apparatus.

Images