JP2008040258A - Musical piece practice assisting device, dynamic time warping module, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To evaluate the level of reproduction of various techniques when a user practices in singing or musical performance modeled after the singing or musical performance that uses the techniques. <P>SOLUTION: A karaoke device which compares a first audio signal representing a user's singing voice with a second audio signal representing a singing voice of a singer as a model, and scores and outputs how much they match each other is provided with a DTW module which normalize a first acoustic parameter obtained by analyzing the first audio signal and a second acoustic parameter obtained by analyzing the second audio signal, and then performs dynamic time warping based upon those acoustic parameters. The karaoke device evaluates how much the first and the second audio signals match each other between time units made to correspond to each other by the DTW module. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for comparing and evaluating a user's song or performance with a model.

In the karaoke device, the user's singing sound is picked up by a microphone, and the time change of the pitch (pitch) of the singing sound is compared with the time change of the pitch of the karaoke accompaniment (hereinafter “guide melody”). Some of them have a function to score and output the degree of coincidence between them (hereinafter, scoring function) (for example, Patent Document 1 and Patent Document 2), and this kind of karaoke device makes it easy to practice singing. It is possible to do.
JP 2005-215493 A JP 2003-15673 A

By the way, skilled singers are not singing faithfully to the content of the score, but intentionally shifting the beginning and end of singing, changing the voice quality and volume, using vibrato and fist, etc. There are cases where emotions and tastes are expressed using singing techniques. Such feelings and tastes are expressed in various ways by the singer. For example, each singer has characteristics such as always applying vibrato to the end of the phrase or always starting the singing (deliberately delaying the timing of the beginning of the singing). There are often.
On the other hand, users who practice singing using a karaoke device often want to imitate singing techniques of their favorite singers, and when performing singing practice using a karaoke device, You may want to receive an evaluation of how well you can reproduce the singing technique.

However, the techniques disclosed in Patent Document 1 and Patent Document 2 are not only able to meet the above needs, but singing techniques such as always starting to sing are subject to deduction as deviations from the score content. There is also a case. This is because the guide melody, which is the evaluation standard in the techniques disclosed in Patent Document 1 and Patent Document 2, faithfully reproduces the change in the pitch of the music in accordance with the contents of the musical score. This is because the technique disclosed in Patent Document 2 is intended to evaluate whether or not the musical score is sung faithfully. This is not limited to the singing of music, and the same applies to the performance of musical instruments.
The present invention has been made in view of the above problems, and evaluates the degree of reproduction of a technique when a user performs a singing practice or a performance practice using a singing or a performance using various techniques as a model. The purpose is to provide technology that makes this possible.

  In order to solve the above problems, the present invention analyzes a first audio signal representing a waveform of a singing sound or performance sound by a user, and measures the degree of temporal change in volume in each time unit, spectrum And an extraction means for extracting a first acoustic parameter representing the degree of time change of the spectrum, and a second audio signal representing a waveform of a singing sound or a performance sound modeled by the user, thereby analyzing the time unit. A second acoustic parameter obtained for each time, the second acoustic parameter representing the degree of temporal change in volume, the spectrum, and the degree of temporal change in spectrum of the sound represented by the second audio signal in each time unit And obtaining means for obtaining the first acoustic parameter for each time unit, and converting the average value and the standard deviation into predetermined values. While normalizing, the normalizing means for normalizing the second acoustic parameter of each time unit, and the time axis of the first audio signal as one coordinate axis, the second audio signal An evaluation value calculated from the difference between the normalized first acoustic parameter and the normalized second acoustic parameter on the coordinate plane with the time axis as the other coordinate axis is coordinated with the time unit. A calculation means for calculating each grid point as a value, and on the coordinate plane, a path from a start point where the coordinate values are the minimum to the end point where the coordinate values are the maximum. Specifying means for specifying a path that minimizes the sum of the evaluation values at grid points on the path; and along the path specified by the specifying means, the time unit in the first audio signal and the second Audio signal A dynamic time matching module that associates the time units in the first and second audio signals with the dynamic time matching module. The dynamic time matching module uses the dynamic time matching module to match the signal waveform of the first audio signal and the signal waveform of the second audio signal. There is provided a music practice support device characterized in that a comparison is made for each time unit in which correspondence is made, and the degree of coincidence between the two is scored and output.

  In a more preferred aspect, the extraction means extracts the pitch of the sound represented by the first audio signal for each time unit, while the acquisition means obtains the pitch of the sound represented by the second audio signal. The normalization means performs the normalization on the pitch of each time unit of the first audio signal, and also the pitch of each second time unit of the second audio signal. The normalization is performed, and when the calculation unit calculates the evaluation value, a coefficient corresponding to the value of the pitch in the time unit corresponding to the coordinate value of the grid point with which the evaluation value is associated is expressed in the time unit. And multiplying the difference between the degree of temporal change in the spectrum of the first audio signal and the degree of temporal change in the spectrum of the second audio signal in FIG. It is characterized by calculating an evaluation value.

  In another preferred embodiment, the apparatus further comprises storage means for storing one or a plurality of sets of a music identifier for uniquely identifying a music and the first audio signal representing the waveform of the model song or model performance of the music. The acquisition means reads out second audio data corresponding to the music identifier designated by the user from the storage means, analyzes the second audio signal, and analyzes the second audio parameter for each time unit. It is characterized by acquiring.

  In order to solve the above-mentioned problem, the present invention analyzes a first audio signal representing a waveform of a singing sound or performance sound by a user, and the degree of temporal change in volume in a predetermined time unit. Extracting the first acoustic parameter representing the spectrum and the degree of time variation of the spectrum, and analyzing the second audio signal representing the waveform of the singing sound or performance sound modeled by the user A second acoustic parameter obtained for each time unit, the second acoustic parameter representing a degree of temporal change in volume, a spectrum, and a degree of temporal change in spectrum of the sound represented by the second audio signal in each time unit. An acquisition means for acquiring an acoustic parameter, and the average value and standard deviation of the first acoustic parameter for each time unit are set to predetermined values. Normalization means for performing normalization on the second acoustic parameters of each time unit, and the time axis of the first audio signal as one coordinate axis, and the second audio parameter An evaluation value calculated from the difference between the normalized first acoustic parameter and the normalized second acoustic parameter on the coordinate plane with the signal time axis as the other coordinate axis is the time unit. Calculating means for calculating each grid point with coordinate values, and on the coordinate plane, a path from the start point where the coordinate values are the minimum to the end point where the coordinate values are the maximum Specifying means for specifying a path that minimizes the sum of the evaluation values at the lattice points on the path; and along the path specified by the specifying means, the time unit and the first in the first audio signal. 2 Ode Providing dynamic time alignment module and having a a correlating means for correlating the time unit at the O signal.

  In order to solve the above-described problem, the present invention analyzes a first audio signal representing a waveform of a singing sound or performance sound by a user, and calculates the volume of the sound volume in a predetermined time unit. Extraction means for extracting the degree of time change, the spectrum and the first acoustic parameter representing the degree of time change of the spectrum, and the second audio signal representing the waveform of the singing sound or performance sound modeled by the user The second acoustic parameter obtained for each time unit, the degree of temporal change in volume of the sound represented by the second audio signal in each time unit, the spectrum, and the degree of temporal change in the spectrum. An acquisition means for acquiring a second acoustic parameter to be expressed, and an average value of the first acoustic parameter for each time unit, and Normalization means for performing normalization to convert the quasi-deviation into a predetermined value, while normalizing means for normalizing the second acoustic parameter in each time unit, and the time axis of the first audio signal as one coordinate axis And calculated from the difference between the normalized first acoustic parameter and the normalized second acoustic parameter on a coordinate plane having the time axis of the second audio signal as the other coordinate axis. An evaluation value is calculated for each grid point having the time unit as a coordinate value, and on the coordinate plane, from the start point where both coordinate values are the minimum grid point, Among the routes to a certain end point, specifying means for specifying a route that minimizes the sum of the evaluation values at the lattice points on the route, and the first audio signal along the route specified by the specifying means In the time unit It is made to function as a correlating means for correlating the time units in the second audio signal and provides a program characterized.

  According to the present invention, when a user performs a singing practice or a performance practice using a singing or a performance using various techniques as an example, it is possible to evaluate the degree of reproduction of those techniques. .

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(A: Configuration)
FIG. 1 is a block diagram showing an example of a hardware configuration of a karaoke apparatus 1 which is an embodiment of a music practice support apparatus according to the present invention. As shown in FIG. 1, the karaoke apparatus 1 includes a control unit 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage unit 14, a display unit 15, an operation unit 16, an audio processing unit 18, and the like. Has a bus 10 that mediates data exchange.
The control unit 11 is, for example, a CPU (Central Processing Unit), reads out a control program stored in the ROM 12, loads it into the RAM 13, and executes it to control each unit of the karaoke apparatus 1.

  The storage unit 14 is a large-capacity storage unit such as a hard disk, and has an accompaniment / lyric data storage area 14a, an exemplary voice data storage area 14b, and a trainer voice data storage area 14c.

  The display unit 15 is, for example, a liquid crystal display and a driving circuit thereof, such as a menu screen for prompting the use of the karaoke device 1 under the control of the control unit 11 or a karaoke screen in which lyrics telop is superimposed on a background image. Display various screens. The operation unit 16 includes various keys such as a numeric keypad, and outputs a signal corresponding to the pressed key to the control unit 11.

  A microphone 17 and a speaker 19 are connected to the sound processing unit 18. The microphone 17 collects the singing sound of a user (hereinafter, a practitioner) who practice singing using the karaoke device 1 and outputs an audio signal (analog data) corresponding to the singing sound to the audio processing unit 18. . The audio processing unit 18 converts the audio signal (analog data) output from the microphone 17 into audio data (digital data) and outputs the audio data (digital data) to the control unit 11, while converting the audio data delivered from the control unit 11 into an audio signal. The data is converted and output to the speaker 19. The speaker 19 emits a sound corresponding to the sound signal output from the sound processing unit 18.

In the accompaniment / lyric data storage area 14a of the storage unit 14, accompaniment data in which performance sounds (so-called guide melodies) of various musical instruments performing accompaniment for each of one or a plurality of music pieces are recorded in the order of progress of the music pieces, and the music pieces. Lyric data indicating the lyrics of the words are stored in association with each other. The accompaniment data is, for example, data in the MIDI (Musical Instruments Digital Interface) format, and is reproduced when the practitioner sings a karaoke song. The lyrics data is displayed on the display unit 15 as a lyrics telop at the time of the karaoke song.
The accompaniment data and lyrics data stored in the accompaniment / lyric data storage area 14a to be described in more detail include an identifier for uniquely identifying a karaoke song (for example, a music code consisting of English letters, symbols, numbers, etc .: (Music identifier) is associated, and accompaniment data and lyrics data are associated with each other by this music identifier. This music identifier is used when the trainee is allowed to specify the music to be practiced.

  In the exemplary voice data storage area 14b, a WAVE that represents a voice waveform of a singing sound (hereinafter, exemplary voice) of a song by a singer who has a song identified by the song identifier in association with the song identifier described above. Format audio data (hereinafter, exemplary audio data) is stored. This model voice data is used as a reference when evaluating a practitioner's song.

  In the practitioner voice data storage area 14c, voice data (hereinafter, practitioner voice data) A / D converted from the microphone 17 via the voice processing unit 18 is stored, for example, in the WAVE format.

  Next, the functional configuration of the karaoke apparatus 1 will be described with reference to the block diagram shown in FIG. The basic analysis module 21, the dynamic time warping (hereinafter referred to as DTW) module 22, and the evaluation module 23 shown in FIG. 2 are software modules realized by the control unit 11 executing the control program described above. It is. The arrows in the figure schematically show the flow of data. In addition to the above three software modules, the karaoke performance module for reproducing the accompaniment sound of the karaoke song designated by the practitioner and synthesizing and outputting the accompaniment sound and the singing sound of the practitioner is also the control program. However, the function of the karaoke performance module is not different from the function of the conventional karaoke apparatus, and illustration and detailed description thereof are omitted.

The basic analysis module 21 performs acoustic parameters (pitch, volume, spectrum) for each of the model voice data read from the model voice data storage area 14b and the trainer voice data read from the trainer voice data storage area 14c. , The degree of temporal change in volume, and the degree of temporal change in spectrum) are extracted every predetermined time unit (one frame unit in this embodiment). In the present embodiment, the case where the time unit for extracting the acoustic parameter from each of the model voice data and the trainer voice data is set to one frame. However, the acoustic parameter is set in units of subframes obtained by further dividing one frame. Of course, the acoustic parameters may be extracted in units of a plurality of frames. In short, the time unit for extracting the acoustic parameters from the model voice data and the time unit for extracting the acoustic parameters from the trainer voice data need only coincide with each other, and the length of the time unit is not limited.
As shown in FIG. 2, the basic analysis module 21 includes pitch detection means 211, volume detection means 212, spectrum detection means 213, and differentiation means 214 a to 214 c, and the voice delivered to the basic analysis module 21. The data (that is, the model voice data or the practice person voice data) is divided into three parts as shown in FIG. 2 and delivered to each of the pitch detection means 211, the volume detection means 212, and the spectrum detection means 213.

  The pitch detection unit 211 obtains autocorrelation for the audio data for the predetermined time unit, detects the pitch in the time unit, and outputs pitch data indicating the detection result. The pitch data output from the pitch detector 211 is delivered to the DTW module 22 as shown in FIG. In this embodiment, the case where the pitch is detected in each time unit by obtaining the autocorrelation has been described. However, for example, the pitch may be detected by obtaining the cepstrum for each time unit. good.

  The sound volume detection means 212 calculates the average of the absolute values of the amplitudes of the samples (256 samples in the present embodiment: see FIG. 3) included in the audio data for the predetermined time unit, and calculates the calculation result. Output as volume data indicating the volume in that time unit. The volume data output from the volume detection means 212 is divided into two as shown in FIG. 2, one of which is delivered to the DTW module 22, and the other is delivered to the differentiation means 214a.

  The differentiating unit 214a calculates a primary differential (hereinafter referred to as “speed”) for the volume from the volume data for a plurality of continuous time units, and outputs volume velocity data indicating the calculation result. The volume speed data indicates whether the volume of the voice represented by the voice data tends to increase or decrease over the plurality of time units. In the present embodiment, as shown in FIG. 3, the differentiating unit 214a generates volume speed data from the volume data for five consecutive frames. The volume speed data is divided into two as shown in FIG. 2, one of which is delivered to the DTW module 22, and the other is delivered to the differentiating means 214b.

  The differentiating means 214b calculates the first derivative (that is, the second derivative of the volume: hereinafter, the acceleration of the volume) from the volume speed data for a plurality of continuous time units, and obtains the volume acceleration data indicating the calculation result. Output. This volume acceleration data indicates whether the degree of change in volume speed represented by the volume speed data tends to increase or decrease over a plurality of continuous time units. As shown in FIG. 2, the volume acceleration data output from the differentiating means 214 b is delivered to the DTW module 22.

  As shown in FIG. 3, the spectrum detection means 213 performs FFT (Fast Fourier Transform) on the audio data for two continuous time units, and further, a bandpass filter having a predetermined pass band (in this embodiment, singing Since sound data of sound is input, it passes through a 1/2 octave bandpass filter from 0 to 2 kHz and a 1/4 octave bandpass filter from 2 to 8 kHz), and its output is in the above time unit. Output as spectrum data representing a spectrum (that is, each passband component). The spectrum data output from the spectrum detection means 213 is divided into two as shown in FIG. 2, one of which is delivered to the DTW module 22 and the other is delivered to the differentiation means 214c.

The differentiating means 214c calculates the spectral velocity from the spectral data for a plurality of continuous time units (in this embodiment, five consecutive frames), and outputs spectral velocity data indicating the calculation result. The spectral velocity data output from the differentiating means 14c is delivered to the DTW module 22 as shown in FIG.
The above is the configuration of the basic analysis module 21.

Next, the functional configuration of the DTW module 22 will be described.
As shown in FIG. 4, the DTW module 22 is for specifying the correspondence between each time unit of the trainee voice and each time unit of the model voice. As shown in FIG. 2, the normalizing means 221 is used. , A difference matrix generation unit 222, and an optimum route specifying unit 223.
The normalizing means 221 receives acoustic parameters in each time unit from the beginning of the singing to the end of singing from the basic analysis module 21 for each of the model voice and the practice person voice, and normalizes each of the voices to normalize the difference matrix generating means 222. Hand over to Here, the normalization of data means that a series of acoustic parameters delivered from the basic analysis module 21 for each time unit is subjected to conversion such that the addition average and standard deviation become constant values. In the present embodiment, the normalization is performed according to the following formula 1.
(Equation 1) AfterDat [i] = (BeforDat [i])-AVR) / STD
In Equation 1, BeforDat [i] is an acoustic parameter for the i-th frame delivered from the basic analysis module 21, SDV is a standard deviation for the acoustic parameter, and AVR is an addition average for the acoustic parameter. Yes, AfterDat [i] is a normalized acoustic parameter for the i-th frame.
By applying the normalization shown in Equation 1, the acoustic parameters delivered from the basic analysis module 21 are acoustic parameters having an addition average of “0” and a standard deviation of “1” (that is, data according to a standardized normal distribution). ) Will be converted. By performing such normalization, it is possible to remove the difference between the sound collection environment of the singer's voice and the sound collection environment of the model voice and compare the singer's voice and the model voice. In addition, there is a difference in volume level between the model voice and the practitioner voice, and the fact that the pitch is different in octave units is almost unrelated to the skill of singing. It is also possible to remove factors such as inherent differences, and to mitigate the effects caused by sudden changes in pitch and volume.

The difference matrix generation unit 222 obtains the Euclidean distance (hereinafter also referred to as “difference”) between the acoustic parameter for each time unit of the trainee voice and the acoustic parameter for each time unit of the model voice, and uses the difference value as a component. A matrix (hereinafter referred to as a difference matrix) is generated in the RAM 13. For example, when the practicing voice singing starts at the 0th frame and the singing end is the Nth frame, the singing of the model voice starts at the 0th frame and the singing end is the Mth frame (N, M Is a natural number greater than or equal to 1), and the difference matrix generation means 222 sets the value represented by the following formula 2 as (i, j) components (where 0 ≦ i ≦ N, 0 ≦ j ≦ M) (N + 1) rows ( A difference matrix of M + 1) columns is generated.
(Equation 2) Sqr {(Σ (GuideSpectrum [j] [k] −SingerSpectrum [i] [k]) ^ 2) * WeightScalar [k]
+ (Σ (ΔGuideSpectrum [j] [k] −ΔSingerSpectrum [i] [k]) ^ 2) * WeightVector [k]
+ (ΔGuidePower [j] −ΔSingerPower [i]) ^ 2)
+ (ΔΔGuidePower [j] −ΔΔSingerPower [i]) ^ 2)
} / num
In this equation 2,
GuideSpectrum [j] [k]: Spectral component of the kth passband of the jth frame of the model voice
SingerSpectrum [i] [k]: Spectral component of the k-th passband of the i-th frame of the trainee voice ΔGuideSpectrum [j] [k]: k-th spectral velocity of the j-th frame of the model voice ΔSingerSpectrum [i] [k]: k-th spectral speed of the i-th frame of the trainer voice ΔGuidePower [j]: volume speed of the j-th frame of the model voice ΔSingerPower [i]: volume speed of the i-th frame of the trainer voice ΔΔGuidePower [j]: Volume acceleration of the j-th frame of the model voice ΔΔSingerPower [i]: Volume acceleration of the i-th frame of the trainer voice
WeightScalar [k]: Weighting coefficient
WeightVector [k]: Weighting factor
num: number of parameters for obtaining the Euclidean distance (for example, when the trainer voice is from the 0th frame to the Nth frame and the model voice is from the 0th frame to the Mth frame, num = (N + 1 ) × (M + 1)).
However, WeightScalar [k] is a coefficient that weights the acoustic parameters that do not depend on time change, and the trainer's singing sound and the model voice are sound (periodic sound) or silence (non-periodic) It is a value that is appropriately selected depending on whether it is a voice. Specifically, if both the practitioner's singing sound and the model voice are voiced, the value is selected so that the low-frequency spectrum is weighted, and both the practitioner's singing sound and the model voice are silent. In the case of, the value is selected so that a weight is given to the high-frequency spectrum. Note that whether the trainer singing sound and the model voice are voiced or silent is determined based on each pitch and volume. Specifically, the difference matrix generation unit 222 determines that the corresponding time unit is sound when the pitch is equal to or greater than a predetermined threshold and the volume is equal to or greater than the predetermined threshold. Is determined to be silent.
On the other hand, WeightVector [k] is a coefficient for weighting the acoustic parameter depending on time change, and is a coefficient for assigning a weight to the mid-range spectrum.
In Equation 2, the symbol Σ means the sum for the subscript k (that is, the sum of the spectral components for all passbands), “^ 2” means the square, and Sqr {} is the square root. Means.

The optimum path specifying means 223 is a path from the lower left corner (that is, (0, 0) component) to the upper right corner (that is, (N, M) component) in the difference matrix generated by the difference matrix generating means 222. Among them, the route that minimizes the cumulative value of each component located on the route is identified as the optimum route that represents the correspondence between the trainer's voice and the model voice in each time unit, and the correspondence between the times indicated by the route Is transferred to the evaluation module 23.
More specifically, the optimum route specifying unit 223 specifies the optimum route according to the rules described below.
(Rule 1) The route search is started from the lower left corner of the difference matrix, and the process of selecting the movement destination is repeated until reaching the upper right corner so that the accumulated value of the movement destination components is minimized.
However, one movement is limited to one of right, top, and top right. For example, movement from the (i, j) component is limited to movement to the (i, j + 1) component, (i + 1, j) component, or (i + 1, j + 1) component.
If the cumulative value when moving to the right is equal to the cumulative value when moving upward, priority is given to the movement to the right. Similarly, when the accumulated value of the movement to the right and the movement to the upper right is equal, priority is given to the movement to the right, and when the accumulated value of the movement to the upper and the movement to the upper right is equal, Give priority to movement.
(Rule 2) The route selected in accordance with Rule 1 is traced backward from the upper right corner to the lower left corner to identify the optimum route.

The evaluation module 23 in FIG. 2 compares the signal waveform for each time unit corresponding to the model voice and the trainer voice that have been associated with each time unit by the DTW module 22, and the trainer voice for the model voice. The matching degree is scored and displayed on the display unit 15.
As described above, the hardware configuration of the karaoke apparatus 1 according to the present embodiment is the same as the hardware configuration of a general computer apparatus, and the functions characteristic of the music practice support apparatus according to the present invention are software. This is realized by modules (that is, basic analysis module 21 and DTW module 22). In this embodiment, the case where the basic analysis module and the DTW module characteristic of the music practice support device according to the present invention are implemented by software modules has been described. However, these modules may be implemented by hardware. Of course.
The above is the configuration of the karaoke apparatus 1.

(B: Operation)
Next, among the operations performed by the karaoke apparatus 1, the operations that remarkably show the characteristics (that is, the operations of the basic analysis module 21 and the DTW module 22) will be described with reference to the drawings. In the operation example described below, it is assumed that the power (illustrated) of the karaoke apparatus 1 has been turned on, and the control unit 11 operates according to a control program loaded from the ROM 12 to the RAM 13.

  A practitioner who wants to practice singing using the karaoke apparatus 1 appropriately operates the operation unit 16 while referring to a menu screen or the like displayed on the display unit 15, thereby obtaining a song identifier of a song for which singing practice is desired. Specify the music to be practiced by entering it. When the music to be practiced is designated in this way, the control unit 11 loads accompaniment data and lyrics data corresponding to the music identifier from the storage unit 14 to the RAM 13. Then, when the practitioner performs an operation to instruct the start of performance on the operation unit 16, the control unit 11 causes the audio processing unit 18 to start reproducing the accompaniment sound according to the accompaniment data read to the RAM 13. At the same time, a karaoke screen in which the lyrics telop represented by the lyrics data is embedded is displayed on the display unit 15, and the lyrics are wiped as the music progresses.

  A practitioner who visually recognizes the karaoke screen and listens to the accompaniment sound emitted from the speaker starts singing the song when the singing timing of the song is reached. The practicing person's singing sound is picked up by the microphone 17, and the practicing person's voice data corresponding to the singing sound is sequentially written in the practicing person's voice data storage area 14c. When the trainer voice data is stored in the trainer voice data storage area 14c in this manner, the control unit 11 associates the trainer voice data with the song identifier designated by the user on the menu screen. The model voice data stored in the voice data storage area 14b is read out, and the scoring process shown in FIG. 5 is executed. In this operation example, the trainee voice is spread over four frames from the 0th frame to the third frame, while the model voice is spread over 5 frames from the 0th frame to the fourth frame. Shall.

  FIG. 5 is a flowchart showing the flow of scoring processing performed by the control unit 11 according to the control program. As shown in FIG. 5, the control unit 11 extracts the acoustic parameters for each time unit from the beginning of the singing to the end of the singing, and the time from the beginning of the singing to the end of the singing. An acoustic parameter is extracted for each unit (step SA0100). The process of step SA0100 is executed by the basic analysis module 21 described above.

  Next, the control unit 11 normalizes the acoustic parameters extracted in step SA0100 (step SA0110), and generates a difference matrix from the normalized acoustic parameters (step SA0120). Note that the processing in step SA0110 is executed by the normalization means 221 described above, and the processing in step SA0120 is executed by the difference matrix generation means 222. In this operation example, as a result of executing the processing up to step SA 0120, the difference matrix of 4 rows and 5 columns shown in FIG. 6 is generated and stored in the RAM 13.

Next, the control unit 11 specifies the optimum route from the difference matrix generated in Step SA0120 (Step SA0130). The process of step SA0130 is a process executed by the optimum route specifying unit 223 described above. Specifically, the optimum route specifying unit 223 specifies the optimum route in the procedure described below.
First, the optimum path specifying unit 223 calculates the cumulative value associated with the movement according to the above (Rule 1) for the component belonging to the 0th column (that is, the leftmost column) of the difference matrix (see FIG. 7). For example, since the value of the (0, 0) component is “1” and the value of the (1,0) component is “4” (see FIG. 6), from the (0, 0) component to (1, 0) The accumulated value accompanying the movement to the component is “5” (see FIG. 7). Since the value of the (2, 0) component is “1”, the cumulative value accompanying the movement of (0, 0) component → (1, 0) component → (2, 0) component is “6”. (See FIG. 7). Hereinafter, the cumulative value of the components accompanying the movement is calculated up to the (3, 0) component, and the result shown in FIG. 7 is obtained. When calculating the cumulative value associated with the movement, the optimum route specifying unit 223 uniquely identifies the movement source component (in this embodiment, two subscripts of the component) and the movement destination. The identifiers indicated by the components are stored in the RAM 13 in association with each other. For example, when moving from the (0,0) component to the (1,1) component, character string data “(0,0) → (1,1,)” is stored in the RAM 13. The data stored in the RAM 13 in this way is used as a back pointer in the trace back when specifying the optimum route.

As in the case of the 0th column described above, the optimum route specifying unit 223 accumulates components accompanying movement in the 1st column. Specifically, the optimum route specifying means 223 first accumulates components accompanying movement from the (0, 0) component to the (0, 1) component. As shown in FIG. 8, since the value of the (0, 0) component is “1” and the value of the (0, 1) component is “3”, the (0, 0) component to (0, 1) The cumulative value accompanying the movement to the component is “4”.
Next, the optimum route specifying means 223 accumulates the components accompanying the movement to the (1,1) component, but the point to be noted here is that the movement pattern to the (1,1) component is as follows: There are three possible patterns described below. That is, upward movement from the (0, 1) component to the (1, 1) component, upward movement from the (0, 0) component to the (1, 1) component, and (1, 0) It is a rightward movement from the component to the (1, 1) component.

The optimum route specifying means 223 selects a movement pattern in which the cumulative value of the component accompanying the movement becomes the minimum among the three movement patterns, and the cumulative value accompanying the movement to the (1, 1) component according to the movement pattern. Is calculated. As shown in FIG. 7, the cumulative value accompanying the movement from the (0, 1) component to the (1, 1) component is “5”, and the movement from the (0, 0) component to the (1, 1) component. Since the accumulated value accompanying the movement from the (1, 0) component to the (1, 1) component is “6”, the optimum path specifying means 223 is (1, 1). ) “2” (that is, the cumulative value accompanying the movement in the upper right direction from the (0, 0) component) is adopted as the cumulative value accompanying the movement to the component.
Hereinafter, similarly, the optimum route specifying unit 223 repeats accumulation of component values accompanying the movement until reaching the upper right corner (that is, the (4, 5) component) of the difference matrix (see FIG. 9).

As shown in FIG. 9, when the accumulation of the components accompanying the movement is completed up to the upper right corner of the difference matrix, the optimum route specifying unit 223 starts the upper right corner as the starting point and performs the process of tracing the back pointer described above at the lower left corner. Repeat until it reaches the lattice point, and identify the optimal route candidate. As a result, for the difference matrix shown in FIG. 7, the optimum route candidate shown in FIG. 10 (that is, (4,5) → (3,4) → (2,3) → (2,2) → (1,1) )) Is identified.
Next, the optimum route specifying means 223 reversely traces the optimum route candidate specified as described above, and when moving away from the optimum route candidate, the accumulated value increases with the movement. This is confirmed, and the optimum route is specified (see FIG. 11).

  The optimum route identified as described above represents a correspondence relationship between the time axis of the model voice and the time axis of the trainee voice. Specifically, the optimum route shown in FIG. 11 indicates that each time unit for the model voice corresponds to each time unit for the trainee voice as shown in FIG. The optimum route specifying means 223 generates data indicating the correspondence shown in FIG. 12, and outputs the data to the evaluation module 23 (step SA0140).

Hereinafter, the evaluation module 23 performs time alignment (expansion and contraction of the time axis) on the trainer voice data so as to satisfy the correspondence specified by the optimum route specifying means 223, and then compares the result with the model voice data, and the comparison result is obtained. The score is displayed on the display unit 15.
What should be noted here is that the reference data for the evaluation by the evaluation module 23 is not a guide melody such as MIDI data, but a model representing a singer's singing sound with a song that the practitioner is singing. It is a point that is voice data. The model voice data reflects techniques that are characteristic of the singer, such as “for” the singing, but the timing of singing deviates from the content of the score because these techniques are fully utilized. In the past, it was difficult to compare with the practitioner's singing sound. However, in the karaoke apparatus 1 according to the present embodiment, as a result of the dynamic time alignment performed by the DTW module 22, the time axis of the trainer voice is associated with the time axis of the model voice, and both are compared and evaluated. Is possible.
Thus, according to the karaoke apparatus 1 according to the present embodiment, when a user performs a singing practice using a singing in which various techniques are used as a model, it is possible to evaluate the degree of reproduction of those techniques. The effects are as follows.

(C: deformation)
Although one embodiment of the present invention has been described above, it is needless to say that the embodiment may be modified as described below.
(1) In the above-described embodiment, when the basic analysis module 21 and the DTW module 22 are incorporated into the karaoke apparatus, when singing is performed using various techniques, the singing is used as a model. The case where it is possible to evaluate the degree of coincidence with the technique is described. However, the acoustic parameter extraction target by the basic analysis module 21 and the dynamic time alignment processing target by the DTW module 22 are not limited to the above-mentioned singing sound. The performance sound data and the model performance data that serves as an example may be used, and it can also be used to learn foreign languages such as English conversation.

(2) In the above-described embodiment, when performing dynamic time matching between the practitioner singing sound and the model voice, the model voice data stored in the model voice data storage area 14b is used as the basic analysis module 21 each time. The case where the acoustic parameters for the singing sound represented by the exemplary voice data are calculated has been described. However, it is of course possible to obtain the acoustic parameters for the model voice data in advance and store the acoustic parameters in association with the music identifiers in the storage unit 14.
Further, in the above-described embodiment, the case where the model audio data is stored in the storage unit 14 provided in the karaoke apparatus 1 has been described, but a CD-ROM (Compact Disk-Read Only Memory) or a DVD (Digital Versatile Disk) is described. Model audio data and acoustic parameters extracted from the exemplary audio data are written and distributed on a computer-readable recording medium such as), and the exemplary audio data is read by reading the exemplary audio data and acoustic parameters from such a recording medium. And acoustic parameters may be acquired, or acoustic parameters for model voices may be acquired via a telecommunication line such as the Internet.

(3) In the above-described embodiment, the basic analysis module 21 that extracts the acoustic parameters from the trainer speech data and the model speech data, and the time axis of the model speech and the trainer speech are correlated based on the acoustic parameters. Although the case where the DTW module 22 to be performed is realized as a separate software module has been described, it is needless to say that it may be configured as one software module. Specifically, the function to extract acoustic parameters from the trainer's voice data and the model voice data to the dynamic time matching module that performs normalization of the acoustic parameters and dynamic time matching using the normalized acoustic parameters It is sufficient to have a function of acquiring acoustic parameters for the model voice by extracting the data or reading from the recording medium or the like.

(4) In the above-described embodiment, the case has been described in which the pitch, volume, and spectrum, and the first and second derivatives of the volume and the first derivative of the spectrum are used as the acoustic parameters for performing dynamic time matching. Of these acoustic parameters, the first and second derivatives of the volume represent the degree of temporal change in volume, but the second derivative is not necessarily essential. Of course, the spectrum may be obtained up to the second derivative in order to accurately reflect the degree of time change by dynamic time matching.

(5) In the above-described embodiment, the model voice data representing the singing sound of the karaoke song by the singer who has the karaoke song identified by the song identifier in association with the song identifier that uniquely identifies the karaoke song is used. The case where it memorize | stores in the memory | storage part 14 was demonstrated.
However, if a plurality of singers individually have a song as a song, different song identifiers may be given as different songs for each singer, and each song is uniquely identified. A singer identifier that uniquely identifies each of the plurality of singers is associated with the song identifier to be identified, and further, the set of the song identifier and the singer identifier is identified by the singer identifier of the song identified by the song identifier The model voice data representing the singing sound by the singer may be stored in the storage unit 14 in association with each other. As mentioned above, the singing technique is generally different for each singer, and even if the singer is different even if it is the same song, the feeling and taste that can be put into the singing is generally different. . If it is configured so that singers can be identified as described above, even if a plurality of singers have a single piece of music as a song, the user can respond to his / her preference from among those singers. It becomes possible to select a song by a singer and practice singing by imitating the song.

(6) In the above-described embodiment, it is determined whether the voice is voiced or silent based on the pitch and volume of the trainer voice and the model voice, and an acoustic parameter that does not depend on time change according to the determination result (above In the embodiment, the case of switching the weight to be given to the spectrum) has been described, but it is of course possible to determine whether sound is present or not based on only the pitch or only the volume. It should be noted that when the sound / silence determination is performed based only on the pitch, it is needless to say that the volume data need not be transferred from the basic analysis module 21 to the DTW module 22, and the sound / silence is determined based only on the volume. Needless to say, when the silence is determined, the basic analysis module 21 does not need to detect the pitch (that is, it is not necessary to provide the pitch detection means 211). In addition, since the weight switching as described above is not always essential, it is not necessary to detect the pitch or deliver the volume data from the basic analysis module 21 to the DTW module 22 in an aspect in which such switching is not performed. Not too long.

(7) In the above-described embodiment, for each of (N + 1) frames representing from the beginning to the end of singing of the practitioner, (M + 1) frames representing from the beginning of the example to the end of singing A case has been described in which a Euclidean distance from each of the above is calculated, a difference matrix of (N + 1) rows (M + 1) columns is generated, and an optimum route is searched using all components of the difference matrix.
However, since the singing of the karaoke song is performed in accordance with the progress of the music shown by the wipe display of the guide melody and the lyrics telop, the progress of the singer voice and the progress of the example are extremely shifted (for example, The 0th frame of the trainee voice does not correspond to the Mth frame of the model). For this reason, when searching for the optimum route, the process of step SA0130 is performed only for components in which the difference between two subscripts uniquely representing the components of the difference matrix (frame number difference: that is, time difference) is within a predetermined range. In addition, when generating a difference matrix, only a component having a difference between two subscripts uniquely representing the component (frame number difference: that is, time difference) within a predetermined range is expressed by Equation 2. Of course, the Euclidean distance shown in FIG. 6 may be calculated, and the process of step SA0130 may be executed only for the component thus calculated.
In this way, it is possible to reduce the number of calculations required for searching for the optimum route (in the latter mode, the number of calculations required for generating the difference matrix), and to reduce the hardware resources required for dynamic time alignment ( For example, the storage capacity of the RAM 13 and the usage rate of the control unit 11 can be reduced.

(8) In the above-described embodiment, a case has been described in which a control program for causing the control unit 11 to realize functions characteristic of the music practice support device according to the present invention is written in the ROM 12 in advance. The control program may be recorded and distributed on a computer-readable recording medium such as a DVD, or the control program may be distributed by downloading via a telecommunication line such as the Internet. By installing the control program distributed in this way into a general computer device and operating the computer device according to the control program, the computer device has the same function as the dynamic time alignment module according to the present invention. Can be granted.

It is a block diagram which shows an example of the hardware constitutions of the karaoke apparatus 1 which concerns on one Embodiment of this invention. It is a block diagram which shows the function structural example of the karaoke apparatus. It is a figure for demonstrating the acoustic parameter extraction performed by the basic analysis module 21. FIG. It is a figure which shows an example of the execution result of the dynamic time alignment process performed by the same DTW module. It is a flowchart which shows the flow of the scoring process which the same karaoke apparatus 1 performs. It is a figure which shows an example of a difference matrix. It is a figure which shows an example of the difference matrix in the optimal path | route specific process. It is a figure which shows an example of the difference matrix in the optimal path | route specific process. It is a figure which shows an example of the difference matrix in the optimal path | route specific process. It is a figure which shows an example of the difference matrix in the optimal path | route specific process. It is a figure which shows an example of the optimal path | route identified by the optimal path | route identification process. It is a figure for demonstrating the processing result of a dynamic time alignment process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Karaoke apparatus, 11 ... Control part, 12 ... ROM, 13 ... RAM, 14 ... Memory | storage part, 15 ... Display part, 16 ... Operation part, 17 ... Microphone, 18 ... Sound processing part, 19 ... Speaker, 21 ... Basics Analysis module 211 ... Pitch detection means 212 ... Volume detection means 213 ... Spectrum detection means 214a, 214b, 214c ... Differentiation means 22 ... DTW module 221 ... Normalization means 222 ... Difference matrix generation means 223 ... Optimal route specifying means, 23... Evaluation module.

Claims (5)

A first audio signal representing a waveform of a singing sound or performance sound by a user is analyzed, and a first time representing a degree of temporal change in volume, a spectrum, and a degree of temporal change in spectrum for each predetermined time unit. Extraction means for extracting acoustic parameters;
A second acoustic parameter obtained for each time unit by analyzing a second audio signal that represents a waveform of a singing sound or performance sound that is modeled by the user, and represents the second audio signal Obtaining means for obtaining a second acoustic parameter representing a degree of temporal change in volume in each time unit of sound, a spectrum, and a degree of temporal change in spectrum;
Normalizing the first acoustic parameter for each time unit with normalization for converting the average value and the standard deviation into predetermined values, and applying the normalization to the second acoustic parameter for each time unit And
The normalized first acoustic parameter and the normalized are in a coordinate plane with the time axis of the first audio signal as one coordinate axis and the time axis of the second audio signal as the other coordinate axis. Calculating means for calculating an evaluation value calculated from a difference from the second acoustic parameter for each lattice point having the time unit as a coordinate value;
On the coordinate plane, the total sum of the evaluation values at the grid points on the path from the start point where the coordinate values are the minimum to the end point where the coordinate values are the maximum is the minimum. A specific means of identifying the route to become,
Association means for associating the time unit in the first audio signal and the time unit in the second audio signal along the path specified by the specifying means;
A dynamic time alignment module having,
The signal waveform of the first audio signal and the signal waveform of the second audio signal are compared for each time unit associated by the dynamic time matching module, and the degree of coincidence between the two is scored. A music practice support device characterized by output. The extracting means extracts the pitch of the sound represented by the first audio signal for each time unit, while the obtaining means acquires the pitch of the sound represented by the second audio signal for each time unit. ,
The normalizing means performs the normalization also on the pitch of each time unit of the first audio signal, while performing the normalization on the pitch of the second audio signal in each time unit,
When calculating the evaluation value, the calculating means calculates a coefficient corresponding to the pitch value in the time unit corresponding to the coordinate value of the grid point with which the evaluation value is associated with the first audio in the time unit. 2. The music practice support device according to claim 1, wherein the evaluation value is calculated by multiplying a difference between a degree of temporal change in the spectrum of the signal and a degree of temporal change in the spectrum of the second audio signal. . A storage means for storing one or a plurality of sets of a song identifier for uniquely identifying a song and the first audio signal representing the waveform of the model song or model performance of the song;
The acquisition means reads out second audio data corresponding to the music identifier designated by the user from the storage means, analyzes the second audio signal, and calculates the second acoustic parameter for each time unit. The music practice support device according to claim 1, wherein the music practice support device is acquired. A first audio signal representing a waveform of a singing sound or performance sound by a user is analyzed, and a first time representing a degree of temporal change in volume, a spectrum, and a degree of temporal change in spectrum for each predetermined time unit. Extraction means for extracting acoustic parameters;
A second acoustic parameter obtained for each time unit by analyzing a second audio signal that represents a waveform of a singing sound or performance sound that is modeled by the user, and represents the second audio signal Obtaining means for obtaining a second acoustic parameter representing a degree of temporal change in volume in each time unit of sound, a spectrum, and a degree of temporal change in spectrum;
Normalizing the first acoustic parameter for each time unit with normalization for converting the average value and the standard deviation into predetermined values, and applying the normalization to the second acoustic parameter for each time unit And
The normalized first acoustic parameter and the normalized are in a coordinate plane with the time axis of the first audio signal as one coordinate axis and the time axis of the second audio signal as the other coordinate axis. Calculating means for calculating an evaluation value calculated from a difference from the second acoustic parameter for each lattice point having the time unit as a coordinate value;
On the coordinate plane, the total sum of the evaluation values at the grid points on the path from the start point where the coordinate values are the minimum to the end point where the coordinate values are the maximum is the minimum. A specific means of identifying the route to become,
Association means for associating the time unit in the first audio signal and the time unit in the second audio signal along the path specified by the specifying means;
A dynamic time alignment module comprising: Computer equipment,
A first audio signal representing a waveform of a singing sound or performance sound by a user is analyzed, and a first time representing a degree of temporal change in volume, a spectrum, and a degree of temporal change in spectrum for each predetermined time unit. Extraction means for extracting acoustic parameters;
A second acoustic parameter obtained for each time unit by analyzing a second audio signal that represents a waveform of a singing sound or performance sound that is modeled by the user, and represents the second audio signal Obtaining means for obtaining a second acoustic parameter representing a degree of temporal change in volume in each time unit of sound, a spectrum, and a degree of temporal change in spectrum;
Normalizing the first acoustic parameter for each time unit with normalization for converting the average value and the standard deviation into predetermined values, and applying the normalization to the second acoustic parameter for each time unit And
The normalized first acoustic parameter and the normalized are in a coordinate plane with the time axis of the first audio signal as one coordinate axis and the time axis of the second audio signal as the other coordinate axis. Calculating means for calculating an evaluation value calculated from a difference from the second acoustic parameter for each lattice point having the time unit as a coordinate value;
On the coordinate plane, the total sum of the evaluation values at the grid points on the path from the start point where the coordinate values are the minimum to the end point where the coordinate values are the maximum is the minimum. A specific means of identifying the route to become,
Association means for associating the time unit in the first audio signal and the time unit in the second audio signal along the path specified by the specifying means;
A program characterized by functioning as

Images