JP2008040260A - Musical piece practice assisting device, dynamic time warping module, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a karaoke device capable of evaluating skill in singing or musical performance using various techniques. <P>SOLUTION: Acoustic parameters (pitch, loudness, and spectrum) are detected from a model voice and a practicer's voice respectively. A section wherein detected loudness exceeds a threshold is determined as a voiced section and DTW (dynamic time wrapping) is performed for the voiced section to determine how both the voices correspond to each other. Then how much waveforms of corresponding parts of both the voices match each other is evaluated. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for comparing and evaluating a user's song or performance with a model.

Conventionally, in a karaoke apparatus, various scoring functions for evaluating a singer's singing have been proposed. This type of karaoke apparatus generates singing data indicating the singing characteristics such as the pitch, volume or tempo of the voice generated by the singer from the singer's voice input from the microphone. Then, the karaoke apparatus compares the singing data with scoring reference data such as a guide melody, and assigns a predetermined score based on the comparison result to generate scoring data. When the singing part ends, the scores in the scoring data are totaled to calculate the total score.
For example, in Patent Document 1, the pitch of the sound extracted from the karaoke guide melody and the pitch and volume of the voice generated by the singer are detected and evaluated by comparing the two, while the singing rate (the part that was actually sung) / Parts to be sung), and a technique for adding the singing rate to the above evaluation is disclosed. This solves the problem of the conventional problem, that is, when the number of parts actually sung is small, the overall evaluation is determined only by a small part of the sung voice. In Patent Document 2, the voice recorded by a singer in a karaoke box or the like is sent to a remote singing instructor via a network or the like, and the singing instructor provides the singing instruction content to the singer through a network or the like individually. A communication system that enables singing instruction is disclosed.
JP 2005-215493 A JP 2003-15673 A

By the way, skilled singers are not singing faithfully to the content of the score, but intentionally shifting the beginning and end of singing, changing the voice quality and volume, using vibrato and fist, etc. There are cases where emotions and tastes are expressed using singing techniques. Such feelings and tastes are expressed in various ways by the singer. For example, always add vibrato to the end of a phrase or always start singing (deliberately delay the timing of singing). Often has characteristics.
On the other hand, users who practice singing using a karaoke device often want to imitate the singing technique of their favorite singer, and when performing singing practice using a karaoke device, the singing technique You may want to receive an evaluation as to how much you can reproduce.

  However, the techniques disclosed in Patent Document 1 and Patent Document 2 are not only able to meet the needs as described above, but singing techniques such as starting to sing are subject to deduction as deviations from the score content. Sometimes it ends up. This is because the guide melody, which is the evaluation standard in the techniques disclosed in Patent Document 1 and Patent Document 2, faithfully reproduces the change in the pitch of the music in accordance with the contents of the musical score. This is because the technique disclosed in Patent Document 2 is intended to evaluate whether or not the score is sung faithfully. This is not limited to the singing of music, and the same applies to the performance of musical instruments.

  In view of the above problems, an object of the present invention is to provide a technique that makes it possible to more efficiently execute evaluations related to singing and playing using various singing techniques.

  The first configuration of the music practice support apparatus according to the present invention analyzes a first audio signal representing a waveform of a singing sound or performance sound by a user, and performs pitch, volume, and spectrum for each predetermined time unit. And extracting means for extracting a first acoustic parameter representing the degree of temporal change in volume and spectrum, and analyzing a second audio signal representing a waveform of a singing sound or performance sound as a model by the user A second acoustic parameter obtained for each time unit, which is a second acoustic parameter representing the pitch, volume, spectrum, and degree of time change of the volume and spectrum of the sound represented by the second audio signal in each time unit. Acquisition means for acquiring the acoustic parameters of the first audio signal, and at least one of pitch and volume from the first audio signal exceeds a predetermined threshold. A section is selected with reference to the first acoustic parameter, and a voiced section in which at least one of pitch or volume from the second audio signal exceeds a predetermined threshold is referred to the second acoustic parameter. Section selection means for selecting from the second audio signal, and the average value and standard deviation of the first acoustic parameters for each time unit in the sounded section selected by the section selection means are set to predetermined values. Normalization means for performing the normalization on the second acoustic parameter of each time unit in the sounded section selected by the section selection means while performing normalization to convert, and the time of the first audio signal The normalized first acoustic parameter and the previous position on a coordinate plane having one axis as the coordinate axis and the time axis of the second audio signal as the other coordinate axis. A calculation means for calculating an evaluation value calculated from a difference from the normalized second acoustic parameter for each lattice point having the time unit as a coordinate value, and a difference between both coordinate values on the coordinate plane is A grid point selection unit that selects only grid points that are smaller than a predetermined value, and on the coordinate plane, from a start point that is the minimum grid point of both coordinate values to an end point that is the maximum grid point of both coordinate values. A specifying means for specifying a path that passes only through the lattice points selected by the lattice point selecting means and has a minimum sum of the evaluation values at the lattice points on the route, and is specified by the specifying means. A dynamic time alignment module having an association means for associating the time unit in the first audio signal and the time unit in the second audio signal along the path, The signal waveform of the audio signal and the signal waveform of the second audio signal are compared for each time unit associated by the dynamic time matching module, and the degree of coincidence of both is scored and output. It is characterized by.

  In a second configuration of the music practice support device according to the present invention, in the first configuration, the section selection unit is configured such that at least one of pitch and volume is predetermined in the first and second audio signals. When the silent section below the predetermined time continues for a longer period, a section excluding the corresponding silent section is selected from the first and second audio signals.

  According to a third configuration of the music practice support device according to the present invention, in the first configuration, the music identifier that uniquely identifies the music, and the second acoustic parameter that corresponds to the music identified by the music identifier, Storage means storing one or a plurality of sets, wherein the acquisition means reads out and acquires the second acoustic parameter corresponding to the music identifier selected by the user from the storage device. .

  The first configuration of the dynamic time alignment module according to the present invention analyzes a first audio signal representing a waveform of a singing sound or performance sound by a user, and for each predetermined time unit, the pitch, volume, Extracting means for extracting a spectrum, a first acoustic parameter representing a volume and a degree of temporal change of the spectrum, and analyzing a second audio signal representing a waveform of a singing sound or a performance sound modeled by the user Is a second acoustic parameter obtained for each time unit, and represents the pitch, volume, spectrum, and degree of temporal change in volume and spectrum of the sound represented by the second audio signal in each time unit. Acquisition means for acquiring two acoustic parameters, and at least one of pitch and volume from the first audio signal exceeds a predetermined threshold. A voiced section is selected with reference to the first acoustic parameter, and a voiced section in which at least one of pitch or volume exceeds a predetermined threshold from the second audio signal is selected as the second acoustic parameter. The section selection means for selecting from the second audio signal with reference to the first acoustic parameter of each time unit in the sounded section selected by the section selection means, the average value and the standard deviation are predetermined. Normalizing means for performing the normalization to the second acoustic parameter in each time unit in the sounded section selected by the section selecting means, The normalized first acoustic parameter in a coordinate plane having the time axis of the signal as one coordinate axis and the time axis of the second audio signal as the other coordinate axis. And an evaluation value calculated from a difference between the normalized second acoustic parameter and a calculation unit that calculates the evaluation value for each lattice point having the time unit as a coordinate value, A grid point selecting means for selecting only grid points whose difference is smaller than a predetermined value and a grid point having both maximum coordinate values on the coordinate plane from the start point having the minimum grid point on both coordinate values. Among the routes to the end point, a specifying unit that specifies only a route that passes through only the lattice point selected by the lattice point selecting unit and minimizes the sum of the evaluation values at the lattice points on the route, and the specifying unit Along with the identified path, there is provided an association means for associating the time unit in the first audio signal with the time unit in the second audio signal.

  In the first configuration of the program according to the present invention, a computer device analyzes a first audio signal representing a waveform of a singing sound or performance sound by a user, and a pitch, volume, and volume in a predetermined time unit. Extracting means for extracting a spectrum, a first acoustic parameter representing a volume and a degree of temporal change of the spectrum, and analyzing a second audio signal representing a waveform of a singing sound or a performance sound modeled by the user Is a second acoustic parameter obtained for each time unit, and represents the pitch, volume, spectrum, and degree of temporal change in volume and spectrum of the sound represented by the second audio signal in each time unit. Acquisition means for acquiring two acoustic parameters, and at least one of pitch and volume is predetermined from the first audio signal A voiced section exceeding a value is selected with reference to the first acoustic parameter, and a voiced section in which at least one of pitch or volume from the second audio signal exceeds a predetermined threshold value is selected as the second sound. Section selection means for selecting from the second audio signal with reference to parameters, and the first acoustic parameter of each time unit in the sounded section selected by the section selection means, its average value and standard deviation Normalizing means for converting the second acoustic parameter in each time unit in the sounded section selected by the section selecting means, and normalizing means for converting the first acoustic parameter into a predetermined value; The normalized first sound on a coordinate plane with the time axis of the audio signal as one coordinate axis and the time axis of the second audio signal as the other coordinate axis A calculation means for calculating an evaluation value calculated from a difference between a parameter and the normalized second acoustic parameter for each lattice point having the time unit as a coordinate value, and both coordinate values on the coordinate plane; A grid point selection means for selecting only grid points whose difference is smaller than a predetermined value, and on the coordinate plane, from the start point where both coordinate values are the minimum grid point, A specifying unit that specifies only a path that passes through only the grid point selected by the grid point selection unit and has a minimum sum of the evaluation values at the grid points on the path, and the specifying unit; The time unit in the first audio signal and the time unit in the second audio signal are made to function as an association unit that associates the time unit in the first audio signal along the path specified by the above.

  According to the present invention, there is an effect that it is possible to more efficiently execute evaluations related to singing and playing using various techniques.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(A: Configuration)
FIG. 1 is a block diagram illustrating a hardware configuration of a karaoke apparatus 1 as a music practice support apparatus according to an embodiment of the present invention. As shown in FIG. 1, the karaoke apparatus 1 includes a control unit 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage unit 14, a display unit 15, an operation unit 16, an audio processing unit 18, and the like. Has a bus 10 that mediates data exchange.
The control unit 11 is, for example, a CPU (Central Processing Unit), reads out a control program stored in the ROM 12, loads it into the RAM 13, and executes it to control each unit of the karaoke apparatus 1.

  The storage unit 14 is a large-capacity storage unit such as a hard disk, and has an accompaniment / lyric data storage area 14a, an exemplary voice data storage area 14b, and a trainer voice data storage area 14c.

The display unit 15 is, for example, a liquid crystal display and a driving circuit thereof, and various types such as a menu screen for operating the karaoke apparatus 1 under the control of the control unit 11 and a karaoke screen in which lyrics telop is superimposed on a background image. Display the screen.
The operation unit 16 includes various keys such as a numeric keypad, and outputs a signal corresponding to the pressed key to the control unit 11.

  A microphone 17 and a speaker 19 are connected to the sound processing unit 18. The microphone 17 collects the singing sound of a user (hereinafter, a practitioner) who practice singing using the karaoke device 1, and outputs an audio signal (analog data) corresponding to the singing sound to the audio processing unit 18. . The audio processing unit 18 converts the audio signal (analog data) output from the microphone 17 into audio data (digital data) and outputs the audio data (digital data) to the control unit 11, while converting the audio data delivered from the control unit 11 into an audio signal. The data is converted and output to the speaker 19. The speaker 19 emits a sound corresponding to the sound signal output from the sound processing unit 18.

  In the accompaniment / lyric data storage area 14a of the storage unit 14, accompaniment data in which performance sounds (so-called guide melodies) of various musical instruments that accompany the music are recorded in the order of progress of the music, and lyrics data indicating the lyrics of the music are included. One or more music pieces are stored in association with each other. More specifically, the accompaniment data and lyrics data stored in the accompaniment / lyric data storage area 14a include identifiers for uniquely identifying karaoke music (for example, music codes consisting of English letters, symbols, numbers, etc.) , Music identifier) are associated with each other, and the accompaniment data and the lyrics data are associated with each other by this music identifier. The accompaniment data is, for example, data in the MIDI (Musical Instruments Digital Interface) format and is reproduced when the practitioner sings a karaoke song. The lyrics data is displayed on the display unit 15 as a lyrics telop at the time of the karaoke song.

  In the model voice data storage area 14b, the WAVE format that represents the voice of the singing sound (hereinafter, model voice) of the song by the singer who has the song identified by the song identifier in association with the song identifier described above. Voice data (hereinafter, model voice data) is stored. This model voice data is used as a reference when evaluating a practitioner's song.

  In the practitioner voice data storage area 14c, voice data (hereinafter, practitioner voice data) generated by A / D conversion from the microphone 17 via the voice processing unit 18 is stored, for example, in the WAVE format.

  Next, the functional configuration of the karaoke apparatus 1 will be described with reference to the block diagram shown in FIG. The basic analysis module 21, the dynamic time warping (hereinafter referred to as DTW) module 22, and the evaluation module 23 illustrated in FIG. 2 are software modules that are realized by the control unit 11 executing the control program described above. It is. The arrows in the figure schematically show the flow of data. In addition to the above three software modules, playback of accompaniment sounds according to accompaniment data of a karaoke song specified by the practitioner, and karaoke performance that combines and outputs the accompaniment sound and the practitioner's singing sound The module is also realized by the control unit 11 executing the control program. However, since the function of the karaoke performance module is not different from that of the conventional karaoke apparatus, illustration and detailed description are omitted.

The basic analysis module 21 detects acoustic parameters (in this embodiment, parameters relating to pitch, volume, and spectrum) in units of frames each having a predetermined time length for the model voice data and the practice person voice data. Hereinafter, for each parameter, a frame number is assigned from 0 in order from the frame with the earliest time (the i-th frame is referred to as the i-th frame).
In the present embodiment, the case where the time unit for extracting the acoustic parameter from each of the model voice data and the trainer voice data is set to one frame. However, the acoustic parameter is set in units of subframes obtained by further dividing one frame. May be extracted, or the acoustic parameters may be extracted from the acoustic parameters in units of a plurality of frames. In short, the time unit for extracting the acoustic parameters from the model voice data and the time unit for extracting the acoustic parameters from the trainer voice data need only be the same, and the length of the time unit does not matter. .

  Hereinafter, the basic analysis module 21 will be described in detail. As shown in FIG. 2, the basic analysis module 21 includes pitch detection means 211, volume detection means 212, spectrum detection means 213, and differentiation means 214a to 214c. The voice data delivered to the basic analysis module 21 (that is, model voice data or practice person voice data) is divided into three parts as shown in FIG. 2, and each of the pitch detection means 211, the volume detection means 212, and the spectrum detection means 213 is distributed. Delivered to.

  The pitch detection unit 211 obtains autocorrelation for the audio data for the predetermined time unit, detects the pitch in the time unit, and outputs pitch data indicating the detection result. The pitch data output from the pitch detector 211 is delivered to the DTW module 22 as shown in FIG. In the present embodiment, the case where the pitch in the time unit is detected by obtaining the autocorrelation has been described. However, for example, the pitch may be detected by obtaining the cepstra for each time unit.

  The sound volume detection means 212 calculates the average of the absolute values of the amplitudes of the samples (256 samples in the present embodiment: see FIG. 3) included in the audio data for the predetermined time unit, and calculates the calculation result. Output as volume data indicating the volume of the frame. The volume data output from the volume detection means 212 is divided into two as shown in FIG. 2, one of which is delivered to the DTW module 22 and the other is delivered to the differentiation means 214a.

  The differentiating means 214a calculates a primary differential (hereinafter referred to as “speed”) for the sound volume from the sound volume data for a plurality of continuous time units (5 in this operation example), and the sound volume speed indicating the calculation result. Output data. In the present embodiment, as shown in FIG. 3, the differentiating means 214a generates volume speed data from volume data for five consecutive frames, and this volume speed data is divided into two as shown in FIG. One of them is delivered to the DTW module 22, and the other is delivered to the differentiating means 214b.

  The differentiating means 214b calculates the first derivative (that is, the second derivative of the volume: hereinafter, the acceleration of the volume) from the volume speed data for a plurality of continuous time units (5 in this operation example), and the calculation result Volume acceleration data indicating is output. The volume acceleration data output from the differentiating means 214b is delivered to the DTW module 22.

  As shown in FIG. 3, the spectrum detection unit 213 performs a fast Fourier transform (hereinafter referred to as FFT) on two consecutive time units of audio data, and then a bandpass filter having a predetermined pass band. (In this embodiment, the voice data of the singing sound is input, so that 0 to 2 kHz is a 1/2 octave bandpass filter, and 2 to 8 kHz is a 1/4 octave bandpass filter) The output is output as spectrum data representing the spectrum in time units. The spectrum data output from the spectrum detecting means 213 is divided into two as shown in FIG. 2, one of which is delivered to the DTW module 22 and the other is delivered to the differentiating means 214c.

The differentiating means 214c calculates a first derivative for each frequency band of the spectrum from a plurality of continuous (5 in this operation example) time-series spectral data, and spectral velocity data indicating the calculation result is shown in FIG. Delivered to the DTW module 22 as shown.
The above is the configuration of the basic analysis module 21.

Next, the functional configuration of the DTW module 22 will be described. The DTW module 22 is for specifying the correspondence between the time axis of the model voice and the time axis of the trainee voice as shown in FIG. 4, and as shown in FIG. A normalizing unit 221, a difference matrix generating unit 222, and an optimum route specifying unit 223 are included.
The DTW execution section limiting means 220 is a means for limiting a section for performing dynamic time matching (DTW) processing described below in the model voice and the practice voice. The function will be described in detail below.

  The DTW execution section limiting unit 220 executes the DTW when the volume extracted from the model voice data by the volume detection unit 212 exceeds a predetermined threshold and the period exceeding the predetermined threshold exceeds a predetermined threshold. Interval. This is because the corresponding music portion of the model voice data is a voiced section that is actually sung.

Similarly, when the volume extracted from the trainer voice data by the volume detection means 212 exceeds a predetermined threshold and the period exceeding the threshold exceeds a predetermined threshold, the section is set as the DTW implementation section. . In the model voice and the trainee voice, just one time unit is added immediately before and after the voiced section (the sections are referred to as offset sections), and these are combined as the DTW implementation section.
Next, the DTW execution section limiting means 220 generates data consisting only of the DTW execution section limited by the DTW execution section limiting means 220 from the various acoustic parameters of the model voice and the practice person voice extracted by the basic analysis module 21. .

The normalizing means 221 receives the acoustic parameters of the model voice and the trainer voice extracted for each frame in the DTW execution section from the DTW execution section limiting means 220, normalizes each, and delivers them to the difference matrix generation means 222. . Here, the normalization of data means that a series of acoustic parameters delivered from the DTW execution section limiting unit 220 is converted in a frame unit so that the addition average and standard deviation become constant values. In the present embodiment, the normalization is performed according to the following formula 1.
(Equation 1) AfterDat [i] = (BeforDat [i])-AVR) / STD

In Equation 1, BeforDat [i] is an acoustic parameter for the i-th frame delivered from the DTW implementation section limiting unit 220, SDV is a standard deviation for the acoustic parameter, and AVR is an addition for the acoustic parameter. It is an average, and AfterDat [i] is a normalized acoustic parameter for the i-th frame.
By performing the normalization shown in Equation 1, the acoustic parameters delivered from the DTW execution section limiting unit 220 for each of the model voice and the practice voice have an addition average of “0” and a standard deviation of “1”. Each is converted into acoustic parameters (that is, data according to a standardized normal distribution).
By performing the normalization described above, it is possible to compare the model voice and the practitioner voice by removing factors such as a difference in the sound pickup environment. In addition, there is a difference in volume level between the model voice and the practitioner voice, and the fact that the pitch is different in octave units is almost unrelated to the skill of singing. It is also possible to remove factors such as inherent differences.

The difference matrix generation means 222 calculates a Euclidean distance (hereinafter also referred to as a difference) between acoustic parameters for each frame of the model voice and each frame of the trainee voice, and a matrix having the difference as a component (hereinafter referred to as a difference matrix). Is stored in the RAM 13. For example, when the practicing voice singing starts at the 0th frame and the singing end is the Nth frame, the singing of the model voice starts at the 0th frame and the singing end is the Mth frame (N, M Is a natural number), and the difference matrix generation means 222 uses the value represented by the following formula 2 as an (i, j) component (where 0 ≦ i ≦ N, 0 ≦ j ≦ M) (N + 1) rows (M + 1) columns The difference matrix is generated.
(Equation 2) Sqr {(Σ (GuideSpectrum [j] [k] −SingerSpectrum [i] [k]) ^ 2) * WeightScalar [k]
+ (Σ (ΔGuideSpectrum [j] [k] −ΔSingerSpectrum [i] [k]) ^ 2) * WeightVector [k]
+ (ΔGuidePower [j] −ΔSingerPower [i]) ^ 2)
+ (ΔΔGuidePower [j] −ΔΔSingerPower [i]) ^ 2)
} / num

In Equation 2,
GuideSpectrum [j] [k]: Spectral component of the kth passband of the jth frame of the model voice
SingerSpectrum [i] [k]: Spectral component of the k-th passband of the i-th frame of the trainee voice ΔGuideSpectrum [j] [k]: k-th spectral velocity of the j-th frame of the model voice ΔSingerSpectrum [i] [k]: k-th spectral speed of the i-th frame of the trainer voice ΔGuidePower [j]: volume speed of the j-th frame of the model voice ΔSingerPower [i]: volume speed of the i-th frame of the trainer voice ΔΔGuidePower [j]: Volume acceleration of the j-th frame of the model voice ΔΔSingerPower [i]: Volume acceleration of the i-th frame of the trainer voice
WeightScalar [k]: Weighting coefficient
WeightVector [k]: Weighting factor
num: The number of parameters for obtaining the Euclidean distance (in this embodiment, (N + 1) × (M + 1)).

However, WeightScalar [k] is a coefficient that weights the acoustic parameters that do not depend on time change, and the trainer's singing sound and the model voice are sound (periodic sound) or silence (non-periodic) It is a value that is appropriately selected depending on whether it is a voice. Specifically, when both the practitioner's singing sound and the model voice are voiced, a value is selected so that a weight is given to the low-frequency spectrum, and both the practitioner's singing sound and the model voice are silent. In the case of, a value is selected so that a weight is given to a high-frequency spectrum. Note that whether the trainer singing sound and the model voice are voiced or silent is determined based on each pitch and volume. Specifically, the difference matrix generation unit 222 determines that the corresponding time unit is sound when the pitch is equal to or greater than a predetermined threshold and the volume is equal to or greater than the predetermined threshold. Is determined to be silent.
On the other hand, WeightVector [k] is a coefficient for weighting the acoustic parameter depending on time change, and is a coefficient for assigning a weight to the mid-range spectrum.
In Equation 2, the Σ symbol means the sum of the subscript k, “^ 2” means the square, and Sqr {} means the square root.

  The optimum route specifying unit 223 determines the frame number of the model voice frame and the trainer voice frame corresponding to each component from the (N + 1) × (M + 1) difference matrix components generated by the difference matrix generation unit 222. Select a component that does not exceed the specified value. For example, the difference matrix shown in FIG. 13 shows the case where the specified value is “2”. In the difference matrix, only the difference matrix component satisfying the above condition is shown.

  Next, the optimum route specifying unit 223 will be described. The optimum route specifying unit 223 specifies the optimum route for the difference matrix (such as the difference matrix shown in FIG. 13) subjected to the difference matrix limiting process. Hereinafter, description will be made using a difference matrix that is not subjected to component limitation processing as shown in FIG. In the difference matrix shown in FIG. 6, among the paths from the lower left corner (that is, (0, 0) component) to the upper right corner (that is, (N, M) = (3,4) component), The path that minimizes the accumulation of the value of each component located in is identified as the path that represents the correspondence between each time unit of the trainer voice and the model voice, and data representing the correspondence between the times indicated by the path is evaluated. Delivered to 23. More specifically, the optimum route specifying unit 223 specifies the optimum route according to the rules described below.

(Rule 1) The route search is started from the lower left corner of the difference matrix, and the process of selecting the movement destination is repeated until reaching the upper right corner so that the accumulated value of the movement destination component values is minimized. However, one movement is limited to one of right, top, and top right. For example, movement from the (i, j) component is limited to movement to the (i, j + 1) component, (i + 1, j) component, or (i + 1, j + 1) component. If the cumulative value when moving to the right is equal to the cumulative value when moving upward, priority is given to the movement to the right. Similarly, when the accumulated value of the movement to the right and the movement to the upper right is equal, priority is given to the movement to the right, and when the accumulated value of the movement to the upper and the movement to the upper right is equal, Give priority to movement.
(Rule 2) The route selected in accordance with Rule 1 is traced backward from the upper right corner to the lower left corner to identify the optimum route.

The evaluation module 23 in FIG. 2 compares the signal waveforms of the model voice data and the trainer voice data that have been associated with the time axis by the DTW module 22, and determines the degree of match between the model voice and the practice person voice. The score is displayed on the display unit 15. The evaluation module 23 matches the time axis of the model voice according to the result of the dynamic time matching performed by the DTW module 22 when comparing the waveform of the trainee voice and the waveform of the model voice. After expanding and contracting the time axis of the practitioner's voice, the waveforms of both are compared.
The above is the configuration of the karaoke apparatus 1. As described above, in the present embodiment, the case where the basic analysis module 21 and the DTW module 22 responsible for the characteristic functions of the musical tone practice support device according to the present invention are implemented by software modules has been described. Of course, the module may be realized by a hardware module.

(B: Operation)
Next, the scoring process performed by the karaoke apparatus 1 will be described with reference to the drawings, centering on the operations remarkably showing the characteristics (that is, the operations of the basic analysis module 21 and the DTW module 22). In the operation example described below, it is assumed that the power source (illustrated) of the karaoke apparatus 1 has been turned on, and the control unit 11 operates according to a control program loaded from the ROM 12 to the RAM 13.

  A practitioner who wants to practice singing using the karaoke apparatus 1 inputs a song identifier of a song to be practiced by appropriately operating the operation unit 16 while referring to a menu screen or the like displayed on the display unit 15. You can specify the music to be practiced. When the music to be practiced is designated in this way, the control unit 11 loads accompaniment data and lyrics data corresponding to the music identifier from the storage unit 14 to the RAM 13. Then, when the practitioner performs an operation to instruct the start of performance on the operation unit 16, the control unit 11 causes the audio processing unit 18 to reproduce the accompaniment sound according to the accompaniment data read to the RAM 13. At the same time, a karaoke screen in which the lyrics telop represented by the lyrics data is embedded is displayed on the display unit 15, and the lyrics are wiped as the music progresses.

  The practitioner visually recognizes the karaoke screen and sings the music in accordance with the accompaniment sound emitted from the speaker. The singing sound of the practitioner is picked up by the microphone 17, and the practitioner voice data corresponding to the singing sound is sequentially written in the practitioner voice data storage area 14c. When the trainer voice data is stored in the trainer voice data storage area 14c in this way, the control unit 11 stores the trainer voice data in the model voice data storage area 14b in association with the music identifier. The exemplary voice data is read out, and the scoring process shown in FIG. 5 is executed.

  FIG. 5 is a flowchart showing the flow of scoring processing performed by the control unit 11 according to the control program. As shown in FIG. 5, the control unit 11 analyzes the model voice data and the trainer voice data, and sets the acoustic parameters for each predetermined time unit (frame in the present embodiment) from the beginning to the end of the music. Extract (step SA100). The process of step SA100 is executed by the basic analysis module 21 described above.

  Next, the control unit 11 performs dynamic time matching (DTW) processing on the part where the singing is performed in the model voice data extracted in step SA100, and on the part where the singing is actually performed in the trainer voice data. Is specified as the music section to be performed (step SA110). Next, data including only the DTW implementation period specified as described above is generated from the data regarding various acoustic parameters extracted in step SA100, and the data is transferred to the normalizing means 221. The process of step SA110 is executed by the DTW execution section limiting means 220 of the DTW module 22 described above.

In the karaoke apparatus 1 according to the present embodiment, it is naturally not necessary to evaluate the singing for an interlude portion where the singing is not performed. Therefore, by generating data corresponding to the part where the singing is performed as described above, it is possible to improve the efficiency of the processing executed thereafter.
Next, the control unit 11 normalizes the acoustic parameters for each parameter type (step SA120), and generates a difference matrix from the normalized acoustic parameters (step SA130). The process of step SA120 is executed by the normalizing unit 221 of the DTW module 22 described above, and the process of step SA130 is executed by the difference matrix generating unit 222 of the DTW module 22. In this operation example, it is assumed that the difference matrix shown in FIG. 6 is generated as a result of executing the processing up to step SA130.

  Here, the optimum route specifying means 223 determines that a difference between the frame number of the model voice corresponding to the difference matrix component and the frame number of the trainee voice from the difference matrix component generated as described above is a predetermined specified value. The following difference matrix component is selected. For example, in the difference matrix shown in FIG. 13, the specified value is “2”, and the difference matrix component that satisfies the above condition is limited (step SA140).

  Next, the control unit 11 specifies the optimum route from the components of the difference matrix limited in step SA140 (step SA150). The process of step SA150 is a process executed by the optimum route specifying unit 223 described above. Specifically, the optimum route specifying unit 223 specifies an optimum route in the procedure described below.

  First, the optimum route specifying unit 223 accumulates component values accompanying the movement of the route along the first column of the difference matrix (see FIG. 7). For example, the value of the (0, 0) component that is the starting point of the route along the first column is “1”, and the value of the (1, 0) component is “4” (see FIG. 6). The accumulated value accompanying the movement from the (0, 0) component to the (1, 0) component is “5” (see FIG. 7). Since the value of the (2, 0) component is “1”, the cumulative value accompanying the movement of (0, 0) component → (1, 0) component → (2, 0) component is “6”. (See FIG. 7). Thereafter, the component values accompanying the movement are accumulated until reaching the (3, 0) component, and the result shown in FIG. 7 is obtained.

  Next, as in the case of the first column described above, the optimum route specifying unit 223 accumulates component values associated with movement in the second column (see FIG. 8). Similarly, the accumulation of component values accompanying the movement is repeated until reaching the upper right corner of the difference matrix (that is, the (3, 4) component) (see FIG. 9).

As shown in FIG. 9, when the accumulation of the component values accompanying the movement to the upper right corner of the difference matrix is completed, the optimum route specifying means 223 can move toward the starting point using the upper right corner as the starting point. Among the lattice points (that is, the left, lower left, or lower lattice point of the starting point), the lattice point with the minimum accumulation of component values along the route to the lattice point is specified as the route candidate. . Then, the optimum route specifying unit 223 repeats the process of specifying the next route candidate using the specified route candidate as the starting point until the route candidate matches the lattice point in the lower left corner. As a result, the optimum matrix candidate shown in FIG. 10 (that is, (3,4) → (2,3) → (1,2) → (1,1) → (0,0) is obtained for the difference matrix shown in FIG. )) Is identified.
Next, the optimum route specifying means 223 reversely traces the optimum route candidate specified as described above, and confirms that the cumulative value increases when moving away from the optimum route candidate. A route is specified (see FIG. 11).

  The optimum route identified as described above represents a correspondence relationship between the time axis of the model voice and the time axis of the trainee voice. Specifically, the optimum route shown in FIG. 11 indicates that each frame for the model voice corresponds to each frame for the trainee voice as shown in FIG. The optimum route specifying means 223 generates data indicating the correspondence shown in FIG. 12, and outputs the data to the evaluation module 23 (step SA160).

  Thereafter, the evaluation module 23 performs time alignment on the trainer voice data so as to satisfy the correspondence specified by the optimum route specifying means 223, and then compares the trained voice data with the model voice data. To display.

  As described above, according to the karaoke apparatus 1 according to the present embodiment, in the process of evaluating the singing by the practitioner, the DTW execution section is limited to the above-described voiced section (in other words, the silent section is excluded). To do). By limiting the DTW implementation interval and by limiting the components in the difference matrix, there is an effect that the amount of calculation required can be reduced.

(C: deformation)
Although one embodiment of the present invention has been described above, it is needless to say that the embodiment may be modified as described below.
(1) In the embodiment described above, when a singing is performed using various techniques by incorporating a function for performing dynamic time alignment (DTW) processing characteristic of the present invention into a karaoke apparatus, We explained the case where it is possible to evaluate the difference from the technique used in the example song. However, the processing target of the dynamic time alignment processing by the DTW module 22 is not limited to the singing sound, and the performance sound data of a musical instrument played using various techniques and a model example thereof. It may be performance data, and can also be used to learn foreign languages such as English conversation.

(2) In the above-described embodiment, it is determined whether the voice is voiced or silent based on the pitch and volume of the trainer voice and the model voice, and an acoustic parameter that does not depend on time change according to the determination result (above In the embodiment, the case of switching the weight to be given to the spectrum) has been described, but it is of course possible to determine whether sound is present or not based on only the pitch or only the volume. In addition, since the weight switching as described above is not always essential, it is not necessary to detect the pitch or deliver the volume data from the basic analysis module 21 to the DTW module 22 in an aspect in which such switching is not performed. Not too long.

(3) In the above-described embodiment, each time the dynamic time matching between the practitioner singing sound and the model voice is performed, the model voice data stored in the model voice data storage area 14b is used as the basic analysis module 21. The case where the acoustic parameters for the singing sound represented by the exemplary voice data are calculated has been described. However, it is of course possible to obtain the acoustic parameters for the model voice data in advance and store the acoustic parameters in association with the music identifiers in the storage unit 14.
Further, in the above-described embodiment, the case where the model audio data is stored in the storage unit 14 provided in the karaoke apparatus 1 has been described, but a CD-ROM (Compact Disk-Read Only Memory) or a DVD (Digital Versatile Disk) is described. Model audio data and acoustic parameters extracted from the exemplary audio data are written and distributed on a computer-readable recording medium such as), and the exemplary audio data is read by reading the exemplary audio data and acoustic parameters from such a recording medium. And acoustic parameters may be acquired, or acoustic parameters for model voices may be acquired via a telecommunication line such as the Internet.

(4) In the above-described embodiment, the basic analysis module 21 that extracts the acoustic parameters from the trainer voice data and the model voice data, and the time axis of the model voice and the trainer voice are associated with each other based on the acoustic parameters. Although the case where the DTW module 22 to be performed is realized as a separate software module has been described, it is needless to say that it may be configured as one software module. Specifically, the function to extract acoustic parameters from the trainer's voice data and the model voice data to the dynamic time matching module that performs normalization of the acoustic parameters and dynamic time matching using the normalized acoustic parameters It is sufficient to have a function of acquiring acoustic parameters for the model voice by extracting the data or reading from the recording medium or the like.

(5) In the above-described embodiment, a case has been described in which a control program for causing the control unit 11 to realize a characteristic function of the music practice support device according to the present invention is written in the ROM 12 in advance. The control program may be recorded and distributed on a computer-readable recording medium such as a DVD, or the control program may be distributed by downloading via a telecommunication line such as the Internet.

(6) In the above-described embodiment, a case has been described in which pitch, volume, and spectrum, and primary and secondary derivatives of volume and primary derivative of spectrum are used as acoustic parameters for performing dynamic time matching. Of these acoustic parameters, the first and second derivatives of the volume represent the degree of temporal change in volume, but the second derivative is not necessarily essential. Of course, the spectrum may be obtained up to the second derivative in order to accurately reflect the degree of time change by dynamic time matching.

(7) In the above-described embodiment, the difference matrix generation unit 222 calculates values for all the components of the difference matrix (step SA130), and then the optimum route specifying unit 223 determines a component that is a route candidate from those components. The case of limiting (step SA140) has been described. However, the difference matrix generating unit 222 may not generate the components excluded in step SA140 in advance in step SA130.
The reason is as follows. Since the singer sings while watching the lyrics telop displayed on the display unit 15, the singer sings the music part for which the lyrics telop is not yet displayed, or sings the music part for which the lyrics telop has been displayed. The possibility of singing extremely different from the model voice is low. Such extreme singing corresponds to a case in which the frame number is extremely different between the model voice and the trainee voice. Therefore, in the difference matrix, for example, a component made up of differences calculated for frames whose numbers are extremely different between the model voice and the practitioner voice, such as the (N, 0) component, is low.

(8) In the above-described embodiment, the DTW execution section limiting unit 220 determines a section in which the volume of the model voice or the practice person voice exceeds a predetermined threshold and the period exceeding the predetermined threshold exceeds the predetermined threshold. The case where it was set as the implementation section was explained. However, the DTW execution period may be determined based on the pitch of the model voice in addition to the volume or instead of the volume. Specifically, (a) a section in which the pitch of the model voice exceeds a predetermined threshold and the period exceeding the threshold exceeds a predetermined threshold; (b) both the volume and the pitch of the model voice are predetermined. (C) a section in which the volume of the model voice exceeds a predetermined threshold (the time of the section is not considered), (d) the model voice A section in which the pitch exceeds a predetermined threshold (the section time is not considered) may be set as the DTW implementation section.

(9) In the above-described embodiment, the model voice data representing the singing sound of the karaoke song by the singer who has the karaoke song identified by the song identifier in association with the song identifier that uniquely identifies the karaoke song is used. The case where it memorize | stores in the memory | storage part 14 was demonstrated.
However, if a plurality of singers individually have a song as a song, different song identifiers may be given as different songs for each singer, and each song is uniquely identified. A singer identifier that uniquely identifies each of the plurality of singers is associated with the song identifier to be identified, and further, the set of the song identifier and the singer identifier is identified by the singer identifier of the song identified by the song identifier The model voice data representing the singing sound by the singer may be stored in the storage unit 14 in association with each other. As mentioned above, the singing technique is generally different for each singer, and even if the singer is different even if it is the same song, the feeling and taste that can be put into the singing is generally different. . If it is configured so that singers can be identified as described above, even if a plurality of singers have a single piece of music as a song, the user can respond to his / her preference from among those singers. It becomes possible to select a song by a singer and practice singing by imitating the song.

It is a block diagram which shows an example of the hardware constitutions of the karaoke apparatus 1 which concerns on one Embodiment of this invention. It is a block diagram which shows the function structural example of the karaoke apparatus. It is a figure for demonstrating the acoustic parameter extraction performed by the basic analysis module 21. FIG. It is a figure which shows an example of the execution result of the dynamic time alignment process performed by the same DTW module. It is a flowchart which shows the flow of the scoring process which the same karaoke apparatus 1 performs. It is a figure which shows an example of a difference matrix. It is a figure which shows an example of the difference matrix in the optimal path | route specific process. It is a figure which shows an example of the difference matrix in the optimal path | route specific process. It is a figure which shows an example of the difference matrix in the optimal path | route specific process. It is a figure which shows an example of the difference matrix in the optimal path | route specific process. It is a figure which shows an example of the optimal path | route identified by the optimal path | route identification process. It is a figure for demonstrating the processing result of a dynamic time alignment process. It is a figure which shows an example of the difference matrix by which the component was limited.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Karaoke apparatus, 10 ... Bus, 11 ... Control part, 12 ... ROM, 13 ... RAM, 14 ... Memory | storage part (14a; Accompaniment / lyrics data storage area, 14b; Model voice data storage area, 14c; Trainer voice data (Storage area), 15 ... display unit, 16 ... operation unit, 17 ... microphone, 18 ... audio processing unit, 19 ... speaker, 21 ... basic analysis module, 22 ... DTW module, 23 ... evaluation module, 211 ... pitch detection means, 212 ... Volume detection means, 213 ... Spectrum detection means, 214, 214a, 214b, 214c ... Differentiation means, 220 ... DTW implementation section limiting means, 221 ... Normalization means, 222 ... Difference matrix generation means, 223 ... Optimum path specifying means

Claims (5)

The first audio signal representing the waveform of the singing sound or performance sound by the user is analyzed, and the first, representing the pitch, volume, spectrum, and degree of time change of the volume and spectrum for each predetermined time unit. Extraction means for extracting acoustic parameters;
A second acoustic parameter obtained for each time unit by analyzing a second audio signal that represents a waveform of a singing sound or performance sound that is modeled by the user, and represents the second audio signal Acquisition means for acquiring a second acoustic parameter representing a pitch, volume, spectrum, and degree of temporal change of volume and spectrum in each time unit of sound;
A voiced section in which at least one of pitch or volume exceeds a predetermined threshold from the first audio signal is selected with reference to the first acoustic parameter, and at least pitch or volume from the second audio signal is selected. Section selection means for selecting a voiced section in which one exceeds a predetermined threshold from the second audio signal with reference to the second acoustic parameter;
The first acoustic parameter of each time unit in the sounded section selected by the section selecting unit is subjected to normalization to convert the average value and standard deviation into a predetermined value, and selected by the section selecting unit. Normalization means for performing the normalization on the second acoustic parameter of each time unit in a sounded section,
The normalized first acoustic parameter and the normalized are in a coordinate plane with the time axis of the first audio signal as one coordinate axis and the time axis of the second audio signal as the other coordinate axis. Calculating means for calculating an evaluation value calculated from a difference from the second acoustic parameter for each lattice point having the time unit as a coordinate value;
On the coordinate plane, grid point selection means for selecting only grid points where the difference between the coordinate values is smaller than a predetermined value;
On the coordinate plane, only the grid point selected by the grid point selection means passes through the path from the start point, which is the grid point having the smallest coordinate value, to the end point, the grid point having the largest coordinate value. Identifying means for identifying a path that minimizes the sum of the evaluation values at the grid points on the path;
A dynamic time alignment module comprising: an association means for associating the time unit in the first audio signal with the time unit in the second audio signal along the path specified by the specifying means;
The signal waveform of the first audio signal and the signal waveform of the second audio signal are compared for each time unit associated by the dynamic time matching module, and the degree of coincidence between the two is scored. A music practice support device characterized by output. In the first and second audio signals, the section selection unit is configured to perform a corresponding silence when a silent section in which at least one of pitch and volume falls below a predetermined threshold value continues for a predetermined time. The music practice support device according to claim 1, wherein a section excluding a section is selected from the first and second audio signals. A storage unit that stores one or a plurality of sets of a song identifier that uniquely identifies a song and the second acoustic parameter corresponding to the song identified by the song identifier;
The music practice support device according to claim 1, wherein the acquisition unit reads and acquires the second acoustic parameter corresponding to the music identifier selected by the user from the storage device. The first audio signal representing the waveform of the singing sound or performance sound by the user is analyzed, and the first, representing the pitch, volume, spectrum, and degree of time change of the volume and spectrum for each predetermined time unit. Extraction means for extracting acoustic parameters;
A second acoustic parameter obtained for each time unit by analyzing a second audio signal that represents a waveform of a singing sound or performance sound that is modeled by the user, and represents the second audio signal Acquisition means for acquiring a second acoustic parameter representing a pitch, volume, spectrum, and degree of temporal change of volume and spectrum in each time unit of sound;
A voiced section in which at least one of pitch or volume exceeds a predetermined threshold from the first audio signal is selected with reference to the first acoustic parameter, and at least pitch or volume from the second audio signal is selected. Section selection means for selecting a voiced section in which one exceeds a predetermined threshold from the second audio signal with reference to the second acoustic parameter;
The first acoustic parameter of each time unit in the sounded section selected by the section selecting unit is subjected to normalization to convert the average value and standard deviation into a predetermined value, and selected by the section selecting unit. Normalization means for performing the normalization on the second acoustic parameter of each time unit in a sounded section,
The normalized first acoustic parameter and the normalized are in a coordinate plane with the time axis of the first audio signal as one coordinate axis and the time axis of the second audio signal as the other coordinate axis. Calculating means for calculating an evaluation value calculated from a difference from the second acoustic parameter for each lattice point having the time unit as a coordinate value;
On the coordinate plane, grid point selection means for selecting only grid points where the difference between the coordinate values is smaller than a predetermined value;
On the coordinate plane, only the grid point selected by the grid point selection means passes through the path from the start point, which is the grid point having the smallest coordinate value, to the end point, the grid point having the largest coordinate value. Identifying means for identifying a path that minimizes the sum of the evaluation values at the grid points on the path;
Dynamic time alignment, comprising: association means for associating the time unit in the first audio signal with the time unit in the second audio signal along the path specified by the specifying means module. Computer equipment,
The first audio signal representing the waveform of the singing sound or performance sound by the user is analyzed, and the first, representing the pitch, volume, spectrum, and degree of time change of the volume and spectrum for each predetermined time unit. Extraction means for extracting acoustic parameters;
A second acoustic parameter obtained for each time unit by analyzing a second audio signal that represents a waveform of a singing sound or performance sound that is modeled by the user, and represents the second audio signal Acquisition means for acquiring a second acoustic parameter representing a pitch, volume, spectrum, and degree of temporal change of volume and spectrum in each time unit of sound;
A voiced section in which at least one of pitch or volume exceeds a predetermined threshold from the first audio signal is selected with reference to the first acoustic parameter, and at least pitch or volume from the second audio signal is selected. Section selection means for selecting a voiced section in which one exceeds a predetermined threshold from the second audio signal with reference to the second acoustic parameter;
The first acoustic parameter of each time unit in the sounded section selected by the section selecting unit is subjected to normalization to convert the average value and standard deviation into a predetermined value, and selected by the section selecting unit. Normalization means for performing the normalization on the second acoustic parameter of each time unit in a sounded section,
The normalized first acoustic parameter and the normalized are in a coordinate plane with the time axis of the first audio signal as one coordinate axis and the time axis of the second audio signal as the other coordinate axis. Calculating means for calculating an evaluation value calculated from a difference from the second acoustic parameter for each lattice point having the time unit as a coordinate value;
On the coordinate plane, grid point selection means for selecting only grid points where the difference between the coordinate values is smaller than a predetermined value;
On the coordinate plane, only the grid point selected by the grid point selection means passes through the path from the start point, which is the grid point having the smallest coordinate value, to the end point, the grid point having the largest coordinate value. Identifying means for identifying a path that minimizes the sum of the evaluation values at the grid points on the path;
A program that causes the time unit in the first audio signal and the time unit in the second audio signal to correspond to each other along the path specified by the specifying unit.

Images