JP2007004153A - Device, method, and program for processing sound signal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for processing sound signals capable of making a decision on an expression of a performance which is highly consistent and guaranteed to be the best. <P>SOLUTION: A plurality of expression modes of a song are modeled as respective states, the probability that a section including a frame or a plurality of continuous frames lies in a specific state is calculated with respect to a prescribed observed section based on the characteristic parameters obtained by dividing the sound signals at intervals of 25 ms, and the optimum route of state transition in the prescribed observed section is determined based on the calculated probabilities to decide expression modes of the sound signals and sections thereof. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an audio signal processing device, an audio signal processing method, and an audio signal processing program for detecting an expression mode of performance or singing input as an audio signal.

Various music evaluation devices have been proposed in which musical sound data such as singing information and performance data is compared with a reference melody to determine the beginning, duration, and end of a sound (for example, Patent Document 1).
JP-A-3-242700

  However, in the above-described conventional apparatus, a model is not assumed, and the musical sound on the spot shows such a feature, so it is determined that it is such an expression. Judgment may be made. In addition, there is a problem that there is no guarantee that the determination result is the best according to some criteria.

  According to the present invention, an audio signal processing apparatus, an audio signal processing method, and an audio signal processing program capable of performing determination with higher consistency and guaranteeing the best by adopting a technique often used in voice recognition. The purpose is to provide.

  According to the first aspect of the present invention, an audio signal input unit for inputting an audio signal for playing or singing a musical sound, and the input audio signal are divided into frames for every predetermined time, and the characteristic parameters of the audio signal are set for each frame. A feature parameter detection unit to detect, a buffer for storing the detected feature parameter, and a plurality of expression modes of performance or singing are each modeled as one state, and one frame or a plurality of consecutive ones are based on the feature parameter of each frame. By calculating the probability that the frame is a section of a specific state over a predetermined observation section, and determining the optimum path of state transition in the predetermined observation section based on this probability, the expression mode of the speech signal and its section are determined. An expression determination unit that performs expression determination processing for determination is provided.

  According to a second aspect of the present invention, the expression determination unit further determines, as the expression determination process, a section in which the expression form is determined based on the feature parameter, and determines the detailed contents of the expression form. It is characterized by.

  According to a third aspect of the present invention, the expression determination unit performs the expression determination process for a partial section of the music in the middle of the music, and the buffer has a storage capacity for the partial section of the music. It is characterized by that.

  The invention of claim 4 is characterized in that the expression determination unit detects a transition point of the musical sound in the input audio signal and executes the expression determination process at the transition point.

  The invention according to claim 5 is characterized in that the expression determination unit determines the optimum route using a Viterbi algorithm.

  According to the sixth aspect of the present invention, an audio signal input procedure for inputting an audio signal for playing or singing a musical sound, and the input audio signal is divided into frames at predetermined intervals, and the characteristic parameters of the audio signal are set for each frame. Probability that a characteristic parameter detection procedure to be detected and a plurality of expression forms of performance or singing are each modeled as one state, and one frame or a plurality of consecutive frames are sections of a specific state based on the feature parameter of each frame Is calculated over a predetermined observation interval, and the voice that performs the expression mode of the audio signal and the expression determination procedure for determining the interval by determining the optimum path of state transition in the predetermined observation interval based on this probability This is a signal processing method.

  According to the seventh aspect of the present invention, a sound signal input procedure for inputting a sound signal for playing or singing a musical sound to the signal processing device, and the input sound signal is divided into frames for every predetermined time, and the sound signal is divided for each frame. A feature parameter detection procedure for detecting a feature parameter of a signal and a plurality of expression modes of performance or singing are each modeled as one state, and one frame or a plurality of consecutive frames are specified based on the feature parameter of each frame. The expression determination procedure for determining the voice signal expression mode and the section by calculating the probability of the section of the voice signal over a predetermined observation section and determining the optimum path of state transition in the predetermined observation section based on the probability And an audio signal processing program for executing

  In the invention of the present application, an expression transition model is first generated from melody information serving as a reference. Features such as pitch, volume, and spectral change are sequentially obtained from the input musical sound. From these values, the probability that the expression changes according to a predetermined rule and the probability that the expression stays without changing are calculated. Based on this probability, the expression transition route with the highest probability is selected in the expression transition model. Then, the position of the transition of the expression is determined, and the characteristic tendency in each expression is labeled. In particular, stream processing is realized by adopting a data structure that discards the probability calculation results in the determined section one after another.

  Thereby, according to this invention, it becomes possible to perform the state determination with the highest probability in each section while observing the previous and subsequent states.

<< Summary of Embodiment >>
A karaoke apparatus to which an audio signal processing apparatus according to an embodiment of the present invention is applied will be described with reference to the drawings. In this karaoke apparatus, a singing voice signal (musical sound signal) for singing a karaoke song is input, and an expression mode (expression type) such as vibrato or shackle is detected from this musical sound signal.

  In this embodiment, the following procedure is used to accurately detect a plurality of expression modes. A plurality of expression modes are each modeled as one state, and an expression transition model that is a hidden Markov model (HMM) is configured. FIG. 1 is a diagram showing this expression transition model. As shown in the figure, when the note F shown as the reference melody data is set as one observation section, seven expression modes (states) corresponding to the observation section are detected (i., E, silence, attack, sustain, vibrato, release, silence, transition). However, in the same figure, since the expression mode is detected continuously for each note, twelve expression modes (states) are detected. That is, silence and transition between the notes F and E are represented in common by the notes F and E.

  FIG. 2 is a diagram illustrating an example of state transition in an expression transition model in which each state is arranged in the vertical direction and frames are arranged in time series in the horizontal direction. In this figure, the persistence probability of each state (each expression mode) is expressed as a cost probability Pcost (n), and the path with the largest product (logarithmic sum) of the cost probabilities of each state is determined.

  The cost probability is obtained by applying an expression mode determination rule to a feature parameter included in each frame from a specific frame to a subsequent specific frame. The cost probability is calculated for all combinations of frames from the start frame to the end frame in each state, and the route with the largest product may be selected from the cost probability group.

  However, in this embodiment, since the optimum route is searched using the Viterbi algorithm, it is not necessary to obtain cost probabilities for all frame combinations from the start to the end frame.

≪Configuration of karaoke device to which the present invention is applied≫
FIG. 3 is a block diagram schematically showing the configuration of the karaoke apparatus 1. The karaoke apparatus 1 is connected to a microphone 2 for collecting the singer's singing voice and a speaker 3 for emitting karaoke performance music.

  The karaoke apparatus 1 includes an automatic performance unit 11 for reproducing karaoke music, an AD (Analog / Digital) converter 12 for digitizing a singing voice signal input from the microphone 2, and a digitized singing voice signal. An FFT processing unit 13, a feature parameter acquisition unit 14, a feature parameter buffer 15, a rule storage unit 16, a reference buffer 17, and an expression determination unit 18, which are various functional units for detecting various expression modes from (singing voice data). I have.

  The automatic performance unit 11 includes a storage unit that stores karaoke song data, a sequencer that plays the karaoke song data, a sound source, and the like, and also includes an operation unit that receives user operations. The automatic performance unit 11 synthesizes the singer's singing voice signal input from the microphone 2 via the AD converter 12 and the automatically performed karaoke performance sound and inputs the synthesized sound to the speaker 3.

  The AD converter 12 converts the analog singing voice signal input from the microphone 2 connected to the connection terminal 12 a into digital data, and inputs the digital data to the FFT processing unit 13 and the feature parameter acquisition unit 14. The FFT processing unit 13 performs fast Fourier transform (FFT) by dividing the singing voice data that is the input sampling data string every 25 msec. At the time of fast Fourier transform, a sampling function is multiplied by a window function in order to suppress an error spectrum due to a finite time window. The frequency spectrum obtained by the FFT is input from the FFT processing unit 13 to the feature parameter acquisition unit 14.

  The feature parameter acquisition unit 14 is realized by a CPU or the like, for example. Singing voice data, which is a signal waveform in the time domain, is directly input from the AD converter 12 to the feature parameter acquisition unit 14, and a frequency spectrum as information in the frequency domain is input from the FFT processing unit 13. The feature parameter acquisition unit 14 acquires a plurality of feature parameters representing various features of the singing voice from these singing voice data and its frequency spectrum. This feature parameter is acquired every 25 ms frame.

  FIG. 4 is a block diagram showing the configuration of the feature parameter acquisition unit 14 shown in FIG. 3 in more detail. The feature parameter acquisition unit 14 calculates the frequency domain feature parameters from the time domain information acquisition unit 141 that calculates the time domain feature parameters from the singing voice data input from the AD converter 12 and the frequency spectrum input from the FFT processing unit 13. A frequency domain information acquisition unit 142 is provided.

  The time domain information acquisition unit 141 divides the input singing voice data into 25 msec-interval frames synchronized with the FFT processing unit 13 and acquires time domain feature parameters for each frame. The characteristic parameters acquired by the time domain information acquisition unit 141 are as follows.

Zero crossing timing: Zero crossing
Energy: Energy
Energy change: Delta energy
Duration (duration): Duration
Pitch interval: Pitch interval
Pitch slope
Pitch range: Pitch range
Pitch stability
Etc. The time domain information acquisition unit 141 acquires the average and deviation of the parameters as necessary. The English notation indicates the notation in FIG. 4 for each characteristic parameter.

  The frequency domain information acquisition unit 142 acquires frequency domain feature parameters from the frequency spectrum of the 25 ms waveform input from the FFT processing unit 13. The characteristic parameters acquired by the frequency domain information acquisition unit 142 are as follows.

Low frequency energy: LF energy
High frequency energy: HF energy
Filter bank (40 elements): Filter bank
Cepstrum (24 elements): Cepstrum
Spectral smoothness: Spectral flatness
Filter bank change rate: Delta filter bank
Cepstrum change: Delta cepstrum
Tone change: Delta timbre
Pitch: Pitch
Pitch change degree: Delta pitch
Vibrato depth
Vibrato rate: Vibrato rate
Overtone frequency: Harmonic frequency
Overtone level: Harmonic amplitude
Overtone phase: Harmonic phase
Harmonic stability
Pureness: Sinusoidality
Etc. Note that the English notation represents the notation of each characteristic parameter in FIG.

  The pitch is acquired from the fundamental frequency of the audio signal, and the energy is acquired from the instantaneous value of the volume of the audio signal. For vibrato, the energy and pitch time fluctuations are approximated by a sine function, the frequency of the approximated sine wave is obtained as the vibrato rate, and the maximum amplitude of the approximated sine wave is obtained as the vibrato depth. Acquired as (Vibrato depth).

  The timbre change degree is a value indicating a change amount between frames regarding a value (cepstrum) obtained by performing inverse Fourier transform on the logarithm of the amplitude spectrum, and is a parameter that well represents a change in the frequency spectrum. By using this timbre change degree as a characteristic parameter for determination to be described later, it is possible to better detect a sound change due to a state transition. In particular, a change in sound from a “vowel” to a “vowel”, which is difficult to detect with other feature parameters, can be well detected by the feature parameter of the timbre change degree.

  The feature parameters acquired by the time domain information acquisition unit 141 and the frequency domain information acquisition unit 142 are input to the feature parameter buffer 15.

  The feature parameter buffer 15 stores time information added to the inputted feature parameters. This time information is information representing the position on the time axis of the frame which is the original data of the feature parameter. The feature parameter buffer 15 stores only the latest feature parameters for several seconds, and discards old feature parameters. The storage time may be about the time required to execute a feature determination process repeatedly executed by the expression determination unit 18 described later. As a result, the feature parameter buffer 15 does not need to store the feature parameters for all the songs, and the memory capacity can be effectively reduced.

  The rule storage unit 16 is a storage unit that stores various rules used in the expression determination process performed by the expression determination unit 18. What rules are stored will be described in the description of the expression determination unit 18.

  Reference melody data synchronized with the performance of the karaoke song (music data) is input from the automatic performance unit 11 to the reference buffer 17. The reference melody data may be guide melody data for guiding a song. The reference buffer 17 stores the reference melody data for an amount of time required to execute the feature determination process that the expression determination unit 18 repeatedly executes, and discards old data.

  The expression determination unit 18 configures an HMM that models the expression mode for each sound of the reference melody data, and determines how the expression mode changes in each sound. Each time the determination of the expression mode is confirmed, the expression determination unit 18 inputs expression mode information relating to the determined expression mode to the scoring unit 19 described later. The expression mode information includes information indicating the type of the expression mode, and the start timing and end timing of the expression mode.

  The scoring unit 19 receives the expression mode information from the expression determination unit 18 and the reference melody data from the automatic performance unit 11. The scoring unit 19 positions the input expression mode information on the reference melody, thereby determining at which position of the song the expression mode is performed, and scores the song based on this. The scoring unit 19 scores this evaluation with, for example, 100 points as a full score, and inputs the scoring result to the display unit 20. The display unit 20 displays the scoring result to the singer. This scoring process may be performed in real time for each short singing section (for example, about 10 seconds), or may be performed as a whole after the karaoke song ends. Moreover, you may make it evaluate comprehensively after completion | finish of a singing, performing in real time for every short singing area.

≪Explanation of expression judgment process≫
Hereinafter, the expression determination process executed by the expression determination unit 18 will be described. In this expression determination process, in addition to the expression form (expression type) as the state of the HMM included in each sound of the reference melody data, an expression label that represents more detailed features in each expression form ( Details). The expression modes determined by the expression determination unit 18 and the expression labels (shown in parentheses) of the respective expression modes are as follows.

Attack: Attack
(Normal / scoop up / scoop fry)
Sustain: Sustain
(Normal, fall down)
Vibrato: Vibrato
(Normal)
Release: Release
(Normal, fall down)
Transition: Transition
(Normal, portamento, scoopup / down, staccato).

  The expression determination unit 18 generates an expression transition model for determining the transition of the expression mode for each sound of the reference melody data. FIG. 1 is a diagram showing an example of this expression transition model.

  The expression determination unit 18 is registered in advance with conditions that cause the transition of the expression mode in one sound, and an expression transition model as shown in FIG. 1 is generated based on the condition of the transition. This condition is as follows.

  For example, “the sound after silence always starts with an attack”, “the transition from a state other than silence does not shift to an attack”, “the last sound of a phrase, ie the sound immediately before the silence, is always It ends with release, "" There is no transition from release to a state other than silence. " Moreover, there are two ways of transition of the expression mode performed between two consecutive sounds. One is “release → silence → attack”, and the other is “transition”. In the middle of a note, the expressions sustain and vibrato can be included.

  Although FIG. 1 shows an expression transition model for two sounds, the expression determination unit 18 represents an expression transition model corresponding to one sound or a plurality of sounds belonging to the observation section at that time in each determination cycle. Are generated by concatenating them in a “silence” state and a “transition” state.

  The expression determination unit 18 may repeat the expression determination process at every appropriate timing, but if the expression determination process is executed at the transition timing of the sound in the reference melody data, the transition of the expression mode is determined over at least two sounds. be able to. In this case, the observation interval of one expression determination process may be about 2 seconds, 1 second before and after the transition timing.

In the figure, the state transition starts from silence at the left end, and the start of sound generation is always an attack. And it changes to the direction of the state transition shown by the arrow. Due to this transition, the pronunciation of the musical sound that started with the attack was:
End with release via Sustain, Vibrato or both (normal singing)
After an attack, transition to direct release and end (song that can be played)
From attack, sustain or vibrato to the next musical sound via transition (legato, portamento, etc.)
Multiple types of transitions can be taken.

  The determination performed by the expression determination unit 18 determines which expression mode is included in a section including the frame for one frame or a plurality of consecutive frames, and the section including the frame is the start of the expression mode. A method of obtaining the probability of whether or not the end timing is appropriate from the timing (that is, the probability of whether or not the length of the section is appropriate) and determining the transition of the expression mode having the highest probability as the entire observation section and its transition timing Is executed.

  Therefore, the determination rule does not evaluate the feature parameter in units of frames, but if there is any feature in the continuous frame sequence (if it shows any tendency), the expression form with the continuous frame sequence is It describes whether the probability of determination is high. The probability value is expressed by a real value between 0.0 and 1.0.

In setting the judgment rule, the measurement error required for the judgment often forms a Gaussian distribution.
gaussian (mean, var) = exp (-0.5 * ((mean-x) / var) ^ 2)
It is reasonable to use Alternatively, mainly from the viewpoint of saving the calculation amount, for example, the determination rule may be expressed by a broken line (approximate straight line), or the determination rule may be expressed by the concept of fuzzy logic. Further, the Gaussian distribution, the broken line (approximate straight line), and the fuzzy logic may be combined piecewise.

  Furthermore, final probabilities may be obtained by combining probabilities obtained by a plurality of rules. As the method, for example, the product of the probabilities obtained by each rule is taken.

  The determination rules described below are heuristic rules set by humans based on common sense and empirical rules, but may be based on machine learning.

● Judgment rules regarding silence ・ The probability is lowered according to the ratio of sound frames in all frames.

  This means that a probability distribution such as a Gaussian distribution having an average value of 0 with respect to the ratio of sound frames is defined, and a probability corresponding to the ratio of sound frames measured from the input speech is obtained.

  If the number of sound frames is 3 or more for 10 frames from the first frame (the first half of all frames if there are no 10 frames), the probability is lowered. This means that a probability distribution such as a Gaussian distribution having an average value of 3 with respect to the number of sound frames included in 10 frames from the first frame is defined, and the probability is obtained based on the measured value of the number of sound frames. It is.

  If the number of sound frames is 3 or more for 10 frames from the last frame (the latter half of all frames if there are no 10 frames), the probability is lowered.

  This means that a probability distribution such as a gown distribution having an average value of 3 with respect to the number of sound frames included in the 10 frames from the last frame is defined, and the probability is obtained based on the measured value of the number of sound frames. It is.

  The multiplication value of the above three probabilities is calculated, and the calculation result is set as the cost probability of the silent mode.

● Judgment rules and durations related to attacks If the length of the target voiced section is shorter than the preset threshold, the probability is low.

  For example, 6 frames are set as the threshold value, and the calculation is performed so that the probability is determined as 1.0 when the length of the voiced section is larger than that and as gaussian (6, 1.8) when the length is smaller.

・ Pitch A pitch must exist in the last frame of the judgment section.

  In this case, the value of the probability distribution is either “1” or “0”. That is, if this condition is satisfied, the probability is 1.

・ Energy is low at the beginning of the judgment section.

  The energy increment is large at the beginning of the judgment section.

  The energy increment is small at the end of the judgment section.

  These mean that a probability distribution such as a Gaussian distribution according to an energy value is defined and a probability according to a measured value is obtained. As each condition is satisfied, each probability increases.

  The head portion means the first few frames, and the tail portion means the last few frames. Hereinafter, it has the same meaning.

・ Pitch change degree Pitch fluctuation should be small at the end of the judgment section.

  This means that a probability distribution such as a Gaussian distribution corresponding to the value of the degree of pitch change is defined, and a probability corresponding to the measured value is obtained. The more this condition is met, the greater the probability.

・ Tone change degree The tone change degree is large at the beginning of the section.

  This means that a probability distribution such as a Gaussian distribution corresponding to the value of the timbre change degree is defined, and a probability corresponding to the measured value is obtained. The more this condition is met, the greater the probability.

・ Vibrato Vibrato does not exist in the judgment section.

  In this case, the value of the probability distribution is either “1” or “0”. That is, if this condition is satisfied, the probability is 1.

  A multiplication value of a plurality of probabilities calculated for the above duration, pitch, energy, pitch change degree, tone color change degree, and vibrato is calculated, and the calculation result is set as the cost probability of the attack mode.

◆ Attack expression labeling rules (normal / scoop-fry / scoop-up)
If the pitch variation is small at the beginning of the determination section, the attack expression is labeled normal.

If the pitch variation is large at the beginning of the judgment section, label the attack expression as follows:
If the number of voiced frames is small at the beginning of the judgment section, label the attack expression as SCOOP-FRY (spoken from a very weak utterance).

  If the number of voiced frames is large at the beginning of the judgment section, the attack expression is labeled SCOOP-UP (normal screaming).

● Judgment rules and duration related to release If the length of the target section is shorter than the preset threshold, the probability is low.

  For example, 4 frames are set as the threshold value, and the probability is determined as 1.0 when the length of the section is larger than that, and as gaussian (4, 1.2) when it is smaller.

  In general, the probability of determining a release is low when the sound section becomes long.

  To express this, for example, gaussian (0, c) (c is the number of frames corresponding to 2 seconds, for example, 80 in the case of 40 frame processing per second) is used.

-Pitch There must be a pitch before the judgment section, that is, in the past two frames. In this case, the value of the probability distribution is “1” or “0”. On the other hand, when there is no pitch in the past two frames, the probability is zero.

・ Energy is low at the end of the judgment section.

  The energy is greatly reduced at the end of the judgment section.

  The increase in energy is small at the beginning of the judgment section.

  There is little increase in energy during the judgment section.

  These mean that a probability distribution such as a Gaussian distribution according to an energy value is defined and a probability according to a measured value is obtained. The more this condition is met, the greater the probability.

・ Pitch change degree The change degree of the pitch should be small at the beginning of the judgment section.

  This means that a probability distribution such as a Gaussian distribution corresponding to the value of the degree of pitch change is defined, and a probability corresponding to the measured value is obtained. The more this condition is met, the greater the probability.

・ Tone change degree The tone change degree is large at the end of the judgment section.

  This means that a probability distribution such as a Gaussian distribution corresponding to the value of the timbre change degree is defined, and a probability corresponding to the measured value is obtained. The more this condition is met, the greater the probability.

-Harmonic stability The harmonic stability is low at the beginning of the judgment section.

  This means that a probability distribution such as a Gaussian distribution according to the harmonic stability value is defined, and the probability according to the measured value is obtained. The more this condition is met, the greater the probability.

・ Vibrato Vibrato in the judgment section is low (vibrato speed is slow and vibrato is shallow).

  This means that a probability distribution such as a Gaussian distribution according to the value of vibrato (vibrato speed and vibrato depth) is defined, and the probability according to the measured value is obtained. The more this condition is met, the greater the probability.

◆ Release expression labeling rules (fall-down)
・ To determine whether the expression label of the release is fall-down, calculate the probability of newly labeling it as fall-down.

  If the length of the judgment section is shorter than the set minimum value, the release expression label is fixed as normal instead of fall-down.

  When the length of the determination section is longer than the set shortest value, the process proceeds to the next determination.

  If the difference between the highest pitch in the first half of the judgment section and the lowest pitch in the second half of the judgment section is small, it is unlikely that the release expression label is fall-down, so the probability of labeling fall-down is lowered. .

  When the approximate straight line of the pitch is found from the beginning (first frame) of the section to the last sounded frame, if the slope is negative, the release expression label is likely to be fall-down So increase the probability of labeling fall-down.

  As a result, if the probability of labeling fall-down is higher than the set value, the section ends with labeling fall-down. On the other hand, if the probability of labeling fall-down is lower than the set value, the section is labeled as normal.

● Judgment rules / duration regarding transitions The length of the sounded section to be judged must be longer than the set minimum value.

  In this case, the value of the probability distribution is either “1” or “0”. That is, if this condition is satisfied, the probability is 1.

・ Staccato If there is a part that can be regarded as silence in the first half of the judgment section, it is considered to be staccato. Staccato is treated as a transition type (not release-silence-attack).

・ Pitch The two frames from the beginning of the judgment section must have a pitch.

  There is a pitch in the first frame of the judgment section.

 The last frame of the judgment section must have a pitch.

 If two subsequent (future) frames can be referenced from the end of the judgment section, there must be a pitch for them.

  In these cases, the value of the probability distribution is either “1” or “0”. That is, if this condition is satisfied, the probability is 1.

・ Energy There must be energy that is larger than the set minimum value at both the beginning and end of the judgment section.

・ Vibrato Vibrato in the judgment section is low (vibrato speed is slow and vibrato is shallow).

  This means that a probability distribution such as a Gaussian distribution according to the value of vibrato (vibrato speed and vibrato depth) is defined, and the probability according to the measured value is obtained. The more this condition is met, the greater the probability.

-Harmonic stability If the length of the judgment section is longer than the set shortest value and the stability is high at the beginning of the target section, the probability is lowered.

  If the length of the determination section is longer than the set shortest value and the stability is high at the end of the target section, the probability is lowered.

  When the state of high stability continues for a long time and there is no staccato in the section, the probability is lowered.

-Pitch For the sound before the transition and the sound after the transition that are assumed to be in the judgment section, the average pitch is obtained and the width is calculated.

  The value is compared with the corresponding two-tone pitch in the reference melody information.

・ Pitch change degree The pitch change degree is low at both ends of the judgment section.

  This means that a probability distribution such as a Gaussian distribution corresponding to the value of the degree of pitch change is defined, and a probability corresponding to the measured value is obtained. The more this condition is met, the greater the probability.

  A multiplication value of a plurality of probabilities calculated for the above duration, pitch, energy, vibrato, harmonic stability, pitch, and pitch change is calculated, and the calculation result is used as the cost probability of the transition mode.

◆ Transition labeling calculation, final result calculation, and scoop-up determination are performed by separately calculating probabilities as follows.

  The length of the sound after the transition is shorter or longer than the set minimum value, and there is a section where the pitch is stable at the beginning of the sound after the transition.

    The difference between the average pitch of the entire judgment section and the pitch at the end is at least a semitone.

  There should be no pitch shift of 60 or more in the part of the sound after transition until it becomes stable.

  The last part of the judgment section has low stability.

  There should be little vibrato at the end of the judgment section.

  As each condition is satisfied, each probability increases. A multiplication value of a plurality of calculated probabilities is calculated, and if the calculation result is higher than the set minimum value, the transition expression label is labeled scoop-up.

・ Portamento is determined by calculating the probability separately as follows.

  The length of the sound before the transition is longer than the set minimum value.

  The sound part after the transition must be stable.

  The part of the sound before the transition must have a large degree of pitch change.

  The pitch range is 1 semitone or more in the part of the sound before the transition.

  If there is an unvoiced frame in the part of the sound before the transition, lower the probability.

After the above judgment, if it does not fall under scoop-up or portamento but falls under normal:
If the pitch change is larger than the set positive value and staccato exists, the transition expression label is labeled staccato-normal-up.

  Otherwise, label the transition expression label as normal-up.

  If the pitch change is smaller than the set negative value and staccato exists, the transition expression label is labeled staccato-normal-down.

  Otherwise, label the transition label as normal-down.

● Judgment rules and duration regarding sustain If the length of the section to be judged is shorter than a preset threshold, the probability is low.

  This means that a probability distribution such as a Gaussian distribution according to the value of the duration is specified, and the probability according to the measurement value is obtained. The more this condition is met, the greater the probability.

・ Pitch The pitch must be in the first frame of the judgment section.

  There is a pitch in the last frame of the judgment section.

 If two subsequent frames from the end of the decision section can be referenced, there must be a pitch for them.

・ Pitch change degree There should be no large pitch change at the beginning of the target section.

・ Energy Value must be larger than the set minimum value.

  The energy value is stable.

  This means that a probability distribution such as a Gaussian distribution according to the energy value is defined and the probability according to the measured value is obtained. The more this condition is met, the greater the probability.

・ Tone change rate The tone change rate must be within the preset range.

  This means that a probability distribution such as a Gaussian distribution corresponding to the value of the timbre change degree is defined, and a probability corresponding to the measured value is obtained. The more this condition is met, the greater the probability.

・ Vibrato There should be little vibrato in the judgment section.

  In this case, the value of the probability distribution is either “1” or “0”. That is, if this condition is satisfied, the probability is 1.

◆ The sustain expression labeling rule label is only normal, but the probability of normality is calculated according to the following rules and reflected in the final result.

  The pitch must be stable.

  The slope of the approximate straight line of the pitch in the entire judgment section is close to 0

● Vibrato Judgment Rules / Duration Time The probability is low when the length of the voiced section to be judged is shorter than a preset threshold.

  This means that a probability distribution such as a Gaussian distribution according to the value of the duration is specified, and the probability according to the measurement value is obtained. The more this condition is met, the greater the probability.

・ Pitch The pitch must be in the first frame of the judgment section.

  There is a pitch in the last frame of the judgment section.

 If two subsequent (future) frames can be referenced from the end of the judgment section, there must be a pitch for them.

  In these cases, the value of the probability distribution is either “1” or “0”. That is, if this condition is satisfied, the probability is 1.

・ Energy Value must be larger than the set minimum value.

  This means that a probability distribution such as a Gaussian distribution according to the energy value is defined and the probability according to the measured value is obtained. The more this condition is met, the greater the probability.

・ Tone variation The timbre variation is within the preset range.

  In this case, the value of the probability distribution is either “1” or “0”. That is, if this condition is satisfied, the probability is 1.

・ Pitch change degree About the maximum and minimum pitch change degree stopped in the judgment section,
The maximum value is larger than the set lower limit value.

  The minimum value is smaller than the set upper limit value.

  In these cases, the value of the probability distribution is either “1” or “0”. That is, if this condition is satisfied, the probability is 1.

・ Vibrato There are many vibrato in the judgment section.

  This means that a probability distribution such as a Gaussian distribution according to the vibrato value is specified, and the probability according to the measured value is obtained. The more this condition is met, the greater the probability.

  A multiplication value of a plurality of probabilities calculated with respect to the above duration, pitch, energy, timbre change degree, pitch change degree and vibrato is calculated, and the calculation result is set as the cost probability of the vibrato mode.

  Note that the probability distribution is not limited to the Gaussian distribution, and may be a Gaussian distribution approximated by a straight line.

  Next, the method of expression determination and optimum route search will be described with reference to FIGS.

  FIG. 6 is a diagram in which the optimal route of the expression transition searched by the Viterbi algorithm is described on the lattice diagram. The probability at each node (frame) gives the best path to reach that node starting from the beginning of the performance. Here, two probabilities are considered. One is TransitionProbability (transition probability) P trans and the other is Cost Probability (cost probability) Pcost. In this embodiment, all transition probabilities for all branches at all nodes are set to 1 in order to facilitate calculation. Therefore, the overall probability is. It is determined only by the cost probability.

Since the transition probabilities are all 1, for example, the path probabilities shown in FIG.
P = Pcost (1) Pcost (2) Pcost (3) Pcost (4)
Given in.

  The cost probabilities are given as n probabilities when the state of 0 to n frame trains before that frame is the end frame at each node (frame). The cost probability is given by the product of the probabilities of all the rules.

  After analyzing the entire observation interval, the Viterbi matrix backtracking method is executed to determine the path with the highest probability.

  In FIG. 5, the path of expression transition of silence, attack, transition, vibrato, release, silence is selected as the most probable path. It is a figure which shows an example. FIG. 6 shows the state up to the transition among the above in the Viterbi matrix.

  When the path of the expression mode is determined in this way, the most likely label is determined for each expression mode. This determination is also calculated based on the heuristic rules described above.

  The upper part of FIG. 5 shows the exact time position of the start and end of the final expression type. The attack is scoop up, the transition is normal, the vibrato is regular, and the release is fall down.

  The above expression determination process may be performed for an observation interval of about 2 seconds, but may be determined over the whole after the end of the song.

≪Ingenuity of real time≫
The expression determination process has been described above, but in order to determine the expression in real time every 2 seconds, in particular, while performing score matching that detects transition points of the actual sound (deviation from the reference) of the singing voice. Further, it is preferable to perform the following operation.

  That is, here, a process of “determining the path of expression transition with the highest probability for the ended sound every time the start (or end) of a sound is determined by score matching” is performed.

  This processing procedure will be described with reference to FIGS.

  In FIG. 7, the time at which the first sound of the reference melody starts is known, but its duration (end point) is unconfirmed, and the first sound section is unconfirmed. However, only the first expression mode in this sound can be determined to be an attack. This is because the first sound of the phrase, that is, the sound immediately after the silence.

  FIG. 8 shows a situation when the start of the second sound in the reference melody is confirmed as a result of the score matching. In this state, the route with the highest probability is determined within the range of the first sound. As a result, paths (expression modes) such as attack (scoop-up), vibrato (normal), and transition are determined.

  However, the duration and label of Transition cannot be determined yet. This is because the end point is determined in the next sound. Therefore, after the duration of the next sound is determined by score matching, the transition duration (end point) and label are determined.

  In this way, instead of determining the expression mode of all the sections within the observation interval of about 2 seconds, by sequentially determining the expression mode for sounds whose actual duration has been determined by score matching, It is possible to determine the expression mode in real time with high accuracy.

  As described in detail above, according to the present embodiment, an expression transition model is first generated from melody information serving as a reference. Characteristic parameters such as pitch, volume, and spectral change are sequentially detected from the input musical sound. From these values, the probability that the expression changes according to a predetermined rule and the probability that the expression stays without changing are calculated. Based on this probability, the expression transition route with the highest probability is selected in the expression transition model. Then, the position of the transition of the expression (the length of the section) is determined, and the characteristic tendency in each expression is labeled. Thereby, the expression mode and expression transition of singing or performance can be detected more accurately from the input musical sound signal. If this is made into a karaoke apparatus, for example, singing can be scored more accurately.

  Also, stream processing is realized by adopting a data structure in which the probability calculation results of the determined section are discarded one after another. Thereby, the expression mode and expression transition of singing or performance can be detected more accurately in real time from the input audio signal.

  In addition, although this embodiment showed the example which applied the audio | voice signal processing apparatus to the karaoke apparatus 1, application of this invention is not limited to this. The present invention can be applied to any device as long as it can determine the expression of the input performance sound. The input voice is not limited to singing, and may be a performance sound of a musical instrument.

It is a figure which shows the reference | standard melody data and expression transition model which are used for the expression determination process by this invention. It is a figure which shows an example of the state transition in the expression transition model which arranged each state to the vertical direction, and arranged the frame to the horizontal direction in time series. It is a block diagram which shows roughly the structure of the karaoke apparatus as an audio | voice signal processing apparatus. It is a block diagram which shows the structure of the characteristic parameter acquisition part of the karaoke apparatus in detail. It is the figure explaining the expression determination process which the karaoke apparatus performs with an expression transition model. It is a figure explaining the said expression determination process. It is a figure explaining the process in the case of performing the said expression determination process in real time. It is a figure explaining the process in the case of performing the said expression determination process in real time.

Explanation of symbols

1-Karaoke device 12a-Connection terminal 13-FFT processing unit 14-Feature parameter acquisition unit 15-Feature parameter buffer 17-Reference buffer 18-Expression determination unit

Claims (7)

An audio signal input unit for inputting an audio signal for playing or singing a musical sound;
A feature parameter detection unit that divides an input speech signal into frames for each predetermined time and detects a feature parameter of the speech signal for each frame;
A plurality of expression modes of performance or singing are each modeled as one state, and the probability that one frame or a plurality of consecutive frames is a specific state section is calculated over a predetermined observation section based on the feature parameter of each frame. An expression determination unit that performs an expression determination process for determining an expression mode of the voice signal and the section by determining an optimum path of state transition in a predetermined observation section based on the probability;
An audio signal processing apparatus.   The audio signal according to claim 1, wherein the expression determination unit further determines, based on the feature parameter, for a section in which an expression mode is determined as the expression determination process, and determines the detailed contents of the expression mode. Processing equipment. A buffer for storing the feature parameter detected by the feature parameter detector;
The said expression determination part performs the said expression determination process about the partial area of a music in the middle of a music, The said buffer has a storage capacity for the partial area of the said music. 2. The audio signal processing device according to 2.   The audio signal processing apparatus according to claim 3, wherein the expression determination unit detects a transition point of the musical sound in the input audio signal, and executes the expression determination process at the transition point.   The audio signal processing apparatus according to claim 1, wherein the expression determination unit determines the optimum route using a Viterbi algorithm. An audio signal input procedure for inputting an audio signal for playing or singing a musical sound;
A feature parameter detection procedure for dividing the input speech signal into frames at predetermined time intervals and detecting feature parameters of the speech signal for each frame;
A plurality of expression modes of performance or singing are each modeled as one state, and the probability that one frame or a plurality of consecutive frames is a specific state section is calculated over a predetermined observation section based on the feature parameter of each frame. , By determining the optimum path of state transition in a predetermined observation section based on this probability, an expression determination procedure for determining the expression mode of the voice signal and the section;
An audio signal processing method comprising: In signal processing equipment,
An audio signal input procedure for inputting an audio signal for playing or singing a musical sound;
A feature parameter detection procedure for dividing the input speech signal into frames at predetermined time intervals and detecting feature parameters of the speech signal for each frame;
A plurality of expression modes of performance or singing are each modeled as one state, and the probability that one frame or a plurality of consecutive frames is a specific state section is calculated over a predetermined observation section based on the feature parameter of each frame. , By determining the optimum path of state transition in a predetermined observation section based on this probability, an expression determination procedure for determining the expression mode of the voice signal and the section;
An audio signal processing program characterized in that

Images