JP6187132B2 - Score alignment apparatus and score alignment program - Google Patents

Description

  The present invention analyzes the captured sound signal while capturing the sound signal representing the performance sound of the music, thereby allowing the currently played portion of the score of the music (hereinafter referred to as the score position) in real time. The present invention relates to a score alignment apparatus to be estimated and a computer program (score alignment program) applied to a computer provided in the score alignment apparatus.

  Conventionally, for example, as shown in Non-Patent Documents 1 and 2 below, score alignment devices (automatic accompaniment devices) are known. When a performer performs a piece of music, it is rare that the performer performs according to the score of the piece of music, and the same part is played repeatedly or a part that cannot be played may be skipped. In order to cope with any musical score position transition as described above, in the score alignment apparatuses of Non-Patent Document 1 and Non-Patent Document 2, the performance process (transition of the musical score position) is described as a probability model.

  When transition from the current score position to all other score positions is possible, the amount of calculation when estimating the score position after the transition is remarkably increased. Therefore, in Non-Patent Document 1, an increase in calculation amount is suppressed by setting an appropriate assumption for the transition of the musical score position.

  Further, the estimation accuracy of the score position estimated in real time is lower than the estimation accuracy of the score position estimated by batch processing (non-real time processing). Therefore, in Non-Patent Document 2, a musical score position that is a predetermined time before the current time is estimated, a tempo trajectory is estimated, and the current musical score position is estimated using both estimation results.

Eita Nakamura, Haruto Takeda, Ryuichi Yamamoto, Yasuyuki Saito, Shinji Sakai, Shigeki Hatakeyama, “Score tracking and automatic accompaniment that can follow performances including replaying and skipping to any location”, Transactions of Information Processing Society of Japan April 2013, vol. 54, no. 4, p. 1338-1349 Ryuichi Yamamoto, Shinji Sakaki, Tadashi Kitamura, “Ryry: an automatic audio input accompaniment system for polyphonic instruments”, Information Processing Society of Japan Interaction, March 2, 2013, 3EXB-13

  In the said nonpatent literature 1, although the increase in calculation amount is suppressed, it is not enough. In Non-Patent Document 2, since the certainty of each state constituting the state series is not taken into account when estimating the tempo, there is a possibility that the estimation accuracy of the score position is lowered. Therefore, when such a score alignment device is applied to a media player (automatic accompaniment device, image display device, etc. (see Japanese Patent No. 4399996, Japanese Patent No. 4534926 etc.)), the progress position of the performance by the performer and other media There is a possibility that the playback position of (accompaniment, images, etc.) is shifted. In other words, playback of other media in response to the performer's performance may feel unnatural.

  The present invention has been made to address the above problems, and an object of the present invention is to provide a score alignment apparatus that more effectively suppresses an increase in the amount of calculation and improves the accuracy of estimating a score position. . In addition, in the description of each constituent element of the present invention below, in order to facilitate understanding of the present invention, reference numerals of corresponding portions of the embodiment are described in parentheses, but each constituent element of the present invention is The present invention should not be construed as being limited to the configurations of the corresponding portions indicated by the reference numerals of the embodiments.

In order to achieve the above object, a feature of the present invention represents a portion of a musical score currently being played in the musical score of the musical piece by analyzing the captured acoustic signal while taking in an acoustic signal representing the musical performance sound. A score alignment apparatus (10) for estimating a score position and a tempo in real time, a probability model (HSMM) expressed as a series of states each representing a score position, the current state depending on the immediately preceding state Score position probability density / tempo probability density calculation that calculates the probability density of the score position and the probability density of the tempo based on a probability model that has the property of being able to transition to any state from the current state and means (S151~S155), the true score position _(x t), the transition rates of the true score position _(v t), and the true score position of the transition acceleration _{(a t)} using A score position / tempo determination means (S162 to S165) for determining the current score position and tempo using the calculated probability density sequence of the score position based on the expressed autoregressive process. The score alignment apparatus is used. In the state transition of the probability model (HSMM), the state before the transition and the state after the transition may be the same.

In this case, the state (S _{i, n, T} (t)) includes the section (i) including the current score position among the plurality of sections obtained by dividing the score and the current score position. Is specified using a time (n) required to play from the beginning of the section including the current score position and a time (T) required to play the entire section, and the probability model is A hidden semi-Markov model (HSMM) represented as a sequence of states, wherein the score position probability density / tempo probability density calculating means applies a forward algorithm to the hidden semi-Markov model, Calculate the probability density of the tempo.

  In the score alignment apparatus configured as described above, first, a score position probability density and a tempo probability density are calculated. Then, the score position and the tempo are determined based on the higher-order autoregressive process using the score position probability density sequence and the tempo probability density sequence. According to this, it is possible to express the property of the music acoustic signal that the temporal differentiation of the tempo (that is, the acceleration of the musical score position) is continuous and shows a tendency to return to “0”. Further, in a section where the distribution of the score position probability density and the tempo probability density is large, a behavior is obtained in which the path of the score position probability density sequence and the tempo probability density sequence is smoothed. Thereby, the estimation accuracy of the score position and the tempo can be improved.

  Another feature of the present invention is that the score position probability density / tempo probability density calculating means includes a section including a current score position among a plurality of sections obtained by dividing the score, Expressed as a sequence of states specified using the time taken to play from the beginning of the section containing the score position to the current score position and the average time taken to play the entire section The score alignment apparatus includes section search means (S153) for searching a plurality of sections to which the forward algorithm is applied among sections of the hidden semi-Markov model based on a hidden Markov model (HMM).

  According to this, a forward variable is calculated by applying a forward algorithm to a normal hidden Markov model, and a state (section) in which the forward variable is maximized is searched. Then, in the hidden semi-Markov model, a forward algorithm is applied to a plurality of sections including a section corresponding to the searched state (section), and a score position probability density and a tempo probability density are calculated. Therefore, an increase in the amount of calculation can be suppressed as compared with the case where the forward algorithm is applied to all the sections constituting the hidden semi-Markov model.

  In this case, it is preferable that the number of sections of the hidden Markov model (HMM) is larger than the number of sections of the hidden semi-Markov model (HSMM). According to this, among the sections of the hidden semi-Markov model (HSMM), the number of sections of the hidden Markov model (HMM) is identical to the section function of the hidden semi-Markov model (HSMM). Compared to the case where it is, it can search more appropriately.

  The present invention can also be implemented as a computer program applied to a computer provided in the score alignment apparatus.

It is a block diagram showing the structure of the score alignment apparatus which concerns on one Embodiment of this invention. It is a functional block diagram of a score alignment apparatus. It is a state transition diagram of a hidden semi-Markov model and a hidden Markov model. It is a graph showing an example of a sound model. It is a conceptual diagram showing the process of estimating the tempo trajectory model for the sequence of score position observation density It is a flowchart showing a score alignment process. It is a flowchart showing a score position probability density / tempo probability density calculation process. It is a flowchart showing a score position / tempo determination process.

  A score alignment apparatus 10 according to an embodiment of the present invention will be described. The score alignment apparatus 10 analyzes the captured acoustic signal while capturing the acoustic signal representing the performance of the music, and estimates which part of the musical score of the music is currently being played. In the present embodiment, standard MIDI file format data is used as musical score data representing a musical score.

  As shown in FIG. 1, the score alignment apparatus 10 includes an input operator 11, a computer unit 12, a display 13, a storage device 14, an external interface circuit 15, and a sound system 16, which are connected via a bus BS. It is connected.

  The input operator 11 includes a switch corresponding to an on / off operation (for example, a numeric keypad for inputting a numerical value), a volume or rotary encoder corresponding to a rotation operation, a volume or linear encoder corresponding to a slide operation, a mouse, a touch panel, etc. Composed. These operators are operated by a player's hand and used for starting or stopping the score alignment process, setting various parameters related to the score alignment process, and the like. When the input operator 11 is operated, operation information indicating the operation content is supplied to the computer unit 12 described later via the bus BS.

  The computer unit 12 includes a CPU 12a, a ROM 12b, and a RAM 12c connected to the bus BS. The CPU 12a reads out from the ROM 12b and executes a score alignment program representing a procedure of score alignment processing described later. In addition to the program, the ROM 12b stores various data such as initial setting parameters, graphic data for generating display data representing an image displayed on the display 13, and character data. The RAM 12c temporarily stores data necessary for executing the program.

  The display 13 is configured by a liquid crystal display (LCD). The computer unit 12 generates display data representing contents to be displayed using graphic data, character data, and the like, and supplies the display data to the display unit 13. For example, the computer unit 12 supplies the display unit 13 with display data representing a musical score position estimated by score alignment processing described later. The display device 13 displays an image based on the display data supplied from the computer unit 12.

  The storage device 14 includes a large-capacity nonvolatile recording medium such as an HDD, FDD, CD, or DVD, and a drive unit corresponding to each recording medium. The storage device 14 stores musical score data (standard MIDI file) representing a musical score. The musical score data may be stored in the storage device 14 in advance or may be taken in from the outside via the external interface circuit 15 described later.

  The external interface circuit 15 includes a connection terminal that enables the score alignment device 10 to be connected to an external device such as an electronic music device or a personal computer. The score alignment apparatus 10 can be connected to a communication network such as a LAN (Local Area Network) or the Internet via the external interface circuit 15.

  The sound system 16 includes a sound source circuit that generates a digital sound signal, a D / A converter that converts the generated digital sound signal into an analog sound signal, an amplifier that amplifies the converted analog sound signal, and an amplified analog A speaker that converts a sound signal into an acoustic signal and outputs the sound signal is provided. The sound system 16 includes a microphone for collecting a musical sound emitted by playing a musical piece, an A / D converter for converting an analog sound signal representing the collected musical sound into a digital sound signal, and a converted sound. A buffer for temporarily storing sample data representing the digital sound signal is also provided. That is, the sound system 16 samples the musical sound at a predetermined sampling period (for example, 1/444100 sec), and stores the sample data obtained by the sampling in the buffer.

  Next, a score position and tempo estimation method will be described. As shown in FIG. 2, the score alignment apparatus 10 first analyzes the captured acoustic signal while capturing the acoustic signal representing the performance of the music through the microphone, and the probability density of the currently played musical score position and the current Calculate the probability density of the tempo. Then, an optimal score position and tempo are determined using the calculated probability density series. The determined musical score position and tempo are used to control a control target (display 13, sound system 16, etc.).

  Next, a method for calculating the score density and the tempo probability density will be described. In the present embodiment, as will be described below, the series of the plurality of sections is modeled as a hidden semi-Markov model HSMM (see FIG. 3, equation (2)). First, as shown in FIG. 3, the musical score is divided into a plurality of sections i (= 1, 2,..., I). The length of each section is the same. For example, the length of each section is the length of one quarter note. “I” is an index indicating the number of sections counted from the beginning of the music. The score corresponding to the actual performance can be expressed as a series of a plurality of sections divided as described above.

  The time taken to play one section (the time during which the performance stays in one section) depends on the tempo. For example, when the tempo is 60 BPM (Beats Per Minute), it takes 1 second to play one section. When the tempo is 120 BPM, it takes 0.5 seconds to play one section. Here, if expressed as the number of frames with a unit time of 0.1 seconds, for example, when the tempo is 60 BPM, it takes 10 frames to play one section, and when the tempo is 120 BPM. It takes time for 5 frames to play one section.

The section i is played at time t (t-th frame from the beginning of the music), and it is determined that it takes time corresponding to the number of frames T to play the section i. The state in which the performance has been completed up to the nth frame from the beginning (the state in which it took n frames to perform from the beginning of the section i to the current score position) is the state S _{i, n, T} (t ). The circles in FIG. 3 correspond to the states S _{i, n, T} (t). It is assumed that the tempo does not change within each section. That is, in each series of circles connected by arrows in the figure, the transition from the left circle to the right circle is sequentially performed. Therefore, the number of circles constituting each series corresponds to the tempo. That is, the tempo is faster as the series has a smaller number of circles, and the tempo is slower as the series has a larger number of circles. Therefore, if one initial state is selected in each section, the length (number of frames T) that the performance stays in that section is determined.

In general, since the performance progresses in order from the beginning to the end of the score, when the performance of one section is completed, only the transition to the next section is allowed. Here, the probability of transition from section i to section _j is denoted as probability τ _{i, j} . Further, when transitioning from the last state of one section to the initial state of the next section, it is possible to transition to an arbitrary initial state. That is, when transitioning from one section to the next section, the tempo can be changed. Here, the probability of transition from the tempo corresponding to the number of frames _{T ′} to the tempo corresponding to the number of frames _T is expressed as probability τ _{T ′, T.} Then, the state _{S i', n',} T'state from _{(t) S i, n,} T (t + 1) to the probability of transition state transition probability _{τ (i', n', T')} ~ (i, _{n, T)} is expressed as the following formula (1). In this embodiment, in order to simplify the description, it is assumed that performance symbols (da capo, repetition symbols, etc.) that shift the musical score position to a distant section are not included in the musical score.

However, when a performer actually plays a piece of music, a part that is not specified in the score may be played repeatedly or a part that cannot be played may be skipped. That is, there is a possibility that the musical score position changes (jumps) to a section far away from the adjacent section. Therefore, the probability γ that the interval transitions according to the hidden Markov model HMM or the hidden semi-Markov model HSMM _, the probability π _{i, n, T} that observes the state S _{i, n, T} , and the observation likelihood of the state S _{i, n, T} A model as shown in the following equation (2) is set using O _{i, n, T} (t). The observation likelihood O _{i, n, T} (t) will be described later.

Next, the power y _m (t) of each pitch m included in the musical sound frame t sampled by the sound system 16 and the power increase Δy _m (t) from the previous frame are the characteristics of the acoustic signal. Calculated as a quantity. Here, the observation likelihood of the power y _m (t) and the observation likelihood of the power increase amount Δy _m (t) follow the von Mises-Fisher distribution, as shown in the equations (3) and (4), respectively. Assume that

Then, the observation likelihood O _{i, n, T} (t) of the equation (2) is expressed as the following equation (5).

Note that “κ” in the above formulas (3) and (4) represents the degree of concentration of the von Mises Fisher distribution. That is, as “κ” is larger, a sharper peak is formed around the average value in the distribution of observation likelihood of the power y _m (t) and the increase amount Δy _m (t) of the power. The value of “κ” is set to “100”, for example. Further, “w (k)” is a template for a characteristic amount of an acoustic signal (hereinafter referred to as a sound model). “K” is an index for specifying a sound model. Each sound model is data obtained by generating musical tones (single notes) of each pitch using each musical instrument, and calculating and recording characteristic quantities of those musical tones. For example, “w (k = 1)” is data obtained by playing a piano to generate a sound having a pitch corresponding to a MIDI note number “69” and recording the characteristic amount (power). Further, for example, “w (k = 2)” is data in which a musical tone having a pitch corresponding to a MIDI note number “69” is generated by playing a violin and the characteristic amount (power) is recorded. “H” represents the intensity of each sound model. In the case of music with a large number of pronunciations, there may be a large difference between the intensity of each set sound model and the intensity of the musical sound actually played. In this case, the value of “κ” may be decreased to increase the dispersion.

The observation likelihood (formula (3)) of the power y _m (t) will be specifically described. In order to simplify the explanation, it is assumed that the music to be analyzed is a music played by a single instrument, and that “k” as the sound model index matches the MIDI note number NN. Here, it is assumed that the current state is the state S _{i = 4, n = 6, T = 12} . Consider the observation likelihood of power y _m (t) at this time. In this case, since i + n / T = 4 + 6/12 = 4.5, “h (4.5)” corresponding to the score position 4.5 is extracted. In FIG. 4, the intensity (that is, the magnitude of “h”) of each sound model is shown as a gray scale graph. The strength of elements shown dark in the figure is large. In this example, the intensity of the element of k = 69 is large, and as a result, the observation likelihood of the power y _m (t) has an average value in which the element of “w (k = 69)” is dominant. Distributed.

The forward variable α _{i, n, T} (t) in the hidden semi-Markov model HSMM is expressed as the following equation (6).

If this formula (6) is arranged, the recurrence formula shown in the following formula (7) is obtained.

Here, in order to simplify the description, a model that can uniformly transition to any musical score position will be considered. In this case, the probability π of observing the state S is expressed as in Equation (8) using the number of states | S |.

If the probability (1-γ) of transition to the initial state is “0.01”, the observation likelihood O _i (t) of state i, transition probability τ _{i, j} from state i to state _j , state Using i's forward variable α _i (t), an update equation for the forward variable α is expressed as in Equation (9).

The part of “τ _{i, j} × 0.99” and the part of “0.01 / | S |” in equation (9) can be calculated when the musical score data is read. On the other hand, if the value of “γ” is set to “1” in equation (7), a recurrence formula of the forward variable in the normal hidden Markov model HMM is obtained as shown in equation (10).

Therefore, the difference (overhead) between the computation of the forward variable update in the hidden semi-Markov model HSMM and the computation of the forward variable update in the normal hidden Markov model HMM is a process of adding “0.01 / | S |”. Only. In this example, it is possible to make uniform transition to any musical score position, but the overhead is the same as in this example even when state transition is restricted.

  In the present embodiment, the time series of the divided sections is modeled as a hidden semi-Markov model HSMM, so the number of states is enormous compared to the case of modeling as a normal hidden Markov model, and the section i, frame The number of combinations of the number n and the frame number T is also enormous. Therefore, if the probability density of the score position is calculated using a forward algorithm, the calculation amount becomes enormous. Therefore, as will be described below, the score alignment apparatus 10 includes section search means for narrowing down a section of a hidden semi-Markov model HSMM to which a forward algorithm is applied using a normal hidden Markov model HMM.

A normal hidden Markov model HMM is defined as follows. That is, the score is divided in the same manner as the above-described hidden semi-Markov model HSMM, and a state variable is assigned to each of the divided sections. However, the musical score is divided so that the number of sections of the hidden Markov model HMM is larger than the number of sections of the hidden semi-Markov model HSMM. For example, in the hidden semi-Markov model HSMM, the score is divided so that the length of each section is the length of a quarter note. In the hidden Markov model HMM, the length of each section is a 32nd note. Divide the score into lengths. Each state (section) can also transition to itself. That is, in the hidden Markov model HMM, the probability of transition from a certain state to itself is “τ ^(HMM) ”, and the probability of transition from one state to the next state is “1-τ ^(HMM) ”. is there. A forward algorithm is applied to such a hidden Markov model HMM in real time to search for a state in which the forward variable is maximum in each frame t. Then, the forward algorithm is applied only to a predetermined number (for example, 16 pieces (four bars of music of four beats)) adjacent to the section of the hidden semi-Markov model HSMM corresponding to the searched state.

Note that “τ ^(HMM) ” can be regarded as the number of sections that transition per frame. This “section” is a section of the hidden Markov model HMM. Therefore, if “τ ^(HMM) ” representing the probability of transition to itself does not match the currently estimated tempo (that is, the number of frames T), the section ΔS may not be appropriately obtained. . Therefore, the distribution (= Σ _{i, n} α _{i, n, T} (t)) with respect to the current frame number T is calculated using the probability density calculated based on the hidden semi-Markov model HSMM. Then, “τ ^(HMM) ” is determined by calculating the expected value of the number of transitions per frame using the current frame number T. Thereby, the tempos of the hidden Markov model HMM and the hidden semi-Markov model HSMM are matched.

Next, a method for determining the current score position based on the calculated score position probability density and tempo probability density series will be described. In Non-Patent Document 2, tempo continuity is modeled as a first-order autoregressive process. That is, with respect to an independent tempo change amount ε _{t according} to a normal distribution in which the tempo at frame t is “ν _t ”, the average value is “0”, and the variance σ ² is greater than “0”, ν _t = ν a model in _t-1 + _{ε t} has been assumed. However, in a music acoustic signal, when the tempo is fast (slow), ε _t takes a positive (negative) value in a certain continuous section, and the time derivative of the tempo (that is, the acceleration of the musical score position) is It tends to return to “0”. That is, the tempo change amount ε _t in a certain frame depends on the tempo change amount ε _t of a frame adjacent to the frame.

Therefore, in this embodiment, higher order information is adopted. The probability density of the musical score position related to the frame t is expressed as a musical score position probability density U _q (t), and the probability density of the tempo is expressed as tempo probability density V _T (t). Here, “q” is a variable defined by an arithmetic expression “q = round (M (i + n / T))” for an arbitrary “M”. That is, “V _T (t)” is a probability that “q” transitions by “M / T” per frame.

Here, denoted the true score position score position x _t, tempo v _t the transition speed of the true score _position, the acceleration a _t the transition acceleration of the true score position in the frame t. In other words, the tempo v _t, which corresponds to the first derivative of the score position x _t, the acceleration a _t corresponds to the second differential of the score position x _t. Then, a state space model (higher order autoregressive process) defined by the following equations (11) to (13) is set. That is, the musical score position trajectory model representing the transition of the musical score position, the tempo trajectory model representing the tempo transition, and the acceleration trajectory model representing the acceleration transition are set.

Note that "r" in the equation (13) is the attenuation coefficient of the acceleration _{a t.} By the action of the damping coefficient, the acceleration a _t is a tendency that continuously changes, and return to "0". In addition, when “r” is large, the change in tempo becomes gentle, and when “r” is small, the change in tempo tends to become intense. “R” is set to “0.5”, for example. “R” may be set to an optimal value based on the tempo data of the actual performance.

If the state space model can model what observation values (that is, score position probability density U _q (t) and tempo probability density V _T (t) are generated, state transition and observation likelihood are considered simultaneously. Thus, the state variable can be inferred, so that the mean value μ (U _q (t)) and variance σ ² (U _q (t)) of the score position probability density U _q (t), and the tempo probability The average value μ (V _T (t)) and variance σ ² (V _T (t)) of the density V _T (t) are calculated using the following formulas (14) to (17).

That is, the average value and variance of the score position probability density U _q (t) and the tempo probability density V _T (t) for the frames around the estimated current score position x _t are calculated. Then, the observation likelihood is defined as shown in the following equation (18).

That is, first, from the sequence of the sequence and the tempo probability density of score position probability density between the current frame and the current frame located N pieces prior frame, a sequence of probability density of the frame around the score position x _t Are extracted respectively. Here, the probability density calculated in a frame located by ΔT pieces before is regarded as a normal distribution. That is, the mean and variance of the histogram of probability density calculated in the frame located by ΔT before are regarded as the mean and variance of the normal distribution. The score position x _t, using the tempo v _t and the acceleration a _t, calculates the likelihood of the tempo track model and acceleration trajectory model in frame located before ΔT pieces only. FIG. 5 is a conceptual diagram showing a process of estimating a tempo trajectory model for the calculated musical score position observation density sequence. In practice, an acceleration trajectory model for the tempo probability density series is also estimated. If the Kalman filter is used, the musical score position trajectory model, the tempo trajectory model, and the acceleration trajectory model as described above can be estimated in real time. An update step of the Kalman filter is executed, and the average value <x _t > of the musical score position x _t and the average value <v _t > of the tempo v _t are calculated using the updated estimated value. Then, the calculated average value <x _t > of the musical score position x _t and the average value <v _t > of the tempo v _t are determined as the current musical score position and tempo.

  Next, the operation of the score alignment apparatus 10 will be specifically described. As shown in FIG. 6A, the CPU 12a reads the score alignment program from the ROM 12b in step S10, and starts the score alignment process. Next, the CPU 12a displays a list of score data on the display unit 13 in step S11. The user uses the input operator 11 to select the musical score data of the music to be subjected to the score alignment process (that is, the music to be played) from the displayed list. Next, in step S12, the CPU 12a reads the selected score data from the storage device 14 and divides it into a plurality of sections i (= 1, 2,..., I).

  Next, in step S13, the CPU 12a causes the sound system 16 to start sampling of musical sounds. Next, in step S14, the CPU 12a sets the processing target frame as the first frame. That is, the value of “t” that is the index of the frame is set to “1”.

Next, in step S15, the CPU 12a executes a score position probability density / tempo probability density calculation process. As shown in FIG. 6B, the CPU 12a starts the score position probability density / tempo probability density calculation process in step S150. Next, the CPU 12a reads the acoustic signal (sample data) included in the frame t from the buffer of the sound system 16 in step S151. Next, in step S152, the CPU 12a calculates the observation likelihood O _{i, n, T} (t) based on the above formulas (3) to (5). Next, in step S153, the CPU 12a applies a forward algorithm to the normal hidden Markov model HMM using the calculated observation likelihood O _{i, n, T} (t), and the forward variable is changed in the frame t. Detect the maximum state. As a result, the interval in which the forward algorithm is applied to the hidden semi-Markov model HSMM is determined. Next, in step S154, the CPU 12a applies a forward algorithm to the determined section among the plurality of sections constituting the hidden semi-Markov model HSMM (see formula (7)). Next, in step S155, the CPU 12a uses the forward variable calculated by applying the forward algorithm to the hidden semi-Markov model HSMM, and uses the score position probability density U _q (t) and the tempo probability density V _T (t). Calculate Then, in step S156, the CPU 12a ends the score position probability density / tempo probability density calculation process, and proceeds to step S16 of the alignment calculation process.

Next, in step S16, the CPU 12a executes score position / tempo determination processing. As shown in FIG. 6C, the CPU 12a starts the musical score position / tempo determination process in step S160. Next, in step S161, the CPU 12a determines whether or not the musical score position has jumped. Specifically, based on the difference between the score position probability density U _q (t) calculated for the current frame and the score position probability density U _q (t−1) calculated for the previous frame, It is determined whether or not the score position has jumped. For example, based on the score position probability density U _q (t) in the current frame and the score position probability density U _q (t−1) in the previous frame, the most likely score position (hidden semi-Markov model HSMM) And the score position jumps when the score position of the detected current frame and the score position of the previous frame are 4 bars or more apart from each other. judge. If the score position is not jump, CPU 12a determines "No" at step S162, the equation (14) to, based on the equation (17), in the vicinity of the frame of the score position _{x t} The average value and variance of the score position probability density U _q (t) and the tempo probability density V _T (t) are calculated. Next, in step S163, the CPU 12a calculates an observation likelihood based on the above equation (18), and estimates a tempo transition model and an acceleration transition model using a Kalman filter.

On the other hand, if the musical score position jumps, the CPU 12a determines “Yes” in step S161. Then, CPU 12a, at step S164, sets score position _{x t,} tempo _{v t,} and the value of the acceleration _{a t} as follows. For example, the most likely score position is detected based on the score position probability density U _q (t) in the current frame, and the detected score position is set as the score position x _t . Further, the tempo v _t is set to a specified value (for example, “120 BPM”). Also, setting the acceleration a _t prescribed value (e.g. "0").

  In step S165, the CPU 12a determines the score position and tempo using the estimation results of the score position transition model, the tempo transition model, and the acceleration transition model, and sets the control target according to the determined score position and tempo. Control.

  For example, the musical score of the music is displayed on the display 13 and the current musical score position is clearly indicated by making the determined musical score position (note) color different from the color of other parts. Further, for example, a still image or a moving image corresponding to the estimated score position is displayed. For example, a file name of still image data representing a still image and a score position may be associated in advance, and the still image may be displayed on the display unit 13 using still image data corresponding to the estimated score position. Further, for example, a reproduction position (for example, a frame number) of moving image data representing a moving image is associated with a musical score position in advance, and a portion corresponding to the estimated musical score position of the moving image is displayed on the display unit 13. Good. Further, for example, a reproduction position (for example, a measure number) of accompaniment data representing accompaniment is associated with a musical score position in advance, and a portion of data corresponding to the estimated musical score position is transmitted to the sound source circuit of the sound system 16. Accompanied musical sounds may be emitted. When reproducing the accompaniment, the accompaniment tempo may be set to the determined tempo.

Next, the CPU 12a updates the state transition probability of the hidden Markov model HMM in step S166 using the calculated forward variable α _{i, n, T} (t). Specifically, first, an expected value <T> of the current frame number T is calculated using a forward variable of the semi-Markov model HSMM. In the hidden semi-Markov model HSMM, when the value of the number of frames T is an average value <T>, the number of sections that transition per frame is expressed as “1 / <T>”. Further, as described above, “τ ^(HMM) ” can be regarded as the number of sections of the hidden Markov model HMM that changes per frame. Therefore, if the ratio between the section length of the hidden semi-Markov model HSMM and the section length of the hidden Markov model HMM is “φ” (= section length of the hidden semi-Markov model HSMM / section length of the hidden Markov model HMM), “τ ^{( HMM)} = φ / <T> ”. Since “τ ^(HMM) ” is defined as a value not less than “0” and not more than “1”, “τ ^(HMM) ” is updated based on the following equation (19).

  Then, in step S167, the CPU 12a ends the score position determination process, and proceeds to step S17 of the alignment calculation process.

  Next, in step S17, the CPU 12a sets the processing target frame as the next frame. That is, the value of “t” that is the index of the frame is incremented. Thereafter, the CPU 12a repeatedly executes steps S15 to S17. However, when the user uses the input operator 11 to instruct the end of the score alignment process, the CPU 12a stops the operation to be controlled and ends the score alignment process.

In the score alignment apparatus 10 configured as described above, first, a score position probability density and a tempo probability density are calculated. Then, the score position and the tempo are determined based on the higher-order autoregressive process using the sequence of the score position probability density U _q (t) and the sequence of the tempo probability density V _T (t). According to this, it is possible to express the property of the music acoustic signal that the temporal differentiation of the tempo (that is, the acceleration of the musical score position) is continuous and shows a tendency to return to “0”. In addition, in a section where the variance of the score position probability density U _q (t) and the tempo probability density V _T (t) calculated using the hidden semi-Markov model HSMM is large, the score position probability density series and the tempo probability density series The behavior of smoothing the path is obtained. Thereby, the estimation accuracy of the score position can be improved. Therefore, if the score alignment apparatus 10 is applied to a media player, it is possible to suppress the shift between the performance position of the performance by the performer and the playback position of other media (automatic accompaniment, images, etc.). That is, it is possible to suppress the reproduction of other media in response to the performance by the performer from being unnatural.

Further, a forward variable is calculated by applying a forward algorithm to a normal hidden Markov model HMM, and a state (section) in which the forward variable is maximized is searched. Then, in the hidden semi-Markov model HSMM, the score position probability density U _q (t) and the tempo probability density V are applied by applying a forward algorithm to a plurality of sections ΔS including a section corresponding to the searched state (section). _T (t) is calculated. Therefore, an increase in the amount of calculation can be suppressed as compared with the case where the forward algorithm is applied to all the sections constituting the hidden semi-Markov model HSMM.

  Further, the musical score was divided so that the number of sections of the hidden Markov model HMM was larger than the number of sections of the hidden semi-Markov model HSMM. Accordingly, a plurality of sections to which the forward algorithm is applied among the sections of the hidden semi-Markov model HSMM are more appropriate as compared with the case where the number of sections of the hidden Markov model HMM and the section function of the hidden semi-Markov model HSMM are the same. Can be searched.

  Furthermore, in carrying out the present invention, the present invention is not limited to the above embodiment, and various modifications can be made without departing from the object of the present invention.

  For example, in the above embodiment, it is assumed that the score does not include da-capo, repeated symbols, etc., but if the score includes da-capo, repeated symbols, etc. What is necessary is just to set a transition probability appropriately. For example, when a repetition symbol is included, the probability of transitioning from the end section of the repeat section to the start of the repeat section is 50%, and one end of the end section from the end section of the repeat section. The probability of transition to this section may be 50%.

Further, for example, a step of determining whether or not there is silence when the acoustic signal data is read in step S151 may be added. If it is silence, it may be updated only based on a model of the score position x _t to the state space model. That is, an arithmetic expression of x _t = x _t−1 + v _t−1 + a _t−1 / 2 may be used. In this case, with respect to the forward variable Hidden Markov Models HMM and hidden semi Markov model HSMM, it sets only variable portion corresponding to the score position x _t a uniform distribution, by setting the rest to "0" Good.

In addition, when fermata exists in the score, the hidden semi-Markov model HSMM may be set so as to allow self-transition in the section where the fermata is written. That is, when fermata exists in the interval i, the probability τ _{i, i} may be set to “ρ” and the probability τ _{i, j} may be set to “1-ρ”. In this case, the number of times of self-transition in the section i may be counted, and the length of the performance remaining in the section i may be evaluated according to the count result. For example, the length remaining in the section i may be determined in three stages of “too short”, “normal”, and “too long”, and the determination result may be output as performance evaluation information.

  Moreover, you may evaluate the performance speed of a music using the estimated tempo and its dispersion | distribution. For example, the performance speed of the music may be determined in three stages, “too slow”, “normal”, and “too fast”, and the determination result may be output as performance evaluation information.

10 ... score alignment apparatus, HMM ··· hidden Markov model, HSMM ··· hidden semi-Markov _{model, x} t ··· score _{position, v} t ··· _{tempo, a} t ··· acceleration

Claims (4)

A score alignment device that estimates in real time a musical score position and a tempo representing a currently played portion of the musical score of the musical piece by analyzing the captured acoustic signal while capturing an acoustic signal representative of the musical performance sound There,
Probability model expressed as a sequence of states that represent each musical score position, with the probability that the current state depends on the immediately preceding state and the property that it can transition to any state from the current state A score position probability density / tempo probability density calculation means for calculating the score position probability density and the tempo probability density based on the model;
Based on the autoregressive process expressed using the true score position, the transition speed of the true score position, and the transition acceleration of the true score position, using the calculated probability density sequence of the score position, A score alignment device comprising: a score position / tempo determination means for determining a score position and a tempo. The score alignment apparatus according to claim 1,
The state depends on playing from the beginning of the section including the current score position to the current score position among the plurality of sections obtained by dividing the score and including the current score position. And the time taken to play the entire section,
The stochastic model is a hidden semi-Markov model represented as a sequence of the states;
The score position probability density / tempo probability density calculation means calculates a score position probability density and a tempo probability density by applying a forward algorithm to the hidden semi-Markov model. The score alignment apparatus according to claim 2,
The musical score position probability density / tempo probability density calculation means includes:
Of the plurality of sections obtained by dividing the score, the section including the current score position, the time taken to play from the beginning of the section including the current score position to the current score position, Based on a hidden Markov model represented as a sequence of states specified using an average time taken to play the entire section, a plurality of forward-looking algorithms are applied among the sections of the hidden semi-Markov model. A score alignment apparatus comprising section search means for searching for a section. A score alignment device that estimates in real time a musical score position and a tempo representing a currently played portion of the musical score of the musical composition by analyzing the captured acoustic signal while capturing an acoustic signal representing a musical performance sound Computer
A probabilistic model expressed as a sequence of states that represent each musical score position, with the property that the current state depends on the immediately preceding state and the property that it can transition from the current state to any other state A score position probability density / tempo probability density calculation step of calculating a score density and a tempo probability density based on the probability model;
Based on the autoregressive process expressed using the true score position, the transition speed of the true score position, and the transition acceleration of the true score position, using the calculated probability density sequence of the score position, A computer program for executing a musical score position / tempo determination step for determining a musical score position and tempo.

Images