US10665216B2 - Control method and controller - Google Patents

Control method and controller Download PDF

Info

Publication number
US10665216B2
US10665216B2 US16/247,717 US201916247717A US10665216B2 US 10665216 B2 US10665216 B2 US 10665216B2 US 201916247717 A US201916247717 A US 201916247717A US 10665216 B2 US10665216 B2 US 10665216B2
Authority
US
United States
Prior art keywords
event
sound
musical instrument
closely
performance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US16/247,717
Other languages
English (en)
Other versions
US20190147837A1 (en
Inventor
Akira MAEZAWA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yamaha Corp
Original Assignee
Yamaha Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yamaha Corp filed Critical Yamaha Corp
Assigned to YAMAHA CORPORATION reassignment YAMAHA CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Maezawa, Akira
Publication of US20190147837A1 publication Critical patent/US20190147837A1/en
Application granted granted Critical
Publication of US10665216B2 publication Critical patent/US10665216B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10G—REPRESENTATION OF MUSIC; RECORDING MUSIC IN NOTATION FORM; ACCESSORIES FOR MUSIC OR MUSICAL INSTRUMENTS NOT OTHERWISE PROVIDED FOR, e.g. SUPPORTS
    • G10G3/00—Recording music in notation form, e.g. recording the mechanical operation of a musical instrument
    • G10G3/04—Recording music in notation form, e.g. recording the mechanical operation of a musical instrument using electrical means
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/061—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for extraction of musical phrases, isolation of musically relevant segments, e.g. musical thumbnail generation, or for temporal structure analysis of a musical piece, e.g. determination of the movement sequence of a musical work
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/066—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental

Definitions

  • the present disclosure relates to a control method and to a controller.
  • Patent Document 1 There is known in the art a technique by which in a music score a position being played by a player is estimated based on sound signals indicative of a sound output in the performance (e.g., Patent Document 1).
  • Patent Document 1 Japanese Patent Application Laid-Open Publication No. 2015-79183
  • processing is executed to predict, for example, a timing of an event in which the automated musical instrument outputs a subsequent sound based on a result of estimating a position in the music score currently being played by the player.
  • a control method in accordance with some embodiments includes: detecting a first sound signal from a musical instrument through a sensor arranged by the musical instrument; receiving a result of detection related to a first event, in which the first sound signal is output by the musical instrument played by a musician, in a music performance; determining a tracking coefficient that indicates how closely a second event, in which a second sound signal is output by an automated musical instrument, follows the first event in the music performance; determining an operation mode, for outputting music from the automated musical instrument, of the second event based on the tracking coefficient; and outputting the music from the automated musical instrument based on the operation mode.
  • a controller in accordance with some embodiments includes: a sensor arranged by a musical instrument that detects a first sound signal from the musical instrument; a receiver configured to receive a result of detection related to a first event, in which the first sound signal is output by the musical instrument played by a musician, in a music performance; a coefficient determiner configured to determine a tracking coefficient that indicates how closely a second event, in which a second sound signal is output by an automated musical instrument, follows the first event in the music performance; and an operation determiner configured to determine an operation mode, for outputting music from the automated musical instrument, of the second event based on the tracking coefficient; wherein the music is output by the automated musical instrument based on the operation mode.
  • FIG. 1 is a block diagram of an ensemble system 1 in accordance with one or more embodiment.
  • FIG. 2 is a functional block diagram of a controller 10 .
  • FIG. 3 is a block diagram of the controller 10 .
  • FIG. 4 is a sequence diagram of the controller 10 .
  • FIG. 5 is an explanatory diagram illustrating sound output positions u[n] and observation noises q[n].
  • FIG. 6 is a flowchart for determining a coupling coefficient ⁇ according to Modification 5 or more embodiments.
  • FIG. 7 is a flowchart of the controller 10 .
  • FIG. 1 is a block diagram showing a configuration of an ensemble system 1 in accordance with one or more embodiments.
  • the ensemble system 1 is a system that enables a human player P and an automated musical instrument 30 to play music together.
  • the automated musical instrument 30 plays music in tune with music played by the player P.
  • the ensemble system 1 includes a controller 10 , a group of sensors 20 , and the automated musical instrument 30 .
  • the controller 10 stores music data indicative of a music score of the music piece that the player P and the automated musical instrument 30 play together.
  • the player P plays a musical instrument.
  • the group of sensors 20 detects information related to a music performance of the player P.
  • the group of sensors 20 includes, for example, a microphone placed in front of the player P.
  • the microphone collects sound output from the musical instrument played by the player P, and converts the collected sound into a sound signal and outputs the sound signal.
  • the controller 10 controls timings with which the automated musical instrument 30 plays music in a manner in which the music played by the automated musical instrument 30 follows music played by the player P. Based on sound signals provided by the group of sensors 20 , the controller 10 executes the following three processes: (1) estimation of a position in a music score that is currently being played (hereafter, may be referred to as “estimation of a performance position”); (2) prediction of a time (timing) at which a subsequent sound is to be output by the automated musical instrument 30 playing music (hereafter, may be referred to as “prediction of a sound output time”); and (3) output of a performance instruction to the automated musical instrument 30 (hereafter, may be referred to as “output of a performance instruction”).
  • the estimation of a performance position is a process for estimating a position in the music score, that is being played together by the player P and the automated musical instrument 30 .
  • the prediction of a sound output time is a process for predicting a time at which the automated musical instrument 30 outputs a subsequent sound, based on a result of the estimation of the performance position.
  • the output of a performance instruction is a process for outputting a performance instruction directed to the automated musical instrument 30 in accordance with the predicted sound output time.
  • Output of sound by the player P during his/her performance is an example of a “first event”
  • output of sound by the automated musical instrument 30 during its performance is an example of a “second event”.
  • the first event and the second event may generally be referred to as “events”.
  • the automated musical instrument 30 is a musical instrument that is capable of playing music without human operation, in accordance with performance instructions provided by the controller 10 .
  • An example of the automated musical instrument 30 is a self-playing piano.
  • FIG. 2 is a block diagram showing an exemplary functional configuration of the controller 10 .
  • the controller 10 includes a storage medium 11 , an estimator 12 , a predictor 13 , an outputter 14 , and a display 15 .
  • the storage medium 11 stores various data.
  • the storage medium 11 stores music data.
  • the music data includes at least information that indicates output timings and pitches of sound specified in the music score.
  • the output timings of sound indicated by the music data are expressed on the basis of, for example, a time unit (e.g., a thirty-second note) set for the music score.
  • the music data may also include information indicating at least one of length, tone, or volume of sound specified in the music score.
  • music data may be data in a MIDI (Musical Instrument Digital Interface) format.
  • the estimator 12 analyzes a sound signal input thereto, and estimates a position that is currently being played in the music score.
  • the estimator 12 first extracts information of an onset time (sound output start time) and of pitches from the sound signal. Next, the estimator 12 calculates from the extracted information a probabilistic estimated value indicative of a position currently being played in the music score.
  • the estimator 12 outputs the estimated value obtained by the calculation.
  • estimated values output by the estimator 12 include a sound output position u, an observation noise q, and a sound output time T.
  • the sound output position u is a position (e.g., the second beat in the fifth bar) in the music score, that corresponds to sound output during performance of either the player P or the automated musical instrument 30 .
  • the observation noise q is an observation noise (probabilistic fluctuation) of the sound output position u.
  • the sound output position u and the observation noise q are expressed, for example, on the basis of the time unit set for the music score.
  • the sound output time T is a time (a position along a time axis) at which output of sound is observed during performance of the player P.
  • u[n] a natural number satisfying n ⁇ 1).
  • Other estimated values are similarly expressed.
  • the predictor 13 by using the estimated values provided by the estimator 12 as observation values, predicts a time at which a subsequent sound is to be output during the performance of the automated musical instrument 30 (prediction of a sound output time is executed).
  • prediction of a sound output time is executed.
  • an exemplary case is assumed in which the predictor 13 executes prediction of a sound output time using a so-called Kalman filter.
  • prediction of a sound output time according to related arts is described, before description is given of prediction of a sound output time according to this embodiment. More specifically, as a prediction of a sound output time according to the related arts, prediction of a sound output time using a regression model, and prediction of a sound output time using a dynamic model, are described.
  • the regression model is a model that estimates a subsequent sound output time by using histories of sound output times for the player P and the automated musical instrument 30 .
  • the regression model is expressed by the following equation (1), for example.
  • a sound output time S[n] is a sound output time for the automated musical instrument 30 .
  • a sound output position u[n] is a sound output position for the player P.
  • prediction of a sound output time is executed using “j+1” observation values (j is a natural number satisfying 1 ⁇ j ⁇ n).
  • j is a natural number satisfying 1 ⁇ j ⁇ n.
  • the matrix G n and the matrix H n are matrices corresponding to regression coefficients.
  • the subscript n shown in the matrix G n , the matrix H n , and the coefficient a n indicates that the matrix G n , the matrix H n , and the coefficient a n are elements that correspond to a note that is the n-th to have been played.
  • the matrix G n , the matrix H n , and the coefficient a n can be set to correspond one-to-one with notes included in the music score of a music piece.
  • the matrix G n , the matrix H n , and the coefficient a n can be set in accordance with the position in the music score.
  • the regression model shown in equation (1) enables prediction of a sound output time S based on the position in the music score.
  • the dynamic model generally updates, by the following exemplary process, a state vector V that indicates a state of a dynamic system to be predicted by the dynamic model.
  • the dynamic model first predicts an after-change state vector V from a before-change state vector V by use of a state transition model, which is a theoretical model that expresses changes in the dynamic system over time. Secondly, by use of an observation model, the dynamic model predicts an observation value from the value of the state vector V predicted by the state transition model, where the observation model is a theoretical model that expresses a relationship between the state vector V and the observation value. Thirdly, the dynamic model calculates an observation residual based on the observation value predicted by the observation model and an actual observation value provided from outside the dynamic model. Fourthly, by correcting the value of the state vector V predicted by the state transition model using the observation residual, the dynamic model calculates an updated value of the state vector V. In this way, the dynamic model updates the state vector V.
  • a state transition model which is a theoretical model that expresses changes in the dynamic system over time.
  • the dynamic model predicts an observation value from the value of the state vector V predicted by the state transition model, where the observation model is a theoretical model that express
  • the state vector V includes a performance position x and a speed v as its elements.
  • the performance position x is a state variable that represents an estimated value of a position in the music score that is being played by the player P or the automated musical instrument 30 .
  • the speed v is a state variable that represents an estimated value of a speed (tempo) of the performance of the player P or the automated musical instrument 30 in the music score.
  • the state vector V may include state variables other than the performance position x and the speed v.
  • the state vector V[n] is a k-dimensional vector having as its elements multiple state variables including the performance position x[n] and the speed v[n] corresponding to a note that is the n-th to have been played (k is a natural number satisfying k ⁇ 2).
  • the process noise e[n] is a k-dimensional vector that represents noise included in a state transition in the state transition model.
  • the matrix A n is a matrix that shows coefficients related to updating of the state vector V in the state transition model.
  • the matrix O n is a matrix that shows a relationship between the observation value (the sound output position u in this example) and the state vector V in the observation model.
  • the subscript n appended to each of the elements such as matrices and variables indicates that the subject element corresponds to the n-th note.
  • Equations (2) and (3) can be made more specific as in the following equations (4) and (5), for example.
  • the performance position x[t] at a future time t can be obtained by the following equation (6).
  • x [ t ] x [ n ]+ v [ n ]( t ⁇ T [ n ]) (6)
  • a demand may arise whereby a degree of synchronization between the performance of the player P and the performance of the automated musical instrument 30 be made adjustable.
  • a demand may arise whereby how closely the performance of the automated musical instrument 30 follows the performance of the player P be made adjustable.
  • a degree of synchronization is adjusted by use of variables such as a process noise e[n].
  • the sound output time S[n+1] is calculated based on observation values related to output of sound by the player P, such as a sound output time T[n]
  • the predictor 13 While still based on the dynamic model according to related arts, the predictor 13 according to the present embodiment predicts the sound output time S[n+1] by a configuration in which how closely the performance of the automated musical instrument 30 follows the performance of the player P can be adjusted more flexibly as compared to the related arts.
  • exemplary processing performed by the predictor 13 of the present embodiment will be described.
  • the predictor 13 in this embodiment updates a state vector (referred to as “state vector Vu”) representative of a state of a dynamic system that is related to the performance of the player P, and a state vector (referred to as “state vector Va”) representative of a state of a dynamic system that is related to the performance of the automated musical instrument 30 .
  • the state vector Vu is a vector that includes as its elements: a performance position xu that is a state variable for representing an estimated position in the music score that is currently being played by the player P; and a speed vu that is a state variable for representing an estimated value of a speed in the music score at which the player P is playing.
  • the state vector Va is a variable that includes as its elements: a performance position xa that is a state variable for representing an estimated value of a position in the music score that is currently being played by the automated musical instrument 30 ; and a speed va that is a state variable for representing an estimated value of a speed in the music score at which the automated musical instrument 30 is playing.
  • the state variables (the performance position xu and the speed vu) included in the state vector Vu are referred to in general as “first state variables”, while the state variables (the performance position xa and the speed va) included in the state vector Va are referred to in general as “second state variables”.
  • the predictor 13 updates the first state variables and the second state variables using a state transition model as shown, for example, in the following equations (8) to (11).
  • the first state variables are updated by equations (8) and (11).
  • Equations (8) and (11) are equations produced by making equation (4) more specific.
  • the second state variables are updated by equations (9) and (10) below, instead of by equation (4) described above.
  • xu [ n ] xu [ n ⁇ 1]+( T [ n ] ⁇ T [ n ⁇ 1]) vu [ n ⁇ 1]+ exu [ n ] (8)
  • xa [ n ] ⁇ [ n ] ⁇ xa [ n ⁇ 1]+( T [ n ] ⁇ T [ n ⁇ 1]) va [ n ⁇ 1]+ exa [ n ] ⁇ +(1 ⁇ [ n ]) ⁇ xu [ n ⁇ 1]+( T [ n ] ⁇ T [ n ⁇ 1]) vu [ n ⁇ 1]+ exu [ n ] ⁇ (9)
  • va [ n ] va [ n ⁇ 1]+ eva [ n ] (10)
  • vu [ n ] vu [ n ⁇ 1]+ evu [ n ] (11)
  • the process noise exu[n] is representative of noise that is generated when the performance position xu[n] is updated by the state transition model.
  • the process noise exa[n] is representative of noise that is generated when the performance position xa[n] is updated by the state transition model.
  • the process noise eva[n] is representative of noise that is generated when the speed va[n] is updated by the state transition model.
  • the process noise evu[n] is representative of noise that is generated when the speed vu[n] is updated by the state transition model.
  • the coupling coefficient ⁇ [n] is a real number that satisfies 0 ⁇ [n] ⁇ 1. In equation (9), the value “1 ⁇ [n]” by which the performance position xu, which is a first state variable, is multiplied is an example of a “tracking coefficient”.
  • the predictor 13 predicts the performance position xu[n] and the speed vu[n], which are first state variables, using the performance position xu[n ⁇ 1] and the speed vu[n ⁇ 1], which are first state variables.
  • the predictor 13 predicts the performance position xa[n] and the speed va[n], which are second state variables, using: the performance position xu[n ⁇ 1] and the speed vu[n ⁇ 1], which are first state variables; or the performance position xa[n ⁇ 1] and the speed va[n ⁇ 1], which are second state variables; or the performance position xu[n ⁇ 1], the speed vu[n ⁇ 1], the performance position xa[n ⁇ 1], and the speed va[n ⁇ 1].
  • the predictor 13 In updating the performance position xu[n] and the speed vu[n], which are first state variables, the predictor 13 according to this embodiment uses the state transition model shown in equations (8) and (11) and the observation model shown in equation (5). In updating the performance position xa[n] and the speed va[n], which are second state variables, the predictor 13 according to this embodiment uses the state transition model shown in equations (9) and (10) and does not use the observation model.
  • the predictor 13 predicts the performance position xa[n], which is a second state variable, based on a product resulting from multiplying a first state variable (e.g., performance position xu[n ⁇ 1]) by the tracking coefficient (1 ⁇ [n]), and based on a product resulting from multiplying a second state variable (e.g., performance position xa[n ⁇ 1]) by the coupling coefficient ⁇ [n].
  • a first state variable e.g., performance position xu[n ⁇ 1]
  • a second state variable e.g., performance position xa[n ⁇ 1]
  • the predictor 13 of this embodiment can adjust the degree of synchronization between the performance of the automated musical instrument 30 and the performance of the player P.
  • a property of the performance of the automated musical instrument 30 to follow the performance of the player P can be strengthened.
  • the predictor 13 includes a receiver 131 , a coefficient determiner 132 , a state variable updater 133 , and a predicted time calculator 134 .
  • the receiver 131 receives input of observation values related to timings of music performance.
  • the observation values related to timings of music performance include first observation values related to timings of music performance of the player P.
  • the observation values related to timings of music performance may include second observation values related to timings of music performance of the automated musical instrument 30 .
  • first observation values are a generic term for a sound output time T and a sound output position u (hereafter referred to as “sound output position uu”) related to the performance of the player P.
  • Second observation values are a generic term for a sound output time S and a sound output position u (hereafter referred to as “sound output position ua”) related to the performance of the automated musical instrument 30 .
  • the receiver 131 receives input of observation values accompanying the observation values related to timings of music performance.
  • the accompanying observation values represent observation noise q related to the performance of the player P.
  • the receiver 131 stores the received observation values in the storage medium 11 .
  • the coefficient determiner 132 determines the value of the coupling coefficient ⁇ .
  • the value of the coupling coefficient ⁇ is set in advance in accordance with a position that is in the music score and is being played, for example.
  • the storage medium 11 of this embodiment stores profile information in which a position that is being played in the music score and a value of the coupling coefficient ⁇ corresponding to the position are associated to each other.
  • the coefficient determiner 132 refers to the profile information stored in the storage medium 11 and retrieves the value of the coupling coefficient ⁇ corresponding to the position that is being played in the music score. The coefficient determiner 132 then sets the value retrieved from the profile information as the value of the coupling coefficient ⁇ .
  • the coefficient determiner 132 may determine the value of the coupling coefficient ⁇ to be a value corresponding to an instruction by an operator (an example of “user”) of the controller 10 , for example.
  • the controller 10 includes a UI (User Interface) for receiving operations that indicate instructions from the operator.
  • the UI may be software UI (UI through a screen displayed by software) or hardware UI (e.g., a fader).
  • the operator and the player P are different persons; however, the player P may act as the operator.
  • the state variable updater 133 updates state variables (first state variables and second state variables). Specifically, the state variable updater 133 according to this embodiment updates state variables using above mentioned equation (5) and equations (8) to (11). More specifically, the state variable updater 133 according to this embodiment updates first state variables using equations (5), (8), and (11), and updates second state variables using equations (9) and (10). The state variable updater 133 then outputs the updated state variables.
  • the state variable updater 133 updates second state variables based on the coupling coefficient ⁇ that has a value determined by the coefficient determiner 132 .
  • the state variable updater 133 updates second state variables based on the tracking coefficient (1 ⁇ [n]).
  • the controller 10 adjusts a mode of outputting sound by the automated musical instrument 30 in music performance based on the tracking coefficient (1 ⁇ [n]).
  • the predicted time calculator 134 calculates the sound output time S[n+1], which is a time at which sound is output next by the automated musical instrument 30 .
  • the predicted time calculator 134 calculates the performance position x[n] at a future time t by applying the state variables updated by the state variable updater 133 to equation (6). More specifically, the predicted time calculator 134 calculates the performance position x[n+1] at the future time t by applying to equation (6) the performance position xa[n] and the speed va[n], both of which have been updated by the state variable updater 133 . Next, using equation (7), the predicted time calculator 134 calculates the sound output time S[n+1] at which the automated musical instrument 30 is to output sound corresponding to the (n+1)th note.
  • the predicted time calculator 134 may calculate the performance position x[n] at the future time t by applying to equation (6) the performance position xu[n] and the speed vu[n], which have been updated by the state variable updater 133 .
  • the outputter 14 outputs toward the automated musical instrument 30 a performance instruction corresponding to a note whose sound is to be output by the automated musical instrument 30 next, in accordance with the sound output time S[n+1] input from the predictor 13 .
  • the controller 10 includes an internal clock (illustration omitted) and measures the time.
  • the performance instruction is written according to a predetermined data format.
  • An example of the predetermined data format is MIDI.
  • the performance instruction includes, for example, a note-on message, a note number, and a velocity.
  • the display 15 displays information related to a result of the estimation of a performance position and information related to a result of the prediction of time at which a subsequent sound is output by the automated musical instrument 30 .
  • the information related to the estimation of a performance position includes, for example, at least one of the music score, a frequency spectrogram of the input sound signal, or a probability distribution of the estimated value of the performance position.
  • the information related to the result of the prediction of time at which a subsequent sound is output includes, for example, state variables. From the display 15 displaying the information related to the result of the estimation of the performance position and the information related to the result of the prediction of the sound output time, the operator (user) of the controller 10 is able to understand operating statuses of the ensemble system 1 .
  • FIG. 3 is a diagram showing an exemplary hardware configuration of the controller 10 .
  • the controller 10 is a computer device that includes a processor 101 , a memory 102 , a storage 103 , an input-output IF 104 , and a display device 105 .
  • the processor 101 is a CPU (central processing unit), for example, and controls each element of the controller 10 .
  • the processor 101 may be configured to include a programmable logic device, such as a DSP (digital signal processor) or an FPGA (field programmable gate array), instead of or in addition to the CPU.
  • the processor 101 may include multiple CPUs (or multiple programmable logic devices).
  • the memory 102 is a non-transient recording medium and is a volatile memory, such as a RAM (random access memory). The memory 102 serves as a work area when the processor 101 executes a control program described later.
  • the storage 103 is a non-transient recording medium and is a non-volatile memory, such as an EEPROM (electrically erasable programmable read-only memory).
  • the storage 103 stores various programs, such as the control program for controlling the controller 10 , and various data.
  • the input-output IF 104 is an interface for receiving input signals from or outputting signals to other devices. Functions carried out by the input-output IF 104 include microphone input and MIDI output, for example.
  • the display device 105 is a device that outputs various information, and includes an LCD (liquid crystal display), for example.
  • the processor 101 serves as the estimator 12 , the predictor 13 , and the outputter 14 by executing the control program stored in the storage 103 and operating in accordance with the control program.
  • Either one or both of the memory 102 and the storage 103 provide functions as the storage medium 11 .
  • the display device 105 provides functions as the display 15 .
  • FIG. 4 is a sequence diagram showing exemplary operations of the controller 10 .
  • the sequence diagram shown in FIG. 4 is initiated, for example, upon the control program being activated by the processor 101 .
  • the estimator 12 receives input of a sound signal.
  • the sound signal is an analog signal
  • the sound signal is converted into a digital signal by a DA convertor (illustration omitted) provided in the controller 10 , for example, and the sound signal that has been converted into the digital signal is input to the estimator 12 .
  • the estimator 12 analyzes the sound signal to estimate a position that is currently played in the music score.
  • Processing related to step S 2 is executed, for example, as follows.
  • the transition of the performance position in the music score (music-score time series) is described using a probability model.
  • the probability model in describing the music-score time series, it is possible to address problems, such as errors in the performance, omission of repetition in the performance, fluctuations in tempo during the performance, or uncertainties in pitches or sound output times during the performance.
  • the Hidden Semi-Markov Model HSMM
  • HSMM Hidden Semi-Markov Model
  • the estimator 12 obtains a frequency spectrogram by dividing the sound signal into frames and performing the constant-Q transform on the frames, for example.
  • the estimator 12 extracts onset times and pitches from the frequency spectrogram.
  • the estimator 12 sequentially estimates, with a delayed-decision technique, distributions of a probabilistic estimated value indicating the position currently being played in the music score, and when the peak of each distribution passes a location that is considered as an onset in the music score, the estimator 12 outputs a Laplace approximation of the distribution and outputs one or more statistics.
  • the estimator 12 outputs: the sound output time T[n] at which output of the sound is detected; and the mean and variance of the position in the music score in the distribution indicating probabilistic positions of the output sound in the music score.
  • the mean position in the music score is an estimated value of the sound output position u[n]
  • the variance is an estimated value of the observation noise q[n]. Details of sound output position estimation is described in Japanese Patent Application Laid-Open Publication No. 2015-79183, for example.
  • FIG. 5 is an explanatory diagram showing exemplary sound output positions u[n] and observation noises q[n].
  • the estimator 12 calculates probability distributions P[ 1 ] to P[ 4 ] that correspond one-to-one with four sound outputs corresponding to the four notes included in the bar.
  • the estimator 12 then outputs the sound output times T[n], the sound output positions u[n], and the observation noises q[n] based on the result of the calculation.
  • step S 3 the predictor 13 performs prediction of a time at which a subsequent sound is output by the automated musical instrument 30 using the estimated values provided by the estimator 12 as the observation values.
  • the estimator 12 performs prediction of a time at which a subsequent sound is output by the automated musical instrument 30 using the estimated values provided by the estimator 12 as the observation values.
  • step S 3 the receiver 131 receives input of observation values (first observation values), such as the sound output position uu, the sound output time T, and the observation noise q, provided by the estimator 12 (step S 31 ).
  • the receiver 131 stores these observation values in the storage medium 11 .
  • step S 3 the coefficient determiner 132 determines the value of the coupling coefficient ⁇ used in updating the state variables (step S 32 ). More specifically, referring to the profile information stored in the storage medium 11 , the coefficient determiner 132 acquires the value of the coupling coefficient ⁇ corresponding to the current performance position in the music score, and sets the acquired value to the coupling coefficient ⁇ . In this way, it is possible to adjust, in accordance with the position currently being played in the music score, a degree of synchronization between the performance of the player P and the performance of the automated musical instrument 30 .
  • the controller 10 in the present embodiment enables the automated musical instrument 30 to execute, in some parts of the music piece, an automated performance that follows the performance of the player P, and in other parts of the music piece, an automated performance that is independent of the performance of the player P.
  • a performance by the automated musical instrument 30 can be made to sound less mechanical (can be made to sound relatively natural).
  • the controller 10 of this embodiment may cause the automated musical instrument 30 to execute an automated performance at a tempo that follows the tempo of the performance of the player P more closely than the tempo of the performance predetermined by the music data.
  • the controller 10 of this embodiment may cause the automated musical instrument 30 to execute an automated performance at a tempo that follows the tempo of the performance predetermined by the music data more closely than the tempo of the performance of the player P.
  • step S 3 the state variable updater 133 updates the state variables using the observation values input thereto (step S 33 ).
  • the state variable updater 133 updates first state variables by use of equations (5), (8), and (11), and updates second state variables using equations (9) and (10). Additionally, in step S 33 , the state variable updater 133 updates the second state variables based on the tracking coefficient (1 ⁇ [n]) as shown in equation (9).
  • step S 3 the state variable updater 133 outputs the state variables updated in step S 33 toward the predicted time calculator 134 (step S 34 ). More specifically, in step S 34 , the state variable updater 133 of this embodiment outputs the performance position xa[n] and the speed va[n] updated in step S 33 to the predicted time calculator 134 .
  • step S 3 the predicted time calculator 134 applies the state variables input from the state variable updater 133 to equations (6) and (7), and calculates the sound output time S[n+1] at which sound of the (n+1)th note is to be output (step S 35 ). More specifically, in step S 35 , the predicted time calculator 134 calculates the sound output time S[n+1] based on the performance position xa[n] and the speed va[n] input from the state variable updater 133 . The predicted time calculator 134 then outputs the sound output time S[n+1] obtained by the calculation to the outputter 14 .
  • the outputter 14 Upon arrival of the sound output time S[n+1] input by the predictor 13 , the outputter 14 outputs to the automated musical instrument 30 a performance instruction corresponding to the (n+1)th note whose sound is to be output next by the automated musical instrument 30 (step S 4 ).
  • the performance instruction needs to be output earlier than the sound output time S[n+1] predicted by the predictor 13 due to a delay in the processing by the outputter 14 and the automated musical instrument 30 , but description thereof will be omitted here.
  • the automated musical instrument 30 outputs sound in accordance with the performance instruction provided by the controller 10 (step S 5 ).
  • the predictor 13 determines whether performance has ended. More specifically, the predictor 13 determines completion of performance based on the performance position estimated by the estimator 12 , for example. In a case where the performance position has reached a predetermined end point, the predictor 13 determines that the performance has ended. In a case where the predictor 13 determines that the performance has ended, the controller 10 ends the processing shown in the sequence diagram in FIG. 4 . In a case where the predictor 13 determines that the performance has not ended, the controller 10 and the automated musical instrument 30 repeatedly execute the processing in steps S 1 to S 5 .
  • step S 1 the estimator 12 receives input of a sound signal.
  • step S 2 the estimator 12 estimates the position currently being played in the music score.
  • step S 31 the receiver 131 receives input of observation values provided by the estimator 12 .
  • step S 32 the coefficient determiner 132 determines the coupling coefficient ⁇ [n].
  • step S 33 the state variable updater 133 updates each state variable in the state vector V using the observation values received by the receiver 131 and the coupling coefficient ⁇ [n] determined by the coefficient determiner 132 .
  • step S 34 the state variable updater 133 outputs the state variables updated in step S 33 to the predicted time calculator 134 .
  • the predicted time calculator 134 calculates the sound output time S[n+1] using the updated state variables output from the state variable updater 133 .
  • step S 4 the outputter 14 outputs the performance instruction to the automated musical instrument 30 based on the sound output time S[n+1].
  • the device for which the timing is controlled by the controller 10 is not limited to the automated musical instrument 30 .
  • the “event” for which the predictor 13 predicts a timing is not limited to output of sound by the automated musical instrument 30 .
  • the device to be controlled may be, for example, a device that generates an image that changes with the performance of the player P synchronously (e.g., a device that generates computer graphics that change in a real-time manner), or a display device that causes images to change in synchronization with the performance of the player P (e.g., a projector or a direct-view-type display).
  • the device to be controlled may be a robot that makes movements, such as dancing, in synchronization with the performance of the player P.
  • the player P need not be a human being.
  • performance sound by another automated musical instrument different from the automated musical instrument 30 may be input to the controller 10 .
  • the number of players P and the number of automated musical instruments 30 are not limited to the examples shown in the embodiment.
  • the ensemble system 1 may include: two or more players P; two or more automated musical instruments 30 ; or two or more players P and two or more automated musical instruments 30 .
  • the functional configuration of the controller 10 is not limited to the example shown in the embodiment. A part of the functional elements shown as examples in FIG. 2 may be omitted.
  • the controller 10 is not required to include the predicted time calculator 134 .
  • the controller 10 may simply output state variables updated by the state variable updater 133 .
  • a device to which the state variables updated by the state variable updater 133 are input the device being other than the controller 10 , may calculate the timing of the subsequent event (e.g., the sound output time S[n+1]).
  • a device other than the controller 10 may perform processing (e.g., display of images that visualize the state variables) other than the calculation of the timing of the subsequent event.
  • the controller 10 may not include the display 15 .
  • the coefficient determiner 132 determines the coupling coefficient ⁇ to have a value corresponding to the position currently being played in the music score. In an embodiment, the coefficient determiner 132 may determine the value of the coupling coefficient ⁇ to be a predetermined default value, a value corresponding to a result of analyzing the music score, or a value corresponding to an instruction from the user.
  • FIG. 6 is a flowchart that shows an exemplary method of determination of the coupling coefficient ⁇ by the coefficient determiner 132 according to Modification 5 or more embodiments. Each process in the flowchart is a process executed during the process in step S 32 shown in FIG. 4 .
  • step S 32 the coefficient determiner 132 sets the value of the coupling coefficient ⁇ [n] to a default value (step S 321 ).
  • the storage medium 11 stores the default value of the coupling coefficient ⁇ [n], wherein the default value does not depend on the music piece (or position that is currently being played in the music score).
  • the coefficient determiner 132 retrieves the default value of the coupling coefficient ⁇ [n] stored in the storage medium 11 , and sets the retrieved default value as the value of the coupling coefficient ⁇ [n].
  • step S 32 the coefficient determiner 132 analyzes the music score, and sets a value that is in accordance with the result of the analysis as the value of the coupling coefficient ⁇ [n] (step S 322 ).
  • step S 322 by analyzing the music score, the coefficient determiner 132 first calculates a ratio (hereafter referred to as “music-note density ratio”) of a density of notes indicating sound output by the player P to a density of notes indicating sound output by the automated musical instrument 30 .
  • the coefficient determiner 132 sets a value corresponding to the calculated music-note density ratio as the value of the coupling coefficient ⁇ [n]. In other words, the coefficient determiner 132 determines the tracking coefficient (1 ⁇ [n]) based on the music-note density ratio.
  • the coefficient determiner 132 sets the value of the coupling coefficient ⁇ [n] such that the value of the coupling coefficient ⁇ [n] is smaller in a case where the music-note density ratio is greater than a predetermined threshold value, than in a case where the music-note density ratio is equal to or smaller than the predetermined threshold value.
  • the coefficient determiner 132 sets the value of the coupling coefficient ⁇ [n] such that in a case where the music-note density ratio is greater than the predetermined threshold value, the value of the tracking coefficient (1 ⁇ [n]) is greater compared to a case where the music-note density ratio is equal to or smaller than the predetermined threshold value.
  • the coefficient determiner 132 sets the value of the coupling coefficient ⁇ [n] such that in a case where the music-note density ratio is greater than the predetermined threshold value, the performance of the automated musical instrument 30 follows the performance of the player P more closely as compared to a case where the music-note density ratio is equal to or smaller than the predetermined threshold value.
  • the coefficient determiner 132 may set the value of the coupling coefficient ⁇ [n] based on a density DA n of notes indicating sound output by the automated musical instrument 30 and a density DU n of notes indicating sound output by the player P, as shown in the following equation (12).
  • DA n may indicate a density of sound output by the automated musical instrument 30
  • DU n may indicate a density of sound output by the player P.
  • step S 32 the coefficient determiner 132 analyzes the music score, and determines whether the part played by the automated musical instrument 30 is a main melody part (step S 323 ).
  • a publicly known technique is used to determine whether the part played by the automated musical instrument 30 is a main melody part or not.
  • the coefficient determiner 132 advances the processing to step S 324 . In a case where it is determined that the part played by the automated musical instrument 30 is not the main melody part (S 323 : NO), the coefficient determiner 132 advances the processing to step S 325 .
  • step S 32 the coefficient determiner 132 updates the value of the coupling coefficient ⁇ [n] to a greater value (step S 324 ).
  • the coefficient determiner 132 updates the value of the coupling coefficient ⁇ [n] to a value that is greater than the value indicated in the right-hand side of equation (12). For example, the coefficient determiner 132 may calculate the updated coupling coefficient ⁇ [n] by adding a predetermined non-negative addition value to the value indicated in the right-hand side of equation (12). Alternatively, for example, the coefficient determiner 132 may calculate the updated coupling coefficient ⁇ [n] by multiplying the value indicated in the right-hand side of equation (12) by a predetermined coefficient greater than one. The coefficient determiner 132 may determine the updated coupling coefficient ⁇ [n] to be a value equal to or smaller than a predetermined upper limit.
  • step S 32 the coefficient determiner 132 updates the value of the coupling coefficient ⁇ [n] according to an instruction from the user at a rehearsal, etc. (step S 325 ).
  • the storage medium 11 stores instruction information that indicates content of instructions from the user at a rehearsal, etc.
  • the instruction information includes information that identifies a lead part in the performance.
  • the information identifying the lead part in the performance is, for example, information identifying whether the lead part in the performance is a part played by the player P or a part played by the automated musical instrument 30 .
  • the information identifying a lead part in the performance may be set in accordance with the position that is currently being played in the music score.
  • the instruction information may be information indicating that there are no instructions from the user.
  • step S 325 in a case where the instruction information is information indicating that the player P holds a lead position, the coefficient determiner 132 updates the value of the coupling coefficient ⁇ [n] to a smaller value. In a case where the instruction information is information indicating that the automated musical instrument 30 holds a lead position, the coefficient determiner 132 updates the value of the coupling coefficient ⁇ [n] to a greater value. In a case where the instruction information is information indicating that there are no instructions from the user, the coefficient determiner 132 does not update the value of the coupling coefficient ⁇ [n].
  • the instruction information is not limited to these examples. There may be more than three types of user instruction content that can be indicated by the instruction information.
  • content of the user's instructions indicated by the instruction information may be information that can indicate levels (e.g., high, middle, and low) indicating the strength of the lead position, and may specify one of the levels.
  • step S 32 the coefficient determiner 132 outputs the value of the coupling coefficient ⁇ [n] determined through the processing in steps S 321 to S 325 to the state variable updater 133 (step S 326 ).
  • the coupling coefficient ⁇ [n] there are shown four exemplary decision factors for determining the coupling coefficient ⁇ [n]: “the user's instruction (result of a rehearsal)”; “the part corresponding to a main melody”; “music-note density ratio”; and “default value”.
  • the order of priority for these four decision factors in determination of the coupling coefficient ⁇ [n] is: “the user's instruction”>“the part corresponding to a main melody”>“music-note density ratio”>“default value”.
  • the coefficient determiner 132 may use only a part of the four decision factors described above when determining the coupling coefficient ⁇ [n].
  • the processing by which the coefficient determiner 132 determines the coupling coefficient ⁇ [n] may include, from among processes of steps S 321 to S 326 shown in FIG. 6 : at least one of the process of step S 321 , the process of step S 322 , the processes of steps S 323 and S 324 , or the process of step S 325 ; and the process of step S 326 .
  • the order of priority for the decision factors in determination of the coupling coefficient ⁇ [n] is not limited to the example shown in FIG. 6 , and may be freely chosen.
  • “the part corresponding to a main melody” may have a higher priority than “the user's instruction”
  • “music-note density ratio” may have a higher priority than “the user's instruction”
  • “music-note density ratio” may have a higher priority than “the part corresponding to a main melody”.
  • the processes of steps S 321 to S 326 shown in FIG. 6 may be rearranged as appropriate in a different order
  • state variables are updated using observation values (the sound output position u[n] and the observation noise q[n]) at a single time point.
  • observation values the sound output position u[n] and the observation noise q[n]
  • the present disclosure is not limited to such a form, and state variables may be updated using observation values at multiple time points. More specifically, in the observation model used in the dynamic model, the following equation (13) may be used instead of equation (5).
  • the matrix O n is a matrix that shows relationships between observation values (sound output positions u[n ⁇ 1], u[n ⁇ 2], . . . , u[n ⁇ j] in this example), the performance position x[n], and the speed v[n], in the observation model.
  • observation values sound output positions u[n ⁇ 1], u[n ⁇ 2], . . . , u[n ⁇ j] in this example
  • the performance position x[n] the speed v[n] in the observation model.
  • state variables are updated using first observation values.
  • present disclosure is not limited to such a form, and the state variables may be updated using both first observation values and second observation values.
  • equation (14) may be used instead of equation (9).
  • equation (9) only the sound output time T, which is a first observation value, is used as the observation value; whereas in equation (14), the sound output time T, which is a first observation value, and the sound output time S, which is a second observation value, are used as the observation values.
  • xa [ n ] ⁇ [ n ] ⁇ xa [ n ⁇ 1]+( S [ n ] ⁇ S [ n ⁇ 1]) va [ n ⁇ 1]+ exa [ n ] ⁇ +(1 ⁇ [ n ]) ⁇ xu [ n ⁇ 1]+( T [ n ] ⁇ T [ n ⁇ 1]) vu [ n ⁇ 1]+ exu [ n ] ⁇ (14)
  • Equation 15 may be used instead of equation (8), and the following equation (16) may be used instead of equation (9).
  • the sound output time Z appearing in equations (15) and (16) is a generic term for the sound output time S and the sound output time T.
  • xu [ n ] xu [ n ⁇ 1]+( Z [ n ] ⁇ Z [ n ⁇ 1]) vu [ n ⁇ 1]+ exu [ n ]
  • xa [ n ] ⁇ [ n ] ⁇ xa [ n ⁇ 1]+( Z [ n ] ⁇ Z [ n ⁇ 1]) va [ n ⁇ 1]+ exa [ n ] ⁇ +(1 ⁇ [ n ]) ⁇ xu [ n ⁇ 1]+( Z [ n ] ⁇ Z [ n ⁇ 1]) vu [ n ⁇ 1]+ exu [ n ] ⁇
  • both first observation values and second observation values may be used in the observation model also. More specifically, in the observation model, state variables may be updated using equation (17), which is derived by making equation (5) of the embodiment more specific, and equation (18) below.
  • ua [ n ] xa [ n ]+ q [ n ] (18)
  • the state variable updater 133 may receive from the receiver 131 the first observation values (the sound output position uu and the sound output time T), and may receive from the predicted time calculator 134 the second observation values (the sound output position ua and the sound output time S).
  • the controller 10 controls the time (timing) at which sound is output by the automated musical instrument 30 .
  • the present disclosure is not limited to such a form, and the controller 10 may control a volume of sound output by the automated musical instrument 30 . That is, the form for sound outputting performed by the automated musical instrument 30 , which form is controlled by the controller 10 , may be the volume of sound output in the sound outputting performed by the automated musical instrument 30 .
  • the controller 10 may adjust how closely the volume of sound output during the performance of the automated musical instrument 30 follows the volume of sound output during the performance of the player P.
  • controller 10 may control both the time (timing) at which sound is output by the automated musical instrument 30 and the volume of sound output by the automated musical instrument 30 .
  • the predicted time calculator 134 calculates the performance position x[t] at a future time t using equation (6), but the present disclosure is not limited to such a form.
  • the state variable updater 133 may calculate the performance position x[n+1] using the dynamic model that updates state variables.
  • Behaviors of the player P detected by the group of sensors 20 are not limited to sound that is played.
  • the group of sensors 20 may detect movements of the player P, in addition to or instead of the played sound.
  • the group of sensors 20 includes a camera or a motion sensor.
  • the algorithm for estimating the performance position in the estimator 12 is not limited to the example shown in the embodiment. Any algorithm may be applied so long as the algorithm enables the estimator 12 to estimate the position that is currently being played in the music score based on the music score that is given in advance and a sound signal input from the group of sensors 20 .
  • the observation values input to the predictor 13 from the estimator 12 are not limited to the examples shown in the embodiment. Any observation values other than the sound output position u and the sound output time T may be input to the predictor 13 , so long as the observation values are related to the timing of the music performance.
  • the dynamic model used in the predictor 13 is not limited to the example shown in the embodiment.
  • the predictor 13 updates the state vector Va (a second state variable) without using the observation model.
  • the predictor 13 may update the state vector Va using both the state transition model and the observation model.
  • the predictor 13 updates the state vector Vu using the Kalman filter.
  • the predictor 13 may update the state vector V using an algorithm other than the Kalman filter.
  • the predictor 13 may update the state vector V using a particle filter.
  • the state transition model used in the particle filter may be equation (2), (4), (8), or (9) described above, or a state transition model different from these equations may be used.
  • the observation model used in the particle filter may be equation (3), (5), (10), or (11), or an observation model different from these equations may be used.
  • each device forming the ensemble system 1 is not limited to the example shown in the embodiment. Each device may have any specific hardware configuration so long as required functions can be realized.
  • the controller 10 may include multiple processors each corresponding to one of the estimator 12 , the predictor 13 , and the outputter 14 .
  • physically separate devices may cooperate with each other to serve as the controller 10 in the ensemble system 1 .
  • the control program executed by the processor 101 in the controller 10 may be provided in a format stored in a non-transitory recording medium, such as an optical disk, a magnetic disc, or a semiconductor memory, or may be provided by being downloaded via a communication line, such as the Internet.
  • the control program need not include all of the steps shown in FIG. 4 .
  • this program may include steps S 31 , S 33 , and S 34 alone.
  • a control method includes: receiving a result of detection related to a first event in a music performance; determining a tracking coefficient that indicates how closely a second event follows the first event in the music performance; and determining an operation mode of the second event based on the tracking coefficient.
  • a control method includes: receiving a first observation value related to a first event in a music performance; updating a first state variable related to the first event by using the first observation value; and updating a second state variable related to a second event in the music performance by using a product resulting from multiplication of the updated first state variable by a tracking coefficient.
  • a control method is the control method according to the first aspect, wherein in determining the tracking coefficient, determining the tracking coefficient to be a value corresponding to a position that is being played in a piece of the music.
  • a control method is the control method according to the first aspect, wherein in determining the tracking coefficient, determining the tracking coefficient to be a value corresponding to a ratio of a density of music notes involved in the first event to a density of music notes involved in the second event.
  • a control method is the control method according to the fourth aspect, wherein in determining the tracking coefficient, causing the tracking coefficient to be greater in a case where the ratio is greater than a predetermined threshold value as compared to a case where the ratio is equal to or smaller than the predetermined threshold value.
  • a control method is the control method according to the first aspect, wherein in determining the tracking coefficient, causing the tracking coefficient to be smaller in a case where the second event is related to a main melody as compared to a case where the second event is not related to the main melody.
  • a control method is the control method according to the first aspect, wherein in determining the tracking coefficient, determining the tracking coefficient to be a value corresponding to an instruction from a user.
  • a control method is the control method according to any one of the first to the seventh aspects, wherein the first event and the second event are each an event in which sound is output in the music performance, and wherein the tracking coefficient indicates either how closely a timing of outputting sound in the second event follows a timing of outputting sound in the first event, or how closely a volume of sound output in the second event follows a volume of sound output in the first event.
  • a control method is the control method according to the first aspect or any one of the third to the seventh aspects, wherein the second event is an event in which sound is output by an automated musical instrument, and wherein the method further includes causing the automated musical instrument to output sound based on the determined operation mode.
  • a control method is the control method according to the second aspect, wherein the method further includes determining the tracking coefficient in accordance with a position that is being played in a piece of music.
  • a control method is the control method according to the second aspect, wherein the method further includes determining the tracking coefficient in accordance with a ratio of a density of music notes involved in the first event to a density of music notes involved in the second event.
  • a control method is the control method according to the second aspect, wherein the method further includes determining the tracking coefficient such that the tracking coefficient is smaller in a case where the second event is related to a main melody as compared to a case where the second event is not related to the main melody.
  • a control method is the control method according to the second aspect, wherein the method further includes setting the tracking coefficient to a value corresponding to an instruction from a user.
  • a control method is the control method according to the second aspect or any one of the tenth to the thirteenth aspects, wherein the first event and the second event are each an event in which sound is output in the music performance, and wherein the first state variable and the second state variable are each a variable related to a timing of outputting sound in the music performance or a volume of sound output in the music performance.
  • a control method is the control method according to the second aspect or any one of the tenth to the fourteenth aspects, wherein the second event is an event in which sound is output by an automated musical instrument, and wherein the method further includes causing the automated musical instrument to output sound at a timing determined based on the updated second state variable.
  • a controller includes: a receiver configured to receive a result of detection related to a first event in music performance; a coefficient determiner configured to determine a tracking coefficient that indicates how closely a second event follows the first event in the music performance; and an operation determiner configured to determine an operation mode of the second event based on the tracking coefficient.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Auxiliary Devices For Music (AREA)
  • Electrophonic Musical Instruments (AREA)
US16/247,717 2016-07-22 2019-01-15 Control method and controller Active US10665216B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2016-144349 2016-07-22
JP2016144349 2016-07-22
PCT/JP2017/026525 WO2018016637A1 (ja) 2016-07-22 2017-07-21 制御方法、及び、制御装置

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2017/026525 Continuation WO2018016637A1 (ja) 2016-07-22 2017-07-21 制御方法、及び、制御装置

Publications (2)

Publication Number Publication Date
US20190147837A1 US20190147837A1 (en) 2019-05-16
US10665216B2 true US10665216B2 (en) 2020-05-26

Family

ID=60992633

Family Applications (1)

Application Number Title Priority Date Filing Date
US16/247,717 Active US10665216B2 (en) 2016-07-22 2019-01-15 Control method and controller

Country Status (5)

Country Link
US (1) US10665216B2 (zh)
EP (1) EP3489944A4 (zh)
JP (1) JP6729699B2 (zh)
CN (1) CN109478398B (zh)
WO (1) WO2018016637A1 (zh)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6642714B2 (ja) * 2016-07-22 2020-02-12 ヤマハ株式会社 制御方法、及び、制御装置
JP6631714B2 (ja) * 2016-07-22 2020-01-15 ヤマハ株式会社 タイミング制御方法、及び、タイミング制御装置
WO2018016636A1 (ja) * 2016-07-22 2018-01-25 ヤマハ株式会社 タイミング予想方法、及び、タイミング予想装置
JP7243026B2 (ja) * 2018-03-23 2023-03-22 ヤマハ株式会社 演奏解析方法、演奏解析装置およびプログラム
CN115244614A (zh) * 2020-03-17 2022-10-25 雅马哈株式会社 参数推论方法、参数推论系统及参数推论程序
JP2022075147A (ja) * 2020-11-06 2022-05-18 ヤマハ株式会社 音響処理システム、音響処理方法およびプログラム
CN112669798B (zh) * 2020-12-15 2021-08-03 深圳芒果未来教育科技有限公司 一种对音乐信号主动跟随的伴奏方法及相关设备

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5652403A (en) * 1995-04-19 1997-07-29 Yamaha Corporation Keyboard musical instrument allowing player to perform ensemble together with electronic sound system
US20010015122A1 (en) 1999-12-24 2001-08-23 Nobuhiro Yamada Electronic musical instrument performance position retrieval system
US20030024375A1 (en) * 1996-07-10 2003-02-06 Sitrick David H. System and methodology for coordinating musical communication and display
US20030133700A1 (en) * 2002-01-15 2003-07-17 Yamaha Corporation Multimedia platform for recording and/or reproducing music synchronously with visual images
US20030177890A1 (en) * 2002-03-25 2003-09-25 Yamaha Corporation Audio system for reproducing plural parts of music in perfect ensemble
US20040177747A1 (en) * 1998-02-09 2004-09-16 Minoru Tsuji Digital signal processing method and apparatus thereof, control data generation method and apparatus thereof, and program recording medium
US6979769B1 (en) * 1999-03-08 2005-12-27 Faith, Inc. Data reproducing device, data reproducing method, and information terminal
JP2007241181A (ja) 2006-03-13 2007-09-20 Univ Of Tokyo 自動伴奏システム及び楽譜追跡システム
US7863513B2 (en) * 2002-08-22 2011-01-04 Yamaha Corporation Synchronous playback system for reproducing music in good ensemble and recorder and player for the ensemble
US8680388B2 (en) * 2001-01-13 2014-03-25 Native Instruments Software Synthesis Gmbh Automatic recognition and matching of tempo and phase of pieces of music, and an interactive music player
US20150013533A1 (en) 2013-07-13 2015-01-15 Apple Inc. System and method for determining an accent pattern for a musical performance
JP2015036710A (ja) 2013-08-12 2015-02-23 ヤマハ株式会社 音響信号のアライメント装置
JP2015045731A (ja) 2013-08-28 2015-03-12 ヤマハ株式会社 音響信号のアライメント装置
JP2015079183A (ja) 2013-10-18 2015-04-23 ヤマハ株式会社 スコアアライメント装置及びスコアアライメントプログラム

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2518742B2 (ja) * 1991-04-05 1996-07-31 ヤマハ株式会社 自動演奏装置
US5521323A (en) * 1993-05-21 1996-05-28 Coda Music Technologies, Inc. Real-time performance score matching
US5641927A (en) * 1995-04-18 1997-06-24 Texas Instruments Incorporated Autokeying for musical accompaniment playing apparatus
JP3720004B2 (ja) * 2002-07-22 2005-11-24 ヤマハ株式会社 音楽制御装置
JP5654897B2 (ja) * 2010-03-02 2015-01-14 本田技研工業株式会社 楽譜位置推定装置、楽譜位置推定方法、及び楽譜位置推定プログラム
JP6191459B2 (ja) * 2012-06-26 2017-09-06 ヤマハ株式会社 オーディオ波形データを使用する自動演奏技術

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5652403A (en) * 1995-04-19 1997-07-29 Yamaha Corporation Keyboard musical instrument allowing player to perform ensemble together with electronic sound system
US20030024375A1 (en) * 1996-07-10 2003-02-06 Sitrick David H. System and methodology for coordinating musical communication and display
US20040177747A1 (en) * 1998-02-09 2004-09-16 Minoru Tsuji Digital signal processing method and apparatus thereof, control data generation method and apparatus thereof, and program recording medium
US6979769B1 (en) * 1999-03-08 2005-12-27 Faith, Inc. Data reproducing device, data reproducing method, and information terminal
US20010015122A1 (en) 1999-12-24 2001-08-23 Nobuhiro Yamada Electronic musical instrument performance position retrieval system
US8680388B2 (en) * 2001-01-13 2014-03-25 Native Instruments Software Synthesis Gmbh Automatic recognition and matching of tempo and phase of pieces of music, and an interactive music player
US20030133700A1 (en) * 2002-01-15 2003-07-17 Yamaha Corporation Multimedia platform for recording and/or reproducing music synchronously with visual images
US20030177890A1 (en) * 2002-03-25 2003-09-25 Yamaha Corporation Audio system for reproducing plural parts of music in perfect ensemble
US7863513B2 (en) * 2002-08-22 2011-01-04 Yamaha Corporation Synchronous playback system for reproducing music in good ensemble and recorder and player for the ensemble
JP2007241181A (ja) 2006-03-13 2007-09-20 Univ Of Tokyo 自動伴奏システム及び楽譜追跡システム
US20150013533A1 (en) 2013-07-13 2015-01-15 Apple Inc. System and method for determining an accent pattern for a musical performance
JP2015036710A (ja) 2013-08-12 2015-02-23 ヤマハ株式会社 音響信号のアライメント装置
JP2015045731A (ja) 2013-08-28 2015-03-12 ヤマハ株式会社 音響信号のアライメント装置
JP2015079183A (ja) 2013-10-18 2015-04-23 ヤマハ株式会社 スコアアライメント装置及びスコアアライメントプログラム

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
English translation (Japanese-language Written Opinion (PCT/ISA/237) previously filed on Jan. 15, 2019) issued in PCT Application No. PCT/JP2017/026525 dated Oct. 17, 2017 (six pages).
Extended European Search Report issued in European Application No. 17831153.6 dated Mar. 6, 2020 (nine (9) pages).
International Search Report (PCT/ISA/210) issued in PCT Application No. PCT/JP2017/026525 dated Oct. 17, 2017 with English translation (four (4) pages).
Japanese-language Office Action issued in counterpart Japanese Application No. 2018-528901 dated Nov. 12, 2019 with English translation (seven pages).
Japanese-language Written Opinion (PCT/ISA/237) issued in PCT Application No. PCT/JP2017/026525 dated Oct. 17, 2017 (four (4)pages).

Also Published As

Publication number Publication date
EP3489944A1 (en) 2019-05-29
US20190147837A1 (en) 2019-05-16
CN109478398B (zh) 2023-12-26
JP6729699B2 (ja) 2020-07-22
JPWO2018016637A1 (ja) 2019-01-24
EP3489944A4 (en) 2020-04-08
WO2018016637A1 (ja) 2018-01-25
CN109478398A (zh) 2019-03-15

Similar Documents

Publication Publication Date Title
US10665216B2 (en) Control method and controller
US10636399B2 (en) Control method and control device
US10580393B2 (en) Apparatus for analyzing musical performance, performance analysis method, automatic playback method, and automatic player system
US10586520B2 (en) Music data processing method and program
US10650794B2 (en) Timing control method and timing control device
US10699685B2 (en) Timing prediction method and timing prediction device
JP7448053B2 (ja) 学習装置、自動採譜装置、学習方法、自動採譜方法及びプログラム
CN111508508A (zh) 一种超分辨率音频生成方法及设备
JP2017519255A (ja) 楽譜の追従方法及びそれに関連したモデル化方法
JP6690181B2 (ja) 楽音評価装置及び評価基準生成装置
CN110959172B (zh) 演奏解析方法、演奏解析装置以及存储介质
US10810986B2 (en) Audio analysis method and audio analysis device
JP2018146782A (ja) タイミング制御方法
JP6323159B2 (ja) 音響解析装置
Canazza et al. Expressive Director: A system for the real-time control of music performance synthesis
WO2023170757A1 (ja) 再生制御方法、情報処理方法、再生制御システムおよびプログラム
US20230419929A1 (en) Signal processing system, signal processing method, and program
JP2005308992A (ja) 学習支援システム

Legal Events

Date Code Title Description
AS Assignment

Owner name: YAMAHA CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MAEZAWA, AKIRA;REEL/FRAME:048013/0423

Effective date: 20181225

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4