WO2021193032A1 - 演奏エージェントの訓練方法、自動演奏システム、及びプログラム - Google Patents

演奏エージェントの訓練方法、自動演奏システム、及びプログラム Download PDF

Info

Publication number
WO2021193032A1
WO2021193032A1 PCT/JP2021/009361 JP2021009361W WO2021193032A1 WO 2021193032 A1 WO2021193032 A1 WO 2021193032A1 JP 2021009361 W JP2021009361 W JP 2021009361W WO 2021193032 A1 WO2021193032 A1 WO 2021193032A1
Authority
WO
WIPO (PCT)
Prior art keywords
performance
performer
data
satisfaction
acquired
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/009361
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
陽 前澤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yamaha Corp
Original Assignee
Yamaha Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yamaha Corp filed Critical Yamaha Corp
Priority to CN202180020161.5A priority Critical patent/CN115298734A/zh
Priority to JP2022509544A priority patent/JP7388542B2/ja
Publication of WO2021193032A1 publication Critical patent/WO2021193032A1/ja
Priority to US17/950,804 priority patent/US12367854B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0033—Recording/reproducing or transmission of music for electrophonic musical instruments
    • G10H1/0041—Recording/reproducing or transmission of music for electrophonic musical instruments in coded form
    • G10H1/0058—Transmission between separate instruments or between individual components of a musical system
    • G10H1/0066—Transmission between separate instruments or between individual components of a musical system using a MIDI interface
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10G—REPRESENTATION OF MUSIC; RECORDING MUSIC IN NOTATION FORM; ACCESSORIES FOR MUSIC OR MUSICAL INSTRUMENTS NOT OTHERWISE PROVIDED FOR, e.g. SUPPORTS
    • G10G1/00—Means for the representation of music
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/36—Accompaniment arrangements
    • G10H1/361—Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/091—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/441—Image sensing, i.e. capturing images or optical patterns for musical purposes or musical control purposes
    • G10H2220/455—Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/311—Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation

Definitions

  • the present invention relates to a performance agent training method, an automatic performance system, and a program.
  • Patent Document 1 proposes a technique for performing an automatic performance in synchronization with the progress of a performance by an estimated performer.
  • it is proposed to use an estimation model trained by machine learning for a configuration in which an automatic performance is controlled according to a signal motion by a specific performer and for identification of the signal motion.
  • a performance agent suitable for one performer is not always suitable for another performer. If the performance agent is manually generated for each performer, the cost of generating the performance agent becomes enormous.
  • the present invention has been made in view of the above circumstances on one aspect, and an object of the present invention is to provide a technique for reducing the cost of generating a performance agent suitable for a performer.
  • the performance agent training method realized by one or more computers is the second performance performed in parallel with the first performance observed by the performance agent.
  • the performance data is generated, the performance data is output so that the second performance is performed in parallel with the first performance of the performer, and the performer's satisfaction with the second performance based on the output performance data. It comprises a process of acquiring a degree and using the acquired satisfaction as a reward by reinforcement learning to train the performance agent so as to maximize the sum of the satisfactions acquired in the future.
  • FIG. 1 shows an example of the configuration of the automatic performance system according to the embodiment.
  • FIG. 2 shows an example of the hardware configuration of the performance control device according to the embodiment.
  • FIG. 3 shows an example of the hardware configuration of the estimation device according to the embodiment.
  • FIG. 4 shows an example of the software configuration of the automatic performance system according to the embodiment.
  • FIG. 5 is a flowchart showing an example of training processing of an estimation model for estimating satisfaction in the embodiment.
  • FIG. 6 is a flowchart showing an example of the training process of the performance agent in the embodiment.
  • FIG. 7 shows an example of the processing process of reinforcement learning of the performance agent in the embodiment.
  • FIG. 1 shows an example of the configuration of the automatic performance system S according to the present embodiment.
  • the automatic performance system S of the present embodiment includes a performance control device 100, a performance device 200, and an estimation device 300.
  • the performance control device 100 and the estimation device 300 are realized by, for example, an information processing device (computer) such as a personal computer, a server, a tablet terminal, or a mobile terminal (for example, a smartphone).
  • the performance control device 100 is a computer configured to automatically generate performance data for controlling the performance device 200 and supply the performance data to the performance device 200.
  • the performance device 200 may be appropriately configured to perform the second performance according to the performance data of the second performance.
  • the performance device 200 is an automatic performance instrument such as a keyboard instrument having a sounding mechanism and a drive mechanism, and is a computer configured to perform unmanned performance based on performance data supplied from the performance control device 100.
  • the sounding mechanism of the playing device 200 is a string striking mechanism that sounds a string (sounding body) in conjunction with the displacement of each key on the keyboard.
  • the drive mechanism of the performance device 200 realizes the performance of the target musical piece by driving the sounding mechanism based on the above performance data.
  • the estimation device 300 is a computer configured to estimate the satisfaction (favorability) of the performer in the co-starring of the performer and the performance agent 160.
  • the "satisfaction" in the present invention means the personal satisfaction of a specific performer.
  • the performer of the present embodiment typically plays using the electronic musical instrument EM connected to the performance control device 100.
  • the electronic musical instrument EM of the present embodiment may be, for example, an electronic keyboard instrument (electronic piano or the like), an electronic stringed instrument (electric guitar or the like), an electronic wind instrument (wind synthesizer or the like) or the like.
  • the musical instrument used by the performer for performance is not limited to the electronic musical instrument EM.
  • the performer may perform with an acoustic instrument.
  • the performer according to the present embodiment may be a singer of a musical piece that does not use a musical instrument. In this case, the performance by the performer may be performed without using an instrument.
  • the performance by the performer is referred to as "first performance”
  • the performance by the performance agent 160 is referred to as "second performance”.
  • the automatic performance system S observes the first performance of the musical piece by the performer, and the performance agent 160 described later performs the performance of the second performance in parallel with the observed first performance. Generate data. Subsequently, the automatic performance system S outputs performance data so that the second performance is performed in parallel with the first performance of the performer, and acquires the satisfaction of the performer with respect to the second performance based on the output performance data. .. Then, the automatic performance system S trains the performance agent 160 so as to maximize the sum of the satisfactions acquired in the future by using the satisfaction acquired by the reinforcement learning as a reward. According to this automatic performance system S, a performance agent 160 suitable for the performer can be automatically generated. Therefore, it is possible to reduce the cost of generating a performance agent 160 suitable for the performer.
  • FIG. 2 shows an example of the hardware configuration of the performance control device 100 according to the present embodiment.
  • the CPU 101, the RAM 102, the storage 103, the input unit 104, the output unit 105, the sound collecting unit 106, the imaging unit 107, the transmitting / receiving unit 108, and the drive 109 are electrically operated by the bus B1.
  • a computer connected to.
  • the CPU 101 is composed of one or a plurality of processors for executing various operations in the performance control device 100.
  • the CPU 101 is an example of a processor resource.
  • the type of processor may be appropriately selected depending on the embodiment.
  • the RAM 102 is a volatile storage medium, and operates as a working memory that holds information such as set values used by the CPU 101 and develops various programs.
  • the storage 103 is a non-volatile storage medium that stores various programs and data used by the CPU 101.
  • the RAM 102 and the storage 103 are examples of memory resources that hold programs executed by processor resources.
  • the storage 103 stores various information such as the program 81.
  • the program 81 performs a second performance in parallel with the first performance of the musical piece by the performer, and causes the performance control device 100 to execute information processing (FIGS. 6 and 7 described later) for training the performance agent 160 described later.
  • Program 81 includes a series of instructions for the information processing.
  • the input unit 104 is composed of an input device for receiving an operation on the performance control device 100.
  • the input unit 104 may be composed of one or a plurality of input devices such as a keyboard and a mouse connected to the performance control device 100, for example.
  • the output unit 105 is composed of an output device for outputting various information.
  • the output unit 105 may be composed of one or a plurality of output devices such as a display and a speaker connected to the performance control device 100, for example.
  • Information may be output by, for example, a video signal, a sound signal, or the like.
  • the input unit 104 and the output unit 105 may be integrally configured by an input / output device such as a touch panel display that receives a user's operation on the performance control device 100 and outputs various information.
  • an input / output device such as a touch panel display that receives a user's operation on the performance control device 100 and outputs various information.
  • the sound collecting unit 106 is configured to convert the collected sound into an electric signal and supply it to the CPU 101.
  • the sound collecting unit 106 is composed of, for example, a microphone.
  • the sound collecting unit 106 may be built in the performance control device 100, or may be connected to the performance control device 100 via an interface (not shown).
  • the imaging unit 107 is configured to convert the captured image into an electric signal and supply it to the CPU 101.
  • the image pickup unit 107 is composed of, for example, a digital camera.
  • the imaging unit 107 may be built in the performance control device 100, or may be connected to the performance control device 100 via an interface (not shown).
  • the transmission / reception unit 108 is configured to transmit / receive data to / from another device wirelessly or by wire.
  • the performance control device 100 is connected to the performance device 200 to be controlled, the electronic musical instrument EM used by the performer to play a musical piece, and the estimation device 300 via the transmission / reception unit 108, and data is input. You may send and receive.
  • the transmission / reception unit 108 may include a plurality of modules (for example, a Bluetooth (registered trademark) module, a Wi-Fi (registered trademark) module, a USB (Universal Serial Bus) port, a dedicated port, etc.).
  • the drive 109 is a drive device for reading various information such as programs stored in the storage medium 91.
  • the storage medium 91 electrically, magnetically, optically, mechanically or chemically acts on the information of the program or the like so that the computer or other device, the machine or the like can read various information of the stored program or the like. It is a medium that accumulates by.
  • the storage medium 91 is, for example, a floppy disk, an optical disk (for example, a compact disk, a digital versatile disk, a Blu-ray disk), a magneto-optical disk, a magnetic tape, a non-volatile memory card (for example, a flash memory), or the like. good.
  • the type of the drive 109 may be arbitrarily selected according to the type of the storage medium 91.
  • the program 81 may be stored in the storage medium 91, and the performance control device 100 may read the program 81 from the storage medium 91.
  • Bus B1 is a signal transmission line that electrically connects the hardware components of the performance control device 100 to each other.
  • the components can be omitted, replaced, or added as appropriate according to the embodiment.
  • at least one of the input unit 104, the output unit 105, the sound collecting unit 106, the imaging unit 107, the transmitting / receiving unit 108, and the drive 109 may be omitted.
  • FIG. 3 shows an example of the hardware configuration of the estimation device 300 according to the present embodiment.
  • the CPU 301, the RAM 302, the storage 303, the input unit 304, the output unit 305, the sound collecting unit 306, the imaging unit 307, the transmission / reception unit 309, and the drive 310 are electrically operated by the bus B3. It is a connected computer.
  • the CPU 301 is composed of one or a plurality of processors for executing various operations in the estimation device 300.
  • the CPU 301 is an example of the processor resource of the estimation device 300.
  • the type of processor may be appropriately selected depending on the embodiment.
  • the RAM 302 is a volatile storage medium, and operates as a working memory that holds various information such as setting values used by the CPU 301 and develops various programs.
  • the storage 303 is a non-volatile storage medium and stores various programs and data used by the CPU 301.
  • the RAM 302 and the storage 303 are examples of memory resources of the estimation device 300 that holds a program executed by the processor resource.
  • the storage 303 stores various information such as the program 83.
  • the program 83 is a program for causing the estimation device 300 to execute information processing for training the satisfaction estimation model (FIG. 5 described later) and information processing for estimating satisfaction using the trained estimation model.
  • Program 83 includes a series of instructions for the information processing.
  • the input unit 304 to the imaging unit 307, the drive 310, and the storage medium 93 may be configured in the same manner as the input unit 104, the imaging unit 107, the drive 109, and the storage medium 91 of the performance control device 100.
  • the program 83 may be stored in the storage medium 93, and the estimation device 300 may read the program 83 from the storage medium 93.
  • the biological sensor 308 is configured to acquire biological signals indicating the performer's biological information in time series.
  • the biological information of the performer may be composed of one or a plurality of types of data such as heart rate, sweating amount, and blood pressure, for example.
  • the biosensor 308 may be composed of, for example, a sensor such as a heart rate monitor, a sweat meter, and a sphygmomanometer.
  • the transmission / reception unit 309 is configured to transmit / receive data to / from another device wirelessly or by wire.
  • the estimation device 300 may be connected to the electronic musical instrument EM and the performance control device 100 used when the performer plays a musical piece via the transmission / reception unit 309 to transmit / receive data.
  • the transmission / reception unit 309 may include a plurality of modules like the transmission / reception unit 108.
  • Bus B3 is a signal transmission line that electrically connects the hardware components of the estimation device 300 to each other.
  • the components can be omitted, replaced, or added as appropriate according to the embodiment.
  • at least one of the input unit 304, the output unit 305, the sound collecting unit 306, the imaging unit 307, the biosensor 308, the transmitting / receiving unit 309, and the drive 310 may be omitted.
  • FIG. 4 shows an example of the software configuration of the automatic performance system S according to the present embodiment.
  • the performance control device 100 has a control unit 150 and a storage unit 180.
  • the control unit 150 is configured to integrally control the operation of the performance control device 100 by the CPU 101 and the RAM 102.
  • the storage unit 180 is configured to store various data used in the control unit 150 by the RAM 102 and the storage 103.
  • the CPU 101 of the performance control device 100 expands the program 81 stored in the storage 103 into the RAM 102, and executes the instructions included in the program 81 expanded in the RAM 102.
  • the performance control device 100 operates as a computer including an authentication unit 151, a performance acquisition unit 152, a video acquisition unit 153, a performance agent 160, and an agent training unit 170 as software modules.
  • the authentication unit 151 is configured to authenticate the user (performer) in cooperation with an external device such as the estimation device 300.
  • the authentication unit 151 transmits authentication data such as a user identifier and a password input by the user using the input unit 104 to the estimation device 300, and permits the user's access based on the authentication result received from the estimation device 300. Or it is configured to refuse.
  • the external device that authenticates the user may be an authentication server other than the estimation device 300.
  • the authentication unit 151 may be configured to supply the user identifier of the authenticated (access-authorized) user to other software modules.
  • the performance acquisition unit 152 is configured to observe the first performance of the musical piece by the performer and acquire performance data (hereinafter, also referred to as "first performance data") indicating the first performance.
  • the first performance data is, for example, a MIDI data string with a time stamp supplied from the electronic musical instrument EM.
  • the performance acquisition unit 152 may be configured to acquire the performance sound indicated by the electric signal that the sound collecting unit 106 collects and outputs the first performance as the first performance data.
  • the first performance data is data showing the characteristics of sounds included in the performance (for example, sound generation time and pitch), and is a kind of high-dimensional time series data expressing the first performance by the performer.
  • the performance acquisition unit 152 is configured to supply the acquired first performance data to the performance agent 160.
  • the performance acquisition unit 152 may be configured to transmit the acquired first performance data to the estimation device 300.
  • the video acquisition unit 153 is configured to acquire video data related to the first performance by the performer.
  • the video acquisition unit 153 may be configured to acquire video data based on an electrical signal indicating a performer's video in the first performance taken by the imaging unit 107.
  • the video data is motion data showing the characteristics of the performer's movement in the performance, and is a kind of high-dimensional time series data expressing the performance by the performer.
  • the motion data is, for example, data obtained by acquiring the skeleton of the performer in time series.
  • the video acquisition unit 153 is configured to supply the acquired video data to the performance agent 160.
  • the video acquisition unit 153 may be configured to transmit the acquired video data to the estimation device 300.
  • the performance agent 160 is configured to cause the performance device 200 to perform an automatic performance that is a co-star with the performer.
  • the performance agent 160 is, for example, a method disclosed in International Publication No. 2018/070286, "Study on real-time sheet music tracking by acoustic signals and active performance support system” (Shinji Sakou (Nagoya Institute of Technology), Telecommunications Promotion Foundation " It may be configured to perform automatic performance control based on any method such as the method disclosed in "Research Grant Report" No. 31, 2016).
  • the automatic performance (second performance) may be, for example, an accompaniment to the first performance or a counter-melody.
  • the performance agent 160 is set to, for example, the state at that time (for example, “volume difference between the two (performer and performance agent)", “volume of the performance agent”, “tempo of the performance agent”, “timing difference between the two”, etc.). Actions to be performed accordingly (for example, “increase tempo”, “decrease tempo”, “decrease tempo by 10", ..., “increase volume by 3", “increase volume by 1", “increase volume” It is composed of an arithmetic model having a plurality of parameters for determining "1 lower”, etc.).
  • the performance agent 160 may be appropriately configured to determine an action according to the state at that time based on the plurality of parameters, and to change the performance being performed at that time according to the determined action.
  • the performance agent 160 is configured to include a performance analysis unit 161 and a performance generation unit 162 according to the calculation model. Non-limiting and general automatic performance control will be illustrated below.
  • the performance analysis unit 161 estimates the performance position, which is the position on the music actually played by the performer, based on the first performance data and the video data supplied from the performance acquisition unit 152 and the video acquisition unit 153. It is composed of.
  • the estimation of the performance position by the performance analysis unit 161 may be continuously (for example, periodically) executed in parallel with the performance by the performer.
  • the performance analysis unit 161 is configured to estimate the performance position of the performer by mutually comparing the series of sounds indicated by the first performance data with the series of notes indicated by the music data for automatic performance. May be done.
  • the music data includes reference data corresponding to the first performance (performer part) by the performer and automatic performance data indicating the second performance (automatic performance part) by the performance agent 160. Any music analysis technique (score alignment technique) may be appropriately adopted for the estimation of the performance position by the performance analysis unit 161.
  • the performance generation unit 162 performs the performance data of the second performance (hereinafter, hereinafter,) based on the automatic performance data in the music data so as to synchronize with the progress (movement on the time axis) of the performance position estimated by the performance analysis unit 161. It is configured to automatically generate (also referred to as "second performance data") and supply the generated second performance data to the performance device 200.
  • the automatically generated second performance data is data for a second performance performed in parallel with the first performance, and is instruction data for operating the drive mechanism of the performance device 200 according to a series of notes.
  • the performance generation unit 162 is configured to operate as a sequencer that supplies the second performance data (for example, a MIDI data string with a time stamp) that realizes the music corresponding to the music data to the performance device 200. ..
  • the performance generation unit 162 may be configured to supply the second performance data to the estimation device 300 as well.
  • the performance device 200 is configured to perform a second performance, which is an automatic performance of a musical piece, according to the second performance data supplied from the performance generation unit 162.
  • the configuration of the performance agent 160 does not have to be limited to such an example.
  • the performance agent 160 (performance analysis unit 161 and performance generation unit 162) improvises a second performance based on the first performance data indicating the performer's first performance, not based on the existing music data.
  • the performance device 200 may be configured to perform automatic performance (improvisational performance).
  • the agent training unit 170 is configured to train the performance agent 160 so as to maximize the performer's satisfaction with the second performance.
  • the operation of the agent training unit 170 will be described in detail later.
  • the degree of satisfaction of the performer with respect to the second performance may be obtained by any method.
  • acquiring the satisfaction level may be configured by acquiring the performer information related to the first performance of the performer and acquiring the satisfaction level from the acquired performer information.
  • obtaining satisfaction from performer information is configured by estimating satisfaction from performer information using a trained estimation model (satisfaction estimation model) generated by machine learning, which will be described later. May be done.
  • the performer information may be configured to include a video of the performer performing the first performance.
  • the performer information may be configured to include at least one of the performer's facial expressions and postures extracted from the video.
  • the performer information may be configured to include the performer's biological signal acquired by the performer during the first performance. This "during the first performance” may include the period during the first performance and the period during which the lingering sound remains after the end of the first performance.
  • the performer information may be configured to include performance data of the first performance by the performer.
  • the estimation device 300 has a control unit 350 and a storage unit 380.
  • the control unit 350 is configured to integrally control the operation of the estimation device 300 by the CPU 301 and the RAM 302.
  • the storage unit 380 is configured to store various data (particularly, a satisfaction estimation model described later) used in the control unit 350 by the RAM 302 and the storage 303.
  • the CPU 301 of the estimation device 300 expands the program 83 stored in the storage 303 into the RAM 302, and executes the instructions included in the program 83 expanded in the RAM 302.
  • the estimation device 300 (control unit 350) includes the authentication unit 351, the performance acquisition unit 352, the reaction acquisition unit 353, the satisfaction acquisition unit 354, the data preprocessing unit 355, the model training unit 356, and the satisfaction estimation unit 357. Operates as a computer equipped as a software module.
  • the authentication unit 351 is configured to authenticate the user (performer) in cooperation with the performance control device 100. In one example, the authentication unit 351 determines whether or not the authentication data provided by the performance control device 100 matches the authentication data stored in the storage unit 380, and determines the authentication result (permission or denial) of the performance control device. It is configured to send to 100.
  • the performance acquisition unit 352 is configured to acquire the first performance data (performer information) by the performer.
  • the first performance data is a note sequence, and is data in which the sounding timing, pitch, pitch, and intensity of each note are defined.
  • the performance acquisition unit 352 may be configured to acquire performance data indicating the first performance supplied from the electronic musical instrument EM directly from the electronic musical instrument EM or via the performance control device 100.
  • the performance acquisition unit 352 may be configured to acquire the performance sound indicating the first performance using the sound collecting unit 306 or via the performance control device 100.
  • the performance acquisition unit 352 is configured to store the acquired first performance data in the storage unit 380.
  • the performance acquisition unit 352 may be configured to associate the acquired first performance data with the user identifier of the performer authenticated by the authentication unit 351.
  • the reaction acquisition unit 353 is configured to acquire reaction data (player information) indicating the reaction of the performer performing the first performance.
  • the reaction acquisition unit 353 may be configured to acquire a video image of the reaction of the performer performing the first performance, which is captured by the imaging unit 307, as reaction data.
  • the reaction acquisition unit 353 may acquire at least one of the facial expressions and postures of the performer extracted from the acquired video as reaction data.
  • the reaction acquisition unit 353 may acquire the biological signal of the performer acquired by the biological sensor 308 at the time of the first performance by the performer as reaction data.
  • the biological signal may be composed of one or more kinds of data such as heart rate, sweating amount, blood pressure and the like.
  • Satisfaction acquisition unit 354 is configured to acquire a satisfaction label (correct answer label) indicating the performer's personal satisfaction in co-starring with the performance agent 160 (performance device 200).
  • the satisfaction label is data indicating the satisfaction of the performer with respect to the second performance by the performance agent 160 (or any method of simulating the performance agent 160). Satisfaction may be expressed as a discrete value representing a stepwise evaluation, or may be expressed as a continuous value.
  • the performer may input the satisfaction label via an input device such as an input unit 104 of the performance control device 100 and an input unit 304 of the estimation device 300.
  • the satisfaction label When the satisfaction label information is input to the performance control device 100, the satisfaction label may be transferred to the estimation device 300 by the control unit 150 (CPU101), and the satisfaction acquisition unit 354 transfers from the performance control device 100. It may be configured to receive a satisfaction label that is to be received.
  • the satisfaction acquisition unit 354 is configured to store the acquired satisfaction label in the storage unit 380 in association with the performer information (first performance data, reaction data) related to the first performance.
  • the data preprocessing unit 355 is suitable for calculating the estimation model by inputting data (performer information, etc.) to be input to an estimation model for estimating the satisfaction of the performer (hereinafter, also referred to as “satisfaction estimation model”). It is configured to be preprocessed so that it has a different format.
  • the data preprocessing unit 355 is configured to supply the preprocessed data to the model training unit 356 in the training stage and to supply the preprocessed data to the satisfaction estimation unit 357 in the estimation stage.
  • the model training unit 356 uses the performer information and the satisfaction label supplied from the data preprocessing unit 355 as input data (training data) and a teacher signal (correct answer data), respectively, to generate a satisfaction estimation model by machine learning. Configured to train.
  • the satisfaction estimation model may be composed of any machine learning model having a plurality of parameters.
  • a feedforward neural network (FFNN) composed of a multi-layer perceptron, a hidden Markov model (HMM), or the like may be used.
  • Machine learning models that make up the satisfaction estimation model include, for example, a recurrent neural network (RNN) adapted to time-series data, a derived configuration (long / short-term memory (LSTM)), and a gated recurrent unit. (GRU) etc.), convolutional neural network (CNN) and the like may be used.
  • RNN recurrent neural network
  • LSTM long / short-term memory
  • GRU gated recurrent unit.
  • CNN convolutional neural network
  • Machine learning consists of training the satisfaction estimation model so that the satisfaction estimated from the performer information for training by the satisfaction estimation model matches the true value indicated by the satisfaction label.
  • the machine learning method may be appropriately selected depending on the type of machine learning model to be adopted.
  • the trained satisfaction estimation model generated by machine learning may be appropriately stored in a storage area such as a storage unit 380 in the form of learning result data.
  • the satisfaction estimation unit 357 includes a trained satisfaction estimation model generated by the model training unit 356.
  • the satisfaction estimation unit 357 is configured to estimate the satisfaction of the performer from the performer information obtained at the time of inference by using the trained satisfaction estimation model.
  • the satisfaction estimation unit 357 inputs the performer information after the pre-processing supplied from the data pre-processing unit 355 into the trained satisfaction estimation model as input data, and the trained satisfaction Execute the arithmetic processing of the degree estimation model. By this arithmetic processing, the satisfaction estimation unit 357 acquires the output corresponding to the result of estimating the satisfaction of the performer from the input performer information from the trained satisfaction estimation model.
  • the estimated satisfaction level (estimation result of satisfaction level) is supplied to the agent training unit 170 of the performance control device 100.
  • each software module of the performance control device 100 and the estimation device 300 is realized by a general-purpose CPU.
  • some or all of the software modules may be implemented by one or more dedicated processors.
  • Each of the above modules may be realized as a hardware module.
  • software modules may be omitted, replaced, or added as appropriate according to the embodiment.
  • FIG. 5 is a flowchart showing an example of training processing of a satisfaction estimation model by the automatic performance system S according to the present embodiment.
  • the following processing procedure is only an example, and each step may be changed as much as possible. Further, with respect to the following processing procedures, steps may be omitted, replaced, and added as appropriate according to the embodiment.
  • step S510 the CPU 301 of the estimation device 300 acquires the performer information related to the first performance of the performer.
  • the performer information includes the first performance data indicating the first performance by the performer, the biological signal of the performer acquired at the time of the first performance by the performer, and the image of the performer performing the first performance. , And at least one of the performer's facial expressions and postures extracted from the video.
  • the CPU 301 operates as the performance acquisition unit 352 and acquires the first performance data indicating the first performance by the performer.
  • the performer information is at least one of the biological signal of the performer acquired at the time of the first performance by the performer, the image of the performer performing the first performance, and the facial expression and posture of the performer extracted from the image.
  • the CPU 301 operates as the reaction acquisition unit 353 and acquires reaction data indicating the reaction of the performer performing the first performance.
  • the CPU 301 stores the acquired performer information in the storage unit 380.
  • the second performance is performed in parallel with the first performance by an arbitrary method (a method using the performance agent 160 or another method). You can do it.
  • step S520 the CPU 301 operates as the satisfaction acquisition unit 354 and acquires a satisfaction label indicating the satisfaction of the performer with respect to the second performance.
  • the CPU 301 may acquire the satisfaction label by inputting the performer via the input device.
  • the CPU 301 stores the acquired satisfaction label in the storage unit 380 in association with the performer information.
  • steps S510 and S520 may be executed in parallel or sequentially.
  • step S530 the CPU 301 operates as a data preprocessing unit 355, performs data preprocessing on the performer information stored in the storage unit 380, and supplies the performer information after the preprocessing to the model training unit 356. ..
  • step S540 the CPU 301 operates as a model training unit 356, and uses the preprocessed performer information and the satisfaction label as input data (training data) and a teacher signal (correct answer data), respectively, to obtain a satisfaction estimation model.
  • Perform machine learning That is, the CPU 301 trains the satisfaction estimation model so that the satisfaction estimated from the performer information for training by the satisfaction estimation model matches the true value indicated by the satisfaction label.
  • This machine learning generates a trained satisfaction estimation model that has acquired the ability to estimate the satisfaction of the performer from the performer information.
  • the CPU 301 generates learning result data showing a trained satisfaction estimation model, and stores the generated learning result data in a storage area such as a storage unit 380.
  • the CPU 301 updates the learning result data stored in the storage area such as the storage unit 380 with the newly generated learning result data.
  • the training process of the satisfaction estimation model related to this operation example is completed.
  • the training process may be executed periodically, or may be executed in response to a request from the user (performance control device 100).
  • the CPU 101 of the performance control device 100 and the CPU 301 of the estimation device 300 may operate as authentication units (151, 351), respectively, to authenticate the performer. This may collect performer information and satisfaction labels for certified performers and generate a trained satisfaction estimation model.
  • FIG. 6 is a flowchart showing an example of a performance agent training process by the automatic performance system S according to the present embodiment.
  • the following processing procedure is an example of a performance agent training method. However, the following processing procedure is only an example, and each step may be changed as much as possible. Further, with respect to the following processing procedures, steps may be omitted, replaced, and added as appropriate according to the embodiment.
  • step S610 at least one of the CPU 101 of the performance control device 100 and the CPU 301 of the estimation device 300 observes the first performance of the musical piece by the performer and acquires the first performance data indicating the first performance.
  • the CPU 101 may operate as the performance acquisition unit 152 and acquire the first performance data.
  • the CPU 301 may operate as the performance acquisition unit 352 and acquire the first performance data.
  • the acquired first performance data may be stored in at least one of the storage unit 180 and the storage unit 380.
  • step S620 the CPU 101 generates second performance data indicating a second performance performed in parallel with the observed first performance by the performance agent 160.
  • the CPU 101 operates as a performance analysis unit 161 and a performance generation unit 162, estimates the performance position of the performer by executing arithmetic processing of the arithmetic model constituting the performance agent, and synchronizes with the estimated performance position.
  • the second performance data is generated in.
  • step S630 the CPU 101 operates as a performance generation unit 162, and outputs second performance data so that the second performance is performed in parallel with the first performance of the performer.
  • the CPU 101 supplies the generated second performance data to the performance device 200, and controls the operation of the performance device 200 so as to execute an automatic performance according to the second performance data.
  • step S640 at least one of the CPU 101 of the performance control device 100 and the CPU 301 of the estimation device 300 acquires the performer information related to the first performance of the performer.
  • the process of step S640 may be executed in common with step S610.
  • the performer information is at least one of the biological signal of the performer acquired at the time of the first performance by the performer, the image of the performer performing the first performance, and the facial expression and posture of the performer extracted from the image.
  • the CPU 301 may operate as the reaction acquisition unit 353 and acquire reaction data indicating the reaction of the performer performing the first performance.
  • the CPU 101 may operate as the image acquisition unit 153 to acquire at least one of the image of the performer performing the first performance and the facial expression and posture of the performer extracted from the image.
  • the acquired performer information may be stored in at least one of the storage unit 180 and the storage unit 380.
  • step S650 the acquired performer information is supplied to the data preprocessing unit 355.
  • the CPU 301 of the estimation device 300 operates as a data pre-processing unit 355, performs data pre-processing on the performer information, and supplies the performer information after the pre-processing to the satisfaction estimation unit 357.
  • the CPU 301 operates as a satisfaction estimation unit 357, and uses a trained estimation model to estimate the satisfaction of the performer with respect to the second performance based on the output second performance data from the acquired performer information. do.
  • the estimated satisfaction level is supplied from the satisfaction level estimation unit 357 to the agent training unit 170 of the performance control device 100.
  • step S660 the CPU 101 of the performance control device 100 determines whether or not to start training the performance agent 160. For example, the CPU 101 may determine that the training of the performance agent 160 is started at an arbitrary timing such as when the data used for reinforcement learning is collected or when the first performance by the performer is completed. If it is determined that the training is to be started, the CPU 101 proceeds to step S670. On the other hand, when it is determined that the training is not started, the CPU 101 returns the process to step S610, repeatedly executes a series of processes from step S610 to step S650, and collects learning data for the performance agent 160 to use for training. To continue.
  • the CPU 101 may determine that the training of the performance agent 160 is started at an arbitrary timing such as when the data used for reinforcement learning is collected or when the first performance by the performer is completed. If it is determined that the training is to be started, the CPU 101 proceeds to step S670. On the other hand, when it is determined that the training is not started, the CPU 101 returns the process to step S610, repeatedly executes a
  • steps S610 to S650 may be sequentially executed in the co-starring of the performer and the performance agent 160, and as a result, the satisfaction level of the performer may be estimated in chronological order. Satisfaction may be estimated for each unit time, or for any performance unit (eg, phrase).
  • the processes of steps S610 to S650 may be executed in real time in parallel with the performer performing the first performance, or the first performance stored in at least one of the storage unit 180 and the storage unit 380. May be executed ex post facto.
  • step S670 the CPU 101 operates as the agent training unit 170, and uses the learning data collected by the processes up to step S660 to execute machine learning of the performance agent 160.
  • FIG. 7 shows an example of processing (machine learning) of the performance agent 160 by the automatic performance system S according to the present embodiment.
  • the CPU 101 executes reinforcement learning of the performance agent 160 by using the satisfaction of the performer for the second performance as a reward.
  • the reinforcement learning of the present embodiment corresponds to the "state” in which the first performance by the performer is observed, and the satisfaction level of the performer estimated by the satisfaction estimation unit 357 is the "reward".
  • the second performance by the performance agent 160 and the performance device 200 corresponds to "action”.
  • the first performance by the performer who is the "state” changes, and the satisfaction level of the performer who is the "reward” is estimated.
  • the first performance, the second performance, and the satisfaction level are all time series data.
  • the "state” may include other information about the performance (eg, music data, reaction data).
  • the CPU 101 trains the performance agent 160 so as to maximize the satisfaction of the performer with respect to the second performance by using the satisfaction obtained by the reinforcement learning as a reward. More specifically, the performance agent 160 is such that the CPU 101 automatically generates second performance data that maximizes the sum of the performer's satisfactions (that is, "revenue") acquired in the future. To train. During this training process, the values of the parameters that make up the playing agent 160 are gradually changed to increase the rewards obtained.
  • the performance agent 160 may be configured to include a value function (for example, an action value function), and as the reinforcement learning method, for example, a method such as Q-learning or a Monte Carlo method may be adopted.
  • the performance agent 160 may be configured to include a policy function, and a method such as a policy gradient method may be adopted as the method of reinforcement learning. The method of reinforcement learning and the configuration of the performance agent 160 may be appropriately selected according to the embodiment.
  • the training process of the performance agent 160 is completed.
  • the training process may be executed at any timing.
  • the CPU 101 of the performance control device 100 and the CPU 301 of the estimation device 300 operate as authentication units (151 and 351) and perform before executing the process of step S610.
  • the automatic performance system S may collect learning data of the authenticated performer and use the collected learning data to perform training of the performance agent 160.
  • the automatic performance system S performs automatic performance by the performance agent 160 by executing the processes of steps S610 to S630 (that is, the processes after step S640 are omitted) without training the performance agent 160. You may.
  • a performance agent 160 suitable for the performer can be automatically generated. Therefore, it is possible to reduce the cost of generating a performance agent 160 suitable for the performer.
  • the satisfaction level of the performer can be automatically acquired by using the performer information. As a result, it is possible to reduce the time and effort required to obtain satisfaction. Further, in the present embodiment, the satisfaction level can be appropriately acquired by using the trained satisfaction level estimation model generated by machine learning.
  • the performer information includes the first performance data indicating the first performance by the performer, the biological signal of the performer acquired at the time of the first performance by the performer, and the performer performing the first performance. It may be configured to include at least one of the image of the performer and the performer's facial expression and posture extracted from the image. As a result, the degree of satisfaction can be estimated with high accuracy.
  • the automatic performance system S includes a performance control device 100, a performance device 200, an estimation device 300, and an electronic musical instrument EM as separate devices.
  • a performance control device 100 may be integrally configured.
  • the performance control device 100 and the performance device 200 may be integrally configured.
  • the performance control device 100 and the estimation device 300 may be integrally configured.
  • the performance device 200 may be omitted.
  • the automatic performance system S may be configured to realize the automatic performance of the second performance by supplying the second performance data to the external performance device by the performance control device 100.
  • the trained satisfaction estimation model generated by machine learning is used to acquire the satisfaction.
  • the method of obtaining satisfaction does not have to be limited to such an example. Satisfaction may be obtained in other ways from the performer information. Alternatively, the satisfaction level may be acquired independently of the performer information. The satisfaction level of the performer used for training the performance agent 160 may be obtained by any method. As another example, the satisfaction level may be calculated from the performer information by a predetermined algorithm. As yet another example, the satisfaction level may be directly input by the player's operation via an input device such as an input unit 104 of the performance control device 100 and an input unit 304 of the estimation device 300.
  • the method of reinforcement learning does not have to be limited to the above method.
  • arithmetic model of the performance agent 160 for example, a regression (Bayesian optimization) model of the Gaussian process may be used.
  • a genetic algorithm which is a method that imitates the process of biological evolution, may be adopted as the machine learning method.
  • Each of the above storage media (91, 93) may be composed of a non-transient recording medium that can be read by a computer. Further, the program (81, 83) may be supplied via a transmission medium or the like.
  • the "non-transient recording medium that can be read by a computer” is, for example, a computer system that constitutes a server, a client, or the like when a program is transmitted via a communication network such as the Internet or a telephone line. It may include a recording medium that holds a program for a certain period of time, such as an internal volatile memory (for example, DRAM (Dynamic Random Access Memory)).

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Social Psychology (AREA)
  • Psychiatry (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Electrophonic Musical Instruments (AREA)
  • Auxiliary Devices For Music (AREA)
PCT/JP2021/009361 2020-03-23 2021-03-09 演奏エージェントの訓練方法、自動演奏システム、及びプログラム Ceased WO2021193032A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN202180020161.5A CN115298734A (zh) 2020-03-23 2021-03-09 演奏代理的训练方法、自动演奏系统及程序
JP2022509544A JP7388542B2 (ja) 2020-03-23 2021-03-09 演奏エージェントの訓練方法、自動演奏システム、及びプログラム
US17/950,804 US12367854B2 (en) 2020-03-23 2022-09-22 Performance agent training method, automatic performance system, and program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-051490 2020-03-23
JP2020051490 2020-03-23

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/950,804 Continuation US12367854B2 (en) 2020-03-23 2022-09-22 Performance agent training method, automatic performance system, and program

Publications (1)

Publication Number Publication Date
WO2021193032A1 true WO2021193032A1 (ja) 2021-09-30

Family

ID=77891473

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/009361 Ceased WO2021193032A1 (ja) 2020-03-23 2021-03-09 演奏エージェントの訓練方法、自動演奏システム、及びプログラム

Country Status (4)

Country Link
US (1) US12367854B2 (https=)
JP (1) JP7388542B2 (https=)
CN (1) CN115298734A (https=)
WO (1) WO2021193032A1 (https=)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7147384B2 (ja) * 2018-09-03 2022-10-05 ヤマハ株式会社 情報処理方法および情報処理装置
WO2021186928A1 (ja) * 2020-03-17 2021-09-23 ヤマハ株式会社 演奏情報に対する評価を推論する方法、システム、及びプログラム
CN115298733A (zh) * 2020-03-24 2022-11-04 雅马哈株式会社 训练好的模型的建立方法、推定方法、演奏代理的推荐方法、演奏代理的调整方法、训练好的模型的建立系统、推定系统、训练好的模型的建立程序及推定程序
US20260105276A1 (en) * 2024-10-16 2026-04-16 Turing Dream SL Self-learning massive agentic system, apparatus, and method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011095530A (ja) * 2009-10-30 2011-05-12 Casio Computer Co Ltd 自動伴奏装置およびプログラム
JP2015191194A (ja) * 2014-03-28 2015-11-02 パイオニア株式会社 演奏評価システム、サーバ装置、端末装置、演奏評価方法及びコンピュータプログラム
JP2018078975A (ja) * 2016-11-15 2018-05-24 株式会社ｇｌｏｏｐｓ 端末装置、端末装置のゲーム実行方法、ゲーム実行プログラム、及びゲーム実行プログラム記録媒体
JP2019162207A (ja) * 2018-03-19 2019-09-26 富士ゼロックス株式会社 情報処理装置及び情報処理プログラム

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3215058B2 (ja) * 1996-11-06 2001-10-02 株式会社エイ・ティ・アール知能映像通信研究所 演奏支援機能付楽器
KR100561315B1 (ko) * 2003-10-16 2006-03-16 티제이미디어 주식회사 노래반주기의 장르별 최적음역 자동설정장치 및 방법
US8134061B2 (en) * 2006-04-21 2012-03-13 Vergence Entertainment Llc System for musically interacting avatars
US9082380B1 (en) * 2011-10-31 2015-07-14 Smule, Inc. Synthetic musical instrument with performance-and/or skill-adaptive score tempo
CN108040497B (zh) * 2015-06-03 2022-03-04 思妙公司 用于自动产生协调的视听作品的方法和系统
US20220313083A1 (en) * 2015-10-09 2022-10-06 Senseye, Inc. Cognitive, emotional, mental and psychological diagnostic engine via the eye
CN105390130B (zh) * 2015-10-23 2019-06-28 施政 一种乐器
JP6776788B2 (ja) 2016-10-11 2020-10-28 ヤマハ株式会社 演奏制御方法、演奏制御装置およびプログラム
JP6708180B2 (ja) * 2017-07-25 2020-06-10 ヤマハ株式会社 演奏解析方法、演奏解析装置およびプログラム
KR102122195B1 (ko) * 2018-03-06 2020-06-12 주식회사 웨이테크 인공지능 합주 시스템 및 인공지능 합주 방법
CN115298733A (zh) * 2020-03-24 2022-11-04 雅马哈株式会社 训练好的模型的建立方法、推定方法、演奏代理的推荐方法、演奏代理的调整方法、训练好的模型的建立系统、推定系统、训练好的模型的建立程序及推定程序
EP4341907A4 (en) * 2021-05-18 2025-01-15 Attune Media Labs, PBC SYSTEMS AND METHODS FOR AUTOMATED REAL-TIME GENERATING AN INTERACTIVE, DISCONTINUOUS, DISCRETE AVATAR

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011095530A (ja) * 2009-10-30 2011-05-12 Casio Computer Co Ltd 自動伴奏装置およびプログラム
JP2015191194A (ja) * 2014-03-28 2015-11-02 パイオニア株式会社 演奏評価システム、サーバ装置、端末装置、演奏評価方法及びコンピュータプログラム
JP2018078975A (ja) * 2016-11-15 2018-05-24 株式会社ｇｌｏｏｐｓ 端末装置、端末装置のゲーム実行方法、ゲーム実行プログラム、及びゲーム実行プログラム記録媒体
JP2019162207A (ja) * 2018-03-19 2019-09-26 富士ゼロックス株式会社 情報処理装置及び情報処理プログラム

Also Published As

Publication number Publication date
CN115298734A (zh) 2022-11-04
JPWO2021193032A1 (https=) 2021-09-30
US20230014736A1 (en) 2023-01-19
US12367854B2 (en) 2025-07-22
JP7388542B2 (ja) 2023-11-29

Similar Documents

Publication Publication Date Title
JP7388542B2 (ja) 演奏エージェントの訓練方法、自動演奏システム、及びプログラム
US20230014315A1 (en) Trained model establishment method, estimation method, performance agent recommendation method, performance agent adjustment method, trained model establishment system, estimation system, trained model establishment program, and estimation program
US10580393B2 (en) Apparatus for analyzing musical performance, performance analysis method, automatic playback method, and automatic player system
US11557269B2 (en) Information processing method
US20190156809A1 (en) Music data processing method and program
JP7383943B2 (ja) 制御システム、制御方法、及びプログラム
JP7718536B2 (ja) 演奏データに対する観衆の評価を推論する方法、システム、及びプログラム
JP7528971B2 (ja) 情報処理方法、情報処理システムおよびプログラム
JP2022115956A (ja) 情報処理方法、情報処理装置およびプログラム
JP7424468B2 (ja) パラメータ推論方法、パラメータ推論システム、及びパラメータ推論プログラム
WO2021186928A1 (ja) 演奏情報に対する評価を推論する方法、システム、及びプログラム
JP2025170154A5 (ja) データ処理方法、データ処理システムおよびプログラム
JP2025170154A (ja) データ出力方法、プログラム、データ出力装置および電子楽器
JP2022149157A (ja) 演奏解析方法、演奏解析システムおよびプログラム
JP7846831B2 (ja) 同期通信のためのオーディオ合成
JP7651901B2 (ja) 画像処理方法、画像処理システムおよびプログラム
JP7845491B2 (ja) データ処理方法、プログラムおよびシステム
JP7615818B2 (ja) 画像処理方法、画像処理システムおよびプログラム
CN121191526A (zh) 音频音色转换方法、电子设备及计算机存储介质
JP2024175437A (ja) 情報処理方法および情報処理装置
Shayda et al. Stanford Research Series: Grand Digital Piano: Multimodal Transfer of Learning of Sound and Touch
WO2023170757A1 (ja) 再生制御方法、情報処理方法、再生制御システムおよびプログラム
Shayda et al. Grand digital piano: multimodal transfer of learning of sound and touch

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21774626

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022509544

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21774626

Country of ref document: EP

Kind code of ref document: A1