WO2021193033A1 - 訓練済みモデルの確立方法、推定方法、演奏エージェントの推薦方法、演奏エージェントの調整方法、訓練済みモデルの確立システム、推定システム、訓練済みモデルの確立プログラム及び推定プログラム - Google Patents

訓練済みモデルの確立方法、推定方法、演奏エージェントの推薦方法、演奏エージェントの調整方法、訓練済みモデルの確立システム、推定システム、訓練済みモデルの確立プログラム及び推定プログラム Download PDF

Info

Publication number
WO2021193033A1
WO2021193033A1 PCT/JP2021/009362 JP2021009362W WO2021193033A1 WO 2021193033 A1 WO2021193033 A1 WO 2021193033A1 JP 2021009362 W JP2021009362 W JP 2021009362W WO 2021193033 A1 WO2021193033 A1 WO 2021193033A1
Authority
WO
WIPO (PCT)
Prior art keywords
performance
satisfaction
performer
data
estimation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/009362
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
陽 前澤
克己 石川
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yamaha Corp
Original Assignee
Yamaha Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yamaha Corp filed Critical Yamaha Corp
Priority to CN202180020523.0A priority Critical patent/CN115298733A/zh
Priority to JP2022509545A priority patent/JP7420220B2/ja
Publication of WO2021193033A1 publication Critical patent/WO2021193033A1/ja
Priority to US17/952,077 priority patent/US20230014315A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10G—REPRESENTATION OF MUSIC; RECORDING MUSIC IN NOTATION FORM; ACCESSORIES FOR MUSIC OR MUSICAL INSTRUMENTS NOT OTHERWISE PROVIDED FOR, e.g. SUPPORTS
    • G10G1/00—Means for the representation of music
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0033—Recording/reproducing or transmission of music for electrophonic musical instruments
    • G10H1/0041—Recording/reproducing or transmission of music for electrophonic musical instruments in coded form
    • G10H1/0058—Transmission between separate instruments or between individual components of a musical system
    • G10H1/0066—Transmission between separate instruments or between individual components of a musical system using a MIDI interface
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/091—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/371—Vital parameter control, i.e. musical instrument control based on body signals, e.g. brainwaves, pulsation, temperature or perspiration; Biometric information
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
    • G10H2220/155—User input interfaces for electrophonic musical instruments
    • G10H2220/441—Image sensing, i.e. capturing images or optical patterns for musical purposes or musical control purposes
    • G10H2220/455—Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
    • G10H2240/075—Musical metadata derived from musical analysis or for use in electrophonic musical instruments
    • G10H2240/085—Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/311—Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation

Definitions

  • the present invention relates to a trained model establishment method, an estimation method, a performance agent recommendation method, a performance agent adjustment method, a trained model establishment system, an estimation system, a trained model establishment program, and an estimation program.
  • Patent Document 1 proposes a technique for evaluating a performance operation by selectively targeting a part of the entire played music.
  • the accuracy of the performance by the performer can be evaluated.
  • the conventional technique has the following problems. That is, in general, a performer often plays (co-stars) with another performer (for example, another person, a performance agent, etc.). In the co-starring, the first performance by the performer and the second performance by another performer are performed in parallel. The second performance performed by the other performers is basically not the same as the first performance. Therefore, it is difficult to estimate the co-starring of the performer or the degree of satisfaction with the co-star from the accuracy of the performance.
  • the present invention has been made in view of the above circumstances on one aspect, and an object of the present invention is to appropriately estimate the satisfaction of the performer of the first performance with respect to the second performance performed together with the first performance by the performer. It is to provide a technique for performing, a technique for recommending a performance agent using the technique, and a technique for adjusting a performance agent.
  • the method of establishing the trained model realized by one or more computers is performed together with the first performance data of the first performance by the performer and the first performance.
  • a plurality of data sets each composed of a combination of the second performance data of the second performance and a satisfaction label configured to indicate the satisfaction of the performer are acquired, and the plurality of data sets are used.
  • perform machine learning of the satisfaction estimation model Provide processing.
  • the result of estimating the satisfaction level of the performer from the first performance data and the second performance data is matched with the satisfaction level indicated by the satisfaction level label. It is configured by training the satisfaction estimation model.
  • the estimation method realized by one or more computers uses the first performance data of the first performance by the performer and the second performance data of the second performance performed together with the first performance.
  • the satisfaction of the performer is estimated from the acquired first performance data and the second performance data, and the satisfaction is calculated. It is equipped with a process that outputs information about the estimated result.
  • the method of recommending a performance agent realized by a computer is to supply a plurality of first performer data related to the first performance to each of the plurality of performance agents.
  • the second performance data of the two performances is generated, and the satisfaction of the performer with respect to each of the plurality of performance agents is estimated by using the trained satisfaction estimation model by the above estimation method, and the estimated plurality of performances are performed.
  • a process is provided in which a recommended performance agent is selected from the plurality of performance agents based on the degree of satisfaction with each of the performance agents.
  • the method of adjusting the performance agent realized by the computer is to supply the performance agent with the data of the first performer related to the first performance, thereby supplying the second performance data of the second performance.
  • the satisfaction estimation model is used to estimate the satisfaction of the performer with respect to the performance agent by the estimation method, and the inside of the performance agent used when generating the second performance data. It has a process to change the value of a parameter. By iteratively performing the generation, the estimation, and the modification, the values of the internal parameters are adjusted so that the satisfaction is high.
  • a technique for appropriately estimating the satisfaction of the performer of the first performance with respect to the second performance performed together with the first performance by the performer, and a technique for recommending a performance agent using the technique can be provided.
  • FIG. 1 shows an example of the configuration of the information processing system according to the first embodiment.
  • FIG. 2 shows an example of the hardware configuration of the performance control device according to the first embodiment.
  • FIG. 3 shows an example of the hardware configuration of the estimation device according to the first embodiment.
  • FIG. 4 shows an example of the software configuration of the information processing system according to the first embodiment.
  • FIG. 5 is a flowchart showing an example of training processing of the satisfaction estimation model according to the first embodiment.
  • FIG. 6 is a flowchart showing an example of the estimation process according to the first embodiment.
  • FIG. 7 is a sequence diagram showing an example of the recommendation process according to the second embodiment.
  • FIG. 8 is a sequence diagram showing an example of the adjustment process according to the third embodiment.
  • FIG. 1 shows an example of the configuration of the information processing system S according to the first embodiment.
  • the information processing system S according to the first embodiment includes a performance control device 100 and an estimation device 300.
  • the information processing system S according to the first embodiment is an example of a trained model establishment system.
  • the information processing system S according to the first embodiment is also an example of an estimation system.
  • the performance control device 100 and the estimation device 300 may be realized by, for example, an information processing device (computer) such as a personal computer, a server, a tablet terminal, or a mobile terminal (for example, a smartphone).
  • the performance control device 100 and the estimation device 300 may be configured to be communicable via the network NW or directly.
  • the performance control device 100 is a computer configured to include a performance agent 160 that controls a performance device 200 such as an automatic player piano to play a musical piece.
  • the performance device 200 may be appropriately configured to perform the second performance according to the second performance data indicating the second performance.
  • the estimation device 300 according to the first embodiment is a computer configured to generate a trained satisfaction estimation model by machine learning. Further, the estimation device 300 is a computer configured to estimate the satisfaction (favorability) of the performer with respect to the co-starring of the performer and the performance agent 160 by using the trained satisfaction estimation model.
  • the process of generating the trained satisfaction estimation model and the process of estimating the satisfaction of the performer using the trained satisfaction estimation model may be executed by the same computer or separately. It may be run by a computer.
  • the "satisfaction" in the present invention means the personal satisfaction of a specific performer.
  • the performer according to the present embodiment typically performs using the electronic musical instrument EM connected to the performance control device 100.
  • the electronic musical instrument EM of the present embodiment may be, for example, an electronic keyboard instrument (electronic piano or the like), an electronic stringed instrument (electric guitar or the like), an electronic wind instrument (wind synthesizer or the like) or the like.
  • the musical instrument used by the performer for performance is not limited to the electronic musical instrument EM.
  • the performer may perform with an acoustic instrument.
  • the performer according to the present embodiment may be a singer of a musical piece that does not use a musical instrument. In this case, the performance by the performer may be performed without using an instrument.
  • the performance by the performer is referred to as "first performance”
  • the performance by a subject (performance agent 160, others, etc.) who is not the performer who performs the first performance is referred to as "second performance”.
  • the information processing system S of the first embodiment includes the first performance data of the first performance for training by the performer, and the second performance of the second performance for training performed together with the first performance.
  • machine learning of the satisfaction estimation model is performed.
  • the satisfaction (true value / correct answer) in which the result of estimating the satisfaction of the performer from the first performance data and the second performance data for training is indicated by the satisfaction label for each data set.
  • Satisfaction estimation model is trained to be suitable.
  • the information processing system S of the first embodiment acquires the first performance data of the first performance by the performer and the second performance data of the second performance performed together with the first performance at the estimation stage, and is subjected to machine learning.
  • the satisfaction of the performer is estimated from the acquired first performance data and the second performance data, and information on the result of estimating the satisfaction is output.
  • the co-starring feature amount is calculated based on the first performance data and the second performance data, and the performance is performed from the calculated co-starring feature amount. It may be configured by estimating the satisfaction of the person.
  • FIG. 2 shows an example of the hardware configuration of the performance control device 100 according to the present embodiment.
  • the CPU 101, the RAM 102, the storage 103, the input unit 104, the output unit 105, the sound collecting unit 106, the imaging unit 107, the transmitting / receiving unit 108, and the drive 109 are electrically operated by the bus B1.
  • a computer connected to.
  • the CPU 101 is composed of one or a plurality of processors for executing various operations in the performance control device 100.
  • the CPU 101 is an example of a processor resource.
  • the type of processor may be appropriately selected depending on the embodiment.
  • the RAM 102 is a volatile storage medium, and operates as a working memory that holds information such as set values used by the CPU 101 and develops various programs.
  • the storage 103 is a non-volatile storage medium that stores various programs and data used by the CPU 101.
  • the RAM 102 and the storage 103 are examples of memory resources that hold programs executed by processor resources.
  • the storage 103 stores various information such as the program 81.
  • the program 81 is a performance control device that generates information processing that generates second performance data indicating a second performance performed in parallel with the first performance of the music by the performer, and information processing that adjusts the value of the internal parameter of the performance agent 160. It is a program for making 100 execute.
  • Program 81 includes a series of instructions for the information processing.
  • the input unit 104 is composed of an input device for receiving an operation on the performance control device 100.
  • the input unit 104 may be composed of one or a plurality of input devices such as a keyboard and a mouse connected to the performance control device 100, for example.
  • the output unit 105 is composed of an output device for outputting various information.
  • the output unit 105 may be composed of one or a plurality of output devices such as a display and a speaker connected to the performance control device 100, for example.
  • Information may be output by, for example, a video signal, a sound signal, or the like.
  • the input unit 104 and the output unit 105 may be integrally configured by an input / output device such as a touch panel display that receives a user's operation on the performance control device 100 and outputs various information.
  • an input / output device such as a touch panel display that receives a user's operation on the performance control device 100 and outputs various information.
  • the sound collecting unit 106 is configured to convert the collected sound into an electric signal and supply it to the CPU 101.
  • the sound collecting unit 106 is composed of, for example, a microphone.
  • the sound collecting unit 106 may be built in the performance control device 100, or may be connected to the performance control device 100 via an interface (not shown).
  • the imaging unit 107 is configured to convert the captured image into an electric signal and supply it to the CPU 101.
  • the image pickup unit 107 is composed of, for example, a digital camera.
  • the imaging unit 107 may be built in the performance control device 100, or may be connected to the performance control device 100 via an interface (not shown).
  • the transmission / reception unit 108 is configured to transmit / receive data to / from another device wirelessly or by wire.
  • the performance control device 100 is connected to the performance device 200 to be controlled, the electronic musical instrument EM used by the performer to play a musical piece, and the estimation device 300 via the transmission / reception unit 108, and data is input. You may send and receive.
  • the transmission / reception unit 108 may include a plurality of modules (for example, a Bluetooth (registered trademark) module, a Wi-Fi (registered trademark) module, a USB (Universal Serial Bus) port, a dedicated port, etc.).
  • the drive 109 is a drive device for reading various information such as programs stored in the storage medium 91.
  • the storage medium 91 electrically, magnetically, optically, mechanically or chemically acts on the information of the program or the like so that the computer or other device, the machine or the like can read various information of the stored program or the like. It is a medium that accumulates by.
  • the storage medium 91 is, for example, a floppy disk, an optical disk (for example, a compact disk, a digital versatile disk, a Blu-ray disk), a magneto-optical disk, a magnetic tape, a non-volatile memory card (for example, a flash memory), or the like. good.
  • the type of the drive 109 may be arbitrarily selected according to the type of the storage medium 91.
  • the program 81 may be stored in the storage medium 91, and the performance control device 100 may read the program 81 from the storage medium 91.
  • Bus B1 is a signal transmission line that electrically connects the hardware components of the performance control device 100 to each other.
  • the components can be omitted, replaced, or added as appropriate according to the embodiment.
  • at least one of the input unit 104, the output unit 105, the sound collecting unit 106, the imaging unit 107, the transmitting / receiving unit 108, and the drive 109 may be omitted.
  • the CPU 301 is composed of one or a plurality of processors for executing various operations in the estimation device 300.
  • the CPU 301 is an example of a processor resource.
  • the type of processor may be appropriately selected depending on the embodiment.
  • the RAM 302 is a volatile storage medium, and operates as a working memory that holds various information such as setting values used by the CPU 301 and develops various programs.
  • the storage 303 is a non-volatile storage medium and stores various programs and data used by the CPU 301.
  • the RAM 302 and the storage 303 are examples of memory resources that hold programs executed by processor resources.
  • the storage 303 stores various information such as the program 83.
  • the program 83 is an information processing device that performs machine learning of the satisfaction estimation model (FIG. 5 described later) and information processing that estimates satisfaction using the trained satisfaction estimation model (FIG. 6 described later). It is a program for making 300 execute.
  • Program 83 includes a series of instructions for the information processing.
  • the instructional portion of program 83, which implements machine learning of the satisfaction estimation model, is an example of a trained model establishment program.
  • the instruction portion of the program 83 for estimating the satisfaction level is an example of the estimation program.
  • the establishment program and the estimation program may be contained in the same file, or may be kept in separate files.
  • the input unit 304 to the imaging unit 307, the drive 310, and the storage medium 93 may be configured in the same manner as the input unit 104, the imaging unit 107, the drive 109, and the storage medium 91 of the performance control device 100.
  • the program 83 may be stored in the storage medium 93, and the estimation device 300 may read the program 83 from the storage medium 93.
  • the biological sensor 308 is configured to acquire biological signals indicating the performer's biological information in time series.
  • the biological information of the performer may be composed of one or a plurality of types of data such as heart rate, sweating amount, and blood pressure, for example.
  • the biosensor 308 may be composed of, for example, a sensor such as a heart rate monitor, a sweat meter, and a sphygmomanometer.
  • the transmission / reception unit 309 is configured to transmit / receive data to / from another device wirelessly or by wire.
  • the estimation device 300 may be connected to the electronic musical instrument EM and the performance control device 100 used when the performer plays a musical piece via the transmission / reception unit 309 to transmit / receive data.
  • the transmission / reception unit 309 may include a plurality of modules like the transmission / reception unit 108.
  • Bus B3 is a signal transmission line that electrically connects the hardware components of the estimation device 300 to each other.
  • the components can be omitted, replaced, or added as appropriate according to the embodiment.
  • at least one of the input unit 304, the output unit 305, the sound collecting unit 306, the imaging unit 307, the biosensor 308, the transmitting / receiving unit 309, and the drive 310 may be omitted.
  • the performance control device 100 has a control unit 150 and a storage unit 180.
  • the control unit 150 is configured to integrally control the operation of the performance control device 100 by the CPU 101 and the RAM 102.
  • the storage unit 180 is configured to store various data used in the control unit 150 by the RAM 102 and the storage 103.
  • the CPU 101 of the performance control device 100 expands the program 81 stored in the storage 103 into the RAM 102, and executes the instructions included in the program 81 expanded in the RAM 102.
  • the performance control device 100 (control unit 150) operates as a computer including the authentication unit 151, the performance acquisition unit 152, the video acquisition unit 153, and the performance agent 160 as software modules.
  • the authentication unit 151 is configured to authenticate the user (performer) in cooperation with an external device such as the estimation device 300.
  • the authentication unit 151 transmits authentication data such as a user identifier and a password input by the user using the input unit 104 to the estimation device 300, and permits the user's access based on the authentication result received from the estimation device 300. Or it is configured to refuse.
  • the external device that authenticates the user may be an authentication server other than the estimation device 300.
  • the authentication unit 151 may be configured to supply the user identifier of the authenticated (access-authorized) user to other software modules.
  • the first performer data may be configured to include at least one of the performance sound, the first performance data, and the image in the first performance by the performer.
  • the performance acquisition unit 152 is configured to acquire first performer data regarding the sound of the first performance by the performer.
  • the performance acquisition unit 152 may acquire the data of the performance sound indicated by the electric signal that the sound collection unit 106 collects and outputs the sound of the first performance as the first performer data.
  • the performance acquisition unit 152 may acquire, for example, first performance data (for example, a MIDI data string with a time stamp) indicating the first performance supplied from the electronic musical instrument EM as the first performer data.
  • the first performer data may be composed of information indicating the characteristics (for example, sound generation time and pitch) of the sound included in the performance, and is a kind of high-dimensional time series data expressing the first performance by the performer. It may be there.
  • the performance acquisition unit 152 is configured to supply the first performer data regarding the acquired sound to the performance agent 160.
  • the performance acquisition unit 152 may be configured to transmit the first performer data regarding the sound to the estimation device 300.
  • the video acquisition unit 153 is configured to acquire the first performer data relating to the video of the first performance by the performer.
  • the video acquisition unit 153 is configured to acquire video data indicating the video of the performer performing the first performance as the first performer data.
  • the image acquisition unit 153 may acquire image data based on an electric signal indicating the image of the performer in the first performance taken by the imaging unit 107 as the first performer data.
  • the video data may be composed of motion data showing the characteristics of the performer's movement in the performance, and may be a kind of high-dimensional time series data expressing the performance by the performer.
  • the motion data is, for example, data obtained by acquiring the whole image or the skeleton (skeleton) of the performer in time series.
  • the image included in the first performer data is not limited to a moving image (moving image), and may be a still image.
  • the video acquisition unit 153 is configured to supply the first performer data regarding the acquired video to the performance agent 160.
  • the video acquisition unit 153 may be configured to transmit the first performer data regarding the acquired video to the estimation device 300.
  • the performance agent 160 is configured to generate second performance data indicating a second performance performed in parallel with the first performance of the performer, and to control the operation of the performance device 200 based on the generated second performance data. Will be done.
  • the performance agent 160 may be configured to automatically perform the second performance based on the first performer data related to the first performance of the performer.
  • the performance agent 160 is, for example, a method disclosed in International Publication No. 2018/070286, "Study on real-time sheet music tracking by acoustic signals and active performance support system” (Shinji Sakou (Nagoya Institute of Technology), Telecommunications Advancement Foundation " It may be configured to perform automatic performance control based on any method such as the method disclosed in "Research Grant Report" No. 31, 2016).
  • the automatic performance (second performance) may be, for example, an accompaniment to the first performance or a counter-melody.
  • the performance agent 160 is in a state at that time (for example, "volume difference between the two (performer and performance agent)", “volume of the performance agent”, “tempo of the performance agent”, “timing difference between the two”, etc.).
  • Actions to be performed according to for example, “increase tempo by 1", “decrease tempo by 1", “decrease tempo by 10", ..., “increase volume by 3", “increase volume by 1", “volume”
  • the performance agent 160 may be appropriately configured to determine an action according to the state at that time based on the plurality of internal parameters, and to change the performance being performed at that time according to the determined action.
  • the performance agent 160 is configured to include a performance analysis unit 161 and a performance control unit 162 according to the calculation model. Non-limiting and general automatic performance control will be illustrated below.
  • the performance analysis unit 161 determines the performance position, which is the position on the music actually played by the performer, based on the first performer data related to the first performance supplied from the performance acquisition unit 152 and the video acquisition unit 153. It is configured to estimate.
  • the estimation of the performance position by the performance analysis unit 161 may be continuously (for example, periodically) executed in parallel with the performance by the performer.
  • the performance analysis unit 161 estimates the performance position of the performer by mutually comparing the series of sounds indicated by the first performer data with the series of notes indicated by the music data for automatic performance. It may be configured.
  • the music data includes reference part data corresponding to the first performance (performer part) by the performer, and automatic part data indicating the second performance (automatic performance part) by the performance agent 160. Any music analysis technique (score alignment technique) may be appropriately adopted for the estimation of the performance position by the performance analysis unit 161.
  • the performance device 200 may be configured to control the performance device 200 so as to execute an automatic performance according to the generated second performance data. That is, the performance control unit 162 operates as a performance data converter that adds an expression to the automatic part data (for example, a MIDI data string with a time stamp) and supplies it to the performance device 200.
  • the expression added here is similar to the human performance expression, for example, the timing of a certain note is slightly shifted forward or backward, an accent is given to a certain note, crescendo or decrescendo is performed over a plurality of notes, and the like. You can.
  • the performance control unit 162 may be configured to supply the generated second performance data to the estimation device 300 as well.
  • the performance device 200 may be appropriately configured to perform a second performance, which is an automatic performance of a musical piece, according to the second performance data supplied from the performance control unit 162.
  • the configuration of the performance agent 160 does not have to be limited to such an example.
  • the performance agent 160 impulsively generates the second performance data based on the first performer data related to the first performance of the performer without using the existing music data, and the generated second performance data is generated. 2
  • the performance device 200 may be configured to perform automatic performance (improvisational performance).
  • the estimation device 300 has a control unit 350 and a storage unit 380.
  • the control unit 350 is configured to integrally control the operation of the estimation device 300 by the CPU 301 and the RAM 302.
  • the storage unit 380 is configured to store various data (particularly, a satisfaction estimation model and an emotion estimation model, which will be described later) used in the control unit 350 by the RAM 302 and the storage 303.
  • the CPU 301 of the estimation device 300 expands the program 83 stored in the storage 303 into the RAM 302, and executes the instructions included in the program 83 expanded in the RAM 302.
  • the estimation device 300 (control unit 350) includes the authentication unit 351 and the performance acquisition unit 352, the reaction acquisition unit 353, the satisfaction acquisition unit 354, the data preprocessing unit 355, the model training unit 356, and the satisfaction estimation unit 357. It operates as a computer equipped with a satisfaction output unit 358 as a software module.
  • the authentication unit 351 is configured to authenticate the user (performer) in cooperation with the performance control device 100. In one example, the authentication unit 351 determines whether or not the authentication data provided by the performance control device 100 matches the authentication data stored in the storage unit 380, and determines the authentication result (permission or denial) of the performance control device. It is configured to send to 100.
  • the performance acquisition unit 352 is configured to acquire (receive) the first performance data of the performance by the performer and the second performance data of the performance by the performance device 200 controlled by the performance agent 160.
  • the first performance data and the second performance data are data indicating a note sequence, respectively, and may be configured to define the sounding timing, sound length, pitch, and intensity of each note.
  • the first performance data is the performance data of the actual performance by the performer, or the performance data including the features extracted from the actual performance by the performer (for example, the extracted features are converted into plain performance data. It may be the performance data generated by giving).
  • the performance acquisition unit 352 is configured to acquire, for example, the first performance data indicating the first performance supplied from the electronic musical instrument EM directly from the electronic musical instrument EM or via the performance control device 100. It's okay.
  • the performance acquisition unit 352 acquires a performance sound indicating the first performance using the sound collecting unit 306 or via the performance control device 100, and the first performance is performed based on the acquired performance sound data. It may be configured to generate data.
  • the performance acquisition unit 352 extracts the features from the actual performance by the performer, and adds the extracted features to the performance data to which the expression is not given to obtain the first performance data. It may be configured to generate.
  • the performance acquisition unit 352 may be configured to acquire, for example, the second performance data indicating the second performance generated by the performance agent 160 from the performance control device 100 or the performance device 200.
  • the performance acquisition unit 352 is configured to acquire the performance sound indicating the second performance by using the sound collecting unit 306 and generate the second performance data based on the acquired performance sound data. You can.
  • the performance acquisition unit 352 may be configured to associate the acquired first performance data and the second performance data with a common time axis and store them in the storage unit 380.
  • the first performance indicated by the first performance data at a certain time and the second performance indicated by the second performance data at the same time are two performances (that is, ensemble) performed at the same time.
  • the performance acquisition unit 352 may be configured to associate the user identifier of the performer authenticated by the authentication unit 351 with the first performance data and the second performance data.
  • the reaction acquisition unit 353 is configured to acquire reaction data indicating the reaction of the performer performing the first performance.
  • the performer's reaction may be configured to include at least one of the performer's audio, image, and biometric information in the co-star.
  • the reaction acquisition unit 353 may be configured to acquire reaction data based on a performer image that reflects the reaction (facial expression, etc.) of the performer during the co-starring image taken by the imaging unit 307.
  • the performer image is an example of an image of the performer.
  • the reaction acquisition unit 353 may be configured to acquire reaction data based on at least one of the performance (first performance) reflecting the reaction of the performer and the biological information.
  • the first performance used to acquire the reaction data may be, for example, the first performance data acquired by the performance acquisition unit 352.
  • the biological information used to acquire the reaction data is composed of one or a plurality of biological signals (for example, heart rate, sweating amount, blood pressure, etc.) acquired by the biological sensor 308 at the time of the first performance by the performer. good.
  • Satisfaction acquisition unit 354 is configured to acquire a satisfaction label indicating the performer's personal satisfaction (true value / correct answer) in co-starring with the performance agent 160 (performance device 200).
  • the satisfaction indicated by the satisfaction label may be estimated from the reaction data acquired by the reaction acquisition unit 353.
  • the storage unit 380 may hold the correspondence table data showing the correspondence between the value indicated by the reaction data and the satisfaction level, and the satisfaction acquisition unit 354 may use the reaction data based on the correspondence table data. It may be configured to obtain satisfaction from the indicated performer's reaction.
  • an emotion estimation model may be used to estimate satisfaction. The emotion estimation model may be appropriately configured to have the ability to estimate satisfaction from the player's reaction.
  • the emotion estimation model may consist of a trained machine learning model generated by machine learning.
  • the emotion estimation model for example, any machine learning model such as a neural network may be adopted.
  • Such a trained emotion estimation model is, for example, a machine using a plurality of learning data sets composed of a combination of training reaction data indicating the player's reaction and a correct answer label indicating the true value of satisfaction. It can be generated by learning.
  • the satisfaction acquisition unit 354 inputs the reaction data indicating the reaction of the performer into the trained emotion estimation model, and executes the arithmetic processing of the trained emotion estimation model to execute the arithmetic processing of the trained emotion estimation model. It may be configured to obtain the result of estimating satisfaction from.
  • the trained emotion estimation model may be stored in the storage unit 380.
  • the satisfaction acquisition unit 354 generates a data set by associating the satisfaction label with the first performance data and the second performance data acquired by the performance acquisition unit 352, and stores each generated data set in the storage unit 380. It may be configured to do so.
  • the data preprocessing unit 355 so that the data (first performance data, second performance data, etc.) input to the satisfaction estimation model for estimating the satisfaction of the performer is in a format suitable for the calculation of the satisfaction estimation model. Is configured to be preprocessed.
  • the data preprocessing unit 355 uses an arbitrary method (for example, phrase detection based on chord progression, phrase detection using a neural network, etc.) to generate a plurality of first performance data and second performance data at a common position (time). It may be configured to break down into phrases of.
  • the data preprocessing unit 355 may be configured to analyze the first performance data and the second performance data related to the co-starring and calculate the co-starring feature amount.
  • the co-starring feature amount is data relating to the co-starring of the first performance by the performer and the second performance by the performance agent 160, and may be composed of values expressing the following features, for example.
  • the average and variance of the timing lag is the average of the similarity (eg, Euclidean distance) for each change type of the shape of the change curve classified and normalized by the change type (eg, Ritaldand, Accelerando, etc.).
  • the "follow-up degree” is, for example, a value corresponding to the "follow-up coefficient” or “coupling coefficient” disclosed in International Publication No. 2018/016637.
  • the "pitch sequence histogram” shows a frequency distribution that counts the number of notes for each pitch.
  • the data preprocessing unit 355 is configured to supply the preprocessed data to the model training unit 356.
  • the data preprocessing unit 355 is configured to supply the preprocessed data to the satisfaction estimation unit 357.
  • the model training unit 356 uses the first performance data and the second performance data of each data set supplied from the data preprocessing unit 355 as training data (input data), and uses the satisfaction label as a teacher signal (correct answer data). It is configured to perform machine learning of the satisfaction estimation model.
  • the training data may be composed of co-starring features calculated from the first performance data and the second performance data. In each data set, the first performance data and the second performance data may be acquired in a state of being converted into co-starring features in advance.
  • the satisfaction estimation model may be composed of any machine learning model having a plurality of parameters.
  • a feedforward neural network composed of a multi-layer perceptron, a hidden Markov model (HMM), or the like
  • Other machine learning models that make up the satisfaction estimation model include, for example, a recurrent neural network (RNN) adapted to time-series data, a derived configuration (long / short-term memory (LSTM)), and a gated recurrent unit. (GRU) etc.), convolutional neural network (CNN) and the like may be used.
  • RNN recurrent neural network
  • LSTM long / short-term memory
  • GRU gated recurrent unit.
  • CNN convolutional neural network
  • the satisfaction level (true value / true value /) is shown by the satisfaction level label as the result of estimating the satisfaction level of the performer from the first performance data and the second performance data using the satisfaction estimation model. It is composed by training the satisfaction estimation model so that it conforms to the correct answer).
  • the result of estimating the satisfaction of the performer from the co-starring features calculated based on the first performance data and the second performance data is indicated by the satisfaction label. It may be configured by training the satisfaction estimation model to be suitable for the degree.
  • the machine learning method may be appropriately selected depending on the type of machine learning model to be adopted.
  • the trained satisfaction estimation model generated by machine learning may be appropriately stored in a storage area such as a storage unit 380 in the form of learning result data.
  • the satisfaction estimation unit 357 includes a trained satisfaction estimation model generated by the model training unit 356.
  • the satisfaction estimation unit 357 is configured to estimate the satisfaction of the performer from the first performance data and the second performance data acquired at the time of inference by using the trained satisfaction estimation model.
  • the estimation is to estimate the satisfaction of the performer from the co-starring features calculated based on the first performance data and the second performance data by using the trained satisfaction estimation model. It may be composed of.
  • the satisfaction estimation unit 357 inputs the co-star feature amount supplied from the data preprocessing unit 355 into the trained satisfaction estimation model as input data, and executes arithmetic processing of the trained satisfaction estimation model. ..
  • the satisfaction estimation unit 357 acquires the output corresponding to the result of estimating the satisfaction of the performer from the input co-starring features from the trained satisfaction estimation model.
  • the estimated satisfaction level (satisfaction level estimation result) is supplied to the satisfaction level output unit 358.
  • the satisfaction level output unit 358 is configured to output information regarding the result of estimating the satisfaction level (estimated satisfaction level) by the satisfaction level estimation unit 357.
  • the output destination and output format may be appropriately selected according to the embodiment.
  • the output of the information regarding the satisfaction estimation result may be configured by simply outputting the information indicating the estimation result to an output device such as the output unit 305.
  • outputting information about the satisfaction estimation result may be configured by executing various control processes based on the satisfaction estimation result. A specific control example by the satisfaction output unit 358 will be described later.
  • each software module of the performance control device 100 and the estimation device 300 is realized by a general-purpose CPU.
  • some or all of the software modules may be implemented by one or more dedicated processors.
  • Each of the above modules may be realized as a hardware module.
  • software modules may be omitted, replaced, or added as appropriate according to the embodiment.
  • FIG. 5 is a flowchart showing an example of training processing of the satisfaction estimation model by the information processing system S according to the present embodiment.
  • the following processing procedure is an example of how to establish a trained model.
  • the following processing procedure is only an example, and each step may be changed as much as possible. Further, with respect to the following processing procedures, steps may be omitted, replaced, and added as appropriate according to the embodiment.
  • step S510 the CPU 301 of the estimation device 300 is configured to show the first performance data of the first performance by the performer, the second performance data of the second performance performed together with the first performance, and the satisfaction of the performer. Acquire a plurality of data sets each composed of a combination of satisfaction labels.
  • the CPU 301 may store each acquired data set in the storage unit 380.
  • the CPU 301 may operate as the performance acquisition unit 352 and acquire the first performance data of the first performance and the second performance data of the second performance by the performer.
  • the second performance may be a performance by a performance agent 160 (performance device 200) that co-stars with the performer.
  • the CPU 101 of the performance control device 100 operates as the performance analysis unit 161 and the performance control unit 162, so that the performance agent 160 automatically performs the second performance based on the first performer data related to the first performance of the performer. You can do it.
  • the CPU 101 may operate as at least one of the performance acquisition unit 152 and the video acquisition unit 153 to acquire the first performer data.
  • the acquired first performer data may be configured to include at least one of a performance sound, a first performance data, and an image in the first performance by the performer.
  • the image may be appropriately acquired so as to capture the performer during the first performance.
  • the image may be a moving image (video) or a still image.
  • the CPU 301 may appropriately acquire a satisfaction label.
  • the CPU 301 may directly acquire the satisfaction label by the input of the performer via an input device such as the input unit 304.
  • the CPU 301 may obtain satisfaction from the reaction of the performer during the first performance indicated by the first performance data for training.
  • the CPU 301 operates as the reaction acquisition unit 353, acquires reaction data indicating the reaction of the performer at the time of the first performance, and supplies the acquired reaction data to the satisfaction acquisition unit 354.
  • the CPU 301 may acquire the satisfaction level from the reaction data by an arbitrary method (for example, calculation by a predetermined algorithm).
  • the CPU 301 may estimate the satisfaction level from the reaction of the performer indicated by the reaction data by using the emotion estimation model.
  • Satisfaction labels may be configured to indicate estimated satisfaction.
  • the "during the first performance” may include the period during the first performance and the period during which the lingering sound remains after the end of the first performance.
  • the performer's reaction may include at least one of the performer's audio, image, and biometric information in the co-star.
  • step S520 the CPU 301 operates as the data pre-processing unit 355, and performs pre-processing on the first performance data and the second performance data of each data set supplied from the performance acquisition unit 352.
  • This preprocessing includes calculating the co-starring feature amount based on the first performance data and the second performance data of each data set.
  • the CPU 301 supplies the preprocessed co-star feature amount and the satisfaction label to the model training unit 356. If the first performance data and the second performance data of each data set obtained in step S510 are converted into co-starring features in advance, the processing of step S520 may be omitted.
  • step S530 the CPU 301 operates as the model training unit 356, and machine learning of the satisfaction estimation model is performed using each acquired data set.
  • the CPU 301 uses a satisfaction label to estimate the satisfaction of the performer from the co-starring features calculated based on the first performance data and the second performance data for each data set. Satisfaction estimation models may be trained to be compatible with. As a result of this machine learning, a trained satisfaction estimation model that has acquired the ability to estimate the satisfaction of the performer from the first performance data and the second performance data (co-starring features) is generated.
  • step S540 the CPU 301 saves the result of the machine learning.
  • the CPU 301 may generate learning result data indicating a trained satisfaction estimation model, and store the generated learning result data in a storage area such as a storage unit 380.
  • the CPU 301 may update the learning result data stored in the storage area such as the storage unit 380 with the newly generated learning result data.
  • the training process of the satisfaction estimation model related to this operation example is completed.
  • the training process may be executed periodically, or may be executed in response to a request from the user (performance control device 100).
  • the CPU 101 of the performance control device 100 and the CPU 301 of the estimation device 300 may operate as authentication units (151, 351), respectively, to authenticate the performer. This may collect a dataset of authenticated performers and generate a trained satisfaction estimation model.
  • FIG. 6 is a flowchart showing an example of estimation processing by the information processing system S according to the present embodiment.
  • the following processing procedure is an example of the estimation method. However, the following processing procedure is only an example, and each step may be changed as much as possible. Further, with respect to the following processing procedures, steps may be omitted, replaced, and added as appropriate according to the embodiment.
  • step S610 the CPU 301 of the estimation device 300 operates as a performance acquisition unit 352, and acquires and acquires the first performance data of the first performance by the performer and the second performance data of the second performance performed together with the first performance.
  • the generated first performance data and second performance data are supplied to the data preprocessing unit 355.
  • the second performance in the estimation stage may be performed by the performance agent 160 (performance device 200) co-starring with the performer.
  • step S620 the CPU 301 operates as the data pre-processing unit 355, and executes pre-processing on the first performance data and the second performance data supplied from the performance acquisition unit 352.
  • This preprocessing includes calculating the co-starring feature amount based on the acquired first performance data and second performance data.
  • the CPU 301 supplies the preprocessed data (co-starring feature amount) to the satisfaction estimation unit 357.
  • the calculation of the co-starring feature amount may be executed in advance by another computer. In this case, the process of step S620 may be omitted.
  • step S630 the CPU 301 operates as the satisfaction estimation unit 357, and uses the trained satisfaction estimation model generated by the machine learning, based on the acquired first performance data and the second performance data.
  • the satisfaction level of the performer is estimated from the calculated co-starring features.
  • the CPU 301 inputs the co-starring feature amount supplied from the data preprocessing unit 355 as input data to the trained satisfaction estimation model held in the storage unit 380, and trains the satisfaction estimation model. Executes the arithmetic processing of. As a result of this arithmetic processing, the CPU 301 acquires an output corresponding to the result of estimating the personal satisfaction of the performer from the co-starring features from the trained satisfaction estimation model.
  • the estimated satisfaction level is supplied from the satisfaction level estimation unit 357 to the satisfaction level output unit 358.
  • step S640 the CPU 301 operates as the satisfaction output unit 358 and outputs information regarding the result of estimating the satisfaction.
  • the output destination and output format may be appropriately selected according to the embodiment.
  • the CPU 301 may output the information indicating the estimation result to the output device such as the output unit 305 as it is.
  • the CPU 301 may execute various control processes as the output process based on the result of estimating the satisfaction level. Specific examples of the control process will be described in detail in other embodiments.
  • steps S610 to S640 may be executed in real time in parallel with the first performance data and the second performance data being input to the estimation device 300 in response to the performers performing together.
  • the processes of steps S610 to S640 may be executed ex post facto with respect to the first performance data and the second performance data stored in the estimation device 300 or the like after the co-starring is performed.
  • the training process generates a trained satisfaction estimation model capable of appropriately estimating the satisfaction of the performer of the first performance with respect to the second performance performed together with the first performance by the performer. be able to. Further, in the above estimation process, the satisfaction of the performer can be appropriately estimated by using the trained satisfaction estimation model thus generated.
  • step S520 and step S620 by converting the input data (first performance data and second performance data) for the satisfaction estimation model into the co-starring feature amount by the preprocessing of step S520 and step S620, the amount of information of the input data is reduced and the satisfaction is satisfied.
  • the degree estimation model will be able to accurately capture the characteristics of co-starring. Therefore, the satisfaction level can be estimated more appropriately, and the calculation processing load of the satisfaction level estimation model can be reduced.
  • the second performance may be automatically performed by the performance agent 160 based on the first performer data related to the first performance by the performer.
  • the first performer data may include at least one of a performance sound, performance data, and an image in the first performance by the performer.
  • the performance agent 160 can automatically generate the second performance data that matches the first performance, so that the time and effort required to generate the second performance data can be reduced, and the second performance can be performed through the second performance. It is possible to generate a trained satisfaction estimation model capable of estimating the satisfaction of the performer with respect to the performance agent 160.
  • the satisfaction level indicated by the satisfaction level label may be obtained from the reaction of the performer.
  • An emotion estimation model may be used to obtain satisfaction. As a result, it is possible to reduce the time and effort required to acquire the plurality of data sets. Therefore, it is possible to reduce the cost required for machine learning of the satisfaction estimation model.
  • the information processing system S according to the first embodiment generates a trained satisfaction estimation model by machine learning, and uses the generated trained satisfaction estimation model to perform an individual performer with respect to the performance agent 160. It is configured to estimate the degree of satisfaction.
  • the information processing system S estimates the satisfaction of the performer with respect to the plurality of performance agents, and based on the estimated satisfaction, the performance agent suitable for the performer from among the plurality of performance agents. Is configured to recommend.
  • one performance control device 100 may include a plurality of performance agents 160.
  • each of the plurality of performance control devices 100 may include one or more performance agents 160.
  • one performance control device 100 adopts a configuration having a plurality of performance agents 160. Except for these points, the second embodiment may be configured in the same manner as the first embodiment.
  • FIG. 7 is a sequence diagram showing an example of recommendation processing by the information processing system S according to the second embodiment.
  • the following processing procedure is an example of a performance agent recommendation method. However, the following processing procedure is only an example, and each step may be changed as much as possible. Further, with respect to the following processing procedures, steps may be omitted, replaced, and added as appropriate according to the embodiment.
  • step S710 the CPU 101 of the performance control device 100 supplies the first performer data related to the first performance by the performer to each of the plurality of performance agents 160, so that a plurality of cases corresponding to each performance agent 160 are provided. Generates the second performance data of the second performance of. More specifically, the CPU 101 operates as the performance analysis unit 161 and the performance control unit 162 of each performance agent 160, as in the first embodiment, and corresponds to each performance agent 160 from the first performer data. 2 Generate performance data. The CPU 101 may cause the performance device 200 to perform an automatic performance (second performance) by appropriately supplying the second performance data of each performance agent 160 to the performance device 200. The generated second performance data of each performance agent 160 is supplied to the estimation device 300.
  • second performance automatic performance
  • step S730 the CPU 301 operates as a data preprocessing unit 355 and a satisfaction estimation unit 357, and estimates the performer's satisfaction with the second performance of each performance agent 160 using the trained satisfaction estimation model. do.
  • the process of estimating the satisfaction level for each performance agent 160 in step S720 may be the same as the process of steps S620 and S630 in the first embodiment.
  • step S740 the CPU 301 of the estimation device 300 operates as the satisfaction output unit 358, and selects a recommended performance agent from the plurality of performance agents 160 based on the estimated satisfaction with each of the plurality of performance agents 160. do.
  • the CPU 301 may select a performance agent 160 having the highest degree of satisfaction or a predetermined number of performance agents 160 selected in order from the one with the highest degree of satisfaction as a performance agent to recommend to a user (performer).
  • the CPU 301 displays the recommended performance agent 160 on the output unit 305 of the estimation device 300 (or the output unit 105 of the performance control device 100) by a message.
  • the avatar corresponding to the recommended performance agent 160 may be displayed. The user may select a performance agent to co-star with himself according to or with reference to this recommendation.
  • the satisfaction of the performer with respect to each of the plurality of performance agents 160 can be estimated by using the trained satisfaction estimation model generated by the machine learning. Then, by using the estimation result of the satisfaction level, the performance agent 160 having a high possibility of matching the attributes of the performer can be recommended to the performer.
  • the information processing system S uses the generated trained satisfaction estimation model to estimate the satisfaction of the performer with respect to the performance agent 160, and to improve the satisfaction of the performer. Is configured to adjust the value of the internal parameters of the playing agent 160. Except for this point, the third embodiment may be configured in the same manner as the first embodiment.
  • FIG. 8 is a sequence diagram showing an example of adjustment processing by the information processing system S according to the third embodiment.
  • the following processing procedure is an example of a performance agent adjustment method. However, the following processing procedure is only an example, and each step may be changed as much as possible. Further, with respect to the following processing procedures, steps may be omitted, replaced, and added as appropriate according to the embodiment.
  • step S810 the CPU 101 of the performance control device 100 supplies the performance agent 160 with the first performer data related to the first performance by the performer, thereby generating the second performance data of the second performance.
  • the process of step S810 may be the same as the process of generating the second performance data by each performance agent 160 in step S710.
  • the CPU 101 may cause the performance device 200 to execute an automatic performance (second performance) by appropriately supplying the generated second performance data to the performance device 200.
  • the generated second performance data is supplied to the estimation device 300.
  • step S820 the CPU 301 of the estimation device 300 operates as the performance acquisition unit 352, and acquires the first performance data of the first performance by the performer and the second performance data generated in step S810.
  • the first performance data and the second performance data may be acquired in the same manner as in step S610 of the first embodiment.
  • step S830 the CPU 301 operates as a data preprocessing unit 355 and a satisfaction estimation unit 357, and uses a trained satisfaction estimation model to estimate the performer's satisfaction with the second performance of the performance agent 160. ..
  • the process of estimating the satisfaction level with respect to the performance agent 160 in step S830 may be the same as the process of steps S620 and S630 in the first embodiment.
  • the CPU 301 operates as the satisfaction output unit 358 and supplies information indicating the result of estimating the satisfaction to the performance control device 100.
  • step S840 the CPU 101 of the performance control device 100 changes the value of the internal parameter of the performance agent 160 used when generating the second performance data.
  • the information processing system S according to the third embodiment is estimated to be satisfied by iteratively executing the above-mentioned generation (step S810), estimation (step S830), and modification (step S840).
  • the value of the internal parameter of the performance agent 160 is adjusted so that the degree becomes high.
  • the CPU 101 may stochastically and gradually change the value of each of the plurality of internal parameters of the performance agent 160.
  • the CPU 101 discards the value of the internal parameter used in the previous iterative process and discards the value of the internal parameter used in the previous iterative process.
  • the value of the internal parameter in the process may be adopted.
  • the information processing system S is inside the performance agent 160 so that the estimated satisfaction level is increased by repeating the above series of processes by an arbitrary method (for example, value iterative method, policy iterative method, etc.). You may adjust the value of the parameter.
  • the satisfaction of the performer with respect to the performance agent 160 can be estimated by using the trained satisfaction estimation model generated by the machine learning. Then, by using the estimation result of the satisfaction level, the value of the internal parameter of the performance agent 160 can be adjusted so that the satisfaction level of the performer with respect to the second performance by the performance agent 160 is improved. As a result, it is possible to reduce the time and effort required to generate a performance agent 160 suitable for the performer.
  • the second performance may be automatically performed by the performance agent 160.
  • the second performance does not have to be limited to such an example.
  • the second performance may be performed by another person (second performer) other than the performer who performs the first performance.
  • the generated trained satisfaction estimation model can be used to appropriately estimate the satisfaction of a performer with respect to the second performance by another actual performer.
  • the information processing system S includes a performance control device 100, a performance device 200, an estimation device 300, and an electronic musical instrument EM as separate devices.
  • a performance control device 100 may be integrally configured.
  • the performance control device 100 and the performance device 200 may be integrally configured.
  • the performance control device 100 and the estimation device 300 may be integrally configured.
  • the estimation device 300 is configured to execute both the training process and the estimation process.
  • the training process and the estimation process may be performed by separate computers.
  • the trained satisfaction estimation model (learning result data) may be provided from the first computer that executes the training process to the second computer that executes the estimation process at an arbitrary timing.
  • the number of the first computer and the second computer may be appropriately determined according to the embodiment.
  • the second computer can perform the estimation process using the trained satisfaction estimation model provided by the first computer.
  • Each of the above storage media (91, 93) may be composed of a non-transient recording medium that can be read by a computer. Further, the program (81, 83) may be supplied via a transmission medium or the like.
  • the "non-transient recording medium that can be read by a computer” is, for example, a computer system that constitutes a server, a client, or the like when a program is transmitted via a communication network such as the Internet or a telephone line. It may include a recording medium that holds a program for a certain period of time, such as an internal volatile memory (for example, DRAM (Dynamic Random Access Memory)).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Electrophonic Musical Instruments (AREA)
  • Auxiliary Devices For Music (AREA)
PCT/JP2021/009362 2020-03-24 2021-03-09 訓練済みモデルの確立方法、推定方法、演奏エージェントの推薦方法、演奏エージェントの調整方法、訓練済みモデルの確立システム、推定システム、訓練済みモデルの確立プログラム及び推定プログラム Ceased WO2021193033A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN202180020523.0A CN115298733A (zh) 2020-03-24 2021-03-09 训练好的模型的建立方法、推定方法、演奏代理的推荐方法、演奏代理的调整方法、训练好的模型的建立系统、推定系统、训练好的模型的建立程序及推定程序
JP2022509545A JP7420220B2 (ja) 2020-03-24 2021-03-09 訓練済みモデルの確立方法、推定方法、演奏エージェントの推薦方法、演奏エージェントの調整方法、訓練済みモデルの確立システム、推定システム、訓練済みモデルの確立プログラム及び推定プログラム
US17/952,077 US20230014315A1 (en) 2020-03-24 2022-09-23 Trained model establishment method, estimation method, performance agent recommendation method, performance agent adjustment method, trained model establishment system, estimation system, trained model establishment program, and estimation program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020052757 2020-03-24
JP2020-052757 2020-03-24

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/952,077 Continuation US20230014315A1 (en) 2020-03-24 2022-09-23 Trained model establishment method, estimation method, performance agent recommendation method, performance agent adjustment method, trained model establishment system, estimation system, trained model establishment program, and estimation program

Publications (1)

Publication Number Publication Date
WO2021193033A1 true WO2021193033A1 (ja) 2021-09-30

Family

ID=77891460

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/009362 Ceased WO2021193033A1 (ja) 2020-03-24 2021-03-09 訓練済みモデルの確立方法、推定方法、演奏エージェントの推薦方法、演奏エージェントの調整方法、訓練済みモデルの確立システム、推定システム、訓練済みモデルの確立プログラム及び推定プログラム

Country Status (4)

Country Link
US (1) US20230014315A1 (https=)
JP (1) JP7420220B2 (https=)
CN (1) CN115298733A (https=)
WO (1) WO2021193033A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210005173A1 (en) * 2018-03-23 2021-01-07 Yamaha Corporation Musical performance analysis method and musical performance analysis apparatus

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7147384B2 (ja) * 2018-09-03 2022-10-05 ヤマハ株式会社 情報処理方法および情報処理装置
WO2021186928A1 (ja) * 2020-03-17 2021-09-23 ヤマハ株式会社 演奏情報に対する評価を推論する方法、システム、及びプログラム
WO2021193032A1 (ja) * 2020-03-23 2021-09-30 ヤマハ株式会社 演奏エージェントの訓練方法、自動演奏システム、及びプログラム
JP7714543B2 (ja) * 2020-06-09 2025-07-29 株式会社Ｎｔｔドコモ 推奨情報提供装置
JP7682175B2 (ja) * 2020-06-09 2025-05-23 株式会社Ｎｔｔドコモ 予測装置

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018078975A (ja) * 2016-11-15 2018-05-24 株式会社ｇｌｏｏｐｓ 端末装置、端末装置のゲーム実行方法、ゲーム実行プログラム、及びゲーム実行プログラム記録媒体
JP2019162207A (ja) * 2018-03-19 2019-09-26 富士ゼロックス株式会社 情報処理装置及び情報処理プログラム
JP2019191937A (ja) * 2018-04-25 2019-10-31 Kddi株式会社 感情推測方法、感情推測装置及びプログラム

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3900187B2 (ja) * 1999-08-09 2007-04-04 ヤマハ株式会社 演奏データ作成装置
JP3945429B2 (ja) * 2003-03-19 2007-07-18 ヤマハ株式会社 自動演奏装置および自動演奏プログラム
US9798805B2 (en) * 2012-06-04 2017-10-24 Sony Corporation Device, system and method for generating an accompaniment of input music data
JP2015191190A (ja) * 2014-03-28 2015-11-02 パイオニア株式会社 演奏評価システム、サーバ装置、端末装置、演奏評価方法及びコンピュータプログラム
KR102122195B1 (ko) * 2018-03-06 2020-06-12 주식회사 웨이테크 인공지능 합주 시스템 및 인공지능 합주 방법
WO2021193032A1 (ja) * 2020-03-23 2021-09-30 ヤマハ株式会社 演奏エージェントの訓練方法、自動演奏システム、及びプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018078975A (ja) * 2016-11-15 2018-05-24 株式会社ｇｌｏｏｐｓ 端末装置、端末装置のゲーム実行方法、ゲーム実行プログラム、及びゲーム実行プログラム記録媒体
JP2019162207A (ja) * 2018-03-19 2019-09-26 富士ゼロックス株式会社 情報処理装置及び情報処理プログラム
JP2019191937A (ja) * 2018-04-25 2019-10-31 Kddi株式会社 感情推測方法、感情推測装置及びプログラム

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210005173A1 (en) * 2018-03-23 2021-01-07 Yamaha Corporation Musical performance analysis method and musical performance analysis apparatus
US11869465B2 (en) * 2018-03-23 2024-01-09 Yamaha Corporation Musical performance analysis method and musical performance analysis apparatus

Also Published As

Publication number Publication date
US20230014315A1 (en) 2023-01-19
CN115298733A (zh) 2022-11-04
JP7420220B2 (ja) 2024-01-23
JPWO2021193033A1 (https=) 2021-09-30

Similar Documents

Publication Publication Date Title
JP7420220B2 (ja) 訓練済みモデルの確立方法、推定方法、演奏エージェントの推薦方法、演奏エージェントの調整方法、訓練済みモデルの確立システム、推定システム、訓練済みモデルの確立プログラム及び推定プログラム
EP3803846B1 (en) Autonomous generation of melody
JP7388542B2 (ja) 演奏エージェントの訓練方法、自動演奏システム、及びプログラム
JP4876207B2 (ja) 認知機能障害危険度算出装置、認知機能障害危険度算出システム、及びプログラム
EP3454328A1 (en) Computer implemented method for providing feedback of harmonic content relating to music track
KR102495888B1 (ko) 사운드를 출력하기 위한 전자 장치 및 그의 동작 방법
CN112992109B (zh) 辅助歌唱系统、辅助歌唱方法及其非瞬时计算机可读取记录媒体
JP7383943B2 (ja) 制御システム、制御方法、及びプログラム
JP7718536B2 (ja) 演奏データに対する観衆の評価を推論する方法、システム、及びプログラム
CN113674723B (zh) 一种音频处理方法、计算机设备及可读存储介质
Fu et al. Ultrasr: Silent speech reconstruction via acoustic sensing
JP7424468B2 (ja) パラメータ推論方法、パラメータ推論システム、及びパラメータ推論プログラム
US11397799B2 (en) User authentication by subvocalization of melody singing
Liu et al. Humtrans: A novel open-source dataset for humming melody transcription and beyond
Papiotis A computational approach to studying interdependence in string quartet performance
JP7846831B2 (ja) 同期通信のためのオーディオ合成
US10861428B2 (en) Technologies for generating a musical fingerprint
JP7651901B2 (ja) 画像処理方法、画像処理システムおよびプログラム
Zhu et al. TimbreSense: Timbre Abnormality Detection for Bel Canto with Smart Devices
KR20210081704A (ko) 음원 분석 방법
JP4319054B2 (ja) 利用者の声域を追跡的に分析して演奏キーに反映させる通信カラオケ応用システム
JP7845491B2 (ja) データ処理方法、プログラムおよびシステム
JP7615818B2 (ja) 画像処理方法、画像処理システムおよびプログラム
Bohak et al. Transcription of polyphonic vocal music with a repetitive melodic structure
CN120727206A (zh) 基于多模态分析的心理疗愈音乐推荐方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21776230

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022509545

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21776230

Country of ref document: EP

Kind code of ref document: A1