WO2022044314A1 - 学習装置、学習方法および学習プログラム - Google Patents

学習装置、学習方法および学習プログラム Download PDF

Info

Publication number
WO2022044314A1
WO2022044314A1 PCT/JP2020/032848 JP2020032848W WO2022044314A1 WO 2022044314 A1 WO2022044314 A1 WO 2022044314A1 JP 2020032848 W JP2020032848 W JP 2020032848W WO 2022044314 A1 WO2022044314 A1 WO 2022044314A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature amount
feature
candidate
learning
objective function
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/032848
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
力 江藤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to JP2022545246A priority Critical patent/JP7529028B2/ja
Priority to PCT/JP2020/032848 priority patent/WO2022044314A1/ja
Priority to US18/023,225 priority patent/US20230306270A1/en
Publication of WO2022044314A1 publication Critical patent/WO2022044314A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/092—Reinforcement learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks

Definitions

  • the present invention relates to a learning device, a learning method, and a learning program for performing reverse reinforcement learning.
  • Non-Patent Document 1 discloses a technique for selecting a feature amount based on “Teaching Risk”. In the method described in Non-Patent Document 1, ideal parameters in the objective function are assumed and compared with the parameters of the learning process, and the feature quantity that makes the difference between the two parameters smaller is selected as the important feature quantity.
  • Non-Patent Document 1 The method described in Non-Patent Document 1 is premised on assuming ideal parameters, but in the first place, the method itself for deriving such ideal parameters is unclear. Therefore, it is difficult to use the method described in Non-Patent Document 1 as it is for selecting the feature amount of reverse reinforcement learning.
  • an object of the present invention is to provide a learning device, a learning method, and a learning program that can support the selection of the feature amount of the objective function used in the inverse reinforcement learning.
  • the learning device executes the first inverse reinforcement learning to derive each weight of the candidate features included in the first objective function by the inverse reinforcement learning using the candidate features which are a plurality of candidate features.
  • the feature amount that is estimated that the reward expressed using that feature amount is closest to the ideal reward result. It is characterized by having a feature amount selection unit for selecting a feature amount and a second inverse reinforcement learning execution unit for generating a second objective function by inverse reinforcement learning using the selected feature amount.
  • each weight of the candidate feature quantity included in the first objective function is derived by inverse reinforcement learning using the candidate feature quantity which is a plurality of candidate feature quantities, and each weight is derived.
  • the candidate feature quantity which is a plurality of candidate feature quantities
  • each weight is derived.
  • the first inverse that derives each weight of the candidate features included in the first objective function by inverse reinforcement learning using the candidate features, which are a plurality of candidate features, on the computer.
  • candidate features which are a plurality of candidate features
  • the first inverse that derives each weight of the candidate features included in the first objective function by inverse reinforcement learning using the candidate features, which are a plurality of candidate features, on the computer.
  • a second inverse reinforcement learning execution process for generating a second objective function is executed by a feature quantity selection process for selecting a feature quantity and an inverse reinforcement learning using the selected feature quantity.
  • FIG. 1 is a block diagram showing a configuration example of the first embodiment of the learning device according to the present invention.
  • the learning device 100 of the present embodiment is a device that performs reverse reinforcement learning that estimates a reward (function) from the behavior of a subject.
  • the learning device 100 includes a storage unit 10, an input unit 20, a first reverse reinforcement learning execution unit 30, a feature amount selection unit 40, a second reverse reinforcement learning execution unit 50, an information amount standard calculation unit 60, and the like.
  • a determination unit 70 and an output unit 80 are provided.
  • the storage unit 10 stores information necessary for the learning device 100 to perform various processes.
  • the storage unit 10 is characterized by expert decision-making history data (sometimes referred to as trajectory) used for learning by the first reverse reinforcement learning execution unit 30 and the second reverse reinforcement learning execution unit 50, which will be described later, and features of the objective function. You may remember the quantity candidates. Further, the storage unit 10 may store the candidate of the feature amount and the information (label) indicating the content of the feature amount in association with each other.
  • the storage unit 10 may store a mathematical optimization solver for realizing the first reverse reinforcement learning execution unit 30 and the second reverse reinforcement learning execution unit 50, which will be described later.
  • the content of the mathematical optimization solver is arbitrary and may be determined according to the environment and the device to be executed.
  • the storage unit 10 is realized by, for example, a magnetic disk or the like.
  • the input unit 20 receives input of information necessary for the learning device 100 to perform various processes.
  • the input unit 20 may accept, for example, the input of the above-mentioned decision-making history data.
  • the first inverse reinforcement learning execution unit 30 sets an objective function (hereinafter referred to as a first objective function) using a plurality of candidate feature quantities (hereinafter referred to as candidate feature quantities). Specifically, the first inverse reinforcement learning execution unit 30 may set the first objective function with all the features assumed as candidates as candidate features. Then, the first inverse reinforcement learning execution unit 30 derives each weight w * of the candidate feature quantity included in the first objective function by inverse reinforcement learning.
  • a first objective function hereinafter referred to as a first objective function
  • candidate feature quantities hereinafter referred to as candidate feature quantities
  • the list including the entire candidate feature quantity used when learning the first objective function may be referred to as a feature quantity list A.
  • the feature amount selection unit 40 selects one feature amount from the candidate feature amounts from which each weight w * is derived, the reward expressed using the feature amount is closest to the ideal reward result. Select the feature amount estimated to be. It can be said that such a feature amount is a feature amount that can most affect the reward among the candidate feature amounts. In other words, it can be said that the feature amount selection unit 40 is performing a process of selecting one feature amount from the feature amount list A described above.
  • the feature amount selection unit 40 may select, for example, the feature amount that the expert determines to be the most important as the feature amount that is estimated to be closest to the ideal reward result. Further, in order to enable selection of a feature amount that is not even conscious of such an expert, the feature amount selection unit 40 uses the method described in Non-Patent Document 1 from among the candidate feature amounts. You may select the feature amount.
  • Teaching Risk described in Non-Patent Document 1 is a value indicating (potential) partial optimality of the objective function learned by inverse reinforcement learning.
  • the objective function optimized (learned) by inverse reinforcement learning may be partially optimal, but not totally optimal (potential). This is because the features are arbitrarily selected, so that optimization (learning) based on the unselected features cannot be considered.
  • Teaching Risk is the maximum state in the objective function for which the feature quantity is not selected. In this state, selecting a feature that reduces Teaching Risk selects a feature that reduces potential partial optimality by reducing the difference between the ideal feature vector and the actual feature vector. Therefore, it corresponds to the selection of the feature amount that is estimated to approach the ideal reward result.
  • WorldView can be represented by a matrix.
  • Equation 1 the left side represents the maximum value of the inner product of the ideal weight and the vector belonging to the kernel of WorldView.
  • the kernel of a matrix is a vector set that becomes a zero vector by linear transformation by the matrix, and in the case of Teaching Risk, it corresponds to the cosine of this vector set and the ideal weight.
  • the feature amount selection unit 40 may consider each weight w * of the derived candidate feature amount as the optimum parameter, and select the feature amount that minimizes the teaching risk from the candidate feature amounts.
  • the feature amount selected by the feature amount selection unit 40 is added to the feature amount list B. Specifically, the feature amount selection unit 40 removes the selected feature amount from the above-mentioned feature amount list A and adds it to the feature amount list B. In the initial state, the feature amount list B may be initialized to the empty set.
  • the second inverse reinforcement learning execution unit 50 generates a second objective function by inverse reinforcement learning using the selected feature amount. Specifically, the second inverse reinforcement learning execution unit 50 uses the selected feature amount (specifically, the feature amount added to the feature amount list B) to perform an objective function (hereinafter, a second objective function). It is written as.). Then, the second inverse reinforcement learning execution unit 50 derives each weight w of the feature amount included in the second objective function by the inverse reinforcement learning.
  • the second reverse reinforcement learning execution unit 50 is newly added.
  • a second objective function including the selected feature quantity and the already selected feature quantity is set, and each weight of the feature quantity included in the set second objective function is derived.
  • the information criterion calculation unit 60 calculates the information criterion of the generated second objective function.
  • the calculation method of the information criterion is arbitrary, and for example, any calculation method such as AIC (Akaike's Information Criterion), BIC (Bayesian Information Criterion), and FIC (Focused Information Criterion) can be used. Which calculation method to use may be determined in advance.
  • the determination unit 70 determines whether or not to further select a feature amount from the candidate feature amounts based on the learning result of the second objective function.
  • the determination unit 70 determines whether or not to further select a feature amount from the candidate feature amounts based on whether or not a predetermined condition such as the number of learning times of the second objective function and the execution time is satisfied. You may. This condition may be determined according to, for example, the number of sensors that can be mounted in robot control or the like.
  • the determination unit 70 may determine whether or not to further select the feature amount based on the information criterion calculated by the information criterion calculation unit 60. Specifically, the determination unit 70 determines that the feature amount is further selected when the information criterion is monotonically increasing.
  • the feature amount selection unit 40 further selects a feature amount other than the already selected feature amount from the candidate feature amounts, and the second reverse reinforcement is performed.
  • the learning execution unit 50 generates a second objective function by adding a newly selected feature amount and executing inverse reinforcement learning, and the information quantity standard calculation unit 60 generates the generated second objective function. Calculate the information amount standard of. After that, these processes are repeated.
  • the feature amount selection unit 40 further selects the feature amount from the feature amount list A and adds the feature amount to the feature amount list B, and the second.
  • the second inverse reinforcement learning execution unit 50 derives the weight of the second objective function including the feature amount included in the feature amount list B.
  • the learning device 100 determines whether or not to further select a feature amount from the candidate feature amounts based on whether or not a predetermined condition is satisfied without using the information criterion.
  • the information criterion calculation unit 60 may not be provided.
  • the trade-off between the number of feature quantities and the fitting can be realized by determining whether or not the determination unit 70 further selects the feature quantity using the information criterion calculated by the information criterion calculation unit 60. That is, by expressing the objective function using all the features, the fitting to the existing data can be improved, but overfitting may occur.
  • the information criterion it is possible to realize a sparse objective function while expressing the objective function with a more preferable feature quantity.
  • the output unit 80 outputs information about the generated second objective function. Specifically, the output unit 80 outputs a set of features included in the generated second objective function and the weight of the features. The output unit 80 may output, for example, a set of features when the information criterion is maximized and the weight of the features.
  • the output unit 80 may output information regarding the previous second objective function.
  • the output unit 80 may output the feature amount in the order selected by the feature amount selection unit 40. Since the order of the features selected by the feature selection unit 40 is the order of approaching the ideal reward result, the user can grasp the order of the features that can more affect the reward. Become. Further, the output unit 80 may also output information (label) indicating the content of the feature amount. By outputting the feature amount in this way, it becomes possible to improve the interpretability of the user.
  • the input unit 20, the first reverse reinforcement learning execution unit 30, the feature amount selection unit 40, the second reverse reinforcement learning execution unit 50, the information amount standard calculation unit 60, the determination unit 70, and the output unit 80 are , It is realized by a computer processor (for example, CPU (Central Processing Unit), GPU (Graphics Processing Unit)) that operates according to a program (learning program).
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • the program is stored in the storage unit 10 included in the learning device 100, the processor reads the program, and the input unit 20, the first reverse reinforcement learning execution unit 30, the feature amount selection unit 40, and the second reverse according to the program. It may operate as the reinforcement learning execution unit 50, the information amount standard calculation unit 60, the determination unit 70, and the output unit 80. Further, the function of the learning device 100 may be provided in the SaaS (Software as a Service) format.
  • SaaS Software as a Service
  • each may be realized by dedicated hardware.
  • a part or all of each component of each device may be realized by a general-purpose or dedicated circuit (circuitry), a processor, or a combination thereof. These may be composed of a single chip or may be composed of a plurality of chips connected via a bus. A part or all of each component of each device may be realized by the combination of the circuit or the like and the program described above.
  • each component of the learning device 100 when a part or all of each component of the learning device 100 is realized by a plurality of information processing devices and circuits, the plurality of information processing devices and circuits may be centrally arranged or distributed. It may be arranged.
  • the information processing device, the circuit, and the like may be realized as a form in which each is connected via a communication network, such as a client-server system and a cloud computing system.
  • FIG. 2 is an explanatory diagram showing an operation example of the learning device 100 of the present embodiment.
  • the operation of selecting a feature amount based on the information criterion will be described using the Teaching Risk and the feature amount list.
  • the first reverse reinforcement learning execution unit 30 stores all the features in the feature list A and initializes the feature list B as an empty set (step S11).
  • the first inverse reinforcement learning execution unit 30 estimates the weight w * of the objective function by inverse reinforcement learning using all the features (step S12).
  • steps S14 to S17 are repeated. That is, the determination unit 70 controls to repeatedly execute the processes from step S14 to step S17 when it is determined that the information criterion is monotonically increasing (step S13).
  • the feature amount selection unit 40 selects one feature amount from the feature amount list A that minimizes the teaching risk using the weight w * and the feature amount stored in the feature amount list B (step S14). ). Then, the feature amount selection unit 40 deletes the feature amount selected from the feature amount list A and adds it to the feature amount list B (step S15).
  • the second inverse reinforcement learning execution unit 50 executes inverse reinforcement learning with the features included in the feature quantity list B (step S16), and the information quantity criterion calculation unit 60 calculates the information quantity criterion of the generated objective function. (Step S17).
  • the output unit 80 outputs information about the generated objective function (step S18).
  • the first inverse reinforcement learning execution unit 30 derives each weight of the candidate feature quantity included in the first objective function by the inverse reinforcement learning using the candidate feature quantity, and features.
  • the quantity selection unit 40 selects a feature amount estimated to be closest to the ideal reward result from the candidate feature amounts from which each weight is derived.
  • the second inverse reinforcement learning execution unit 50 generates a second objective function by inverse reinforcement learning using the selected feature amount. Therefore, it is possible to support the selection of the feature amount of the objective function used in the inverse reinforcement learning.
  • Embodiment 2 Next, a second embodiment of the learning device of the present invention will be described.
  • an embodiment in which a user is presented with a candidate for a feature amount to be used for learning the second objective function and is selected will be described.
  • FIG. 3 is a block diagram showing a configuration example of a second embodiment of the learning device according to the present invention.
  • the learning device 200 of the present embodiment includes a storage unit 10, an input unit 20, a first reverse reinforcement learning execution unit 30, a feature amount selection unit 41, a feature amount presentation unit 42, an instruction reception unit 43, and a first. It includes a two-reverse reinforcement learning execution unit 51, an information amount standard calculation unit 60, a determination unit 70, and an output unit 80.
  • the learning device 200 of the present embodiment has a feature amount selection unit 41 and features instead of the feature amount selection unit 40 and the second reverse reinforcement learning execution unit 50, as compared with the learning device 100 of the first embodiment. It differs in that it includes a quantity presentation unit 42, an instruction reception unit 43, and a second reverse reinforcement learning execution unit 51. Other than that, the configuration is the same as that of the first embodiment.
  • the feature amount selection unit 41 selects a feature amount from the candidate feature amounts, as in the feature amount selection unit 40 of the first embodiment. At that time, the feature amount selection unit 41 of the present embodiment selects one or more higher-order feature amounts of a predetermined number, which are estimated to be closer to the result of the ideal reward. When the number of feature quantities to be selected is one, the processing performed by the feature quantity selection unit 41 is the same as the processing performed by the feature quantity selection unit 40 of the first embodiment.
  • the feature amount presentation unit 42 presents the feature amount selected by the feature amount selection unit 41 to the user. For example, when a plurality of feature quantities are selected, the feature quantity presenting unit 42 may display the feature quantities in order from the higher rank. Further, when the feature amount label is present, the feature amount presentation unit 42 may also display the label corresponding to the feature amount.
  • FIG. 4 is an explanatory diagram showing an example of a candidate feature amount presented to the user.
  • the feature amount presenting unit 42 displays a graph in which the reciprocal of the Teaching Task illustrated in the first embodiment is set on the horizontal axis and the candidate feature amount is set on the vertical axis from the top of the values. Indicates that four are selected and displayed.
  • the instruction receiving unit 43 receives a selection instruction from the user for the feature amount candidate presented by the feature amount presenting unit 42.
  • the instruction receiving unit 43 may receive, for example, a feature amount selection instruction from the user via a pointing device.
  • the selection instruction received by the instruction receiving unit 43 may be the selection of one feature amount or the selection of a plurality of feature amounts. Further, when the user determines that the corresponding feature amount does not exist, the instruction receiving unit 43 may accept an instruction not to select.
  • the second inverse reinforcement learning execution unit 51 generates a second objective function by inverse reinforcement learning using the feature amount selected by the user. For example, when one feature amount is selected by the user, the second reverse reinforcement learning execution unit 51 may perform the same processing as the second reverse reinforcement learning execution unit 50 of the first embodiment. Further, for example, when a plurality of features are selected, the second inverse reinforcement learning execution unit 51 adds the plurality of features (for example, to the feature list B) to generate a second objective function. May be good. When the feature amount is not selected, the second inverse reinforcement learning execution unit 51 does not have to generate the second objective function.
  • the 60, the determination unit 70, and the output unit 80 are realized by a computer processor that operates according to a program (learning program).
  • FIG. 5 is an explanatory diagram showing an operation example of the learning device 200 of the present embodiment.
  • the process from step S11 to step S12 until the first objective function is generated is the same as the process illustrated in FIG. After that, while the information criterion is monotonically increasing, the processes of steps S22 to S24 and steps S15 to S17 are repeated. That is, the determination unit 70 controls to repeatedly execute the processes of steps S22 to S24 and steps S15 to S17 when it is determined that the information criterion is monotonically increasing (step S21).
  • the feature amount selection unit 41 selects a plurality of features in ascending order of Teaching Risk (step S22).
  • the feature amount presenting unit 42 presents the feature amount selected by the feature amount selection unit 41 to the user (step S23).
  • the instruction receiving unit 43 receives a feature amount selection instruction from the user (step S24).
  • the feature amount selection unit 41 performs the processes from step S15 to step S17 illustrated in FIG. 2. After that, the process of step S18 for outputting the information regarding the generated objective function is performed.
  • the feature amount selection unit 41 selects one or more higher-order feature amounts of a predetermined number, which are estimated to be closer to the ideal reward result, and the feature amount presentation unit. 42 presents the user with one or more selected features. Then, the instruction receiving unit 43 receives an instruction for selection from the user for the presented feature amount, and the second reverse reinforcement learning execution unit 51 performs the second reverse reinforcement learning using the feature amount selected by the user. Generate an objective function of.
  • FIG. 6 is a block diagram showing an outline of the learning device according to the present invention.
  • the learning device 90 according to the present invention performs each of the candidate feature quantities included in the first objective function by inverse reinforcement learning using the candidate feature quantities which are a plurality of (specifically, all) feature quantities as candidates.
  • One from the first reverse reinforcement learning execution unit 91 for example, the first reverse reinforcement learning execution unit 30 for deriving the weight (for example, w * ) and the candidate feature quantity from which each weight (for example, w * ) is derived.
  • the feature amount selection unit 92 (for example, the feature amount selection unit) that selects the feature amount that is estimated that the reward expressed using the feature amount is closest to the ideal reward result. 40) and a second inverse reinforcement learning execution unit 93 (for example, a second inverse reinforcement learning execution unit 50) that generates a second objective function by inverse reinforcement learning using selected features. ..
  • the feature amount selection unit 92 considers each weight (for example, w * ) of the derived candidate feature amount as the optimum parameter, and partially optimizes the objective function (for example, Teaching Risk) from the candidate feature amounts. You may select the feature amount that minimizes.
  • the learning device 90 includes a determination unit 94 (for example, a determination unit 70) for determining whether or not to further select a feature amount from the candidate feature amounts based on the learning result of the second objective function. You may. Then, when it is determined that the feature amount is further selected, the feature amount selection unit 92 newly selects a feature amount other than the already selected feature amount from the candidate feature amounts, and the second reverse reinforcement learning execution unit. 93 may generate a second objective function by performing inverse reinforcement learning by adding newly selected features.
  • a determination unit 94 for example, a determination unit 70 for determining whether or not to further select a feature amount from the candidate feature amounts based on the learning result of the second objective function. You may. Then, when it is determined that the feature amount is further selected, the feature amount selection unit 92 newly selects a feature amount other than the already selected feature amount from the candidate feature amounts, and the second reverse reinforcement learning execution unit. 93 may generate a second objective function by performing inverse reinforcement learning by adding newly selected features.
  • the learning device 90 may include an information criterion calculation unit (for example, the information criterion calculation unit 60) that calculates the information criterion of the generated second objective function. Then, the determination unit 94 may determine whether or not to further select a feature amount from the candidate feature amounts based on the information criterion. With such a configuration, a trade-off between the number of features and the fitting can be realized.
  • an information criterion calculation unit for example, the information criterion calculation unit 60
  • the determination unit 94 may determine whether or not to further select a feature amount from the candidate feature amounts based on the information criterion.
  • the determination unit 94 may determine that the feature amount is further selected from the candidate feature amounts when the information criterion increases monotonically.
  • the learning device 90 includes an output unit 95 (for example, an output unit 80) that outputs the weight of the feature amount included in the second objective function and the corresponding feature amount when the information criterion is maximized. You may.
  • an output unit 95 for example, an output unit 80
  • the output unit 95 may output the feature amount in the order selected by the feature amount selection unit 92.
  • the learning device 90 (for example, the learning device 200) has a feature amount presenting unit (for example, a feature amount presenting unit 42) that presents the feature amount selected by the feature amount selection unit 92 to the user, and the presented feature amount. It may be provided with an instruction receiving unit (for example, an instruction receiving unit 43) that receives an instruction for selection from the user. Then, the feature amount selection unit 92 selects one or more higher-order feature amounts of a predetermined number that are estimated to be closer to the ideal reward result, and the feature amount presentation unit is one or more selected. The feature amount of the above may be presented to the user, and the second inverse reinforcement learning execution unit 93 may generate the second objective function by the inverse reinforcement learning using the feature amount selected by the user.
  • a feature amount presenting unit for example, a feature amount presenting unit 42
  • an instruction receiving unit 43 that receives an instruction for selection from the user.
  • the feature amount selection unit 92 selects one or more higher-order feature amounts of a predetermined number that are estimated to be closer to the
  • FIG. 7 is a schematic block diagram showing a configuration of a computer according to at least one embodiment.
  • the computer 1000 includes a processor 1001, a main storage device 1002, an auxiliary storage device 1003, and an interface 1004.
  • the above-mentioned learning device 90 is mounted on the computer 1000.
  • the operation of each of the above-mentioned processing units is stored in the auxiliary storage device 1003 in the form of a program (learning program).
  • the processor 1001 reads a program from the auxiliary storage device 1003, expands it to the main storage device 1002, and executes the above processing according to the program.
  • the auxiliary storage device 1003 is an example of a non-temporary tangible medium.
  • non-temporary tangible media include magnetic disks, magneto-optical disks, CD-ROMs (Compact Disc Read-only memory), DVD-ROMs (Read-only memory), which are connected via interface 1004. Examples include semiconductor memory.
  • the program may be for realizing a part of the above-mentioned functions. Further, the program may be a so-called difference file (difference program) that realizes the above-mentioned function in combination with another program already stored in the auxiliary storage device 1003.
  • difference file difference program
  • a learning device including a feature amount selection unit to be selected and a second inverse reinforcement learning execution unit that generates a second objective function by inverse reinforcement learning using the selected feature amount.
  • the feature amount selection unit regards each weight of the derived candidate feature amount as the optimum parameter, and selects the feature amount that minimizes the partial optimization of the objective function from the candidate feature amounts.
  • a determination unit for determining whether or not to further select a feature amount from the candidate feature amounts based on the learning result of the second objective function is provided, and the feature amount selection unit further selects the feature amount. If so, a feature amount other than the already selected feature amount is newly selected from the candidate feature amounts, and the second reverse reinforcement learning execution unit adds the newly selected feature amount and reverses.
  • Appendix 4 It is equipped with an information criterion calculation unit that calculates the information criterion of the generated second objective function.
  • the learning device according to Appendix 3, wherein the determination unit determines whether or not to further select a feature amount from the candidate feature amounts based on the information criterion.
  • Appendix 5 The learning device according to Appendix 3, wherein the determination unit determines to further select a feature amount from the candidate feature amounts when the information criterion increases monotonically.
  • Appendix 6 Any one of Appendix 1 to Appendix 5 provided with an output unit that outputs the weight of the feature amount included in the second objective function and the corresponding feature amount when the information criterion is maximized.
  • Appendix 7 The learning device according to Appendix 6, wherein the output unit outputs the feature amount in the order selected by the feature amount selection unit.
  • a feature amount selection unit is provided with a feature amount presentation unit that presents the feature amount selected by the feature amount selection unit to the user, and an instruction reception unit that receives an instruction for selection from the user for the presented feature amount. Selects one or more of a predetermined number of higher-level features that are estimated to be closer to the ideal reward result, and the feature amount presenting unit gives the user one or more selected features.
  • the second reverse reinforcement learning execution unit is described in any one of Supplementary note 1 to Supplementary note 7 that generates a second objective function by reverse reinforcement learning using a feature amount selected by the user. Learning device.
  • each weight of the candidate feature quantity included in the first objective function is derived, and each weight is derived.
  • the feature estimated that the reward expressed using the feature is closest to the ideal reward result is selected, and the selected feature is selected.
  • a learning method characterized by generating a second objective function by inverse reinforcement learning using.
  • Appendix 12 The computer considers each weight of the derived candidate feature quantity as the optimum parameter in the feature quantity selection process, and selects the feature quantity that minimizes the partial optimization of the objective function from the candidate feature quantities.
  • the program storage medium according to Appendix 11 for storing a learning program for making the learning program.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Machine Translation (AREA)
PCT/JP2020/032848 2020-08-31 2020-08-31 学習装置、学習方法および学習プログラム Ceased WO2022044314A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2022545246A JP7529028B2 (ja) 2020-08-31 2020-08-31 学習装置、学習方法および学習プログラム
PCT/JP2020/032848 WO2022044314A1 (ja) 2020-08-31 2020-08-31 学習装置、学習方法および学習プログラム
US18/023,225 US20230306270A1 (en) 2020-08-31 2020-08-31 Learning device, learning method, and learning program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/032848 WO2022044314A1 (ja) 2020-08-31 2020-08-31 学習装置、学習方法および学習プログラム

Publications (1)

Publication Number Publication Date
WO2022044314A1 true WO2022044314A1 (ja) 2022-03-03

Family

ID=80354958

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/032848 Ceased WO2022044314A1 (ja) 2020-08-31 2020-08-31 学習装置、学習方法および学習プログラム

Country Status (3)

Country Link
US (1) US20230306270A1 (cg-RX-API-DMAC7.html)
JP (1) JP7529028B2 (cg-RX-API-DMAC7.html)
WO (1) WO2022044314A1 (cg-RX-API-DMAC7.html)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119359082B (zh) * 2024-10-14 2025-12-12 上海交通大学宁波人工智能研究院 利用强化学习生成非线性协同金融产品alpha因子的方法

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111401556A (zh) * 2020-04-22 2020-07-10 清华大学深圳国际研究生院 一种对抗式模仿学习中奖励函数的选择方法
JP2020119384A (ja) * 2019-01-25 2020-08-06 富士通株式会社 解析プログラム、解析装置、及び解析方法

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020119384A (ja) * 2019-01-25 2020-08-06 富士通株式会社 解析プログラム、解析装置、及び解析方法
CN111401556A (zh) * 2020-04-22 2020-07-10 清华大学深圳国际研究生院 一种对抗式模仿学习中奖励函数的选择方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LUIS HAUG; SEBASTIAN TSCHIATSCHEK; ADISH SINGLA: "Teaching Inverse Reinforcement Learners via Features and Demonstrations", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 21 October 2018 (2018-10-21), 201 Olin Library Cornell University Ithaca, NY 14853 , XP080926150 *

Also Published As

Publication number Publication date
US20230306270A1 (en) 2023-09-28
JP7529028B2 (ja) 2024-08-06
JPWO2022044314A1 (cg-RX-API-DMAC7.html) 2022-03-03

Similar Documents

Publication Publication Date Title
US11861474B2 (en) Dynamic placement of computation sub-graphs
KR102036968B1 (ko) 전문화에 기반한 신뢰성 높은 딥러닝 앙상블 방법 및 장치
US12327180B2 (en) Convolutional neural networks with soft kernel selection
JP6954003B2 (ja) データベースのための畳み込みニューラルネットワークモデルの決定装置及び決定方法
JP7003753B2 (ja) 探索点決定プログラム、探索点決定方法および探索点決定装置
KR102748213B1 (ko) 국부적으로 해석 가능한 모델에 기반한 강화 학습
JP7279821B2 (ja) 意図特徴量抽出装置、学習装置、方法およびプログラム
KR102413588B1 (ko) 학습 데이터에 따른 객체 인식 모델 추천 방법, 시스템 및 컴퓨터 프로그램
US20220261685A1 (en) Machine Learning Training Device
WO2014199920A1 (ja) 予測関数作成装置、予測関数作成方法、及びコンピュータ読み取り可能な記録媒体
US20230401361A1 (en) Generating and analyzing material structures based on neural networks
US20210342707A1 (en) Data-driven techniques for model ensembles
JPWO2020121378A1 (ja) 学習装置および学習方法
JP7529028B2 (ja) 学習装置、学習方法および学習プログラム
WO2020115903A1 (ja) 学習装置、学習方法、および学習プログラム
JP2021174330A (ja) 異種機械学習のアンサンブル学習による予測装置
JP6114679B2 (ja) 制御方策決定装置、制御方策決定方法、制御方策決定プログラム、及び制御システム
JP7439923B2 (ja) 学習方法、学習装置及びプログラム
US20210117831A1 (en) Computer System
JP7147874B2 (ja) 学習装置、学習方法および学習プログラム
WO2022137520A1 (ja) 学習装置、学習方法および学習プログラム
JP6726312B2 (ja) シミュレーション方法、システム、及びプログラム
WO2024203905A1 (ja) 条件予測方法、プログラム、および装置
JP5942998B2 (ja) 線形制約条件生成装置及び方法、半正定値最適化問題求解装置、計量学習装置、並びにコンピュータ・プログラム
US20240061906A1 (en) System and method for downsampling data

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20951563

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022545246

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20951563

Country of ref document: EP

Kind code of ref document: A1