WO2022091261A1 - 評価システム、評価方法および評価プログラム - Google Patents

評価システム、評価方法および評価プログラム Download PDF

Info

Publication number
WO2022091261A1
WO2022091261A1 PCT/JP2020/040484 JP2020040484W WO2022091261A1 WO 2022091261 A1 WO2022091261 A1 WO 2022091261A1 JP 2020040484 W JP2020040484 W JP 2020040484W WO 2022091261 A1 WO2022091261 A1 WO 2022091261A1
Authority
WO
WIPO (PCT)
Prior art keywords
function
evaluation
value
utility
mission
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/040484
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
亮太 比嘉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to US18/032,873 priority Critical patent/US20230394970A1/en
Priority to JP2022558680A priority patent/JP7548327B2/ja
Priority to PCT/JP2020/040484 priority patent/WO2022091261A1/ja
Publication of WO2022091261A1 publication Critical patent/WO2022091261A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles

Definitions

  • the present invention relates to an evaluation system, an evaluation method, and an evaluation program for evaluating a plan performed by an own agent.
  • the automatic negotiation technology is effective as a mechanism for securing or providing a flight route and a flight area for UAV to fly safely.
  • Patent Document 1 describes an area evaluation system that evaluates an area used for the operation of a moving body.
  • the region evaluation system described in Patent Document 1 when a first mission that uses the first region and a second mission that does not use the first region are given, the utility of the first region is used. , Evaluate based on the difference between the utility of the first mission and the utility of the second mission.
  • Non-Patent Document 1 describes AOP (Alternating Offers Protocols), which is an example of a protocol for performing automatic negotiation.
  • the utility at the mission level more specifically, all the tasks included in the mission and the utility based on it are taken into consideration. It is necessary to evaluate it.
  • a mission is defined as "a task (duty) managed by an agent with practicability (concrete planning group)".
  • the mission is "a flight plan group obtained by allocating resources (mobile resources, area resources, etc.) so that each of one or more tasks managed by the agent can be executed”.
  • resources mobile resources, area resources, etc.
  • a task can be defined as a task that involves the operation of a moving object and includes at least the designation of space and time.
  • the operation plan group obtained by allocating mobile resources for a certain task and determining the operation route for each allocated mobile resource is an example of a mission.
  • the utility of a mission is evaluated based on a predetermined function with the reward and cost of the mission as arguments.
  • the utility of the mission will vary depending on what each agent plans. Therefore, it is preferable not only to evaluate the mission based on the existing reward function, but also to perform an original evaluation that reflects the internal plan of each agent.
  • the evaluation method according to the present invention is a mission including the above actions so that the computer maximizes the value of the mission evaluation function that calculates the value of the action of the own agent in a certain state, or the expected value of the cumulative sum of the values.
  • the computer learns the planning evaluation function that evaluates the internal value of its own agent when planning, and the computer transfers the target resource, which is a resource that is a candidate for negotiation, to another agent, or the target resource is transferred from another agent. It is characterized in that the utility of the mission in such a case is evaluated using the utility function that defines the difference in the internal value calculated by using the planning evaluation function.
  • the evaluation program according to the present invention is a mission including the above actions so as to maximize the value of the mission evaluation function that calculates the value of the action of the own agent in a certain state, or the expected value of the cumulative sum of the values.
  • a learning process that learns a plan evaluation function that evaluates the internal value of the own agent when planning, and when the target resource, which is a candidate for negotiation, is transferred to another agent or the target resource is transferred from another agent. It is characterized in that an evaluation process is executed in which the utility of the mission in the case of being evaluated is evaluated using the utility function that defines the difference in the internal value calculated by using the planning evaluation function.
  • the utility of the planned mission can be appropriately evaluated for the own agent even when there is an interaction with another agent.
  • FIG. 1 is a block diagram showing a configuration example of an embodiment of the negotiation system according to the present invention.
  • the negotiation system 100 of the present embodiment includes a mission evaluation unit 10, a planning unit 20, and a negotiation unit 40.
  • the mission evaluation unit 10 gives the planning unit 20 a function (hereinafter referred to as a mission evaluation function) for evaluating a mission from the viewpoint of economic or business value (hereinafter, also referred to as reward). input.
  • a mission evaluation function for evaluating a mission from the viewpoint of economic or business value (hereinafter, also referred to as reward). input.
  • the mission evaluation function is a function that calculates the value (that is, the reward) of the action a (Action) of the own agent in a certain state s (state) (for the mission). This action is the result of the decision (policy) of the own agent, and in the case of reinforcement learning, it can be said that it is the result based on the policy function.
  • the mission evaluation function is a function that derives the reward obtained when the action a is taken in a certain state s and the next phase s'is moved to.
  • the mission evaluation function corresponds to the objective function of mathematical optimization (OR: Operations Research), and when used in reinforcement learning, it corresponds to the reward function.
  • the mission evaluation function of this embodiment is preset by an operator or the like specialized in the industry.
  • the mode of the mission evaluation function is arbitrary.
  • the mission evaluation function may include consideration for actions as a term used for calculating the value. For example, in a situation where repeated negotiations are conducted, when compensation for an action can be obtained from another agent, it is possible to feed back this consideration to the value function by including the consideration term in the mission evaluation function.
  • this corresponds to the function of updating the value function assuming the own agent, including some information of other agents. Therefore, it is also possible to support so-called multi-agent reinforcement learning, in which reinforcement learning is performed by reflecting the information of the other party in the environmental information.
  • the planning unit 20 is a device that plans independently for each agent.
  • the planning unit 20 includes an input unit 22, a learning unit 24, a planning evaluation function storage unit 26, a utility function storage unit 28, and an evaluation unit 30.
  • the input unit 22 accepts the input of the reward function.
  • the input unit 22 may accept the input of the plan evaluation function described later and store it in the plan evaluation function storage unit 26. Further, the input unit 22 may accept input of information regarding the reward by negotiation from, for example, an operator or another device.
  • the learning unit 24 learns a function for evaluating the internal value of its own agent when planning a mission including the above actions (hereinafter referred to as a plan evaluation function).
  • the learning unit 24 of the present embodiment learns the planned evaluation function so as to maximize the input value of the mission evaluation function or the expected value of the cumulative sum of the values over the future.
  • the learning unit 24 may learn the plan evaluation function by arbitrary machine learning. Specifically, the learning unit 24 clearly separates the state s and the action a, and then performs reinforcement learning using MDP (Markov decision process) mathematically modeled in time series. , May generate a plan evaluation function.
  • MDP Markov decision process
  • the learning unit 24 may generate a policy function ⁇ (at
  • the state st there is information indicating the position of the moving object on the map at a certain time t , the weather, the wind direction, the cost of the mission, and the like. That is, the state st may include not only information that is directly dependent on negotiations, such as location information at a certain time, but also information that is not directly dependent on negotiations, such as weather and mission costs. Further, as an example of the action at, there is information indicating in which direction the vehicle moves at a certain time t .
  • the learning unit 24 generates a state action value function Q ( st , at) as a plan evaluation function by enhanced learning using the mission evaluation function as a reward function, and the generated state action value function Q (
  • s) and the state value function V (s) may be generated using s t , at ) .
  • the policy function for generating an action can be defined as the following equation 1.
  • the policy function can be defined as in Equation 2 illustrated below.
  • the value function can be defined as the following equation 3 by using the above-mentioned policy function.
  • the cumulative sum (that is, the sum of rewards) G over the entire time can be defined by the formula 4 illustrated below.
  • the value function (Q (s, a), V (s)) is obtained by calculating the expected value of Equation 4 using a time-series model in MDP.
  • s) and the value function V (s) is not limited to the above-mentioned method.
  • function approximations may be used to replace policy and value functions with neural nets.
  • a policy function and a value function may be defined by using the Q function. That is, the learning unit 24 may learn the Q function, and the policy function and the value function may be defined using the learned Q function.
  • the learning unit 24 may learn the planning evaluation function by using the state transition model which is a function representing the time evolution.
  • the state transition model can be represented by, for example, Equation 5 illustrated below.
  • the learning unit 24 may learn the plan evaluation function by using a simulator or a function representing a predetermined state transition.
  • the plan evaluation function By learning the plan evaluation function using the state transition model, not only the temporary value but also the value reflecting the prediction as a whole can be evaluated at the same time. For example, in the case of the above-mentioned flight plan, not only the value of a certain point but also the value and the route that can be understood from the route information can be generated at the same time. That is, since decision-making based on sequential prediction is a feature of MDP, route negotiation can be optimized by reflecting long-term prediction for real problems.
  • the state transition model may be generated by machine learning, or an external simulator, a given function, a database, or the like may be used as the state transition model. For example, when planning an operation, the function as a simulator may be realized by referring to information from an external engine.
  • the learning unit 24 may perform learning by reflecting the result of negotiations by the negotiation unit 40, which will be described later.
  • the learning unit 24 may redefine the sum of the mission evaluation function and the negotiation reward as the reward function, and perform learning using the defined reward function.
  • the information input to the input unit 22 may be used as a reward for negotiation. This makes it possible to generate a model that links route planning and rewards obtained through negotiations, which had been considered independently until now.
  • knowing the likelihood of negotiation in advance is considered to be effective information for each agent. Therefore, for example, when the other party is an operator (person), by reflecting the result of the negotiation in the model, it is possible to efficiently proceed with the transaction with the company that adjusts well.
  • the learning unit 24 calculates a plan evaluation function that maximizes the internal plan of each agent, using the mission evaluation function as a reward. Then, the learning unit 24 stores the generated plan evaluation function (more specifically, a value function such as Q or V, a policy function, a state transition function, etc.) in the plan evaluation function storage unit 26.
  • the generated plan evaluation function more specifically, a value function such as Q or V, a policy function, a state transition function, etc.
  • the plan evaluation function storage unit 26 stores at least one of the plan evaluation function generated as a result of machine learning by the learning unit 24 and the plan evaluation function accepted by the input unit 22.
  • the plan evaluation function storage unit 26 is realized by, for example, a magnetic disk or the like.
  • the utility of the mission when the resource (hereinafter referred to as the target resource) ⁇ which is a candidate for negotiation, is transferred to another agent, or when the target resource is transferred from another agent.
  • the utility function indicating is defined by the difference in internal value calculated using the plan evaluation function.
  • the target resource is a resource to be negotiated and is used exclusively with other agents (for example, time, space, etc.).
  • the state st includes, for example, the position information of the moving body at a certain time t .
  • the internal value V (s) when there is no negotiation with other agents is described as the original internal value.
  • the utility function U ( ⁇ ) can be defined as in Equation 6 illustrated below.
  • the above-mentioned formula 6 can be said to be a formula in which payment ( ⁇ ) is added to the difference between the internal value ( ⁇ ) adjusted by negotiation and the original internal value.
  • the utility function U may be defined as the equation 7 illustrated below.
  • the evaluation unit 30 acquires a utility function from the utility function storage unit 28, and evaluates the utility of the mission using the acquired utility function. Specifically, the evaluation unit 30 evaluates the utility of the mission when the target resource is transferred to another agent or when the target resource is transferred from another agent, and obtains the utility.
  • the evaluation unit 30 may evaluate the utility by using, for example, a utility function using the state value function as shown in the above equation 3 as the planning evaluation function.
  • the evaluation unit 30 evaluates, one is when the target resource is transferred to another agent, and the other is when the target resource is transferred from another agent.
  • Which situation may be determined, for example, according to the instruction of the negotiation unit 40, which will be described later.
  • the evaluation unit 30 uses the plan evaluation function to calculate the internal value of the mission when the target resource requested by another agent is not used.
  • the calculated internal value is calculated lower than the original internal value because the use of the target resource is restricted. Then, the evaluation unit 30 evaluates the utility when the target resource is transferred to another agent by using the utility function.
  • the evaluation unit 30 calculates the internal value of the mission when the target resource requested by the own agent is used by using the plan evaluation function.
  • the calculated internal value is calculated higher than the original internal value because the target resources that can be used increase. Then, the evaluation unit 30 evaluates the utility when the target resource is transferred from another agent by using the utility function.
  • the utility value may be calculated by inquiring about the data such as the above and acquiring the information.
  • the planning unit 20 of the present embodiment evaluates the utility for the own agent
  • the planning unit 20 can be called an evaluation system.
  • the input unit 22, the learning unit 24, and the evaluation unit 30 are realized by a computer processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) that operates according to a program (evaluation program).
  • a computer processor for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • the program may be stored in a storage unit (not shown) of the planning unit 20, and the processor may read the program and operate as an input unit 22, a learning unit 24, and an evaluation unit 30 according to the program.
  • the function of the planning unit 20 may be provided in the SaaS (Software as a Service) format.
  • the input unit 22, the learning unit 24, and the evaluation unit 30 may each be realized by dedicated hardware. Further, a part or all of each component of each device may be realized by a general-purpose or dedicated circuit (circuitry), a processor, or a combination thereof. These may be composed of a single chip or may be composed of a plurality of chips connected via a bus. A part or all of each component of each device may be realized by the combination of the circuit or the like and the program described above.
  • each component of the planning unit 20 when a part or all of each component of the planning unit 20 is realized by a plurality of information processing devices and circuits, the plurality of information processing devices and circuits may be centrally arranged or distributed. It may be arranged.
  • the information processing device, the circuit, and the like may be realized as a form in which each is connected via a communication network, such as a client-server system and a cloud computing system.
  • plan evaluation function storage unit 26 and the utility function storage unit 28 are realized by, for example, a magnetic disk or the like.
  • the negotiation unit 40 negotiates with other agents based on the utility calculated by the planning unit 20. Specifically, when a request for a target resource is received from another agent, the negotiation unit 40 notifies the planning unit 20 of the request from the other agent, and receives the calculated utility from the planning unit 20. Then, when the calculated utility exceeds a predetermined threshold value, the negotiation unit 40 may decide to allow the request of the target resource from another agent.
  • the negotiation unit 40 notifies the planning unit 20 of the proposed request and receives the calculated utility from the planning unit 20.
  • the proposed request is input to the negotiation unit 40 by, for example, an operator or the like. Then, when the calculated utility exceeds a predetermined threshold value, the negotiation unit 40 may decide to propose the use of the target resource to another agent.
  • the protocol used by the negotiation department 40 for negotiations is arbitrary.
  • the negotiation unit 40 may negotiate with another agent using, for example, AOP (Alternating Offers Protocols) described in Non-Patent Document 1.
  • the negotiation target to be another agent may be a computer (information processing device) that automatically negotiates, or a device that makes a manual judgment.
  • a device (personal computer, etc.) for which a manual judgment is made is, for example, a device (keyboard) capable of responding (accepting or rejecting) to a proposal (acceptance / rejection) transmitted from the negotiation unit 40 and inputting a value used for a counter offer. Etc.).
  • FIG. 2 is a flowchart showing an operation example of the planning unit 20 of the present embodiment.
  • the learning unit 24 learns the plan evaluation function so as to maximize the value of the mission evaluation function or the expected value of the cumulative sum of the values (step S11).
  • the evaluation unit 30 evaluates the utility of the mission using the utility function defined by using the plan evaluation function (step S12).
  • FIG. 3 is a flowchart showing an operation example of the negotiation system 100 of the present embodiment.
  • the negotiation unit 40 notifies the planning unit 20 of the request for the target resource received from another agent (step S21).
  • the evaluation unit 30 of the planning unit 20 evaluates the utility when the requested target resource is transferred to another agent by using the utility function (step S22). Then, the evaluation unit 30 transmits the utility to the negotiation unit 40 (step S23).
  • the negotiation unit 40 determines that the request for the target resource from the other agent is allowed (step S25), and notifies the other agent that the request is allowed. (Step S26). On the other hand, when the calculated utility is equal to or less than the threshold value (No in step S24), the negotiation unit 40 decides to reject the request for the target resource from another agent (step S27), and the content of the negotiation to reject the request or Notify the other agent of the counter offer (step S28).
  • FIG. 4 is a flowchart showing another operation example of the negotiation system 100 of the present embodiment.
  • the negotiation unit 40 notifies the planning unit 20 of the request to be proposed to another agent (step S31).
  • the evaluation unit 30 of the planning unit 20 evaluates the utility when the target resource is proposed to another agent by using the utility function (step S32). Then, the evaluation unit 30 transmits the utility to the negotiation unit 40 (step S33).
  • step S34 When the calculated utility exceeds a predetermined threshold value (Yes in step S34), the negotiation unit 40 decides to propose the use of the target resource to the other agent (step S35), and notifies the other agent of the content of the proposal. (Step S36). On the other hand, when the calculated utility is equal to or less than the threshold value (No in step S34), the negotiation unit 40 decides not to propose the use of the target resource to other agents (step S37).
  • the learning unit 24 learns the planned evaluation function so as to maximize the value of the mission evaluation function or the expected value of the cumulative sum of the values, and the evaluation unit 30 is the target resource.
  • the utility of the mission when the product is transferred to another agent or the target resource is transferred from another agent is evaluated using the utility function that defines the difference in internal value calculated using the planning evaluation function. Therefore, even if there is interaction with other agents regarding the mission to be planned, the utility for the own agent can be appropriately evaluated.
  • FIG. 5 is a block diagram showing an outline of the evaluation system according to the present invention.
  • the evaluation system 80 (for example, the planning unit 20) according to the present invention is the value of the mission evaluation function for calculating the value (for the mission) of the action (for example, a) of the own agent in a certain state (for example, s), or the value thereof. Learning to learn a plan evaluation function that evaluates the internal value of the own agent when planning a mission that includes the action so as to maximize the expected value of the cumulative sum of values (for example, G in Equation 4 shown above).
  • the utility of the mission when the means 81 (for example, the learning unit 24) and the target resource (for example, ⁇ ) which is a candidate for negotiation are transferred to another agent or the target resource is transferred from another agent. It is provided with an evaluation means 82 (for example, an evaluation unit 30) for evaluation using a utility function (for example, the equation 6 shown above) that defines a difference in internal value calculated by using a planning evaluation function.
  • a utility function for example, the equation 6 shown above
  • the utility of the planned mission can be appropriately evaluated for the own agent even if there is interaction with other agents.
  • the learning means 81 learns a policy function (for example, ⁇ (a
  • the utility may be evaluated using a utility function whose design evaluation function is the state value function.
  • the learning means 81 learns the state-behavior value function (for example, Q (s, a)) by using the mission evaluation function as the reward function, and uses the learned state-behavior value function as a policy function and a policy function.
  • a state value function may be generated, and the evaluation means 82 may evaluate the utility by using a utility function using the generated state value function as a planning evaluation function.
  • the evaluation means 82 calculates the internal value of the mission when the target resource requested by the other agent is not used by using the planning evaluation function, and uses the utility when the target resource is transferred to the other agent. It may be evaluated using a function.
  • the evaluation means 82 calculates the internal value of the mission when the target resource requested by the own agent is used by using the planning evaluation function, and uses the utility function when the target resource is transferred from another agent. It may be evaluated using.
  • the state evaluated by the plan evaluation function may include the position information of the moving body, and the target resource as an argument of utility may include the position information of the moving body at a certain time.
  • the state evaluated by the plan evaluation function may include information that does not directly depend on negotiation (for example, weather, mission cost, etc.).
  • the mission evaluation function may include the consideration for the action as a term used for calculating the value.
  • the learning means 81 may learn the plan evaluation function by using at least one of a state transition model, a simulator, and a predetermined function.
  • the utility function may be defined by a function (for example, Equation 6 shown above) in which consideration is added to the difference between the internal value adjusted by negotiation and the original internal value.
  • FIG. 6 is a schematic block diagram showing a configuration of a computer according to at least one embodiment.
  • the computer 1000 includes a processor 1001, a main storage device 1002, an auxiliary storage device 1003, and an interface 1004.
  • the evaluation system 80 described above is mounted on the computer 1000.
  • the operation of each of the above-mentioned processing units is stored in the auxiliary storage device 1003 in the form of a program (evaluation program).
  • the processor 1001 reads a program from the auxiliary storage device 1003, expands it to the main storage device 1002, and executes the above processing according to the program.
  • the auxiliary storage device 1003 is an example of a non-temporary tangible medium.
  • non-temporary tangible media include magnetic disks, magneto-optical disks, CD-ROMs (Compact Disc Read-only memory), DVD-ROMs (Read-only memory), which are connected via interface 1004. Examples include semiconductor memory.
  • the program may be for realizing a part of the above-mentioned functions. Further, the program may be a so-called difference file (difference program) that realizes the above-mentioned function in combination with another program already stored in the auxiliary storage device 1003.
  • difference file difference program
  • (Appendix 1) Self when planning a mission that includes the action so as to maximize the value of the mission evaluation function that calculates the value of the action of the own agent in a certain state, or the expected value of the cumulative sum of the values.
  • a learning method for learning a planning evaluation function that evaluates the internal value of an agent The utility of the mission when the target resource, which is a resource that is a candidate for negotiation, is transferred to another agent or the target resource is transferred from another agent, is the internal value calculated using the plan evaluation function.
  • An evaluation system characterized by having an evaluation means for evaluating using a utility function that defines a difference.
  • the learning means uses the mission evaluation function as a reward function to learn the policy function and the state value function.
  • the evaluation means is the evaluation system according to Appendix 1 in which the utility is evaluated using a utility function having the state value function as a planning evaluation function.
  • the learning means learns the state-action value function using the mission evaluation function as a reward function, and generates the policy function and the state-value function using the learned state-action value function.
  • the evaluation means is the evaluation system according to Appendix 1 in which the utility is evaluated using a utility function whose generated state value function is a planned evaluation function.
  • the evaluation means calculates the internal value of the mission when the target resource requested by another agent is not used using the planning evaluation function, and the utility when the target resource is transferred to another agent. Evaluation system according to any one of Supplementary note 1 to Supplementary note 3 for evaluation using a utility function.
  • the evaluation means calculates the internal value of the mission when the target resource requested by the own agent is used using the planned evaluation function, and the utility function when the target resource is transferred from another agent.
  • the evaluation system according to any one of Supplementary note 1 to Supplementary note 4.
  • the state evaluated by the plan evaluation function includes the position information of the moving object.
  • the evaluation system according to any one of Supplementary note 1 to Supplementary note 5, which includes the position information of a moving object at a certain time in the target resource which is an argument of utility.
  • Appendix 7 The evaluation system according to any one of Appendix 1 to Appendix 6, which includes information that does not directly depend on negotiation in the state evaluated by the plan evaluation function.
  • Appendix 8 The evaluation system according to any one of Appendix 1 to Appendix 7, which includes the consideration for the action as a term used for calculating the value of the mission evaluation function.
  • Appendix 9 The evaluation system according to any one of Appendix 1 to Appendix 8 for learning a planning evaluation function using at least one of a state transition model, a simulator, and a predetermined function as a learning means. ..
  • the utility function is defined by a function obtained by adding consideration to the difference between the internal value adjusted by negotiation and the original internal value. The evaluation described in any one of Appendix 1 to Appendix 9. system.
  • a mission including the action was planned so that the computer maximizes the value of the mission evaluation function that calculates the value of the action of the own agent in a certain state, or the expected value of the cumulative sum of the values.
  • the utility of the mission when the computer transfers the target resource, which is a resource that is a candidate for negotiation, to another agent or the target resource is transferred from another agent is calculated by using the plan evaluation function.
  • An evaluation method characterized in that evaluation is performed using a utility function that defines the difference in internal value.
  • Appendix 12 Using the mission evaluation function as a reward function, learn the policy function and the state value function, The evaluation method according to Appendix 11 for evaluating utility using a utility function using the state value function as a planning evaluation function.
  • the mission evaluation function is used as a reward function to learn the state-action value function, and the learned state-action value function is used to generate the policy function and the state-value function.
  • the program storage medium according to Appendix 14 which stores an evaluation program for evaluating utility by using a utility function having the state value function as a planned evaluation function in the evaluation process.
  • the mission evaluation function is used as a reward function to learn the state-action value function, and the learned state-action value function is used to generate the policy function and the state-value function.
  • the program storage medium according to Appendix 14 which stores an evaluation program for evaluating utility by using a utility function having a generated state value function as a planned evaluation function in the evaluation process.
  • the mission evaluation function is used as a reward function to learn the state-action value function, and the learned state-action value function is used to generate the policy function and the state-value function.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Traffic Control Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
PCT/JP2020/040484 2020-10-28 2020-10-28 評価システム、評価方法および評価プログラム Ceased WO2022091261A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US18/032,873 US20230394970A1 (en) 2020-10-28 2020-10-28 Evaluation system, evaluation method, and evaluation program
JP2022558680A JP7548327B2 (ja) 2020-10-28 2020-10-28 評価システム、評価方法および評価プログラム
PCT/JP2020/040484 WO2022091261A1 (ja) 2020-10-28 2020-10-28 評価システム、評価方法および評価プログラム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/040484 WO2022091261A1 (ja) 2020-10-28 2020-10-28 評価システム、評価方法および評価プログラム

Publications (1)

Publication Number Publication Date
WO2022091261A1 true WO2022091261A1 (ja) 2022-05-05

Family

ID=81383824

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/040484 Ceased WO2022091261A1 (ja) 2020-10-28 2020-10-28 評価システム、評価方法および評価プログラム

Country Status (3)

Country Link
US (1) US20230394970A1 (cg-RX-API-DMAC7.html)
JP (1) JP7548327B2 (cg-RX-API-DMAC7.html)
WO (1) WO2022091261A1 (cg-RX-API-DMAC7.html)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116562692A (zh) * 2023-05-11 2023-08-08 南京航空航天大学 一种城市低空无人机航路网络评价方法
JP2024055243A (ja) * 2022-10-07 2024-04-18 パナソニックIpマネジメント株式会社 生産プログラム作成支援装置および生産プログラム作成支援方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019004081A1 (ja) * 2017-06-30 2019-01-03 日本電気株式会社 領域評価システム、方法およびプログラム
JP2019087096A (ja) * 2017-11-08 2019-06-06 本田技研工業株式会社 行動決定システム及び自動運転制御装置
WO2019167161A1 (ja) * 2018-02-28 2019-09-06 日本電気株式会社 領域評価システム、方法および記録媒体

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019004081A1 (ja) * 2017-06-30 2019-01-03 日本電気株式会社 領域評価システム、方法およびプログラム
JP2019087096A (ja) * 2017-11-08 2019-06-06 本田技研工業株式会社 行動決定システム及び自動運転制御装置
WO2019167161A1 (ja) * 2018-02-28 2019-09-06 日本電気株式会社 領域評価システム、方法および記録媒体

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2024055243A (ja) * 2022-10-07 2024-04-18 パナソニックIpマネジメント株式会社 生産プログラム作成支援装置および生産プログラム作成支援方法
CN116562692A (zh) * 2023-05-11 2023-08-08 南京航空航天大学 一种城市低空无人机航路网络评价方法
CN116562692B (zh) * 2023-05-11 2024-03-22 南京航空航天大学 一种城市低空无人机航路网络评价方法

Also Published As

Publication number Publication date
US20230394970A1 (en) 2023-12-07
JP7548327B2 (ja) 2024-09-10
JPWO2022091261A1 (cg-RX-API-DMAC7.html) 2022-05-05

Similar Documents

Publication Publication Date Title
US11631333B2 (en) Multi-agent reinforcement learning for order-dispatching via order-vehicle distribution matching
JP6760501B2 (ja) 領域評価システム、方法およびプログラム
CN113287124A (zh) 用于搭乘订单派遣的系统和方法
US20210150388A1 (en) Model estimation system, model estimation method, and model estimation program
US20200311560A1 (en) Integrated Intelligence Systems and Processes
Wu et al. Two-level vehicle path planning model for multi-warehouse robots with conflict solution strategies and improved ACO
Khorasgani et al. Challenges of applying deep reinforcement learning in dynamic dispatching
JP7548327B2 (ja) 評価システム、評価方法および評価プログラム
JP2023021150A (ja) コンテナ積載計画装置、方法、および、プログラム
Ho et al. Preference-based multi-objective multi-agent path finding: F. Ho, S. Nakadai
CN119396005A (zh) 基于信息协同的无人机控制系统和方法
Mondal et al. How to coordinate uavs and ugvs for efficient mission planning? optimizing energy-constrained cooperative routing with a drl framework
US20240103519A1 (en) Intelligent assignment of robotic edge devices in an edge computing ecosystem
Ge et al. Robust optimization for integrated vehicle and crew scheduling based on uncertainty in the main inputs
CN120124906A (zh) 基于多智能体的任务解决方法、系统及多智能体控制设备
JP7464115B2 (ja) 学習装置、学習方法および学習プログラム
Buyer et al. Data-driven merging of car-following models for interaction-aware vehicle speed prediction
WO2022137520A1 (ja) 学習装置、学習方法および学習プログラム
JP7524961B2 (ja) 交渉装置、交渉システム、交渉方法、および、交渉プログラム
JP7420236B2 (ja) 学習装置、学習方法および学習プログラム
Franco et al. An innovative heuristic for planning-based urban traffic control
AU2018102123A4 (en) Knowledge-based Recommendation System for Infrastructure Project Design
WO2022180680A1 (ja) コンテナ管理装置、コンテナ積載管理システム、方法、および、プログラム
JP7563593B2 (ja) 学習装置、学習システム、方法およびプログラム
JP7480909B2 (ja) コンテナ積載管理システム、および、コンテナ積載管理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20959779

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022558680

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20959779

Country of ref document: EP

Kind code of ref document: A1