JP5046149B2 - Technology to determine the most appropriate measures to get rewards - Google Patents

Description

  The present invention relates to a system for determining measures for minimizing risk and maximizing reward. In particular, the present invention relates to a system for determining a measure so that a future accumulated reward is maximized for an object whose state is changed by executing the measure.

  Conventionally, research on portfolio theory has been carried out (see Non-Patent Document 1). The portfolio theory is a theory for determining the operation ratio of each product in a situation where there are a plurality of products such as stocks and bonds that have a risk in return. That is, for example, when a user wants to obtain a desired return as an expected value, by applying portfolio theory, it is possible to determine an operation ratio that minimizes risk in order to obtain the return. Conventionally, research on the Markov decision process has been carried out (see Non-Patent Documents 2 to 8). The Markov decision process problem is a problem of calculating the accumulated profit obtained from a target when the state transition is performed a plurality of times in accordance with a predetermined rule. According to the existing solution of the Markov decision process problem, given the measure that determines the behavior, the expected value of the accumulated profit can be calculated.

H. Markowitz, "Portfolio Selection," Journal of Finance, vol. 7, pp. 77-91, Mar. 1952. G. Tirenni, A. Labbi, A. Elisseeff, and C. Berrospi, "Efficient allocation of marketing resources using dynamic programming," in Proceedings of the SIAM International Conference on Data Mining, 2005. J. A. Filar, L. C. M. Kallenberg, and H. Lee, "Variance-penalized markov decision processes," Mathematics of Operations Research, vol. 14, pp. 147-161, 1989. D. J. White, "Mean, variance, and probabilistic criteria in finite Markov decision processes: A review," Journal of Optimization Theory and Applications, vol. 56, no. 1, pp. 1-29, 1988. R. Munos and A. W. Moore, "Variable Resolution Discretization for High-Accuracy Solutions of Optimal Control Problems," in Proceedings of the International Joint Conference on Artificial Intelligence, 1999, pp. 1348-1355. R. Neuneier, "Enhancing Q-learning for Optimal Asset Allocation," in Advances in Neural Information Processing Systems, 1998, vol. 10, pp. 936-942. H. Kawai, "A variance minimization problem for a Markov decision process," European Journal of Operational Research, vol. 31, pp. 140-145, 1987. M. L. Puterman, Markov Decision Process, John Wiley and Sons, 1994.

  In an environment with a large number of agents (for example, customers), when trying to decide what measures (marketing campaigns, etc.) should be applied to each agent, measures to maximize short-term rewards are long-term. Not necessarily optimal. This is because the policy changes the state of the agent. Also, it is reasonable to model the rewards that the agent brings as a random variable instead of being constant. Therefore, maximizing the expected value of remuneration has a risk with a large risk. In practice, it is desirable to determine the most appropriate measure from the viewpoints of both return and risk.

  For such a problem, a method of calculating the risk of the accumulated reward obtained by the measure for each of a large number of measures and determining the measure that minimizes the risk as the optimum measure among the calculated risks is conceivable. However, conventionally, in order to calculate a risk that occurs as a result of trying to obtain a predetermined expected value from a certain measure, a simulation that requires a long time for calculation has been required. Furthermore, if the measure differs depending on the state of the target agent, and if the measure may be deterministic and probabilistic, it is assumed that the number of simulations will increase explosively and will not be completed in a realistic time. The

  In order to solve the same problem, it can be considered that the change in the state of customers can be modeled as a Markov model, and portfolio theory can be applied to decision making considering return and risk. However, the technology that combines the Markov decision process problem and portfolio theory has not been sufficiently studied. For example, techniques for reducing risk in order to obtain a predetermined reward from an object modeled as a Markov model have been proposed (see Non-Patent Documents 2 to 6). However, in the technique of Non-Patent Document 2, there are cases where the risk can be reduced to some extent, but there are cases where the risk becomes extremely high in a specific situation. Although the techniques of Non-Patent Documents 3 to 6 can avoid an extremely high risk, they can obtain a measure that uniquely determines an action to be taken for an agent in a certain state (hereinafter, decisive). Measure), which is not a measure for selecting an action for an agent in a certain state from a plurality of action candidates (hereinafter, a probabilistic measure).

  On the other hand, in the technique of Non-Patent Document 7, it is possible to obtain a probabilistic measure that minimizes the risk of reward per period in a steady state obtained from one agent. However, in this technique, only the risk associated with the reward focusing only on the steady state is minimized, and the risk associated with the accumulated reward including the intermediate stage until the steady state is reached is not minimized. For this reason, the problem setting is different from the problem to be solved in reality and is not appropriate. Further, in an actual problem such as when a marketing measure is determined, a plurality of agents may make a state transition while having correlation with each other, and it is not appropriate to limit the number of agents to one.

  Therefore, an object of the present invention is to provide a system, a method, and a program that can solve the above-described problems. This object is achieved by a combination of features described in the independent claims. The dependent claims define further advantageous specific examples of the present invention.

In order to solve the above-mentioned problem, in the present invention, a system for calculating a probability distribution of cumulative rewards obtained as a result of sequentially taking a plurality of actions for a plurality of agents that change state according to the action, comprising: For each of the plurality of states that can be taken, for each of the plurality of states, a probability storage unit that stores transition probabilities of transitioning to each state when each action is taken with respect to the agent of the state, A parameter storage unit storing parameters of probability distribution of rewards obtained when transitioning to each state as a result of taking respective actions with respect to the plurality of agents each of which is in the state; and the plurality of states Measures that determine the probability of taking each action for an agent in that state in association with each A measure acquiring unit that receives an input, the parameters of the probability distribution of the cumulative reward obtained after this term from the plurality of agents, by the transition probability to the behavior probability and the next term of the state of this term behavior, obtained by this term actions The recursion formula is calculated by weighting the values based on the parameters of the probability distribution of the rewards and the parameters indicating the probability distribution of the cumulative rewards obtained from the next period onward, and summing up each action and the state of the next period. If the initial state is the same in the recurrence formula after this term and the following term in the recurrence formula, a first calculation is performed to calculate the parameter by solving an equation that the cumulative reward probability distribution parameter is assumed to converge to the same value And an output unit that outputs the calculated parameter as information indicating the probability distribution of the accumulated reward A program for causing a computer to function as the system and a method for calculating a probability distribution by the system are provided.
The above summary of the invention does not enumerate all the necessary features of the present invention, and sub-combinations of these feature groups can also be the invention.

  According to the present invention, it is possible to obtain a probabilistic measure that minimizes the risk of cumulative reward obtained from a plurality of objects that undergo state transition.

  Hereinafter, the present invention will be described through the best mode for carrying out the invention (hereinafter referred to as an embodiment). However, the following embodiment does not limit the invention according to the claims, and Not all the combinations of features described therein are essential to the solution of the invention.

  FIG. 1 shows the overall configuration of the information system 10. The information system 10 includes a probability storage unit 20, a parameter storage unit 30, and a measure determination system 40. The probability storage unit 20 stores, for each of a plurality of states that can be taken by the agent, transition probabilities of transition to the respective states when the respective actions are taken with respect to the agent in the state. An agent is, for example, a consumer to be marketed, and an action is, for example, a marketing action performed on those consumers. The state indicates consumer behavior characteristics, for example, an attribute that the consumer can change. For example, in marketing, it is state 1 that the monthly consumption amount belongs to a customer segment that has a certain amount of money, and the monthly consumption amount belongs to another customer segment that has a different amount of money. State 2 and so on. That is, the state transition means, for example, that a consumer belongs to another customer segment as a result of taking an action such as a discount or a campaign for a certain consumer.

  The parameter storage unit 30 is a parameter indicating a probability distribution of rewards obtained when each of the plurality of states transitions to each state as a result of taking each action for a plurality of agents that are in the state. Is remembered. The reward means, for example, the amount of sales or profit. In the above-described marketing, for example, a reward is the amount of sales obtained from a plurality of consumers as a result of conducting a certain campaign for a plurality of consumers in a certain state. The parameter indicating the probability distribution of the reward is, for example, an average value and a variance value when the probability distribution follows a normal distribution. The data stored in the probability storage unit 20 or the parameter storage unit 30 may be generated in advance by analyzing information such as past marketing history.

  The measure determination system 40 selects a plurality of expected values in advance for the accumulated reward obtained as a result of sequentially taking a plurality of actions for the plurality of agents. Then, the measure determination system 40 determines a measure that defines an action for minimizing the risk with respect to each of these expected values. The measure determination system 40 draws points indicating the expected value and the minimum risk index for obtaining the accumulated reward on a plane constituted by the coordinate axes indicating the expected value and the risk index, and connects the respective points. To draw an efficient frontier curve 60 and display it to the user. The user selects a desired expected value and a risk index from points on the efficient frontier curve 60. The measure determination system 40 outputs the measure for obtaining the selected expected value to the user as the optimum measure 70 for minimizing the risk.

  As described above, the information system 10 according to the present embodiment is provided with a risk in order to obtain a desired cumulative reward from a plurality of agents when data such as a state transition probability of the agent and a reward for one action is given in advance. The objective is to output the most appropriate measures to minimize the risk.

FIG. 2 shows an example of the data structure of the probability storage unit 20. The probability storage unit 20 stores, for each of a plurality of states that can be taken by the agent, transition probabilities of transition to the respective states when the respective actions are taken with respect to the agent in the state. It represents the state of the agent by the variable s, specifically, states _{s 1, s} 2, ..., and the like s _m. Also, let S be a set of states that the agent can take. That is, sεS. Also, it represents the action that can be taken by the variable a, specifically, the action is _{a 1,} a 2, ..., and the like _{a n.} Also, let A be a set of actions that can be taken. That is, aεA. And the probability memory | storage part 20 memorize | stores the transition probability for every combination of the pair (s, a) of the state and action of a transition origin, and the state s of a transition destination. For example, as a result of taking the action a ₁ for the agent in the state s _1, the probability that the state s ₁ is not changed is 25%, and the probability that the state s ₂ is changed is 40%. In the following description, this transition probability is expressed as p _{s ′ | s, a} . In this notation, s indicates a transition source state, a indicates an action, and s ′ indicates a transition destination state.

  FIG. 3 shows an exemplary data structure of the parameter storage unit 30. The parameter storage unit 30 is a parameter indicating a probability distribution of rewards obtained when each of the plurality of states transitions to each state as a result of taking each action for a plurality of agents that are in the state. Is remembered. FIG. 3 shows an average value of reward as an example of a parameter for determining the probability distribution. The parameter storage unit 30 is obtained as a result of transition from the transition source state to the transition destination state for each combination of the transition source state and action (s, a) and the transition destination state s. It memorizes the average value of rewards.

As an example, if there are 10,000 agents, and all 10,000 agents are in a state s ₁ , the state of all the agents as a result of taking action a ₁ for all those agents The average value of the reward obtained when the state transits to state s ₂ is $ 2.10. The average value here is an average value as a parameter of the probability distribution, and means an average value of rewards obtained as a result of taking action a ₁ a plurality of times under the same conditions as described above. Note that, depending on the reward distribution, the probability distribution cannot be determined only by the average value, so the parameter storage unit 30 further stores the same table as FIG. 3 for each of the variance value and other parameters. The other parameters are, for example, a characteristic index and a skewness in a stable distribution. Since the data structure for storing these parameters is substantially the same as that shown in FIG. 3 except that the data contents are replaced with the average value and other parameters are used, the description thereof will be omitted.

In the following description, the reward obtained as a result of the transition to the state s ′ as a result of taking the action a for the agent in the state s is expressed as r _{s ′ | s, a} . The probability distribution is expressed as P (r _{s ′ | s, a} ), and the average value or the position parameter is expressed as μ _{s ′ | s, a} .

FIG. 4 shows an example of the optimum measure 70. The optimal measure 70 determines the action probability to take each action for each state agent. For example in FIG. 4, the probability value of 20% of the intersections of the row state s ₁ and row action a ₁ is a behavior probability that the agent of the state s ₁ take action a _1. The user, if the agent of the state _{s 1} is of 100 people, may be taking the action _{a 1} for 20 of them people, for each of the 100 people of the agent, taking action _{a 1} with a probability of 20% Also good. By acting in accordance with this optimum measure 70, the risk for the desired expected value can be minimized.
In the following description, a measure that defines an action probability of taking action a for an agent in state s is denoted as π _{s, a} . Further, the values of π _{s, a} for all states s ∈ S and action a ∈ A are collectively expressed as π = {π _{s, a} ; s ∈ _{S, a} ∈ A}.

但し、γは割引率を示し０より大きく１より小さい値をとる。ｒ_ｔはこの施策πによって得られる期間ｔにおける報酬を示し、Ｅ_π［ｒ_ｔ］はその期待値を示す。 Next, details of the processing function by which the measure determination system 40 obtains the optimum measure based on these transition probabilities and parameters will be described. Prior to explanation, first, the expected value of cumulative reward is defined. Cumulative reward is the total of rewards obtained from all agents for the current term and the future from this term. Note that this term refers to a period in which execution of the optimum measure to be requested is started among a plurality of periods that are sequentially passed, and the next term refers to a period subsequent to this term. In an actual problem, even if the amount is the same, a reward that can be obtained earlier has a higher value. Therefore, a future reward is multiplied by a discount rate and its value is evaluated low. Specifically, the expected value of the accumulated reward from period 0 to period (T-1) for a certain measure π is expressed as in the following formula (1).

However, γ indicates a discount rate and takes a value larger than 0 and smaller than 1. r _t represents the reward in the period t obtained by this measure π, and E _π [r _t ] represents its expected value.

  FIG. 5 shows a functional configuration of the measure determination system 40. The measure determination system 40 includes a position parameter range calculator 400, a position parameter acquirer 410, a measure acquirer 420, a first calculator 430, a second calculator 440, a convergence determination unit 450, and an output unit 460. And a display control unit 470. The position parameter range calculation unit 400 is a position parameter indicating a probability distribution of cumulative rewards obtained from a plurality of agents based on the transition probabilities stored in the probability storage unit 20 and the parameters stored in the parameter storage unit 30. Calculate the maximum and minimum values. The details of the calculation process will be described later, but based on the position parameter of the accumulated reward obtained from the next period and the position parameter of the reward of the current period, the minimum value (or the maximum value) of the position parameter of the accumulated reward obtained from the current period. This is realized by generating a recursion formula to be obtained and obtaining a convergence value of the value by value iteration.

In the following, since the process differs depending on how the position parameter is determined, each will be described.
The agent can take states s ₁ to s _m as described above. The reward obtained from the agent differs depending on the state of the transition source. Therefore, even if it acts according to the same measure, the accumulated rewards obtained from agents with different initial states are different. For this reason, the position parameter of the accumulated reward may be desired to be determined for each initial state. This is effective when there are few types of customer segments in the marketing example and it is desired to finely determine the rewards to be obtained from each of the customer segments. On the other hand, there is a case where it is desired to determine the position parameter of the probability distribution of the total reward amount obtained from a plurality of agents having various states as initial states. This is effective in the marketing example when there are many types of customer segments and it is difficult to determine a reward to be obtained from each of them. Hereinafter, each of these cases will be described.

(1) When Position Parameters are Determined for Each Initial State The position parameter acquisition unit 410 calculates a value within a range from the minimum value to the maximum value calculated by the position parameter range calculation unit 400 for each of a plurality of states. It is acquired as a position parameter indicating a probability distribution of rewards to be obtained from a plurality of agents having the state as an initial state. The position parameter acquisition unit 410 may receive an input for specifying the position parameter from the user, or may acquire a plurality of values from the minimum value to the maximum value according to a predetermined rule.

  For each of a plurality of states, the measure acquisition unit 420 generates one of the measures for matching the position parameter of the probability distribution of the cumulative reward obtained from a plurality of agents having the state as an initial state with the acquired position parameter. And acquire that measure as the initial measure. It is sufficient for this initial measure to match the position parameter with the given value, regardless of whether it minimizes the risk. The first calculation unit 430 calculates, for each state, a position parameter and a scale parameter indicating a probability distribution of accumulated rewards obtained as a result of acting according to the initial measure from a plurality of agents having the state as an initial state. A method for calculating these parameters will be described later.

  The second calculation unit 440 uses the action probability of each action that can be taken by the agent for each of the plurality of states as a variable, and the scale parameter of the probability distribution of the accumulated reward based on the scale parameter calculated by the first calculation unit 430. By solving the linear programming problem that minimizes the value of the objective function for obtaining, measures for determining the respective action probabilities are calculated. This linear programming problem is constrained in that the position parameter of the cumulative reward probability distribution obtained as a result of the action according to the action probability of each action matches the position parameter calculated by the first calculation unit 430. Also, the sum of the action probabilities for the same state is 1, and the action probabilities are 0 or more.

  The convergence determination unit 450 performs the second calculation on the condition that the scale parameter calculated by the first calculation unit 430 and the scale parameter calculated by the second calculation unit 440 have converged to a value within a predetermined range. The measure calculated by the unit 440 is output to the output unit 460, and on the condition that the measure is not converged, the measure calculated by the second calculator 440 is given to the first calculator 430 instead of the initial measure. As a result, the first calculation unit 430 further calculates the position parameter and the scale parameter of the probability distribution of the cumulative reward obtained as a result of acting according to the given measure.

  The output unit 460 outputs the converged scale parameter and the first acquired position parameter to the user. The output unit 460 may calculate and output a risk index value, such as a value at risk, based on the converged scale parameter and the acquired position parameter. Furthermore, when a plurality of position parameters are acquired, the output unit 460 outputs each position parameter and the corresponding scale parameter to the display control unit 470.

  The display control unit 470 draws an efficient frontier curve based on the position parameter and the scale parameter received from the output unit 460, and outputs them to the user. That is, for example, each time the position parameter is acquired by the position parameter acquisition unit 410, the display control unit 470 performs the position parameter on the plane including the coordinate axis indicating the position parameter and the coordinate axis indicating the risk index value, and The point is drawn at the coordinate value represented by the risk index value based on the scale parameter calculated by the second calculation unit 440 corresponding to the position parameter and determined to be converged by the convergence determining unit 450. Then, the display control unit 470 draws and displays a curve by complementing the drawn points.

  In this case, the output unit 460 may accept designation of coordinate values on the displayed curve from the user. In response to the designation of the coordinate value, the output unit 460 performs a second calculation on the set of the position parameter and the risk index represented by the coordinate value to obtain a cumulative reward of the probability distribution indicated by the position parameter and the risk index. The information is output in association with the measure calculated by the unit 440. In this way, the user not only obtains the optimum measure corresponding to the specified position parameter, but also arbitrarily selects a desired reward amount from various position parameters represented on the curve and responds to it. Measures to do.

(2) When determining the position parameter of the total reward for all initial states The position parameter acquisition unit 410 indicates a probability distribution of the total accumulated rewards obtained from a plurality of agents that can take different states as initial states. Get parameters. This position parameter is equal to or greater than the total weighted by a value given in advance as the ratio of agents having the state as the initial state to the minimum value of the position parameter of the probability distribution of rewards obtained from the agent having the state as the initial state. It is desirable that the value of In addition, this position parameter is obtained by weighting the maximum value of the position parameter of the probability distribution of rewards obtained from an agent having each state as an initial state by a value given in advance as a ratio of agents having the state as an initial state. It is desirable that the value is less than the total.

  The measure acquisition unit 420 generates one of the measures that matches the position parameter indicating the total probability distribution of accumulated rewards obtained from a plurality of agents with the acquired position parameter, and acquires it as an initial measure. It is sufficient for this initial measure to match the position parameter with the given value, regardless of whether it minimizes the risk. The first calculation unit 430 calculates a position parameter and a scale parameter indicating a probability distribution of cumulative rewards obtained from a plurality of agents as a result of acting according to the initial measure.

  For each of the plurality of states, the second calculation unit 440 uses the action probability of each action that can be taken for the agent as a variable, and sets each state as an initial state based on the scale parameter calculated by the first calculation unit 430. Solving a linear programming problem that minimizes the value of the objective function that sums the scale parameter of the cumulative reward probability distribution obtained as a result of acting according to the action probability by weighting the number of agents with the state as the initial state Based on the above, a measure for determining each action probability is calculated. This linear programming problem is a value obtained by weighting the position parameter of the probability distribution of the cumulative reward obtained as a result of acting according to the action probability with each state as the initial state, weighted by the number of agents having the state as the initial state. Have a constraint that matches the position parameter calculated by the first calculation unit 430. In addition, there is a constraint that the total action probability for the same state is 1, and each action probability is 0 or more.

The convergence determination unit 450 performs the second calculation on the condition that the scale parameter calculated by the first calculation unit 430 and the scale parameter calculated by the second calculation unit 440 have converged to a value within a predetermined range. The measure calculated by the unit 440 is output. On the other hand, the convergence determination unit 450 gives the measure calculated by the second calculation unit 440 to the first calculation unit 430 instead of the initial measure on the condition that the convergence has not occurred. Thereby, the 1st calculation part 430 further calculates the position parameter and scale parameter of the probability distribution of the cumulative reward obtained as a result of acting according to this measure.
The functions of the output unit 460 and the display control unit 470 are the same as those described in (1). The processing contents are partially different from the case of (1), but details will be described later with reference to a flowchart.

但し、σ_{ｓ´｜ｓ，ａ}は標準偏差である。また、状態ｓからの合計報酬は、互いに独立な無数のエージェントによる微小報酬の合計となり、中心極限定理が適用できる。 FIG. 6 shows a functional configuration of the first calculation unit 430. The first calculation unit 430 includes a first unit 500, a second unit 530, and a third unit 570. The first unit 500 includes an average value calculation unit 510 and a variance value calculation unit 520. The first unit 500 is given when the reward amount probabilistically obtained from each of a plurality of agents is determined independently, and the total reward obtained from a plurality of agents in the same state in one period follows a normal distribution. The purpose is to calculate an average value and a variance value that define a probability distribution of cumulative reward for a measure (for example, an initial measure). In this case, the probability distribution of the reward obtained when everyone moves to the state s ′ after taking action a when they are in the state s is expressed as the following equation (2).

However, σ _{s ′ | s, a} is a standard deviation. Further, the total reward from the state s is the sum of minute rewards by countless agents independent from each other, and the central limit theorem can be applied.

ただし、Ｍ_ｓ（π）は平均値を表し、Ｓ_ｓ（π）は標準偏差を表す。これらのパラメータは全ての状態ｓ∈Ｓおよび行動ａ∈Ａに対する施策π_ｓ，ａが決まることで初めて決定される。したがってπ={π_ｓ，ａ; ｓ∈Ｓ, ａ∈Ａ}の関数であるため、・_ｓ（π）と表記する。 When the reward for each period follows the normal distribution, the cumulative reward R _s (π) also follows the normal distribution, and therefore the cumulative reward R _s (π) is expressed as the following equation (3).

However, M _s (π) represents an average value, and S _s (π) represents a standard deviation. These parameters are determined for the first time by determining the measures π _{s, a} for all states sεS and action aεA. Therefore, since it is a function of π = {π _{s, a} ; s ∈ _{S, a} ∈ A}, it is expressed as · _s (π).

This average value M _s (π) represents an average value of accumulated rewards obtained from the plurality of agents after this term. And since an average value (or expected value) has linearity, this average value is represented as a recurrence formula which totals the average value of the reward of this term and the average value of the accumulated reward obtained in the following term. Specifically, this recurrence formula is an accumulation obtained from the state s ′ of the next period onward by the action probability π _{s, a} of the action of the current period and the transition probability p _{s ′ | s, a} of the next period. Multiplying the average reward value by the discount rate r and adding the average reward value μ _{s ′ | s, a} obtained by the action of the current term, weights each action a (∈A) and the state s of the next period By summing up ′ (∈S), an equation for obtaining an average value of accumulated rewards from this term is obtained.

平均値算出部５１０は、この方程式を解くことにより、累積報酬の平均値Ｍ_ｓ（π）を初期状態ｓ毎に算出する。 In this recurrence formula, assuming that the average value of the probability distribution of cumulative reward converges to the same value if the initial state is the same in this period and subsequent periods, the state for the average value M _s (π) of the cumulative reward | S | The original simultaneous equation is generated, which becomes the Bellman equation. This equation is shown in equation (4).

The average value calculation unit 510 calculates the average value M _s (π) of the accumulated reward for each initial state s by solving this equation.

The standard deviation S _s (π) represents the standard deviation of accumulated rewards obtained from the plurality of agents after this term. The variance value, which is the square of the standard deviation, is expressed as a recurrence formula that performs calculations based on the variance value of the current term reward and the variance value of the accumulated reward obtained in the following term. Specifically, this recurrence formula is an accumulation obtained from the state s ′ of the next period onward by the action probability π _{s, a} of the action of the current period and the transition probability p _{s ′ | s, a} of the next period. Weighting the value obtained by multiplying the variance value of the reward by the square of the discount rate r and adding the variance value σ ² _{s ′ | s, a} of the reward obtained by the action of the current term, each action a (∈A) and By summing up the state s ′ (∈S) in the next term, the equation for obtaining the variance value of the accumulated rewards from this term is obtained.

分散値算出部５２０は、この方程式を解くことにより、累積報酬の分散値Ｓ^２ _ｓ（π）を初期状態ｓ毎に算出する。算出結果は第２算出部４４０に出力される。 In this recurrence formula, if the initial value is the same in this period and the following period, assuming that the variance value of the cumulative reward probability distribution converges to the same value, the cumulative reward variance value S ² _s (π) The number of states | S |. The original simultaneous equation is generated and becomes the Bellman equation. This equation is shown in equation (5).

The variance value calculation unit 520 calculates the variance value S ² _s (π) of the accumulated reward for each initial state s by solving this equation. The calculation result is output to the second calculation unit 440.

The second unit 530 includes a skewness calculation unit 540, a position parameter calculation unit 550, and a scale parameter calculation unit 560. The second unit 530 is given when the amount of reward probabilistically obtained from each of a plurality of agents is determined independently, and the total reward obtained from a plurality of agents in the same state in one period follows a stable distribution. The purpose is to calculate an average value and a variance value that define a probability distribution of cumulative reward for a measure (for example, an initial measure). By extending the reward distribution from the normal distribution to the stable distribution, it is possible to handle a distribution having a heavy tail property. The distribution having heavy tail property means a distribution having a thicker base than the normal distribution and a dispersion value that does not become a finite value. As a result, it is possible to solve practical problems that need to be taken into consideration, such as a stock price crash or a loan loss due to chain bankruptcy.

In this case, the probability distribution of the reward obtained when everyone moves to the state s ′ after taking action a when they are in the state s is expressed as the following equation (6).

ここで、Β_ｓ（π）は歪度を示し、Ｍ_ｓ（π）は位置パラメータを示し、Ｓ_ｓ（π）はスケールパラメータを示す。 α _{s ′ | s, a} is a characteristic index of the stable distribution, and indicates the degree of attenuation of probability density in a region where the reward is large. β _{s ′ | s, a} is the skewness and indicates the asymmetry of the distribution. μ _{s ′ | s, a} corresponds to an expected value / average value in _a normal distribution, and an expected value may not be determined in the case of a stable distribution, and is therefore referred to as a position parameter. σ _{s ′ | s, a} is a scale parameter. When α = 2, the variance value is finite, and the stable distribution matches the normal distribution. Further, when 1 <α ≦ 2, an expected value exists, and the position parameter indicates the expected value. The total reward from a plurality of agents in the same state is the sum of minute rewards from an infinite number of independent agents, and the extended central limit theorem can be applied. That is, assuming that α _{s ′ | s, a} is the same for all s, s ′, a (this value is simply expressed as α), the accumulated reward follows a stable distribution. Under this assumption, the accumulated reward R _s (π) is expressed by the following equation (7).

Here, Β _s (π) indicates the skewness, M _s (π) indicates a position parameter, and S _s (π) indicates a scale parameter.

The skewness Β _s (π) is expressed by a recurrence formula based on the skewness indicating the probability distribution of the reward for the current period and the skewness indicating the probability distribution of the reward for the next period or later. Specifically, this recurrence formula is obtained by calculating the distortion of the probability distribution of the reward obtained by the current period action by the action probability π _{s, a} of the current period action and the transition probability p _{s ′ | s, a} to the next period state. Degree β _{s ′ | s, a} and α of the scale parameter σ ^α _{s ′ | s, a} , and the value based on the skewness of the probability distribution of the cumulative reward obtained from the next period s ′ onwards and the scale parameter and the scale parameter Then, by summing up each action a and the state s ′ of the next term, an equation for calculating the skewness Β _s (π) of the probability distribution of the cumulative reward after this term is obtained.

ただし、スケールパラメータは後述する式（１０）に示す方程式を解くことで算出されるものとする。歪度算出部５４０は、式（８）を解くことにより、今期以降の累積報酬の確率分布の歪度であるΒ_ｓ（π）を算出できる。 In this recurrence formula, assuming that the skewness of the probability distribution of the cumulative reward converges to the same value if the initial state is the same in this period and after, the skewness 累積_s (π) of the probability distribution of the cumulative reward The number of states | S | of the original simultaneous equations is generated, which becomes the Bellman equation. This equation is shown in equation (8).

However, the scale parameter is calculated by solving an equation shown in the equation (10) described later. The skewness calculation unit 540 can calculate 歪_s (π), which is the skewness of the probability distribution of the cumulative reward from this period onward, by solving Equation (8).

Further, the position parameter M _s (π) is represented by a recurrence formula based on a position parameter indicating the probability distribution of the reward for the current period and a position parameter indicating the probability distribution of the reward for the next period or later. Specifically, this recurrence formula is based on the behavioral probability π _{s, a} of the current period and the transition probability p _{s ′ | s, a} of the next period. By multiplying the position parameter by the discount rate r and weighting the value obtained by adding the position parameter μ _{s ′ | s, a} of the reward obtained by this period's action, the total is obtained for each action a and the state s ′ of the next period. This is an expression for calculating the position parameter M _s (π) of the probability distribution of the cumulative reward from this term.

位置パラメータ算出部５５０は、式（９）を解くことにより、今期以降の累積報酬の確率分布の位置パラメータであるＭ_ｓ（π）を算出できる。 In this recurrence formula, if the initial state is the same in this period and the following period, if it is assumed that the position parameter of the cumulative reward probability distribution converges to the same value, the position parameter M _s (π) of the cumulative reward probability distribution The number of states | S | of the original simultaneous equations is generated, which becomes the Bellman equation. This equation is shown in equation (9).

The position parameter calculation unit 550 can calculate M _s (π), which is a position parameter of the probability distribution of the cumulative reward from this period, by solving the equation (9).

スケールパラメータ算出部５６０は、式（１０）を解くことにより、今期以降の累積報酬の確率分布のスケールパラメータであるＳ_ｓ ^α（π）を算出できる。 Similarly, the α of the scale parameter S _s ^α (π) is a recurrence based on the α power of the scale parameter indicating the probability distribution of the reward for the current period and the α parameter of the scale parameter indicating the probability distribution of the reward for the next period or later. Represented by an expression. Specifically, this recurrence formula is based on the behavioral probability π _{s, a} of the current period and the transition probability p _{s ′ | s, a} of the next period. Weighting the value obtained by multiplying the α power of the scale parameter by the α power of the discount rate γ and adding the α power σ ^α _{s ′ | s, a} of the scale parameter of the reward obtained by the current behavior, By summing up the state s ′ of the next term, an equation for calculating the α power S _s ^α (π) of the scale parameter of the probability distribution of the accumulated rewards from this term onwards is obtained. This equation is shown in equation (10).

The scale parameter calculation unit 560 can calculate S _s ^α (π), which is a scale parameter of the probability distribution of the cumulative reward from this term, by solving the equation (10).

The third unit 570 includes an average value calculation unit 580 and a scale parameter calculation unit 590. The third unit 570 is given when there is a correlation between the amount of reward probabilistically obtained from each of a plurality of agents, and the total reward obtained from a plurality of agents in the same state in one period follows a normal distribution. The purpose is to calculate an average value and a variance value that define a probability distribution of cumulative reward for a measure (for example, an initial measure). Since the reward for each period follows a normal distribution, the reward obtained when everyone moves to state s ′ after taking action a when everyone is in state s, as in the case of independent and uncorrelated It can be modeled as equation (11).

Since a plurality of agents in the same state correlate with each other, the total reward obtained from the plurality of agents in the same state is the sum of minute rewards by an infinite number of agents correlated with each other. In this case, the central limit theorem cannot be applied as it is.
Or in order to model the distribution of total rewards from agents that correlates to such other, firstly, the random variables X _1, X 2, which correlate with each other, _..., the probability distribution of the sum follows a X _n will look like think of. As one modeling, the following equation (12) can be used.

H is a correlation index value indicating the degree to which the reward obtained from each agent correlates with the reward obtained from each other agent, and this is called a Hurst index. H = 1/2 indicates a case where the remuneration amounts are independent from each other, which is consistent with the situation where the central limit theorem is applied. On the other hand, in an extreme case, it can be assumed that all other X _j (i ≠ j) are linked to the behavior of X _i . Since each random variable alone has the same mean and variance, this is a situation where “X ₂ , X ₃ ,..., X _n are always equal to X ₁ ”. At this time, H = 1. That, n times the standard deviation of X ₁ is as appears in the sum. Including the case of inverse correlation, the domain of H is 0 <H ≦ 1.

Apply this model. That is, in the case of that there are n _s number of agents in state s, the next fiscal year of state in the n _s · [pi _{s, a} number of agents that measures [pi _{s, a} is the distribution for the total compensation of those s' , The Hurst exponent H _{s ′ | s, a} is considered to be dependent. When H _{s ′ | s, a} is the same for all s ′, a (this value is H _s ), the shape of the distribution is relatively easy, and the period per period when the measure π _s is performed The reward r _{s ′ | s, πs} is _expressed by the following equation (13).

This is because, instead of the variance σ ² _{s ′ | s, a} , σ ^{1 / Hs} _{s ′ | s, a obtained by raising the} standard deviation to the power of H _s ^−1.
It means that additivity holds in the domain of. Although it is necessary to know the property of the accumulated reward R _s (π) on the premise of the equation (13), there are two points to consider when considering time evolution. They are shown below.
1. Whether there is a correlation between rewards brought by different states s ₁ and s ₂ . That is, when a high reward is obtained from the agent in the state s _1, a high reward is also obtained from the agent in the state s ₂ , and when a low reward is obtained from the agent in the state s _1, a low reward is also obtained from the agent in the state s _2. Whether there is a correlation such as
2. Are the rewards for period t and period t + 1 independent or linked? In other words, even in the same current state, if the reward for t period is high, the reward for t + 1 period is high, and if the reward for t period is low, the reward for t + 1 period is also low. .
It has been confirmed that both correlations based on these points of interest exist in fields such as marketing. For example, when a global phenomenon that affects all states occurs and is not considered in the definition of the state, the correlation shown in the above 1 exists. Further, when there are seasonal fluctuations and up / down trends, and these are not taken into account in the definition of the state, the correlation shown in 2 above exists.

H _s is a Hurst index indicating a correlation between agents belonging to the state s in the same period. Let this be H _s = H for all states s. Under this condition, if the Hearst index H is the same as the Hearst index indicating the interlocking with respect to 1 and 2, the Bellman equation shown in the following equation (14) can be derived.

In other words, the equation shows that the cumulative reward scale parameter S ^{1 / H} _s (π _s ) obtained from the current period, the reward scale parameter σ ^{1 / H} _{s ′ |} In the recurrence formula calculated based on the scale parameter of the accumulated reward obtained in the above, it is generated by regarding that the scale parameter of the accumulated reward converges to the same value in and after this term. The recurrence formula is, specifically, the cumulative reward obtained from the next period to the next period on the basis of the action probability π _{s, a} of the current period and the transition probability p _{s ′ | s, a} to the next period. Is multiplied by the value obtained by multiplying the scale parameter of the discount rate r by the reciprocal of the Hearst index H, and the value obtained by adding the scale parameter σ ^{1 / H} _{s ′ |} By summing up the action a and the state s ′ of the next term, the equation for calculating the scale parameter S ^{1 / H} _s (π) of the cumulative reward probability distribution from this term onwards is obtained.

The scale parameter calculation unit 590 can calculate S ^{1 / H} _s (π), which is a scale parameter of the probability distribution of the cumulative reward from this period onward, by solving the equation (14). Note that the average value calculation processing by the average value calculation unit 580 is the same as the calculation processing by the average value calculation unit 510, and thus description thereof is omitted.
As described above with reference to FIG. 6, according to the measure determination system 40 according to the present embodiment, it is possible to analytically calculate parameters that determine the probability distribution of the accumulated reward for a given measure. Thereby, even if it is a case where a parameter is repeatedly calculated with respect to various measures, the time required for calculation can be reduced.

  FIG. 7 shows an example of an efficient frontier curve 60. For each of the plurality of position parameters, display control unit 470 draws a point at the coordinates represented by the position parameter and the risk index value based on the scale parameter calculated correspondingly. A curve obtained by connecting the drawn points by spline interpolation or the like becomes an efficient frontier curve 60. FIG. 7 shows an efficient frontier curve for three cases where the value of the Hurst index is changed when there is a correlation between the rewards obtained from each agent. In detail, the horizontal axis of the figure shows the expected value, the vertical axis shows the value at risk, and the three cases where the Hurst index is 0.5, 0.556, and 0.667 are shown. The efficient frontier curve for the case is illustrated.

That is, for example, for H = 0.667, if a point on the efficient frontier curve 60 is selected, a measure for minimizing the risk is obtained in order to obtain a predetermined expected value regardless of which point is selected. be able to.
As described above, according to the information system 10 in the present embodiment, given a transition probability, a distribution of rewards for one period, etc., a measure for minimizing the risk in the environment is obtained, and a set of these measures is converted into a frontier. It can be displayed as a curve. As a result, the user can select a measure that maximizes profits according to the risk tolerance of the user, and the flexibility of policy determination can be increased.

位置パラメータ範囲算出部４００は、式（１５）に基づくバリュー・イテレーションによる収束値を位置パラメータの最小値として算出する。また、位置パラメータ範囲算出部４００は、式（１５）における「ｍｉｎ」を「ｍａｘ」に換えた式に基づくバリュー・イテレーションによる収束値を位置パラメータの最大値として算出する。なお、各変数の定義は上述の通りである。 Hereinafter, further detailed processing will be described with reference to FIGS. 8 to 9.
FIG. 8 shows a flowchart of processing in which the optimum measure is determined by the measure determination system 40. The position parameter range calculation unit 400 is a position parameter indicating a probability distribution of cumulative rewards obtained from a plurality of agents based on the transition probabilities stored in the probability storage unit 20 and the parameters stored in the parameter storage unit 30. The maximum value and the minimum value are calculated (S800). This calculation process is shown in Expression (15).

The position parameter range calculation unit 400 calculates a convergence value based on value iteration based on Expression (15) as the minimum value of the position parameter. Further, the position parameter range calculation unit 400 calculates a convergence value by value iteration based on an expression obtained by replacing “min” in Expression (15) with “max” as the maximum value of the position parameter. The definition of each variable is as described above.

  Hereinafter, similar to the case of FIG. 4, different processing is performed depending on how the position parameter is given, and therefore the case will be described separately.

効率的フロンティア曲線を効率的に描画するため、位置パラメータ取得部４１０は、複数の位置パラメータとして、この範囲の中の所定の数の値を、差分値を均等として順次取得することが望ましい。 (1) When Position Parameters are Determined for Each Initial State The position parameter acquisition unit 410 calculates a value within a range from the minimum value to the maximum value calculated by the position parameter range calculation unit 400 for each of a plurality of states. It is acquired as a position parameter indicating the probability distribution of rewards to be obtained from a plurality of agents having the state as the initial state (S810). When the position parameter to be acquired is M _s ^obj , the minimum value is M _s ^min , and the maximum value is M _s ^max , the following equation (16) is satisfied.

In order to efficiently draw an efficient frontier curve, it is desirable that the position parameter acquisition unit 410 sequentially acquires a predetermined number of values in the range as a plurality of position parameters, with the difference values being equal.

Next, for each of the plurality of states, the measure acquisition unit 420 selects one of the measures for matching the position parameter of the probability distribution of the cumulative reward obtained from the plurality of agents having the state as the initial state with the acquired position parameter. And the measure is acquired as an initial measure (S820). It is sufficient for this initial measure to match the position parameter with the given value, regardless of whether it minimizes the risk. Specifically, for example, the measure acquisition unit 420 has the constraint shown in the following equation (17), and solves the linear program problem having a predetermined objective function, thereby obtaining the initial measure π ⁽⁰⁾ _{s, a} Can be sought.

  Next, the first calculation unit 430 calculates, for each state, a position parameter and a scale parameter indicating a probability distribution of accumulated rewards obtained as a result of acting according to the initial measure from a plurality of agents having the state as an initial state. (S830). This is calculated by the first unit 500, the second unit 530, or the third unit 570 described in FIG. 6 according to the probability distribution of the reward in one period. Next, for each of the plurality of states, the second calculation unit 440 uses the action probability of each action that can be taken for the agent as a variable, and the first calculation unit 430 determines the scale parameter in the cumulative reward probability distribution from the next term. Each behavioral probability is solved by solving a linear programming problem that minimizes the value of the scale parameter and the objective function for obtaining the scale parameter from this term on the basis of the variable, assuming that it matches the calculated scale parameter. Measures for determining (S840). This linear programming problem is constrained in that the position parameter of the cumulative reward probability distribution obtained as a result of the action according to the action probability of each action matches the position parameter calculated by the first calculation unit 430. In addition, the sum of the action probabilities for the same state is 1, and the action probabilities are 0 or more.

Specifically, the second calculation unit 440 first obtains the values of Expression (18) and Expression (19).

この目的関数は、来期以降の累積報酬の確率分布におけるスケールパラメータを固定、つまり、来期以降の行動確率は、前回に第２算出部４４０で算出された施策（初回の場合は初期施策）によって定められることを前提としている。そして、この目的関数は、今期の行動確率のみを変数である行動確率で置換えた場合に得られる新たな累積報酬の確率分布におけるスケールパラメータを算出している。 Then, using the values of these equations, the objective function of the linear programming problem is expressed as equation (20). In addition, the constraints of the linear programming problem are expressed as in Equation (21).

This objective function fixes the scale parameter in the probability distribution of accumulated rewards from the next term, that is, the action probability from the next term is determined by the measure previously calculated by the second calculation unit 440 (initial measure in the first case). It is assumed that The objective function calculates a scale parameter in a new cumulative reward probability distribution obtained when only the action probability of the current period is replaced with an action probability that is a variable.

これにより、リスクを最小化するために費やすことのできる費用の制約を定めることができ、より現実の課題に即した解を得ることができる。 Note that the value of the objective function minimized by the second calculation unit 440 does not necessarily need to be a logical minimum value. For example, the second calculation unit 440 improves by minimizing the value of the objective function, that is, by obtaining a value smaller than the scale parameter calculated by the second calculation unit 440 last time. As a result, the scale parameter may be converged to the vicinity of the minimum value.
In addition, the linear programming problem may be further constrained by the fact that a value obtained by weighting the cost required for an action by the action probability of the action and totaling the actions is equal to or less than a predetermined reference budget (Cs). This constraint is expressed as equation (22).

As a result, it is possible to set a cost constraint that can be spent to minimize the risk, and to obtain a solution that is more realistic.

  The convergence determination unit 450 determines whether the scale parameter calculated by the first calculation unit 430 and the scale parameter calculated by the second calculation unit 440 have converged to a value within a predetermined range (S850). On the condition that it has converged (S850), the measure calculated by the second calculation unit 440 is output to the output unit 460. At this time, the second calculation unit 440 separately records the calculated scale parameter and the corresponding position parameter for later use in drawing a frontier curve.

On the other hand, on the condition that the convergence is not converged (S850: NO), the convergence determination unit 450 changes the measure calculated by the second calculation unit 440 to the measure previously calculated by the second calculation unit 440 (in the first case). Is given to the first calculation unit 430 instead of the initial measure. As a result, the first calculation unit 430 further calculates the position parameter and the scale parameter of the probability distribution of the cumulative reward obtained as a result of acting according to the given measure (S830). In addition, the measures that have been premised on the accumulated remuneration from the next term in the objective functions shown in the equations (20) and (19) are replaced with the measures calculated by the second calculation unit 440 this time, and a new linear programming problem is generated. The new linear programming problem is generated and solved by the second calculation unit 440 (S840).
By repeating the processes from S830 to S850, measures π ⁽¹⁾ _{s, a} , π ⁽²⁾ _{s, a} ,..., Π ⁽ⁿ⁾ _{s, a} are sequentially calculated until the scale parameters converge. When the scale parameter converges, the scale parameter becomes a scale parameter of a reward obtained by a measure for minimizing the risk.

  When the scale parameter converges (S850: YES), the position parameter range calculation unit 400 determines whether or not position parameters to be acquired remain within a range that the position parameter value can take (S860). If it remains (S860: NO), the position parameter range calculation unit 400 returns to S810 to acquire the next position parameter, and calculates the optimum measure corresponding to it. On the other hand, if it does not remain (S860: YES), the display control unit 470 draws an efficient frontier curve and displays it to the user (S870). This efficient frontier curve is drawn for each initial state of the agent.

  The output unit 460 indicates a set of the position parameter and the risk index represented by the coordinate value by the position parameter and the risk index when the coordinate value on the displayed curve is designated by the user. In order to obtain the cumulative reward of the probability distribution, it is output in association with the measure calculated by the second calculation unit 440 (S880). The measure may be recorded every time convergence is determined in S850, or may be calculated in response to the coordinate value on the curve being designated. For example, the output unit 460 causes the position parameter acquisition unit 410 to acquire and acquire the position parameter indicated by the coordinate value specified by the user, thereby causing the processing from S810 to S850 to be performed again, and the scale parameter to converge The measures may be output.

The above is the processing in the case of (1) for determining the position parameter for each initial state. Thereby, the user can determine the optimum measure for each customer segment (that is, the initial state), for example.
Next, the case where the position parameter of the total reward for all initial states is determined will be described.

(2) When determining the position parameter of the total reward for all initial states The position parameter acquisition unit 410 indicates a probability distribution of the total accumulated rewards obtained from a plurality of agents that can take different states as initial states. The parameter is acquired (S810). The position parameter M _obj is expressed by the following equation (23). However, w _s is the ratio of agents whose initial state is state s. w _s may be a number of customers in the example of, for example, marketing and the like.

In addition, the position parameter is weighted by the value given in advance as the ratio of the agent having the state as the initial state to the minimum value of the position parameter of the probability distribution of the reward obtained from the agent having the state as the initial state. It is desirable that the value is greater than the total. In addition, this position parameter is obtained by weighting the maximum value of the position parameter of the probability distribution of rewards obtained from an agent having each state as an initial state by a value given in advance as a ratio of agents having the state as an initial state. It is desirable that the value is less than the total. That is, the range of the position parameter M _obj is expressed by the following equation (24).

The measure acquisition unit 420 generates one of the measures that matches the position parameter indicating the total probability distribution of the accumulated rewards obtained from a plurality of agents with the acquired position parameter, and acquires it as an initial measure (S820). It is sufficient for this initial measure to match the position parameter with the given value, regardless of whether it minimizes the risk. For example, the measure acquisition unit 420 first sets, for each state s, a value within a range that can be taken by the position parameter of the probability distribution of the cumulative reward when the state is the initial state, and each value is weighted w. M _s ^tmp is generated as a position parameter M _obj when weighted by _s and summed. For example, the measure acquisition unit 420 can calculate M _s ^tmp by the following equation (25).

次に、第１算出部４３０は、初期施策に従って行動した結果として複数のエージェントから得られる累積報酬の確率分布を示す位置パラメータおよびスケールパラメータを算出する（Ｓ８３０）。第２算出部４４０は、施策を算出するべために、まず、Ｓ８３０において算出されたパラメータに基づき式（２７）および式（２８）の値を算出する。

Then, the measure acquisition unit 420 obtains an initial measure π ⁽⁰⁾ _{s, a} that satisfies the constraint shown in the following formula (26) for each state s. This is achieved by solving a linear programming problem having the constraints shown in equation (26).

Next, the first calculation unit 430 calculates a position parameter and a scale parameter indicating the probability distribution of cumulative rewards obtained from a plurality of agents as a result of acting according to the initial measure (S830). In order to calculate the measure, the second calculation unit 440 first calculates the values of Expression (27) and Expression (28) based on the parameters calculated in S830.

Then, the second calculation unit 440 uses the behavior probability of each action that can be taken for the agent as a variable, and obtains a scale parameter in the probability distribution of the total reward obtained from the agent having each of a plurality of states as an initial state. Solve a linear programming problem that minimizes the value of a function. The objective function is based on the value of the scale parameter on the assumption that the scale parameter in the cumulative reward probability distribution from the next term matches the scale parameter calculated by the first calculation unit 430 for each of the plurality of states. It is a function that calculates the scale parameter of the probability distribution of cumulative rewards from this term obtained as a result of acting according to the action probability with each state as the initial state, and sums it by weighting the number of agents with the state as the initial state . As a result, the second calculation unit 440 can calculate a measure that determines each action probability that minimizes the value of the objective function. Specifically, the objective function of this linear programming problem is expressed by equation (29).

This objective function fixes the scale parameter in the probability distribution of accumulated rewards from the next term, that is, the action probability from the next term is determined by the measure previously calculated by the second calculation unit 440 (initial measure in the first case). It is assumed that The objective function calculates a scale parameter in a new cumulative reward probability distribution obtained when only the action probability of the current period is replaced with an action probability that is a variable.
In addition, this linear programming problem is calculated by weighting the position parameter of the probability distribution of the cumulative reward obtained as a result of acting according to the action probability with each state as the initial state, weighted by the number of agents having the state as the initial state. The obtained value has a constraint that matches the position parameter calculated by the first calculation unit 430. In addition, there is a constraint that the total action probability for the same state is 1, and each action probability is 0 or more. These constraints are expressed by equation (30).

In addition, this linear programming problem has a predetermined value determined by weighting the cost required for an action by the action probability of the action and the ratio of agents having each state as an initial state, and adding up the combination of each state and action. It may be further included as a constraint that the budget is equal to or less than the standard budget C _total . This restriction is expressed as the following equation (31).

  The convergence determination unit 450 determines whether the scale parameter calculated by the first calculation unit 430 and the scale parameter calculated by the second calculation unit 440 have converged to a value within a predetermined range (S850). The convergence determination unit 450 sets the measure calculated by the second calculation unit 440 last time by the second calculation unit 440 on the condition that it has not converged (S850: NO) (initially in the first case) Instead of the measure) is provided to the first calculation unit 430. Thereby, the 1st calculation part 430 further calculates the position parameter and scale parameter of the probability distribution of the cumulative reward obtained as a result of acting according to this measure (S830). In addition, the measures that have been premised on the accumulated remuneration from the next term in the objective functions shown in the equations (29) and (28) are replaced with the measures calculated by the second calculation unit 440 this time, and a new linear programming problem is obtained. And the new linear programming problem is solved by the second calculation unit 440 (S840).

  On the other hand, on the condition that the scale parameter has converged (S850: YES), the convergence determination unit 450 separately stores the converged scale parameter and the corresponding position parameter in association with the measure calculated in S840. Subsequently, the position parameter range calculation unit 400 determines whether or not position parameters to be acquired remain within a range that the position parameter value can take (S860). If it remains (S860: NO), the position parameter range calculation unit 400 returns to S810 to acquire the next position parameter, and calculates the optimum measure corresponding to it. On the other hand, if it does not remain (S860: YES), the display control unit 470 draws an efficient frontier curve and displays it to the user (S870). Unlike the case of (1), only one efficient frontier curve is drawn.

  The output unit 460 indicates a set of the position parameter and the risk index represented by the coordinate value by the position parameter and the risk index when the coordinate value on the displayed curve is designated by the user. In order to obtain the cumulative reward of the probability distribution, it is output in association with the measure calculated by the second calculation unit 440 (S880). The measure is desirably recorded every time convergence is determined in S850. For the calculation procedures shown in S830 to S850, although it is guaranteed that the risk is minimized, the position parameter acquired in S810 is the position parameter of the probability distribution of the reward brought about by the measure calculated in S840. This is because they do not necessarily match.

  The above is the processing in the case of (2) for determining the position parameter for the total reward obtained from the agent having various initial states. As a result, the user can determine a measure that optimizes the relationship between the entire reward amount and the risk even when it is difficult to determine the position parameters individually because there are many customer segments.

  FIG. 9 is a flowchart showing details of the processing in S830. The target in FIG. 9 is (A) when the variance is finite (that is, normal distribution) and the agents are independent with respect to the total reward obtained from a plurality of agents in one period. (B) The variance is infinite. However, in the case where the agents are independent, (C) a process for obtaining a scale parameter from the measure in the case where the variance is finite and the agents have a correlation according to the Hurst index. Each will be described below.

The first calculation unit 430 determines whether or not the variance value of rewards obtained from a plurality of agents in one period is finite (S900). When the variance value is finite (S900: YES), the first calculation unit 430 determines whether or not the reward amount obtained from each agent is independent (S910). When the reward amount is independent (S910: YES), the first calculation unit 430 causes the first unit 500 to calculate the average value and the variance value of the cumulative reward probability distribution (S920). This calculation process is realized by solving the simultaneous equations of the above equations (4) and (5). These equations are expressed as equation (32). That is, the first unit 500 solves the equation for M and the equation for S shown in Equation (32) by an existing numerical method such as the LU decomposition method or the Gaussian elimination method, thereby calculating the probability of the accumulated reward. Parameters that determine the distribution can be calculated.

However, as shown in Expression (33), M, S ² , μ, and σ are vectors of the number | S | of the states, and P is a matrix of | S | × | S |.

但し、式（３５）に示すように、Ｍ、Ｓ^２、μ、およびσは、状態の数｜Ｓ｜次のベクトルであり、Ｐは｜Ｓ｜×｜Ｓ｜の行列である。

On the other hand, if the variance does not have to be finite (S900: NO), the first calculation unit 430 causes the second unit 530 to calculate a parameter that determines the probability distribution of the accumulated reward (S930). This calculation process is realized by solving the equations shown in the above equations (9) and (10). These equations are expressed as the following equation (34). That is, the second unit 530 solves the equation for M shown in Equation (34) and the equation for S by an existing numerical method such as the LU decomposition method or the Gaussian elimination method, thereby calculating the probability of the accumulated reward. A position parameter and a scale parameter that define the distribution can be calculated.

However, as shown in Expression (35), M, S ² , μ, and σ are vectors of the number | S | of the states, and P is a matrix of | S | × | S |.

Further, the second unit 530 calculates the product of the skewness Β and the scale parameter S by solving an equation shown in the following equation (36). However, the definition of each variable is defined in Expression (37).

Then, the second unit 530 can calculate the skewness に よ っ て by calculating Expression (38).

On the other hand, when there is a correlation between agents (S910: NO), the first calculation unit 430 causes the third unit 570 to calculate a parameter that determines the probability distribution of the accumulated reward (S940). This calculation process is realized by solving the simultaneous equations of the above equations (4) and (14). These equations are expressed as equation (39). That is, the third unit 570 solves the equation for M shown in Equation (39) and the equation for S by an existing numerical method such as the LU decomposition method or the Gaussian elimination method, thereby calculating the probability of the accumulated reward. Parameters that determine the distribution can be calculated.

However, as shown in Expression (40), M, S ² , μ, and σ are vectors of the number | S | of the states, and P is a matrix of | S | × | S |.

  FIG. 10 illustrates an example of a hardware configuration of the information processing apparatus 600 that functions as the information system 10. The information processing apparatus 600 includes a CPU peripheral unit including a CPU 1000, a RAM 1020, and a graphic controller 1075 connected to each other by a host controller 1082, a communication interface 1030, a hard disk drive 1040, and the like connected to the host controller 1082 by an input / output controller 1084. And an input / output unit having a CD-ROM drive 1060 and a legacy input / output unit having a ROM 1010 connected to an input / output controller 1084, a flexible disk drive 1050, and an input / output chip 1070.

  The host controller 1082 connects the RAM 1020 to the CPU 1000 and the graphic controller 1075 that access the RAM 1020 at a high transfer rate. The CPU 1000 operates based on programs stored in the ROM 1010 and the RAM 1020, and controls each unit. The graphic controller 1075 acquires image data generated by the CPU 1000 or the like on a frame buffer provided in the RAM 1020 and displays it on the display device 1080. Alternatively, the graphic controller 1075 may include a frame buffer that stores image data generated by the CPU 1000 or the like.

  The input / output controller 1084 connects the host controller 1082 to the communication interface 1030, the hard disk drive 1040, and the CD-ROM drive 1060, which are relatively high-speed input / output devices. The communication interface 1030 communicates with an external device via a network. The hard disk drive 1040 stores programs and data used by the information processing apparatus 600. The CD-ROM drive 1060 reads a program or data from the CD-ROM 1095 and provides it to the RAM 1020 or the hard disk drive 1040.

  The input / output controller 1084 is connected to the ROM 1010 and relatively low-speed input / output devices such as the flexible disk drive 1050 and the input / output chip 1070. The ROM 1010 stores a boot program executed by the CPU 1000 when the information processing apparatus 600 is activated, a program depending on the hardware of the information processing apparatus 600, and the like. The flexible disk drive 1050 reads a program or data from the flexible disk 1090 and provides it to the RAM 1020 or the hard disk drive 1040 via the input / output chip 1070. The input / output chip 1070 connects various input / output devices via a flexible disk 1090 and, for example, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

  A program provided to the information processing apparatus 600 is stored in a recording medium such as the flexible disk 1090, the CD-ROM 1095, or an IC card and provided by the user. The program is read from the recording medium via the input / output chip 1070 and / or the input / output controller 1084, installed in the information processing apparatus 600, and executed. The operations that the program causes the information processing apparatus 600 to perform are the same as the operations in the information system 10 described with reference to FIGS.

  The program shown above may be stored in an external storage medium. As the storage medium, in addition to the flexible disk 1090 and the CD-ROM 1095, an optical recording medium such as a DVD or PD, a magneto-optical recording medium such as an MD, a tape medium, a semiconductor memory such as an IC card, or the like can be used. Further, a storage device such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a recording medium, and the program may be provided to the information processing device 600 via the network.

  As described above, according to the information system 10 according to the present embodiment, since it is possible to obtain an optimal measure for a plurality of agents, an appropriate solution can be given to an actual problem such as marketing. In addition, it is possible to obtain a measure that fully considers not only the expected return but also the risk and the budget, and it is easy to apply to the actual problem. Further, since a probabilistic measure can be obtained instead of a definitive measure, actions that can be taken for the same state can be mixed. Thereby, it is possible to obtain a more optimal measure. In addition, it is possible to determine a measure for a more realistic problem, considering not only the case where each agent is independent but also the case of acting with correlation.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

FIG. 1 shows the overall configuration of the information system 10. FIG. 2 shows an example of the data structure of the probability storage unit 20. FIG. 3 shows an exemplary data structure of the parameter storage unit 30. FIG. 4 shows an example of the optimum measure 70. FIG. 5 shows a functional configuration of the measure determination system 40. FIG. 6 shows a functional configuration of the first calculation unit 430. FIG. 7 shows an example of an efficient frontier curve 60. FIG. 8 shows a flowchart of processing in which the optimum measure is determined by the measure determination system 40. FIG. 9 is a flowchart showing details of the processing in S830. FIG. 10 shows an example of the hardware configuration of the information processing apparatus 600 that functions as the measure determination system 40.

Explanation of symbols

10 Information System 20 Probability Storage Unit 30 Parameter Storage Unit 40 Policy Determination System 60 Efficient Frontier Curve 70 Optimal Policy 400 Position Parameter Range Calculation Unit 410 Position Parameter Acquisition Unit 420 Policy Acquisition Unit 430 First Calculation Unit 440 Second Calculation Unit 450 Convergence Determination unit 460 Output unit 470 Display control unit 500 First unit 510 Average value calculation unit 520 Dispersion value calculation unit 530 Second unit 540 Skewness calculation unit 550 Position parameter calculation unit 560 Scale parameter calculation unit 570 Third unit 580 Average value calculation 590 Scale parameter calculation unit 600 Information processing apparatus

Claims (11)

A system for calculating a probability distribution of cumulative rewards obtained as a result of sequentially taking a plurality of actions for a plurality of agents,
For each of a plurality of states that an agent can take, a probability storage unit that stores transition probabilities of transition to each state when each action is taken with respect to the agent in the state;
For each of the plurality of states, a parameter storing a parameter of a probability distribution of rewards obtained when transition is made to each state as a result of taking action for each of the plurality of agents that are in the state. A storage unit;
In association with each of the plurality of states, a measure acquisition unit that acquires a measure that determines the action probability of taking each action against the agent in the state,
The probability distribution parameters of cumulative rewards obtained from the plurality of agents from this term onward are the parameters of the probability distribution of the rewards obtained by this term behavior based on the behavior probability of this term behavior and the transition probability to the state of the next term, and by weighting the value based on the parameters of the probability distribution of the cumulative reward from next term of the state obtained after the next fiscal year, to generate a recurrence formula to calculate the sum for each of the behavior and the next term of the state, this term in the recurrence formula A first calculation unit that calculates the parameter by solving an equation that the parameters of the cumulative reward probability distribution converge to the same value if the initial state is the same in the following and subsequent periods;
An output unit that outputs the calculated parameter as information indicating a probability distribution of cumulative reward. The parameter storage unit stores an average value and a variance value of rewards as the parameters of the probability distribution of rewards when the sum of rewards obtained from the plurality of agents follows a normal distribution.
The first calculation unit includes:
The average value of accumulated rewards obtained from the plurality of agents after this term is changed to the average value of accumulated rewards obtained from the next term to the next term based on the behavior probability of this term and the transition probability to the next term. Based on a recurrence formula that is calculated by multiplying the discount rate and adding the average value of rewards for this period's actions and summing up each action and the state of the next period, the average of accumulated rewards for this period and beyond An average value calculation unit for calculating a value;
The variance value of the cumulative reward obtained from the plurality of agents after this term is changed to the variance value of the cumulative reward obtained from the next term to the next term based on the behavior probability of the behavior of this term and the transition probability to the next term. Accumulated after the current term based on a recurrence formula that is calculated by multiplying the discount rate squared and adding the variance value of the rewards obtained by this term's behavior and summing up the status of each behavior and next term The system according to claim 1, further comprising: a variance value calculation unit that calculates a variance value of the reward. The parameter storage unit obtains the average value of the reward, the variance value, and the reward obtained from each of the agents from each of the other agents when the total of the rewards obtained from the plurality of agents follows a normal distribution. Memorize the correlation index value indicating the degree of correlation with the reward,
The first calculation unit includes:
The average value of accumulated rewards obtained from the plurality of agents after this term is changed to the average value of accumulated rewards obtained from the next term to the next term based on the behavior probability of this term and the transition probability to the next term. Based on a recurrence formula that is calculated by multiplying the discount rate and adding the average value of rewards for this period's actions and summing up each action and the state of the next period, the average of accumulated rewards for this period and beyond An average value calculation unit for calculating a value;
The scale parameter of the probability distribution of cumulative rewards obtained from the plurality of agents from this term onward is used to calculate the cumulative rewards obtained from the next term from the next term onwards, based on the behavior probability of the current term and the transition probability to the next state . by weighting the value obtained by adding the scale parameter of the probability distribution of compensation obtained by this term behavioral scale parameter to the discount rate of the probability distribution is multiplied by the value raised to the power of a reciprocal of the correlation index value of each behavior and the next term The system according to claim 2, further comprising: a scale parameter calculation unit that calculates a scale parameter of a probability distribution of cumulative rewards from this term on the basis of a recurrence formula that is calculated by summing up the states. When the probability distribution of rewards obtained from the plurality of agents follows a stable distribution, the parameter storage unit includes a characteristic index indicating a degree of probability density attenuation in a region where the reward is large in the stable distribution, and an asymmetry of the distribution in the stable distribution Memorizes the skewness indicating the stability, the position parameter of the stable distribution, and the scale parameter of the stable distribution,
The first calculation unit includes:
The skewness of the probability distribution of cumulative rewards obtained from the plurality of agents from this term onward is calculated based on the behavior probability of the current term and the transition probability to the state of the next term. The recurrence formula is calculated by weighting the degree and scale parameters and the value based on the skewness of the probability distribution of cumulative rewards obtained from the next term onward and the scale parameters and the scale parameters, and summing up each behavior and next state. Based on the skewness calculation unit that calculates the skewness of the probability distribution of the cumulative reward from this term,
The position parameter of the probability distribution of cumulative rewards obtained from the plurality of agents from the current term onward is calculated based on the behavioral probability of the current behavior and the transition probability to the next state . A recurrence formula that calculates the sum of each behavior and the state of the next term by weighting the value obtained by multiplying the location parameter of the probability distribution by the discount rate and adding the location parameter of the probability distribution of the reward obtained by this behavior. Based on the position parameter calculation unit for calculating the position parameter of the cumulative reward probability distribution from this term,
The scale parameter of the probability distribution of cumulative rewards obtained from the plurality of agents from this term onward is used to calculate the cumulative rewards obtained from the next term from the next term onwards, based on the behavior probability of the current term and the transition probability to the next state . by weighting the value obtained by adding the scale parameter of the probability distribution of compensation obtained by this term behavior is multiplied by the value raised to the power of the scale parameter to the discount rate of the probability distribution in the value of the quality index for each of the behavior and the next term of the state The system according to claim 1, further comprising: a scale parameter calculation unit that calculates a scale parameter of a probability distribution of cumulative rewards from this term on the basis of a recurrence formula calculated in total. For each of the plurality of states, further comprising a position parameter acquisition unit that acquires a position parameter of a probability distribution of rewards to be obtained from the plurality of agents having the state as an initial state,
For each of the plurality of states, the measure acquisition unit includes one of the measures for matching the position parameter of the probability distribution of the cumulative reward obtained from the plurality of agents having the state as an initial state with the acquired position parameter. Generated as an initial measure,
The first calculation unit calculates a position parameter and a scale parameter of a probability distribution of cumulative rewards obtained from the plurality of agents as a result of acting according to the initial measure,
The system
With respect to each of the plurality of states, the behavior probability of each action that can be taken with respect to the agent is used as a variable, and the position parameter of the probability distribution of the cumulative reward obtained as a result of acting according to the action probability is calculated by the first calculation unit. Based on the value of the scale parameter based on the value of the scale parameter on the assumption that the scale parameter of the probability distribution of the cumulative reward from the next period matches the scale parameter calculated by the first calculation unit by solving the linear programming problem to minimize the value of the objective function for calculating the scale parameter of the probability distribution of cumulative rewards, a second calculation unit for calculating a measure for determining the respective action probability,
Calculated by the second calculation unit on condition that the scale parameter calculated by the first calculation unit and the scale parameter calculated by the second calculation unit have converged to a value within a predetermined range. Cumulative reward obtained as a result of giving a measure calculated by the second calculation unit to the first calculation unit instead of the initial measure and acting in accordance with the measure, on the condition that the measure is output and has not converged The system according to claim 1, further comprising: a convergence determination unit that calculates a position parameter and a scale parameter of the probability distribution. Based on the transition probability stored in the probability storage unit and the parameter stored in the parameter storage unit, the maximum value and the minimum value of the position parameter of the probability distribution of the cumulative reward obtained from the plurality of agents are determined. A position parameter range calculation unit for calculating,
The system according to claim 5, wherein the position parameter acquisition unit accepts an input of a value within a range from the calculated minimum value to the maximum value, and does not accept an input of a value outside the range. The second calculation unit further adds a constraint that a value obtained by weighting the cost required for an action by the action probability of the action and totaling each action is equal to or less than a predetermined reference budget, and the linear programming problem The system according to claim 5 or 6 . A position parameter acquisition unit that acquires a position parameter of a probability distribution of a total cumulative reward obtained from the plurality of agents, each of which can take a different state as an initial state;
The measure acquisition unit generates one of the measures for matching the position parameter of the total probability distribution of cumulative rewards obtained from the plurality of agents with the acquired position parameter, and acquires it as an initial measure,
The first calculation unit calculates a position parameter and a scale parameter of a probability distribution of cumulative rewards obtained from the plurality of agents as a result of acting according to the initial measure,
The system
For each of the plurality of states, the action probability of each action that can be taken with respect to the agent as a variable, the position parameter of the probability distribution of the cumulative reward obtained as a result of acting according to the action probability with each state as an initial state, The scale parameter of the probability distribution of the cumulative reward from the next term is the constraint that the sum of weighted by the number of agents having the state as the initial state matches the position parameter calculated by the first calculation unit. Based on the value of the scale parameter on the assumption that it matches the scale parameter calculated by the first calculation unit, the probability distribution of cumulative rewards from this term obtained as a result of acting according to the action probability with each state as the initial state An agent that obtains the scale parameter and sets that state as the initial state By solving the linear programming problem to minimize the value of the objective function of total weighted by the number, a second calculation unit for calculating a measure for determining the respective action probability,
Calculated by the second calculation unit on condition that the scale parameter calculated by the first calculation unit and the scale parameter calculated by the second calculation unit have converged to a value within a predetermined range. Cumulative reward obtained as a result of giving a measure calculated by the second calculation unit to the first calculation unit instead of the initial measure and acting in accordance with the measure, on the condition that the measure is output and has not converged The system according to claim 1, further comprising: a convergence determination unit that calculates a position parameter and a scale parameter of the probability distribution. Each time the position parameter is acquired, on the plane composed of the coordinate axis indicating the position parameter and the coordinate axis indicating the risk index value, the coordinates represented by the acquired risk parameter value based on the acquired position parameter and the converged scale parameter It further includes a display control unit that sequentially draws points on the value and draws and displays a curve by complementing between the drawn points.
The output unit indicates a set of a position parameter and a risk index represented by the coordinate value by the position parameter and the risk index when a coordinate value on the displayed curve is designated by the user. The system according to any one of claims 5 to 8, wherein an output is made in association with the measure calculated by the second calculation unit in order to obtain a cumulative reward of the probability distribution. A method of calculating a probability distribution of cumulative rewards obtained as a result of sequentially taking a plurality of actions for a plurality of agents by a system ,
The system
For each of a plurality of states in which the agent can take stores the transition probabilities of transition to each state when taking the respective actions to the agent of the state probability memory unit,
For each of said plurality of states, each stored together with parameters of the probability distribution of reward obtained when the transition to the plurality of agents is the state to each of the actions of taking the results of each state in the parameter storage unit And
The method is
A step in which the measure acquisition unit of the system associates with each of the plurality of states and acquires a measure that defines an action probability of taking an action with respect to the agent in the state;
The first calculation unit of the system determines a parameter of a probability distribution of cumulative rewards obtained from the plurality of agents from this term onward, based on the behavior probability of this term and the transition probability to the state of the next term, according to the behavior of this term. A recurrence formula that weights values based on the probability distribution parameter of the reward and the value based on the parameter of the probability distribution of the cumulative reward obtained from the next period onward, and sums up each action and the state of the next period. Generating and calculating the parameter by solving the equation that the probability distribution parameter of the cumulative reward converges to the same value if the initial state is the same in the recurrence formula from the current term and the next term in the recurrence formula; and
A method in which the output unit of the system outputs the calculated parameter as information indicating a probability distribution of a cumulative reward. A program for causing an information processing device to function as a system for calculating a probability distribution of cumulative rewards obtained as a result of sequentially taking a plurality of actions for a plurality of agents,
The information processing apparatus;
For each of a plurality of states that an agent can take, a probability storage unit that stores transition probabilities of transition to each state when each action is taken with respect to the agent in the state;
For each of the plurality of states, a parameter storing a parameter of a probability distribution of rewards obtained when transition is made to each state as a result of taking action for each of the plurality of agents that are in the state. A storage unit;
In association with each of the plurality of states, a measure acquisition unit that acquires a measure that determines the action probability of taking each action against the agent in the state,
The probability distribution parameters of cumulative rewards obtained from the plurality of agents from this term onward are the parameters of the probability distribution of the rewards obtained by this term behavior based on the behavior probability of this term behavior and the transition probability to the state of the next term, and by weighting the value based on the parameters of the probability distribution of the cumulative reward from next term of the state obtained after the next fiscal year, to generate a recurrence formula to calculate the sum for each of the behavior and the next term of the state, this term in the recurrence formula A first calculation unit that calculates the parameter by solving an equation that the parameters of the cumulative reward probability distribution converge to the same value if the initial state is the same in the following and subsequent periods;
A program that causes the calculated parameter to function as an output unit that outputs information indicating a probability distribution of accumulated rewards.

Images