JP6114679B2 - Control policy determination device, control policy determination method, control policy determination program, and control system - Google Patents

Description

  The present invention relates to an interactive system, a control policy determination device that determines a control policy based on environmental sensing information including uncertainty such as noise in autonomous mobile robots and vehicles, a control system including the control policy, and a control policy determination method It is.

  Conventionally, a technology for automatically acquiring a value function for determining an optimal control strategy of a system based on a partially observable Markov Decision Process (POMDP) by a reinforcement learning framework has been known. .

  When the number of states and actions (behavior) are finite (discrete), it is known that the value function of POMDP is represented by a piecewise linear function in the belief space. The belief space is given as a hyperplane in which each coordinate value is positive and the sum thereof is 1 in the Euclidean space whose dimension is equal to the number of states. Each coordinate value of a point on the belief space is a probability that the system takes a corresponding state. Each linear function corresponds to one of the actions.

  This piecewise linear value function is given as the upper bound of the lower half space determined by a plurality of linear functions. The number of these linear functions increases exponentially with respect to the number of steps read ahead to determine the control strategy. In order to determine the linear function that gives the upper bound from such many linear functions, a huge amount of calculation is required. Therefore, it is difficult to apply this POMDP formula as it is to a practical problem.

  Therefore, methods for calculating the value function approximately have been proposed. In particular, a method called PBVI (Point-Based Value Iteration) is easy to implement and is often used. In addition, various extensions of PBVI have been made (for example, Non-Patent Document 1).

G. Shani, J. Pineau and R. Kaplow, "A Survey of Point-Based POMDP solvers", Auton Agent Multi-Agent Syst (Published Online, 08 June 2012), DOI 10.1007 / s10458-012-9200-2. C. Thurau, K. Kersting and C. Bauckhage, "Convex Non-Negative Matrix Factorization in the Wild", (ICDM 2009, pp. 523-532), DOI: 10.1109 / ICDM.2009.55.

  However, the PBVI method has the following problems, although the calculation amount is greatly reduced as compared with pure POMDP. In other words, the PBVI method has a problem that there is no clear standard for what point should be selected theoretically because the approximation accuracy greatly depends on how the points used for approximation are determined. For this reason, in order to increase the accuracy of approximation, it is necessary to adopt many points, and as a result, still large computational resources are required. Usually, the number of points increases exponentially with respect to the number of states, so that there are still significant restrictions when applied to practical problems.

  The present invention has been made in view of the above-described problem, modeled system control by a partially observable Markov decision process (POMDP), and reduced the calculation cost for automatically acquiring the value function by reinforcement learning, The purpose is to make such a system control model applicable to practical problems.

  In order to solve the above problems, the control policy determination device of the present invention is a control policy determination device that determines a control policy based on environmental sensing information including uncertainty, and based on the environmental sensing information, A linear function generation unit that generates a linear function candidate set that gives a linear element of a value function on a belief space; a dual conversion unit that converts the candidate set on the belief space into a plurality of points on a dual space; and A convex hull approximation calculation unit that calculates an approximate convex hull that approximates the convex hull of a plurality of points, a membership determination unit that determines a membership function of a vertex of the approximate convex hull, and an upper side of the approximate convex hull Based on a convex hull upper side extraction unit, an inverse dual transformation unit that inversely transforms vertices belonging to the upper side into a linear function on the belief space, and a linear function obtained by the inverse transformation, according to the number of backup steps A linear function update unit that updates a shape function, and the dual conversion unit further converts the linear function updated according to the number of backup steps into a plurality of points on a dual space as the candidate set, The control policy determining device further includes a value function determining unit for obtaining a plurality of linear elements of the approximate value function based on the linear function obtained by the inverse transformation after the update of the linear function of the number of backup steps, and the approximation And a policy determining unit that assigns an action to each of a plurality of linear elements of the value function according to the membership function.

  With this configuration, the candidate set in the belief space is mapped to the dual space, and the vertex of the approximate convex hull that approximates the convex hull and its membership function are obtained, so the number of backup steps (value iteration interval) ) Becomes large, the approximate value function can be calculated at high speed and actions can be assigned to the elements of the candidate set.

  In the control policy determination apparatus, the membership determination unit may use a membership function that obtains a correct membership when applied to a plurality of points on the dual space as a membership function of a vertex of the approximate convex hull. You may decide.

  With this configuration, when the approximate convex hull matches the convex hull of a plurality of points on the dual space, it is possible to obtain a membership function that satisfies the condition of reducing to a correct membership.

  In the control policy determination apparatus, the membership determination unit converts a membership function of a plurality of points on the dual space with a matrix that converts the plurality of points on the dual space to the approximate convex hull. May be determined as a membership function of the vertices of the approximate convex hull.

  In the control policy determining apparatus, the linear function update unit may update the linear function for each observation value in each backup step.

  The control system of the present invention outputs the control policy determining device, the environment sensing information input unit for inputting the environment sensing information, and a control command for executing an action assigned by the control policy determining device. And an output unit.

  Even with this configuration, the candidate set in the belief space is mapped to the dual space, and the vertex of the approximate convex hull that approximates the convex hull and its membership function are obtained, so the value iteration interval (backup step) Even when the number is large, the approximate value function can be calculated at high speed and actions can be assigned to the elements of the candidate set.

  The control policy determination method of the present invention is a control policy determination method for acquiring a control policy based on environmental sensing information including uncertainty, the environmental sensing information input step for inputting the environmental sensing information, and the environmental sensing A linear function generation step for generating a linear function candidate set that gives a linear element of a value function on the belief space based on the information; and a dual function that converts the candidate set on the belief space into a plurality of points on the dual space. A conversion step; an approximate convex hull calculation step for calculating an approximate convex hull approximating the convex hull of the plurality of points; a membership determination step for determining a membership function of a vertex of the approximate convex hull; and the approximate convex hull A convex hull upper side extracting step for extracting the upper side of the vertices, an inverse dual transformation step for inversely transforming vertices belonging to the upper side to linear functions on the belief space, A linear function updating step that updates a linear function according to the number of backup steps based on a linear function obtained by inverse transformation, and the dual transformation step further includes a linear function updated according to the number of backup steps Is converted into a plurality of points on the dual space as the candidate set, and the control policy determination method is further based on the linear function obtained by the inverse transformation after the update of the linear function of the number of backup steps. A value function determining step for obtaining a plurality of linear elements of the approximate value function, and a policy determining step for assigning an action to each of the plurality of linear elements of the approximate value function according to the membership function Yes.

  Even with this configuration, the candidate set in the belief space is mapped to the dual space, and the vertex of the approximate convex hull that approximates the convex hull and its membership function are obtained, so the value iteration interval (backup step) Even when the number is large, the approximate value function can be calculated at high speed and actions can be assigned to the elements of the candidate set.

  Yet another aspect of the present invention is a control policy determination program for causing a computer to function as the control policy determination device.

  According to the present invention, it is possible to perform approximate calculation of value iteration in POMDP at high speed, and the number of states such as dialog control in a voice dialog system that is robust against voice recognition errors, real-world independent robots and vehicle control, etc. It is possible to apply the POMDP framework to large real problems.

The block diagram which shows the structure of the dialog system in embodiment of this invention The block diagram which shows the structure of the value function calculation part in embodiment of this invention. Graph showing an example of a candidate set (T = 1) in the embodiment of the present invention (A) Graph showing an example of a candidate set in an embodiment of the present invention (b) Graph showing an example of a value function in an embodiment of the present invention Graph showing an example of a candidate set (T = 2) in the embodiment of the present invention The graph which shows the example of the candidate set (T = 3) in embodiment of this invention (A) Graph showing an example of a candidate set (T = 1) in an embodiment of the present invention (b) Graph showing an example of a dual-transformed point set in an embodiment of the present invention (A) Graph showing an example of a candidate set (T = 1) in an embodiment of the present invention (b) Graph showing an example of a dual-transformed point set in an embodiment of the present invention The graph which shows the example of the approximate convex hull in embodiment of this invention (A) Graph showing an example of the approximate convex hull in the embodiment of the present invention (b) Graph showing an example of an inverse dual transformed linear function in the embodiment of the present invention (A) Graph showing an example of a candidate set (T = 2) in the embodiment of the present invention (b) Graph showing an example of a dual-transformed point set in the embodiment of the present invention (A) Graph showing an example of a candidate set (T = 3) in the embodiment of the present invention (b) Graph showing an example of a dual-transformed point set in the embodiment of the present invention (A) Graph showing an example of the candidate set (T = 2) in the embodiment of the present invention (b) Graph showing an example of the approximate convex hull in the embodiment of the present invention (A) Graph showing an example of a candidate set (T = 3) in the embodiment of the present invention (b) Graph showing an example of an approximate convex hull in the embodiment of the present invention The graph which shows the example of the approximate value function (T = 3) in embodiment of this invention (A) Graph showing an example of the approximate value function in the embodiment of the present invention (b) Graph showing an example of the strict value function in the embodiment of the present invention The graph which shows the example of the value function in embodiment of this invention, and the optimal action policy Operation flow diagram of control policy determination apparatus in the embodiment of the present invention Operation flow diagram of control policy determination device in modified example of embodiment of the present invention

  First, the outline of the principle for solving the problem in learning of the value function in POMDP as described above according to the present invention will be described.

  In one embodiment of the present invention, first, a set of linear functions on a belief space is made to correspond to a point on another Euclidean space (hereinafter referred to as “dual space”). That is, a set of linear functions in belief space is treated as a set of points in dual space. The dimension of the dual space is equal to the number of states.

  Next, a convex hull for a set of points in the dual space is obtained. Furthermore, the upper half of the convex hull obtained previously is obtained for the vertical relationship in the dual space determined from the vertical relationship of the value function in the belief space. A linear function in the belief space corresponding to the points on the upper half becomes a part of the value function. Therefore, a desired value function can be obtained by inversely transforming a point on the upper half of the convex hull in the dual space into a linear function in the belief space.

  In general, the amount of calculation for accurately obtaining a convex hull in a high-dimensional space increases exponentially with the number of dimensions. In order to solve this problem, in one aspect of the present invention, the problem of obtaining a convex hull is changed to a continuous problem called a minimization problem of obtaining a minimum volume among convex sets including all point sets. ease. In one aspect of the present invention, the approximate solution of the minimization problem is configured at high speed. When obtaining this approximate solution, the number of vertices can be made smaller than the actual convex hull, and in that case, the amount of calculation can be made smaller.

  The vertex of the convex hull obtained by approximation (approximate convex hull) generally does not match the point mapped to the dual space from the linear function of the belief space, so when the vertex is converted back to the belief space, What action should be taken is a problem. Therefore, in one aspect of the present invention, the membership function of each point of the point set mapped from the linear function in the belief space to each vertex is defined. Note that the number of such maps is not limited to one, but one is selected based on an appropriate criterion. In one aspect of the present invention, the action corresponding to a vertex of a certain approximate convex hull divides a point set mapped from a linear function in the belief space into subsets having the same action, and the vertex corresponding to the point of each subset Add the membership function to, and define it as the subset action that maximizes the sum.

  In this way, by determining the action for each vertex of the approximate convex hull, returning it to the belief space by inverse transformation, and determining the upper bound of those lower half spaces, it is possible to construct an approximate value function of POMDP by value iteration.

  Hereinafter, a system control apparatus according to an embodiment of the present invention will be described with reference to the drawings. The embodiment described below shows an example when the present invention is implemented, and the present invention is not limited to the specific configuration described below. In carrying out the present invention, a specific configuration according to the embodiment may be adopted as appropriate.

  Hereinafter, for the sake of simplicity, a case where the number of states is two will be described. However, the mathematical formulas and the like of the embodiment of the present invention can be applied to any number of states N as long as the states are discrete and finite. Also, in the figure in this embodiment, the case of two states will be described for the sake of simplicity. Similarly, as long as the states are discrete and finite, they can be extended to a high-dimensional space for any number of states. It is possible to think about what you did.

  In describing the embodiment of the present invention, specific problem setting is not necessarily required, but in order to help understanding, the following control of the interactive system is considered as an example. In this interactive system, it is assumed that the user has an intention of “I want to eat” or “I want to shop”, and the user's intention is determined by voice dialogue.

Hereinafter, these user intentions are referred to as “state”, the state of “I want to eat” is expressed as x ₁ , and the state of “I want to shop” is expressed as x ₂ . It is assumed that the user responds to the inquiry about the intention from the interactive system by speaking “meal” or “shopping” by voice. These responses are called observations, and the observation of “meal” is represented as o ₁ , and the observation of “shopping” is represented as o ₂ . In general, M discrete observation values are obtained. In the state x, it is assumed that the observation amount o is obtained according to the probability p (o | x). This probability p (o | x) is called a perceptual model.

  FIG. 1 is a block diagram showing the configuration of the interactive system according to the present embodiment. As shown in FIG. 1, the dialogue system 100 includes a voice input unit 10, a voice recognition unit 20, a control policy determination device 30, and an output unit 40. The voice input unit 10, the voice recognition unit 20, and the output unit 40 are environments E that input environment sensing information to the control policy determination device 30 and perform output in accordance with the policy determined by the control policy determination device 30. . Each function described below of the control policy determination device 30 is realized by a computer including an arithmetic processing unit, a memory, a storage device, an input unit, an output unit, and the like executing a predetermined program.

  The voice input unit 10 receives voice input and converts it into a voice signal. The speech recognition unit 20 performs speech recognition processing on the speech signal generated by the speech input unit 10 to recognize speech content, and inputs the recognition result to the control policy determination device 30 as environment sensing information. The configuration including the voice input unit 10 and the voice recognition unit 20 senses the environment and supplies the environmental sensing information to the control policy determination device 30, and corresponds to the “environmental sensing information input unit” of the present invention.

  The control strategy determination device 30 determines a control strategy based on the voice content recognized by the voice recognition unit 20. Based on the control strategy determined by the control calculation section 30, the output section 40 outputs a control command for executing the strategy to the controlled object. The control policy determination device 30 includes a belief calculation unit 310, a system model unit 320, an environment model unit 330, a value function calculation unit 340, and a policy determination unit 350. The belief calculation unit 310 includes an initial belief generation unit 311 and a belief update unit 312. The system model unit 320 includes a behavior model unit 321, a perceptual model unit 322, and a policy model unit 323. The environment model unit 330 includes a reward function model unit 331.

  The belief calculation unit 311 generates an initial belief and updates the belief. The system model unit 320 stores an action model, a perceptual model, and a policy model. The environment model unit 330 stores a reward function model. The value function calculation unit 340 calculates a value function. The policy determination unit 350 uses the value function calculated by the value function calculation unit 340 to determine a control policy according to the policy model.

  In the interactive system 100 according to the present embodiment, the voice recognition unit 20 recognizes a speech as voice and recognizes it as “meal” or “shopping”. In general, in speech recognition, even if the user speaks “meal” due to misrecognition, it is problematic that the user is recognized as “shopping”. In other words, the result of speech recognition does not specify the user's state, but can be regarded as merely observing a certain stochastic phenomenon reflecting the user's state. This is the reason for handling the probabilistic perceptual model p (o | x) in the control strategy determination device 30. Note that the perceptual model is stored in the perceptual model unit 322.

Due to the probabilistic nature of this perceptual model, the user's state can only be grasped probabilistically even if the system observes it. Therefore, the probability that the state is x ₁ is p ₁ = p (x ₁ ), the probability that the state is x ₂ is p ₂ = p (x ₂ ), and a set of these probabilities b = (p ₁ , p ₂ ) is called belief. However, p ₁ + p ₂ = 1. The entire possible values of p ₁ and p ₂ are called belief space. That is, in the case of this system, the belief space is a straight line that satisfies p ₁ + p ₂ = 1 in a square area of [0, 1] × [0, 1] (p ₁ , p ₂ ).

Thus, it can be considered that the state of the system is given in an expanded sense by the belief b, instead of being definitively determined as x ₁ or x ₂ . The conventional state space given in two states x ₁ and x ₂ can be regarded as a case in which the belief space consists of only (1, 0) and (0, 1). In that sense, beliefs are sometimes referred to as states below.

The dialogue system 100 executes a process of “presenting information about meals”, “presenting information about shopping”, or “re-listening” to the user again according to the value of the belief. Shall be able to. Hereinafter, the processing of these systems will be referred to as “behavior” in particular, the action of “presenting information on meals” is u ₁ , the action of “presenting information on shopping” is u ₂ , and the action of “listening” is called. expressed as u _3. In general, L discrete actions can be taken. The policy determining unit 350 determines which of these actions is taken.

In general, actions are classified into two large action categories, such as "behavior that observes the state", such as "rehearse" action, and "behavior that makes final decisions" such as actions u ₁ and u _2. be able to. When an action belonging to the “behavior for final decision” category is selected, the problem of system control is once completed. The problem of system control continues as long as an action belonging to the “behavior performing state observation” category is selected until an action belonging to the “behavior performing final decision” category is selected.

Hereinafter, it is assumed that when the action u is executed in the state x, the state transitions to x ′ according to the probability p (x ′ | x, u). This probability p (x ′ | x, u) is called an action model. This behavior model is stored in the behavior model unit 321. Hereinafter, when an action u ^* belonging to the “behavior for final decision” category is selected, it is assumed that p (x ′ | x, u ^* ) = 0 for any state x, x ′. In this case, strictly speaking, p cannot be interpreted as a probability, but is defined as such for convenience.

In the state x, when the action u is executed, the system receives a reward r (x, u). The reward model r (x, u) is stored in the reward function model unit 331 of the environment model unit 330. The maximum expected value of the sum of rewards received from the state where the belief is b to the future T steps ahead is called the value function of that state, and is expressed as V _r (b). In addition, in order to obtain the value function, an action to be taken when the belief is b is called an optimal policy, and is expressed as π _T (b).

However, when calculating the expected value of the sum of rewards, in general, the reward obtained in the future is weighted by multiplying the discount rate γ (0 <γ ≦ 1) by the number of backup steps (ie, τ regarding remuneration obtained after step, by multiplying the gamma ^tau), takes the expectation. This optimal policy π _T (b) is stored in the policy model unit 323 as a policy model.

  As described above, in the situation where the state is estimated only probabilistically by observation, the optimal system action at that time is selected based on the current belief value, that is, the optimal policy is determined. This is the basic problem setting when applying POMDP to interactive system control.

In such a problem setting, the optimal strategy is determined as maximizing the value function in each state x _i . Therefore, if the value function can be obtained, dialogue control by POMDP can be performed. The value function calculation unit 340 of the system control unit obtains this value function. Below, the value function calculation part 340 of this Embodiment demonstrates the method of calculating | requiring the value function by a value iteration method.

  FIG. 2 is a block diagram showing the configuration of the value function calculation unit 340. The value function calculation unit 340 includes an initial linear function generation unit 1, a dual conversion unit 2, a dual value iteration processing unit 3, an inverse dual conversion unit 4, a value function determination unit 5, and a linear function update unit 6. The dual value iterative processing unit 3 includes a labeled convex hull approximation calculating unit 31 and a pruning unit 35. The labeled convex hull approximation calculation unit 31 includes a convex hull approximation calculation unit 32 including a ConvexNMF calculation unit 33 and a membership determination unit 34.

  First, it is theoretically shown that when the number of states is discrete and finite as in the present embodiment, the value function of POMDP is expressed as a piecewise linear function in a belief space (Non-patent Document 1). reference). It can also be theoretically shown that this piecewise linear function is generally convex downward. The value function calculation unit 340 calculates the value function using the property that this piecewise linear function is convex downward. This will be specifically described below.

A linear function (hereinafter also referred to as “linear element”) that gives each section is set as shown in Expression (1). In the following, it is generally assumed that the number of states is N, the number of observed values is M, and the number of actions is L. In the present embodiment, N = 2, M = 2, and L = 3.
The linear function φν (b) is the coefficient vector
Can be identified.

(Initial linear function generation process)
First, the initial linear function generation unit 1 sets the number of backup steps as T = 1, and calculates a linear function that initializes a set of linear functions that give linear elements of the value function at this time. Specifically, the initial linear function generation unit 1 calculates Equation (2) from the reward function γ obtained in each state x _n for each action u _l (l = 1, 2,..., L). calculate.

The initial linear function generation unit 1 acquires a candidate set Γ ^(1), which is a linear element of the value function, using Equation (3), based on the set of initial linear functions thus obtained.
FIG. 3 is a graph showing an example of this candidate set Γ ⁽¹⁾ . In the graph of FIG. 3, the horizontal axis represents belief space and the vertical axis represents belief value. FIG. 3 shows the candidate set Γ ⁽¹⁾ when L = 6. Each line in FIG. 3 represents an initial linear function. Since L = 6, the number of linear functions is also six.

Here, the value function V ₁ (b) is given by the equation (4) from the candidate set Γ ⁽¹⁾ .
The left side of FIG. 4 is a graph showing an example of a certain candidate set Γ ⁽¹⁾ , and the right side of FIG. 4 is a graph showing the value function V ₁ (b) obtained from the candidate set Γ ⁽¹⁾ . The value of the value function V ₁ (b) is also called belief value.

As shown in FIG. 4, not all elements of the candidate set Γ ⁽¹⁾ are linear elements of the value function V ₁ (b). In fact, only a small part contributes. Only elements indicated by solid lines in the left graph of FIG. 4 are linear elements of the value function V ₁ (b). The right graph of FIG. 4 shows only elements that contribute as linear elements of the value function V ₁ (b). In this manner, eliminating elements that are not linear elements of the value function V ₁ (b ⁾ from the candidate set Γ ⁽¹⁾ is called pruning.

FIG. 5 is a graph showing the candidate set Γ ⁽¹⁾ when the number of backup steps T = 2 in the example of FIG. 3, and FIG. 6 shows the number of backup steps T = 3 in the example of FIG. It is a graph which shows candidate set Γ ⁽¹⁾ in the case of becoming. As illustrated in FIGS. 5 and 6, in general, this pruning process is performed because the number of elements in the candidate set increases exponentially with an increase in the number of backup steps (also referred to as a value iteration interval). The amount becomes extremely large, the calculation load increases, and the calculation time also increases. This is a problem of the calculation method of the value function by value iteration in POMDP.

Therefore, in the present embodiment, in order to solve the pruning problem, the value function V ₁ is not strictly selected from the candidate set Γ ⁽¹⁾ but becomes a linear element of the value function V ₁ (b). ₁ Calculate a linear function approximating the linear element of (b).

(Dual transformation)
For this purpose, first, the dual transformation unit 2 uses the elements of the candidate set Γ ⁽¹⁾ .
Is made to correspond to a point in a certain N-dimensional space as follows. Coefficient (ν ⁽¹⁾ ) Linear function by considering _n as coordinate value of point in N-dimensional space
To point in N-dimensional space
A mapping to can be defined. This is called dual transformation, and the mapped N-dimensional space is called dual space.

FIG. 7 is a graph showing an example of dual transformation. In the example of FIG. 7, the linear functions 101 to 106 are mapped to the points 201 to 206 in the dual space, respectively. The dual transformation shown here is only one example, and is not limited to the one described here. In addition, a set composed of points obtained by mapping the elements of the candidate set Γ ⁽¹⁾ to the dual space is represented as Γ ′ ⁽¹⁾ .

(Convex hull and value function)
The hierarchical relationship of linear functions in belief space is preserved even in dual space. That is, as shown in FIG. 8, due to the property that the value function is given by a downwardly convex piecewise linear function, the linear element giving the value function corresponds to the point set Γ ′ ⁽¹⁾ of the candidate set mapped to the dual space. The upper side of the convex hull. Therefore, the problem of obtaining the linear element of the value function from the candidate set Γ ⁽¹⁾ results in the problem of obtaining the upper side of the convex hull for the point set Γ ′ ^{(1) in} the dual space.

  When the number of states N = 2, the dual space is two-dimensional, in which case the algorithm for obtaining the convex hull is essentially a sorting problem, and a fast processing algorithm is known. For example, the Kirkpatrick-Seidel quick convex hull algorithm can be used. Therefore, the value function can be calculated at high speed by mapping the problem to the dual space. In other words, when the number of states N = 2 by this process, it is possible to solve the difficulty of value function calculation due to POMDP value iteration.

  However, in a space where the number of states is N = 3 or more, it is known that the amount of calculation for obtaining the convex hull increases exponentially with respect to the number of dimensions even if the best algorithm known so far is used. . Therefore, simply mapping to the convex hull calculation problem in the dual space cannot easily solve the difficulty of the value function calculation due to the POMDP value iteration.

A further consideration is that as the dimension N of the space increases, the number of vertices of the convex hull increases in proportion to the number of points in the point set Γ ′ ⁽¹⁾ . In other words, the value function is subdivided into many categories. Strictly speaking, even if it is exactly the same, the difference between adjacent segments becomes smaller, and it becomes a smooth function. Practically, it is desirable to approximate with a piecewise linear function with a small number of sections that approximate it better than such a smooth function.

(Approximate calculation of convex hull by NMF)
For the reasons described above, in the present embodiment, the dual value iteration processing unit 3 of the value function calculation unit 340 includes the convex hull approximation calculation unit 32. The convex hull approximation calculation unit 32 does not calculate the convex hull exactly, but calculates a polytope that closely approximates it by non-negative matrix factorization (NMF). Hereinafter, such a polytope is referred to as an “approximate convex hull” for simplicity. Considering the approximate convex hull in this way means that the discrete combination problem of finding the convex hull for the discrete point set Γ ′ ⁽¹⁾ is the point set Γ ′ in the entire continuous N-dimensional space. ^This means that the problem of computing a polytope that closely approximates the convex hull in ⁽¹⁾ is alleviated.

Now, assuming that the number of elements of the point set Γ ′ ⁽¹⁾ is | Γ ′ ⁽¹⁾ |, an N × | Γ ′ ⁽¹⁾ | matrix in which these elements are arranged is set as in Expression (5).
Here, φ _r ′ ⁽¹⁾ is assumed to be a vertical vector. The number of vertices of the convex hull of the point set Γ ′ ⁽¹⁾ is unknown, but here it is assumed that the number of vertices is given by K. In other words, here, a problem is considered in which the number of vertices K is suppressed so that the calculation amount can be calculated at a sufficiently high speed, and a polytope that best approximates the convex hull is obtained.

Now, let the vertex of the desired polytope be w _k ⁽¹⁾ (k = 1, 2,..., K). These are N-dimensional vectors, and the condition that this polytope includes the point set Γ ′ ⁽¹⁾ is given by the equations (6) and (7).

Here, h _r ⁽¹⁾ is a K-dimensional vector, and Expression (8) is established.
W ⁽¹⁾ is an N × K matrix, H ⁽¹⁾ is a K × | Γ ′ ⁽¹⁾ | matrix, and the following formulas (9) and (10) hold.

Furthermore, when the number of vertices of this polytope is equal to the number of vertices of the actual convex hull, in order to match the convex hull, the vertex w _k ⁽¹⁾ of the polytope is an element of the point set Γ ′ ⁽¹⁾ . Request that it be represented by That is, it is as shown in Formula (11) and Formula (12).
However, g _k ⁽¹⁾ is | Γ ′ ⁽¹⁾ | dimensional vector and satisfies Expression (13).
G ⁽¹⁾ is a | Γ ′ ⁽¹⁾ | × K matrix, as shown in Equation (14).
From the equations (6), (7), (11), and (12), the following self-consistent factorization problem of the following equations (15) and (16) is obtained.

An algorithm for solving this problem is given in Non-Patent Document 2, for example. According to the method in this document, when actually solving the problems of the equations (15) and (16), first, a projection to a certain two-dimensional space is taken, and a convex hull is obtained in the two-dimensional space, The operation of returning it to the original space is repeated. According to this method, even if the size | Γ ′ ⁽¹⁾ | of the point set Γ ′ ⁽¹⁾ increases, it can be calculated at high speed.

By solving the equations (15) and (16), the vertex W ⁽¹⁾ = V ⁽¹⁾ G ^{(1) of} the approximate convex hull is obtained. FIG. 9 is a graph showing an example of calculating the approximate convex hull. In the example of FIG. 9, K = 5. In the figure, the broken line is the original convex hull, and the solid line is an approximate convex hull approximating it.

(Membership function definition)
A certain action u is assigned to each element φ of the candidate set Γ ⁽¹⁾ . To which action u the element φ is assigned is called membership. Generalizing this, the ratio of the contribution of the element φ to the action u _l is expressed as α _l (φ) ∈ [0, 1], and α (φ) = (α ₁ (φ), α ₂ (φ),.・ ・ Α _L (φ)) is called the membership function.

The membership determination unit 34 determines the membership of the vertex w _k ⁽¹⁾ of the approximate convex hull. The membership of the element φ ′ of the dual set Γ ′ ⁽¹⁾ is given by the membership of the element φ of the corresponding candidate set Γ ⁽¹⁾ . That is, when the element φ of the candidate set Γ ⁽¹⁾ corresponds to the action u _l , the membership of the element φ ′ ⁽¹⁾ is given by Expression (17).
Hereinafter, for ^{_{simplicity, α l'(φ l '(}} 1)) = denoted as [delta] _LL'.

By removing the restriction on the point set Γ ′ ⁽¹⁾ and performing the approximation, the membership of the vertex w _k ⁽¹⁾ of the obtained approximate convex hull becomes non-trivial. Therefore, the membership determination unit 34 adds the membership of the vertex w _k ⁽¹⁾ by adding the weight of how the vertex w _k ⁽¹⁾ is associated with each element of the point set Γ ′ ^(1). decide. In other words, the membership function of the vertex w _k ⁽¹⁾ of the approximate convex hull is a function that obtains correct membership when it is applied to each element of the point set Γ ′ ⁽¹⁾ . By determining membership in this way, it can be expected that if the approximate convex hull exactly matches the convex hull of the point set Γ ′ ⁽¹⁾ , it will result in correct membership.

Specifically, the membership determination unit 34 obtains a membership function for the vertex w _k ⁽¹⁾ by the following equation (18).
As shown in Expression (18), the membership determination unit 34 uses a transformation matrix G that converts each element of the point set Γ ′ ⁽¹⁾ into an approximate convex hull, and calculates each element of the point set Γ ′ ⁽¹⁾ . The converted membership is determined as the membership of the vertex of the approximate convex hull.

Action corresponding to vertex w _k ⁽¹⁾
Is determined by the following equation (19).

(Upper side of approximate convex hull)
When the labeled convex hull approximation calculation unit 31 determines the vertex w _k ⁽¹⁾ of the approximate convex hull and its membership as described above, the convex hull upper side extraction unit 36 of the pruning unit 35 determines from the approximate convex hull. By extracting the upper side, a function corresponding to the approximate value function is obtained.

In extracting the upper side of the convex hull, the coordinates of the N-dimensional space are used.
Against its direction cosine
think of.

In the extraction of the upper side of the convex hull, the point at which the distance from the origin is the largest among the intersections of a convex hull in the dual space and the half line extending from the origin parallel to the previous cosine in the dual space and the convex hull And a set of points left in this way can be extracted as the upper side of the convex hull in the dual space. The vertex of the upper side of the convex hull of this dual set Γ ′ ⁽¹⁾ gives a linear function corresponding to the value function.

Similarly, the upper side is defined for the approximate convex hull, and the vertexes w _k ⁽¹⁾ (k = 1, 2,..., K) of the approximate convex hull belong to the upper side.
far. Here, K ≦ J and 1 ≦ k _s ≠ k _t ≦ K (s ≠ t).

(Inverse transformation of approximation result and value function)
The inverse dual transform unit 4 is a vertex belonging to the upper side of the approximate convex hull.
Is returned to a linear function in belief space by inverse dual transformation. The inverse dual transform unit 4 is a vertex belonging to the upper side of the approximate convex hull.
Is used to obtain a linear function in the belief space by Expression (20).

  FIG. 10 is a graph showing an example of inverse dual transformation. In the left graph of FIG. 10, vertices 211, 212, 213, and 214 belonging to the upper side of the approximate convex hull are converted into linear functions 111, 112, 113, and 114 in the belief space of the right graph of FIG. Yes.

Based on these linear functions, the value function determining unit 5 uses the approximate value function according to the equation (21).
Find piecewise linear elements of.

With the above processing, the value function calculation unit 340 can obtain an approximate value function with the number of backup steps T = 1. Note that if the convex hull is obtained not strictly but approximate, a strict value function V ₁ (b) is obtained.

(Belief update by observation)
In the above description, the value function calculation unit 340 calculates the value function using only the reward that can be obtained immediately. However, in general, a better value function can be obtained by considering the rewards obtained in the future.

  This means that before selecting an action that belongs to the “behavior for final decision” category, selecting an action that belongs to the “behavior for state observation” category, ie, performing an observation before the next action. Means that the state of belief can be updated. This observation makes it possible to recognize the value of the belief more correctly, so that the expected value of the sum of the finally obtained rewards is improved.

First, it is assumed that the belief is b = (p ₁ , p ₂ ,..., P _N ). Here, by executing the action u, a transition to another belief state is assumed. Therefore, if further observation is performed and the observed value o is obtained, a posterior distribution regarding the belief is obtained by Expression (22).

The optimal action strategy for this posterior distribution of beliefs is the value function
Given in.

Therefore, the linear function updating unit 6 updates the linear function in order to improve the expected value of the sum of rewards obtained in the future. Specifically, the linear function update unit 6 updates the elements of the candidate set Γ ⁽¹⁾ with Expression (23).
The updated candidate set is output to the dual conversion unit 2 and a value function of the number of backup steps T = 2 is obtained by the same process as described above.

From the equation (22), the value function V ₂ (b) at the backup step number T = 2 is calculated by the equation (24).

Expression (24) can be rewritten as the following Expression (25) and Expression (26).
here,
Was abbreviated.

The value function calculation unit 340 calculates the value function V ₂ (b) from the set of linear functions of the candidate set Γ ⁽²⁾ of Expression (27) according to Expression (28).

In addition, the membership determination unit 34
Is obtained by Expression (29).

Approximate value function from candidate set Γ ⁽¹⁾
Linear elements of
Is repeated from the candidate set Γ ⁽²⁾ to the value function V ₂ (b) or its approximate function.
Linear elements of
Can be requested.

In addition, by increasing the backup step, linear elements
By generating the candidate set Γ ⁽²⁾ from the same procedure, the linear element
A candidate set Γ ⁽³⁾ can be generated from

As described above, the value function V _T (b) in which the backup step is performed for an arbitrary value repeating section T or its approximate function.
Can be calculated. The value function calculation unit 340 performs the value function V _T (b) and its approximate function as described above.
Calculate

The policy decision unit 350 performs an action for each linear element.
Assign.

11 and 12 are graphs showing examples of candidate sets Γ ⁽²⁾ and Γ ⁽³⁾ and their dual transformations in the backup steps T = 2 and T = 3. In the example of FIG. 11, the linear functions 121 to 124 are mapped to the points 221 to 224 in the dual space, respectively.

  13 and 14 are graphs showing an example of approximate convex hull calculation in the backup steps T = 2 and T = 3. FIG. 15 is a graph showing an example of the approximate value function when the backup step T = 3. FIG. 16A is a graph showing an example of an approximate value function, and FIG. 16B is a graph showing an example of a strict value function. As can be seen by comparing FIG. 16A and FIG. 16B, the difference between the strict value function and the approximate value function is small, and it can be seen that the calculation method of the present embodiment is effective.

The weighting coefficient of the linear function that gives the value function at the point b on the belief space is given by Expression (30).
The policy determination unit 350 determines an optimal policy from the membership of the linear function according to Expression (31).

  FIG. 17 is a graph showing an example of the value function and the optimum action policy in the present embodiment. As shown in FIG. 17, if the belief space is 0 to 3.2 according to the value function, the dialogue system 100 outputs a control command for executing an action “present information about meals”, and the belief space is If 0.32 to 8.2, execute the action “rehearse”, and if the belief space is 8.2 to 1, output a control command to execute the action “present information about shopping” To do.

Next, the operation of the control policy determination device 30 will be described. FIG. 18 is an operation flow diagram of the control policy determination device 30. The value function calculator 340 loads the perceptual model, the behavior model, and the reward function from the perceptual model unit 322, the behavior model unit 321, and the reward function model unit 331, respectively (step S11). Next, the number of backup steps is set to T = 1 (step S12), and the initial linear function generation unit 1 generates a set of initial linear functions (step S13). Then, it is determined whether T has reached the upper limit (T _max ) (step S14), and if not reached (NO in step S14), the dual conversion unit 2 performs dual conversion on the set of linear functions. (Step S15).

  Next, the convex hull approximation calculation unit 32 calculates an approximate convex hull of a plurality of points obtained by the dual transformation (step S16). Then, the membership determination unit 14 calculates the membership of the vertex of the approximate convex hull (step S17), and the convex hull upper side extraction unit 36 extracts the upper side of the approximate convex hull (step S18). When the upper side of the approximate convex hull is extracted, the inverse dual transform unit 4 performs an inverse dual transform on the vertex of the upper side of the approximate convex hull (step S19). Then, the value function determining unit 5 updates the set of linear functions with the linear elements of the value function obtained by inverse dual transformation (step S20).

Thereafter, the backup step number T is incremented (step S21), and it is determined again whether the backup step number exceeds the upper limit (T _max ) (step S14). In this way, steps S15 to S21 are repeated until the number of backup steps exceeds the upper limit. When the number of backup steps exceeds the upper limit (YES in step S14), value function calculation unit 340 is obtained at that time. A set of linear functions giving a value function is output (step S22). The value function at this time is the above approximate value function. The policy determination unit 350 determines an optimal policy using this approximate value function.

  In the above embodiment, when the number of backup steps is 2 or more, the update of the candidate set of linear elements is performed at once for all the observed values as in Expression (26). However, as a modification, Expression (25) May be sequentially executed for each observation value. In other words, the update of the candidate set of linear elements is performed sequentially for each observation value, and the candidate set of linear elements for all observation values is updated while pruning the partially updated linear function set. Processing may be executed.

  The operation of the control policy determination device 30 in this modification will be described. FIG. 19 is an operation flowchart of the control policy determination device 30 according to the modification. The value function calculator 340 loads the perceptual model, the behavior model, and the reward function from the perceptual model unit 322, the behavior model unit 321 and the reward function model unit 331, respectively (step S31). Next, the number of backup steps is set to T = 1 (step S32), and the initial linear function generation unit 1 generates a set of initial linear functions (step S33). Then, the dual conversion unit 2 dual converts the set of initial linear functions (step S34).

  Next, the convex hull approximation calculation unit 32 calculates the approximate convex hull of a plurality of points obtained by the dual transformation (step S35). Then, the membership determination unit 14 calculates the membership of the vertex of the approximate convex hull (step S36), and the convex hull upper side extraction unit 36 extracts the upper side of the approximate convex hull (step S37). When the upper side of the approximate convex hull is extracted, the inverse dual transform unit 4 performs an inverse dual transform on the vertex of the upper side of the approximate convex hull (step S38). With the above processing, a value function with the number of backup steps T = 1 is obtained.

Thereafter, the backup step number T is incremented (step S39), and it is determined whether the backup step number T exceeds the upper limit (T _max ) (step S40). If T has not reached the upper limit (T _max ) (NO in step S40), the observed value m is set to 1 (step S41). Then, the linear function update unit 6 updates the linear function group related to the observation value m (step S42).

  Next, the dual transformation unit 2 dual transforms the set of linear functions (step S43), and the convex hull approximation calculation unit 32 calculates an approximate convex hull of a plurality of points obtained by the dual transformation (step S44). The membership determination unit 14 calculates the membership of the vertex of the approximate convex hull (step S45), the convex hull upper side extraction unit 36 extracts the upper side of the approximate convex hull (step S46), and the inverse dual transform unit 4 However, by performing inverse dual transformation on the top vertex of the approximate convex hull (step S47), a value function is acquired. Then, the observed value m is incremented (step S48), and it is determined whether m exceeds the observed value number M (step S49). If not exceeded (NO in step S49), the process returns to step S42. Steps S42 to S49 are repeated.

If the incremented observation value m exceeds the observation value number M (YES in step S49), that is, if the value function is calculated by updating the linear function group for all observation values, the process proceeds to step S39. Returning, the backup step number T is incremented, and it is determined whether the backup step number T exceeds the upper limit (T _max ). In this way, steps S39 to S49 are repeated until the number of backup steps exceeds the upper limit. When the number of backup steps exceeds the upper limit (YES in step S40), the value function calculation unit 340 is obtained at that time. A set of linear functions giving a value function is output (step S50). The value function at this time is the above approximate value function. The policy determination unit 350 determines an optimal policy using this approximate value function.

  As described above, the embodiment has been described by taking the control of the interactive system as an example, but the application of the control policy determining apparatus of the present invention is not limited to the control of the interactive system.

  The present invention has an effect that it is possible to perform approximate execution of a value iteration calculation at high speed, and is useful as a control policy determination device or the like that determines a control policy based on environment sensing information.

DESCRIPTION OF SYMBOLS 1 Initial linear function production | generation part 2 Dual transformation part 3 Dual value iterative processing part 4 Inverse dual transformation part 5 Value function determination part 6 Linear function update part 10 Speech input part 20 Speech recognition part 30 Control policy decision apparatus 31 Convex hull approximation with label Calculation unit 32 Convex hull approximation calculation unit 33 ConvexNMF calculation unit 34 Membership determination unit 35 Pruning unit 36 Convex hull upper side extraction unit 40 Output unit 100 Interactive system

Claims (7)

A control policy determination device that determines a control policy based on environmental sensing information including uncertainty,
A linear function generation unit that generates a candidate set of linear functions that give linear elements of a value function in a belief space based on the environment sensing information;
A dual transform unit for transforming the candidate set on the belief space into a plurality of points on the dual space;
A convex hull approximation calculator for calculating an approximate convex hull approximating the convex hull of the plurality of points;
A membership determination unit for determining a membership function of a vertex of the approximate convex hull;
A convex hull upper side extraction unit for extracting the upper side of the approximate convex hull;
An inverse dual transformation unit that inversely transforms vertices belonging to the upper side into a linear function on the belief space;
A linear function updating unit that updates the linear function according to the number of backup steps based on the linear function obtained by the inverse transformation;
With
The dual transform unit further transforms the linear function updated according to the number of backup steps into a plurality of points on the dual space as the candidate set,
The control policy determination device further includes:
A value function determining unit for obtaining a plurality of linear elements of the approximate value function based on the linear function obtained by the inverse transformation after the update of the linear function of the number of backup steps;
A policy determining unit that assigns an action according to the membership function to each of a plurality of linear elements of the approximate value function;
A control policy decision device comprising:   The membership determining unit determines a membership function that obtains a correct membership when applied to a plurality of points on the dual space as a membership function of a vertex of the approximate convex hull. Item 2. The control policy determination device according to Item 1.   The membership determination unit converts a membership function of a plurality of points on the dual space with a matrix that converts a plurality of points on the dual space to the approximate convex hull, The control policy determining apparatus according to claim 2, wherein the control policy determining apparatus is determined as a vertex membership function.   4. The control policy determination device according to claim 1, wherein the linear function update unit updates the linear function in each backup step for each observation value. 5. A control policy determination device according to any one of claims 1 to 4,
An environmental sensing information input unit for inputting the environmental sensing information;
An output unit for outputting a control command for executing an action assigned by the control policy determination device;
A control system characterized by comprising: A control policy determination method for obtaining a control policy based on environmental sensing information including uncertainty,
Environmental sensing information input step for inputting the environmental sensing information;
Generating a linear function candidate set that gives a linear element of a value function in a belief space based on the environmental sensing information; and
A dual transformation step of transforming the candidate set on the belief space into a plurality of points on the dual space;
An approximate convex hull calculating step of calculating an approximate convex hull approximating the convex hull of the plurality of points;
A membership determination step for determining a membership function of a vertex of the approximate convex hull;
A convex hull upper side extracting step of extracting the upper side of the approximate convex hull;
An inverse dual transformation step of inversely transforming vertices belonging to the upper side to a linear function on the belief space;
A linear function updating step for updating the linear function according to the number of backup steps based on the linear function obtained by the inverse transformation;
Including
The dual transformation step further transforms a linear function updated according to the number of backup steps into a plurality of points on the dual space as the candidate set,
The control strategy determination method further includes:
A value function determining step for obtaining a plurality of linear elements of the approximate value function based on the linear function obtained by the inverse transformation after the update of the linear function of the number of backup steps;
A policy determining step of assigning an action according to the membership function to each of a plurality of linear elements of the approximate value function;
A control policy determination method comprising: Computer
A control policy determination device that determines a control policy based on environmental sensing information including uncertainty,
A linear function generation unit that generates a candidate set of linear functions that give linear elements of a value function in a belief space based on the environment sensing information;
A dual transform unit for transforming the candidate set on the belief space into a plurality of points on the dual space;
A convex hull approximation calculator for calculating an approximate convex hull approximating the convex hull of the plurality of points;
A membership determination unit for determining a membership function of a vertex of the approximate convex hull;
A convex hull upper side extraction unit for extracting the upper side of the approximate convex hull;
An inverse dual transformation unit that inversely transforms vertices belonging to the upper side into a linear function on the belief space;
A linear function updating unit that updates the linear function according to the number of backup steps based on the linear function obtained by the inverse transformation;
With
The dual transform unit further transforms the linear function updated according to the number of backup steps into a plurality of points on the dual space as the candidate set,
The control policy determination device further includes:
A value function determining unit for obtaining a plurality of linear elements of the approximate value function based on the linear function obtained by the inverse transformation after the update of the linear function of the number of backup steps;
A policy determining unit that assigns an action according to the membership function to each of a plurality of linear elements of the approximate value function;
A control policy determination program for functioning as a control policy determination device.