JP7338858B2 - Behavior learning device, behavior learning method, behavior determination device, and behavior determination method - Google Patents

Description

The present invention relates to an action learning device, an action learning method, an action determining device, and an action determining method.

In recent years, as a machine learning method, deep learning using multi-layer neural networks has attracted attention. Deep learning uses a calculation method called back propagation to calculate the output error when a large amount of teacher data is input to a multi-layer neural network and perform learning to minimize the error.

Patent Documents 1 to 3 disclose a neural network processing device that enables the construction of a neural network with a small amount of labor and computational processing by defining a large-scale neural network as a combination of a plurality of sub-networks. ing. Further, Patent Document 4 discloses a structure optimization device that optimizes a neural network.

JP-A-2001-051968 JP-A-2002-251601 Japanese Patent Application Laid-Open No. 2003-317073 JP-A-09-091263

However, deep learning requires a large amount of high-quality data as teacher data, and requires a long time for learning. Patent Documents 1 to 4 propose a method for reducing the labor and computational processing amount for constructing a neural network, but in order to further reduce the system load, etc., it is possible to learn behavior using a simpler algorithm. A possible action learning device has been desired.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an action learning device and an action decision device that can learn and select an action according to the environment and one's own situation with a simpler algorithm.

According to one aspect of the present invention, an action selection unit that selects a candidate action to be executed with respect to the environment based on situation information data representing the environment and one's own situation, and the action selected by the action selection unit an evaluation acquisition unit that acquires an evaluation of the candidate for the candidate, which is an evaluation indicating a judgment to execute or not to execute the action candidate in the situation indicated by the situation information data, together with a reason; and based on the reason in the evaluation, a slot generation unit that generates a slot indicating a point of interest in the situation information data; and user learning that generates a user learning model in which the action candidate is associated with the situation information data, the slot, and the judgment in the evaluation. A behavior learning device having a model generation unit is provided.

According to another aspect of the present invention, for each of a plurality of action candidates, situation information data representing the environment and the self's situation; A storage unit that holds a user learning model that is associated with data and a decision to execute or not execute the action candidate in the situation indicated by the slot, and a current situation that represents the current environment and the self's situation an action selection unit that selects an action candidate to be executed in the environment based on information data; and the user learning model linked to the action candidate selected by the action selection unit from the storage unit , a user learning model extracting unit for extracting the user learning model having the situation information data most suitable to the current situation information data; and the slots of the user learning model extracted with the current situation information data. and a behavior determination unit that determines whether or not to execute the behavior candidate selected by the behavior selection unit based on the relationship of the behavior determination device.

According to still another aspect of the present invention, a step of selecting a candidate action to be executed with respect to the environment based on situation information data representing the environment and one's own situation; a step of acquiring a user's evaluation of the action candidate, the evaluation indicating a judgment to execute or not to execute the action candidate in the situation indicated by the situation information data together with a reason, based on the reason in the evaluation; generating a slot indicating a point of interest in the situation information data; and generating a user learning model in which the action candidate is associated with the situation information data, the slot, and the judgment in the evaluation. A behavioral learning method is provided.

According to still another aspect of the present invention, based on current situation information data representing the current environment and one's own situation, a step of selecting a candidate action to be executed with respect to the environment; For each candidate, situation information data representing the environment and one's own situation, a slot indicating a point of interest in the situation information data, and executing or performing the action candidate in the situation indicated by the situation information data and the slot and the user learning model linked to the action candidate selected in the selecting step from among the user learning models linked to the current situation information data. extracting the user learning model having the most relevant contextual information data; and selecting in the selecting step based on the relationship between the current contextual information data and the slots of the extracted user learning model. and determining whether to execute the proposed action.

ADVANTAGE OF THE INVENTION According to this invention, action learning and selection according to an environment and one's own situation can be implement|achieved with a simpler algorithm. In addition, users' comments on actions selected according to situation information can be accumulated and used as know-how, and more appropriate actions can be selected.

BRIEF DESCRIPTION OF THE DRAWINGS It is the schematic which shows the structural example of the action learning apparatus by 1st Embodiment of this invention. It is a schematic diagram showing an example of composition of a situation learning part in an action learning device by a 1st embodiment of the present invention. It is a schematic diagram showing an example of composition of a score acquisition part in an action learning device by a 1st embodiment of the present invention. It is a schematic diagram showing a configuration example of a neural network unit in the action learning device according to the first embodiment of the present invention. It is a schematic diagram showing a configuration example of a learning cell in the behavior learning device according to the first embodiment of the present invention. It is a schematic diagram showing a configuration example of a usage learning unit in the action learning device according to the first embodiment of the present invention. 4 is a flow chart showing a learning method of the situation learning unit in the action learning device according to the first embodiment of the present invention; It is a figure which shows an example of the situation information data which a situation information production|generation part produces|generates. It is a figure which shows an example of the situation information data which a situation information production|generation part produces|generates, and its element value. 4 is a flow chart showing a learning method of a usage learning unit in the action learning device according to the first embodiment of the present invention; FIG. 4 is a diagram showing an example of situation information data generated from situation information by a situation information generation unit; FIG. 10 is a diagram showing a display example of information about the action selected by the situation information and the action selection unit, and an example of a user episode; FIG. 10 is a diagram showing an example of a method of generating a slot indicating a point of interest in status information data; 4 is a flow chart showing an action determination method in the action learning device according to the first embodiment of the present invention; FIG. 2 is a diagram illustrating an example of a method for evaluating suitability of a user learning model for contextual information; BRIEF DESCRIPTION OF THE DRAWINGS It is the schematic which shows the hardware structural example of the action-learning apparatus by 1st Embodiment of this invention. It is a flowchart which shows the learning method of the situation learning part in the action learning apparatus by 2nd Embodiment of this invention. It is a schematic diagram showing an example of composition of an action learning device by a 3rd embodiment of the present invention. It is a schematic diagram showing an example of composition of an action deciding device by a 4th embodiment of the present invention.

[First embodiment]
A schematic configuration of a behavior learning device according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 6. FIG. FIG. 1 is a schematic diagram showing a configuration example of an action learning device according to this embodiment. FIG. 2 is a schematic diagram showing a configuration example of the situation learning unit in the action learning device according to this embodiment. FIG. 3 is a schematic diagram showing a configuration example of the score acquisition unit in the action learning device according to this embodiment. FIG. 4 is a schematic diagram showing a configuration example of the neural network unit in the action learning device according to this embodiment. FIG. 5 is a schematic diagram showing a configuration example of a learning cell in the action learning device according to this embodiment. FIG. 6 is a schematic diagram showing a configuration example of the usage learning unit in the action learning device according to this embodiment.

The action learning device 100 according to this embodiment has a situation learning section 110 and a usage learning section 120, as shown in FIG. The situation learning unit 110 performs learning (situation learning) based on the information received from the environment 200 and its own situation, and selects an action to be executed with respect to the environment 200 . The usage learning unit 120 receives the user's evaluation (advice) of the action selected by the situation learning unit 110, and generates a user learning model that associates the action selected by the situation learning unit 110 with the user's evaluation (usage learning). Also, the usage learning unit 120 determines actions to be performed on the environment 200 based on the actions selected by the situation learning unit 110 and the user learning model. Action learning device 100 constitutes action learning system 400 together with environment 200 .

For example, as shown in FIG. 2, the situation learning unit 110 can be configured by an action candidate acquisition unit 10, a situation information generation unit 20, a score acquisition unit 30, an action selection unit 70, and a score adjustment unit 80. .

The action candidate acquisition unit 10 has a function of extracting possible actions (action candidates) based on the information received from the environment 200 and the situation of the agent itself (agent). An agent is a subject that learns and selects actions. An environment is an object on which an agent works.

The situation information generation unit 20 has a function of generating situation information data representing information related to actions based on information received from the environment 200 and the situation of the user (situation information). Information included in the situation information data is not particularly limited as long as it relates to actions, and may include, for example, environment information, time, number of times, self-state, past actions, and the like.

The score acquisition unit 30 has a function of acquiring scores for the situation information data generated by the situation information generation unit 20 for each of the action candidates extracted by the action candidate acquisition unit 10 . Here, the score is a variable used as an index representing the expected effect of the action result. For example, the score is large when the evaluation of the action result is expected to be high, and the score is small when the evaluation of the action result is expected to be low.

The action selection unit 70 selects the action candidate with the highest score acquired by the score acquisition unit 30 from among the action candidates extracted by the action candidate acquisition unit 10 . The action selection unit 70 also has a function of executing the selected action on the environment 200 or notifying the usage learning unit 120 of the selected action.

The score adjustment unit 80 has a function of adjusting the score value associated with the selected action according to the result given to the environment 200 by the action selected by the action selection unit 70 . For example, if the evaluation of the action result is high, the score is raised, and if the evaluation of the action result is low, the score is lowered.

The score acquisition unit 30 may be configured including a neural network unit 40, a determination unit 50, and a learning unit 60, as shown in FIG. 3, for example. The learning unit 60 can be configured including a weight correction unit 62 and a learning cell generation unit 64 .

The neural network unit 40 may be composed of a two-layer artificial neural network including an input layer and an output layer, as shown in FIG. 4, for example. The input layer has a number of cells (neurons) 42 corresponding to the number of element values extracted from one piece of situation information data. For example, if one context information data includes M element values, the input layer includes at least M cells 42 ₁ , 42 ₂ , . . . , _{42 i} , . The output layer comprises at least a number of cells (neurons) 44 corresponding to the number of possible actions. For example, the output layer includes N cells 44 ₁ , 44 ₂ , . . . , _{44 j} , . Each of the cells 44 that make up the output layer is associated with one of possible actions. A predetermined score is set in each cell 44 .

M element _values I _{1 ,} I _{2 ,} . _{. .} , I _i , . be done. Each of the _cells 42 _{1 ,} 42 _{2 ,} . . . , _{42 i , .}

For each branch (axon) connecting the cell 42 and the cell 44, a weighting coefficient ω for giving a predetermined weighting to the element value I is set. For example, the branches _connecting the cells 42 ₁ , _{42 2} , . _. . , _{42 i ,} . , ω _Mj are set. Accordingly, the cell 44 _j performs the calculation shown in the following equation (1) and outputs the output value O _j .

In this specification, one cell 44, a branch (input node) that inputs the element values I ₁ to I _M to the cell 44, and a branch (output node) that outputs the output value O from the cell 44 may be collectively referred to as a learning cell 46.

The determining unit 50 compares the correlation value between the plurality of element values extracted from the situation information data and the output value of the learning cell with a predetermined threshold, and determines whether the correlation value is greater than or equal to the threshold or less than the threshold. judge. An example of a correlation value is the likelihood for the output value of a learning cell. Note that each of the learning cells 46 may have the function of the determination unit 50 .

The learning unit 60 is a functional block that performs learning of the neural network unit 40 according to the determination result of the determination unit 50 . The weight correction unit 62 updates the weighting coefficient ω set to the input node of the learning cell 46 when the correlation value is equal to or greater than a predetermined threshold. Also, the learning cell generation unit 64 adds a new learning cell 46 to the neural network unit 40 when the correlation value is less than the predetermined threshold.

The usage learning unit 120 can be configured by a selection behavior acquisition unit 130, an evaluation acquisition unit 140, an behavior evaluation unit 150, an behavior determination unit 160, and a storage unit 170, as shown in FIG. 6, for example.

The selected action acquisition unit 130 has a function of acquiring information about the action selected by the action selection unit 70 from the situation learning unit 110 . The evaluation acquisition unit 140 has a function of acquiring a user's (advisor's) evaluation of the information on the action selected by the action selection unit 70 . This evaluation indicates the judgment to perform or not to perform the action selected by the action selection unit 70 in the situation indicated by the situation information data, together with the reason.

The behavior evaluation unit 150 may include a slot generation unit, a user learning model generation unit, and a user learning model extraction unit. The slot generation unit has a function of generating a slot indicating a point of interest in the situation information data based on the reason in the user's evaluation. The user learning model generation unit has a function of generating a user learning model in which the action selected by the action selection unit 70 is linked to the situation information data, the slot, and the judgment in the evaluation, and storing it in the storage unit 170 . The user learning model extraction unit retrieves from the storage unit 170 the user learning model having the situation information data that is most compatible with the current situation information data among the user learning models linked to the action selected by the action selection unit 70. It has a function to extract

The action determination unit 160 determines whether or not to execute the action selected by the action selection unit 70 on the environment 200 based on the relationship between the current situation information data and the slot of the user learning model extracted by the user learning model extraction unit. It has a function to judge whether

That is, the usage learning unit 120 learns the user's evaluation (advice) for the action selected by the action selection unit 70, and determines the action to be executed in the environment 200 based on the learning result.

Next, a behavior learning method using the behavior learning device 100 according to this embodiment will be described with reference to FIGS. 7 to 15. FIG. In order to facilitate understanding, the player's actions in the card game "Millionaire" will be used as an example to supplement the description as appropriate. However, the behavior learning device 100 according to this embodiment can be widely applied to various uses for selecting behavior according to the situation of the environment 200 .

First, the learning method in the situation learning unit 110 of the behavior learning device 100 according to this embodiment will be described with reference to FIGS. 7 to 9. FIG. FIG. 7 is a flow chart showing a learning method in the situation learning unit of the action learning device according to this embodiment. FIG. 8 is a diagram showing an example of situation information data. FIG. 9 is a diagram showing an example of situation information data and its element values.

The action candidate acquisition unit 10 extracts actions (action candidates) that can be taken under the situation based on the information received from the environment 200 and the situation (step S101). Although the method for extracting action candidates is not particularly limited, for example, extraction can be performed using a rule-based program.

In the case of the “millionaire”, the information received from the environment 200 includes, for example, the type of cards on the field (for example, whether it is a single card or a plurality of cards), strength, and whether other players have passed. information such as whether or not Examples of the player's own situation include information on cards in hand, information on cards played so far, number of rounds, and the like. The action candidate acquisition unit 10 extracts all actions (action candidates) that can be taken under the environment 200 and the self's situation in accordance with the "millionaire" rule. For example, if the player has a plurality of stronger cards of the same type as those on the field in his or her hand, each of the actions of playing one of these cards becomes an action candidate. Also, passing one's turn is one of the action candidates.

Next, it is checked whether each action candidate extracted by the action candidate acquisition unit 10 is linked to at least one learning cell 46 included in the neural network unit 40 of the score acquisition unit 30 . If there is an action candidate that is not linked to the learning cell 46 , the learning cell 46 that is linked to the action candidate is newly added to the neural network unit 40 . If all possible actions are known, learning cells 46 associated with all possible actions may be set in advance in the neural network unit 40 .

A predetermined score is set for each of the learning cells 46 as described above. When a learning cell 46 is added, an arbitrary value is set as the initial value of the score for that learning cell 46 . For example, when the score is set in a numerical range of -100 to +100, 0, for example, can be set as the initial value of the score.

Next, based on the information received from the environment 200 and the user's own situation, the situation information generating unit 20 generates situation information data mapping information related to actions (step S102). The situation information data is not particularly limited, but can be generated, for example, by expressing information based on the environment or one's own situation as bitmap image data. The situation information data may be generated before step S101 or in parallel with step S101.

FIG. 8 is a diagram showing an example of situation information data in which the cards on the table, the number of times, the cards in hand, and the past information among the information indicating the environment 200 and one's own situation are represented as bitmap images. In the figure, the "number" shown on the horizontal axis of the images shown as "play cards", "hand cards", and "past information" represents the strength of the cards. That is, a smaller "number" indicates a weaker card, and a larger "number" indicates a stronger card. In the figure, "pairs" shown on the vertical axis of the images shown as "play cards", "hand cards", and "past information" represent the number of sets of cards. For example, in a hand consisting of one type of number, the value of "pair" increases in the order of 1, 2 (pair), 3 (three of a kind), and 4 (four of a kind). In the figure, "number of times" is a two-dimensional expression along the horizontal axis of the current turn from the start to the end of one game. In the illustrated plot, the boundaries of each point are blurred in order to improve the generalization performance, but the boundaries of each point do not necessarily need to be blurred.

Regarding the mapping of situation information, for the purpose of shortening the processing time, reducing the amount of learning cells, and improving the accuracy of action selection, layering is performed step by step while extracting a part of the information, information conversion, Processing such as combination of information may be performed.

FIG. 9 shows the "hand" portion extracted from the situation information data shown in FIG. For this situation information data, one pixel can be associated with one element value, as shown in the enlarged view on the right side, for example. An element value corresponding to a white pixel can be defined as 0, and an element value corresponding to a black pixel can be defined as 1. For example, in the example of FIG. 9, the element value Ip corresponding to the _p -th pixel is 1, and the element value Iq corresponding to the _q -th pixel is 0. Element values corresponding to one piece of situation information data are element values I ₁ to I _M .

Next, the element values I ₁ to I _M of the situation information data generated by the situation information generation section 20 are input to the neural network section 40 (step S103). The element values I ₁ to I _M input to the neural network unit 40 are input to each of the learning cells 46 linked to the action candidates extracted by the action candidate acquisition unit 10 via the cells 42 ₁ to 42 _M. be done. Each of the learning cells 46 to which the element values I ₁ to I _M are input outputs an output value O based on equation (1). Thus, the output value O from the learning cell 46 for the element values I ₁ to I _M is obtained (step S104).

When the learning cell 46 is in a state in which no weighting coefficient ω is set for each input node, that is, in an initial state in which no learning has been performed, the input element values I ₁ to I _M are It is set as the initial value of the weighting coefficient ω of the input node of the learning cell 46 . For example, in the example of FIG. 9, the weighting coefficient ω _pj of the input node corresponding to the p-th pixel of the learning cell 46 _j is 1, and the weighting coefficient ω _qj of the input node corresponding to the q-th pixel of the learning cell 46 _j . becomes 0. The output value O in this case is calculated using the weighting coefficient ω set as the initial value.

Next, the determining unit 50 acquires the correlation value (here, the likelihood P regarding the output value of the learning cell) between the element values I ₁ to I _M and the output value O from the learning cell 46 (step S105). A method for calculating the likelihood P is not particularly limited. For example, the likelihood P _j of the learning cell 46 _j can be calculated based on Equation (2) below.

Equation (2) indicates that the likelihood P _j is expressed by the ratio of the output value O _j of the learning cell 46 _j to the cumulative value of the weighting factors ω _ij of the multiple input nodes of the learning cell 46 _j . . Alternatively, the likelihood P _j is expressed as the ratio of the output value of the learning cell 46 _j when a plurality of element values are input to the maximum value of the output of the learning cell 46 _j based on the weighting coefficients ω _ij of the plurality of input nodes. It indicates that

Next, the determination unit 50 compares the obtained value of likelihood P with a predetermined threshold value, and determines whether the value of likelihood P is equal to or greater than the threshold value (step S106).

In each action candidate, if there is one or more learning cells 46 whose likelihood P value is equal to or greater than the threshold among the learning cells 46 linked to the action candidate ("Yes" in step S106) , the process proceeds to step S107. In step S107, the weighting factor ω of the input node of the learning cell 46 having the largest likelihood P value among the learning cells 46 linked to the action candidate is updated. The weighting factors ω _ij of the input nodes of learning cell 46 _j can be modified, for example, based on Equation (3) below.
ω _ij =(number of appearances of black in i-th pixel)/(number of times of learning) (3)

Equation (3) indicates that the weighting factor ω of each of the multiple input nodes of the learning cell 46 is determined by the cumulative average value of the element values I input from the corresponding input nodes. In this way, by accumulating the information of the situation information data whose value of likelihood P is equal to or greater than the predetermined threshold in the weighting coefficient ω of each input node, it is possible to correspond to the pixels in which black (1) appears frequently. The value of the weighting factor ω increases for an input node that The learning algorithm of such a learning cell 46 approximates the Hebbian law known as the learning principle of the human brain.

On the other hand, in each action candidate, if there is not even one learning cell 46 whose likelihood P value is equal to or greater than the threshold among the learning cells 46 linked to the action candidate (“No” in step S106), moves to step S108. In step S108, a new learning cell 46 linked to the action candidate is generated. For each input node of the newly generated learning cell 46, the element values I ₁ to I _M are set as the initial values of the weighting coefficients ω in the same way as when the learning cell 46 was in the initial state. An arbitrary value is set as the initial value of the score in the learning cell 46 to be added. In this way, by adding the learning cells 46 linked to the same action candidate, it becomes possible to learn various forms of situation information data belonging to the same action candidate, and to select a more appropriate action. becomes possible.

It should be noted that the addition of the learning cell 46 need not always be performed when there is not even one learning cell 46 whose likelihood P value is equal to or greater than the threshold in any of the action candidates. For example, a learning cell 46 may be added only when there is not even one learning cell 46 with a value of likelihood P equal to or greater than a threshold among all action candidates. In this case, the learning cell 46 to be added can be associated with one of the action candidates randomly selected from among the plurality of action candidates.

As the threshold value used to determine the likelihood P increases, the adaptability to the situation information data increases, but the number of learning cells 46 also increases, requiring more time for learning. Conversely, the smaller the threshold value, the lower the adaptability to the situation information data, but the smaller the number of learning cells 46 and the shorter the time required for learning. It is desirable that the set value of the threshold is appropriately set according to the type and form of the situation information data so that the desired relevance rate and learning time can be obtained.

Next, for each action candidate, the learning cell 46 with the highest correlation (likelihood P) to the situation information data is extracted from the learning cells 46 linked to the action candidate (step S109).

Next, the learning cell 46 with the highest score is extracted from the learning cells 46 extracted in step S109 (step S110).

Next, the action selection unit 70 selects the action candidate linked to the learning cell 46 with the highest score, and executes it on the environment 200 (step S111). As a result, an action that is expected to have the highest evaluation as a result of the action can be executed with respect to the environment 200 .

Next, the score adjustment unit 80 adjusts the score of the learning cell 46 extracted as the learning cell 46 with the highest score based on the evaluation of the result of executing the action selected by the action selection unit 70 on the environment 200. (Step S112). For example, if the evaluation of the action result is high, the score is raised, and if the evaluation of the action result is low, the score is lowered in step S112. By adjusting the scores of the learning cells 46 in this way, the neural network unit 40 advances learning so that the scores of the learning cells 46 that are expected to be highly evaluated as a result of execution on the environment 200 will have higher scores. can be done.

In the case of "millionaire", it is difficult to evaluate the result based on one action in one game, so the score of the learning cell 46 can be adjusted based on the ranking at the end of one game. . For example, if the game moves up in first place, the score of the learning cell 46 extracted as the learning cell 46 with the highest score in each turn during the game is increased by 10. In the case of second place, the score of the learning cell 46 extracted as the learning cell 46 with the highest score in each turn during the game is increased by 5. There will be no score adjustment for third-place finishers. If the player moves up to fourth place, the score of the learning cell 46 extracted as the learning cell 46 with the highest score in each turn during the game is reduced by 5. If the player moves up to fifth place, the score of the learning cell 46 extracted as the learning cell 46 with the highest score in each turn during the game is reduced by 10.

With this configuration, the neural network section 40 can learn based on the situation information data. Further, by inputting the situation information data to the neural network unit 40 that has advanced in learning, it is possible to select an action expected to be highly evaluated as a result of execution on the environment 200 from among a plurality of action candidates.

The learning method of the neural network unit 40 in the situation learning unit 110 does not apply the error back propagation method (back propagation) used in deep learning and the like, and learning in one pass is possible. Therefore, the learning process of the neural network section 40 can be simplified. Moreover, since each learning cell 46 is independent, addition, deletion, and update of data are easy. Moreover, any information can be mapped and processed, and the versatility is high. In addition, the situation learning unit 110 can perform so-called dynamic learning, and can easily perform additional learning processing using the situation information data.

Next, a learning method in the usage learning unit 120 of the behavior learning device 100 according to this embodiment will be described with reference to FIGS. 10 to 13. FIG. FIG. 10 is a flow chart showing a learning method in the usage learning unit of the action learning device according to this embodiment. 11 is a diagram illustrating an example of situation information data generated from situation information by the situation information generation unit; FIG. 12A and 12B are diagrams showing a display example of situation information and information on actions selected by the action selection unit, and an example of user episodes. FIG. 13 is a diagram showing an example of a method of generating slots indicating points of interest in situation information data.

For the learning of the usage learning section 120, the situation learning section 110 after learning according to the above procedure is used.

First, the selected action acquisition unit 130 acquires information about the action selected by the action selection unit 70 based on the situation information from the situation learning unit 110 (step S201). The information acquired from the situation learning section 110 includes situation information data generated by the situation information generating section 20 and actions selected by the action selecting section 70 .

FIG. 11 is a diagram showing an example of situation information data generated from situation information by the situation information generation unit 20. As shown in FIG. FIG. 11 shows an example of situation information data in which "cards in play", "cards in hand", "number of turns", and "cards put out last time" among the situation information in the example of "millionaire" are expressed as bitmap images. showing. In the figure, the vertical axis represents the suit, and the horizontal axis represents the strength of the cards in the "play cards", "hand cards" and "previous cards". The "number of turns" is a two-dimensional representation along the horizontal axis of the current turn from the start to the end of one game.

Although FIG. 11 shows various kinds of information as simple bitmap images, as in FIG. 8, the boundaries between points may be blurred to improve the generalization performance. Also, here, the situation information data is represented by a bitmap image so that it can be easily visualized, but the form of the situation information data is not limited to the bitmap image. For example, the status information data can also be expressed as a numeric string in which element values are arranged.

Here, as an example, the selected action acquisition unit 130 obtains the situation information data shown in FIG. is received from the situation learning unit 110 .

Next, the usage learning unit 120 presents information (situation information) based on the current environment and one's own situation and the action selected by the action selection unit 70 to the user (advisor) via a display device or the like. The evaluation acquisition unit 140 acquires the user's evaluation of the action selected by the action selection unit 70 for the current situation via an input device or the like (step S202).

For example, as shown in FIG. 12, the usage learning unit 120 displays information 144 on the action selected by the situation information and the action selection unit 70 on the display device 142 . The user reviews this information and enters user episodes 146 that strategically comment on the evaluation of the actions selected by action selector 70 .

Here, the user episode describes the judgment as to whether or not to perform the action selected by the action selection unit 70, together with the reason. For example, in the case of the example of "millionaire", it is possible to set a user episode composed of three words: "target", "reason", and "issue/not issue". Here, the "target" includes cards in hand, cards in the field, number of turns (for example, early stage, middle stage, final stage), previously played cards, and the like. The "reasons" include strong, weak, and the like. For example, ``Since the ``play card'' is ``weak'', the hand selected by the situational learning is ``play'', or ``the ``hand'' is ``weak'', so the hand selected by the situational learning is ``do not play''. User episodes can be assumed. To facilitate understanding, a simple user episode composed of three words is assumed here, but a more complicated user episode may be set according to the dimension of the situation information. .

Next, the slot generation unit of the behavior evaluation unit 150 generates a slot indicating a point of interest in the bitmap image corresponding to the situation information data from the user episode input by the user (step S203). For example, in the case of the example of "millionaire", the bitmap image indicating the "target" (Fig. 13(a)) and the bitmap image indicating the "reason" (Fig. 13(b)) are regarded as a two-dimensional matrix, Corresponding element values of these matrices are multiplied (taken the product for each element). As a result, a slot (FIG. 13(c)) indicating a point of interest in a user episode can be generated. In other words, the behavior evaluation unit 150 grammatically interprets the user episode input by the user and generates a map representing the meaning.

Note that the positions on the bitmap image that give the element value "1" according to the "target" and "reason" of the user episode may be stored in advance as common sense data.

Next, the user learning model generation unit of the behavior evaluation unit 150 generates a user learning model by linking the behavior selected by the behavior selection unit 70 with the situation information data, the slot, and the evaluation of “do/not do” in the user episode. Generate. Then, the generated user learning model is stored in the storage unit 170 (step S204).

By repeatedly performing the processing from step S201 to step S204 on the usage learning unit 120, the storage unit 170 accumulates a user learning model indicating the user's evaluation of the action selected by the situation learning unit 110. FIG. In other words, the user's strategy can be learned according to the situation information by linking the situation information and the user's comments (words). The operation performed by the usage learning unit 120 is, so to speak, collecting situation information and corresponding user comments to generate know-how.

Next, a behavior determination method using the behavior learning device 100 according to this embodiment will be described with reference to FIGS. 14 and 15. FIG. FIG. 14 is a flow chart showing an action determination method in the action learning device according to this embodiment. FIG. 15 is a diagram showing an example of a method of evaluating suitability of a user learning model for situation information.

First, the selected action acquisition unit 130 acquires information about the action selected by the action selection unit 70 based on the situation information from the situation learning unit 110 (step S301). The information acquired from the situation learning section 110 includes situation information data generated by the situation information generating section 20 and actions selected by the action selecting section 70 .

Next, the user learning model extraction unit of the behavior evaluation unit 150 searches for user learning models linked to the behavior selected by the behavior selection unit 70 from among the user learning models stored in the storage unit 170 ( step S302).

When at least one user learning model linked to the action selected by the action selecting unit 70 exists among the user learning models stored in the storage unit 170 as a result of the search (“Yes” in step S303) to step S304. On the other hand, if there is no user learning model linked to the action selected by the action selecting unit 70 among the user learning models stored in the storage unit 170 (“No” in step S303), , to step S307.

Next, the user learning model extraction unit of the behavior evaluation unit 150 selects from among the user learning models linked to the behavior selected by the behavior selection unit 70 the suitability of the situation information data to the current situation information data. Extract the highest user learning model (step S304).

For example, the current situation information data and the situation information data of the user learning model linked to the selected action are regarded as two-dimensional vectors, and the inner product value of these vectors is calculated. Then, among the combinations of the current situation information data and the situation information data of the user learning model linked to the selected action, the user learning model including the situation information data in the combination with the largest inner product value is selected as the most suitable. Extract as a high user learning model.

Alternatively, an algorithm similar to that of the score acquisition section 30 of the situation learning section 110 may be used to extract the user learning model with the highest suitability based on likelihood and score.

Next, the action evaluation unit 150 determines whether the extracted slot of the user learning model matches the current situation information data (step S305). Specifically, it is checked whether there is matching information between the slot of the user learning model extracted in step S304 and the current situation information data. For example, it is assumed that the slot of the user learning model extracted in step S304 is represented by the bitmap image shown in FIG. 15(a), and the current situation information data is represented by the bitmap image shown in FIG. 15(b). . Considering these bitmap images as a two-dimensional matrix and taking the product of each element value, the bitmap image shown in FIG. 15(c) is obtained. In this case, since there is information that matches the slot and the status information data, it is determined that the extracted slot corresponds to the current status information data.

Alternatively, an algorithm similar to that of the score acquisition section 30 of the situation learning section 110 may be used to determine whether or not the extracted slot corresponds to the current situation information data based on the likelihood and score.

As a result of the determination, if the extracted slot corresponds to the current situation information data ("Yes" in step S305), the process proceeds to step S306. On the other hand, if the extracted slot of the user learning model does not correspond to the current situation information data as a result of determination in step 305 ("No" in step S305), the process proceeds to step S307.

In step S<b>306 , the behavior evaluation unit 150 determines whether or not the user episode of the extracted user learning model determines whether or not the behavior selected by the behavior selection unit 70 is “perform”.

If the determination of the user episode is that the action selected by the action selection unit 70 is to be "executed" ("Yes" in step S306), the process proceeds to step S307. On the other hand, if the determination of the user episode indicates that the action selected by the action selection unit 70 is "not performed" ("No" in step S306), the Next, the action with the highest likelihood is selected. Then, the processing of steps S301 to S307 described above is repeated.

In step S<b>307 , the action determination unit 160 executes the action selected by the action selection unit 70 . The action selected by the action selection unit 70 is executed when the action matches the user learning model, when there is no user learning model linked to the action, and when there is no user learning model linked to the action. This is when the slot of the model does not correspond to the status information data. The action determination unit 160 is configured not to execute the action selected by the action selection unit 70 when there is a user learning model that contradicts the action selected by the action selection unit 70 according to the situation information. ing.

As described above, in the behavior learning device according to the present embodiment, it is possible to realize learning and selection of behavior according to situation information with a simpler algorithm. In addition, users' comments on actions selected according to situation information can be accumulated and used as know-how, and more appropriate actions can be selected.

Next, a hardware configuration example of the action learning device 100 according to this embodiment will be described with reference to FIG. 16 . FIG. 16 is a schematic diagram showing a hardware configuration example of the action learning device according to this embodiment.

The action learning device 100 can be realized by a hardware configuration similar to that of a general information processing device, as shown in FIG. 16, for example. For example, the action learning device 100 can include a CPU (Central Processing Unit) 300 , a main memory section 302 , a communication section 304 and an input/output interface section 306 .

The CPU 300 is a control/arithmetic device that manages overall control and arithmetic processing of the action learning device 100 . The main storage unit 302 is a storage unit used as a data work area and a data temporary save area, and can be configured by a memory such as a RAM (Random Access Memory). A communication unit 304 is an interface for transmitting and receiving data via a network. The input/output interface unit 306 is an interface for transmitting and receiving data by connecting to an external output device 310, input device 312, storage device 314, and the like. The CPU 300 , main storage unit 302 , communication unit 304 and input/output interface unit 306 are interconnected by a system bus 308 . The storage device 314 can be configured by, for example, a hard disk device configured from a non-volatile memory such as a ROM (Read Only Memory), a magnetic disk, or a semiconductor memory.

The main memory unit 302 can be used as a work area for constructing the neural network unit 40 including a plurality of learning cells 46 and executing calculations. The CPU 300 functions as a control section that controls arithmetic processing in the neural network section 40 constructed in the main memory section 302 . Storage device 314 may store learning cell information, including information about learned cells 46 that have been trained. Further, by reading the learning cell information stored in the storage device 314 and constructing the neural network section 40 in the main memory section 302, it is possible to construct a learning environment for various situation information data. Also, the storage unit 170 may be configured by the storage device 314 . The CPU 300 is desirably configured to execute arithmetic processing in parallel in a plurality of learning cells 46 of the neural network section 40 constructed in the main memory section 302 .

The communication unit 304 is a communication interface based on standards such as Ethernet (registered trademark) and Wi-Fi (registered trademark), and is a module for communicating with other devices. The learning cell information may be received from another device via communication section 304 . For example, frequently used learning cell information may be stored in the storage device 314, and infrequently used learning cell information may be read from another device.

The output device 310 includes, for example, a display such as a liquid crystal display. The output device 310 can be used as a display device for presenting situation information and information about actions selected by the action selection unit to the user when the usage learning unit 120 learns. In addition, notification of learning results and action determination to the user can be performed via the output device 310 . The input device 312 is a keyboard, a mouse, a touch panel, or the like, and is used by the user to input predetermined information to the behavior learning device 100, such as a user episode during learning by the usage learning unit 120. FIG.

The status information data can also be configured to be read from another device via the communication unit 304 . Alternatively, the input device 312 can be used as a means for entering context information data.

The functions of each part of the action learning device 100 according to the present embodiment can be implemented in hardware by implementing circuit components, which are hardware components such as LSI (Large Scale Integration) incorporating programs. Alternatively, a program that provides the function can be stored in the storage device 314, loaded into the main storage unit 302, and executed by the CPU 300, thereby realizing software.

As described above, according to the present embodiment, learning and selection of actions according to situation information can be realized with a simpler algorithm. In addition, users' comments on actions selected according to situation information can be accumulated and used as know-how, and more appropriate actions can be selected.

[Second embodiment]
A behavior learning device and behavior learning method according to the second embodiment of the present invention will be described with reference to FIG. Components similar to those of the behavior learning device according to the first embodiment are denoted by the same reference numerals, and description thereof is omitted or simplified.

The basic configuration of the action learning device according to this embodiment is the same as that of the action learning device according to the first embodiment shown in FIG. The behavior learning device according to the present embodiment differs from the behavior learning device according to the first embodiment in that the score acquisition unit 30 is configured by a database. Hereinafter, the behavior learning device according to the present embodiment will be described with reference to FIG. 1, focusing on the differences from the behavior learning device according to the first embodiment.

The situation information generation unit 20 has a function of generating situation information data which is a key for searching the database based on the information received from the environment 200 and its own situation. The situation information data does not need to be mapped as in the first embodiment, and the information received from the environment 200 and the own situation can be applied as they are. For example, in the example of "millionaire", the cards on the table, the number of times, cards in hand, past information, etc. can be used as keys for executing a search.

The score acquisition unit 30 has a database that gives a score for a specific action using situation information data as a key. The database of the score acquisition unit 30 holds scores for all possible actions for all combinations of situation information data. By searching the database of the score acquisition unit 30 using the situation information data generated by the situation information generation unit 20 as a key, the score for each of the action candidates extracted by the action candidate acquisition unit 10 can be acquired.

The score adjustment unit 80 has a function of adjusting the score value registered in the database of the score acquisition unit 30 according to the result given to the environment 200 by the action selected by the action selection unit 70 . By configuring in this way, the database of the score acquisition unit 30 can be learned based on the results of actions.

Next, an action learning method using the action learning device according to this embodiment will be described with reference to FIG.

First, the action candidate acquisition unit 10 extracts possible actions (action candidates) based on the information received from the environment 200 and the situation (step S401). The method of extracting action candidates is not particularly limited, but can be performed, for example, based on rules registered in a rule base.

Next, the situation information generating unit 20 generates situation information data representing information related to behavior based on the information received from the environment 200 and the situation of the user itself (step S402). The situation information data may be generated before step S401 or in parallel with step S401.

Next, the situation information data generated by the situation information generation unit 20 is input to the score acquisition unit 30 (step S403). The score acquisition unit 30 searches the database using the input situation information data as a key, and acquires a score for each action candidate extracted by the action candidate acquisition unit 10 (step S404).

Next, in the action selection unit 70, the action candidate with the highest score acquired by the score acquisition unit 30 is extracted from among the action candidates extracted by the action candidate acquisition unit 10 (step S405), and executed on the environment 200. (Step S406). As a result, an action that is expected to have the highest evaluation as a result of the action can be executed with respect to the environment 200 .

Next, the score adjustment unit 80 adjusts the score value registered in the database of the score acquisition unit 30 based on the evaluation of the result of executing the action selected by the action selection unit 70 with respect to the environment 200 (step S407). For example, if the evaluation of the action result is high, the score is raised, and if the evaluation of the action result is low, the score is lowered. By adjusting the score of the database in this way, the database of the score acquisition unit 30 can be learned based on the result of the action.

As described above, according to the present embodiment, even when the score acquisition unit 30 is configured with a database, as in the case of the first embodiment, it is possible to more easily learn and select an action according to the environment and one's own situation. It can be realized by an algorithm. In addition, users' comments on actions selected according to situation information can be accumulated and used as know-how, and more appropriate actions can be selected.

[Third embodiment]
A behavior learning device according to a third embodiment of the present invention will be described with reference to FIG. Components similar to those of the behavior learning device according to the first or second embodiment are denoted by the same reference numerals, and description thereof is omitted or simplified. FIG. 18 is a schematic diagram showing a configuration example of the action learning device according to this embodiment.

The action learning device 100 according to this embodiment has an action selection unit 70, an evaluation acquisition unit 140, a slot generation unit 152, and a user learning model generation unit 154, as shown in FIG.

The action selection unit 70 has a function of selecting a candidate action to be executed with respect to the environment based on the situation information data representing the environment and one's own situation. The evaluation acquisition unit 140 is a function of acquiring a user's evaluation of the action candidate selected by the action selection unit, which is an evaluation indicating whether or not to execute the action candidate in the situation indicated by the situation information data, together with the reason. Prepare. The slot generation unit 152 has a function of generating a slot indicating a point of interest in the situation information data based on the reason for the evaluation. The user learning model generation unit 154 has a function of generating a user learning model in which action candidates are associated with situation information data, slots, and judgments in evaluations.

As described above, according to the present embodiment, it is possible to accumulate user's comments on actions selected according to situation information and use them as know-how, thereby realizing learning of more appropriate actions.

[Fourth embodiment]
A behavior determination device according to a fourth embodiment of the present invention will be described with reference to FIG. Components similar to those of the behavior learning device according to the first or second embodiment are denoted by the same reference numerals, and description thereof is omitted or simplified. FIG. 19 is a schematic diagram showing a configuration example of the behavior determination device according to this embodiment.

A behavior determination device 500 according to this embodiment includes a behavior selection unit 70, a user learning model extraction unit 156, a behavior determination unit 160, and a storage unit 170, as shown in FIG.

The action selection unit 70 has a function of selecting action candidates to be executed in the environment based on the current situation information data representing the current environment and one's own situation. The storage unit 170 stores, for each of a plurality of action candidates, situation information data representing the environment and one's own situation, a slot indicating a point of interest in the situation information data, and action candidates in the situation indicated by the situation information data and the slot. Holds a user learning model that is associated with a decision to execute or not to execute. The user learning model extraction unit 156 retrieves from the storage unit 170 the user learning model linked to the action candidate selected by the action selection unit that has the situation information data with the highest suitability for the current situation information data. It has a function to extract learning models. The action determination unit 160 determines whether or not to execute the action candidate selected by the action selection unit based on the relationship between the current situation information data and the slot of the user learning model extracted by the user learning model extraction unit 156. It has a function to

As described above, according to the present embodiment, it is possible to accumulate user's comments on actions selected according to situation information and use them as know-how, thereby realizing selection of more appropriate actions.

[Modified embodiment]
The present invention is not limited to the above embodiment, and various modifications are possible.
For example, an example in which a part of the configuration of one of the embodiments is added to another embodiment, or an example in which a part of the configuration of another embodiment is replaced is also an embodiment of the present invention.

Further, in the above-described embodiment, as an application example of the present invention, the action of the player in the card game "Millionaire" has been described as an example. It can be widely applied to learning and selection.

Further, there are various processing methods in which a program for operating the configuration of the embodiment is recorded on a recording medium so as to realize the functions of the above embodiment, the program recorded on the recording medium is read as code, and executed by a computer. It is included in the scope of the embodiment. That is, a computer-readable recording medium is also included in the scope of each embodiment. In addition to the recording medium on which the above program is recorded, the program itself is also included in each embodiment.

For example, a floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, CD-ROM, magnetic tape, nonvolatile memory card, and ROM can be used as the recording medium. Further, not only the program recorded on the recording medium alone executes the process, but also the one that operates on the OS and executes the process in cooperation with other software and functions of the expansion board. included in the category of

All of the above-described embodiments merely show specific examples for carrying out the present invention, and the technical scope of the present invention should not be construed to be limited by these. That is, the present invention can be embodied in various forms without departing from its technical concept or main features.

Some or all of the above embodiments may also be described in the following additional remarks, but are not limited to the following.

(Appendix 1)
an action selection unit that selects an action candidate to be executed with respect to the environment based on situation information data representing the environment and one's own situation;
An evaluation acquisition unit that acquires a user's evaluation of the action candidate selected by the action selection unit, the evaluation indicating a judgment to execute or not to execute the action candidate in the situation indicated by the situation information data, together with a reason. and,
a slot generation unit that generates a slot indicating a point of interest in the situation information data based on the reason in the evaluation;
and a user learning model generation unit that generates a user learning model in which the situation information data, the slot, and the judgment in the evaluation are linked to the action candidate.

(Appendix 2)
an action candidate acquisition unit that extracts a plurality of action candidates that can be taken with respect to the environment based on the situation information data;
a score acquisition unit that acquires a score, which is an index representing an expected effect of the action result, for each of the plurality of action candidates;
The action learning device according to Supplementary note 1, wherein the action selection unit selects, from among the plurality of action candidates, an action candidate with the highest score as the action candidate.

(Appendix 3)
Supplementary Note 2, further comprising: a score adjustment unit that adjusts the score value associated with the selected action candidate based on a result of executing the selected action candidate on the environment. behavioral learning device.

(Appendix 4)
The score acquisition unit has a plurality of input nodes that weight each of the plurality of element values based on the situation information data with a predetermined weight, and an output node that adds and outputs the plurality of weighted element values. has a neural network unit having a plurality of learning cells including
Each of the plurality of learning cells has a predetermined score and is associated with one of the plurality of action candidates,
The score acquisition unit selects the learning cell having the largest correlation value between the plurality of element values and the output value of the learning cell among the learning cells linked to each of the plurality of action candidates. setting a score to the score of the corresponding action candidate;
The action selection unit selects the action candidate with the highest score from among the plurality of action candidates and executes the action candidate on the environment;
Action learning according to Supplementary note 3, wherein the score adjustment unit adjusts the score of the learning cell linked to the selected action candidate based on a result of executing the selected action candidate. Device.

(Appendix 5)
The score acquisition unit further has a learning unit that performs learning of the neural network unit,
The learning unit updates the weighting coefficients of the plurality of input nodes of the learning cell or adds a new learning cell to the neural network unit according to the output value of the learning cell. The action learning device according to appendix 4.

(Appendix 6)
The action according to Supplementary Note 5, wherein the learning unit adds the new learning cell when a correlation value between the plurality of element values and the output value of the learning cell is less than a predetermined threshold. learning device.

(Appendix 7)
The learning unit updates the weighting coefficients of the plurality of input nodes of the learning cell when a correlation value between the plurality of element values and the output value of the learning cell is equal to or greater than a predetermined threshold. The action learning device according to Supplementary Note 5, characterized by:

(Appendix 8)
8. The behavior learning device according to any one of appendices 4 to 7, wherein the correlation value is a likelihood of the output value of the learning cell.

(Appendix 9)
The likelihood is the ratio of the output value of the learning cell when the plurality of element values are input to the maximum value of the output of the learning cell according to the weighting factors set for each of the plurality of input nodes. The action learning device according to appendix 8, characterized by:

(Appendix 10)
10. The action learning according to any one of Appendices 4 to 9, further comprising a situation information generating unit that generates the situation information data that maps information related to actions based on the environment and the self's situation. Device.

(Appendix 11)
The action learning device according to appendix 2 or 3, wherein the score acquisition unit has a database that provides the score for each of the plurality of action candidates using the situation information data as a key.

(Appendix 12)
For each of a plurality of action candidates, situation information data representing the environment and one's own situation, a slot indicating a point of interest in the situation information data, and executing the action candidate in the situation indicated by the situation information data and the slot A storage unit that holds a user learning model that is associated with a judgment of whether to do or not to execute,
an action selection unit that selects a candidate action to be executed with respect to the environment based on current situation information data representing the current environment and one's own situation;
The user learning model having, from the storage unit, the situation information data having the highest compatibility with the current situation information data among the user learning models linked to the action candidate selected by the action selection unit. a user learning model extraction unit that extracts
an action determination unit that determines whether or not to execute the action candidate selected by the action selection unit based on the relationship between the current situation information data and the extracted slot of the user learning model; A behavior decision device characterized by:

(Appendix 13)
an action candidate acquisition unit that extracts a plurality of action candidates that can be taken with respect to the environment based on the situation information data;
a score acquisition unit that acquires a score, which is an index representing an expected effect of the action result, for each of the plurality of action candidates;
13. The action determination device according to Supplementary note 12, wherein the action selection unit selects, from among the plurality of action candidates, an action candidate with the highest score as the action candidate.

(Appendix 14)
If the slot of the extracted user learning model matches the current situation information data and the user learning model is associated with a determination to execute the candidate action, , and the action candidate selected by the action selection unit is determined to be executed.

(Appendix 15)
The action determination unit determines to execute the action candidate selected by the action selection unit when the extracted slot of the user learning model does not match the current situation information data. 14. The behavior determination device according to additional remark 12 or 13.

(Appendix 16)
If the slot of the extracted user learning model matches the current situation information data and the user learning model is associated with a determination not to execute the candidate action, the action selection unit selects the action candidate with the next highest score as the action candidate.

(Appendix 17)
The score acquisition unit has a plurality of input nodes that weight each of the plurality of element values based on the situation information data with a predetermined weight, and an output node that adds and outputs the plurality of weighted element values. has a neural network unit having a plurality of learning cells including
Each of the plurality of learning cells has a predetermined score and is associated with one of the plurality of action candidates,
The score acquisition unit selects the learning cell having the largest correlation value between the plurality of element values and the output value of the learning cell among the learning cells linked to each of the plurality of action candidates. setting a score to the score of the corresponding action candidate;
14. The action determination device according to Supplementary note 13, wherein the action selection unit selects the action candidate with the highest score from among the plurality of action candidates.

(Appendix 18)
18. The action determination device according to appendix 17, wherein the correlation value is a likelihood of the output value of the learning cell.

(Appendix 19)
The likelihood is the ratio of the output value of the learning cell when the plurality of element values are input to the maximum value of the output of the learning cell according to the weighting factors set for each of the plurality of input nodes. 19. The behavior determination device according to appendix 18, characterized by:

(Appendix 20)
14. The action determination device according to Supplementary Note 13, wherein the score acquisition unit has a database that provides the score for each of the plurality of action candidates using the situation information data as a key.

(Appendix 21)
a step of selecting a candidate action to be executed with respect to the environment based on situation information data representing the environment and one's own situation;
a step of acquiring a user's evaluation of the action candidate selected in the selecting step, the evaluation indicating a judgment to execute or not to execute the action candidate in the situation indicated by the situation information data, together with a reason;
generating a slot indicating a point of interest in the context information data based on the reason in the evaluation;
and generating a user learning model in which the situation information data, the slot, and the judgment in the evaluation are linked to the action candidate.

(Appendix 22)
a step of extracting a plurality of action candidates that can be taken based on situation information data representing the environment and one's own situation;
obtaining a score, which is an index representing an expected effect on the result of the action, for each of the plurality of action candidates;
22. The action learning method according to Supplementary note 21, wherein in the selecting step, the action candidate with the highest score is selected from among the plurality of action candidates.

(Appendix 23)
23. The action according to Supplementary Note 22, further comprising: adjusting the score value associated with the selected action candidate based on a result of executing the selected action candidate on the environment. learning method.

(Appendix 24)
In the obtaining step, a plurality of input nodes that weight each of the plurality of element values based on the situation information data with a predetermined weight, and an output node that adds and outputs the plurality of weighted element values, respectively. wherein each of the plurality of learning cells has a predetermined score and is linked to one of the plurality of action candidates, wherein the plurality of The score of the learning cell having the largest correlation value between the plurality of element values and the output value of the learning cell, among the learning cells linked to each of the action candidates, is the score of the corresponding action candidate. set to score,
In the selecting step, the action candidate with the highest score is selected from among the plurality of action candidates;
Action learning according to Supplementary note 23, wherein the adjusting step adjusts the score of the learning cell linked to the selected action candidate based on a result of executing the selected action candidate. Method.

(Appendix 25)
a step of selecting an action candidate to be executed with respect to the environment based on the current situation information data representing the current environment and the self's situation;
For each of a plurality of action candidates, situation information data representing the environment and one's own situation, a slot indicating a point of interest in the situation information data, and executing the action candidate in the situation indicated by the situation information data and the slot the user learning model linked to the action candidate selected in the selecting step from among the user learning models linked to the decision to perform or not to perform, wherein the current situation extracting the user learning model having the context information data that best matches the information data;
determining whether or not to execute the action candidate selected in the selecting step based on the relationship between the current situation information data and the extracted slot of the user learning model. Action decision method.

(Appendix 26)
extracting a plurality of possible action candidates for the environment based on the situation information data;
obtaining a score, which is an index representing an expected effect on the result of the action, for each of the plurality of action candidates;
26. The behavior determination method according to Supplementary note 25, wherein in the selecting step, a behavior candidate with the highest score is selected from among the plurality of behavior candidates as the behavior candidate.

(Appendix 27)
In the determining step, if the slot of the extracted user learning model matches the current situation information data, and the user learning model is associated with a determination to execute the action candidate, 27. The action determination method according to appendix 25 or 26, further comprising: determining to execute the action candidate selected in the selecting step.

(Appendix 28)
In the determining step, if the extracted slot of the user learning model does not match the current situation information data, it is determined to execute the action candidate selected in the selecting step. The behavior determination method according to Supplementary Note 25 or 26.

(Appendix 29)
In the determining step, if the slot of the extracted user learning model matches the current situation information data and the user learning model is associated with a determination not to execute the candidate action. selects the action candidate with the next highest score as the action candidate.

(Appendix 30)
the computer,
means for selecting action candidates to be executed in the environment based on situation information data representing the environment and one's own situation;
means for acquiring a user's evaluation of the action candidate selected by the selecting means, the evaluation indicating a judgment to execute or not to execute the action candidate in the situation indicated by the situation information data, together with a reason;
means for generating a slot indicating a point of interest of the situation information data based on the reason in the evaluation;
A program that functions as means for generating a user learning model in which the situation information data, the slot, and the judgment in the evaluation are linked to the action candidate.

(Appendix 31)
the computer,
Means for selecting action candidates to be executed in the environment based on current situation information data representing the current environment and self's situation;
For each of a plurality of action candidates, situation information data representing the environment and one's own situation, a slot indicating a point of interest in the situation information data, and executing the action candidate in the situation indicated by the situation information data and the slot the user learning model linked to the action candidate selected by the selecting means from among the user learning models linked to the determination of whether to perform or not to perform, wherein the current situation means for extracting the user learning model having the contextual information data that best matches the information data;
A program that functions as means for determining whether or not to execute the action candidate selected in the selecting step based on the relationship between the current situation information data and the extracted slot of the user learning model.

(Appendix 32)
A computer-readable recording medium recording the program according to appendix 30 or 31.

(Appendix 33)
The action learning device according to any one of Appendices 1 to 11;
and an environment to be acted upon by the action learning device.

DESCRIPTION OF SYMBOLS 10... Action candidate acquisition part 20... Situation information generation part 30... Score acquisition part 40... Neural network parts 42, 44... Cell 46... Learning cell 50... Judgment part 60... Learning part 62... Weight correction part 64... Learning cell generation part 70 Action selection unit 80 Score adjustment unit 100 Action learning device 110 Situation learning unit 120 Usage learning unit 130 Selected action acquisition unit 140 Evaluation acquisition unit 150 Action evaluation unit 152 Slot generation unit 154 User learning Model generation unit 156 User learning model extraction unit 160 Action determination unit 170 Storage unit 200 Environment 300 CPU
302...main storage section 304...communication section 306...input/output interface section 308...system bus 310...output device 312...input device 314...storage device 400...action learning system

Claims (12)

an action selection unit that selects an action candidate to be executed with respect to the environment based on situation information data representing the environment and one's own situation;
An evaluation acquisition unit that acquires a user's evaluation of the action candidate selected by the action selection unit, the evaluation indicating a judgment to execute or not to execute the action candidate in the situation indicated by the situation information data, together with a reason. and,
a slot generation unit that generates a slot indicating a point of interest in the situation information data based on the reason in the evaluation;
and a user learning model generation unit that generates a user learning model in which the situation information data, the slot, and the judgment in the evaluation are linked to the action candidate. an action candidate acquisition unit that extracts a plurality of action candidates that can be taken with respect to the environment based on the situation information data;
a score acquisition unit that acquires a score, which is an index representing an expected effect of the action result, for each of the plurality of action candidates;
2. The action learning device according to claim 1, wherein the action selection unit selects, from among the plurality of action candidates, an action candidate with the highest score as the action candidate. 2. The score adjustment unit further comprises a score adjustment unit that adjusts the score value associated with the selected action candidate based on a result of executing the selected action candidate on the environment. Behavioral learning device as described. The score acquisition unit has a plurality of input nodes that weight each of the plurality of element values based on the situation information data with a predetermined weight, and an output node that adds and outputs the plurality of weighted element values. has a neural network unit having a plurality of learning cells including
Each of the plurality of learning cells has a predetermined score and is associated with one of the plurality of action candidates,
The score acquisition unit selects the learning cell having the largest correlation value between the plurality of element values and the output value of the learning cell among the learning cells linked to each of the plurality of action candidates. setting a score to the score of the corresponding action candidate;
The action selection unit selects the action candidate with the highest score from among the plurality of action candidates and executes the action candidate on the environment;
4. The action according to claim 3, wherein the score adjustment unit adjusts the score of the learning cell linked to the selected action candidate based on a result of executing the selected action candidate. learning device. For each of a plurality of action candidates, situation information data representing the environment and one's own situation, a slot indicating a point of interest in the situation information data, and executing the action candidate in the situation indicated by the situation information data and the slot A storage unit that holds a user learning model that is associated with a judgment of whether to do or not to execute,
an action selection unit that selects a candidate action to be executed with respect to the environment based on current situation information data representing the current environment and one's own situation;
The user learning model having, from the storage unit, the situation information data having the highest compatibility with the current situation information data among the user learning models linked to the action candidate selected by the action selection unit. a user learning model extraction unit that extracts
an action determination unit that determines whether or not to execute the action candidate selected by the action selection unit based on the relationship between the current situation information data and the extracted slot of the user learning model; A behavior decision device characterized by: an action candidate acquisition unit that extracts a plurality of action candidates that can be taken with respect to the environment based on the situation information data;
a score acquisition unit that acquires a score, which is an index representing an expected effect of the action result, for each of the plurality of action candidates;
6. The action determination device according to claim 5, wherein the action selection unit selects, from among the plurality of action candidates, an action candidate with the highest score as the action candidate. If the slot of the extracted user learning model matches the current situation information data and the user learning model is associated with a determination to execute the candidate action, 7. The action determination device according to claim 5, wherein the action candidate selected by the action selection unit is determined to be executed. The action determination unit determines to execute the action candidate selected by the action selection unit when the extracted slot of the user learning model does not match the current situation information data. 7. The action determining device according to claim 5 or 6, wherein If the slot of the extracted user learning model matches the current situation information data and the user learning model is associated with a determination not to execute the candidate action, the action selection unit selects the action candidate with the next highest score as the action candidate. The score acquisition unit has a plurality of input nodes that weight each of the plurality of element values based on the situation information data with a predetermined weight, and an output node that adds and outputs the plurality of weighted element values. has a neural network unit having a plurality of learning cells including
Each of the plurality of learning cells has a predetermined score and is associated with one of the plurality of action candidates,
The score acquisition unit selects the learning cell having the largest correlation value between the plurality of element values and the output value of the learning cell among the learning cells linked to each of the plurality of action candidates. setting a score to the score of the corresponding action candidate;
7. The action determination device according to claim 6, wherein the action selection unit selects the action candidate with the highest score from among the plurality of action candidates. a step of selecting a candidate action to be executed with respect to the environment based on situation information data representing the environment and one's own situation;
a step of acquiring a user's evaluation of the action candidate selected in the selecting step, the evaluation indicating a judgment to execute or not to execute the action candidate in the situation indicated by the situation information data, together with a reason;
generating a slot indicating a point of interest in the context information data based on the reason in the evaluation;
and generating a user learning model in which the situation information data, the slot, and the judgment in the evaluation are linked to the action candidate. a step of selecting an action candidate to be executed with respect to the environment based on the current situation information data representing the current environment and the self's situation;
For each of a plurality of action candidates, situation information data representing the environment and one's own situation, a slot indicating a point of interest in the situation information data, and executing the action candidate in the situation indicated by the situation information data and the slot the user learning model linked to the action candidate selected in the selecting step from among the user learning models linked to the decision to perform or not to perform, wherein the current situation extracting the user learning model having the context information data that best matches the information data;
determining whether or not to execute the action candidate selected in the selecting step based on the relationship between the current situation information data and the extracted slot of the user learning model. Action decision method.

Images