JP2018063602A - Program, system, and method for adjusting weighting of neural network using q-learning - Google Patents

Abstract

PROBLEM TO BE SOLVED: To apply Q-learning for improving weighting of neural networks (NN) having game parameters as input values and expected rewards Q corresponding to respective actions of a game character as output values.SOLUTION: A method for adjusting parameters of a neural network using Q-learning executes a set of steps repeatedly for a plurality of times, the set of the steps including: a step for extracting a game parameter relating to a game character as a first input value; a step for extracting, on the basis of the first input value, a first expected reward (Q value) associated with respective actions of the game character from a neural network (NN) as an output value; a step for extracting the game parameter after selecting/executing one out of respective actions as a second input value; a step for calculating, on the basis of the second input value, a second expected reward (Q value) corresponding to respective actions of the game character from the neural network (NN); a step for updating, on the basis of the first expected reward (Q value) and the second expected reward (Q value), a parameter of the neural network (NN) consisting of one or a plurality of layers.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a program, a system, and a method for weighting adjustment of a neural network using Q-learning, and more specifically, an expected reward Q for each action of a game character using a game environment parameter as an input value. The present invention relates to a program, a system, and a method for improving the weighting of a neural network (NN) having Q as an output value by Q learning.

  A neural network is a model that is modeled after the neurons and synapses of the brain, and is composed of two stages of learning and processing. In the learning stage, features are learned from a large number of inputs, and a neural network is constructed for later processing. In the processing stage, a process for extracting an output value based on a new input is performed using a neural network. In recent years, the technology at the learning stage has been greatly developed, and for example, deep learning has been able to construct a multilayer neural network having high expressive power. The effectiveness of this multilayer neural network has been confirmed in various fields, and the effectiveness of deep learning has been widely recognized.

  On the other hand, Q learning (Q-learning) is one of reinforcement learning, which learns appropriate actions according to the situation based on rewards given from the environment without giving correct actions to tasks. It is. In recent years, attempts have been made to improve and update the structure and parameters of neural networks by reinforcement learning. For example, Non-Patent Document 1 proposes DQN (Deep Q-Network) as a technique for improving the parameters of a multilayer neural network by Q learning by combining multilayer neural network (CNN) and Q learning.

“Human-level Control Through Deep Reinforcement Learning”, Volodymyr Minih et al., February 26, 2015

  However, if the DQN method is used to learn the parameters of the neural network for selecting the optimum action of the game character, improvement specific to the game is required. More specifically, for example, the DQN method is used as it is in a game in which the selection of the action of the user character is restricted depending on the progress (such as an action such as a deadly technique that takes time to become selectable). However, there is a problem in that learning of the user character's restricted behavior does not proceed as expected, and learning is biased, making it difficult to improve and update the parameters of the neural network. It was.

  An object of the embodiment of the present invention is to adjust the weight of a neural network by Q-learning in consideration of a situation unique to a game environment. Other objects of the embodiments of the present invention will become apparent by referring to the entire specification.

  A method according to an embodiment of the present invention, when executed on one or more computers, causes the one or more computers to have a game parameter related to one or more game characters as a first input value. A step of extracting, based on the first input value, a step of extracting a first expected reward (Q value) by each action of the game character from the neural network (NN) as an output value; Extracting the game parameter after selection / execution as a second input value, and a second potential reward for each action of the game character from the neural network (NN) based on the second input value Based on the step of calculating (Q value) and the first expected reward (Q value) and the second expected reward (Q value), 1 or The step of updating the parameter of the neural network (NN) composed of several layers is repeatedly executed a plurality of times, and the parameter of the neural network (NN) composed of one or a plurality of layers is adjusted using Q learning. Configured.

  A system according to an embodiment of the present invention, when executed on one or more computers, causes the one or more computers to have a game parameter relating to one or more game characters as a first input value. A step of extracting, based on the first input value, a step of extracting a first expected reward (Q value) by each action of the game character from the neural network (NN) as an output value; Extracting the game parameter after selection / execution as a second input value, and a second potential reward for each action of the game character from the neural network (NN) based on the second input value Based on the step of calculating (Q value), the first expected reward (Q value) and the second expected reward (Q value), 1 Is configured to perform the steps of updating the parameters of the neural network (NN) consists of multiple layers were multiple iterations performed, the adjustment of the parameters of the neural network (NN) using the Q-learning.

  When the program according to the embodiment of the present invention is executed on one or a plurality of computers, the game parameter of one or a plurality of game characters is used as a first input value for the one or a plurality of computers. A step of extracting, based on the first input value, a step of extracting a first expected reward (Q value) by each action of the game character from the neural network (NN) as an output value; Extracting the game parameter after selection / execution as a second input value, and a second potential reward for each action of the game character from the neural network (NN) based on the second input value Based on the step of calculating (Q value) and the first expected reward (Q value) and the second expected reward (Q value), 1 or And updating the parameters of the neural network (NN) consists of several layers was repeatedly performed a plurality of times, configured to adjust the parameters of the neural network (NN) using the Q-learning.

  According to various embodiments of the present invention, a game in which a battle state changes every moment according to the action of one or a plurality of game characters during one battle, in particular, the selection of the action of the user character is limited by the progress. In the game, it is possible to efficiently and surely select the learning of the restricted behavior of the user character, and to effectively improve and update the parameters of the neural network.

The lineblock diagram showing roughly the composition of system 1 concerning one embodiment of the present invention. The block diagram which shows roughly the function which the system 1 in one Embodiment has. The figure which shows an example of the flow which adjusts the parameter of the neural network (NN) in one Embodiment. The figure which shows an example of a structure of the neural network (NN) in one Embodiment. The figure which shows an example of the log | history management table classified by action in one Embodiment.

  FIG. 1 is a configuration diagram schematically showing a configuration of a system 1 according to an embodiment of the present invention. As illustrated, the system 1 according to an embodiment includes a server 10 and a plurality of terminal devices 30 connected to the server 10 via a communication network 20 such as the Internet. To provide electronic commerce services. In addition, the system 1 according to the embodiment includes a service for providing various digital contents other than games such as games using characters, electronic books, moving image contents, and music contents, text chat (mini mail), circles, and avatars. Various Internet services such as a communication platform (SNS platform) service that realizes communication functions between various users such as a diary, a message board, and greetings can be provided to the user of the terminal device 30.

  The server 10 in one embodiment is configured as a general computer, and as illustrated, a CPU (computer processor) 11, a main memory 12, a user I / F 13, a communication I / F 14, and storage (storage). Device) 15, and these components are electrically connected to each other via a bus 17. The CPU 11 loads an operating system and various other programs from the storage 15 into the main memory 12 and executes instructions included in the loaded programs. The main memory 12 is used for storing a program executed by the CPU 11, and is configured by a DRAM or the like, for example. In addition, the server 10 in one embodiment may be configured using a plurality of computers each having a hardware configuration as described above. The CPU (computer processor) 11 described above is merely an example, and it goes without saying that a GPU (graphics processing unit) may be used instead. How to select the CPU and / or GPU can be appropriately determined in consideration of desired cost or efficiency. Hereinafter, the CPU 11 will be described as an example.

  The user I / F 13 includes, for example, an information input device such as a keyboard and a mouse that accepts an operator input, and an information output device such as a liquid crystal display that outputs a calculation result of the CPU 11. The communication I / F 14 is implemented as hardware, firmware, communication software such as a TCP / IP driver or a PPP driver, or a combination thereof, and is configured to be able to communicate with the terminal device 30 via the communication network 20.

  The storage 15 is composed of, for example, a magnetic disk drive, and stores various programs such as a control program for providing various services. The storage 15 can also store various data for providing various services. Various data that can be stored in the storage 15 may be stored in a database server or the like that is physically separate from the server 10 that is communicably connected to the server 10.

  In one embodiment, the server 10 also functions as a web server that manages a website composed of a plurality of hierarchical web pages, and can provide various services to the user of the terminal device 30 via the website. . The storage 15 can also store HTML data corresponding to this web page. HTML data is associated with various image data, and various programs described in a script language such as JavaScript (registered trademark) can be embedded.

  In one embodiment, the server 10 can provide various services via an application (program) executed on the terminal device 30 in an execution environment other than the web browser. Such applications can also be stored in the storage 15. This application is created using a programming language such as Objective-C or Java (registered trademark). The application stored in the storage 15 is distributed to the terminal device 30 in response to the distribution request. Note that the terminal device 30 can also download such an application from a server other than the server 10 (a server that provides an application market) or the like.

  As described above, the server 10 can manage websites for providing various services, and distribute web pages (HTML data) constituting the websites in response to requests from the terminal device 30. In addition, as described above, the server 10 is an application that is executed in the terminal device 30 in place of or in addition to the provision of various services using such a web page (web browser). Various services can be provided based on communication. Regardless of which aspect of the service is provided, the server 10 can transmit and receive various data (including data necessary for screen display) necessary for providing various services to and from the terminal device 30. Moreover, the server 10 can store various data for each identification information (for example, user ID) for identifying each user, and can manage the provision status of various services for each user. Although detailed description is omitted, the server 10 may have a function of performing user authentication processing, billing processing, and the like.

  The terminal device 30 according to the embodiment is an arbitrary information processing device that displays a web page of a website provided by the server 10 on a web browser and implements an execution environment for executing an application. , A wearable device, a personal computer, a game-dedicated terminal, and the like, but are not limited thereto.

  The terminal device 30 is configured as a general computer, and as shown in FIG. 1, a CPU (computer processor) 31, a main memory 32, a user I / F 33, a communication I / F 34, and a storage (storage device) 35. These components are electrically connected to each other via a bus 37.

  The CPU 31 loads an operating system and various other programs from the storage 35 into the main memory 32 and executes instructions included in the loaded programs. The main memory 32 is used for storing a program executed by the CPU 31, and is configured by, for example, a DRAM or the like.

  The user I / F 33 includes, for example, an information input device such as a touch panel that accepts user input, a keyboard, a button, and a mouse, and an information display device such as a liquid crystal display that outputs a calculation result of the CPU 31. The communication I / F 34 is implemented as hardware, firmware, communication software such as a TCP / IP driver or a PPP driver, or a combination thereof, and is configured to be able to communicate with the server 10 via the communication network 20.

  The storage 35 is composed of, for example, a magnetic disk drive, a flash memory, or the like, and stores various programs such as an operating system. The storage 35 can store various applications received from the server 10.

  The terminal device 30 includes, for example, a web browser for interpreting an HTML file (HTML data) and displaying the screen, and interprets and receives the HTML data acquired from the server 10 by the function of the web browser. A web page corresponding to the HTML data thus displayed can be displayed. The web browser of the terminal device 30 can incorporate plug-in software that can execute various types of files associated with HTML data.

  When the user of the terminal device 30 uses a service provided by the server 10, for example, HTML data, an animation instructed by an application, an operation icon, or the like is displayed on the terminal device 30. The user can input various instructions using the touch panel of the terminal device 30 or the like. The instruction input from the user is transmitted to the server 10 via the function of the application execution environment such as the web browser of the terminal device 30 and NgCore (trademark).

  Next, the function which the system 1 in one embodiment comprised in this way has is demonstrated. As described above, the system 1 according to an embodiment can provide various Internet services to the user, and in particular, can provide a game distribution service. Hereinafter, the function of the system 1 according to an embodiment will be described by taking a function of providing a game distribution service as an example.

  FIG. 2 is a block diagram schematically showing functions of the system 1 (the server 10 and the terminal device 30). First, the function which the server 10 in one Embodiment has is demonstrated. As illustrated, the server 10 includes an information storage unit 41 that stores various information, and a character action control unit 42 that determines the action of the game character in one embodiment. These functions are realized by the cooperative operation of hardware such as the CPU 11 and the main memory 12 and various programs and tables stored in the storage 15, for example, instructions included in the loaded program This is realized by the CPU 11 executing. 2 may be realized by the terminal device 30 or may be realized by the cooperation of the server 10 and the terminal device 30.

  The information storage unit 41 in one embodiment is realized by the storage 15 or the like, and as shown in FIG. 2, various statuses of enemies and teammate characters (collectively referred to as game characters) in the game, actionable characters, A game parameter management table 41a for managing game parameters such as a skill that can be acted, a flag of the character that acted most recently, a skill of the character that acted most recently, and each action that can be acted and each action based on the game parameter during the game An action evaluation management table 41b for managing a prospective reward (Q value) when the game is selected, and an action history management table 41c for managing a history of actions selected by each game character in the game. Have.

  Next, the function of the character action control part 42 which determines the action of the game character in one Embodiment is demonstrated. The character action control unit 42 has various statuses of enemies and teammate characters stored in the game parameter management table 41a of the information storage unit 41, an actionable character, an actionable technique, a flag of a character that has recently acted, and an action of the most recent character. The action of each game character is selected and determined based on game parameters such as the character's skill. More specifically, the character action control unit 42 extracts game parameters from the game parameter management table 41a, inputs them as input values to one or more layers of neural networks, and outputs each action of the character that can act as an output. The expected reward (Q value) is extracted as the output value, and the action with the highest output value is normally selected and determined from these. In the learning stage, the expected reward (Q value) for each action is By selecting / determining the actions of characters that can act independently, or by selecting / determining the action with the highest expected reward (Q value) for each action of a character that can act, NN) parameters are adjusted. The character action control unit 42 refers to the action evaluation management table 41b as necessary for action selection / determination, and each action that can be acted and when each action is selected based on the game parameter during the game. Expected reward (Q value) can be obtained. In this way, the character action control unit 42 proceeds with learning by selecting and determining each action of the teammate character until one battle is completed, for example. In addition, in this example, although demonstrated as each action of the friendly character until 1 battle is complete | finished, it is not the intention which excludes being an enemy character, a friendly character, or these parts. Moreover, although it was made until 1 battle was complete | finished, it does not matter even if it is to the time of the division | segmentation which can calculate other expected reward (Q value) appropriately.

Regardless of any action selection / determination method, the character action control unit 42 manages each action that can be performed and the expected reward (Q value) when each action is selected based on the game parameter during the game. For each action a in the state St + 1 after selecting the action at and its expected reward Q (St, at) in a certain state St, with appropriate reference to the action evaluation management table 41b for The expected reward that should have been an appropriate prospective reward according to the following formula using the maximum expected reward maxQ (St + 1, a) among the expected rewards Q (St + 1, a) of Is calculated. It is needless to say that the calculation formula is an example, and is not limited thereto and can be changed as appropriate. Here, the expected reward Q (Q value) is the sum of the reward obtained when the action a is performed and the reward that will be obtained in the future, and r is the actual reward after selecting and executing the action at, α represents a learning rate, and γ represents a discount rate (reward attenuation coefficient).
As a result, when it is determined that the initial expected reward Q (St, at) should be corrected (that is, looking back from the initial expected reward Q (St, at) and after (St + 1) in the state St) If there is a divergence between the estimated reward (St, at) and the expected reward (St, at), the parameters of the neural network described later are updated so that the expected reward (St, at) becomes a more appropriate value. The basic method of Q-learning is not described in detail any more, but those skilled in the art can appropriately apply these methods to the system according to the embodiment.

  FIG. 3 shows an embodiment of the present invention as a flow. First, a game parameter (input value) is extracted (step 110), and this is input to the initial neural network (as a first input value), thereby providing a first for each action of the game character. The expected reward (Q value) is extracted as an output value (step 120), a specific action of the game character is selected and determined from among them (step 130), and the game parameters after the action are updated. The updated game parameter is extracted (step 140), the updated game parameter is input to the neural network as a second input value, and a second potential reward (Q value) for each action of the game character. Is extracted as an output value (step 150), and based on the first and second expected rewards (Q value), a more appropriate expectation Updates parameters of the neural network to be able to calculate the based compensation (step 160). By repeating these steps, for example, until one game battle is completed or by repeating a plurality of generations, the parameters of the neural network related to selection / determination of appropriate action of the game character are evolved. Hereinafter, a method for learning a neural network by Q learning for determining a more optimal action of a game character in the invention according to an embodiment will be described more specifically.

  First, the neural network to be subjected to reinforcement learning may be an arbitrary network having a single-layer or multiple-layer neural network (including CNN) structure. Here, the update of the parameters of the neural network is mainly intended to change the weight between the nodes of the neural network among the structure of the neural network and the weight between the nodes. FIG. 4 shows an example of the structure of this neural network and the weighting between the nodes. As shown in the figure, a neural network (NN) having one hidden layer is connected to three output layers (nodes) through a hidden layer having three nodes for five input layers (nodes). Has been. As shown in the figure, the nodes to be connected are connected with specific weights (W11, W12, W21, W22, etc., in this example, a total of 13 weights). The structure of the neural network is usually composed of one or two hidden layers. The number of nodes in the hidden layer is the same as the number of nodes in the output layer (about 30 to 70). Alternatively, CNN, LSTM, or other neural networks may be used.

  As described above, by using such a neural network, a game parameter at each time point is input as an input value, so that an expected reward (Q value) when selecting each action (for example, attack, magic, special skill, etc.) ) As an output. Especially in the initial stage of learning, when this expected reward (Q value) is not calculated correctly, by applying error propagation of this neural network, etc., when the same game parameter is used as an input value, Neural network parameters (weighting between nodes) are changed so that an appropriate expected reward (Q value) can be calculated. By continuing this over a plurality of generations, the neural network will evolve and improve, and it will lead to the selection and determination of more appropriate actions by game characters in the game.

  However, depending on the game, there are actions that are restricted in selection (actions that are not always selectable) and actions that are difficult to select. In such a complicated game environment, reinforcement learning based on Q-learning is performed. Even if it is adopted as it is, it is known that there is a problem that learning of the neural network does not proceed as expected. As a result of examining the coping method for the problem, it was found that the improvement of the learning method is effective by the following method.

  Hereinafter, although a role playing game will be described as an example as an embodiment, it may be an arbitrary game and is not intended to be limited to a specific game. The role playing game is a command battle game in which a plurality of teammate characters and enemy characters battle each other. Allied characters and enemy characters have action gauges, and the action gauges increase with the passage of time. Then, a character whose action gauge is full is in an actionable state. The game progresses when the player selects a command for the character in the actionable state. In the trial of the game, the state of the game at a certain time is parameterized and input to an individual as an input value, and a command is selected according to the output value from the neural network, and the game is advanced. As described above, various statuses of enemies and teammate characters (collectively referred to as game characters) in the game, actionable game characters, actionable techniques, the flag of the most recently acted game character, and the game that recently acted Character skills are defined as game parameters.

  Next, an example of this input value will be described more specifically. As an input value to the neural network, for example, (1) the status of the game character at a certain time (the time when any game character can act), (2) a command selectable flag for the game character, and (3) A normalization value indicating whether or not the game character has most recently made a command selection is input, but it is not intended to be limited to these. (1) The game character status may be the attack power, magic attack power, defense power, magic defense power, speed, deadly skill gauge, etc. of each game character (including enemy characters). (2) The command selectable flag for all game characters means an available flag (0 or 1) for each command of all game characters (friend characters and enemy characters).

In the game, there are teammate character 1, teammate character 2, teammate character 3, teammate character 4, and enemy character, and five commands (attack, defense, skill 1, skill 2, and special move) are set for each character. Yes. When only the ally character 1 can act at a certain time and the ally character 1 can select attack, defense, skill 1, and skill 2,
Allied character 1_attack: 1
Ally character 1_defense: 1
Ally character 1_skill 1: 1
Ally character 1_skill 2: 1
Allied Character 1_ Special Move: 0
Ally character 2_attack: 0
Ally character 2_defense: 0
Ally character 2_skill 1: 0
Ally character 2_skill 2: 0
Allied Character 2_ Special Moves: 0
... (omitted) ...
Enemy Character_ Special Moves: 0
Will be input.

  Next, the normalized value of whether or not the game character of (3) has most recently selected a command is a value indicating how recently each command has been selected for each command. For example, when a certain command is selected, after the selection, this normalized value is set to 0, and is set to a value that becomes 0.2 at the next command selection timing, and then increases to a maximum value of 1 by 0.2. The If this normalized value is low, it is difficult to select the same command.

Next, examples of output values will be described more specifically. An expected reward (Q value) is extracted as an output value for each of the teammate character commands from the neural network. Thereby, the command of the character that can act at a certain time is determined.
For example, the expected reward (Q value) in the ally character 1 is
Allied character 1_attack: 0.8
Ally character 1_defense: 0.5
Ally character 1_skill 1: 0.4
Ally character 1_skill 2: 0.1
Allied Character 1_ Special Move: 0
If this is the case, in the normal command selection method, the “attack” with the largest expected reward (Q value) is selected.

  In the game battle, each action of the game character is determined by repeating this input and output. Usually, a plurality of actions of a plurality of game characters are combined and a game battle proceeds, and continues until the battle ends. The end of the battle is usually intended, for example, when the HP of the enemy game character is 0, or the HP of all the friend game characters is 0, but in other cases where the game battle can be evaluated. It doesn't matter.

An example of the output value will be described in more detail. Each command and prospective reward (Q value) of the teammate character are extracted and calculated as follows.
Ally character 1_defense: 0.89961839
Allied Character 1_Skilled Fist: 1.16006923
Ally character 1_shout: 0.6012032
Allied character 1_Fight: 0.53579712
Friend Character 1_ Armor Break: 0.52256131
Allied character 1_Fist butt: 0.314955
Allied character 1_wind blade: 0.60766339

Ally character 2_defense: 0.63610768
Ally character 2_ sacred legendary sword: 0.83349609
Ally character 2_protection of the genie: 0.29603601
Ally character 2_shout: -0.20091677
Friend Character 2_Inazuma Sword: 0.4563830
Allied Character 2_ Flame Sword: 0.640000938
Ally character 2_ fight: 0.408863395

Ally character 3_defense: 0.9544442247
Allied Character 3_Ocean Champion: 0.29018426
Allied character 3_ 兜 split: 0.20885491
Ally character 3_shout: -0.25541139
Ally character 3_ fight: 0.509958467
Ally character 3_ magical destruction: 0.36382771
Allied character 3_arm strength destruction: 0.77368307

Allied character 4_Defense: 0.84505749
Allied character 4_ nemesis change: 0.21710753
Friend character 4_friend combination: 0.771493635
Ally character 4_ shout: 0.188607745
Allied character 4_ Fight: 0.68954039
Allied character 4_Destroyed work: 0.8964808
Friend Character 4_Sword Ray: 0.6781354

Ally character 5_defense: 0.370557304
Ally character 5_whole recovery: -1.292669934
Allied character 5_Queen's blessing: -1.283307414
Ally character 5_shout: -1.3387702702
Ally character 5_ healing: -0.21439075
Ally character 5_ armor plus: -0.15570354
Ally character 5_ fight: -2.44153452
In the above example, since each of the 5 characters has 7 commands, a total of 35 commands and the expected reward (Q value) are extracted. Note that this expected reward (Q value) is normalized in the range of −1 to 1, but in Q learning, a future reward may be added, so it may not be within that range.

  When the expected reward (Q value) for each command is calculated and extracted in this way, the action with the highest output value is usually selected and determined from these, but in the learning stage, it is a deadly move. Because the selection and execution of actions such as tricks are restricted (because it takes time to accumulate the Special Moves Gauge, it is not possible to select the Special Moves during that time), there is a problem that the range of learning will be biased It became clear. This problem was particularly noticeable in games such as role playing. Therefore, in the learning stage, the behavior of the character that can act independently of the expected reward (Q value) by each behavior is randomly selected or determined, or the expected reward (Q value) for each behavior of the character that can act The parameters of the neural network (NN) are adjusted each time by selecting / determining the action with the highest). Randomly select / determine the action of the character that can act regardless of the expected reward (Q value) for each action, or select the action with the highest expected reward (Q value) for each action of the character that can act・ Which one to select should be determined in advance by deciding the selection probability of each selection / determination method as a set value. is there. If these set values are 50% and 50%, respectively, one of the selection / determination methods is selected with equal probability. If they are set to 100% and 0%, respectively, the expected remuneration (Q A command to be executed is determined by a method of randomly selecting and determining an action of a character that can act regardless of (value). Such selection / determination selection probability can be changed to a desired set value in accordance with the number of learnings and the like.

  In one embodiment, the selection / execution of actions such as the above-mentioned special moves is restricted (it takes time to accumulate the special technique gauge, so the special techniques cannot be selected during that time), etc. In order to prevent such a situation, the setting value of the selection probability of a method for randomly selecting / determining the behavior of a character that can act independently of the expected reward (Q value) of each behavior is set to 100%. The selection probability of the method for selecting / determining the action with the highest expected reward (Q value) can be set to 0%. By doing in this way, compared with the method of determining the command to be selected based on the expected reward (Q value) by each action, more difficult commands to be selected are selected. Learning is possible. If learning by selecting such difficult-to-select commands is sufficiently advanced, the respective probabilities can be changed. For example, for each learning, a setting value of a selection probability of a method for randomly selecting and determining an action of a character that can act independently of the expected reward (Q value) by each action is set from 100% to 0.0015%. It may be reduced gradually. In this way, as learning progresses, the selection probability setting method for the method of selecting / determining at random decreases, and the selection probability of the method for selecting / determining the action with the highest expected reward (Q value) increases. It will be. Although there is no particular limitation on the change range of the set value, for example, it can be 0.0015% to 0.015%. The set change width can be changed at any time.

  In one embodiment, when a command that is difficult to select (for example, a special move) is selected a predetermined number of times or when a predetermined condition is satisfied, the user can act regardless of the expected reward (Q value) for each action. You may comprise so that the setting value of the selection probability of the method of selecting and determining a character's action at random may be changed. For example, when a command that is difficult to select (for example, a special move) is selected twice or more, the setting value of the selection probability of the method of selecting / determining at random may be changed. In addition, the predetermined conditions can be set as appropriate.

  Here, in one embodiment, since the number of actions of the enemy character is less than the number of actions of the teammate character in calculating the reward for an action, one of the teammate characters selected and acted before and after the enemy character attacked It is known that the action by the command of the character is greatly corrected in the negative direction when only the damage received from the attack from the enemy character is reflected on the character. In order to avoid this, the opponent's attack is distributed by the number of allied characters, and the character that had the action pays one of them, and the actual damage caused by his action is corrected downward. For example, if the enemy character's attack is 1 and the number of friendly characters is 5, the memory to be damaged is set to -0.2 each. When one ally's action (reward 1) fires, it learns 0.8 as a reward. In this case, the memory to be damaged is “−0.2, −0.2, −0.2, −0.2, 0”, and when the action (reward 0.5) of the ally character is ignited, 0.3 Learning as a reward. In this case, the memory to be damaged is “−0.2, −0.2, −0.2, 0, 0”. When the attack of the enemy character is 0.5, the memory to be damaged is “−0.3, −0.3, −0.3, −0.1, 0.1”.

  As described above, in the learning stage, there is a bias in the learning range of action selection / determination in games, particularly games such as role-playing, because the selection and execution of actions such as special moves are restricted. In order to avoid the occurrence, the behavior of the character that can be acted at random is selected or determined at random in the learning stage, regardless of the expected reward (Q value) by each behavior, or the behavior of the character that can be acted By selecting and determining the action with the highest expected reward (Q value) for each, the neural network (NN) parameters can be adjusted each time, but the expected reward (Q value) for each action and Even when the actions of characters that can be acted independently are selected and determined at random, the actions that are restricted from being selected and executed are not always sufficiently selected. There is a situation that it is not. This is because, in the action history management table 41c for managing the history of actions selected by each game character in the game described above, there are more other actions than the actions whose selection / execution is restricted. As a result of being selected, this is because the history of other behaviors occupies a relatively large amount.

  In one embodiment, in order to further improve this, a memory space that is different for each action, or history information is stored for each action, and the same number is selected from all actions (actions), and each action (action) is selected. Can be selected completely randomly from the history of As an example, an action history management table as shown in FIG. 5 can be considered. In FIG. 5, the action history management table is in a state in which the history is managed for each action (behavior 1, action 2... Action n). Furthermore, in this case, two histories are kept for each action, and if the same action is selected more than twice, the old history is deleted, so that the number of histories is limited to two. It has become. In this way, even when selecting randomly, it is surely prevented that many actions that inevitably have a large number of histories are selected, and which actions (behaviors that are difficult to select, It was found that even if it is an action that is easy to select), it is possible to effectively prevent bias in learning by being widely selected. The example shown in FIG. 5 is an example, and the number of action histories for each action is not limited to this, and can be set as appropriate. Further, the behavior history management table for each behavior may be stored in different memory spaces.

  By doing in this way, even in a game where game parameters such as game battles change every moment, the neural network parameters are subjected to reinforcement learning using the Q learning method, and the neural network is updated. It becomes possible to determine a more appropriate action of the enemy character or ally character, to enjoy more naturally without getting tired of the progress of the game, and to enhance the appeal to the entire game.

  In the above, the function which the server 10 has was demonstrated. Next, functions of the terminal device 30 in the embodiment will be described. As shown in FIG. 2, the terminal device 30 includes an information storage unit 51 that stores various information, and a terminal-side control unit 52 that executes control for displaying image information on the terminal side in one embodiment. Have. These functions are realized by the cooperation of hardware such as the CPU 31 and the main memory 32, and various programs and tables stored in the storage 35. For example, instructions included in the loaded program This is realized by the CPU 31 executing. In addition, part or all of the functions of the terminal device 30 illustrated in FIG. 2 can be realized by the cooperation of the server 10 and the terminal device 30, or can be realized by the server 10.

  The information storage unit 51 in the embodiment is realized by the main memory 32 or the storage 35. The terminal-side control unit 52 in one embodiment controls the execution of various terminal-side processes such as selection of actions by the user character and display of received game screen information. For example, when the user selects an action with a user character, the terminal-side control unit 52 transmits the action to the server 10 or when the game parameter changes as a result of the action of an ally character or enemy character, The changed game parameters can be received from the server 10 and displayed.

  The processes and procedures described in this specification are implemented by software, hardware, or any combination thereof other than those explicitly described in the embodiments. More specifically, the processes and procedures described in this specification are performed by mounting logic corresponding to the processes on a medium such as an integrated circuit, a volatile memory, a nonvolatile memory, a magnetic disk, or an optical storage. Realized. Further, the processes and procedures described in this specification can be implemented as a computer program and executed by various computers.

  Even if the processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple Component and / or multiple modules. In addition, even though the data, tables, or databases described herein are described as being stored in a single memory, such data, tables, or databases are provided on a single device. Alternatively, the data can be distributed and stored in a plurality of memories or a plurality of memories arranged in a plurality of devices. Further, the software and hardware elements described herein may be implemented by integrating them into fewer components or by decomposing them into more components.

  In the present specification, when the constituent elements of the invention are described as one or a plurality, or when they are described without being limited to one or a plurality of cases, they should be understood separately in context. The component may be either singular or plural.

DESCRIPTION OF SYMBOLS 10 Server 20 Communication network 30 Terminal device 41 Information storage part 42 Character action control part 51 Information storage part 52 Terminal side control part

Claims (12)

A method for adjusting parameters of a neural network (NN) composed of one or more layers using Q learning,
In response to being executed on one or more computers,
Extracting game parameters relating to one or more game characters as a first input value;
Extracting, based on the first input value, a first expected reward (Q value) due to each action of the game character from the neural network (NN) as an output value;
Extracting the game parameter after selection / execution from one of the actions as a second input value;
Calculating a second expected reward (Q value) for each action of the game character from the neural network (NN) based on the second input value;
Updating a parameter of a neural network (NN) composed of one or more layers based on the first potential reward (Q value) and the second potential reward (Q value);
Is a method of adjusting a parameter of a neural network (NN) using Q-learning, which is characterized in that it is repeatedly executed a plurality of times.   One selection from among the actions of the character selects an action having the highest expected reward (Q value) from the expected rewards (Q value) of the actions, or depends on the actions. The method according to claim 1, wherein the method is performed by determining at random regardless of the expected reward (Q value).   One selection from among the actions of the character includes a set value of a probability of selecting an action having the highest expected reward (Q value) from the expected rewards (Q value) of the actions and the actions. The method according to claim 2, wherein the method is performed based on a set value of a probability that is randomly determined regardless of the expected reward (Q value).   One selection from the actions of the character selects an action having the highest expected reward (Q value) from the expected rewards (Q value) of the actions as the number of selections increases. 4. The method according to claim 3, wherein the probability setting value is increased, and the probability setting value determined at random regardless of the expected reward (Q value) by each action is decreased.   One selection from the actions of the character selects an action having the highest expected reward (Q value) from the expected rewards (Q value) of the actions as the number of selections increases. The probability setting value is increased by about 0.0015%, and the probability setting value determined randomly regardless of the expected reward (Q value) by each action is decreased by about 0.0015%. The method of claim 4.   When one of the actions of the character is selected randomly regardless of the expected reward (Q value) for each action, a plurality of action histories storing action histories for each action 6. The method according to any one of claims 2 to 5, wherein the information is randomly determined from the actions regardless of the amount of history information in the information.   When one of the actions of the character is selected at random regardless of the expected reward (Q value) of each action, the action history in which the number of action histories that can be stored is limited to a predetermined number 6. The method according to any one of claims 2 to 5, wherein the management table is referred to and determined randomly from each action history.   Each of the plurality of action-specific history information is stored in different memory spaces, and one selection of each action of the character is stored in a memory space randomly determined from the plurality of memory spaces. The method according to any one of claims 2 to 7, wherein the action is history information according to action.   9. The method according to claim 2, wherein one of the actions of the character is selected at random from actions whose selection conditions are attached to the actions. The method according to claim 1.   The set value of the probability determined randomly regardless of the expected reward (Q value) by each action is reduced when the action with the selection condition is selected a predetermined number of times or more. 10. A method according to any one of claims 3 to 9. A system for adjusting parameters of a neural network (NN) composed of one or more layers using Q learning,
In response to being executed on one or more computers,
Extracting game parameters of one or more game characters as a first input value;
Extracting, based on the first input value, a first expected reward (Q value) due to each action of the game character from the neural network (NN) as an output value;
Extracting the game parameter after selection / execution from one of the actions as a second input value;
Calculating a second expected reward (Q value) for each action of the game character from the neural network (NN) based on the second input value;
Updating a parameter of a neural network (NN) composed of one or more layers based on the first potential reward (Q value) and the second potential reward (Q value);
Is a system for adjusting parameters of a neural network (NN) using Q-learning, which is characterized in that it is repeatedly executed. A program for adjusting parameters of a neural network (NN) composed of one or more layers using Q learning,
In response to being executed on one or more computers,
Extracting game parameters relating to one or more game characters as a first input value;
Extracting, based on the first input value, a first expected reward (Q value) due to each action of the game character from the neural network (NN) as an output value;
Extracting the game parameter after selection / execution from one of the actions as a second input value;
Calculating a second expected reward (Q value) for each action of the game character from the neural network (NN) based on the second input value;
Updating a parameter of a neural network (NN) composed of one or more layers based on the first potential reward (Q value) and the second potential reward (Q value);
Is a program for adjusting parameters of a neural network (NN) using Q-learning, which is characterized by being repeatedly executed.