CN110555517A - Improved chess game method based on Alphago Zero - Google Patents

Abstract

The invention proposes an improved chess game method based on AlphaGo Zero, expands the application range of the AlphaGo Zero method in the field of man-machine games, and belongs to the field of robot science and technology entertainment. It includes the following steps: using a hybrid network structure including CNN, ResNet and fully connected layers that can effectively avoid gradient dispersion and obtain optimal position convergence, and use a training network to simultaneously output strategies and estimates; 2) using reinforcement learning Strategy, by using the data generated by self-play (Self-Play) for training, solve the problem of large-scale data training of sequential structure, and optimize the model during the game process; 3) neural network training optimization model, define the loss function and Select the corresponding optimizer to iteratively update in the direction of reducing the loss; 4) network model evaluation, use the new model after training for a period of time to play against the model before training, and obtain the performance of the current model according to the number of winning and losing games to determine whether Iterate the model; 5) Use third-party software for visual interactive game testing and evaluation.

Description

Improved Chess Game Method Based on AlphaGo Zero

technical field

The invention proposes an improved chess game method based on AlphaGo Zero, expands the application scope of the AlphaGo Zero method in the field of man-machine games, and belongs to the technical field of robot technology and entertainment.

Background technique

Human-computer game mechanism and its algorithm research Since the birth of the computer, people have not stopped exploring it. Human-computer game is an important branch of artificial intelligence. In the process of studying human-computer game, people have explored many new methods and new ideas of artificial intelligence including machine learning, which have had a profound impact on social life and academic research. .

The reason why the present invention chooses chess as a research example of man-machine game is that in addition to the infinite charm of the game displayed by chess, the larger search space of chess is also more difficult to solve by traditional methods, so that it can better show Excellent performance of deep learning algorithm in solving the problem of too large game tree. The main work of the present invention is to continue to use the construction ideas, network structure and training methods of AlphaGo Lee and AlphaGo Zero, try to construct a neural network model with chess playing ability and carry out machine learning training without human experience; in addition, the present invention It will also introduce the core neural network structure of the chess program and the training mode of the model. Finally, through the self-play of ChessAi engine black and white and the game test with Chess Titans chess program based on traditional search algorithms, fully explore the deep learning algorithm Universality in the field of man-machine games.

Contents of the invention

AlphaGo Zero has adopted single neural network to have finished the function of strategy network (Policynetwork) and valuation network (Value network) in AlphaGo Lee, and the present invention also adopts similar network structure. The network structure built by the model is not a simple convolutional neural network (CNN), but a mixed network structure form. The present invention takes the following technical scheme: a chess game method improved based on AlphaGo Zero comprises the following steps:

1) Adopt a hybrid network structure including CNN, ResNet and fully connected layers that can effectively avoid gradient dispersion and obtain optimal position convergence, and use a training network to simultaneously output strategies and estimates;

2) Using the reinforcement learning strategy, by using the data generated by self-play (Self-Play) for training, to solve the problem of large-scale sequential structure data training, and to optimize the model during the game process;

3) The neural network trains the optimization model, defines the loss function and selects the corresponding optimizer to iteratively update in the direction of reducing the loss;

4) Network model evaluation, use the new model after training for a period of time to play against the model before training, and obtain the performance of the current model according to the number of wins and losses to determine whether to iterate the model;

5) Use third-party software for visual interactive game testing and evaluation.

The step 1) The ResNet network uses a "short-circuit" design, so that the mapping relationship that we originally expected to use F(x) to fit is changed to use H(x)=F(x)+x mapping to introduce and maintain more Rich reference information, so that the network can learn richer content. Using ResNet also produces a smooth forward pass and a smooth reverse residual pass. For example, from the L-th layer of ResNet (the input of the L-th layer is x _l ) to the ResNet of the L-1 layer, it can be deduced that:

①x _l +1＝x _l +F(x _l )

Then there is the following derivation:

②x _l +2＝x _l +1+F(x _l +1)＝x _l +F(x _l )+F(x _l +1)

Therefore, the following general form can be obtained:

③

_The content of the _xL vector of any subsequent layer will have a linear contribution from the xl of a previous layer. The reverse residual transfer also has a similar smoothing process, defining the residual x _lable represents ideally the value corresponding to a certain layer x _L after a given sample and label, according to the chain rule:

④

From this, it can be found that the residual generated by the output x _L of any layer can be transmitted back to x _l of any layer in front of it. This is a "direct" transfer process, so that ResNet has more layers. There will be no obvious efficiency problems; at the same time, according to the equation factor It can be seen that and It is a process of linear superposition rather than a multiplicative relationship, so ResNet has almost no gradient dispersion phenomenon. The hybrid network structure trained by the present invention includes CNN, ResNet and fully connected layers, which includes a 19-layer ResNet structure, and uses a training network to output strategies and estimates at the same time, while simplifying the network structure. The amount of calculation improves the training efficiency and reduces the training time.

Step 2) The entire reinforcement learning process is divided into three asynchronous sub-processes: self-play (Self-Play), model optimization (Optimize, Opt) and network evaluation (Evaluate, Eval).

Self-Play is jointly driven by Monte Carlo Search (Monto Carlo Tree Search, MCTS) and the constructed network model, using a more efficient valuation network for quick judgment, while rule supervision continues to run during this process, providing chess guidance After the end of a single game, it will be formatted, archived and stored, and the chess board will be initialized for the next self-game. This process generates a large number of game records through self-play, and then removes a large amount of training data to provide training data resources for network optimization;

Opt is the training and optimization process of the model network, using the chess record data from the Self-Play stage for model optimization;

Eval is the evaluation process of the new model. After a period of optimization and execution, the game simulation evaluation is carried out, and the performance of the current model is obtained according to the number of wins and losses to judge whether to iterate the model.

Said step 3) neural network training optimization model, defining a loss function and selecting a corresponding optimizer to iteratively update the direction of loss reduction, specifying the optimizer and defining the loss function before starting training after the model definition is completed.

The present invention uses Adam optimizer, the learning rate is set to 0.001, β ₁ is set to 0.9, β ₂ is set to 0.999, ε is set to 10 ^-8 , and the learning rate decay value "dency" is set to 0.0. The Adam optimizer uses the cumulative gradient instead of a single gradient, and uses the root mean square of the cumulative gradient to self-adjust the learning rate, which helps to promote the model to converge in the correct direction and speed up the convergence speed of the model.

The policy probability output of the policy network is defined as multi-class logarithmic loss, and one-hot encoding is used on the input data of the model to reduce the cognitive interference of the network on cross-entropy loss functions such as multi-class logarithmic loss. For the evaluation of the situation winning rate output by the evaluation network, the loss function is defined as the mean square error. For the neural network with single value output, it is common and effective to use the mean square error as the loss function.

Described step 4) network model evaluation, use the new model after training for a period of time to play a game with the model before training, obtain the performance of current model according to the number of winning and losing rounds to determine whether to iterate the model;

Before the model training, the original model needs to be backed up for use in this step. For the training time, it is set indirectly by setting epochs. After training one epochs, the model is evaluated, and each evaluation is performed with 100 games of simulated games. The game process of each game is similar to the Self-Play stage, the difference is that the MCTS of both sides in each round is judged and updated by different model networks. After the game is over, judge whether the new model replaces the original model and becomes the optimal model according to whether the winning rate of the new model reaches the iterative winning rate threshold (hyperparameter, set to 0.55). After the model iteration is completed, it will return to the Self-Play stage, use the new model for self-play and continuous optimization, and complete the continuous improvement of the network game ability.

The step 5) uses third-party software to perform visual interactive game testing and evaluation.

The present invention focuses on the construction and training of the network model. For the design of the front-end interface, a third-party ChessGUI Arena is used for visual interactive game testing. The chess engine built through the model network uses the ChessAi logo on the front end of the GUI, and the Arena GUI and the ChessAi engine use UCI's FEN (Forsyth-Edwards Notation) string for front-end connection and situation interaction.

The present invention uses the Elometer online Elo rating rating system from the University of Duesseldorf to carry out the Elo rating of the ChessAi engine. The Elo rating system is an evaluation method for calculating the relative level of player skills in zero-sum games. Elometer gives Elo by solving 76 puzzles Estimated value of rating.

The present invention has the following advantages due to the adoption of the above technical scheme:

The neural network trained by the present invention is a mixed network structure including CNN, ResNet and fully connected layers. The downsampling method, which can avoid overfitting and reduce a certain amount of calculation, not only exerts its performance, but also leads to the image "blurring" phenomenon that occurs with the increase in the number of network layers, which in turn leads to blurring in the training process. Gradient dispersion, which eventually leads to problems where the model converges in the wrong direction or deviates from the optimal position;

The hybrid neural network of the present invention includes a 19-layer ResNet structure, and uses a training network to simultaneously output strategies and estimates, which not only simplifies the network structure, but also reduces the amount of computation, improves training efficiency, and reduces training time.

Description of drawings

The model network structure of Fig. 1 the present invention;

Figure 2 Model Network Reinforcement Learning Mode;

Figure 3 Schematic diagram of MCTS simulation;

Figure 4 Single game self-play process

Figure 5 Model network training optimization

Figure 6 GUI and running mode

Figure 7 Elometer test feedback

Figure 8 Game test results (10 rounds)

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Step 1. Adopt a hybrid network structure including CNN, ResNet and fully connected layers that can effectively avoid gradient dispersion and obtain optimal position convergence, and use a training network to simultaneously output strategies and estimates;

The ResNet network uses a "short-circuit" design, so that the mapping relationship that we originally expected to use F(x) to fit is changed to use H(x)=F(x)+x mapping to introduce and maintain richer reference information. So that the network can learn richer content. Using ResNet also produces a smooth forward pass and a smooth reverse residual pass. For example, from the L-th layer of ResNet (the input of the L-th layer is x _l ) to the ResNet of the L-1 layer, it can be deduced that

①x _l +1＝x _l +F(x _l )

Then there is the following derivation:

②x _l +2＝x _l +1+F(x _l +1)＝x _l +F(x _l )+F(x _l +1)

Therefore, the following general form can be obtained:

③

_The content of the _xL vector of any subsequent layer will have a linear contribution from the xl of a previous layer. The reverse residual transfer also has a similar smoothing process, defining the residual x _lable represents ideally the value corresponding to a certain layer x _L after a given sample and label, according to the chain rule:

④

From this, it can be found that the residual generated by the output x _L of any layer can be transmitted back to x _l of any layer in front of it. This is a "direct" transfer process, so that ResNet has more layers. There will be no obvious efficiency problems; at the same time, according to the equation factor It can be seen that and It is a process of linear superposition rather than a multiplicative relationship, so ResNet has almost no gradient dispersion phenomenon. The hybrid network structure trained by the present invention includes CNN, ResNet and fully connected layers, which includes a 19-layer ResNet structure, and uses a training network to output strategies and estimates at the same time, which not only simplifies the network structure, but also reduces the calculation The amount improves the training efficiency and reduces the training time.

Step 2, as shown in Figure 2, the entire reinforcement learning process is divided into three asynchronous sub-processes: self-play (Self-Play), model optimization (Optimize, Opt) and network evaluation (Evaluate, Eval).

The specific process of reinforcement learning:

⑴Self-Play

Self-Play is jointly driven by Monte Carlo Search (Monto Carlo Tree Search, MCTS) and the constructed network model. MCTS in the process of Self-Play uses a more efficient valuation network for quick judgment. As shown in the MCTS simulation in Figure 3, the MCTS and the model network drive the game network, while the rule supervision continues to run during this process, providing chess guidance and final judgment of victory and defeat, and formatting, archiving, storage and initialization after a single game The chess surface is prepared for the next self-game. Figure 4 shows the process of a single game of self-play. This process generates a large number of game records through self-play, and then removes a large amount of training data to provide training data resources for network optimization;

⑵Model optimization (Optimize, Opt)

Opt is the training optimization process of the model network, using the chess record data from the Self-Play stage for model optimization.

⑶ network evaluation (Evaluate, Eval)

Eval is the evaluation process of the new model. After the optimization is executed for a period of time, the simulation evaluation of the game is carried out. Based on the evaluation results, it is determined whether to continue the optimization or iteratively update the model. Network evaluation is to use the new model after training for a period of time to play against the model before training, and obtain the performance of the current model according to the number of wins and losses to judge whether to iterate the model.

Step 3, the present invention adopts the neural network training optimization model, defines the loss function and selects the corresponding optimizer to carry out iterative update to the direction of reducing the loss; the designation of the optimizer and the definition of the loss function before starting the training after the model definition is completed .

Specifically include the following steps:

(1) The present invention uses Adam optimizer, the learning rate is set to 0.001, β ₁ is set to 0.9, β ₂ is set to 0.999, ε is set to 10 ^-8 , and the learning rate decay value "dency" is set to 0.0. The Adam optimizer uses the cumulative gradient instead of a single gradient, and uses the root mean square of the cumulative gradient to self-adjust the learning rate, which helps to promote the model to converge in the correct direction and speed up the convergence speed of the model.

(2) The strategy probability output of the strategy network is defined as multi-class logarithmic loss, and one-hot encoding is used on the input data of the model to reduce the cognitive interference of the network on cross-entropy loss functions such as multi-class logarithmic loss. For the evaluation of the situation winning rate output by the evaluation network, the loss function is defined as the mean square error. For the neural network with single value output, it is common and effective to use the mean square error as the loss function.

Step 4: Use network model evaluation, use the new model after training for a period of time to play against the model before training, and obtain the performance of the current model according to the number of wins and losses to determine whether to iterate the model. Figure 5 shows the optimization of the model network training. Before the model training, the original model needs to be backed up for use in this step. For the training time, it is set indirectly by setting the epochs, and the model is evaluated after training one epochs, each time The evaluation carried out 100 rounds of simulated games. The game process of each game is similar to the Self-Play stage. The difference is that the MCTS of both sides in each round is judged and updated by different model networks. After the game is over, judge whether the new model replaces the original model and becomes the optimal model according to whether the winning rate of the new model reaches the iterative winning rate threshold (hyperparameter, set to 0.55). After the model iteration is completed, it will return to the Self-Play stage, use the new model for self-play and continuous optimization, and complete the continuous improvement of the network game ability.

Step 5. The present invention focuses on the realization and training of the model network. For the design of the front-end interface, a third-party Chess GUI Arena is used for visual interactive game testing, and the Elometer online Elo rating from the University of Duesseldorf is used to perform the Elo rating of the ChessAi engine.

The chess engine constructed by the model network in the present invention uses the ChessAi logo on the front end of the GUI, and ArenaGUI and the ChessAi engine use the FEN (Forsyth-Edwards Notation) string of UCI to perform front-end connection and situation interaction. Figure 6 shows the front-end presentation of the Arena GUI and the deployment and operation structure of the entire project. The GUP version of Keras and TensorFlow are used to complete distributed data calculations, and the Master CPU is used to automatically allocate computing resources and thread processing. Elometer estimates the Elo rating by solving 76 puzzles. The test feedback is shown in Figure 7. The final Elo rating is 1545, and the 95% confidence interval is [1392, 1698].

In the actual game testing stage, the present invention uses the ChessAi engine to carry out ten games of games, including six games of ChessAi self-play and four games of ChessAi and Chess Titans. The final result of the game is shown in Figure 8.

Claims (7)

1. the improved chess game method based on Alphago Zero comprises the following steps:

1) Adopting a mixed network structure including CNN, ResNet and full connection layers which can effectively avoid gradient dispersion and can obtain optimal position convergence, and simultaneously outputting strategy and estimation value by using a training network;

2) the method adopts a reinforcement learning strategy, trains by using data generated by Self game (Self-Play), solves the problem of large scale of data training of a sequential structure, and optimizes a model in the game process;

3) training an optimization model by a neural network, defining a loss function and selecting a corresponding optimizer to carry out iterative update in a loss reduction direction;

4) Evaluating a network model, namely playing chess by using a new model after training for a period of time and a model before training, and acquiring the performance of the current model according to the number of wins and losses to judge whether to iterate the model;

5) and carrying out visual interactive game testing and evaluation by adopting third-party software.

2. an AlphaGo Zero-based improved chess gaming method according to claim 1, wherein: the step 1), the ResNet network uses a "short circuit" design, so that the mapping relationship that we would like to fit using f (x) is changed to using h (x) ═ f (x) + x mapping to introduce and maintain richer reference information, thereby enabling the network to use a "short circuit" designLearn richer content. Using ResNet, a smooth forward transfer and a smooth reverse residual transfer process also result. E.g. an Lth layer of ResNet (the input of the Lth layer is x)_l) ResNet to layer L-1, one can deduce

①x_l+1＝x_l+F(x_l)

The following derivation follows:

②x_l+2＝x_l+1+F(x_l+1)＝x_l+F(x_l)+F(x_l+1)

The following general form can be obtained:

③

X of any layer behind_Lthe contents of the vector will have a portion of x from one layer above_lThe linear contribution of (a). The inverse residual transfer also has a similar smoothing process, defining the residualx_lableMeaning that ideally after a given sample and label, at a certain level x_LThe corresponding values are determined by the chain rule:

④

From this, it can be found that the output x of any one layer_Lthe generated residual can be transmitted back to x of any layer before it_lIn the above, the transmission process is a 'direct' transmission process, so that the ResNet does not have obvious efficiency problem when the number of layers is increased; at the same time, according to the equation factorin a clear view of the above, it is known that,Andthe method is a linear superposition process and not a continuous multiplication relation, so that the ResNet hardly has the phenomenon of gradient diffusion. The mixed network structure which is trained by the invention and comprises CNN, ResNet and full connection layers comprises a ResNet structure with 19 layers, and a training network is used for simultaneously outputting strategies and estimated values, thereby simplifying the network structure, reducing the operation amount, improving the training efficiency and reducing the training time.

3. An AlphaGo Zero-based improved chess gaming method according to claim 1, wherein: in the step 2), the whole reinforcement learning process is divided into three asynchronous sub-processes of Self-game (Self-Play), model optimization (Optimize, Opt) and network evaluation (Evaluate, Eval), and the specific steps are as follows:

Step S21) the Self-Play is driven by Monte Carlo Search (MCTS) and the constructed network model, a large amount of chess playing spectrums are generated through Self-game in the process, and then a large amount of training data are obtained to provide training data resources for network optimization;

Step S22) Opt is a training optimization process of the model network, using the chess manual data from the stage of Self-Play to perform model optimization;

Step S23) Eval is the evaluation process of the new model, the simulation evaluation of playing is carried out after the optimization is carried out for a period of time, and whether the optimization is continued or the iterative update of the model is carried out is judged according to the evaluation result.

4. an improved chess gaming method based on AlphaGo Zero as recited in claim 3, wherein the MCTS in the Self-Play process of step S21): and a more efficient estimation network is adopted for quick judgment. The MCTS and the model network drive the playing network, meanwhile, the rule supervision continuously operates in the process, chess playing guidance and final outcome judgment are carried out, and after a single outcome is finished, formatting filing storage is carried out, and chess surfaces are initialized to be played for the next time.

5. a game method of chess according to claim 3 based on the improvement of AlphaGo Zero, characterized by the steps of S22) and S23): and (3) network evaluation, namely, playing games by using a new model after training for a period and a model before training, and acquiring the performance of the current model according to the number of wins and losses to judge whether to iterate the model.

6. the improved chess gaming method based on AlphaGo Zero of claim 1, wherein said step 3): the specification of the optimizer and the definition of the loss function are performed after the model definition is completed and before the training is started. Specifically comprises the following steps

step S31) the invention uses an Adam optimizer with a learning rate set to 0.001, β₁Is set to 0.9, beta₂set to 0.999, ∈ set to 10^-8The learning rate attenuation value "dency" is set to 0.0. The Adam optimizer replaces single gradient with cumulative gradient, uses cumulative gradient root mean square self-adjusting learning rate, and helps to promote the convergence of the model to the right direction and accelerate the convergence speed of the model.

Step S32) the strategy probability output of the strategy network is defined as multi-class logarithmic loss, and the one-hot coding is used on the input data of the model to weaken the cognitive interference of the network on cross entropy class loss functions such as the multi-class logarithmic loss. For the situation win estimation of estimation network output, the loss function is defined as mean square error, and for the neural network of single value output, it is common and effective to use the mean square error as the loss function.

7. The improved chess gaming method based on AlphaGo Zero of claim 1, wherein said step 4): before model training, backup of original models is needed to be carried out for use in the step, the training time is set indirectly by setting epochs, the models are evaluated when one epochs is trained, 100 rounds of simulated Play are carried out for each evaluation, the playing process of each round is similar to that of a Self-Play stage, and the difference is that MCTS of two sides of each round is judged and updated by different model networks. And after the chess playing is finished, judging whether the new model replaces the original model to become the optimal model according to whether the new model rate reaches an iteration rate threshold value (the over-parameter is set to be 0.55). And after the model iteration is finished, returning to a Self-Play stage, and using the new model to carry out Self-Play and continuous optimization to finish the continuous improvement of the network Play capability.