KR20220164323A - Method and device for providing service that deep learning-based imperfect information game - Google Patents

Abstract

The present invention provides a method and a device for providing an imperfect information game service based on deep learning which can determine the best action in accordance with a present play state. According to an embodiment of the present invention, the method for providing an imperfect information game service based on deep learning provides a deep learning-based imperfect information game service through a user terminal by a game play server and comprises: a step of acquiring a training dataset including go-stop board states for a plurality of actions; and a step of training a deep learning model of the imperfect information service based on the acquired training dataset. The step of training the deep learning model includes at least one among: a step of performing state determination training based on the training dataset; a step of performing winning rate estimation training based on the training dataset; a step of performing score estimation training based on the training dataset; and a step of performing genealogy achievement probability training based on the training dataset.

Description

Deep learning-based imperfect information game service providing method and its device

The present invention relates to a deep learning-based imperfect information game service providing method and apparatus therefor. More specifically, it relates to a deep learning-based incomplete information game service providing method and apparatus for performing the best action according to the current play state in an incomplete information game.

BACKGROUND ART [0002] In recent years, as the spread of wired/wireless Internet has rapidly increased with the development of communication and network technology, various types of services have been provided through a homogeneous medium called the Internet.

In particular, the game service is a service used by many users among services provided through the Internet, and various games are provided.

Among them, many battle-type games in which game points are exchanged in exchange for victory are being serviced, and games such as 'GOSTOP', 'janggi' or 'poker' are popular with many users. they are games

On the other hand, in addition to this trend, it has recently been possible to play a game with a programmed artificial intelligence computer rather than a human being through the above game.

In detail, there is a trend in which research is being actively conducted to enhance the power of artificial intelligence computers so that they can play matches at the human level.

Recently, developers have applied the Monte Carlo Tree Search (MCTS) algorithm and deep learning technology to artificial intelligence computers, raising the power of artificial intelligence computers to a considerable level.

However, conventional artificial intelligence computers are specialized in complete information games (e.g., Go and chess) in which all information related to the progress of the game is fully disclosed to all game players, so some information necessary for game progress ( Development of technology for artificial intelligence computers capable of realizing high performance in imperfect information games (eg, GoStop, etc.) This is an obscure situation.

US 2014-0315625 A1

An object of the present invention is to provide a deep learning-based incomplete information game service providing method and apparatus for performing the best action according to the current play state in an incomplete information game in which some information necessary for game progress is not disclosed. there is.

In detail, the present invention is to provide a deep learning-based incomplete information game service providing method and apparatus for determining the best action that can be performed in a current play state in an incomplete information game.

In addition, the present invention is to provide a deep learning-based incomplete information game service providing method and apparatus for estimating a win rate and a score by performing a roll-out based on a current play state in an incomplete information game.

In addition, the present invention is to provide a deep learning-based incomplete information game service providing method and apparatus for predicting whether a genealogy is achieved in an incomplete information game.

In addition, the present invention is to provide a deep learning-based incomplete information game service providing method and apparatus for estimating an opponent's action in an incomplete information game based on deep learning.

In addition, the present invention adds a score-related parameter term to the existing UCT (Upper Confidence Bound 1 applied to Trees) algorithm to develop a deep learning-based algorithm using an algorithm optimized for incomplete information games that must consider not only game wins and losses but also scores. It is intended to provide a method and device for providing incomplete information game service.

However, the technical problems to be achieved by the present invention and the embodiments of the present invention are not limited to the technical problems described above, and other technical problems may exist.

A method for providing a game service based on deep learning based on incomplete information according to an embodiment of the present invention is a method for providing a game service based on deep learning based on incomplete information through a user terminal in a game play server. Obtaining a training data set comprising; and learning a deep learning model of the incomplete information service based on the obtained training data set, wherein the learning of the deep learning model comprises performing state decision training based on the training data set. And, performing odds estimation training based on the training data set, performing score estimation training based on the training data set, and performing genealogy achievement probability training based on the training data set. It includes at least one step.

At this time, the state judgment training takes the first high stop board state as an input and trains the deep learning model to output state determination information that is information obtained by determining at least one action that can be performed in the input first high stop board state. .

In addition, the state determination information is information for determining an activation state, which is a reference element for detecting at least one action that can be performed in the state of the first high-stop board.

In addition, the activation state is a hand state (HAND State), a shake state (SHAKE State), an eating tile selection state (SELECT State), a drinking cup position selection state (SWITCH State), a gostop state (GOSTOP State), a game end state ( END State), hand holding state (CHANCE_HAND State), and deck flipping state (CHANCE_FLIP State).

In addition, the odds estimation training takes the first high-stop board state as an input based on roll-out using the training data set, and the winning rate by actions possible in the input first high-stop board state The deep learning model is trained to output an odds ratio estimation value, which is the estimated information.

In addition, the score estimation training takes the first high-stop board state as an input based on roll-out using the training data set, and scores by actions possible in the input first high-stop board state The deep learning model is trained to output a score estimation value, which is the estimated information.

In addition, the genealogy achievement probability training takes the first high-stop board state as an input and outputs a genealogy achievement probability value, which is information predicting the probability of achieving a predetermined genealogy at the end of roll-out based on the input first high-stop board state The deep learning model is trained to do so.

In addition, the state of the high-stop board is the player's hand, the player's eaten hand, the opposing player's eaten hand, the floor laid on the floor, the score that the player needs to update to win, the score that the opponent player needs to update to win, the player It includes at least some of the selected hand, opponent player's selected hand, bonus hand, hostage type, go execution count, previous player information, knock-inducing information, and 9 blood usage information.

On the other hand, an incomplete information game service providing apparatus according to an embodiment of the present invention includes a communication unit for receiving a training data set; memory for storing deep learning models; and a processor that performs at least one of state judgment training, odds estimation training, score estimation training, and genealogy achievement probability training for the deep learning model based on the received training data set.

At this time, the processor performs the training so that the deep learning model is trained to output at least one of state determination information, odds estimation value, score estimation value, and genealogy achievement probability value for the first high-stop version state.

A deep learning-based incomplete information game service providing method and apparatus according to an embodiment of the present invention, using at least one of state determination information based on a current play state in an incomplete information game, an odds estimate value, a score estimate value, and a genealogy achievement probability value, of actions can be performed.

In addition, a deep learning-based imperfect information game service providing method and apparatus according to an embodiment of the present invention include some information such as the hand (or card, etc.) of an opponent and the hand (or card, etc.) included in a deck It is possible to estimate the action of an opponent player in an incomplete information game that discloses based on deep learning according to a predetermined algorithm rather than a random method.

In addition, the deep learning-based incomplete information game service providing method and apparatus according to an embodiment of the present invention can determine the best action according to the current play state using an improved UCT algorithm in a form more optimized for incomplete information games. have.

However, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood from the description below.

1 is an exemplary view of a deep learning-based incomplete game service system according to an embodiment of the present invention.
2 is an example of a diagram for explaining the structure of an operation model of a game play server according to an embodiment of the present invention.
3 is an example of a diagram for explaining a process in which an operation model according to an embodiment of the present invention performs a Monte Carlo Tree Search (MCTS) according to a pipeline of a search unit.
4 is a flowchart illustrating a method of learning an operation model of a game play server according to an embodiment of the present invention.
5 is a conceptual diagram illustrating a method of learning an operation model of a game play server according to an embodiment of the present invention.
6 is a conceptual diagram for explaining a method of performing state judgment training according to an embodiment of the present invention.
7 is an example of a diagram showing transition relationships between states according to an embodiment of the present invention.
8 is a conceptual diagram illustrating roll-out based training according to an embodiment of the present invention.
9 is an example of a diagram for explaining a method of estimating odds and scores according to an embodiment of the present invention.
10 is a flowchart illustrating a method for providing a deep learning-based incomplete information game service by an operation model of a game play server according to an embodiment of the present invention.
11 is a conceptual diagram for explaining a method for providing a game service based on deep learning based on incomplete information by an operation model of a game play server according to an embodiment of the present invention.
12 is an example of a diagram for explaining a method of estimating an action of an opponent player in MCTS of an incomplete information game service according to an embodiment of the present invention.
13 is an example of a diagram showing a sigmoid function used in the G-UCT algorithm according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention, and methods for achieving them will become clear with reference to the embodiments described later in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first and second are used for the purpose of distinguishing one component from another component without limiting meaning. Also, expressions in the singular number include plural expressions unless the context clearly dictates otherwise. In addition, terms such as include or have mean that features or elements described in the specification exist, and do not preclude the possibility that one or more other features or elements may be added. In addition, in the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, since the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of description, the present invention is not necessarily limited to the illustrated bar.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof will be omitted. .

1 is an exemplary view of a deep learning-based incomplete game service system according to an embodiment of the present invention.

Referring to FIG. 1 , a deep learning-based incomplete game service system according to an embodiment may include a terminal 100, a game service providing server 200, a game play server 300, and a network 400.

Each component of FIG. 1 may be connected through a network 400 . It means a connection structure capable of exchanging information between nodes such as the terminal 100, the game service providing server 200, and/or the game play server 300. An example of such a network is 3GPP (3rd Generation Partnership) Project) network, LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), A personal area network (PAN), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a digital multimedia broadcasting (DMB) network, and the like are included, but are not limited thereto.

On the other hand, the incomplete information game service according to an embodiment of the present invention can be applied to any game service in which some information necessary for game progress is kept private. Explain.

- Terminal (100: Terminal)

First, the terminal 100 is a terminal of a user who wants to receive a go-stop game service. Also, the terminal 100 is one or more computers or other electronic devices used by a user to execute applications that perform various tasks.

For example, it includes a computer, laptop computer, smart phone, mobile phone, PDA, tablet PC, or any other device operable to communicate with the game service providing server 200 and/or the game play server 300.

However, it is not limited thereto, and the terminal 100 is executed on various machines, includes processing logic that interprets and executes commands stored in a plurality of memories, and provides a graphical user interface (GUI) on an external input/output device. It may contain various other elements, such as processes that display graphical information.

In addition, the terminal 100 may be connected to an input device (eg, mouse, keyboard, touch sensitive surface, etc.) and an output device (eg, display device, monitor, screen, etc.).

Applications executed by the terminal 100 may include game applications, web browsers, web applications running on web browsers, word processors, media players, spreadsheets, image processors, security software, or the like. .

In addition, the terminal 100 may include at least one memory 101 storing instructions, at least one processor 102 and a communication unit 103 .

The memory 101 of the terminal 100 may store a plurality of application programs or applications running in the terminal 100, data for operation of the terminal 100, and commands.

Instructions are executable by the processor 102 to cause the processor 102 to perform operations, such as sending a go-stop game play request signal, sending and receiving game data, sending and receiving action information, sending a status determination request signal, status It may include operations for receiving a decision result, requesting game time information, receiving game time information, and receiving various types of information.

In addition, the memory 101 may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware, and the memory 130 performs the storage function of the memory 101 on the Internet. It can also be a web storage that performs.

The processor 102 of the terminal 100 may perform data processing to receive the GoStop game service by controlling overall operations.

When the GoStop game application is executed in the terminal 100, a GoStop game environment is configured in the terminal 100. In addition, the GoStop game application exchanges GoStop game data with the game service providing server 200 through the network 400 so that the GoStop game service is executed on the terminal 100 .

These processors 102 include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, and microcontrollers. It may be micro-controllers, microprocessors, or any type of processor for performing other functions.

The communication unit 103 of the terminal 100 uses the following communication methods (eg, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink (HSUPA)) Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.), WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA ( A wireless signal may be transmitted and received with at least one of a base station, an external terminal, and a server on a network constructed according to Digital Living Network Alliance (WiBro), Wireless Broadband (WiBro), and World Interoperability for Microwave Access (WiMAX).

Such a terminal 100 may perform at least a part of a function operation performed in at least one of the game service providing server 200 and the game play server 300 to be described later.

- Game service providing server (200: Game service providing server)

The GoStop game service provided by the game service providing server 200 may be configured in such a way that a virtual computer user provided by the game service providing server 200 and a real user participate in a game together. In this case, one real user and one computer user play the game together in a go-stop game environment implemented on the user-side terminal 100 .

In another aspect, the Go-Stop game service provided by the game service providing server 200 may be configured in a form in which a plurality of user-side devices participate to play the Go-Stop game.

The game service providing server 200 may include at least one memory 201 storing instructions, at least one processor 202, and a communication unit 203.

The memory 201 of the game service providing server 200 stores a plurality of application programs or applications running in the game service providing server 200 and data for the operation of the game service providing server 200. , can store commands. Instructions are executable by the processor 202 to cause the processor 202 to perform operations, which include receiving a game execution request signal, transmitting and receiving game data, transmitting and receiving action information, transmitting and receiving a state determination request signal, and transmitting and receiving a state determination result. , action preparation time transmission and reception, game information time transmission and reception, and various transmission operations.

In addition, the memory 201 may store a plurality of play data that has been played in the game service providing server 200 or a plurality of play data that has been previously disclosed.

Each of the plurality of play data may include all information from a first action, which is information about a first action at the start of a competitive battle, to a final action at which the competitive game ends. That is, the plurality of play data may include action-related history information.

In addition, the play data may include each play state according to the order of actions performed by the go-stop game service, that is, the go-stop board state in the embodiment.

Here, the play state (hereinafter referred to as the Go-Stop board state) is a set of data according to the progress of the game, including the state in which the Go-Stop hand is placed on the Go-Stop board, and the like. hand, floored hand (i.e. open hand), number of points a player must score to win, number of points opponent player must score to win, hand selected when player is in hand, and opponent player in hand It may include the selected hand, knock-inducing information, bonus hand hostage type, go execution count, previous player information, and/or 9-fold blood usage information.

The game service providing server 200 can provide the game play server 300 with a plurality of stored play data for training of the game play server 300 .

In addition, the memory 201 may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware, and the memory 201 performs the storage function of the memory 201 on the Internet. It can also be a web storage that performs.

The processor 202 of the game service providing server 200 may perform data processing to provide a go-stop game service by controlling overall operations. These processors 202 include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, and microcontrollers. It may be micro-controllers, microprocessors, or any type of processor for performing other functions.

The game service providing server 200 may communicate with the terminal 100 and the game play server 300 via the network 400 through the communication unit 203 .

- Game play server (300: Game playing server)

The game play server 300 may include a separate cloud server or a computing device. In addition, the game play server 300 may be a neural network system installed in the processor of the terminal 100 or the data processing unit of the game service providing server 200. It will be described as a device separate from the service providing server 200.

The game play server 300 may include at least one memory 301 storing instructions, at least one processor 302 and a communication unit 303 .

The game play server 300 builds an action model, which is a deep learning model that learns by itself according to the Go-Stop rules, and is an artificial intelligence computer that can compete with the user of the terminal 100. Actions can be performed in A detailed description of how the game play server 300 trains the motion model follows the description of the motion model of FIGS. 2 to 9 .

The memory 301 of the game play server 300 stores a plurality of application programs or applications running in the game play server 300, data for the operation of the game play server 300, and instructions. can be saved Instructions are executable by the processor 302 to cause the processor 302 to perform operations, which include operation model learning (training) operations, action information transmission and reception, action preparation time reception, game time information reception, and various transmissions. Actions may be included.

Also, the memory 301 may store an operation model that is a deep learning model. In addition, the memory 301 may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware, and the memory 301 performs the storage function of the memory 301 on the Internet. It can also be a web storage that performs.

The processor 302 of the game play server 300 reads the motion model stored in the memory 302, and learns the motion model described below and performs actions in the GoStop game according to the built neural network system.

Depending on the embodiment, the processor 302 may be configured to include a main processor that controls all units and a plurality of graphics processing units (GPUs) that process large-capacity calculations required for driving a neural network according to an operation model. have.

The game play server 300 may communicate with the game service providing server 200 via the network 400 through the communication unit 303 .

<action model>

2 is an example of a diagram for explaining the operation model structure of the game play server 300 according to an embodiment of the present invention.

Referring to FIG. 2 , the operation model according to an embodiment of the present invention is a deep learning model of the game play server 300, and includes a state determination unit 310, a simulation unit 320, an action determination unit 330, and a deep learning A neural network 340 and a search unit 350 may be included.

In detail, the action model is played using the MCTS module 390 including the simulation unit 320, the deep learning neural network 340 and the search unit 350, the state determination unit 310 and the action determination unit 330. It can be learned as a model that determines the action to win in .

Specifically, the search unit 350 may perform a Monte Carlo Tree Search (MCTS) operation according to the guide of the deep learning neural network 340 . MCTS is a heuristic search algorithm for some kind of decision-making. That is, the search unit 350 may perform MCTS based on the movement probability value (p) and/or value value (V) provided by the deep learning neural network 340 .

)을 출력할 수 있다. 시뮬레이션부(320)는 탐색 확률값(

)에 따라 스스로 고스톱 대전을 할 수 있다. 시뮬레이션부(320)는 게임의 승패가 결정되는 시점까지 스스로 고스톱 대전을 진행하고, 자가 대전(즉, self-play)이 종료되면 고스톱판 상태(S), 탐색 확률값(

), 자가 플레이 가치값(z)을 딥러닝 신경망(340)을 학습시키기 위한 정답 데이터로서 제공할 수 있다. 여기서 고스톱판 상태(S)는 고스톱판에 고스톱 패가 놓여진 상태 등을 포함하는 게임 진행에 따른 데이터들의 집합일 수 있다. 자가 플레이 가치값(z)은 고스톱판 상태(S)에서 자가 대전을 하였을 때 승률 값이다. 또한, 시뮬레이션부(320)는 자가 대전의 결과로 승률추정값(W), 점수추정값(G) 및/또는 족보 성취확률값(L)을 액션 결정부(330)로 제공할 수 있다. For example, the search unit 350 guided by the deep learning neural network 340 performs MCTS to search probability values (probability distribution values for actions according to the number of visits) (

) can be output. The simulation unit 320 provides a search probability value (

), you can play Go-Stop by yourself. The simulation unit 320 proceeds with the Go-Stop game by itself until the game's win or loss is decided, and when the self-play (ie, self-play) ends, the Go-Stop board state (S), the search probability value (

), and the self-play value value z may be provided as correct answer data for learning the deep learning neural network 340 . Here, the high-stop board state (S) may be a set of data according to the progress of the game including a state in which a high-stop hand is placed on the high-stop board. The self-play value value (z) is a win rate value when a self-play is played in the high-stop board state (S). In addition, the simulation unit 320 may provide an odds estimation value (W), a score estimation value (G), and/or a genealogy achievement probability value (L) to the action determination unit 330 as a result of the self-play.

)과 동일해지도록 트레이닝되고, 가치값(V)이 자가 플레이 가치값(z)과 동일해지도록 트레이닝될 수 있다. 이후 트레이닝된 딥러닝 신경망(340)은 탐색부(350)를 가이드하고, 탐색부(350)는 이전 탐색 확률값(

)보다 더 좋은 액션을 찾도록 액션 준비 시간 동안 MCTS를 진행하여 새로운 탐색 확률값(

)을 출력하게 한다. 예를 들어, 액션 준비 시간은 MCTS 진행 시간에 따라 평균 액션 준비 시간, 제1 액션 준비 시간 및 제2 액션 준비 시간 중 어느 하나의 액션 준비 시간을 따를 수 있다. 액션 준비 시간은 기본적으로 평균 액션 준비 시간으로 설정되어 있을 수 있다. The deep learning neural network 340 may output a movement probability value (p) and a value value (V). The movement probability value p may be a probability distribution value representing numerically which action to perform according to the high-stop board state S is a good action to win the game. The value value (V) represents the overall win rate value in the current player's state (State). Here, the comprehensive win rate value means data representing the win rate of the corresponding player as a single value by comprehensively considering at least one action that a player can perform and future situations as the at least one or more actions are performed. can do. For example, an action having a high movement probability value p may be a good action. The deep learning neural network 340 determines that the movement probability value (p) is the search probability value (

), and the value (V) may be trained to be equal to the self-play value (z). Then, the trained deep learning neural network 340 guides the search unit 350, and the search unit 350 uses the previous search probability value (

) to find a better action than the new search probability value (

) to output. For example, the action preparation time may follow one action preparation time among an average action preparation time, a first action preparation time, and a second action preparation time according to the MCTS progress time. The action preparation time may be basically set to an average action preparation time.

)에 기초하여 고스톱판 상태(S)에 따른 새로운 자가 플레이 가치값(z)을 출력하고 고스톱판 상태(S), 새로운 탐색 확률값(

), 새로운 자가 플레이 가치값(z)을 딥러닝 신경망(340)에 제공할 수 있다. 딥러닝 신경망(340)은 이동 확률값(p)과 가치값(V)이 새로운 탐색 확률값(

)과 새로운 자가 플레이 가치값(z)으로 출력되도록 다시 트레이닝될 수 있다. The simulation unit 320 provides a new search probability value (

) Based on the high stop board state (S), a new self-play value value (z) is output, and the high stop board state (S), a new search probability value (

), the new self-play value value z may be provided to the deep learning neural network 340 . The deep learning neural network 340 determines that the movement probability value (p) and the value value (V) are a new search probability value (

) and a new self play value (z) can be trained again.

That is, the action model can be trained to find a better action for winning the match by repeating this process for the deep learning neural network 340 . As an example, the action model may use action loss (l). The action loss (l) is shown in Equation 1.

[Equation 1]

는 신경망의 파라미터이고, c는 매우 작은 상수이다.

is a parameter of the neural network, and c is a very small constant.

와 p가 같아 지도록 하는 것은 크로스 엔트로피 손실(cross entropy loss) 텀에 해당되고,

에 c를 곱하는 것은 정규화 텀으로 오버피팅(overfitting)을 방지하기 위한 것이다.Making z and v equal in the action loss (l) of Equation 1 corresponds to the mean square loss term,

and p are equal to the cross entropy loss term,

Multiplying by c is to prevent overfitting with the regularization term.

Meanwhile, the deep learning neural network 340 may have a neural network structure. For example, the deep learning neural network 340 may include one convolution block and 19 residual blocks. A convolution block may have a form in which several 3X3 convolution layers are overlapped. One residual block may have a form in which several 3X3 convolutional layers are overlapped and a skip connection is included. A skip connection is a structure in which an input of a predetermined layer is outputted by summing an output value of a corresponding layer and inputted to another layer.

3 is an example of a diagram for explaining a process of performing a Monte Carlo Tree Search (MCTS) according to the pipeline of the search unit 350 by the operating model according to an embodiment of the present invention.

Referring to FIG. 3 , an operation model according to an embodiment of the present invention performs a Monte Carlo tree search using an MCTS module 390 including a simulation unit 320, a deep learning neural network 340, and a search unit 350. (MCTS).

In detail, the action model is a second high-stop plate having an action having a high activity function (Q) and a high confidence value (U) among branches not searched through MCTS in the current first high-stop plate state (S1) through the selection process (a). State (S1-2) is selected. The activity function (Q) is the average value of value values (V) calculated every time a corresponding branch is passed. The confidence value (U) is inversely proportional to the number of visits (N) passing through the corresponding branch and proportional to the movement probability value (p).

The motion model may be expanded to the third high stop board state (S1-2-1) in the selected action through the expansion and evaluation process (b) and calculate a movement probability value (p).

The motion model calculates the value (V) of the extended third high-stop board state (S1-2-1), and the activity function (Q), number of visits (N), and movement of branches passed through the backup process (c). A probability value (p) can be stored.

)을 출력할 수 있다. 동작 모델은 액션들 중 가장 높은 탐색 확률값(

)을 가지는 액션을 검출할 수 있다. The action model repeats the process of selection (a), expansion and evaluation (b), and backup (c) during the action preparation time, and creates a probability distribution using the number of visits (N) for each action, resulting in a search probability value (

) can be output. The action model has the highest search probability value among actions (

) can be detected.

Meanwhile, the state determination unit 310 of the motion model may receive a predetermined high stop plate state (S) as an input and output state determination information (C) for the input high stop plate state (S). Here, the state determination information (C) may be information that determines the activation state, which is a criterion for detecting an action that can be performed in a specific high-stop board state (S). That is, the activation state may determine at least one action that can be performed by the player in the specific high-stop state (S). A detailed description thereof follows the description of the operation model of FIGS. 6 and 7 .

In addition, the state determination unit 310 performs state determination learning to output state determination information (C) for a predetermined high-stop board state (S) based on the training data set provided by the game service providing server 200. can be performed A detailed description thereof follows the description of the operation model of FIGS. 4 to 9 .

On the other hand, the action determination unit 330 of the action model may determine and perform the optimal action (A), which is the best action for a predetermined high-stop board state (S), based on the MCTS, in conjunction with the MCTS module 390. there is. Here, the optimal action (A) may be a predetermined action determined to have the highest win rate and score in a predetermined high-stop board state (S).

In detail, the action decision unit 330 may include the high-stop board state (S) received from the game service providing server 200, the odds estimate value (W), the score estimate value (G) and/or the genealogy received from the simulation unit 320. Based on the achievement probability value (L) and the state determination information (C) received from the state determination unit 310, MCTS can be performed, and through this, the optimal action (A) for the high-stop state (S) can be determined. can be determined and carried out.

Specifically, the action determination unit 330 may determine which action to perform from among at least one possible action for each activation state according to an embodiment. In detail, the action determination unit 330 determines which of the at least one action possible in the corresponding activation state has the highest odds rate and You can determine whether it is an action that can have points.

For example, the action determination unit 330, when the activation state determined for a predetermined high-stop board state (S) is a hand state in which a hand is to be played from among my hands, actions possible in the corresponding hand state ( For example, if my hand has 3 cards, it is determined which of the actions to perform is the action with the highest winning rate and score (e.g., an action to play a first hand, an action to play a second hand, or an action to play a third hand) can do.

In addition, the action determining unit may be trained to determine an optimal action (A) for a predetermined high-stop board state (S) based on a training data set provided by the game service providing server 200 . A detailed description thereof follows the description of the operation model of FIGS. 4 to 9 .

- Game play server according to an embodiment - Operation model learning

4 is a flowchart for explaining a method for learning an operation model of the game play server 300 according to an embodiment of the present invention, and FIG. 5 shows an operation model of the game play server 300 according to an embodiment of the present invention. It is a conceptual diagram to explain how to learn.

4 and 5, the deep learning-based go-stop game service according to an embodiment of the present invention operates so that the game play server 300 performs an optimal action (A) for a specific go-stop board state (S). model can be trained. Specifically, the game play server 300 may train an action model to determine an activation state for a specific high-stop board state (S). In addition, the game play server 300 may train an operating model to determine an odds ratio estimation value (W) based on a specific high-stop board state (S). In addition, the game play server 300 may train an operating model to determine a score estimation value (G) based on a specific high-stop board state (S). In addition, the game play server 300 may train an operating model to determine a genealogy achievement probability value (L) based on a specific high-stop board state (S).

In detail, further referring to FIG. 4, the deep learning-based GoStop game service method according to an embodiment of the present invention includes a step (S101) of receiving a training data set by the game play server 300. can

In more detail, the game play server 300 may receive a training data set for learning a motion model in conjunction with the game service providing server 200 . Here, each training data of the training data set may be play record data (hereinafter, play data) in a high-stop game. The play data may include all information from a first action, which is information about a first action at the start of a competitive battle, to a final action at which the competitive game ends. That is, the plurality of play data may include action-related history information. In addition, the play data may include each GoStop version state (S) according to the order of actions performed by the GoStop game service. Here, the go-stop board state (S) is a set of data according to the game progress, including the state in which the go-stop hand is placed on the go-stop board, etc. hand, opponent player's eaten hand, floored hand (i.e. open hand), number of points a player must score to win, number of points opponent player must score to win, hand selected by player when in hand, opponent It may include a hand selected when the player is in a hand state, knock-inducing information, a hostage type of a bonus hand, the number of go runs, previous player information, and/or 9-fold blood usage information.

According to the embodiment, the game service providing server 200 obtains a plurality of play data in conjunction with the game service providing server 200, and input characteristics for the acquired play data (ie, in the embodiment, the high-stop version state (S ) data) extraction process.

In detail, the game service providing server 200 may further include an input feature extraction unit that performs an input feature extraction process on the play data. For example, the input feature extraction unit may have a neural network structure and may include a kind of encoder.

In addition, the game service providing server 200 may extract the above-described high-stop version state (S) data for the play data using the input feature extraction unit.

Thus, the game service providing server 200 may build a training data set for learning a motion model based on the play data obtained as above and the high-stop board state (S) information.

In addition, the deep learning-based GoStop game service method according to an embodiment of the present invention may include a step S103 in which the game play server 300 performs supervised learning based on the received training data set.

In detail, the game play server 300 may perform supervised learning on the motion model based on the received training data set. In more detail, the game play server 300 may supervise the state determination unit 310 of the motion model based on the received training data set.

In addition, the deep learning-based GoStop game service method according to an embodiment of the present invention may include a step (S105) of the game play server 300 performing supervised learning-based state judgment training.

6 is a conceptual diagram for explaining a method of performing state judgment training according to an embodiment of the present invention, and FIG. 7 is an example of a diagram showing a transition relationship between states according to an embodiment of the present invention.

Referring to FIG. 6 , the game play server 300 takes a predetermined high-stop board state (S) as an input and outputs state determination information (C) for the corresponding high-stop board state (S) of the action model. State determination training for learning the state determination unit 310 may be performed.

Here, the state determination information (C) may be information that determines the activation state, which is a criterion for detecting an action that can be performed in a specific high-stop board state (S). That is, the activation state may determine at least one action that can be performed by the player in the specific high-stop state (S).

Here, the activation states are HAND State, SHAKE State, SELECT State, SWITCH State, GOSTOP State, END State ), hand holding state (CHANCE_HAND State) and deck flipping state (CHANCE_FLIP State). This is to classify states in an optimized form in an incomplete information game.

At this time, the hand state (HAND State) is a state in which the player decides which hand to play out of his or her hand, and an action of playing one of the player's hands on the go-stop game ground (hereinafter referred to as the floor) may be possible. . For example, when a player has three hands, an action of playing a first hand, an action of playing a second hand, or an action of playing a third hand may be possible. That is, in the hand mode, my hand can be converted to a selected hand (public hand) played on the floor.

Further, the SHAKE State is a state for determining whether or not to perform shaking when shaking is possible according to the go-stop rule, and may be a state in which an action to perform shaking or an action not to perform shaking is possible.

In addition, in the selection state (SELECT State), one of the selected hand or deck included in the deck (hereinafter referred to as a deck hand) is overturned and placed on the floor. A state in which it is decided which floor tile to acquire when at least one of the tiles (hereinafter referred to as “floor tiles”) existing on the floor is acquired, and an action of selecting and acquiring at least one of at least one floor tile placed on the floor is possible. can be

In addition, the wine cup position selection state (SWITCH State) is a state in which the location of the wine cup is determined when the wine cup can fly for the first time in a state in which a wine cup (i.e., a bad hand in September among the high-stop hands) is acquired, and the cup is avoided. An action of placing a drinking cup in a group (ie, an action of using a drinking cup as a pair) or an action of placing a drinking cup in a group (ie, an action of using a drinking cup as a pair) may be in a state in which it is possible.

In addition, the GOSTOP State is a state in which it is determined whether to go or stop when a score can be declared to go or stop. ) and an action to stop may be in a possible state.

In addition, the game end state (END State) is a state in which the game ends when one player declares a stop or the game ends when there is no winner, and may be a state in which an action to end the game is possible.

In addition, the hand handing state (CHANCE_HAND State) is a state in which it is determined which hand is to be handed to the opposing player's hand, and at least one of the remaining private hands excluding the open hand in the corresponding GoStop game is placed in the opposing player's hand. It may be in a state in which an action to replace with is possible.

In addition, the deck flip state (CHANCE_FLIP State) is a state that determines which hand to put in my hand when a deck hand is overturned or a bonus hand is paid and a new hand is brought. It is a state in which the action of putting one of them into my hand is possible.

Meanwhile, the above-described activation states may have a mutual switching relationship as shown in FIG. 7 according to the go-stop game rule, a relationship between activation states or a context, and the like.

Returning again, the game play server 300 may further include a plurality of sub-layer modules that output to the state determining unit 310 whether or not each of a plurality of activation states possible in the GoStop game is activated. Here, the sub-layer module for each activation state may take a predetermined high stop plate state (S) as an input and provide as an output whether or not the corresponding activation state is activated.

In detail, the game play server 300 inputs data based on the high-stop board state (S) of the training data set (for example, the player's hand, the player's eaten hand, the opposing player's eaten hand, and the floor laid (i.e., open hand), the number of points a player must refresh to win, the number of points an opponent must refresh to win, a hand selected by a player while in hand, a hand selected by opponent player in hand, trigger information, bonus hand Hostage type, go execution count, first player information and/or 9th blood usage information, etc.) as input, and based on this, a state determination unit to output activation status for each possible activation state in the GoStop game ( 310) can be learned.

In addition, the game play server 300 may teach the state determination unit 310 to output state determination information (C) for the input high-stop board state (S) based on whether each activation state is activated. That is, the game play server 300 determines that the state determination unit 310 is active among a plurality of activation states, that is, detects the active state facing the player during the current match, and based on the detected activation state State determination information (C) for the corresponding high stop plate state (S) can be output.

Therefore, an apparatus for providing a deep learning-based incomplete information game service according to an embodiment of the present invention determines an activation state for determining at least one action that a player can perform based on a current play state of an incomplete information game such as GoStop. can

In addition, the deep learning-based GoStop game service method according to an embodiment of the present invention may include a step of the game play server 300 performing a roll-out based on the received training data set (S107). .

8 is a conceptual diagram illustrating roll-out based training according to an embodiment of the present invention.

In detail, referring to FIG. 8 , the game play server 300 may learn a motion model based on the received training data set. In addition, the game play server 300 may perform MCTS-based self-play using the learned motion model.

In detail, the game play server 300 may train the MCTS module 390 including the simulation unit 320, the deep learning neural network 340, and the search unit 350 based on the received training data set.

In addition, the game play server 300 may provide an odds ratio estimation value (W) and/or a score estimation value (G) based on an operation model including the learned MCTS module 390 .

9 is an example of a diagram for explaining a method of estimating odds and scores according to an embodiment of the present invention.

Referring to FIG. 9 , the deep learning-based go-stop game service method according to an embodiment of the present invention may include a step in which the game play server 300 performs roll-out-based win rate estimation training (S109).

In detail, the game play server 300 may operate the MCTS module 390 to perform rollout based on the received training data set. In addition, the game play server 300 may perform win rate estimation training for learning an action model to estimate a win rate (ie, win probability) for possible actions in the corresponding high-stop board state (S).

That is, the game play server 300 receives a predetermined high-stop board state (S) as an input and outputs an odds ratio estimation value (W), which is information obtained by estimating the win rate by actions possible in the input high-stop board state (S). A motion model can be trained to do so.

Specifically, the game play server 300 may learn the Policy net and Value net of the motion model based on the received training data set. In addition, the game play server 300 may determine a winner according to a series of actions performed based on a predetermined high-stop board state S by using the learned action model.

At this time, the game play server 300, if the go (go) or stop (stop) is not declared, the player who performs more roll-out and reaches the go (go) or stop (stop) first in the future is the winner can be judged by

Alternatively, when the game play server 300 has declared go, the player who further rolls out and reaches a go or stop first in the future may be determined as the winner. Alternatively, when the game play server 300 is in a state in which a stop is declared, the player who declared the corresponding stop may be determined as the winner.

Therefore, the deep learning-based incomplete information game service providing apparatus according to an embodiment of the present invention can provide a clear criterion for estimating a winner in an incomplete information game such as GoStop.

In addition, the game play server 300 may estimate the winning rate based on the number of roll-outs performed while determining the winner as described above (ie, the total number of roll-outs) and the number of times determined as the winner in each roll-out. .

In detail, in an embodiment, the game play server 300 may calculate a ratio between the total number of rollouts and the number of times determined as a winner in each rollout (ie, the number of times determined as a winner in rollouts/the total number of rollouts). there is. In addition, the game play server 300 may determine an odds ratio estimation value (W) according to a series of actions performed based on the corresponding go-stop board state (S) based on the calculated ratio. At this time, the odds ratio estimation value (W) may have a range of 0 to 1.

That is, the game play server 300 receives a predetermined high-stop board state (S) as an input in the above manner, and an odds ratio estimation value ( A motion model can be trained to output W).

Further, referring to FIG. 9 , the deep learning-based go-stop game service method according to an embodiment of the present invention may include a step (S111) of the game play server 300 performing roll-out-based score estimation training. have.

In detail, the game play server 300 may operate the MCTS module 390 to perform rollout based on the received training data set. In addition, the game play server 300 may perform score estimation training for learning an action model to estimate a score obtainable for each possible action in the corresponding high-stop board state (S).

That is, the game play server 300 receives a predetermined high-stop board state (S) as an input and outputs a score estimation value (G), which is information obtained by estimating scores by actions possible in the input high-stop board state (S). A motion model can be trained to do so.

In more detail, the game play server 300 may determine a score according to a series of actions performed based on a predetermined high-stop board state S by performing roll-out based on the received training data set.

At this time, when the game play server 300 has not declared a go or stop, the score that can be obtained by declaring a stop in the future after further roll-out is a score estimate value (G ) can be determined.

Alternatively, the game play server 300 may determine a score that can be obtained in the future when roll-out is further performed as the score estimation value (G) when the game play server 300 has declared go. Alternatively, when the game play server 300 is in a state in which a stop is declared, a score that can be obtained at the time when the stop is declared may be determined as the score estimation value (G).

Therefore, the deep learning-based incomplete information game service providing apparatus according to an embodiment of the present invention can provide a clear criterion for estimating scores in incomplete information games such as GoStop.

However, in general, the score in the GoStop game has a larger range (eg, '-1000 to 1000') compared to the winning rate having a range of '0 to 1'. Therefore, the game play server 300 may perform score estimation training using a score clipping method or a regression to classification method for more effective score estimation training. Here, the score clipping technique may be a method of limiting the range of the score estimation value G to a predetermined range (eg, '-200 to 200') to perform the above-described roll-out-based score estimation training. Also, the regression to classification technique may be a method of performing score estimation training through roll-out based on a classification model rather than a regression model. In detail, the regression to classification technique expresses a preset possible score estimate (G) in a one-hot vector method (eg, '[-200, -199, ..., 199, 200] ' etc.), obtaining a probability distribution for each score estimate value (G) expressed, and selecting the score estimate value (G) having the largest probability value, thereby performing the rollout-based score estimation training. .

Therefore, the deep learning-based incomplete information game service providing apparatus according to an embodiment of the present invention can more efficiently and quickly perform score estimation training targeting a wide range.

In summary, the game play server 300 can perform rollout based on a predetermined training data set in the above manner to perform odds estimation training and score estimation training, and through this, to a predetermined high-stop board state (S) An operating model can be trained to output an odds estimate (W) and a score estimate (G) for

Therefore, the deep learning-based incomplete information game service providing apparatus according to an embodiment of the present invention can estimate the winning rate and score by a series of actions performed based on a predetermined play state of an incomplete information game such as GoStop. there is.

In addition, the deep learning-based GoStop game service method according to an embodiment of the present invention may include a step (S113) of the game play server 300 performing genealogy achievement probability training.

Here, the genealogy may mean a type of collection of hands promised in advance for a drug that gives a privilege to obtain a special number of points in the go-stop game. In the embodiment, the genealogy may include Godori, Cheongdan, Hongdan, and/or Chodan according to the GoStop rule.

In detail, further referring to FIG. 8 , the game play server 300 may operate the MCTS module 390 to perform rollout based on the received training data set. In addition, the game play server 300 trains an action model to predict the probability of matching genealogies such as Godori, Cheongdan, Hongdan and / or Chodan when the rollout match for the corresponding GoStop board state (S) is completed. Genealogical achievement probability training can be performed.

That is, the game play server 300 receives a predetermined high-stop board state (S) as an input, and as a result of roll-out based on the input high-stop board state (S), the predetermined genealogy (godori, cheongdan, hongdan, and/or first stage) etc.) can be trained to output the genealogy achievement probability value (L), which is information that predicts the probability of achieving (achievement).

More specifically, upon receiving a predetermined high-stop board state (S), the game play server 300 performs a roll-out based on the input high-stop board state (S), and a series of actions possible from the corresponding high-stop board state (S). As a result of the above, when the match is completed, the probability of achieving Godori, the probability of achieving Cheongdan, the probability of achieving Hongdan, and / or the probability of achieving the first stage are calculated and provided as output. The model can be trained.

In addition, the game play server 300 may utilize the genealogy achievement probability value (L) predicted as above when estimating the above-described odds estimate value (W) and score estimate value (G). For example, when the game play server 300 determines that there is a high probability of obtaining a special score in the corresponding Go-Stop match based on the genealogy achievement probability value (L), a predetermined odds ratio estimation value (W) and/or a predetermined score estimation value (G) can increase

Therefore, an apparatus for providing a deep learning-based incomplete information game service according to an embodiment of the present invention is a genealogy according to a corresponding game rule in an incomplete information game such as Go-Stop (in an embodiment, Godori, Cheongdan, Hongdan, and/or Chodan, etc.) It is possible to improve the performance of an artificial intelligence computer playing an imperfect information game by predicting the probability of achieving

- Game play server according to another embodiment - Deep learning-based Go-Stop service provided

10 is a flow chart illustrating a method for providing a deep learning-based incomplete information game service by an operation model of a game play server 300 according to an embodiment of the present invention, and FIG. It is a conceptual diagram for explaining how the operation model of the play server 300 provides a deep learning-based incomplete information game service.

10 and 11, in the deep learning-based GoStop game service according to an embodiment of the present invention, the game play server 300 performs an optimal action (A) for a specific GoStop board state (S) and GoStop can play the game. Here, the optimal action (A) may be a predetermined action determined to have the highest win rate and score in a predetermined high-stop board state (S). Specifically, the game play server 300 may acquire a specific high-stop board state (S). In addition, the game play server 300 may obtain state determination information (C) for determining the activation state for the specific high-stop board state (S), and a win rate estimation value (W) based on the specific high-stop board state (S) and/or It is possible to obtain estimation information including a score estimation value (G) and a genealogy achievement probability value (L) based on the specific high-stop board state (S). In addition, the game play server 300 determines the optimal action (A) for the specific high-stop board state (S) by using the obtained specific high-stop board state (S), state determination information (C) and/or estimation information. and can be performed.

In detail, further referring to FIG. 10, the deep learning-based go-stop game service method according to an embodiment of the present invention may include a step (S201) of receiving, by the game play server 300, a go-stop board state (S). .

In more detail, the game play server 300 may receive the current GoStop board state S for the GoStop game currently in progress in conjunction with the game service providing server 200 .

Here, the current high-stop board state (S) is a set of data according to the current game progress, including the state in which the high-stop hand is placed on the high-stop board, etc. It may contain public information, including open hand), points that need to be updated to win, my selected hand information, ditch trigger information, bonus hand hostage type, go execution count, and/or 9-bit blood usage information. there is. The current go-stop board state (S) may not include private information that is not disclosed while playing the go-stop game, such as the opponent's hand, the opponent's lost hand, and/or the score that the opponent needs to update to win.

In addition, the deep learning-based GoStop game service method according to an embodiment of the present invention includes the step of obtaining, by the game play server 300, state determination information (C) for the received GoStop board state (S) (S203). can do.

Here, the state determination information (C) may be information that determines the activation state, which is a criterion for detecting an action that can be performed in a specific high-stop board state (S). That is, the activation state may determine at least one action that can be performed by the player in the specific high-stop state (S). Here, the activation states are HAND State, SHAKE State, SELECT State, SWITCH State, GOSTOP State, END State ), hand holding state (CHANCE_HAND State) and deck flipping state (CHANCE_FLIP State). This is to classify states in an optimized form in an incomplete information game.

In detail, the game play server 300 may obtain state determination information (C) for the current high-stop board state (S) by using an operation model. Specifically, the game play server 300 takes the current high-stop board state (S) as an input based on the state determination unit 310 of the action model, and state determination information (C) for the current high-stop board state (S). ) can be performed as an output. At this time, the state determination unit 310 may receive a predetermined high stop plate state (S) as an input and may be trained to determine state to output state determination information (C) for the corresponding high stop plate state (S). A detailed description thereof follows the description of the operation model of FIGS. 6 and 7 .

In addition, the game play server 300 may obtain state determination information (C) for the current high-stop board state (S) through the performed deep learning.

For example, the game play server 300 is a hand state (HAND State) in which the activation state for the current go-stop board state (S) determines which hand to play out of my hand, and any one of my hands is a go-stop board (ie, , the bottom) can obtain state determination information (C) providing that the action is possible.

In addition, the deep learning-based go-stop game service method according to an embodiment of the present invention may include a step (S205) of acquiring estimation information on the received go-stop board state (S) by the game play server 300.

Here, the estimation information may be information including an odds ratio estimation value (W) and/or a score estimation value (G) estimated based on deep learning for a predetermined high-stop board state (S). At this time, the odds ratio estimation value (W) may be information obtained by estimating the odds ratio by possible actions in a predetermined high-stop board state (S). The score estimation value (G) may be information obtained by estimating scores by possible actions in a predetermined high-stop board state (S).

In detail, the game play server 300 may obtain estimation information on the current high-stop board state (S) using an operation model. In more detail, the game play server 300 takes the current high-stop board state (S) as an input based on the MCTS module 390 of the action model and obtains an odds estimation value (W) for the current high-stop board state (S) and/or Alternatively, deep learning with a score estimation value (G) as an output may be performed. At this time, the MCTS module 390 may be trained to receive a predetermined high stop plate state (S) as an input and output estimation information for the corresponding high stop plate state (S). A detailed description thereof follows the description of the operation model of FIGS. 8 and 9 .

In addition, the game play server 300 may obtain estimation information on the current high-stop board state (S) through the deep learning performed above.

In addition, in the deep learning-based GoStop game service method according to an embodiment of the present invention, the game play server 300 takes an action according to the obtained GoStop board state (S), state determination information (C), and estimation information. It may include performing step (S207).

In detail, the game play server 300 may perform an optimal action (A) based on the current high-stop board state (S), state determination information (C), and estimation information using an action model. Here, the optimal action (A) may be a predetermined action determined to have the highest win rate and score in a predetermined high-stop board state (S).

Specifically, the game play server 300 includes an odds estimate value (W) and/or a score estimate value (G) obtained for the current high-stop board state (S) based on the action decision unit 330 of the motion model. It is possible to perform deep learning with the estimation information as an input and the optimal action (A) for the current high-stop board state (S) as an output.

In more detail, the game play server 300 determines the current high-stop board state (S), the state determination information (C) obtained for the current high-stop board state (S), and the current high-stop board state (S). Based on the estimated information including the odds estimate value (W) and/or the score estimate value (G) obtained for can At this time, the action determination unit 330 may be trained to output the optimal action (A) for the high-stop board state (S) with the estimation information as described above as an input. A detailed description thereof follows the description of the operation model of FIGS. 4 to 9 .

In addition, the game play server 300 may determine the optimal action (A) for the current high-stop board state (S) through the performed deep learning and perform it.

At this time, the game play server 300 may determine the optimal action (A) based on a Monte Carlo Tree Search (MCTS) algorithm.

In detail, the game play server 300 uses an operation model in which the state determination unit 310, the MCTS module 390, and the action determination unit 330 interoperate, and the current high-stop board state (S) and the current MCTS based on the state determination information (C) and estimation information obtained for the high-stop board state (S) may be performed. In addition, the game play server 300 provides an activation state for the current high-stop board state (S) based on the performed MCTS, a win rate and a score according to actions that can be performed in the active state, and the current high-stop board state ( Considering the genealogy achievement probability according to a series of actions that can be performed based on S), the optimal action (A) that is determined to have the highest rate and score in the current high-stop board state (S) can be determined. . Also, the game play server 300 may perform an operation according to the determined optimal action (A).

Therefore, an apparatus for providing a deep learning-based incomplete information game service according to another embodiment of the present invention can achieve an activation state for a predetermined play state of an incomplete information game such as GoStop, and a win rate and/or score according to a predetermined action. The optimal action (A) for winning the game can be performed by considering the probability.

On the other hand, in general, the MCTS for an incomplete information game (in an embodiment, a go-stop game) in which some information necessary for game progress (in an embodiment, an opponent's hand and a deck hand, etc.) is kept private, is unknown information When estimating an action that can be triggered based on (in the embodiment, an action of an opponent player), the action is based on a random probability.

In detail, in the case of estimating the action of the opposing player (ie, the action of the chance node) when performing the MCTS for the go-stop game in the existing random method, one action among all possible actions for the opposing player is determined. MCTS may be performed by randomly selecting an action according to an algorithm (eg, random policy, etc.) and considering that the randomly selected action of the opposing player is performed by the opposing player.

12 is an example of a diagram for explaining a method of estimating an action of an opponent player in MCTS of an incomplete information game service according to an embodiment of the present invention.

Illustratively, referring to (a) of FIG. 12, when performing MCTS for a go-stop game in the conventional random method, when the opponent's hand is 2 and the number of deck hands is 3, the action of the opponent is estimated In this case, one randomly selected action among all possible actions for the opposing player. For example, the opposing player is in the HAND state, and all private hands that have not yet been revealed in the corresponding GoStop game (i.e., the opposing player's hand) 2 cards and 5 cards including 3 deck hands), the opponent player selects one hand selected at random probability, and the action on the floor is determined as the action of the opponent player, and the simulation can be performed .

However, since the action of the opponent player is not substantially determined in a random manner as described above, performance reliability of the MCTS may deteriorate if the MCTS is performed based on the conventional random method as described above.

Therefore, the game play server 300 according to another embodiment of the present invention provides MCTS for a GoStop game in which some information related to the game progress (in the embodiment, the opponent's hand and deck hand, etc.) is kept private. When performing, a deep learning-based opponent action estimation simulation (hereinafter referred to as opponent action estimation simulation) may be performed to estimate an opponent player's action based on unknown information based on deep learning.

That is, the game play server 300 according to another embodiment of the present invention may perform an opponent action estimation simulation in which an action of an opponent player is estimated using deep learning in MCTS for an incomplete information game such as GoStop.

In detail, the game play server 300 may first determine the activation state of the opponent player when estimating the action of the opponent player when performing the MCTS in the GoStop game service. For example, the game play server 300 may perform a plurality of activation states (in the embodiment, a hand state (HAND State), a shake state (SHAKE State), and a selection state (SELECT State)) possible in the go-stop game according to the embodiment. , SWITCH State, GOSTOP State, END State, CHANCE_HAND State, and CHANCE_FLIP State) Activation status can be determined.

Also, referring to (b) of FIG. 12 , in an embodiment, the game play server 300 may calculate the number of hands of the opponent player when the determined activation state of the opponent player is a hand state (HAND State). For example, when the opponent player has two hands, the game play server 300 may calculate the number of hands of the opponent player as two. At this time, due to the characteristics of the Go-Stop game, the opponent player's hand may be a private hand in which information of the corresponding hand is not disclosed.

In addition, the game play server 300 may detect at least one closed hand excluding the open hand from all hands of the GoStop game, that is, one or more of the other player's hand hand and deck hand. At this time, the game play server 300 may obtain information on a hand corresponding to each of the detected closed hands (that is, information indicating which hand it is) by matching it to each of the corresponding closed hands. That is, due to the nature of the Go-Stop game, which is an incomplete information game, it is possible to determine which paddles are private paddles that have not yet been released in the corresponding Go-Stop game, but which of the corresponding non-public paddles belongs to the opponent's hand may be private. .

In addition, the game play server 300 may randomly extract as many as the calculated number of hand cards of the opponent player from among the detected at least one closed hand. For example, when the number of hands of the opposing player is calculated as two, the game play server 300 may randomly select and extract two hands from among the detected at least one closed hand.

In addition, the game play server 300 may detect the type of at least one extracted hand based on information on a hand matched to the extracted undisclosed hand (hereinafter referred to as extracted hand). For example, the game play server 300 determines which hand each of the first and second extraction hands are based on the information of the randomly extracted first and second extraction hands and matched information. can do.

In addition, the game play server 300 may determine, through deep learning, which extracted hand is the best hand that can have a higher win rate and score among at least one extracted hand, which hand is disclosed as above.

In detail, the game play server 300 may perform self-play in which an opponent player performs an action based on each of the extracted hands based on the corresponding MCTS. For example, the game play server 300 may perform a self-play in which an action of releasing a first extraction hand is an action of an opponent player, and a self-play in which an action of an action of releasing a second extraction hand is an action of an opponent player. .

In addition, the game play server 300 may obtain an estimated win rate value (W) and/or a score estimation value (G) for each extracted hand as a result of performing the self-play for each extracted hand.

In addition, the game play server 300 determines which extraction tile to apply to the action of the opponent player from among the corresponding at least one extraction tile based on the obtained win rate estimation value (W) and/or score estimation value (G) for each extraction hand. can In an embodiment, the game play server 300 may determine a extracted hand having a higher win rate estimation value (W) and/or score estimation value (G) as a extracted hand to be applied to the action of the opponent player as a result of self-play. At this time, according to the embodiment, the game play server 300 may improve the accuracy by applying the softmax temperature method when executing the MCTS algorithm.

Thus, the game play server 300 determines, through deep learning, the extracted hand that is determined to be the best hand that can have a higher win rate and score among at least one or more extracted hands extracted as many as the number of hands of the opposing player in the corresponding go-stop game. can

In addition, the game play server 300 may determine the action of releasing the extracted hand determined as the best hand as the action of the opposing player.

Therefore, an apparatus for providing a game service based on deep learning based on incomplete information according to another embodiment of the present invention is a go-stop game in which some information related to game progress (in an embodiment, an opponent's hand and deck hand, etc.) is confidentially processed. When MCTS is performed for an incomplete information game such as , an action of an opponent player based on unknown information can be estimated based on deep learning.

On the other hand, in general, the above-described MCTS mainly uses a UCT method using a policy network called PUCT and a value network in order to efficiently search for nodes. That is, the existing MCTS generally uses the UCT method, which is a method that is highly dependent on the odds ratio estimation value (W). However, in games such as Go-Stop, factors such as scores in addition to the win rate act as very important indicators.

Thus, the game play server 300 according to another embodiment of the present invention intends to provide a UCT of a new method that performs MCTS by considering not only the odds estimation value (W) but also the score estimation value (G).

In detail, the game play server 300 may perform MCTS outputting an optimal action (A) based on an Upper Confidence Bound 1 applied to Trees (UCT) algorithm. Here, the UCT algorithm may be a known algorithm as a combination of MCTS and UCB1 (Upper Confidence Bound 1), and follows the following formula.

[Formula 1]

In Equation 1, w(winrate) may be an odds ratio estimation value (W), b may represent a child node, N may represent the number of visits, and C may be set to 1 as a weight.

In this case, the game play server 300 may implement a G-UCT algorithm modified from the UCT algorithm by further applying a score estimation value (G) index to the above existing UCT algorithm. The G-UCT algorithm according to another embodiment of the present invention follows the following formula.

[Formula 2]

Specifically, the game play server 300 may add a score term according to another embodiment of the present invention to the existing UCT algorithm (Equation 1). That is, the score term follows the following formula.

[Formula 3]

In Equations 2 and 3, the score of the score term may mean a score estimation value (G), const may mean a predetermined constant, and n may mean the number of simulations.

That is, the game play server 300 may perform a simulation of a predetermined high-stop board state (S) by using a new type of algorithm in which the above score terms are added to the existing UCT.

In this case, the sigmoid function included in the score term may be used to implement normalization of the odds ratio estimation value (W) and the score estimation value (G) of the G-UCT according to the embodiment. In detail, generally, the distribution of scores in a go-stop game may have a wider range than the odds. For example, while the odds estimation value (W) is usually output as a predetermined value between '0 and 1', the score estimate (G) can be theoretically output as a predetermined value between '-1000 and 1000' .

13 is an example of a diagram showing a sigmoid function used in the G-UCT algorithm according to an embodiment of the present invention.

Thus, referring to FIG. 13, the game play server 300 according to another embodiment of the present invention normalizes between the score estimate value (G) and the odds estimate value (W) when performing G-UCT, A sigmoid function may be included in the score term to which the score estimation value (G) is applied in the G-UCT. That is, the game play server 300 may implement normalization that performs scaling by including a sigmoid function in the score term to adjust the distribution of the score estimation value (G) between '0 and 1'.

In addition, at this time, when the game play server 300 implements the G-UCT algorithm (Equation 2) by applying a score term including a sigmoid function, a score estimation value (G) that is larger or smaller than a predetermined threshold value ( For example, in order to prevent a situation in which a score estimate value greater than 4 or less than -4 converges to 0 (or a value close to 0) or 1 (or a value close to 1), the range of the score estimate value (G) A predetermined constant (const) that adjusts within a predetermined range may be further included in the score term.

Thus, the game play server 300 may implement the G-UCT algorithm (Equation 2) in which the score term configured as above is added to the existing UCT algorithm (Equation 1). In addition, the game play server 300 performs deep learning based on the implemented G-UCT algorithm (Equation 2) to consider in more detail not only the win rate according to the predetermined action in the go-stop game, but also the obtainable score to set the current go-stop The optimal action (A) for the board state (S) can be determined.

Therefore, an apparatus for providing a deep learning-based incomplete information game service according to another embodiment of the present invention is an improved UCT algorithm (ie, the G-UCT algorithm according to the embodiment) in a form more optimized for an incomplete information game such as a go-stop game. can be used to determine the best action according to the current play state.

As described above, the deep learning-based incomplete information game service providing method and apparatus according to an embodiment of the present invention provide state determination information (C) based on the current play state in an incomplete information game, an odds estimate (W), and/or a score estimate ( At least one of G) can be used to perform the best action.

In addition, a method and apparatus for providing a game service based on deep learning based on incomplete information according to an embodiment of the present invention discloses an opponent player in an incomplete information game in which some information such as an opponent's hand and a hand included in a deck is not disclosed. The action of can be estimated based on deep learning according to a predetermined algorithm rather than a random random method.

In addition, the deep learning-based incomplete information game service providing method and apparatus according to an embodiment of the present invention can determine the best action according to the current play state using an improved UCT algorithm in a form more optimized for incomplete information games. there is.

In addition, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. medium), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. A hardware device may be modified with one or more software modules to perform processing according to the present invention and vice versa.

Specific implementations described in the present invention are examples and do not limit the scope of the present invention in any way. For brevity of the specification, description of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection of lines or connecting members between the components shown in the drawings are examples of functional connections and / or physical or circuit connections, which can be replaced in actual devices or additional various functional connections, physical connection, or circuit connections. In addition, if there is no specific reference such as “essential” or “important”, it may not be a component necessarily required for the application of the present invention.

In addition, the detailed description of the present invention described has been described with reference to preferred embodiments of the present invention, but those skilled in the art or those having ordinary knowledge in the art will find the spirit of the present invention described in the claims to be described later. And it will be understood that the present invention can be variously modified and changed without departing from the technical scope. Therefore, the technical scope of the present invention is not limited to the contents described in the detailed description of the specification, but should be defined by the claims.

Claims (10)

A method of providing a deep learning-based incomplete information game service through a user terminal in a game play server,
Obtaining a training data set including a high-stop plate state for each of a plurality of actions; and
Learning a deep learning model of the incomplete information service based on the obtained training data set;
The step of learning the deep learning model,
performing state judgment training based on the training data set;
Performing odds estimation training based on the training data set;
performing score estimation training based on the training data set;
Including at least one step of performing genealogy achievement probability training based on the training data set
Imperfect information How to provide game services. According to claim 1,
The state judgment training,
Learning the deep learning model to output state determination information, which is information obtained by determining at least one action that can be performed in the first high stop board state as an input and the input first high stop board state
Imperfect information How to provide game services. According to claim 2,
The state determination information,
Information that determines the activation state, which is a reference element for detecting at least one action that can be performed in the first high-stop board state
Imperfect information How to provide game services. According to claim 3,
The activation state is
HAND State, SHAKE State, SELECT State, SWITCH State, GOSTOP State, END State, hand holding state (CHANCE_HAND State) and deck flip state (CHANCE_FLIP State), including at least one state
Imperfect information How to provide game services. According to claim 1,
The odds estimation training,
Based on the roll-out using the training data set, the first high-stop board state is taken as an input, and the win rate estimation value, which is information estimating the win rate by actions possible in the input first high-stop board state, is output Training the deep learning model to
Imperfect information How to provide game services. According to claim 1,
The score estimation training,
Based on the roll-out using the training data set, the first high-stop board state is taken as an input, and a score estimation value, which is information obtained by estimating scores by possible actions in the input first high-stop board state, is output. Training the deep learning model to
Imperfect information How to provide game services. According to claim 1,
The genealogy achievement probability training,
Training the deep learning model to output a genealogy achievement probability value, which is information predicting the probability of achieving a predetermined genealogy at the end of roll-out based on the first high-stop board state as an input and the input first high-stop board state
Imperfect information How to provide game services. According to claim 1,
The high-stop plate state,
Player's hand, player's lost hand, opponent's lost hand, floored hand, number of points a player needs to update to win, number of points opponent needs to update to win, player's choice hand, opponent's choice Hand, bonus hand, hostage type, go execution count, previous player information, ditch trigger information, and at least part of 9 blood usage information
Imperfect information How to provide game services. a communication unit for receiving a training data set;
memory for storing deep learning models; and
A processor for performing at least one of state judgment training, odds estimation training, score estimation training, and genealogy achievement probability training for the deep learning model based on the received training data set.
An incomplete information game service providing device. According to claim 9,
the processor,
By performing the training, the deep learning model learns to output at least one of state judgment information, odds estimate, score estimate, and genealogy achievement probability value for the first high-stop board state
An incomplete information game service providing device.

Images