CN107145948A - A NPC Control Method Based on Multi-agent Technology - Google Patents

Abstract

The invention discloses a kind of NPC control methods based on multi-agent Technology, it is related to the technical fields such as artificial intelligence, computer game, NPC of the prior art is solved without automated reasoning, adjust automatically behavior, need frequently to change NPC scripts according to the experience of game developer, cause the problems such as human cost is paid wages.The present invention includes defining an automated reasoning model for each NPC --- Interactive Dynamic influence figure, the parameter of defining interactive Dynamic Influence Diagrams according to the characteristics of NPC itself and the characteristics of scene；According to the Interactive Dynamic influence figure for defining parameter, NPC behavior script is constructed；NPC records the implementing result of each action in the behavior script with execution NPC in player's interaction；According to the implementing result of each action of record, updating Interactive Dynamic influences the parameter of figure, reconfigures NPC behavior script applications in new player's interaction.Artificial intelligence approach is used in the online RPG of many people the behavior for automatically controlling non-player role by the present invention.

Description

A NPC Control Method Based on Multi-agent Technology

technical field

An NPC control method based on multi-agent technology uses artificial intelligence methods to automatically control the behavior of non-player characters in multiplayer online role-playing games, and involves technical fields such as artificial intelligence and computer games.

Background technique

A multiplayer online role-playing game is a large-scale online game in which players play a fictional character and control the character's activities, communicate and interact with a large number of characters played by other players in the game, and cooperate to defeat monsters and complete tasks. Different from the individual confrontation between man and machine in the stand-alone game mode, online games allow multiple players to cooperate and compete in real time, and the game content and fun are greatly improved. An excellent multiplayer online role-playing game can attract a large number of players. By providing high-quality game services, it can enrich the spiritual life of players and make people live happier. This has also given birth to a new industry, which has produced huge economic benefits.

In addition to player characters, there are also a large number of non-player characters in the game, namely NPC (Non-player characters). In addition to a class of NPCs that provide game functions, such as item trading and task guide characters, there are a large number of monster NPCs in the game. These NPCs are the core of various game tasks. Players need to defeat a large number of various monsters to complete tasks. By making the behavior of NPCs more intelligent, it is an important content to provide excellent online game services to reasonably adjust the difficulty of the game, taking into account fairness, challenge and fun. Major game manufacturers have invested a lot of manpower and material resources in each game to try to achieve this goal.

Existing NPC behaviors are mostly controlled through pre-programmed scripts. Due to the limitation of development resources such as manpower and time, game developers cannot prepare a method for each NPC in each scene to deal with every possible situation. In order to gradually increase the challenge of the game, developers will greatly enhance the physical attributes of NPCs (such as attack power, defense power, etc.) to increase the difficulty of game players. Therefore, NPCs often appear to act dull but extremely powerful, and it is detrimental to the fairness of the game. An important flaw of pre-programmed scripts is that NPCs only have the ability to deal with players passively, but have no active reasoning ability. That is to say, the player can infer its fixed behavior pattern through multiple trials, so as to find a strategy to defeat the seemingly powerful NPC. More importantly, this strategy will spread quickly among players, making the game less challenging and less interesting. For this reason, game developers will frequently modify NPC scripts based on game logs to ensure the fun of the game. Because frequent modification of NPC scripts is limited by labor costs, only important monsters will be modified for NPC scripts. How to effectively and quantitatively adjust the script according to the game player's strategy also depends on the experience of the game developer.

Contents of the invention

The present invention provides a kind of NPC control method based on multi-agent technology for above-mentioned inadequacy, solves the problem that NPC in the prior art does not have automatic reasoning, automatic adjustment behavior, needs to revise NPC script frequently according to the experience of game developer , resulting in waste of labor costs and other issues.

In order to achieve the above object, the technical scheme adopted in the present invention is as follows:

A kind of NPC control method based on multi-agent technology, is characterized in that, comprises the following steps:

Step (1), define an automatic reasoning model-interactive dynamic influence diagram for each NPC, define the parameters of the interactive dynamic influence diagram according to the characteristics of the NPC itself and the characteristics of the scene;

Step (2), according to the parameter of interactive dynamic influence graph, constructs the behavior script of NPC;

Step (3), the NPC executes the behavior script of the NPC during the interaction with the player, and records the execution result of each action, wherein the action refers to the behavior script;

Step (4), according to the execution result of each action recorded in step (3), update the parameters of the interactive dynamic influence graph, reconstruct the behavior script of the NPC and apply it to the new player interaction.

Further, in the step (1), the specific steps of defining the parameters of the interactive dynamic influence graph according to the characteristics of the NPC itself and the characteristics of the scene are as follows:

Step (11), in the interactive dynamic influence diagram, define an action set according to the executable actions of the NPC Defines a set of actions based on the executable actions of the player character

Step (12), according to the nature of the scene where the NPC is located, the position and attributes of the NPC and the player role define the state collection S ^* ;

Step (13), according to step (11) and step (12) define from time t to time t+1, state s ^t ∈ S ^t is acted by NPC and player character actions The conditional probability function of the possibility of transitioning to state s ^t+1 ∈ S ^t+1 , the state transition function

Step (14), according to the behavior style and preference of the NPC character, define that at time t, the NPC passes the action and player character actions The utility function for transitioning from state s ^t ∈ S ^t to state s ^t+1 ∈ S ^t+1

Step (15), according to the existing empirical knowledge, initialize the state transition function and utility function parameters;

Step (16), according to the priori, the interactive dynamic influence diagram of the NPC contains several strategy descriptions describing the behavior of the player character Description of each policy Indicates the conditional probability that the player character will perform different actions in different states That is, the player character performs an action in state s ^t The probability Describe the N strategies A collection of stored player behavior patterns details as follows:

Further, the concrete steps of described step (2) are as follows:

Step (21), according to the interactive dynamic influence diagram constructed in step (1) and the required NPC behavior script length, the interactive dynamic influence diagram is expanded into an interactive dynamic influence diagram comprising T-step reasoning;

Step (22), use the classic dynamic programming algorithm to solve the interactive dynamic influence diagram that contains T-step reasoning obtained in step (21), maximize the weighted sum of the immediate expected utility and the long-term expected utility of the NPC, and find the one that can The strategy for maximizing the expected utility of the NPC role is to find the actions that can maximize the NPC's actions for various situations at each moment The conditional probability of The formula for maximizing the weighted sum of immediate expected utility and long-term expected utility of NPC is as follows:

in, is the immediate expected utility, is the long-term expected utility, λ is the discount factor, which is used to weaken the impact of the long-term utility on the current action, ER _i is the total utility value, that is, the weighted sum of the immediate expected utility and the long-term expected utility of the NPC;

Step (23), converting the strategy of maximizing NPC expected utility obtained in step (22) into a behavior script format compatible with the current game.

Further, the concrete steps of described step (3) are as follows:

Step (31), NPC adopts NPC actions for different actions of different player characters in the process of interacting with player characters Promptly adopt the behavioral script that obtains in the step (23);

Step (32), record every NPC action Result of execution: at each moment, including the actions performed by the player character Actions performed by NPCs The change to the state, that is, the utility value corresponding to the action under the transition from state s ^t to state s ^t+1 and state s ^t to state s ^t+1

Further, the concrete steps of described step (4) are as follows:

Step (41), record the results of each action execution according to step (3), and count the actions performed by the player role when the NPC is in each state s ^t ∈ S ^t The frequency of , normalize the frequency to get the conditional probability of the behavior frequency in each state That is, it can be regarded as the NPC's strategy description based on the current player's behavior to the NPC By integrating with player behavior patterns Compare to find the most similar player strategy description which is of and of is most similar to:

Strategy description of player behavior based on statistics Updated interactive dynamic influence graph for most similar player strategy descriptions Conditional probability of taking various actions in various situations get the updated policy description The specific formula is as follows:

in, is a policy description of player behavior in existing interactive dynamic influence graphs, is the policy description obtained from the interaction data, and α is the policy update rate, indicating the speed at which the current policy description is updated;

Step (42), according to the result of each action execution recorded in step (3), count when the NPC performs the action Perform actions with the player The conditional probability of the frequency at which the current state transitions from s ^t to state s ^t+1 when The expected utility value corresponding to the action Update state transition function of NPC interactive dynamic influence graph and utility function The updated specific formula is as follows:

in, is the conditional probability in the existing interactive dynamic influence diagram, is the utility function in the existing interactive dynamic influence diagram;

Step (43), updating the interactive dynamic influence diagram with the results obtained in step (41) and step (42), for step (2)-step (4).

In summary, owing to adopting above-mentioned technical scheme, the beneficial effect of the present invention is:

1. The interactive dynamic influence graph in the present invention is one of the latest research results of artificial intelligence. It has powerful modeling capabilities for individuals and can accurately describe the player's behavior. Moreover, the operating efficiency and script quality of the script generation algorithm based on the model are both excellent. Meet the requirements of practical applications;

2. The present invention can realize the automatic online update and generation of behavior scripts through the interactive dynamic influence graph defined in advance for monster NPCs, greatly reducing the workload of game developers;

3. In the present invention, the update of the interactive dynamic influence map contained in the monster NPC is based on real online data, so the update of the NPC script is more targeted to the player. Further, through each monster NPC in multiple copies Sharing data and player strategy descriptions among players, the generated NPC behavior scripts have good versatility.

Description of drawings

Fig. 1 is the interactive dynamic influence figure that the present invention comprises 2 steps reasoning;

Fig. 2 is a schematic diagram describing the strategy of the present invention;

Fig. 3 is an example of execution record data of the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Step (1), define an automatic reasoning model for each NPC—an interactive dynamic influence diagram (hereinafter referred to as the influence diagram), and define the parameters of the influence diagram according to the characteristics of the NPC itself and the characteristics of the scene.

Step (2), according to the parameters of the influence diagram, the behavior script of the NPC is constructed.

Step (3), the NPC executes the behavior script during the interaction with the player, and records each action, wherein the action refers to the behavior script.

Step (4), according to the execution result of each action recorded in step (3), update the parameters of the influence graph, and regenerate the behavior script of the NPC to be applied to the new player interaction.

In the step (1), according to the characteristics of the NPC itself and the characteristics of the scene, the parameters of the influence diagram are defined as follows:

In the influence diagram, define the set of actions according to the executable actions of the NPC (For example, NPC's executable actions such as attack, defense, escape, pursuit, etc.), define the action set according to the executable actions of the player character The executable actions of the player character such as light punch, heavy punch, escape, etc.; define the state set S ^* according to the nature of the scene, the position and attributes of the NPC and the player character; the state transition function from time t to time t+1 According to the NPC's behavior style and preferences (for example, the utility of aggressive NPC attacks will be set higher to encourage attacks, and the utility of conservative characters when they take defense is relatively high), define the utility function of NPC Among them, the state in the state set S ^* contains the basic information of the NPC and the player character: such as the position and attribute value of the NPC and the player. The state transition function is a conditional probability function, specifically, the state transition function Quantitatively express from time t to time t+1, state s ^t ∈ S ^t , through NPC actions and player character actions The possibility of transitioning to the state s ^t+1 ∈ S ^t+1 , that is, the probability; the utility function Quantitatively represent the results of each action performed by the NPC, specifically, the utility function Indicates that at time t, the NPC passes the action and player character actions The utility value obtained by transferring from state s ^t ∈ S ^t to state s ^t+1 ∈ S ^t+1 . When the NPC performs an action, the damage received by the player character is the positive utility of the NPC; the damage received by the NPC itself is the negative utility of the NPC, and the total utility of the action is the sum of these two parts. The utility of the NPC action also depends on the initial state s ^t , the target state s ^t+1 and the action of the player character Figure 1 is a dynamic influence diagram containing 2-step reasoning (t and t+1). When the 3rd step reasoning is required, the model of the t+2 step can be copied by the t+1 time slice (that is, all the t+1-based Superscripted nodes) and the links between the nodes are constructed. Repeating this process, a dynamic influence graph including T-step reasoning can be constructed. According to the existing empirical knowledge, initialize the state transition function and utility function parameters.

According to prior knowledge, the NPC's influence diagram contains several policy descriptions that describe the behavior of the player character Description of each policy Indicates the conditional probability that the player character will perform different actions in different states Specifically, Defines the action that the player character performs in state s ^t The probability. Store N policy descriptions in a collection of player behavior patterns The formula is as follows:

The concrete steps of described step (2) are as follows:

Step (21), expanding the influence diagram constructed in step (1) into an influence diagram including T-step reasoning according to the required script length. Because at time t, the action performed by the NPC will generate utility value The utility value here also depends on the current state S ^t , the target state S ^t+1 and the action of the player character The utility value here quantitatively describes the actions performed by the NPC in a specific situation (including the current state, new state and player's actions) the result of. In addition, the actions performed by the NPC will also randomly lead to changes in the battle situation, that is, state transitions changes, and then affect the utility value that may be obtained in the future. Therefore, the goal of solving this image graph containing T-step reasoning is to maximize the immediate expected utility of the NPC and forward expected utility The weighted sum of , weighting is to reduce the impact of future utility on current utility.

Step (22), use the classic dynamic programming algorithm to solve the interactive dynamic influence diagram that contains T-step reasoning obtained in step (21), maximize the weighted sum of the immediate expected utility and long-term expected utility of the NPC, and find the one that can A strategy for maximizing the expected utility of an NPC role, where the strategy specifies the actions that the NPC should take for various situations (including its own state, player's state, position, etc.) at each moment Each action taken also depends on the current state, where the strategy uses a conditional probability function to describe. The current state depends on the probabilities of the previous state and previous actions taken by the player and NPC The previous actions taken by the player and the NPC are also called observation information; among them, the formula for maximizing the weighted sum of the NPC’s immediate expected utility and long-term expected utility is as follows:

in, is the immediate expected utility, is the long-term expected utility, λ is the discount factor, which is used to weaken the impact of the long-term utility on the current action, and ER _i is the total utility value, that is, the weighted sum of the immediate expected utility and the long-term expected utility of the NPC. The expected utility value is the utility value in various specific situations and the possibility of various situations, that is, the current state is ^st , and the NPC performs an action Goal state s ^t+1 , and the player character performs the action The specific utility value obtained when and the probability of this happening The weighted sum of , where every possible combination needs to be considered.

Step (23), converting the strategy of maximizing NPC expected utility obtained in step (22) into a behavior script format compatible with the current game. The strategy expressed by the conditional probability function to be obtained is converted into a behavior script format based on conditional judgment compatible with the current game.

The concrete steps of described step (3) are as follows:

(31) During the interaction process between the NPC and the player character, execute the behavior script defined in step (22) for different actions of different players. Each action taken also depends on the current state S ^t , where S ^t can also be regarded as the information observed by the NPC at different times.

(32) Record the results of each action execution: at each moment, including the actions performed by the player character Actions performed by NPCs The change to the state, that is, the utility value corresponding to the action under the transition from state s ^t to state s ^t+1 and from state s ^t to state s ^t+1 The data format is shown in Figure 3.

The concrete steps of described step (4) are as follows:

(41) From the results of each action recorded in step (32), count the actions performed by the player character when the NPC is in each state ^st ∈ S ^t The frequency of , the frequency is normalized, and then the conditional probability of the behavior frequency in each state is obtained That is, the NPC describes the player's strategy based on the current player's behavior By integrating with player behavior patterns Compare to find the most similar player strategy which is of and of is most similar to:

Strategy description of player behavior based on statistics Updated interactive dynamic influence graph for most similar player strategy descriptions In various situations (i.e. different current states s ^t , different player character actions ) the conditional probability of taking various actions get the updated policy description The specific formula is as follows:

in, is a policy description of player behavior in existing interactive dynamic influence graphs, is the policy description obtained from the interaction data, and α is the policy update rate, indicating the speed at which the current policy description is updated;

By updating the strategy description of the player's angle in the existing interactive dynamic influence graph based on the interaction data, the player's behavior can be more accurately predicted in the behavior script generation, making the NPC's behavior more intelligent in subsequent battles.

(42) According to the result of each action execution recorded in step (32), count when the NPC character performs the action Action performed with the player Probability of how often the current state transitions from state s ^t to state s ^t+1 The expected utility value corresponding to the action The expected utility value here is the weighted sum of the utility value in various specific situations and the occurrence frequency of various situations. Update the state transition function of the monster NPC influence graph and utility function The updated method is similar to step (41):

By updating the state transition function and utility function in the existing influence diagram based on the interaction data, the game scene where the NPC is located can be more accurately described, making the generated NPC behavior more intelligent in subsequent battles.

(43) Update the interactive dynamic influence diagram with the results obtained in step (41) and step (42), and use it in step (2)-step (4).

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. within range.

Claims (5)

1. An NPC control method based on multi-agent technology is characterized by comprising the following steps:

defining an automatic reasoning model, namely an interactive dynamic influence graph, for each NPC, and defining parameters of the interactive dynamic influence graph according to the characteristics of the NPC and the characteristics of a scene;

step (2), constructing a behavior script of the NPC according to the parameters of the interactive dynamic influence diagram;

step (3), the NPC executes the behavior script of the NPC in the interactive process with the player, and records the execution result of each action, wherein the action refers to the behavior script;

and (4) updating parameters of the interactive dynamic influence diagram according to the execution result of each action recorded in the step (3), and reconstructing the behavior script of the NPC to be applied to new player interaction.

2. The multi-agent technology-based NPC control method as claimed in claim 1, wherein in step (1), the specific steps of defining parameters of the interactive dynamic influence map according to the characteristics of the NPC and the characteristics of the scene are as follows:

step (11), in the interactive dynamic influence diagram, defining action set according to the executable action of NPCDefining a set of actions from executable actions of a player character

Step (12), defining a state set S according to the property of the scene where the NPC is positioned, the position and the property of the NPC and the player character^*；

Step (13), defining a state s from time t to time t +1 according to step (11) and step (12)^t∈S^tBy NPC actionAnd player character actionsTransition to state s^t+1∈S^t+1Conditional probability function of likelihood, i.e. state transition function

Step (14), defining the NPC passing action at the time t according to the action style and preference of the NPC roleAnd player character actionsSlave state s^t∈S^tTransition to state s^t+1∈S^t+1Utility function of

Step (15), initializing the state transfer function according to the prior experience knowledgeAnd utility functionThe parameters of (1);

step (16), according to prior, the interactive dynamic influence diagram of the NPC contains a plurality of strategy descriptions which describe the player character behaviorsEach policy descriptionRepresenting conditional probabilities that a player character will perform different actions in different statesI.e. the player character is in state s^tExecute actions at the timeProbability of (2)Describing N strategiesCollections stored in player behavior patternsThe method comprises the following specific steps:

<mrow> <msubsup> <mi>M</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>=</mo> <mo>{</mo> <msubsup> <mi>m</mi> <mrow> <mi>j</mi> <mo>,</mo> <mn>1</mn> </mrow> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>m</mi> <mrow> <mi>j</mi> <mo>,</mo> <mn>2</mn> </mrow> <mi>t</mi> </msubsup> <mo>...</mo> <mo>,</mo> <msubsup> <mi>m</mi> <mrow> <mi>j</mi> <mo>,</mo> <mi>N</mi> </mrow> <mi>t</mi> </msubsup> <mo>}</mo> <mo>;</mo> </mrow>

<mrow> <msubsup> <mi>m</mi> <mrow> <mi>j</mi> <mo>,</mo> <mi>k</mi> </mrow> <mi>t</mi> </msubsup> <mo>=</mo> <mi>P</mi> <mrow> <mo>(</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>|</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>)</mo> </mrow> <mo>,</mo> <mn>1</mn> <mo>&amp;le;</mo> <mi>k</mi> <mo>&amp;le;</mo> <mi>N</mi> <mo>.</mo> </mrow>

3. the NPC control method based on multi-agent technology as claimed in claim 1, wherein: the specific steps of the step (2) are as follows:

step (21), according to the interactive dynamic influence diagram constructed in the step (1) and the required NPC behavior script length, expanding the interactive dynamic influence diagram into an interactive dynamic influence diagram containing T-step reasoning;

step (22), solving the inclusion obtained in step (21) by adopting a classical dynamic programming solving algorithmAn interactive dynamic influence graph with T-step reasoning, a weighted sum of the instant expected utility and the long-term expected utility of the NPC is maximized, a strategy for maximizing the expected utility of the NPC role is found, namely actions which can be taken by the NPC for various situations at each moment are foundConditional probability of (2)The formula that maximizes the weighted sum of the immediate expected utility and the long-term expected utility of the NPC is as follows:

<mrow> <msub> <mi>ER</mi> <mi>i</mi> </msub> <mo>=</mo> <mi>E</mi> <mo>&amp;lsqb;</mo> <msubsup> <mi>R</mi> <mi>i</mi> <mn>1</mn> </msubsup> <mo>&amp;rsqb;</mo> <mo>+</mo> <mi>&amp;lambda;</mi> <mi>E</mi> <mo>&amp;lsqb;</mo> <msubsup> <mi>R</mi> <mi>i</mi> <mn>2</mn> </msubsup> <mo>&amp;rsqb;</mo> <mo>+</mo> <msup> <mi>&amp;lambda;</mi> <mn>2</mn> </msup> <mi>E</mi> <mo>&amp;lsqb;</mo> <msubsup> <mi>R</mi> <mi>i</mi> <mn>3</mn> </msubsup> <mo>&amp;rsqb;</mo> <mo>+</mo> <mo>...</mo> <mo>+</mo> <msup> <mi>&amp;lambda;</mi> <mrow> <mi>T</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mi>E</mi> <mo>&amp;lsqb;</mo> <msubsup> <mi>R</mi> <mi>i</mi> <mi>T</mi> </msubsup> <mo>&amp;rsqb;</mo> <mo>,</mo> </mrow>

<mrow> <mi>E</mi> <mo>&amp;lsqb;</mo> <msubsup> <mi>R</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>&amp;rsqb;</mo> <mo>=</mo> <msub> <mo>&amp;Sigma;</mo> <mrow> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>A</mi> <mrow> <mi>j</mi> <mo>,</mo> </mrow> <mi>t</mi> </msubsup> <msup> <mi>S</mi> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> </mrow> </msub> <msub> <mi>R</mi> <mi>i</mi> </msub> <mrow> <mo>(</mo> <msup> <mi>s</mi> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>,</mo> <msup> <mi>s</mi> <mi>t</mi> </msup> <mo>,</mo> <msubsup> <mi>a</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>a</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>)</mo> </mrow> <mo>*</mo> <mi>P</mi> <mrow> <mo>(</mo> <msup> <mi>s</mi> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>,</mo> <msup> <mi>s</mi> <mi>t</mi> </msup> <mo>,</mo> <msubsup> <mi>a</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>a</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>)</mo> </mrow> <mo>,</mo> <mn>1</mn> <mo>&amp;le;</mo> <mi>t</mi> <mo>&amp;le;</mo> <mi>T</mi> <mo>,</mo> </mrow>

wherein,is the instant desired utility of the utility,is the expected effect of the future, and λ is a discount factor to reduce the impact of the future effect on the current action, ER_iIs the total utility value, i.e., the weighted sum of the instant expected utility and the future expected utility of the NPC;

and (23) converting the strategy which is obtained in the step (22) and maximizes the expected utility of the NPC into a behavior script format compatible with the current game.

4. The multi-agent technology-based NPC control method as claimed in claim 3, wherein the specific steps of the step (3) are as follows:

step (31), NPC adopts NPC action aiming at different actions of different player characters in the process of interacting with the player charactersAdopting the behavior script obtained in the step (23);

step (32) of recording each NPC actionThe result of the execution is: at each time, including the actions performed by the player characterActions performed by NPCFor change of state, i.e. from state s^tTransition to state s^t+1And state s^tTransition to state s^t+1Utility value corresponding to down action

5. The multi-agent technology-based NPC control method as claimed in claim 4, wherein the specific steps of the step (4) are as follows:

step (41) recording the result of each action execution according to the step (3), and counting the state s of the NPC in each state^t∈S^tEach action performed by a player characterNormalizing the frequency to obtain the conditional probability of the behavior frequency in each stateI.e. can be regarded as a strategy description of NPC construction based on current player's behavior By aggregation with player behavior patternsComparing, finding the most similar player strategy descriptionsNamely, it isIs/are as followsAndis/are as followsIs the most similar:

<mrow> <msubsup> <mi>m</mi> <mrow> <mi>j</mi> <mo>,</mo> <mi>k</mi> </mrow> <mi>t</mi> </msubsup> <mo>=</mo> <mi>arg</mi> <mi> </mi> <msub> <mi>min</mi> <mi>m</mi> </msub> <msub> <mo>&amp;Sigma;</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> </msub> <msup> <mrow> <mo>&amp;lsqb;</mo> <mi>P</mi> <mrow> <mo>(</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>|</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>)</mo> </mrow> <mo>-</mo> <mover> <mi>P</mi> <mo>~</mo> </mover> <mrow> <mo>(</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>|</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>)</mo> </mrow> <mo>&amp;rsqb;</mo> </mrow> <mn>2</mn> </msup> <mo>,</mo> <mi>m</mi> <mo>&amp;Element;</mo> <msubsup> <mi>M</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>,</mo> </mrow>

player behavior strategy description obtained according to statisticsUpdating the most similar player strategy descriptions in an interactive dynamic impact graphConditional probabilities of taking various actions in various situationsObtaining updated policy descriptionsThe specific formula is as follows:

<mrow> <msub> <mover> <mi>P</mi> <mo>&amp;CenterDot;</mo> </mover> <mrow> <mi>n</mi> <mi>e</mi> <mi>w</mi> </mrow> </msub> <mrow> <mo>(</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>|</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>)</mo> </mrow> <mo>=</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>-</mo> <mi>&amp;alpha;</mi> <mo>)</mo> </mrow> <mo>*</mo> <mi>P</mi> <mrow> <mo>(</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>|</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>)</mo> </mrow> <mo>+</mo> <mi>&amp;alpha;</mi> <mo>*</mo> <mover> <mi>P</mi> <mo>~</mo> </mover> <mrow> <mo>(</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>|</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>)</mo> </mrow> <mo>,</mo> </mrow>

wherein,is a strategic description of player behavior in existing interactive dynamic impact diagrams,is the policy description obtained from the interaction data, α is the policy update rate, which represents the speed at which the current policy description is updated;

step (42), according to the result of each action execution recorded in step (3), counting the action executed when NPC executesPerforming an action with a playerTime, current state from s^tTransition to state s^t+1Conditional probability of frequency ofPeriod corresponding to actionValue of hopeful effectUpdating state transition functions of NPC interactive dynamic impact diagramsAnd utility functionThe updated specific formula is as follows:

<mrow> <msub> <mi>P</mi> <mrow> <mi>n</mi> <mi>e</mi> <mi>w</mi> </mrow> </msub> <mrow> <mo>(</mo> <msup> <mi>S</mi> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>|</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>)</mo> </mrow> <mo>=</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>-</mo> <mi>&amp;alpha;</mi> <mo>)</mo> </mrow> <mo>*</mo> <mi>P</mi> <mrow> <mo>(</mo> <msup> <mi>S</mi> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>|</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>)</mo> </mrow> <mo>+</mo> <mi>&amp;alpha;</mi> <mo>*</mo> <mover> <mi>P</mi> <mo>~</mo> </mover> <mrow> <mo>(</mo> <msup> <mi>S</mi> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>|</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>)</mo> </mrow> <mo>,</mo> </mrow>2

<mrow> <msub> <mi>R</mi> <mrow> <mi>i</mi> <mo>,</mo> <mi>n</mi> <mi>e</mi> <mi>w</mi> </mrow> </msub> <mrow> <mo>(</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msup> <mi>S</mi> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>)</mo> </mrow> <mo>=</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>-</mo> <mi>&amp;alpha;</mi> <mo>)</mo> </mrow> <mo>*</mo> <msub> <mi>R</mi> <mi>i</mi> </msub> <mrow> <mo>(</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msup> <mi>S</mi> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>)</mo> </mrow> <mo>+</mo> <mi>&amp;alpha;</mi> <mo>*</mo> <msub> <mover> <mi>R</mi> <mo>~</mo> </mover> <mi>i</mi> </msub> <mrow> <mo>(</mo> <msup> <mi>S</mi> <mi>t</mi> </msup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>i</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msubsup> <mi>A</mi> <mi>j</mi> <mi>t</mi> </msubsup> <mo>,</mo> <msup> <mi>S</mi> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>)</mo> </mrow> <mo>,</mo> </mrow>

wherein,is the conditional probability in the existing interactive dynamic impact graph,is a utility function in the existing interactive dynamic impact graph;

and (43) updating the interactive dynamic influence diagram according to the results obtained in the step (41) and the step (42) for the step (2) to the step (4).