WO2025228010A1 - 模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品 - Google Patents

模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品

Info

Publication number
WO2025228010A1
WO2025228010A1 PCT/CN2025/084695 CN2025084695W WO2025228010A1 WO 2025228010 A1 WO2025228010 A1 WO 2025228010A1 CN 2025084695 W CN2025084695 W CN 2025084695W WO 2025228010 A1 WO2025228010 A1 WO 2025228010A1
Authority
WO
WIPO (PCT)
Prior art keywords
action
pathfinding
information
agent
reward
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/084695
Other languages
English (en)
French (fr)
Inventor
胡欢
廖诗飏
刘若尘
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology (Shenzhen) Co Ltd
Original Assignee
Tencent Technology (Shenzhen) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Co Ltd filed Critical Tencent Technology (Shenzhen) Co Ltd
Publication of WO2025228010A1 publication Critical patent/WO2025228010A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/092—Reinforcement learning
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/55—Controlling game characters or game objects based on the game progress
    • A63F13/56—Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/80—Special adaptations for executing a specific game genre or game mode
    • A63F13/822—Strategy games; Role-playing games
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/80—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
    • A63F2300/807—Role playing or strategy games

Definitions

  • This application relates to the field of artificial intelligence, and in particular to a model training method, apparatus, computer device, computer-readable storage medium, and computer program product.
  • Game agents are one of the important elements in games, which can add depth, fun, and interactivity to games, and enhance players' gaming experience and engagement.
  • the two main techniques are: first, using a navigation grid and scene query system to control the agent's pathfinding; and second, using atomic actions. Both of these techniques employ a single, fixed pathfinding method, leading to poor training results. Ultimately, this results in models failing to deploy online, learning strategies that are ineffective, or even failing to converge.
  • This application provides a model training method, apparatus, computer device, computer-readable storage medium, and computer program product, which can improve the efficiency and accuracy of model training.
  • This application provides a model training method applied to a computer device, the method comprising:
  • the state data of the first intelligent agent in the virtual scene and the first prediction model to be trained corresponding to the first intelligent agent are obtained; the first prediction model to be trained is used to control the first intelligent agent to move based on the state data of the first intelligent agent.
  • the state data is predicted using the first prediction model to be trained to obtain a set of action instructions, which includes a pathfinding method and multiple action information.
  • the first intelligent agent is controlled to move based on the pathfinding method, and performs corresponding actions based on each action information to obtain the execution result;
  • the parameters of the first prediction model to be trained are adjusted based on the reward information of the multiple execution results to obtain the trained first prediction model, wherein the reward information is determined based on the multiple execution results, the main reward coefficient, and the auxiliary reward coefficient.
  • This application provides a model training apparatus, the apparatus comprising:
  • the first acquisition module is configured to acquire the state data of the first intelligent agent in the virtual scene and the first prediction model to be trained corresponding to the first intelligent agent; the first prediction model to be trained is used to control the first intelligent agent to move based on the state data of the first intelligent agent.
  • the instruction prediction module is configured to use the first prediction model to be trained to predict the state data and obtain an action instruction set, the action instruction set including a pathfinding method and multiple action information.
  • the execution module is configured to control the first intelligent agent to move based on the pathfinding method, and to execute corresponding actions based on each action information to obtain the execution result;
  • the adjustment module is configured to adjust the parameters of the first prediction model to be trained based on the reward information of multiple execution results, so as to obtain a trained first prediction model, wherein the reward information is determined based on the multiple execution results, the main reward coefficient and the auxiliary reward coefficient.
  • This application provides a computer device, including:
  • Memory is used to store executable instructions for a computer
  • the processor when executing computer-executable instructions stored in the memory, implements the model training method provided in the embodiments of this application.
  • This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the model training method provided in this application.
  • This application provides a computer program product, including a computer program or computer executable instructions.
  • the computer program or computer executable instructions When executed by a processor, they implement the model training method provided in this application.
  • the game includes a first agent and a second agent that compete against each other.
  • the first agent corresponds to a first prediction model to be trained.
  • the first prediction model to be trained is used to predict the state data of the first agent, obtaining a set of action commands including pathfinding methods. Since the first prediction model can control the first agent to move according to different pathfinding methods based on different scenarios, when the first agent is in different virtual scenarios, it can predict a set of action commands containing different pathfinding methods.
  • the pathfinding methods in this set of action commands match the scenario in which the first agent is located, thus achieving diversification of pathfinding methods.
  • the first agent is controlled to execute corresponding actions, obtaining execution results.
  • the main reward coefficient, and the auxiliary reward coefficient the reward information of multiple execution results is determined, and the parameters of the first prediction model to be trained are adjusted based on the reward information of multiple execution results to obtain a trained first prediction model.
  • This approach enables hybrid pathfinding and movement for the first agent, diversifying pathfinding and movement methods. This variety of pathfinding methods improves the training performance of the first prediction model under training. Furthermore, since the first agent's pathfinding methods are matched to the scene, it also enhances the training accuracy of the first prediction model. In addition, because both primary and secondary reward coefficients are used when determining reward information, the learning efficiency and performance of the first agent in policy learning tasks are improved, thus enhancing the predictive performance of the trained first prediction model.
  • Figure 1 is a schematic diagram of a network architecture of a model training system 100 provided in an embodiment of this application;
  • FIG. 2 is a structural schematic diagram of the server 400 provided in an embodiment of this application.
  • Figure 3 is a schematic diagram of the first process of the model training method provided in the embodiment of this application.
  • Figure 4 is a schematic diagram of the first flow of the prediction method provided in the embodiment of this application.
  • Figure 5 is a flowchart illustrating a feature extraction method provided in an embodiment of this application.
  • Figure 6 is a schematic diagram of the second process of the prediction method provided in the embodiment of this application.
  • Figure 7 is a schematic diagram of the third process of the prediction method provided in the embodiments of this application.
  • Figure 8 is a schematic diagram of the fourth process of the prediction method provided in the embodiments of this application.
  • Figure 9 is a flowchart illustrating a set of construction action instructions provided in an embodiment of this application.
  • Figure 10 is a flowchart illustrating an execution action provided in an embodiment of this application.
  • Figure 11 is a flowchart illustrating an adjustment parameter provided in an embodiment of this application.
  • Figure 12 is a schematic diagram of the second process of the model training method provided in the embodiment of this application.
  • Figure 13 is a schematic diagram of an interface display of a farm map according to an embodiment of this application.
  • Figure 14A is a flowchart illustrating the logic for determining the battle configuration provided in an embodiment of this application.
  • Figure 14B is a schematic diagram of an interface display of generated coordinates provided in an embodiment of this application.
  • Figure 15 is a schematic diagram of the overall structure of the model training stage provided in the embodiments of this application.
  • Figure 16 is a schematic diagram of a synchronous inference mode provided in an embodiment of this application.
  • Figure 17 is a structural flowchart of an asynchronous inference mode provided in an embodiment of this application.
  • Figure 18A is a schematic diagram of a network structure of a hybrid pathfinding action head provided in an embodiment of this application;
  • Figure 18B is a schematic diagram of a time slice structure provided in an embodiment of this application.
  • Figure 19 is a schematic diagram of a hybrid pathfinding process provided in an embodiment of this application.
  • Figure 20 is a schematic diagram of a frame structure of the model structure provided in an embodiment of this application.
  • Figure 21A is a schematic diagram of the distribution of the AI's activity area in the map, trained based on a hybrid mobile pathfinding scheme according to an embodiment of this application.
  • Figure 21B is a schematic diagram of the distribution of the AI's activity area in the map, trained based on the atomic movement pathfinding scheme provided in the embodiments of this application.
  • Figure 22 is a schematic diagram showing the win rate results of AI battle behavior tree provided in an embodiment of this application.
  • Figure 23 is a schematic diagram showing the AI battle behavior tree draw rate results provided in the embodiments of this application.
  • first and second are used only to distinguish different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process.
  • first/second/third are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first/second/third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.
  • module refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof.
  • a processor or multiple processors or memory
  • each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
  • the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.
  • Intelligent agents In the gaming field, intelligent agents typically refer to virtual characters or agent characters within a game. They are controlled by computer programs and exhibit behavioral characteristics that simulate human intelligence. These intelligent agents can be enemies, allies, non-player characters (NPCs), or other roles in the game. The design and implementation of intelligent agents can enhance the playability, challenge, and realism of a game.
  • Game agents In a game scenario, the machine that performs reinforcement learning is called an agent. It is a game artificial intelligence (AI) driven by neural networks. The environment that interacts with the agent is called the environment. The game agent makes reasonable decisions by analyzing the game environment and can cooperate and compete with other agents or players, making it behave like a human.
  • AI game artificial intelligence
  • Reinforcement Learning is a machine learning method that aims to enable an agent to learn how to make decisions through interaction with its environment in order to maximize cumulative rewards.
  • the agent explores the environment by trying different actions and adjusts its behavior based on reward signals from the environment to maximize long-term cumulative rewards. This learning method is similar to trial-and-error learning, where the agent optimizes its behavioral strategies through continuous experimentation and feedback.
  • Reinforcement learning is often used to solve problems requiring dynamic decision-making, such as training game agents, robot control, and autonomous driving.
  • Supervised learning is one of the paradigms of machine learning. It is a data analysis method that iteratively trains an algorithm from a labeled dataset to enable the model to accurately classify data or predict results.
  • Navigation Mesh is the walking surface for navigation and pathfinding in complex spaces. It is a two-dimensional polygonal mesh data structure that defines which areas of the environment the game agent can traverse. Within these areas, the game agent can move freely without being hindered by environmental obstacles. When the game agent receives a command, the navigation mesh will automatically calculate the shortest reachable path to the target point, helping the game agent to find its way in complex airspace and easily achieving the automatic pathfinding effect of AI.
  • EQS Environment Query System
  • Depth Map is a two-dimensional data structure that uses the distance (depth) from the image acquisition device to each point in the scene as pixel values. It can depict the pure depth information in front of the game agent in detail. The closer the value is to 1, the brighter the front is and the clearer the field of vision is.
  • a depth map is an image where the value of each pixel represents the distance from the observer (usually a camera) to the corresponding point in the scene. It is typically presented as a grayscale image, where darker pixels represent farther distances, and brighter pixels represent closer distances.
  • depth maps are frequently used to achieve depth of field effects, simulating the changes in depth of field caused by different camera focal lengths. By using depth maps, game engines can determine which objects are in focus and which are in the blurred area, resulting in more realistic visual effects.
  • Circular rays Starting from itself, several rays are emitted outward around its own perimeter. This is used to describe the terrain information of obstacles around itself and is a method of environmental perception.
  • a ring ray is a technique commonly used in computer graphics and game development to detect collisions between objects or determine the distance between them. Unlike a regular ray (i.e., a line extending from a point in one direction), a ring ray is a ray emitted from a point in all directions, forming a circular or ring-shaped array of rays.
  • Original Action refers to the most basic and fine-grained primitive actions performed by game characters in a game. These actions cannot be further broken down and refined. In concurrent programming, atomic actions are often used to ensure synchronization and consistency between multiple threads or processes.
  • Behavior Tree A tree-like structure used to control the decision-making of NPCs in games. It is a graphical tool and method for describing and controlling the behavior of game characters or intelligent agents. It is a tree structure composed of a series of nodes, each node representing a behavior, condition, or control logic. Behavior trees are widely used in game development, artificial intelligence, and robotics.
  • Proximal Policy Optimization is an online deep reinforcement learning algorithm based on policy gradient optimization, applicable to continuous or discrete action spaces.
  • the core idea of PPO is to control the magnitude of policy changes during each policy update, keeping the update magnitude within a controllable range, thereby improving the algorithm's stability.
  • Auxiliary rewards In reinforcement learning algorithms, to enable agents to achieve various differentiated performances and more targeted goals, rewards are designed to assist in achieving the main objective. During agent training, auxiliary rewards typically refer to additional reward signals used to assist the agent in learning specific aspects of the task or in achieving specific goals. Auxiliary rewards are often used in conjunction with the main task reward to help accelerate the agent's learning process, improve its performance, and in some cases, help solve difficult problems in the task.
  • auxiliary rewards include exploration rewards, time rewards, distance rewards, cover-finding rewards, and action diversity rewards, which can be used in different learning frameworks such as reinforcement learning and supervised learning.
  • auxiliary rewards By designing and using auxiliary rewards appropriately, the learning efficiency and performance of an agent in a task can be improved.
  • PVP Player versus player
  • AI artificial intelligence
  • PVP mode is very common in many games, including esports, massively multiplayer online (MMO) games, shooters, fighting games, and various other genres.
  • PVP mode typically increases the competitiveness and challenge of a game and is one of the most popular game features among many players.
  • PVE Player versus environment
  • Embedding is essentially “data compression”. It uses lower k-dimensional features to describe higher n-dimensional features with redundant information. Information compression usually results in information loss, including the loss of redundant information and some useful information. Embedding can be viewed as the parameter matrix of a one-hot layer in a neural network.
  • “embedding” typically refers to the process of mapping high-dimensional discrete data (such as categorical features or symbolic data) to a low-dimensional continuous vector space. This mapping can be achieved through the embedding layer in a neural network.
  • embeddings are typically used to process various types of input data, including word embeddings, categorical embeddings, action embeddings, and state embeddings.
  • original discrete data can be transformed into continuous vector representations, improving the data's expressive power and the model's generalization ability, thereby enhancing the effectiveness and performance of training agents.
  • Action Mask In deep reinforcement learning training, when the network infers the action distribution, "unreasonable or unexecutable action dimensions" are temporarily masked, and then actions are sampled from “reasonable or executable actions” according to the original probability. This can ensure that the inferred actions can obtain effective data samples in the game environment and avoid wasting computing resources due to returning invalid actions.
  • Action masking is used to restrict the set of available actions when an agent chooses an action. Action masking ensures that the agent only chooses from permitted actions, ignoring those that are not allowed, thus making the agent's decision-making process more efficient and accurate.
  • An action mask can be represented by a binary vector (or mask), where each element corresponds to a possible action. If an action is permitted, the mask value for the corresponding bit is 1; if an action is not permitted, the mask value for that action is 0. For example, assuming there are 5 possible actions, the action mask can be represented as: [1, 0, 1, 0, 0]. This means that the agent can execute the first and third actions, while the other actions are not executable.
  • a zero-sum game also known as a zero-sum game theory concept, refers to a game where the sum of the interests of all players is zero, or a constant. In other words, if one player gains, the others must lose. In a zero-sum game, the players do not cooperate; it is a non-cooperative game. If different combinations of strategies result in all participants either gaining or losing, it is a non-zero-sum game. In a non-zero-sum game, cooperation is possible.
  • BR Best Response
  • Self-Play is an iterative process of finding the optimal response (BR).
  • An agent uses a previous copy of its own policy as an opponent to train and update its current policy, continuously iterating. By training against increasingly stronger opponents, it prevents significant skill differences from hindering policy development, thus effectively increasing training efficiency and continuously improving model capabilities.
  • This training method is highly effective in zero-sum games like chess and Go (AlphaGo), but requires two conditions: the model's ability improves in each round, and the game environment is a transitive game.
  • Nash Equilibrium also known as non-cooperative game equilibrium
  • NE Nash Equilibrium
  • Nash equilibrium is not necessarily the optimal solution for all parties in the game. It expresses that the current game environment has reached a stable equilibrium state.
  • the relevant technologies mainly employ the following two model training methods:
  • Model training method based on navigation grid and scene query system.
  • a scene query system queries several pathfinding target points and returns the pathfinding target points that meet the conditions according to the weights. Then, the AI moves to these target points using a navigation grid. Since the movement path of the navigation grid is guaranteed to be reachable, this ensures the AI's ability to reach the pathfinding target point.
  • Navigation grid movement is a common movement method used by business teams to develop various behavior tree AIs.
  • the aforementioned related technology 1 has the following drawbacks: Although moving from the navigation grid to the pathfinding target point guarantees the AI's accessibility, this query method only provides the AI with a few discrete reachable points. While navigation grid movement ensures reachability, its movement path is singular and fixed, failing to clearly indicate all reachable paths around the AI. This limits the diversity of the AI's movement paths, reduces the completeness of the feasible area, and may even cause the AI to miss a better path simply because the returned pathfinding target point is not on that path. Furthermore, each path query operation based on a pathfinding point consumes server performance.
  • the performance consumption is related to the number of pathfinding points and the distance between the pathfinding points and the AI itself; the more pathfinding points and the greater the distance, the greater the performance consumption. If this pathfinding scheme is used to train reinforcement learning AI, the pathfinding point query frequency will also increase when the model's prediction frequency is high. The AI is prone to swaying and jerking between returned pathfinding points, resulting in poor performance and significant performance consumption, ultimately making such trained models unusable for online deployment.
  • Atomic actions define the smallest unit of instruction operation. They are the finest-grained instruction sets that cannot be further subdivided. The atomic action space can completely cover the operations that players can execute, achieving the level of human players in terms of completeness and flexibility. Movement commands also adopt atomic movement. Atomic movement can completely break free from the constraints of the navigation grid and move freely and without restraint, thereby going to places that the behavior tree cannot reach and completing operations that the behavior tree cannot perform.
  • the shortcomings of the aforementioned related technology 2 include: the granularity of atomic action instructions is too fine. In order to match human player operations and balance the completeness and smoothness of actions, it is necessary to design complex, diverse, and high-dimensional atomic action instruction sets. Only through the coordinated operation and cooperation of these atomic instruction sets can operations and movements similar to those of human players be completed. If a reinforcement learning model is trained using an atomic action scheme, the poor performance of the learned strategy or even the inability to converge may be caused by the overly fine-grained atomic action logic and the high dimensionality of the atomic instruction sets. This effect is particularly prominent when the model predicts at a high frequency. When the prediction frequency is high, the model will oscillate between various atomic instructions, which can easily lead to uncontrollable behavior. Moreover, when the scene map is very large and the terrain is particularly complex, the model's strategy exploration efficiency is low, the training time cost is too high, and ultimately the model performance and production efficiency are affected.
  • the scene query combined with navigation grid movement method makes it difficult to depict surrounding road conditions in detail, resulting in a single AI movement route;
  • the design of pure atomic action instruction sets is relatively complex and has high dimensionality, requiring repeated debugging and verification, which incurs high time costs;
  • the model training method is not suitable for shooting games with complex and varied maps.
  • This application provides a model training method, apparatus, computer device, computer-readable storage medium, and computer program product, which can improve the efficiency and accuracy of model training.
  • the exemplary application of the computer device provided in this application is described below.
  • the computer device provided in this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or it can be implemented as a server.
  • the exemplary application of the computer device as a server will be described below.
  • the server 400 in order to support a model training application, is connected to the terminal 200 through the network 300.
  • the network 300 can be a wide area network or a local area network, or a combination of the two.
  • terminal 200 collects the state data of the first agent during a game and sends the state data to server 400.
  • Server 400 uses a first prediction model to be trained to predict the state data, obtaining a set of action instructions including pathfinding methods. Based on the action instruction set, server 400 controls the first agent to execute corresponding actions and obtains execution results. Based on the reward information of multiple execution results obtained in the game, server 400 adjusts the parameters of the first prediction model to be trained, obtaining a trained first prediction model. The reward information is determined based on multiple execution results, the main reward coefficient, and the auxiliary reward coefficient. Server 400 also returns the trained first prediction model to terminal 200, which can use the trained first prediction model offline to control the first agent to complete a game of the first-person shooter.
  • the server 400 uses the first prediction model to predict the state data of the first agent, obtaining a set of action instructions including pathfinding methods.
  • the first prediction model can control the first agent to move according to different pathfinding methods based on different scenarios, when the first agent is in different virtual scenarios, it can predict a set of action instructions containing different pathfinding methods.
  • the pathfinding methods in this set of action instructions match the scenario in which the first agent is located, thus achieving diversification of pathfinding methods.
  • the trained second prediction model can also control the second agent to move according to different pathfinding methods based on different scenarios, the second agent can also execute different pathfinding methods in different scenarios, thus strengthening the ability of the first agent to execute different pathfinding methods in different scenarios.
  • the server controls the first agent to execute the corresponding actions, obtaining the execution result.
  • the reward information for each execution result is determined. Based on this reward information, the parameters of the first prediction model to be trained are adjusted, resulting in a trained first prediction model. This enables hybrid pathfinding and movement of the first agent, diversifying pathfinding and movement methods.
  • server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
  • Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.
  • the server 400 shown in Figure 2 includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430.
  • the various components in the server 400 are coupled together through a bus system 440.
  • the bus system 440 is used to implement communication between these components.
  • the bus system 440 also includes a power bus, a control bus, and a status signal bus.
  • all buses are labeled as bus system 440 in Figure 2.
  • the processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor can be a microprocessor or any conventional processor, etc.
  • User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and/or one or more visual displays.
  • User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
  • the memory 450 may be removable, non-removable, or a combination thereof.
  • Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc.
  • the memory 450 may optionally include one or more storage devices physically located away from the processor 410.
  • the memory 450 may include volatile memory or non-volatile memory, or both.
  • the non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM).
  • ROM read-only memory
  • RAM random access memory
  • the memory 450 described in this application embodiment is intended to include any suitable type of memory.
  • memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.
  • Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;
  • the network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.
  • Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;
  • information e.g., a user interface for operating peripheral devices and displaying content and information
  • output devices 431 e.g., a display screen, a speaker, etc.
  • the input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.
  • the apparatus provided in this application can be implemented in software.
  • Figure 2 shows a model training apparatus 455 stored in memory 450, which can be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 4551, an instruction prediction module 4552, an execution module 4553, and an adjustment module 4554. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.
  • the apparatus provided in this application can be implemented in hardware.
  • the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method provided in this application.
  • the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.
  • ASICs application-specific integrated circuits
  • DSPs digital signal processor
  • PLDs programmable logic devices
  • CPLDs complex programmable logic devices
  • FPGAs field-programmable gate arrays
  • model training method provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.
  • the computer device implementing the model training method in the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.
  • model training method examples below are illustrated using a first-person shooter game as an example. Those skilled in the art, based on their understanding of the following text, can apply the model training methods provided in this application to model training for other types of games. This application can also be applied to various scenarios, including but not limited to games, artificial intelligence, intelligent transportation, and assisted driving.
  • FIG 3 is a schematic diagram of the first process of the model training method provided in the embodiment of this application
  • the model training method provided in the embodiment of this application will be described in conjunction with the steps shown in Figure 3.
  • the execution subject in Figure 3 is the server.
  • step S101 the state data of the first intelligent agent in the virtual scene and the first prediction model to be trained corresponding to the first intelligent agent are obtained.
  • the virtual scene can be a virtual scene during a game, or it can be an augmented reality virtual scene or a virtual reality scene.
  • the virtual scene includes the first intelligent agent and a second intelligent agent corresponding to a trained second prediction model.
  • the second intelligent agent is an agent that opposes the first intelligent agent.
  • the trained second prediction model is used to control the second intelligent agent to move according to different pathfinding methods based on different scenes in which the second intelligent agent is located.
  • the first prediction model to be trained is used to control the first intelligent agent to move according to different pathfinding methods based on different scenes in which the first intelligent agent is located.
  • a game match refers to a game confrontation or competition between two or more intelligent agents, which may be virtual characters, robots or other autonomous entities controlled by computer programs.
  • Game matches between intelligent agents can take different forms and rules, including but not limited to:
  • the first type of game format and rules is the game theory game: intelligent agents engage in games of strategy, such as chess, Go, and poker, to test their strategies and decision-making abilities.
  • the second type of game format and rules is adversarial games: intelligent agents engage in adversarial games, such as combat games and racing games, to test their reaction speed, skills and tactical abilities.
  • the third type of game format and rules is cooperative games: agents play cooperative games, such as team cooperative games and cooperative puzzle games, to test their teamwork and coordination abilities.
  • the fourth type of game and rules is hybrid games: agents engage in various forms of game play, including hybrid games with elements of strategy, confrontation, and cooperation, to test their versatility and adaptability.
  • Game-based competition between intelligent agents is an important area of research and practice in the field of artificial intelligence. Through game-based competition, we can gain a deeper understanding of the behavior and decision-making processes of intelligent agents, thereby improving their performance and effectiveness.
  • an adversarial game is constructed through a first and a second intelligent agent that are in conflict with each other. That is, in this adversarial game, the first and second intelligent agents belong to different factions and are adversaries.
  • this embodiment involves the second type of game format and rules mentioned above; that is, the game format and rules adopted in this embodiment are adversarial games.
  • the state data of a first agent refers to a set of information describing the current state or situation of the first agent.
  • state data is the foundation for the first agent to perceive and understand the environment.
  • State data contains key features and information about the environment, which helps the first agent make decisions and take actions.
  • State data typically consists of various types of information, including data about the environment, the agent itself, and interactions with other entities. State data forms the basis for the agent's interaction with its environment, providing it with the ability to perceive and understand the environment, helping it make appropriate decisions and actions based on the current situation. In reinforcement learning, state data serves as the input to the agent's policy and value function, directly influencing its learning and decision-making processes.
  • the first prediction model to be trained corresponding to the first agent refers to the machine learning model used to train the first agent.
  • the first agent is typically represented by a machine learning model, which can be various types of models such as neural networks, decision trees, and random forests.
  • the first prediction model to be trained refers to the model that needs to be continuously adjusted and optimized during the training process based on the interaction data between the first agent and the environment.
  • the goal of the first prediction model to be trained is to learn an effective policy that enables the first agent to maximize cumulative rewards or achieve specific task objectives when interacting with the environment.
  • the first prediction model to be trained typically includes the following main components:
  • Policy Network Used to learn the policy of the first agent, that is, the probability distribution of choosing the optimal action given a state.
  • Value Network Used to estimate the value of a state or action, helping the first agent determine which actions are better or which states are more valuable.
  • Action-Value Function Used to estimate the value of taking a specific action in a given state, i.e., the expected cumulative reward of the action.
  • Reward function Used to define the reward signal obtained by the first agent in its interaction with the environment, which helps the first prediction model to be trained learn an effective policy.
  • auxiliary components such as experience replay buffers and exploration strategies, are used to improve the stability and effectiveness of the model.
  • the goal of the first prediction model to be trained is to continuously update and optimize it through interaction data with the environment, so that the first agent can gradually learn a better strategy and thus exhibit better performance and effects in the real environment.
  • step S102 the first prediction model to be trained is used to predict the state data to obtain a set of action instructions.
  • the action instruction set includes a pathfinding method and multiple action information.
  • predicting the state data using a first prediction model to be trained means using the first prediction model to predict the set of action instructions that the first agent will execute in the given state corresponding to the state data.
  • the action instruction set refers to the collection of actions that the first agent is about to execute; for example, the action instruction set may include a pathfinding method, movement direction, posture type, etc.
  • the first prediction model includes a feature extraction module, a pathfinding prediction module, and multiple action prediction modules. Based on this, referring to Figure 4, the above step S102 can be implemented through the following steps S1021 to S1024, which are described in detail below.
  • step S1021 the feature extraction module is used to extract features from the state data to obtain state features.
  • the state data includes environmental information, scalar information, depth map information, and ring ray information
  • the environmental information, scalar information, depth map information, and ring ray information are information of different modalities.
  • the feature extraction module includes an encoding unit and a feature extraction unit.
  • environmental information refers to various information describing the current game environment.
  • Environmental information typically includes various features, attributes, and states of the game scene, and may also include various factors affecting the game progress and the decisions of the first agent.
  • Scalar information refers to simple, single numerical information describing the game state, usually a specific numerical value. Exemplary scalar information could be health points, score, time, resource quantity, distance, etc.
  • Depth map information refers to image information describing the distance between objects in the game scene and the camera or viewpoint. Depth map information can provide information about the distance and spatial position of objects in the game world. Circular ray information is commonly used in agent perception systems to help the first agent understand the surrounding environment, such as detecting obstacles, obtaining target positions, and identifying second agents.
  • step S1021 can be implemented by the following steps S0211 to S0213, which are described in detail below.
  • step S0211 the encoding unit is used to encode the environmental information, scalar information, depth map information and ring ray information respectively, so as to obtain the encoded environmental information, encoded scalar information, encoded depth map information and encoded ring ray information.
  • the encoding unit may include four encoding subunits, namely the first encoding subunit, the second encoding subunit, the third encoding subunit, and the fourth encoding subunit.
  • the first encoding subunit corresponds to environmental information and is used to encode the environmental information to obtain encoded environmental information.
  • the first encoding subunit can be an embedding model that learns the relationships and patterns of environmental information and maps the environmental information to an embedding vector, thus transforming the environmental information into a more compact and efficient representation.
  • the second encoding subunit corresponds to scalar information and is used to encode the scalar information to obtain encoded scalar information. Similar to the first encoding subunit, the second encoding subunit can also be an embedding model.
  • the third encoding subunit corresponds to depth map information and is used to encode the depth map information to obtain encoded depth map information.
  • the third encoding subunit can be a two-dimensional convolutional (Conv2D) model.
  • the fourth encoding subunit corresponds to annular ray information and is used to encode the annular ray information to obtain encoded annular ray information.
  • the fourth encoding subunit can be a one-dimensional convolutional (Conv1D) model.
  • step S0212 the encoded environmental information, encoded scalar information, encoded depth map information, and encoded annular ray information are spliced together to obtain comprehensive encoded information.
  • splicing the encoded environmental information, encoded scalar information, encoded depth map information, and encoded ring ray information means combining the encoded environmental information, encoded scalar information, encoded depth map information, and encoded ring ray information to obtain comprehensive encoded information.
  • the combination can be performed according to the order of the encoded environmental information, encoded scalar information, encoded depth map information, and encoded ring ray information; or it can be performed according to the order of the encoded scalar information, encoded environmental information, encoded depth map information, and encoded ring ray information.
  • step S0213 the feature extraction unit is used to extract features from the comprehensive coding information to obtain state features.
  • the feature extraction unit can be a deep neural network; for example, it can be a long short-term memory artificial neural network.
  • the feature extraction unit encapsulates and fuses the comprehensive encoded information to obtain state features. Through the feature extraction unit, long-term dependencies in the comprehensive encoded information can be captured.
  • the information of different modalities included in the state data is encoded separately. This allows for better capture of the characteristics of each modality, thereby improving the accuracy of data processing and analysis. It also preserves the detailed information of each modality, making the data more comprehensive and richer during transmission and parsing.
  • the feature extraction unit integrates information from different modalities, which can enhance the interaction and information integration between modalities. By extracting information-rich state features, these state features can accurately reflect the state of the game, providing an accurate data foundation for model training and thus improving the accuracy of the trained first prediction model.
  • step S1022 for each action prediction module, the action prediction module is used to predict based on state features to obtain the action information of the first intelligent agent.
  • the action prediction module corresponds to other actions that the first intelligent agent can perform.
  • Other actions refer to actions other than pathfinding. Since this embodiment describes pathfinding and other actions separately, the number of action prediction modules is one less than the number of actions that the first intelligent agent can perform. For example, assuming that the first intelligent agent can perform 9 actions, there are 8 action prediction modules.
  • the action prediction module is used to predict actions based on state features to obtain action information of the first agent, which is used to instruct the first agent to perform the action.
  • step S1022 can be achieved through the following steps S0221 to S0225, which will be explained in detail below.
  • step S0221 the second action identifier corresponding to the action prediction module is obtained.
  • each action has a corresponding action identifier. Therefore, the action prediction module also has a second action identifier for the predicted action.
  • step S0222 based on the action sequence table, the third action identifier that precedes the second action identifier is determined.
  • the action sequence table is used to represent the order of actions.
  • the action sequence table is a pre-set table that stores the action identifier of each action sequentially.
  • the action sequence table can represent the order of actions, and the order of actions reflects the dependencies between them.
  • the action identifiers stored in the action sequence table are 01, 02, 03, 04, 05, 06, 07, 08, and 09.
  • 01 corresponds to the horizontal aiming direction, 02 to the vertical aiming direction, 03 to whether to fire, 04 to the movement mode, 05 to the pathfinding mode, 06 to the movement direction, 07 to the posture type, 08 to the side-stepping type, and 09 to the special type. Therefore, the order of actions is: horizontal aiming direction, vertical aiming direction, whether to fire, movement mode, pathfinding mode, movement direction, posture type, side-stepping type, and special type.
  • the third action identifier refers to the action identifier that is located before and adjacent to the second action identifier. Continuing with the example above, assuming that the second action identifier is 03, then the third action identifier is 02.
  • a preset action sequence table may be obtained before performing step S0222.
  • step S0223 the second action information corresponding to the third action identifier is obtained.
  • the second action information is predicted using the action prediction module corresponding to the third action identifier.
  • the third action is identified as 02, and the second action information is predicted using the action prediction module corresponding to 02.
  • step S0224 the second action information is encoded to obtain the second encoded action information.
  • the second action information can be encoded by embedding the second action information, and the resulting embedding vector can be determined as the second encoded action information.
  • step S0225 the action prediction module corresponding to the second action identifier is used to predict the action information of the first intelligent agent based on the state features and the second encoded action information.
  • the second action is predicted not only based on state features but also based on the second encoded action information of the preceding action, thereby obtaining the action information corresponding to the second action.
  • the action prediction module predicts the action information of the second action, it is based on the state information and the second encoded action information of the preceding action.
  • the action prediction module can better understand the context information of the action occurrence, realize that the dependency relationship between actions has been incorporated when making prediction, and can ensure the continuity between the preceding action and the second action, thereby obtaining a highly accurate prediction result, that is, obtaining highly accurate action information, so as to provide an accurate data foundation for model training, thereby improving the accuracy of the trained first prediction model.
  • action information for each action other than the pathfinding action can be determined based on the steps S0221 to S0225 described above.
  • step S1023 the pathfinding prediction module is used to predict based on state features to obtain the pathfinding method of the first intelligent agent.
  • the path prediction module is used to predict the path prediction method.
  • step S1023 can be achieved through the following steps S0231 to S0235, which will be explained in detail below.
  • step S0231 a preset action sequence table is obtained.
  • the action sequence table is used to represent the order in which actions are arranged.
  • the action sequence table can be obtained by obtaining instructions.
  • step S0232 based on the action sequence table, the first action identifier preceding the pathfinding action identifier is determined.
  • the implementation process of step S0232 is similar to that of step S0222. Therefore, the implementation process of step S0232 can refer to the implementation process of step S0222.
  • the first action identifier preceding the pathfinding action identifier refers to the action identifier that precedes and is adjacent to the pathfinding action identifier. Continuing with the above example, if the pathfinding action identifier is 05, then the first action identifier is 04.
  • step S0233 the first action information corresponding to the first action identifier is obtained.
  • the first action information is predicted using the action prediction module corresponding to the first action identifier.
  • step S0233 is similar to that of step S0223. Therefore, the implementation process of step S0233 can refer to the implementation process of step S0223.
  • step S0234 the first action information is encoded to obtain the first encoded action information.
  • step S0234 is similar to that of step S0224. Therefore, the implementation process of step S0234 can refer to the implementation process of step S0224.
  • step S0235 the pathfinding prediction module uses state features and first encoded action information to predict the pathfinding method of the first intelligent agent.
  • the pathfinding prediction module when predicting the pathfinding method, it is based not only on state features, but also on the first encoded action information, that is, on the encoded action information of the preceding action. In this way, the pathfinding prediction module can predict the impact of subsequent actions on path planning based on the analysis of the preceding actions, thereby optimizing the pathfinding strategy and obtaining a more efficient or state-feature-compliant pathfinding method, thus improving the accuracy of the pathfinding method.
  • step S0235 can be achieved through the following steps S2351 to S2353, which will be explained in detail below.
  • step S2351 according to the prediction cycle, the pathfinding prediction module is used to predict the state features and the first encoded action information to obtain the target pathfinding index.
  • the target pathfinding index is one of a first pathfinding index and a second pathfinding index, wherein the first pathfinding index and the second pathfinding index are different pathfinding indexes, and the first pathfinding index and the second pathfinding index represent different pathfinding methods.
  • the first pathfinding index represents a pathfinding method based on atomic movement
  • the second pathfinding index represents a pathfinding method based on rule-based movement.
  • the prediction period is a pre-set value; for example, the prediction period can be 6 seconds, 7 seconds, or 8 seconds.
  • the path prediction module predicts a target path index, which indicates the target pathing method.
  • step S2352 the target pathfinding method corresponding to the target pathfinding index is determined based on the preset index mapping table.
  • the index mapping table stores the mapping relationship between routing indexes and routing methods. Based on this, the target routing method corresponding to the target routing index can be determined from the index mapping table.
  • step S2353 the target pathfinding method is determined as the pathfinding method of the first intelligent agent.
  • the pathfinding method of the first agent is determined to be a rule-based pathfinding method, also known as a rule-based movement pathfinding method.
  • the pathfinding method of the first agent is determined to be an atomic movement-based pathfinding method.
  • Rule-based pathfinding refers to a path planning strategy that relies on predefined rules or instructions to guide mobile entities (such as robots, autonomous vehicles, etc.) from a starting position to a target position in an environment. These predefined rules can be obstacle avoidance rules, priority rules, greedy search rules, etc.
  • Atom-based pathfinding is a path planning strategy at the microscopic level. It treats mobile entities (such as robots, vehicles, etc.) as freely movable "atoms" and calculates the path based on their physical properties or dynamic models.
  • steps S0231 to S0235 when predicting the pathfinding mode using the pathfinding prediction module, it is based on state information and the first encoded action information. In this way, the dependency between the pathfinding action and the first action is decoupled, and the dependency between the pathfinding action and the first action is already incorporated into the prediction, thereby obtaining a more accurate pathfinding mode.
  • This provides more accurate training data for model training, which not only speeds up the convergence speed, reduces training time and iterations, and improves training efficiency, but also reduces the risk of the model learning noise and irrelevant features, thus avoiding overfitting.
  • training the model on more accurate training data can better identify and handle abnormal situations, enhance the robustness of the model, and thus improve the prediction accuracy and generalization ability of the model.
  • step S1024 an action instruction set is constructed based on the pathfinding method and various action information.
  • the action instruction set includes action information for all actions, and the action instruction set is used to instruct the first intelligent agent to perform the corresponding action.
  • step S1024 can be achieved through the following steps S0241 to S0243, which will be explained in detail below.
  • step S0241 a preset motion mask is obtained.
  • the action mask is matched with the interaction rules in the virtual scene, and the action mask is used to limit the action selection space of the first agent.
  • the interaction rules can be the game rules.
  • action masking can be a mechanism used to limit the range of actions a first agent can perform in a given state, helping the first agent make more appropriate decisions in a specific situation.
  • Action masking can be viewed as a binary vector, where each element corresponds to an action. When an action mask value is 1, it indicates that the action is executable; when the mask value is 0, it indicates that the action is not executable.
  • the interaction rules first need to be defined, including which actions are permissible and which actions are prohibited under specific circumstances. For example, in a board game, the rules might prohibit the king from moving to a position where it would be checkmated. Then, based on the game rules, an action space is created, containing all possible actions the agent can take. During a game, based on the current game state, it is determined which actions are allowed and which are not. The mask values for allowed actions are set to 1, and the mask values for prohibited actions are set to 0, thus obtaining the action mask.
  • step S0242 the pathfinding method and each action information are constructed into a candidate instruction set.
  • the pathfinding method and various action information are combined to form a candidate instruction set.
  • step S0243 the candidate instruction set is masked using an action masking method to obtain an action instruction set.
  • the action information includes a first flag value indicating whether an action is performed.
  • the first flag value can be a first preset value or a second preset value. When the first flag value is the first preset value, it indicates that the action corresponding to the action information is not performed. When the first flag value is the second preset value, it indicates that the action corresponding to the action information is performed.
  • the first preset value can be 0, and the second preset value can be 1.
  • the action mask can be a one-dimensional vector, where each vector value is a mask value, and each mask value corresponds to an action information. The mask value is either the first preset value or the second preset value. When the mask value is the first preset value, it indicates that the action information corresponding to the mask value needs to be masked. When the mask value is the second preset value, it indicates that the action information corresponding to the mask value does not need to be masked.
  • the mask value corresponding to the action information is obtained from the action mask.
  • the mask value corresponding to the action information is a first preset value and the first flag value in the action information is a second preset value
  • the first flag value in the action information is updated to the first preset value. That is, when the mask value corresponding to an action information is 0, it means that the action corresponding to the action information needs to be masked, that is, the action corresponding to the action information is not executed. Then it is necessary to further determine whether the first flag value in the action information of the action is 1.
  • the first flag value is 1, it indicates that the action is executed, so the first flag value in the action information of the action is updated to 0, thus not executing the action. If the first flag value in the action information is not 1, that is, 0, it means that the action information itself indicates that the action is not executed, and the action information remains unchanged.
  • the mask value corresponding to an action information is a second preset value, such as a mask value of 1, it means that the action information is not masked, and the action information can remain unchanged.
  • an action instruction set is obtained.
  • action masking is used to filter and select pathfinding methods and various action information, thereby obtaining a set of action instructions. This constrains the action space, making the action instructions more consistent with the actual situation or task requirements; it also optimizes the behavior strategy, thereby improving the decision-making efficiency and performance of the first agent, shortening the model training time, and improving the efficiency and accuracy of model training.
  • steps S1021 to S1024 different modal information is encoded separately, thus accurately representing the state of the game.
  • the feature extraction unit integrates different modal information to obtain information-rich state features. Then, the dependencies between actions are incorporated into the prediction, resulting in highly accurate prediction results.
  • action masking is used to filter and select pathfinding methods and individual action information, thereby constraining the action space and optimizing the behavior strategy, thus improving the decision-making efficiency and performance of the first agent and enhancing the efficiency and accuracy of model training.
  • step S103 the first intelligent agent is controlled to move based on the pathfinding method and to perform corresponding actions based on each action information to obtain the execution result.
  • the first intelligent agent is controlled to execute various actions in the action instruction set in order to change the game environment and thus obtain the execution result.
  • the "controlling the first agent to move based on the pathfinding method and execute corresponding actions based on each action information" in step S103 can be implemented through the following steps S1031 to S1036, which will be explained in detail below.
  • step S1031 the target location and the current location of the first agent are obtained.
  • the target location can be obtained in advance through a preset method, such as a fixed location or a specific object location.
  • the target location can also be a randomly generated location, thus increasing the randomness and challenge of the game.
  • the target location can also be a location obtained based on environmental information or task settings.
  • the current position refers to the current location of the first intelligent agent.
  • step S1032 the movement path of the first intelligent agent is determined based on the target location and the current location.
  • a path planning algorithm can be used to determine the movement path based on the target location and the current location.
  • the path planning algorithm can be an ant colony algorithm or a Dijkstra algorithm.
  • map data of the virtual scene where the first agent is located is acquired, including multiple nodes.
  • the distance values of all nodes are set to infinity (indicating that the distance from the starting node to that node is unknown).
  • the node corresponding to the current position is determined as the starting node, and its distance value is set to 0 (indicating that the distance from the starting node to itself is 0).
  • a priority queue is created, and all nodes are added to this queue, with the starting node having the highest priority (smallest distance).
  • the unvisited node closest to the starting node is taken from the priority queue and determined as the current node.
  • all unvisited neighboring nodes of the current node are traversed. For each neighboring node, the distance from the starting node to that node is calculated. If the calculated distance is less than the currently recorded distance of the neighboring node, the distance of the neighboring node is updated, and the node is added to the priority queue. The current node is marked as visited to prevent it from being processed again. This process of selecting the nearest unvisited node, updating the distance of neighboring nodes, and marking the current node as visited is repeated until the target node is marked as visited or the priority queue is empty (all nodes have been visited).
  • the target node corresponding to the target location is visited, starting from the target node, based on the predecessor node of each recorded node (the node pointing to it in the shortest path tree), backtrack to the starting node to obtain the path planning result, which is the movement path of the first agent.
  • step S1033 based on the movement path, the first intelligent agent is controlled to move according to the rule-based pathfinding method; and based on the action information, the first intelligent agent is controlled to execute each action.
  • the first intelligent agent is controlled to move according to a movement path, and the first intelligent agent is also controlled to perform various actions based on various action information.
  • step S1034 during the process of controlling the first intelligent agent to move according to the rule-based pathfinding method, when it is determined that the pathfinding method switching condition is met, the target pathfinding index is updated to the first pathfinding index.
  • the target pathfinding index can be updated to the third pathfinding index.
  • the third pathfinding index is used to represent keeping the pathfinding method unchanged or keeping the movement method unchanged, that is, keeping the movement path unchanged, and moving according to the rules based on the movement path, without needing to determine the movement path again.
  • the target routing index when it is determined that the routing mode switching condition is met, can be updated to the fourth routing index first.
  • the fourth routing index is used to represent pausing the rule-based routing mode, thereby pausing the current routing mode in a timely manner. Then, the target routing index is updated from the fourth routing index to the first routing index.
  • the methods for determining whether the pathfinding mode switching conditions are met based on received game data include the following three implementation methods:
  • the pathfinding mode switching condition is met.
  • an occluded scene indicates that there is an obstruction between the first intelligent agent and the second intelligent agent
  • an unoccluded scene indicates that there is no obstruction between the first intelligent agent and the second intelligent agent.
  • the first position coordinates of the first agent and the second position coordinates of the second agent are first obtained. Based on the first and second position coordinates, a straight line between the first and second agents is determined, and ray detection is performed along this line to determine whether the ray intersects with objects in the game scene. If the ray intersects with an object, it indicates that there is an obstruction between the first and second agents; if the ray does not intersect with any object, it indicates that there is no obstruction between them.
  • the absence of obstructions between the first and second agents means they can see each other, signifying a transition from a non-combat scenario to a combat scenario. At this point, the game data satisfies the pathfinding switching conditions, providing path assurance for the battle.
  • the atomic movement pathfinding method allows the first agent to quickly respond to the combat scenario.
  • the third implementation method involves obtaining the movement time of the first intelligent agent according to the rule-based pathfinding method.
  • the movement time reaches the duration threshold, it is determined that the pathfinding method switching condition is met.
  • the duration threshold is less than the prediction period.
  • the target position and movement path do not change or update during the movement time.
  • the game data is determined to meet the pathfinding switching conditions, and the target pathfinding index is updated to the first pathfinding index.
  • the duration threshold is a pre-set value; for example, the duration threshold can be 3 seconds or 4 seconds.
  • step S1035 the pathfinding mode of the first agent is switched to a pathfinding mode based on atomic movement.
  • the pathfinding method of the first agent is switched to the pathfinding method based on atomic movement.
  • step S1036 the first intelligent agent is controlled to perform atomic movement.
  • the first agent is controlled to make atomic movements for the remainder of the prediction period until the prediction period ends.
  • the movement path is first determined, and then the first agent is controlled to move according to the rule-based pathfinding method based on the movement path. In this way, only one movement path needs to be determined, thereby saving computing resources, shortening the model training time, and improving the model training efficiency.
  • the target pathfinding index is updated to the first pathfinding index, and the first agent is controlled to perform atomic movement until the prediction period ends.
  • the pathfinding method is switched in a timely manner when the pathfinding method switching condition is met, which not only improves the flexibility and variability of the pathfinding method, but also improves the adaptability of the pathfinding method to the scene in which the first agent is located, thereby improving the accurate training data for model training and thus improving the prediction accuracy of the model.
  • the first agent's pathfinding mode is a pathfinding mode based on atomic movement
  • the first agent is controlled to perform atomic movement throughout the entire prediction period.
  • step S104 the parameters of the first prediction model to be trained are adjusted based on the reward information of multiple execution results to obtain the trained first prediction model.
  • the reward information is determined based on multiple execution results, the main reward coefficient, and the auxiliary reward coefficient.
  • the parameters of the first prediction model to be trained can be adjusted based on the reward information of each execution result, or the parameters of the first prediction model to be trained can be adjusted based on the sum of the reward information of each execution result.
  • step S104 can be achieved through the following steps S1041 to S1043, which will be explained in detail below.
  • step S1041 the reward function is determined based on the primary reward coefficient and the secondary reward coefficient.
  • the primary reward coefficient represents the reward information assigned when the first agent executes the action corresponding to the action information in the action instruction set
  • the secondary reward coefficient represents the reward coefficient assigned when the first agent executes a preset anthropomorphic action. Both the primary and secondary reward coefficients can be pre-set.
  • the preset anthropomorphic actions can be actions such as actively seeking cover, firing and pulling, or pulling away.
  • R represents reward information
  • ⁇ represents the primary reward coefficient
  • ⁇ represents the secondary reward coefficient
  • R ⁇ sub> main ⁇ /sub> represents the primary reward value
  • R ⁇ sub>auxiliayr ⁇ /sub> represents the secondary reward value
  • Primary reward values are typically closely related to the primary agent's objective of completing its main task. For example, if the agent's primary task is to reach a destination, the primary reward value obtained upon reaching the destination will be relatively high. Primary reward values can be set based on task completion, efficiency, or other relevant metrics. Secondary reward values are rewards derived from secondary objectives or behaviors related to the primary task. Examples include avoiding enemies, maintaining health, and conserving resources. Secondary reward values can help the agent develop more comprehensive and stable strategies during the learning process.
  • step S1042 the reward information for each execution result is determined using a reward function.
  • the execution result refers to the state or feedback obtained by the first agent after performing movement and corresponding actions based on a pathfinding method.
  • the execution result may include at least one of the following: the first agent's current position, the interaction result between the first agent and objects or elements in the virtual environment (such as collecting items, attacking enemies, etc.), the change in the environment after the agent performs the action (such as the state of enemies, the state of objects, etc.), and the game score after the agent performs the action.
  • the primary reward value and the secondary reward value are first determined based on the execution result, and then the primary reward value and the secondary reward value corresponding to the execution result are substituted into formula (1) to obtain the reward information of the execution result.
  • This reward information can be the total reward value.
  • step S1043 the reward information of each execution result is accumulated to obtain the cumulative reward.
  • the cumulative reward is backpropagated to the first prediction model to be trained, and the parameters of the first prediction model to be trained are adjusted based on the cumulative reward until the training termination condition is met, thus obtaining the trained first prediction model.
  • the reward information of the execution results within one or more prediction periods can be accumulated to obtain a cumulative reward. Then, the parameters of the first prediction model to be trained are adjusted based on the cumulative reward. When the preset number of adjustments is reached or the cumulative reward reaches the reward threshold, the training termination condition is determined to be reached, thereby obtaining the trained first prediction model.
  • the first prediction model to be trained can be trained in the direction of a preset anthropomorphic action, so that the trained first prediction model can control the first intelligent agent to have an anthropomorphic style while ensuring the strength of the ability, thereby improving the prediction performance of the trained first prediction model.
  • the trained first prediction model can predict based on the state data of the first agent to obtain a set of action instructions. Then, the first agent is controlled to move based on the set of action instructions and execute corresponding actions based on the action information in the set of action instructions, thereby interacting with a virtual object.
  • the virtual object can be a game character that fights against the first agent.
  • the virtual object can be a game character controlled by a real player or a second agent driven by a second prediction model.
  • the first prediction model to be trained can control the first agent to move according to different pathfinding methods based on different scenarios. Therefore, when the first agent is in different scenarios during a game, it can predict a set of action commands containing different pathfinding methods. The pathfinding methods in this set of action commands match the scenario in which the first agent is located, thus achieving diversification of pathfinding methods.
  • the trained second prediction model can also control the second agent to move according to different pathfinding methods based on different scenarios. That is, the second agent can also execute different pathfinding methods in different scenarios, thus strengthening the ability of the first agent in the adversarial game to execute different pathfinding methods in different scenarios.
  • the first prediction model to be trained can be trained towards a preset anthropomorphic action.
  • the first initial position of the first agent and the second initial position of the second agent are determined first, and the first agent is generated at the first initial position, and the second agent is generated at the second initial position.
  • steps S001 to S009 can also be executed before step S101, which will be explained in detail below.
  • step S001 in response to the game start command, the selected area is determined from each sub-area covered by the virtual scene corresponding to the game, based on the principle of equal probability.
  • the command to start a game can be in the form of touch, gesture, voice, etc.
  • the probability weights of each sub-region can be set to the same value, and each sub-region can be associated with a number. Then, a random number generator can be used to generate a random number, and the sub-region associated with the random number can be determined as the selected region.
  • step S002 the first center point of the selected area is determined.
  • the coordinates of the first center point of the selected area can be determined by methods such as the geometric center method, the centroid method, the point set method, and the distance weighted method.
  • step S003 based on the first center point and the preset first maximum pathfinding distance, the first birth region is determined, and the first set of available points in the first birth region is determined.
  • a circular first spawn area can be determined with the first center point as the center and a preset first maximum pathfinding distance as the radius; alternatively, a square first spawn area can be determined with the first center point as the center and a preset first maximum pathfinding distance as the side length.
  • first navigable points refer to valid locations or areas that the first agent can move to. These points are typically target points that the first agent can consider when planning and navigating during a game.
  • the determination of first navigable points is usually based on the following aspects:
  • terrain features Available points are typically located in flat, passable terrain areas on the game map, such as flat land, roads, stairs, etc. These areas are usually where the first agent can move and operate safely.
  • the first available point usually needs to avoid obstacles or areas composed of obstacles on the game map, such as walls, buildings, and water bodies.
  • obstacles or areas composed of obstacles on the game map such as walls, buildings, and water bodies.
  • the first agent needs to consider how to bypass these obstacles and find a safe path.
  • target points The first available points usually also include target points or target areas in the game, such as mission objectives, enemy locations, resource points, etc.
  • the first agent can find the optimal path to reach these target points through path planning.
  • the first available points can also include points or areas where players can interact, such as shops, NPC locations, and vehicles.
  • the first intelligent agent can select appropriate paths and destinations based on the player's needs and behaviors.
  • the first agent can be provided with effective movement targets and path planning references, thereby enabling it to exhibit more intelligent and flexible behavior in the game.
  • all first available points constitute a first available point set.
  • step S004 the first initial position of the first agent is determined from the first set of available points, and the first agent is generated at the first initial position.
  • any first available point in the first available point set can be randomly determined as the first initial position, and a first agent can be generated at the first initial position, that is, the first initial position is the birth position of the first agent.
  • step S005 the first initial position is determined as the second center point.
  • step S006 the preset second maximum pathfinding distance and the preset second minimum pathfinding distance corresponding to the second agent are obtained.
  • the preset second maximum pathfinding distance and the preset second minimum pathfinding distance can be preset values.
  • step S007 based on the second center point, the preset second maximum pathfinding distance, and the preset second minimum pathfinding distance, the second birth region is determined, and the second set of available points in the second birth region is determined.
  • the second center point can be used as the center of a circle
  • the preset second maximum pathfinding distance can be used as the outer radius of the circle
  • the preset second minimum pathfinding distance can be used as the inner radius of the circle to determine an annulus
  • the annulus can be defined as the second birth area
  • the process of determining the second set of available points in the second birth region is similar to the process of "determining the first set of available points in the first birth region" in step S003 above. Therefore, the process of determining the second set of available points in the second birth region can refer to the process of "determining the first set of available points in the first birth region" in step S003 above.
  • step S008 candidate initial positions are determined from the second set of available points.
  • any second available point in the second available point set may be randomly selected as a candidate initial position.
  • step S009 when there is an obstruction between the candidate initial position and the first initial position, the candidate initial position is determined as the second initial position of the second agent, and the second agent is generated at the second initial position.
  • the candidate initial position is determined as the second initial position of the second agent, and the second agent is generated at the second initial position.
  • the process when there is no obstruction between the candidate initial position and the first initial position, the process returns to step S008 above to redetermine the candidate initial position.
  • the randomness of the generation of the first and second agents can be improved, the probability of each region being selected can be increased, and the complete symmetry between the first and second agents within the selected region can be ensured; on the other hand, by having obstacles between the two spawn positions, the diversity of samples can be guaranteed, and the possibility of one side being killed by the other side at the start of the game can be avoided, which can increase the game time, ensure sufficient training samples, and thus improve the model training efficiency.
  • This application proposes a model training method based on a large-scale reinforcement learning training framework, combined with a proximal policy optimization algorithm.
  • a rule-enhanced hybrid pathfinding scheme is proposed by combining atomic movement and navigation grid movement.
  • the powerful adaptive learning and exploration capabilities of neural networks are utilized to realize a training scheme for a companion AI that can be deployed in PVE and PVP scenarios on ultra-large maps and complex and diverse terrains.
  • the companion AI corresponds to the first intelligent agent in other embodiments.
  • This application's embodiments based on the original action space, additionally introduce navigation grid movement as global movement logic and adopt a model architecture of centralized training and hybrid execution.
  • the AI focuses on learning precise positioning; the neural network prioritizes atomic movement methods to learn extremely small operations, enabling it to perform actions similar to human players and improve its human-likeness.
  • the neural network de-emphasizes pathfinding learning and prefers global movement via the navigation grid to quickly encounter target enemies and switch to the combat scenario. This allows for the generation of more combat samples in each game, improving training efficiency.
  • the target enemy corresponds to the second intelligent agent in other embodiments.
  • a comprehensive auxiliary reward system is introduced to enhance the human-like nature of the strategy.
  • self-play training is used to continuously improve the strength of the strategy.
  • the hybrid pathfinding reinforcement learning training scheme can fully leverage the decision-making and exploration advantages of reinforcement learning, adaptively and flexibly selecting the pathfinding method according to the battle situation. It not only takes into account both full-map pathfinding and combat game capabilities, but also improves training efficiency. It can truly produce high-level companion AI that meets business requirements in ultra-large maps and complex and diverse 3D terrain scenarios, helping business teams to generate content more efficiently.
  • the model training method provided in this application can achieve the following four objectives:
  • the first objective is to accelerate the collection of training data for policy learning through large-scale distributed reinforcement learning.
  • the second objective is to avoid the problem of AI characters getting stuck in a certain location or unable to move as expected due to terrain design by using a rule-enhanced hybrid pathfinding approach combined with centralized reinforcement learning training.
  • the third objective is to train a AI capable of both full-map navigation and combat strategy in ultra-large map environments and complex, varied 3D terrain scenes, not only increasing the upper limit of model performance but also significantly accelerating training efficiency.
  • the fourth objective is to further accelerate model adaptation and deployment by enhancing the model's anthropomorphism and stylistic guidance through a comprehensive reward and punishment mechanism.
  • the model in question corresponds to the first prediction model to be trained in other embodiments.
  • the model training method provided in this application will be described from three aspects.
  • the business objective is the task background upon which this application's embodiments are based, requiring a clear understanding of the AI's tasks and specific roles.
  • the environmental interaction module is universal; this training and inference model can be used in any network game employing state synchronization.
  • the training scheme proposes a differentiated training scheme. The introduction of a hybrid pathfinding method accelerates training efficiency and increases the model's upper limit. A highly integrated neural network and a complete reward and punishment system are conducive to guiding the model towards a human-like style, thus improving the model's anthropomorphism.
  • the farm map is the most active map for players, taking it as an example, existing technologies have deployed behavior tree-based AI for casual play and filling match counts.
  • the logic of these existing behavior tree-based AIs is easily recognized by players, it diminishes the enjoyment of the game. Therefore, the business objective is to produce a high-intensity, highly human-like AI using reinforcement learning technology for use in high-level matches.
  • This AI should be able to move freely across the entire map, effectively utilize cover in combat and kiting, move flexibly and naturally, exhibit human-like firing behavior, and have its elimination replays aligned with the human player's perspective. Its gameplay should demonstrate high levels of human-likeness and diversity, truly achieving the goal of highly human-like casual play, thereby boosting game activity and player engagement.
  • the farm map is divided into eight key combat zones: 1301, 1302, 1303, 1304, 1305, 1306, 1307, and 1308. At the start of each game, one of these combat zones will be selected with equal probability to increase the probability of the main event occurring.
  • the two opposing sides (i.e., the first agent and the second agent) within each battle zone are completely symmetrical, and their spawn points are selected according to specific logic.
  • obstacles are required between their spawn points to ensure sample diversity.
  • a battle zone is randomly selected.
  • a combination of spawn points within it is further randomly selected. The game ends when one side dies or the match times out, thereby improving training efficiency.
  • Figure 14A is a flowchart illustrating the logic for determining the battle configuration provided in an embodiment of this application, including the following steps S1401 to S1410, which are described in detail below.
  • step S1401 the selected combat zone is determined.
  • the selected combat zone is chosen from the eight key combat zones according to the principle of equal probability.
  • the selected combat zone corresponds to the selected area in other embodiments.
  • step S1402 the coordinates of the center point of the selected combat zone are determined.
  • the coordinates of the center point of the selected combat zone can be determined by methods such as the geometric center method, the centroid method, the point set method, and the distance weighted method.
  • step S1403 the maximum pathfinding distance of the first agent is obtained.
  • the maximum pathfinding distance of the first intelligent agent at birth is set, that is, the maximum pathfinding distance of the first intelligent agent at birth is a preset value.
  • step S1404 the first birth region is determined.
  • the first spawn area is determined by using the center point coordinates as the center and the maximum pathfinding distance of the first agent as the radius. See Figure 14B, where 401 is the selected combat zone.
  • step S1405 available points of the navigation grid are sampled within the first birth area to obtain the birth coordinates of the first agent.
  • the birth coordinates of the first intelligent agent can be recorded as the first birth point.
  • 402 is the first birth point.
  • the implementation process of step S1405 is similar to the implementation process of "determining the first set of available points in the first birth region" in step S003 and "determining the first initial position of the first agent from the first set of available points" in step S004. Therefore, the implementation process of step S1405 can refer to the implementation process of "determining the first set of available points in the first birth region" in step S003 and "determining the first initial position of the first agent from the first set of available points" in step S004.
  • step S1406 the minimum pathfinding distance and the maximum pathfinding distance at birth of the second agent are obtained.
  • the minimum pathfinding distance and the maximum pathfinding distance of the second agent at birth are also preset values.
  • step S1407 the second birth region is determined.
  • a ring is defined with the birth coordinates of the first agent as the center, the minimum pathfinding distance of the second agent as the inner radius, and the maximum pathfinding distance of the second agent as the outer radius. This ring is then designated as the second birth region. Referring again to Figure 14B, 403 represents the second birth region.
  • step S1408 available points of the navigation grid are sampled within the second birth area to obtain sampling points.
  • the implementation process of step S1408 is similar to the implementation process of "determining the second set of available points in the second spawn region" in step S007 and "determining the candidate initial position from the second set of available points” in step S008. Therefore, the implementation process of step S1408 can refer to the implementation process of "determining the second set of available points in the second spawn region" in step S007 and "determining the candidate initial position from the second set of available points” in step S008.
  • step S1409 it is determined whether there is occlusion between the sampling point and the birth coordinates of the first agent.
  • step S1410 when there is occlusion between the sampling point and the spawn coordinates of the first agent, the phenomenon of one side being instantly eliminated at the start of the game can be avoided, indicating that the sampling point can be used as the spawn coordinates of the second agent, and then proceed to step S1410.
  • the phenomenon of one side being instantly eliminated at the start of the game will occur, indicating that the sampling point cannot be used as the spawn coordinates of the second agent, and then return to step S1408 to continue sampling.
  • step S1410 the sampling point is determined as the birth coordinates of the second agent.
  • the birth coordinates of the second agent can be denoted as the second birth point.
  • 404 is the second birth point.
  • the second aspect is environmental interaction.
  • FIG 15 is a schematic diagram of the overall structure of the model training phase provided in this embodiment.
  • the game client 1501 refers to the game environment that integrates the AI Service SDK for interface calls.
  • the game client sends a start-up request to the AI server, which returns the corresponding battle configuration data based on training requirements.
  • the game client constructs a game and generates a reinforcement learning agent.
  • Each subsequent request and response game frame sample is processed into state data 1502 and sent to the reinforcement learning framework 1503 for interaction.
  • the AI training service performs action prediction on the current game's request frame, concatenates network actions into executable action instructions 1504, and sends the executable action instructions 1504 back to the client's game engine for execution.
  • the AI training service continues to wait for the next game frame sample request and response.
  • business-side statistical indicators are periodically reported to the training platform to monitor training effectiveness.
  • the game client calls the relevant logic interface to destroy the agent for that game and automatically starts the next game with the same configuration. This process is repeated to collect data and provide it to the reinforcement learning framework for training.
  • the model training phase employs a synchronous inference mode, as shown in Figure 16.
  • Each game during the training phase has no human players, no AI business behavior trees, and no irrelevant game business logic. Only two training AIs are running in the game kernel (DS).
  • the game business logic is simple and pure, requiring minimal server performance for collision and ray detection.
  • the game kernel is in a synchronous waiting state. State request 03 is sent from game kernel 01 to AI training service 1602, and action response packet 04 is returned from AI training service 1602 to game kernel 01. AI training service 1602 corresponds to the first prediction model to be trained in other embodiments.
  • the game kernel logic only continues to run after receiving the action response packet.
  • the game kernel during the training phase is a continuous loop of state request 03 and action response packet 04, with an action request made every n game frames.
  • the synchronous inference method during model training allows the game kernel's own game business logic and reinforcement learning training to run on the same timeline.
  • the value of n ranges from 2 to 10.
  • the model deployment phase adopts an asynchronous inference mode, and the asynchronous inference business flow is shown in Figure 17.
  • a large number of online players exist simultaneously. Interaction logic arises between players and between players and various AIs.
  • the game kernel will perform a large amount of collision and ray detection each time it sends a game frame request response to build state data. This consumes most of the server performance. This server resource consumption is proportional to the increase in the number of AI deployed in each game after deployment. More seriously, because the game kernel is in a synchronous waiting state during each state request and action response in the synchronous inference mode, the game logic itself will not run during this period.
  • Inference service 1702 will pre-store a large amount of static collision data and perform ray detection queries on the status requests 03 sent by game kernel 01 each time, returning action response packets 04 to game kernel 01. This avoids the significant performance consumption of game kernel 01 caused by dynamically performing ray detection on the game kernel 01 side each time. Thus, only the logic of inference service 1702 needs to be focused on, improving development efficiency. Inference service 1702 corresponds to the trained first prediction model in other embodiments.
  • Asynchronous inference mode involves two timelines: the game kernel periodically sends basic state request data to the inference service every 4 frames; the inference service, following a first-in-first-out (FIFO) principle, performs static collision and raycasting queries on the received basic state data, processes it into network state data for inference calculations, and then continues to return action command packets to the game kernel according to the FIFO principle.
  • FIFO first-in-first-out
  • the game kernel after stably sending a state request response every 4 frames, does not need to wait for action responses and continues running its own game business logic. Simultaneously, it checks for action responses every frame; if one is found, it immediately executes the latest action.
  • the response speed may not be able to keep up with the packet sending speed.
  • the packet sending speed can be reduced by increasing the request interval frame number n, or the packet sending speed can be accelerated by optimizing the model structure to reduce inference time.
  • Asynchronous inference can prevent data from blocking the main thread, ensuring that factors such as neural network inference time and action response packet latency caused by network fluctuations do not affect other game business logic. It can also reduce the server load on the business side and ensure the security of model inference. If inference fails due to factors such as network fluctuations, the business logic can be switched to ordinary behavior tree AI to ensure the stable and normal operation of the game logic.
  • This hybrid reinforcement learning access logic is characterized by ease of use, flexibility, and security.
  • the game map is typically open and diverse, with varied gameplay.
  • the realistic 3D environment contains numerous static, dynamic, and environmentally relevant objects with highly detailed and complex terrain.
  • the game control panel also includes many simultaneous joystick actions.
  • Training a high-intensity AI capable of smooth movement across the entire map and effectively utilizing cover in combat within such a vast and complex game environment places high demands on the AI's ability to navigate the map smoothly and move across the entire map. Therefore, this application's embodiment employs a rule-enhanced hybrid pathfinding method, coupled with a comprehensive reward and punishment system, to perform self-playing iterative training in a two-player zero-sum game scenario, ultimately converging to a Nash equilibrium strategy. At this point, the model's strength and human-likeness meet the business objectives.
  • this embodiment employs a hybrid pathfinding scheme. It utilizes a rule-enhanced centralized training and hybrid execution approach, directly integrating the UE4 underlying navigation mesh pathfinding rules for global movement. Simultaneously, the original atomic movements are retained as local movements. To flexibly select between global and local movements, a hybrid pathfinding action head is added to the neural network. This action head determines which pathfinding method to use.
  • the advantages of this hybrid pathfinding scheme are:
  • the AI can flexibly choose between rule-based movement and atomic movement.
  • the network learns more rule-based movement, downplaying its pathfinding learning process.
  • the network learns more about using atomic movement for local positioning and micro-management, allowing the network to focus on learning the combat game process.
  • the AI has adaptive movement strategies for different scenarios, making it highly flexible and controllable, achieving truly smooth pathfinding and movement across the entire map, while also contributing to improved model strength.
  • Training games can generate more combat samples rather than pathfinding samples, which helps the network focus more on learning the combat tactics of mutual game-playing and pulling, thus accelerating training efficiency and saving training time.
  • the training process of the hybrid pathfinding scheme provided in this application embodiment includes: First, a three-finger manipulation modeling scheme is adopted to prune (or filter) a large number of illegal and invalid parallel actions by using the three-finger operation rules that represent the upper limit of human operation, so as to achieve a dual improvement in training efficiency and human-likeness.
  • the network action output header categories and meanings are shown in Table 1.
  • the Path type action header indicates whether to select a rule-based movement method or an atomic movement method.
  • the Path type action header also includes the action meaning of "maintain the previous movement method.” This is because the rule-based movement logic is as follows: first, determine the pathfinding target point; based on this target point, UE4 queries the optimal path to move to that target point; then, using the UE4's underlying navigation mesh, move to the pathfinding target point along the queried optimal path.
  • the pathfinding target point corresponds to the target location in other embodiments
  • the optimal path corresponds to the movement path in other embodiments.
  • querying the optimal pathfinding movement path consumes server performance. If the network action head predicts frequently, it will continuously predict rule-based movements and intensively query the optimal path in a short period, resulting in a significant short-term drain on the business team's server performance, causing lag in the game's business logic and impacting the online player experience. Furthermore, from a top-down perspective, the model may not need to repeatedly predict rule-based movements using the navigation grid in a short time. This is because rule-based movements mainly occur in non-combat scenarios, where the target enemy's position may not change much, meaning the optimal path to the target enemy may not change significantly. The previously queried optimal path remains valid for a period, and the model can continue to use it.
  • the hybrid pathfinding scheme provided in this application's embodiments allows the model to avoid frequent changes to the optimal path in non-combat scenarios. It can still use the optimal path returned by the previous query. A single query avoids the unnecessary performance overhead of repeatedly querying the optimal path. Therefore, the neural network does not need to frequently predict rule-based movement commands in a short period; it only needs to predict the rule-based movement once, and maintain the previously predicted rule-based movement for subsequent periods. The optimal path found by UE4 can then be used for further movement. Rule-based movement uses the navigation grid, meaning that Path type action index 2 in Table 1 indicates that the optimal path from the previous query is being used with the navigation grid.
  • the rule-based movement mode of the navigation grid will be interrupted. This corresponds to Path type action index 3 in Table 1, indicating that assisted movement will stop.
  • the auxiliary rule-based movement or atomic movement will be restarted during the next network prediction.
  • each movement using the navigation grid is subject to a time threshold constraint. Regardless of whether the pathfinding target point has been reached, rule-based movement based on the navigation grid will stop once the time threshold constraint is reached.
  • the pathfinding mode will be reselected during the next neural network decision.
  • optimal path lookup is only performed when predicting action command 1 (Path type), consuming server performance.
  • Subsequent predictions of action command 2 are sufficient to maintain movement based on the navigation grid's auxiliary rules. This achieves full-map navigation with minimal performance overhead, without affecting the agent's strategic maneuvering in combat scenarios. It truly maximizes the decision-making and exploration advantages of reinforcement learning, enabling targeted adaptive learning of pathfinding and game-playing strategies for different scenarios.
  • the focus can be placed on strategic maneuvering, minimizing the pathfinding process in non-combat scenarios, accelerating training efficiency, and improving the model's upper limit performance.
  • Input information 1801 is input into neural network 1802, and neural network 1802 predicts the pathfinding index 1803.
  • the pathfinding index is 0, atomic movement is performed with an action index of (0-15); when the pathfinding index is 1, rule-based movement is performed with an action index of 16; when the pathfinding index is 2, rule-based movement is maintained with an action index of 16; when the pathfinding index is 3, rule-based movement is stopped with an action index of (0-15).
  • Input information 1801 corresponds to state data in other embodiments.
  • the pathfinding target point 1804 and the optimal path 1805 are determined; and when the pathfinding index is 2, movement continues along the optimal path 1805 until the pathfinding target point 1804 is reached.
  • the input information 1801 indicates that the scenario is not a battle scenario, then the pathfinding target point 1804 is moved; if the input information 1801 indicates that any of the three specific logics mentioned above is triggered, then the rule-based movement is stopped.
  • Step 1 The game is divided into several time slices according to the time.
  • the starting time of each time slice T is determined by the network, which pathfinding method is used.
  • time slice T corresponds to the prediction period in other embodiments.
  • the pathfinding method at the start time is atomic movement.
  • Step 2 When the starting point of each time segment T is selected to move according to the rules, UE4 queries the optimal path based on the pathfinding target point, and then moves there with the help of the navigation grid. Subsequently, the optimal path is maintained within the time segment, and the navigation grid is used to move there.
  • Step 3 Within a single time slice, if at least one of the following occurs: the pathfinding target point is reached, the time threshold constraint is met, or one party is seen, the rule-based movement ends immediately, and the remaining time of the time slice is switched back to atomic movement mode.
  • time slice 2 switches from regular movement to atomic movement when the time threshold is reached; time slice 3 switches from regular movement to atomic movement when one party is seen; and time slice 4 switches from regular movement to atomic movement when the pathfinding target point is reached.
  • "one party being seen” corresponds to the first agent in other embodiments switching from an occluded scene to an unoccluded scene.
  • Step four the selection of the pathfinding target point can be output by the model or by rules.
  • the pathfinding target point can be set to the most recent target enemy voiceprint perception point.
  • Figure 18A illustrates the decision logic of the path-type network action head.
  • the input information 1801 consists of multimodal environment information.
  • the meaning of the path-type action instructions is shown in Table 1.
  • the path-type network action head corresponds to the hybrid path-type action head in other embodiments.
  • the game is divided into multiple time slices. At the start of each time slice, the neural network path-type action head selects a specific path-seeking method based on the current game situation (i.e., the AI's scenario).
  • FIG19 is a schematic flowchart of a hybrid pathfinding method provided in this application, which will be described in detail below.
  • step S1901 proceed to the next time slice.
  • step S1902 it is determined whether the game is in a battle scenario.
  • step S1903 if the scenario is not a battle scenario, the process proceeds to step S1905A.
  • step S1903 it is determined whether any one of the three specific logics mentioned above is satisfied.
  • step S1904 if any of the above three specific logics are not satisfied, it indicates that the pathfinding target point has not been reached, the time threshold constraint has not been met, and no party is seen, then proceed to step S1904; if any of the above three specific logics are satisfied, it indicates that the pathfinding target point has been reached, or the time threshold constraint has been met, or one party is seen, then proceed to step S1905D.
  • step S1904 the pathfinding index is obtained.
  • step S1905A when the routing index is 0, the process proceeds to step S1905A; when the routing index is 1, the process proceeds to step S1905B; when the routing index is 2, the process proceeds to step S1905C; and when the routing index is 3, the process proceeds to step S1905D.
  • step S1905A the intelligent agent is controlled to perform atomic movement, and the process ends.
  • step S1905B the control agent moves based on rules.
  • step S1905C the control agent continues to move based on rules and proceeds to step S1907.
  • step S1905D the control agent stops moving based on rules and the process ends.
  • step S1906 the optimal path is determined.
  • the optimal path is determined based on the current location and the target point for navigation.
  • step S1907 the current optimal path is maintained.
  • the characterization does not require updating the optimal path and continues to maintain the current optimal path.
  • step S1908 move to the pathfinding target point.
  • the target point is reached by moving along the optimal path using a navigation grid.
  • the neural network outputs a pathfinding target point, and the agent is trained to approach this target point using reinforcement learning. Only when the agent reaches this pathfinding target point will a new pathfinding target point be set. If the agent does not reach the pathfinding target point, it is encouraged to approach the pathfinding target point with a reward.
  • This method can also achieve the same pathfinding purpose in non-combat scenarios, while avoiding the server performance consumption caused by multiple path query operations.
  • steps S1901 to S1908 atomic movement can be performed in battle scenarios, while rule-based movement can be performed in non-battle scenarios, enabling scene-matched movement and improving model training efficiency. Furthermore, rule-based movement only requires determining the optimal path once, saving computational resources and accelerating model training.
  • Figure 20 is a schematic diagram of a model structure provided in an embodiment of this application.
  • the model structure includes a perception part 2001 and a decision part 2002.
  • the perception part 2001 includes environmental information 0011, scalar information 0012, depth map information 0013, ring ray information 0014, a first encoder 0015, a second encoder 0016, a third encoder 0017, a fourth encoder 0018, a feature extraction unit 0019, and global information 0020.
  • the decision part 2002 includes an action mask 0021, a policy network 0022, and a value network 0023.
  • the first encoder 0015 is used to encode the environmental information 0011 to obtain the encoded environmental information; the second encoder 0016 is used to encode the scalar information 0012 to obtain the encoded scalar information; the third encoder 0017 is used to encode the depth map information 0013 to obtain the encoded depth map information, wherein the third encoder 0017 can be a Conv2D; the fourth encoder 0018 is used to encode the annular ray information 0014 to obtain the encoded annular ray information, wherein the fourth encoder 0018 can be a Conv1D.
  • the encoded environmental information, encoded scalar information, encoded depth map information, and encoded ring ray information are concatenated to obtain the comprehensive encoded information 0024.
  • feature extraction unit 0019 is used to extract features from the comprehensive encoded information 0024 to obtain state features.
  • the feature extraction unit 0019 can be a long short-term memory neural network.
  • the state features are input into the policy network 0022, which, combined with the action mask 0021, predicts the action instruction set 0025.
  • the game kernel acquires runtime state data and divides it into different modal perception data according to Table 2.
  • Each modal perception data is independently processed by different types of neural networks for feature extraction, which allows for a more accurate and detailed representation of the current game situation.
  • the different modal perception data include environmental information, scalar information, depth map information, and ring ray information.
  • the input to the value network 0023 includes not only the hidden state encoding information output by the feature extraction unit 0019, but also global information.
  • This global information comprises all the true information about itself and the target enemy.
  • This global input feature, including "cheating" information can reduce the value estimation bias and variance of the value network 0023, which is beneficial for policy convergence stability.
  • This method only needs to be used in the model training phase, because in the actual online deployment phase, only the policy network is needed, not the value network. Furthermore, this embodiment also introduces autoregressive embedding into the model design.
  • each layer of action head samples actions and performs one-hot encoding into an embedding vector, which is then superimposed with the input vector of the next layer as the input to the next layer's action head.
  • This layer-by-layer influence forms a causal dependency, making the decisions made by subsequent action heads more reasonable and natural.
  • the existing reward and punishment system causes the AI agent to focus solely on killing the player, resulting in a mindless and aggressive playstyle that relies heavily on pre-positioning and high accuracy to win, it lacks strategic maneuvering and the ability to utilize cover for skirmishes.
  • the overall combat lacks human-like realism and fails to meet the deployment standards of the business side. Therefore, this application's embodiment, during model training, incorporates more auxiliary, human-like rewards and punishments based on the existing main reward and punishment system to guide the AI agent's human-like behavior.
  • initial proportions of various reward and punishment coefficients are determined, divided into primary rewards and punishments and secondary anthropomorphic rewards and punishments.
  • the agent is encouraged to train with the ultimate goal of victory. Initially, the weight of the primary reward and punishment coefficient can be increased, while the proportion of anthropomorphic rewards and punishments can be decreased.
  • the proportion of the secondary reward coefficient needs to be dynamically adjusted based on the model's specific performance and the changing trends of the corresponding anthropomorphic indicators. If the agent does not use cover to engage in combat, analysis indicates that the primary kill reward weight is too high.
  • the agent is only focused on "greedyly” obtaining a larger kill reward value and abandons learning anthropomorphic behavior guided by secondary rewards. Therefore, once the agent's strength reaches the required level, the weight of the secondary rewards needs to be dynamically adjusted to ensure that the agent, with a certain level of strength, can further develop anthropomorphic behavior.
  • model performance can be further optimized through business-side experience feedback.
  • the model can be tested periodically, and the corresponding reward coefficients can be adjusted based on the strength or gameplay issues obtained from the experience, thereby guiding the development of anthropomorphic multi-style performance.
  • FPS games need to strike a reasonable balance between strength and anthropomorphism. Sometimes, some strength can be sacrificed for the sake of anthropomorphism.
  • a model that can guarantee strength while exhibiting different anthropomorphic styles and meeting business objectives is obtained.
  • the reward and punishment system is set up as shown in Table 3:
  • This application adopts a self-game iterative process. Since FPS games are similar to real-world three-dimensional scenes, with sufficiently large map areas and complex and varied terrain, it is necessary to train a game to perform like a human in the real world with diverse gameplay. Therefore, it is necessary to construct a two-player zero-sum game framework and continuously iterate through self-game technology until it converges to a Nash equilibrium strategy.
  • the overall training process includes the following steps one through eight:
  • Step 1 Use the game client's built-in behavior tree as the combat opponent to train the model in round 0.
  • human data can be used to guide early behavioral patterns.
  • the model may engage in many repetitive and ineffective explorations, and may even easily fall into unpredictable special states, causing irreversible long-term effects on policy learning, making it difficult for the policy to escape local optima.
  • supervised learning can be used to start the agent at the level of a human player, and then reinforcement learning can be used to further explore strategies, the exploration efficiency can be greatly improved, and the initial policy learning can effectively avoid falling into local optima. Subsequent human-like training can also be much more efficient.
  • Step 2 The model from round 0 is added to the "evaluation model pool" for evaluation, and the evaluation results are obtained.
  • Step 3 Based on the evaluation results, determine whether to store the Round 0 model in the "opponent model pool".
  • Step 4 Sample past models from the "opponent model pool” according to certain rules to serve as opponents, and train the Kth round model until the best response is achieved.
  • uniform sampling rules In the embodiments of this application, uniform sampling rules, elite sampling rules, etc., may be followed.
  • Step 5 The Kth round model is added to the "Evaluation Model Pool" for evaluation to obtain the Kth evaluation result.
  • Step 6 Based on the Kth evaluation result, determine whether to store the Kth round model in the "opponent model pool".
  • Step 7 Return to step 4 to train the model for the K+1th round.
  • Step 8 Continue training iterations until the training converges to a Nash equilibrium policy.
  • This application employs a self-playing reinforcement learning training method based on a hybrid pathfinding scheme. It leverages the decision-making and exploration advantages of reinforcement learning, using centralized neural network training and a hybrid execution of underlying movement.
  • the method adaptively and flexibly selects movement modes based on the battle situation, balancing full-map pathfinding and combat game-playing capabilities while significantly improving training efficiency.
  • Furthermore, coupled with a comprehensive and detailed anthropomorphic reward and punishment system it truly achieves high-level AI for game companionship that meets business requirements in ultra-large maps and complex, diverse 3D terrain scenarios, helping business teams generate content more efficiently.
  • the reinforcement learning-based gaming AI trained in this embodiment of the application has achieved the business objectives in terms of both strength and human-likeness.
  • the advantages of the reinforcement learning-based gaming AI include:
  • the reinforcement learning AI can identify the location of cover and go to take cover without pre-constructing cover points, and can use various unconventional cover to avoid combat.
  • the behavior is flexible and varied, with different combat performances even in the same entanglement scenario, which can increase replayability.
  • the behavior space completely covers the player's behavior space, allowing the player to perform all actions that can be performed, thus going to places that the behavior tree cannot reach and completing operations that the behavior tree cannot perform.
  • Figure 21A shows the distribution of the active area of the agent trained based on the hybrid pathfinding scheme in the map
  • Figure 21B shows the distribution of the active area of the AI trained based on the atomic pathfinding scheme in the map.
  • the active area of the agent i.e., the model
  • the coverage of the whole map is wider, highlighting the improvement of the hybrid pathfinding scheme on the AI's full map navigation capability.
  • curve 1 shows the win rate of the AI combat behavior tree trained based on the hybrid pathfinding scheme
  • curve 2 shows the win rate of the AI combat behavior tree trained based on the atomic pathfinding scheme.
  • curve 3 shows the draw rate of the AI's combat behavior tree trained based on the hybrid movement pathfinding scheme
  • curve 4 shows the draw rate of the AI's combat behavior tree trained based on the atomic movement pathfinding scheme.
  • the AI trained based on the atomic movement scheme has a higher draw rate, indicating that the strategy eventually converges to an increasingly passive style, which is manifested in the agent's unwillingness to take the initiative and only willing to passively wait for the enemy to come to fight. This does not meet the project team's expectations for the agent.
  • the AI trained based on the hybrid movement scheme has a higher win rate with a lower draw rate.
  • the software module stored in the model training device 455 in the memory 450 may include:
  • the first acquisition module 4551 is configured to acquire the state data of the first intelligent agent in the virtual scene and the first prediction model to be trained corresponding to the first intelligent agent; the first prediction model to be trained is used to control the first intelligent agent to move based on the state data of the first intelligent agent; the instruction prediction module 4552 is configured to use the first prediction model to be trained to predict the state data to obtain a set of action instructions, the set of action instructions including a pathfinding method and multiple action information; the execution module 4553 is configured to control the first intelligent agent to move based on the pathfinding method and execute corresponding actions based on each of the action information to obtain an execution result; the adjustment module 4554 is configured to adjust the parameters of the first prediction model to be trained based on the reward information of multiple execution results to obtain a trained first prediction model, wherein the reward information is determined based on the multiple execution results, the main reward coefficient, and the auxiliary reward coefficient.
  • the first prediction model includes a feature extraction module, a pathfinding prediction module, and multiple action prediction modules.
  • the instruction prediction module 4552 is further configured to: extract features from the state data using the feature extraction module to obtain state features; for each action prediction module, make predictions based on the state features using the action prediction module to obtain action information of the first agent; make predictions based on the state features using the pathfinding prediction module to obtain the movement mode of the first agent; and construct the action instruction set based on the movement mode and each action information.
  • the state data includes environmental information, scalar information, depth map information, and ring ray information.
  • the feature extraction module includes an encoding unit and a feature extraction unit.
  • the instruction prediction module 4552 is further configured to: use the encoding unit to encode the environmental information, the scalar information, the depth map information, and the ring ray information respectively to obtain encoded environmental information, encoded scalar information, encoded depth map information, and encoded ring ray information; concatenate the encoded environmental information, encoded scalar information, encoded depth map information, and encoded ring ray information to obtain comprehensive encoded information; and use the feature extraction unit to extract features from the comprehensive encoded information to obtain the state features.
  • the instruction prediction module 4552 is further configured to: obtain a preset action sequence table, the action sequence table being used to characterize the arrangement order between actions; determine a first action identifier preceding the pathfinding action identifier based on the action sequence table; obtain first action information corresponding to the first action identifier, the first action information being predicted by the action prediction module corresponding to the first action identifier; encode the first action information to obtain first encoded action information; and use the pathfinding prediction module to predict the movement mode of the first agent based on the state features and the first encoded action information.
  • the instruction prediction module 4552 is further configured to: predict the state features and the first encoded action information using the pathfinding prediction module according to the prediction period to obtain a target pathfinding index; determine the target movement mode corresponding to the target pathfinding index based on a preset index mapping table; and determine the target movement mode as the movement mode of the first intelligent agent.
  • the target pathfinding index is one of a first pathfinding index and a second pathfinding index.
  • the movement mode of the first agent is determined to be a rule-based movement mode.
  • the execution module 4553 is further configured to: obtain the target position and the current position of the first agent; determine the movement path of the first agent based on the target position and the current position; control the first agent to move according to the rule-based movement mode based on the movement path; and control the first agent to execute various actions based on various action information.
  • the execution module 4553 is further configured to: when controlling the first agent to move according to the rules, when it is determined that the movement mode switching condition is met, update the target pathfinding index to the first pathfinding index; and switch the movement mode of the first agent to a movement mode based on atomic movement.
  • the execution module 4553 is further configured to: determine that the movement mode switching condition is met when it is determined that the first agent has moved to the target position; or, determine that the movement mode switching condition is met when it is determined that the first agent has switched from an occluded scene to an unoccluded scene, wherein the occluded scene indicates that there is an obstruction between the first agent and the second agent, and the unoccluded scene indicates that there is no obstruction between the first agent and the second agent; or, obtain the movement duration of the first agent moving according to the rule movement mode, and determine that the movement mode switching condition is met when the movement duration reaches a duration threshold, wherein the duration threshold is less than the prediction period.
  • the instruction prediction module 4552 is further configured to: obtain a second action identifier corresponding to the action prediction module; determine a third action identifier located before the second action identifier based on the action sequence table; and obtain second action information corresponding to the third action identifier, wherein the second action information is predicted using the action prediction module corresponding to the third action identifier.
  • the second action information is encoded to obtain the second encoded action information; the action prediction module corresponding to the second action identifier is used to predict the action information of the first agent based on the state features and the second encoded action information.
  • the instruction prediction module 4552 is further configured to: obtain a preset action mask, the action mask being matched with the interaction rules in the virtual scene, the action mask being used to limit the action selection space of the first intelligent agent; construct a candidate instruction set from the movement mode and the various action information; and perform masking processing on the candidate instruction set using the action mask to obtain the action instruction set.
  • the action information includes a first flag value for indicating whether an action is performed.
  • the instruction prediction module 4552 is further configured to: for each action information in the candidate instruction set, obtain a mask value corresponding to the action information from the action mask; when the mask value corresponding to the action information is a first preset value and the first flag value in the action information is a second preset value, update the first flag value in the action information to the first preset value, wherein when the first flag value is the first preset value, it indicates that the action corresponding to the action information is not performed.
  • the adjustment module 4554 is further configured to: determine a reward function based on the main reward coefficient and the auxiliary reward coefficient, wherein the auxiliary reward coefficient is used to characterize the reward coefficient assigned when the first agent performs a preset anthropomorphic action; determine the reward information for each execution result using the reward function; accumulate the reward information for each execution result to obtain a cumulative reward; backpropagate the cumulative reward to the first prediction model to be trained; and adjust the parameters of the first prediction model to be trained based on the cumulative reward until the training termination condition is met, thereby obtaining the trained first prediction model.
  • the model training device 455 further includes: a first determining module configured to, in response to a game start command, determine a selected region from each sub-region covered by the virtual scene corresponding to the game game based on the principle of equal probability; a second determining module configured to determine a first center point of the selected region; and a third determining module configured to, based on the first center point and a preset first maximum pathfinding distance, determine a first spawn region and determine a first set of available points in the first spawn region.
  • the first generation module is configured to determine the first initial position of the first agent from the first set of available points, and generate the first agent at the first initial position.
  • the model training device 455 further includes: a fourth determining module configured to determine the first initial position as a second center point; a second acquiring module configured to acquire a preset second maximum pathfinding distance and a preset second minimum pathfinding distance corresponding to the second agent; a fifth determining module configured to determine a second birth region based on the second center point, the preset second maximum pathfinding distance, and the preset second minimum pathfinding distance, and determine a second set of available points in the second birth region; a sixth determining module configured to determine a candidate initial position from the second set of available points; and a second generating module configured to determine the candidate initial position as the second initial position of the second agent when there is an obstruction between the candidate initial position and the first initial position, and generate the second agent at the second initial position.
  • a fourth determining module configured to determine the first initial position as a second center point
  • a second acquiring module configured to acquire a preset second maximum pathfinding distance and a preset second minimum pathfinding distance corresponding to the second
  • This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium.
  • a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the model training method described above in this application.
  • This application provides a computer-readable storage medium storing executable instructions.
  • the processor will execute the model training method provided in this application, such as the model training method shown in Figures 3 and 12.
  • the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.
  • executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., files that store one or more modules, subroutines, or code sections).
  • HTML Hyper Text Markup Language
  • executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Human Computer Interaction (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本申请提供了一种模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品；本申请实施例可应用于游戏、人工智能、智慧交通、辅助驾驶等各种场景，方法包括：获取第一智能体在虚拟场景中的状态数据和第一智能体对应的待训练的第一预测模型；利用待训练的第一预测模型对状态数据进行预测，得到包含寻路方式的动作指令集合；基于动作指令集合控制第一智能体执行对应的动作，得到执行结果；基于多个执行结果的奖励信息调整待训练的第一预测模型的参数，得到训练好的第一预测模型，其中，奖励信息是基于所述多个执行结果、主奖励系数和辅助奖励系数确定的。

Description

模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品
相关申请的交叉引用
本申请基于申请号为202410545713.X、申请日为2024年4月30日的中国专利申请提出，并要求以上中国专利申请的优先权，以上中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请涉及人工智能领域，尤其涉及一种模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品。
背景技术
随着人工智能技术的发展，游戏智能体在游戏领域中的应用前景也越来越广阔。游戏智能体是游戏中的重要元素之一，能够为游戏增加深度、趣味性和交互性，提升玩家的游戏体验和参与度。
相关技术在训练游戏智能体的过程中，第一种是基于导航网格和场景查询系统的方式来控制游戏智能体的寻路方式，第二种是基于原子动作控制游戏智能体的寻路方式。上述相关技术均是采用单一、固定的寻路方式，存在训练效果不佳的问题，最终导致模型无法上线部署、模型学出的策略效果不佳甚至无法收敛。
发明内容
本申请实施例提供一种模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品，能够提升模型训练的效率及准确率。
本申请实施例的技术方案是这样实现的：
本申请实施例提供一种模型训练方法，应用于计算机设备，所述方法包括：
获取第一智能体在虚拟场景中的状态数据和所述第一智能体对应的待训练的第一预测模型；所述待训练的第一预测模型用于基于所述第一智能体的状态数据，控制所述第一智能体进行移动；
利用所述待训练的第一预测模型对所述状态数据进行预测，得到动作指令集合，所述动作指令集合包括寻路方式和多个动作信息；
控制所述第一智能体基于所述寻路方式进行移动，并基于各个所述动作信息执行对应的动作，得到执行结果；
基于多个所述执行结果的奖励信息调整所述待训练的第一预测模型的参数，得到训练好的第一预测模型，其中，所述奖励信息是基于所述多个执行结果、主奖励系数和辅助奖励系数确定的。
本申请实施例提供一种模型训练装置，所述装置包括：
第一获取模块，配置为获取第一智能体在虚拟场景中的状态数据和所述第一智能体对应的待训练的第一预测模型；所述待训练的第一预测模型用于基于所述第一智能体的状态数据，控制所述第一智能体进行移动；
指令预测模块，配置为利用所述待训练的第一预测模型对所述状态数据进行预测，得到动作指令集合，所述动作指令集合包括寻路方式和多个动作信息；；
执行模块，配置为控制所述第一智能体基于所述寻路方式进行移动，并基于各个所述动作信息执行对应的动作，得到执行结果；
调整模块，配置为基于多个执行结果的奖励信息调整所述待训练的第一预测模型的参数，得到训练好的第一预测模型，其中，所述奖励信息是基于所述多个执行结果、主奖励系数和辅助奖励系数确定的。
本申请实施例提供一种计算机设备，包括：
存储器，用于存储计算机可执行指令；
处理器，用于执行所述存储器中存储的计算机可执行指令时，实现本申请实施例提供的模型训练方法。
本申请实施例提供一种计算机可读存储介质，存储有计算机可执行指令，用于引起被处理器执行时，实现本申请实施例提供的模型训练方法。
本申请实施例提供一种计算机程序产品，包括计算机程序或计算机可执行指令，所述计算机程序或计算机可执行指令被处理器执行时，实现本申请实施例提供的模型训练方法。
本申请实施例具有以下有益效果：
在本申请实施例中，游戏对局包括相互对抗的第一智能体和第二智能体，第一智能体与待训练的第一预测模型相对应。接着，可利用待训练的第一预测模型对第一智能体的状态数据进行预测，得到包含寻路方式的动作指令集合，由于待训练的第一预测模型能够基于第一智能体所处的不同场景，控制第一智能体按照不同的寻路方式进行移动，那么当第一智能体处于不同虚拟场景时，则能够预测得到包含不同寻路方式的动作指令集合，该动作指令集合中的寻路方式与第一智能体所处场景相匹配，如此实现寻路方式的多样化。然后，基于动作指令集合控制第一智能体执行对应的动作，得到执行结果。最后，利用多个执行结果、主奖励系数以及辅助奖励系数，确定多个执行结果的奖励信息，并基于多个执行结果的奖励信息调整待训练的第一预测模型的参数，得到训练好的第一预测模型。如此能够实现第一智能体的混合寻路并移动，使得寻路方式、移动方式多样化，通过多样化的寻路方式提升待训练的第一预测模型的训练效果，并且第一智能体的寻路方式是与所处场景相匹配的，因此还能够提升待训练的第一预测模型的训练准确性。此外，由于在确定奖励信息时同时采用了主奖励系数和辅助奖励系数，则能够提高第一智能体在策略学习任务中的学习效率和性能，也即提升训练好的第一预测模型的预测性能。
附图说明
图1是本申请实施例提供的模型训练系统100的一种网络架构示意图；
图2是本申请实施例提供的服务器400的结构示意图；
图3是本申请实施例提供的模型训练方法的第一流程示意图；
图4是本申请实施例提供的预测方法的第一流程示意图；
图5是本申请实施例提供的特征提取方法的一种流程示意图；
图6是本申请实施例提供的预测方法的第二流程示意图；
图7是本申请实施例提供的预测方法的第三流程示意图；
图8是本申请实施例提供的预测方法的第四流程示意图；
图9是本申请实施例提供的构建动作指令集合的一种流程示意图；
图10是本申请实施例提供的执行动作的一种流程示意图；
图11是本申请实施例提供的调整参数的一种流程示意图；
图12是本申请实施例提供的模型训练方法的第二流程示意图；
图13是本申请实施例农场地图的一种界面显示示意图；
图14A是本申请实施例提供的确定对战配置逻辑的一种流程示意图；
图14B是本申请实施例提供的呈现出生坐标的一种界面展示示意图。
图15是本申请实施例提供的模型训练阶段的总体结构示意图；
图16是本申请实施例提供的同步推理模式的一种结构示意图；
图17是本申请实施例提供的异步推理模式的一种结构流程图；
图18A是本申请实施例提供的混合寻路动作头的网络结构的一种结构示意图；
图18B是本申请实施例提供的时间片的的一种结构示意图；
图19是本申请实施例提供的混合寻路的一种流程示意图；
图20是本申请实施例提供的模型结构的一种框架结构示意图；
图21A是本申请实施例提供的基于混合移动寻路方案训练出的AI在地图中活动区域的分布示意图；
图21B是本申请实施例提供的基于原子移动寻路方案训练出的AI在地图中活动区域的分布示意图；
图22是本申请实施例提供的AI对战行为树胜率结果展示示意图；
图23是本申请实施例提供的AI对战行为树平局率结果展示示意图。
需要指出，上述“第一”、“第二”仅用于区分不同的方案，不代表用于区分方案的优劣程度或在实施过程中的优先级。
具体实施方式
为了使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请作进一步地详细描述，所描述的实施例不应视为对本申请的限制，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其它实施例，都属于本申请保护的范围。
在以下的描述中，涉及到“一些实施例”，其描述了所有可能实施例的子集，但是可以理解，“一些实施例”可以是所有可能实施例的相同子集或不同子集，并且可以在不冲突的情况下相互结合。
在以下的描述中，所涉及的术语“第一/第二/第三”仅仅是区别类似的对象，不代表针对对象的特定排序，可以理解地，“第一/第二/第三”在允许的情况下可以互换特定的顺序或先后次序，以使这里描述的本申请实施例能够以除了在这里图示或描述的以外的顺序实施。
本申请实施例中，术语“模块”或“单元”是指有预定功能的计算机程序或计算机程序的一部分，并与其他相关部分一起工作以实现预定目标，并且可以通过使用软件、硬件(如处理电路或存储器)或其组合来全部或部分实现。同样的，一个处理器(或多个处理器或存储器)可以用来实现一个或多个模块或单元。此外，每个模块或单元都可以是包含该模块或单元功能的整体模块或单元的一部分。
除非另有定义，本申请实施例所使用的所有的技术和科学术语与所属技术领域的技术人员通常理解的含义相同。本申请实施例中所使用的术语只是为了描述本申请实施例的目的，不是旨在限制本申请。
本申请实施例中相关数据收集处理在实例应用时应该严格根据相关国家法律法规的要求，获取个人信息主体的知情同意或单独同意，并在法律法规及个人信息主体的授权范围内，开展后续数据使用及处理。
对本申请实施例进行进一步详细说明之前，对本申请实施例中涉及的名词和术语进行说明，本申请实施例中涉及的名词和术语适用于如下的解释。
1)智能体，在游戏领域，智能体通常指游戏中的虚拟角色或角色代理，它们由计算机程序控制，并具有模拟人类智能的行为特征。这些智能体可以是游戏中的敌人、盟友、非真实玩家角色(Non-Player Character，NPC)或其他角色。智能体的设计和实现可以用于增强游戏的可玩性、挑战性和真实感。
2)游戏智能体，在游戏场景下，将执行强化学习的机器叫做智能体(Agent)，是基于神经网络驱动的游戏人工智能(Artificial Intelligence，AI)，与智能体进行交互的称为环境(Environment)，游戏智能体通过分析游戏环境来做合理决策，可以与其它智能体或玩家之间合作竞争，使之表现得像人类一样智能。
3)强化学习(Reinforcement Learning，RL)，是一种机器学习方法，旨在让智能体通过与环境的交互学习如何做出决策，以获得最大的累积奖励。在强化学习中，智能体通过尝试不同的行动来探索环境，并根据环境给出的奖励信号来调整其行为，以最大化长期累积奖励。这种学习方式类似于试错学习，智能体通过不断的试验和反馈来优化其行为策略。强化学习常用于解决需要动态决策的问题，如游戏智能体的训练、机器人控制、自动驾驶等。
4)监督学习(Supervised learning，SL)，是机器学习的范式之一，是一种数据分析方法，从带标签的数据集迭代训练学习的算法，使得模型准确分类数据或预测结果。
5)导航网格(Navigation Mesh，NavMesh)，是复杂空间导航寻路的行走面，是二维多边形网格数据结构，定义了游戏智能体遍历环境的哪些区域，在这些区域内游戏智能体可以自由走动，不受环境障碍物的阻碍，当游戏智能体接收到指令后，导航网格就会自动计算达到目标点的最短可达路径，帮助游戏智能体在复杂空中进行寻路，轻松简单地实现AI的自动寻路效果。
6)场景查询系统(Environment Query System，EQS)，是UE4 AI系统的一个功能，可用来从环境中收集数据，并对当前场景里满足条件的一些信息进行查询，然后划分权重，根据权重返回结果，让AI根据收集的结果做出不同的响应。
7)深度图(Depth Map，DM)，是将图像采集器到场景中各点的距离(深度)作为像素值的二维数据结构，能够详尽描绘游戏智能体前方纯深度信息，值越接近于1表明前方越亮，视野越通透。
深度图指的是一种图像，其中每个像素的值表示从观察者(通常是摄像机)到场景中对应点的距离。它通常以灰度图像的形式呈现，其中较暗的像素表示较远的距离，而较亮的像素表示较近的距离。在游戏开发中，深度图经常用于实现景深效果(depth of field)，即模拟相机焦距不同而产生的景深变化。通过使用深度图，游戏引擎可以确定哪些物体位于焦点范围内，哪些物体位于模糊范围内，从而产生更加逼真的视觉效果。
8)环形射线，以自身为起点，环绕自身周边向外打出若干条射线，用来描述环绕自身周边的障碍物地形信息，是一种环境感知方式。
环形射线是一种在计算机图形学和游戏开发中常用的技术，用于检测物体之间的碰撞或确定物体之间的距离。它与常规的射线(即从一个点沿着一个方向延伸的线)不同，环形射线是从一个点向周围的所有方向发射的射线，形成一个圆形或环形的射线阵列。
9)原子动作(Original Action，OA)，是指在游戏中，游戏角色操作的是最基本、最细粒度的原始动作，这类动作不可进一步进行拆解细化。在并发编程中，原子动作通常用于确保多个线程或进程之间的同步和一致性。
10)行为树，用以控制游戏中NPC决策的树状结构体。是一种用于描述和控制游戏角色或智能体行为的图形化工具和方法。它是一种树形结构，由一系列节点组成，每个节点代表一种行为、条件或控制逻辑。行为树被广泛应用于游戏开发、人工智能和机器人领域。
11)近端策略优化算法(Proximal Policy Optimization，PPO)，是一种基于策略梯度优化的、面向连续或者离散动作空间的在线深度强化学习算法。PPO的核心思想是通过在每次更新策略时，控制策略变化的幅度，以保持更新幅度在一个可控范围内，从而提高算法的稳定性。
12)辅助奖励，在强化学习算法中，为让agent实现各类差异化的表现，更具差异化目标，设计出辅助实现主目标的奖励。在训练智能体时，辅助奖励通常是指一种额外的奖励信号，用于辅助智能体学习任务中的特定方面或辅助达到特定的目标。辅助奖励通常与主要的任务奖励结合使用，可以帮助加速智能体的学习过程，改善其性能，并且在某些情况下可以帮助解决任务中的困难问题。
常见的辅助奖励包括探索奖励、时间奖励、距离奖励、寻找掩体奖励、动作多样性奖励等，它们可以在强化学习、监督学习等不同的学习框架中使用。通过合理设计和使用辅助奖励，可以提高智能体在任务中的学习效率和性能。
13)真实玩家对战真实玩家(Player versus Player，PVP)，在这种模式下，玩家们可以直接与其他玩家进行对抗、竞争或合作，而不是与游戏的人工智能(AI)或电脑控制的角色进行交互。PVP模式在许多游戏中都非常常见，包括电子竞技游戏、多人在线游戏(Massive Multiplayer Online，MMO)、射击游戏、格斗游戏等各种类型的游戏。PVP模式通常会增加游戏的竞争性和挑战性，也是许多玩家喜爱的游戏内容之一。
14)玩家对抗环境(Player versus Environment，PVE)，在这种模式下，玩家们与游戏中的环境、任务、怪物或其他非玩家角色(NPC)进行交互和对抗，而不是与其他玩家直接竞争。PVE模式通常包括玩家完成任务、探索地图、击败敌人、探索副本或地下城等活动。这种模式下，玩家通常需要克服游戏中的各种挑战，提高自己的技能和装备，以达到游戏设定的目标或获得奖励。PVE模式在许多单人游戏和多人在线游戏中都非常常见。
15)嵌入(Embedding)，本质是“压缩数据”，用较低k维特征去描述有冗余信息的较高维度的n维特征，信息压缩过程通常是会丢失信息的，包括冗余信息的丢失和部分有用信息的丢失，嵌入可以看作神经网络独热(one-hot)层的参数矩阵。
在训练智能体的过程中，“嵌入”通常指的是将高维的离散数据(如类别型特征、符号型数据)映射到低维连续向量空间的过程。这种映射可以通过神经网络中的嵌入层(Embedding Layer)来实现。
在训练智能体的过程中，嵌入通常用于处理各种类型的输入数据，包括：词嵌入(Word Embedding)、类别嵌入(Categorical Embedding)、动作嵌入(Action Embedding)、状态嵌入(State Embedding)。
通过使用嵌入技术，可以将原始的离散型数据转换为连续型的向量表示，提高了数据的表达能力和模型的泛化能力，从而改善了训练智能体的效果和性能。
16)动作遮罩(Action Mask，AM)，在深度强化学习训练中，通过对网络推理得出动作分布时将“不合理或者不可执行的动作维度”暂时屏蔽，再从“合理或者可执行动作”中按照原有概率采样动作，这样可以保证推理动作在游戏环境内可获得有效的数据样本，避免因返回无效动作而浪费计算资源。
动作遮罩用于在智能体选择动作时限制可选动作的集合。动作遮罩可以确保智能体只从允许的动作中进行选择，而忽略那些不可用的动作，从而使智能体的决策过程更加高效和准确。动作遮罩可以用一个二进制向量(或称为掩码)表示，其中每个元素对应一个可能的动作。如果动作是允许的，则相应位的掩码值为1；如果动作是不允许的，则该动作相对应的掩码值为0。例如，假设有5个可能的动作，动作遮罩可以表示为：[1，0，1，0，0]。这意味着智能体可以执行第一个和第三个动作，而其他动作不可执行。
17)零和博弈(zero-sum game)，又称为零和游戏，是博弈论的一个概念，它是指博弈各方的利益之和为零，或者是一个常数，即一方有收益，其他方必有所失，在零和博弈中，博弈各方是不合作的，属于非合作博弈。如果博弈各方在不同策略组合下使得全体参与者均收益或均受损的情况就是非零和博弈，非零和博弈中博弈各方存在合作的可能性。
18)最佳响应(Best Response，BR)，是当其他玩家选择固定策略时，当前玩家采取能产生对自己最有利结果的策略。
19)自博弈(Self-Play，SP)，是一种迭代式求解最佳响应(BR)的过程，智能体使用自己(也即策略)的前副本作为对手来训练更新当前策略，并持续迭代下去，通过面对不断增强的对手训练自身的方式，可以防止巨大的能力差异使得策略学不动，从而有效增加训练效率，持续提升模型能力。这种训练方法在象棋、围棋(AlphaGo)这种零和游戏中非常有效，但是需要满足两个条件：每一轮模型能力均有所提升、游戏环境是一个胜负可传递博弈环境(Transitive Game)。
20)纳什均衡(Nash Equilibrium，NE)，又叫做非合作博弈均衡，是博弈论的一个重要概念，在一个博弈过程中，对于每一个玩家来说，当其他玩家不改变策略时候，当前所有玩家的策略都是最佳响应，此时，没有玩家可以通过在其余玩家策略保持不变时通过仅改变自身策略来获取更大收益，则称当前联合策略达到了纳什均衡，纳什均衡并不一定是博弈各方的最优解，它表达的是当前博弈环境达到一种均衡稳定的状态。
为了更好地理解本申请实施例提供的模型训练方法，首先对相关技术中的模型训练方法及存在的缺点进行说明。
相关技术中主要存在以下两种模型训练方法：
相关技术1，基于导航网格和场景查询系统的模型训练方法。
在采用相关技术1进行模型训练时，基于场景查询系统查询若干寻路目标点，并根据权重返回满足条件的寻路目标点，然后借助导航网格移动过去。由于导航网格的移动路径必然是可达的，这为AI达到寻路目标点的通过性提供保证，导航网格移动是业务团队开发各种行为树AI的常用移动方式。
上述相关技术1存在的缺陷包括：虽然从导航网格上移动到寻路目标点为AI通过性提供了保证，但是这种查询方式只是给AI提供了若干离散的可达点，导航网格移动可以保证可达性，但是其移动路径是单一固定的，并不能完整清晰的表明AI自身周边所有的可达路径，会限制AI的移动路径多样性，降低可行区域完备度，甚至如果有更好的移动路径，仅仅因为返回的寻路目标点不在该移动路径上，导致借助导航网格移动会错过这条更好的移动路径。此外，每做一次基于寻路点的路径查询操作都会消耗服务器性能，性能消耗量与寻路点个数、寻路点与AI自身距离都有关系，个数越多、距离越远则越性能消耗越大，如果采用这种寻路方案来训练强化学习AI，那么当模型预测频率较高时，寻路点查询频率也会上升，AI不仅容易在前后两次返回的寻路点之间拉扯摆动，出现抽搐抖动的行为，效果不佳的同时还会造成大量的性能消耗，最终导致这样训练模型无法上线部署。
相关技术2，基于原子动作的模型训练方法。
原子动作定义了最小的指令操作单元，是不能进一步拆分的最细颗粒度指令集，原子动作空间可以完全覆盖玩家可执行的操作，在完备度和灵活度上达到人类玩家水平，移动指令也是采用原子移动方式，原子移动可以彻底摆脱导航网格的束缚，进行自由无约束的移动，从而去到行为树去不到的位置，完成行为树无法做出的操作。
上述相关技术2存在的缺陷包括：原子动作的指令颗粒度太细，为了能拟合人类玩家操作，兼顾动作完备度和流畅度，所以需要设计复杂、多样、高维的原子动作指令集，通过这些原子指令集的协同运转，互相配合，才可能完成类似人类玩家的操作和移动方式。如果用原子动作方案训练强化学习模型，可能会因原子动作逻辑过于精细、原子指令集维度过高等原因，导致模型学出的策略效果不佳甚至无法收敛，这种影响在模型预测频率很高时候尤为突出，预测频率很高时模型会在各种原子指令之间摇摆，容易出现不可控行为，而且在场景地图十分庞大、地形特别复杂的情况下，模型的策略探索效率低下，训练的时间成本太高，最终影响模型效果和生产效率。
结合上述相关技术可知，相关技术的缺点在于，第一，场景查询结合导航网格的移动方式，难以详尽描绘周边路况信息，AI移动路线单一；第二，纯原子动作指令集的设计较为复杂、维度较高，需要反复调试验证，时间成本较高；第三，在庞大地图范围和复杂多样的地形场景下，基于纯原子动作集训练模型，学习效率低下，不利于模型训练和收敛；第四，强度和拟人性难以定义，缺乏量化指标；第五，模型训练方式难以适用于地图复杂且变化多样的射击类游戏。
本申请实施例提供一种模型训练方法、装置、计算机设备、计算机可读存储介质和计算机程序产品，能够提升模型训练的效率及准确率，下面说明本申请实施例提供的计算机设备的示例性应用，本申请实施例提供的计算机设备可以实施为笔记本电脑、平板电脑，台式计算机、机顶盒、智能手机、智能音箱、智能手表、智能电视、车载终端等各种类型的终端，也可以实施为服务器。下面，将说明计算机设备实施为服务器时示例性应用。
参见图1，图1是本申请实施例提供的模型训练系统100的网络架构示意图，为实现支撑一个模型训练应用，服务器400通过网络300连接终端200，网络300可以是广域网或者局域网，又或者是二者的组合。
以第一人称射击类游戏(First-person shooting game，FPS)举例来说，终端200用于采集第一人称射击类游戏对局中第一智能体的状态数据，并将状态数据发送至服务器400；服务器400用于利用待训练的第一预测模型对状态数据进行预测，得到包含寻路方式的动作指令集合；基于动作指令集合控制第一智能体执行对应的动作，得到执行结果；基于游戏对局中得到的多个执行结果的奖励信息调整待训练的第一预测模型的参数，得到训练好的第一预测模型，其中，奖励信息是基于多个执行结果、主奖励系数和辅助奖励系数确定的。服务器400还将训练好的第一预测模型返回至终端200，终端200可通过离线的方式利用训练好的第一预测模型控制第一智能体完成第一人称射击类游戏对局。
在本申请实施例中，服务器400获取到状态数据和待训练的第一预测模型之后，利用待训练的第一预测模型对第一智能体的状态数据进行预测，得到包含寻路方式的动作指令集合，一方面，由于待训练的第一预测模型能够基于第一智能体所处的不同场景，控制第一智能体按照不同的寻路方式进行移动，那么当第一智能体处于不同虚拟场景时，则能够预测得到包含不同寻路方式的动作指令集合，该动作指令集合中的寻路方式与第一智能体所处场景相匹配，如此实现寻路方式的多样化。另一方面，由于训练好的第二预测模型也能够基于第二智能体所处的不同场景，控制第二智能体按照不同的寻路方式进行移动，也即第二智能体也能够在不同的场景下执行不同的寻路方式，如此能够强化进行对抗的第一智能体在不同的场景下执行不同的寻路方式的能力。然后，基于动作指令集合控制第一智能体执行对应的动作，得到执行结果。最后，利用多个执行结果、主奖励系数以及辅助奖励系数，确定多个执行结果的奖励信息，并基于多个执行结果的奖励信息调整待训练的第一预测模型的参数，得到训练好的第一预测模型。如此能够实现第一智能体的混合寻路并移动，使得寻路方式、移动方式多样化，通过多样化的寻路方式提升待训练的第一预测模型的训练效果，并且第一智能体的寻路方式是与所处场景相匹配的，因此还能够提升待训练的第一预测模型的训练准确性。此外，由于在确定奖励信息时同时采用了主奖励系数和辅助奖励系数，因此能够提高第一智能体在策略学习任务中的学习效率和性能，也即能够提升第一预测模型的预测性能。
在一些实施例中，服务器400可以是独立的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、内容分发网络(Content Delivery Network，CDN)、以及大数据和人工智能平台等基础云计算服务的云服务器。终端以及服务器可以通过有线或无线通信方式进行直接或间接地连接，本申请实施例中不做限制。
参见图2，图2为本申请实施例提供的服务器400的结构示意图，图2所示的服务器400包括：至少一个处理器410、存储器450、至少一个网络接口420和用户接口430。服务器400中的各个组件通过总线系统440耦合在一起。可理解，总线系统440用于实现这些组件之间的连接通信。总线系统440除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图2中将各种总线都标为总线系统440。
处理器410可以是一种集成电路芯片，具有信号的处理能力，例如通用处理器、数字信号处理器(Digital Signal Processor，DSP)，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等，其中，通用处理器可以是微处理器或者任何常规的处理器等。
用户接口430包括使得能够呈现媒体内容的一个或多个输出装置431，包括一个或多个扬声器和/或一个或多个视觉显示屏。用户接口430还包括一个或多个输入装置432，包括有助于用户输入的用户接口部件，比如键盘、鼠标、麦克风、触屏显示屏、摄像头、其他输入按钮和控件。
存储器450可以是可移除的，不可移除的或其组合。示例性的硬件设备包括固态存储器，硬盘驱动器，光盘驱动器等。存储器450可选地包括在物理位置上远离处理器410的一个或多个存储设备。
存储器450包括易失性存储器或非易失性存储器，也可包括易失性和非易失性存储器两者。非易失性存储器可以是只读存储器(Read Only Memory，ROM)，易失性存储器可以是随机存取存储器(Random Access Memory，RAM)。本申请实施例描述的存储器450旨在包括任意适合类型的存储器。
在一些实施例中，存储器450能够存储数据以支持各种操作，这些数据的示例包括程序、模块和数据结构或者其子集或超集，下面示例性说明。
操作系统451，包括用于处理各种基本系统服务和执行硬件相关任务的系统程序，例如框架层、核心库层、驱动层等，用于实现各种基础业务以及处理基于硬件的任务；
网络通信模块452，用于经由一个或多个(有线或无线)网络接口420到达其他电子设备，示例性的网络接口420包括：蓝牙、无线相容性认证(WiFi)、和通用串行总线(Universal Serial Bus，USB)等；
呈现模块453，用于经由一个或多个与用户接口430相关联的输出装置431(例如，显示屏、扬声器等)使得能够呈现信息(例如，用于操作外围设备和显示内容和信息的用户接口)；
输入处理模块454，用于对一个或多个来自一个或多个输入装置432之一的一个或多个用户输入或互动进行检测以及翻译所检测的输入或互动。
在一些实施例中，本申请实施例提供的装置可以采用软件方式实现，图2示出了存储在存储器450中的模型训练装置455，其可以是程序和插件等形式的软件，包括以下软件模块：第一获取模块4551、指令预测模块4552、执行模块4553和调整模块4554，这些模块是逻辑上的，因此根据所实现的功能可以进行任意的组合或进一步拆分。将在下文中说明各个模块的功能。
在另一些实施例中，本申请实施例提供的装置可以采用硬件方式实现，作为示例，本申请实施例提供的装置可以是采用硬件译码处理器形式的处理器，其被编程以执行本申请实施例提供的模型训练方法，例如，硬件译码处理器形式的处理器可以采用一个或多个应用专用集成电路(Application Specific Integrated Circuit，ASIC)、DSP、可编程逻辑器件(Programmable Logic Device，PLD)、复杂可编程逻辑器件(Complex Programmable Logic Device，CPLD)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或其他电子元件。
将结合本申请实施例提供的服务器的示例性应用和实施，说明本申请实施例提供的模型训练方法。
下面，说明本申请实施例提供的模型训练方法，如前所述，实现本申请实施例的模型训练方法的计算机设备可以是终端、服务器，又或者是二者的结合。因此下文中不再重复说明各个步骤的执行主体。
需要说明的是，下文中的模型训练方法的示例中，是以场景为第一人称射击游戏为例说明的，本领域技术人员根据对下文的理解，可以将本申请实施例提供的模型训练方法应用于针对其他类型游戏的模型训练。本申请实施例还可应用于各种场景，包括但不限于游戏、人工智能、智慧交通、辅助驾驶等。
参见图3，图3是本申请实施例提供的模型训练方法的第一流程示意图，将结合图3示出的步骤对本申请实施例提供的模型训练方法进行说明，图3的执行主体是服务器。
在步骤S101中，获取第一智能体在虚拟场景中的状态数据和第一智能体对应的待训练的第一预测模型。
在本申请实施例中，虚拟场景可以是游戏对局过程中的虚拟场景，还可以是增强现实虚拟场景、虚拟现实场景。虚拟场景中包括所述第一智能体以及与训练好的第二预测模型对应的第二智能体，第二智能体为与第一智能体进行对抗的智能体，训练好的第二预测模型用于基于第二智能体所处的不同场景，控制第二智能体按照不同的寻路方式进行移动，待训练的第一预测模型用于基于第一智能体所处的不同场景，控制第一智能体按照不同的寻路方式进行移动。
在一些实施例中，游戏对局指的是两个或多个智能体之间进行的游戏对抗或竞争，这些智能体可以是由计算机程序控制的虚拟角色、机器人或其他自主实体。
在人工智能领域，智能体之间的游戏对局通常用于测试和评估不同算法和方法的性能，以及研究智能体在复杂环境下的行为和决策能力。这种游戏对局可以是在虚拟环境中进行的，也可以是在实际物理环境中进行的。
智能体之间的游戏对局可以采用不同的对局形式和规则，包括但不限于：
第一种对局形式和规则，博弈游戏：智能体之间进行博弈游戏，如象棋、围棋、扑克等，以测试它们的策略和决策能力。
第二种对局形式和规则，对抗性游戏：智能体之间进行对抗性游戏，如对战类游戏、竞速游戏等，以测试它们的反应速度、技巧和战术能力。
第三种对局形式和规则，合作游戏：智能体之间进行合作游戏，如团队合作游戏、合作解谜游戏等，以测试它们的团队合作和协调能力。
第四种对局形式和规则，混合游戏：智能体之间进行多种形式的游戏对局，包括博弈、对抗和合作等元素的混合游戏，以测试它们的多样化能力和适应性。
智能体之间的游戏对局是人工智能领域中重要的研究和实践领域，通过游戏对局可以深入理解智能体的行为和决策过程，进而提升智能体的性能和效果。
在本申请实施例中，是通过相互对抗的第一智能体和第二智能体，构建出对抗性游戏，也即在该对抗性游戏中，第一智能体和第二智能体所属不同阵营，为敌对关系。相对应的，本申请实施例涉及到的是上述第二种对局形式和规则，也即本申请实施例中游戏对局采用的对局形式和规则，为对抗性游戏。
在一些实施例中，第一智能体的状态数据指的是描述第一智能体当前状态或情境的信息集合。在强化学习和智能体设计中，状态数据是第一智能体感知和理解环境的基础，状态数据包含了环境的关键特征和信息，用于帮助第一智能体做出决策和行动。
状态数据通常由多种类型的信息组成，这些信息可以是关于环境、第一智能体自身以及与其他实体的交互等方面的数据。状态数据是第一智能体与环境交互的基础，状态数据提供了第一智能体对环境的感知和理解能力，帮助第一智能体根据当前情境做出合适的决策和行动。在强化学习中，状态数据是第一智能体策略和价值函数的输入，直接影响第一智能体的学习和决策过程。
在一些实施例中，第一智能体对应的待训练的第一预测模型指的是，用于训练第一智能体的机器学习模型。在强化学习中，第一智能体通常由一个机器学习模型来表示，该机器学习模型可以是神经网络、决策树、随机森林等各种类型的模型。
待训练的第一预测模型是指在训练过程中，需要根据第一智能体与环境的交互数据来不断调整和优化的模型。待训练的第一预测模型的目标是学习到一个有效的策略，使得第一智能体在与环境交互时能够最大化累积奖励或实现特定的任务目标。
在强化学习中，待训练的第一预测模型通常包括以下几个主要部分：
策略网络(Policy Network)：用于学习第一智能体的策略，即在给定状态下选择最优的行动的概率分布。
价值网络(Value Network)：用于估计状态的价值或行动的价值，帮助第一智能体判断哪些行动是更优的或哪些状态是更有价值的。
动作价值函数(Action-Value Function)：用于估计在给定状态下采取特定行动的价值，即行动的预期累积奖励。
奖励函数(Reward Function)：用于定义第一智能体在与环境交互中所获得的奖励信号，帮助待训练的第一预测模型学习有效的策略。
其他辅助组件：如经验回放缓冲区、探索策略等，用于提高模型的稳定性和效果。
待训练的第一预测模型的目标是通过与环境的交互数据来不断更新和优化，以使得第一智能体能够逐渐学习到一个更加优秀的策略，从而在实际环境中表现出更好的性能和效果。
在步骤S102中，利用待训练的第一预测模型对状态数据进行预测，得到动作指令集合。
其中，动作指令集合包括寻路方式和多个动作信息。在本申请实施例中，利用待训练的第一预测模型对状态数据进行预测指的是，利用待训练的第一预测模型来预测第一智能体在状态数据对应的这一确定状态下，将要执行的动作指令集合。其中，动作指令集合是指第一智能体即将执行的各个动作的集合，示例性的，该动作指令集合中可以包括寻路方式、移动方向、姿态类型等。
在一些实施例中，第一预测模型包括特征提取模块、寻路预测模块和多个动作预测模块，基于此，参见图4，上述步骤S102可通过以下步骤S1021至步骤S1024来实现，下面具体说明。
在步骤S1021中，利用特征提取模块对状态数据进行特征提取，得到状态特征。
在一些实施例中，状态数据包括环境信息、标量信息、深度图信息和环形射线信息，并且环境信息、标量信息、深度图信息和环形射线信息为不同模态的信息。特征提取模块包括编码单元和特征提取单元。
在一些实施例中，环境信息指的是描述当前游戏环境的各种信息，环境信息通常包括了游戏场景的各种特征、属性和状态，环境信息还可以包括影响游戏进程和第一智能体决策的各种因素。标量信息指的是一种描述游戏状态的简单、单一数值信息，通常为具体的数值，示例性标量信息可以为生命值、分数、时间、资源数量、距离等。深度图信息指的是描述游戏场景中物体距离摄像机或者视角的远近关系的图像信息，深度图信息可以提供有关游戏世界中物体的距离和空间位置的信息。环形射线信息通常用于智能体感知系统中，帮助第一智能体获知周围环境的情况，例如检测障碍物、获取目标位置、识别第二智能体等。
在一些实施例中，参见图5，上述步骤S1021可通过以下步骤S0211至步骤S0213来实现，下面具体说明。
在步骤S0211中，利用编码单元，分别对环境信息、标量信息、深度图信息和环形射线信息进行编码处理，对应得到编码后的环境信息、编码后的标量信息、编码后的深度图信息和编码后的环形射线信息。
在本申请实施例中，编码单元可以包括四个编码子单元，分别为第一编码子单元、第二编码子单元、第三编码子单元和第四编码子单元。
在一些实施例中，第一编码子单元与环境信息相对应，第一编码子单元用于对环境信息进行编码，得到编码后的环境信息，其中，第一编码子单元可以为一个嵌入模型，该嵌入模型通过学习环境信息的关系和模式，将环境信息映射到嵌入向量中，如此，使得环境信息转换成一种更加紧凑、高效的表示形式。第二编码子单元与标量信息相对应，第二编码子单元用于对标量信息进行编码，得到编码后的标量信息，第二编码子单元与上述第一编码子单元相类似，也可以为一个嵌入模型。第三编码子单元与深度图信息相对应，第三编码子单元用于对深度图信息进行编码，得到编码后的深度图信息，第三编码子单元可以为二维卷积(Conv2D)模型。第四编码子单元与环形射线信息相对应，第四编码子单元用于对环形射线信息进行编码，得到编码后的环形射线信息，第四编码子单元可以为一维卷积(Conv1D)模型。
在步骤S0212中，拼接编码后的环境信息、编码后的标量信息、编码后的深度图信息和编码后的环形射线信息，得到综合编码信息。
在本申请实施例中，拼接编码后的环境信息、编码后的标量信息、编码后的深度图信息和编码后的环形射线信息，是指将编码后的环境信息、编码后的标量信息、编码后的深度图信息和编码后的环形射线信息组合起来，得到综合编码信息。
本申请实施例对组合时的组合顺序并不做限定。示例性的，可按照编码后的环境信息、编码后的标量信息、编码后的深度图信息、编码后的环形射线信息的排列顺序，进行组合；也可按编码后的标量信息、照编码后的环境信息、编码后的深度图信息、编码后的环形射线信息的排列顺序，进行组合。
在步骤S0213中，利用特征提取单元对综合编码信息进行特征提取，得到状态特征。
在本申请实施例中，特征提取单元可以为一个深度神经网络，示例性的，特征提取单元可以为长短期记忆人工神经网络。特征提取单元对综合编码信息进行封装、融合，得到状态特征，通过特征提取单元可以捕捉到综合编码信息中的长期依赖关系。
通过上述步骤S0211至步骤S0213，对状态数据中包括的不同模态的信息分别进行编码，如此能够更好地捕捉每种模态的特征，从而提高数据处理和分析的准确性，并且还可以保留各自模态的细节信息，使得数据在传递和解析过程中更加全面和丰富；此外，特征提取单元融合了不同模态信息，能够增强模态间的交互和信息整合，通过特征提取得到信息量丰富的状态特征，因此该状态特征能够精确地反映出游戏对局的局势状态，以便为模型训练提供准确的数据基础，从而能够提升训练好的第一预测模型的准确性。
继续参见图4，接续上文的步骤S1021进行说明。
在步骤S1022中，针对每一动作预测模块，利用动作预测模块基于状态特征进行预测，得到第一智能体的动作信息。
在本申请实施例中，动作预测模块与第一智能体所能执行的其他动作相对应，其他动作是指除寻路动作以外的动作，由于本申请实施例将寻路动作与其他动作分开进行说明，那么，动作预测模块的数量比第一智能体所能执行的动作数量少1，示例性的，假设第一智能体能够执行的动作包括9个，则存在8个动作预测模块。
在一些实施例中，动作预测模块用于基于状态特征进行动作预测，从而得到第一智能体的动作信息，该动作信息用于指示第一智能体执行该动作。
参见图6，上述步骤S1022可通过以下步骤S0221至步骤S0225来实现，下面具体说明。
在步骤S0221中，获取动作预测模块对应的第二动作标识。
在本申请实施例中，每个动作具备对应的动作标识，因此，动作预测模块也对应有所预测动作的第二动作标识。
在步骤S0222中，基于动作顺序表，确定位于第二动作标识前序的第三动作标识。
在本申请实施例中，动作顺序表用于表征动作之间的排列顺序，动作顺序表是提前预设好的表，该动作顺序表中按顺序存储每一动作的动作标识，动作顺序表能够表征动作之间的先后顺序，通过动作之间的先后顺序能够反映出动作之间的依赖关系。示例性的，动作顺序表中存储的动作标识为01、02、03、04、05、06、07、08、09，其中，01为枪口水平瞄准朝向对应的动作标识，02为枪口竖直瞄准朝向对应的动作标识，03为是否开火对应的动作标识，04为移动方式对应的动作标识，05为寻路方式对应的动作标识，06为移动方向对应的动作标识，07为姿态类型对应的动作标识，08为侧身类型对应的动作标识，09为特殊类型对应的动作标识，则表征动作之间的排列顺序为枪口水平瞄准朝向、枪口竖直瞄准朝向、是否开火、移动方式、寻路方式、移动方向、姿态类型、侧身类型、特殊类型。
在一些实施例中，第三动作标识是指位于第二动作标识之前，且与第二动作标识相邻的动作标识，承接上面的示例，假设第二动作标识为03，则第三动作标识为02。
在一些实施例中，在执行上述步骤S0222之前，还可先获取预设的动作顺序表。
在步骤S0223中，获取第三动作标识对应的第二动作信息。
在本申请实施例中，第二动作信息是利用第三动作标识对应的动作预测模块预测得到。
承接上面的示例，第三动作标识为02，第二动作信息是利用02对应的动作预测模块预测得到。
在步骤S0224中，对第二动作信息进行编码，得到第二编码后动作信息。
在本申请实施例中，可通过对第二动作信息的嵌入处理，来实现对第二动作信息的编码，将得到的嵌入向量确定为第二编码后动作信息。
在步骤S0225中，利用第二动作标识对应的动作预测模块基于状态特征和第二编码后动作信息进行预测，得到第一智能体的动作信息。
在本申请实施例中，不仅基于状态特征，而且还基于前序动作的第二编码后动作信息，对第二动作进行预测，从而得到第二动作对应的动作信息。
通过上述步骤S0221至步骤S0225，在利用动作预测模块对第二动作进行动作信息预测的时候，是基于状态信息和前序动作的第二编码后动作信息，如此，使得动作预测模块能够更好地理解动作发生的上下文信息，实现在进行预测时已经融入了动作之间的依赖关系，并且能够保证前序动作与第二动作的连贯性，从而得到准确性高的预测结果，也即得到准确性高的动作信息，以便为模型训练提供准确的数据基础，从而能够提升训练好的第一预测模型的准确性。
在一些实施例中，可基于上述步骤S0221至步骤S0225，确定出除寻路动作之外的每一动作的动作信息。
继续参见图4，接续上文的步骤S1022进行说明。
在步骤S1023中，利用寻路预测模块基于状态特征进行预测，得到第一智能体的寻路方式。
在本申请实施例中，寻路预测模块是用于对寻路方式进行预测。
参见图7，上述步骤S1023可通过以下步骤S0231至步骤S0235来实现，下面具体说明。
在步骤S0231中，获取预设的动作顺序表。
在本申请实施例中，动作顺序表用于表征动作之间的排列顺序。在一些实施例中，可通过获取指令来获取动作顺序表。
在步骤S0232中，基于动作顺序表，确定位于寻路动作标识前序的第一动作标识。
在一些实施例中，上述步骤S0232的实现过程与上述步骤S0222的实现过程相类似，因此，上述步骤S0232的实现过程可参考上述步骤S0222的实现过程。其中，位于寻路动作标识前序的第一动作标识，是指位于寻路动作标识之前，且与寻路动作标识相邻的动作标识，承接上述举例，寻路动作标识为05，那么第一动作标识即为04。
在步骤S0233中，获取第一动作标识对应的第一动作信息。
在本申请实施例中，第一动作信息是利用第一动作标识对应的动作预测模块预测得到。
在一些实施例中，上述步骤S0233的实现过程与上述步骤S0223的实现过程相类似，因此，上述步骤S0233的实现过程可参考上述步骤S0223的实现过程。
在步骤S0234中，对第一动作信息进行编码，得到第一编码后动作信息。
在一些实施例中，上述步骤S0234的实现过程与上述步骤S0224的实现过程相类似，因此，上述步骤S0234的实现过程可参考上述步骤S0224的实现过程。
在步骤S0235中，利用寻路预测模块基于状态特征和第一编码后动作信息进行预测，得到第一智能体的寻路方式。
在本申请实施例中，在对寻路方式进行预测时，不仅基于状态特征，而且还基于第一编码后动作信息，也即还基于前序动作的编码后动作信息，如此，使得寻路预测模块能够基于对前序动作的分析，预测后续动作对路径规划的影响，进而优化寻路策略，从而得到更加高效或符合状态特征的寻路方式，提高寻路方式的准确性。
参见图8，上述步骤S0235可通过以下步骤S2351至步骤S2353来实现，下面具体说明。
在步骤S2351中，按照预测周期，利用寻路预测模块对状态特征和第一编码后动作信息进行预测，得到目标寻路索引。
在本申请实施例中，目标寻路索引为第一寻路索引和第二寻路索引中的一种，其中，第一寻路索引和第二寻路索引为不同的寻路索引，并且第一寻路索引和第二寻路索引表征不同的寻路方式。示例性的，第一寻路索引表征基于原子移动的寻路方式，第二寻路索引表征按照规则移动的寻路方式。
在一些实施例中，预测周期为提前设置好的值，示例性的，预测周期的取值可以为6秒、7秒、8秒。
在一些实施例中，寻路预测模块预测得到的是目标寻路索引，目标寻路索引能够指示目标寻路方式。
在步骤S2352中，基于预设的索引映射表，确定目标寻路索引对应的目标寻路方式。
在一些实施例中，索引映射表中存储有寻路索引与寻路方式之间的映射关系，基于此，可从索引映射表中确定出目标寻路索引对应的目标寻路方式。
在步骤S2353中，将目标寻路方式确定为第一智能体的寻路方式。
在一些实施例中，当目标寻路索引为第二寻路索引时，确定第一智能体的寻路方式为按照规则的寻路方式，也可称之为按照规则移动的寻路方式。当目标寻路索引为第一寻路索引时，确定第一智能体的寻路方式为基于原子移动的寻路方式。
按照规则移动的寻路方式是指一种路径规划策略，它依赖于预定义的规则或指令来指导移动实体(如机器人、自动驾驶车辆等)在环境中从起始位置移动到目标位置，预定义的规则可以是避障规则、优先级规则、贪婪搜索规则等。基于原子移动的寻路方式是一种在微观层面上的路径规划策略，它将移动实体(如机器人、车辆等)视为可以自由移动的“原子”，并根据其物理特性或动力学模型来计算路径。
通过上述步骤S0231至步骤S0235，在利用寻路预测模块对寻路动作进行寻路方式预测的时候，是基于状态信息和第一编码后动作信息，如此，解耦寻路动作与第一动作间的依赖关系，实现在进行预测时已经融入了寻路动作与第一动作间的依赖关系，从而得到准确性高的寻路方式，为模型训练提供准确度更高的训练数据，不仅能够加快收敛速度，减少训练时间和迭代次数，提高训练效率，并且准确度更高的训练数据能够降低模型学习到噪声和无关特征的风险，从而避免出现过拟合现象，另外，模型在准确度更好的训练数据上训练，能够更好地识别和处理异常情况，增强模型的鲁棒性，从而提高模型的预测精度和泛化能力。
继续参见图4，接续上文的步骤S1023进行说明。
在步骤S1024中，基于寻路方式和各个动作信息，构建动作指令集合。
在本申请实施例中，动作指令集合中包括所有动作的动作信息，该动作指令集合用于指示第一智能体执行对应的动作。
参见图9，上述步骤S1024可通过以下步骤S0241至步骤S0243来实现，下面具体说明。
在步骤S0241中，获取预设的动作遮罩。
在本申请实施例中，动作遮罩与虚拟场景中的互动规则相匹配，动作遮罩用于限制第一智能体的动作选择空间。当虚拟场景是游戏对局过程中的虚拟场景时，互动规则可以为游戏规则。
在一些实施例中，动作遮罩可以是一种机制，用于在给定的状态下，限制第一智能体可以执行的动作范围，有助于第一智能体在特定情境下做出更合适的决策。动作遮罩可以视为一个二进制向量，其中每个元素对应一个动作。当一个动作遮罩值为1时，表示该动作可执行；而当遮罩值为0时，表示该动作不可执行。
在一些实施例中，首先需要明确互动规则，互动规则中包括哪些动作是可以执行的，哪些动作在特定情况下是被禁止的。例如，在棋类游戏中，游戏规则可能会禁止王移动到会被对方将军的位置。然后根据游戏规则，创建一个动作空间，它包含了智能体可以采取的所有可能动作。在游戏对局中，根据当前游戏状态，确定哪些动作是允许执行的，哪些动作是不允许执行的，进而将允许执行的动作对应的掩码值设置为1，将不允许执行的动作对应的掩码值设置为0，从而得到动作遮罩。
在步骤S0242中，将寻路方式和各个动作信息构建为候选指令集合。
在本申请实施例中，是将寻路方式和各个动作信息组合起来，构成候选指令集合。
在步骤S0243中，利用动作遮罩对候选指令集合进行遮罩处理，得到动作指令集合。
在本申请实施例中，动作信息中包括用于表征是否执行动作的第一标志值，第一标志值可以为第一预设值或第二预设值，其中当第一标志值为第一预设值时，表征不执行该动作信息对应的动作，当第一标志值位第二预设值时，表征执行该动作信息对应的动作。示例性地，第一预设值可以为0，第二预设值可以为1。动作遮罩可以为一个一维向量，每个向量值为一个掩码值，且每个掩码值对应一个动作信息。该掩码值为第一预设值或第二预设值，当掩码值为第一预设值时，说明需要对该掩码值对应的动作信息进行遮罩，当掩码值为第二预设值时，说明不需要对该掩码值对应的动作信息进行遮罩。
在利用所述动作遮罩对所述候选指令集合进行遮罩处理时，针对所述候选指令集合中的每一动作信息，从动作遮罩中获取所述动作信息对应的掩码值；当所述动作信息对应的掩码值为第一预设值，且动作信息中的第一标志值为第二预设值时，将所述动作信息中的第一标志值更新为第一预设值。也就是说当一个动作信息对应的掩码值为0时，说明需要对该动作信息对应的动作进行遮罩，也即不执行该动作信息对应的动作，那么需要继续确定该动作的动作信息中的第一标志值是否为1，如果该第一标志值为1，表征执行该动作，则将该动作的动作信息中的第一标志值更新为0，从而不执行该动作；如果该动作信息中的第一标志值不为1，也即为0，那么说明动作信息本身就表征不执行该动作，此时则保持动作信息不变。
在一些实施例中，当一个动作信息对应的掩码值为第二预设值，例如该掩码值为1，说明不对该动作信息进行遮罩，此时可以保持动作信息不变。
如此，通过动作遮罩对候选指令集合进行过滤，得到动作指令集合。
通过上述步骤S0241至步骤S0243，通过动作遮罩实现对寻路方式和各个动作信息的过滤、筛选，从而得到动作指令集合。实现对动作空间的约束，使得动作指令更符合实际情况或任务要求；还实现了对行为策略的优化，从而提高第一智能体的决策效率和性能，缩短模型训练的时长，提升模型训练的效率和准确率。
通过上述步骤S1021至步骤S1024，对不同模态信息分别进行编码，如此能够精确地表征游戏对局的局势状态，此外，特征提取单元融合了不同模态信息，通过特征提取得到信息量丰富的状态特征；接着，在进行预测时已经融入了动作之间的依赖关系，从而得到准确性高的预测结果；最后，通过动作遮罩实现对寻路方式和各个动作信息的过滤、筛选，实现了对动作空间的约束以及对行为策略的优化，从而提高第一智能体的决策效率和性能，提升模型训练的效率和准确率。
继续参见图3，接续上文的步骤S102进行说明。
在步骤S103中，控制所述第一智能体基于所述寻路方式进行移动，并基于各个所述动作信息执行对应的动作，得到执行结果。
在本申请实施例中，控制第一智能体执行动作指令集合中各个动作，以改变游戏对局的环境，从而得到执行结果。
在一些实施例中，当第一智能体的寻路方式为按照规则的寻路方式时，参见图10，上述步骤S103中的“控制所述第一智能体基于所述寻路方式进行移动，并基于各个所述动作信息执行对应的动作”可通过以下步骤S1031至步骤S1036来实现，下面具体说明。
在步骤S1031中，获取目标位置和第一智能体的当前位置。
在一些实施例中，可通过提前预设的方式得到目标位置，例如目标位置可以是固定位置或者特定的对象位置。目标位置还可以是随机生成的位置，如此能够增加游戏的随机性和挑战性。目标位置还可以是基于环境信息或者任务设定得到的位置。
在一些实施例中，当前位置是指第一智能体当前所处的位置。
在步骤S1032中，基于目标位置和当前位置，确定第一智能体的移动路径。
在一些实施例中，可通过路径规划算法基于目标位置和当前位置确定出移动路径，其中，路径规划算法可以为蚁群算法、迪杰斯特拉(Dijkstra)算法。
这里以路径规划算法为Dijkstra算法为例，说明确定第一智能体的移动路径的实现过程。首先获取第一智能体所在虚拟场景的地图数据，地图数据中包括多个节点；然后将所有节点的距离值设置为无穷大(表示从起始节点到该节点的距离未知)。将当前位置对应的节点确定为起始节点，并将起始节点的距离值设置为0(表示从起始节点到自身的距离为0)。之后创建一个优先队列，将所有节点添加到该优先队列中，起始节点具有最高的优先级(距离最小)；将优先队列中取出距离起始节点最近的未访问节点，确定为当前节点，再遍历当前节点的所有未访问的相邻节点，对于每个相邻节点，计算从起始节点到该节点的距离，如果计算出的距离小于该相邻节点当前记录的距离，则更新该相邻节点的距离，并将该节点放入优先队列。将当前节点标记为已访问，以防止它再次被处理。重复选择最近的未访问节点、更新相邻节点的距离以及标记当前节点为已访问，直到目标节点被标记为已访问或者优先队列为空(所有节点都已访问)。如果目标位置对应的目标节点被访问，从目标节点开始，根据记录的每个节点的前驱节点(在最短路径树中指向它的节点)，回溯到起始节点，从而得到路径规划结果，也即第一智能体的移动路径。
在步骤S1033中，基于移动路径，控制第一智能体按照规则的寻路方式进行移动；并基于各个动作信息，控制第一智能体执行各个动作。
在一些实施例中，控制第一智能体按照移动路径进行移动，还基于各个动作信息控制第一智能体执行各个动作。
在步骤S1034中，在控制第一智能体按照规则的寻路方式进行移动的过程中，当确定满足寻路方式切换条件时，将目标寻路索引更新为第一寻路索引。
在一些实施例中，在控制第一智能体按照规则的寻路方式进行移动的过程中，在确定出移动路径之后，则可将目标寻路索引更新为第三寻路索引，第三寻路索引用于表征保持寻路方式不变或者保持移动方式不变，也即保持移动路径不变，并基于移动路径按照规则进行移动，无需再确定移动路径。
在一些实施例中，当确定满足寻路方式切换条件时，可先将目标寻路索引更新为第四寻路索引，第四寻路索引用于表征暂停基于规则的寻路方式，从而及时暂停当前寻路方式；接着，再将目标寻路索引由第四寻路索引更新为第一寻路索引。
在一些实施例中，基于接收到的游戏对局数据，确定满足寻路方式切换条件的实现方式包括以下三种实现方式：
实现方式一，当确定第一智能体移动至目标位置时，确定满足寻路方式切换条件。
在这种实现方式下，表征第一智能体已经达到目标位置，也表征第一智能体已完成对移动路径的遍历，此时确定满足寻路切换条件。
实现方式二，当确定第一智能体由遮挡场景切换为非遮挡场景时，确定满足寻路方式切换条件，其中，遮挡场景表征第一智能体与第二智能体之间存在遮挡物，非遮挡场景表征第一智能体与第二智能体之间不存在遮挡物。
在这种实现方式下，可以先获取第一智能体的第一位置坐标以及第二智能体的第二位置坐标，基于第一位置坐标和第二位置坐标，确定第一智能体和第二智能体之间的直线，并沿着第一智能体和第二智能体之间的直线进行射线检测，以确定射线是否与游戏场景中的物体相交，如果射线与游戏场景中的物体相交，说明第一智能体与第二智能体之间存在遮挡物；如果射线与游戏场景中的任一物体均不相交，说明第一智能体与第二智能体之间不存在遮挡物。第一智能体与第二智能体之间无遮挡物，说明第一智能体与第二智能体能够相互看见对方，也表征游戏对局由非战斗场景进入至战斗场景，此时确定游戏对局数据满足寻路切换条件，从而为双方战斗提供路径保障，通过原子移动的寻路方式使得第一智能体迅速应对战斗场景。
实现方式三，获取第一智能体按照规则的寻路方式进行移动的移动时长，当移动时长达到时长阈值时，确定满足寻路方式切换条件，其中，时长阈值小于预测周期。
在这种实现方式下，表征目标位置和移动路径在移动时长这段时间内没有变化、更新，为了适应游戏的变化性及实时性，此时则确定游戏对局数据满足寻路切换条件，并将目标寻路索引更新为第一寻路索引。
在一些实施例中，时长阈值为提前设置好的值，示例性的，时长阈值可以为3秒、4秒。
在步骤S1035中，将第一智能体的寻路方式切换为按照原子移动的寻路方式。
在本申请实施例中，由于第一寻路索引与按照原子移动的寻路方式相对应，基于此，则将第一智能体的寻路方式切换为按照原子移动的寻路方式。
在步骤S1036中，控制第一智能体进行原子移动。
在一些实施例中，在预测周期的剩余时间中，控制第一智能体进行原子移动，直至预测周期结束。
通过上述步骤S1031至步骤S1036，在目标寻路索引为第二寻路索引时，先确定出移动路径，再基于移动路径控制第一智能体按照规则的寻路方式进行移动，如此只需要确定一次移动路径，从而节省计算资源，缩短模型训练时长，提升模型训练效率；接着，在第一智能体按照规则的寻路方式进行移动的过程中，如果确定满足寻路方式切换条件，则将目标寻路索引更新为第一寻路索引，并控制第一智能体进行原子移动，直至预测周期结束，如此在满足寻路方式切换条件时及时进行寻路方式切换，不仅能够提高寻路方式的灵活多变性，还能够提高寻路方式与第一智能体所处场景的适配性，从而为模型训练提高准确的训练数据，进而提升模型的预测准确性。
在一些实施例中，当第一智能体的寻路方式为按照原子移动的寻路方式时，则在整个预测周期内控制第一智能体进行原子移动。
继续参见图3，接续上文的步骤S103进行说明。
在步骤S104中，基于多个执行结果的奖励信息调整待训练的第一预测模型的参数，得到训练好的第一预测模型。
在本申请实施例中，奖励信息是基于多个执行结果、主奖励系数和辅助奖励系数确定的。
在一些实施例中，可基于每一执行结果的奖励信息分别调整待训练的第一预测模型的参数，也可基于各个执行结果的奖励信息之和来调整待训练的第一预测模型的参数。
参见图11，上述步骤S104可通过以下步骤S1041至步骤S1043来实现，下面具体说明。
在步骤S1041中，基于主奖励系数和辅助奖励系数确定奖励函数。
在本申请实施例中，主奖励系数用于表征当第一智能体执行了动作指令集合中动作信息对应动作时所赋予的奖励信息，辅助奖励系数用于表征当第一智能体执行预设拟人化动作时所赋予的奖励系数。主奖励系数和辅助奖励系数可以是预先设置好的。
在一些实施例中，预设拟人化动作可以为主动寻找掩体动作、开火拉扯动作、距离拉扯动作等。
在一些实施例中，可通过如下公式(1)来确定出奖励函数：
R＝α·Rmain+β·Rauxiliayr   公式(1)；
在公式(1)中，R表示奖励信息，α表示主奖励系数，β表示辅助奖励系数，Rmain表示主奖励值，Rauxiliayr表示辅助奖励值。
主奖励值通常与第一智能体完成主要任务的目标紧密相关。例如，如果智能体的主要任务是到达某个目的地，那么到达目的地后获得的主奖励值会比较高。主奖励值可以根据任务完成度、效率或者其他相关指标来设定。辅助奖励值是与主任务相关的次要目标或者行为带来的奖励。例如，避免敌人、保持健康状态、节省资源等。辅助奖励值可以帮助智能体在学习过程中形成更加全面和稳定的策略。
在步骤S1042中，利用奖励函数确定每一执行结果的奖励信息。
在一些实施例中，执行结果是指第一智能体在执行基于寻路方式的移动和对应动作后所获得的状态或反馈。执行结果可以包括第一智能体的当前位置、第一智能体与虚拟环境中物体或元素的交互结果(如收集物品、攻击敌人等)、智能体执行动作后环境的改变(如敌人的状态、物体的状态等)、智能体执行动作后的游戏分数中的至少之一。步骤1042在实现时，首先基于执行结果确定主奖励值和辅助奖励值，然后将该执行结果对应的主奖励值和辅助奖励值代入到公式(1)，得到该执行结果的奖励信息。该奖励信息可以为总奖励值。
在步骤S1043中，累加每一执行结果的奖励信息，得到累积奖励，将累积奖励反向传播至待训练的第一预测模型，并基于累积奖励调整待训练的第一预测模型的参数，直至达到训练结束条件，得到训练好的第一预测模型。
在一些实施例中，可累加一个或者多个预测周期内执行结果的奖励信息，得到累积奖励，然后基于累积奖励调整待训练的第一预测模型的参数，当达到预设调整次数或者累积奖励达到奖励阈值时，确定达到训练结束条件，从而得到训练好的第一预测模型。
通过上述步骤S1041至步骤S1043，通过引入辅助奖励系数，能将待训练的第一预测模型朝着预设拟人化动作的方向训练，从而使得训练好的第一预测模型能够控制第一智能体在确保能力强度的同时，控制第一智能体具备拟人化风格，提升训练好的第一预测模型的预测性能。
在一些实施例中，在得到训练好的第一预测模型之后，在游戏对局开始后，可以通过训练好的第一预测模型基于第一智能体的状态数据进行预测，得到动作指令集合，然后控制第一智能体基于动作指令集合移动，并基于动作指令集合中各个动作信息执行对应的动作，从而与虚拟对象进行交互，虚拟对象可以是与第一智能体进行对抗的游戏角色，该虚拟对象可以是由真实玩家控制的游戏角色，也可以是由第二预测模型驱动的第二智能体。
通过上述步骤S101至步骤S104，由于待训练的第一预测模型能够基于第一智能体所处的不同场景，控制第一智能体按照不同的寻路方式进行移动，那么当第一智能体在游戏对局中处于不同场景时，则能够预测得到包含不同寻路方式的动作指令集合，该动作指令集合中的寻路方式与第一智能体所处场景相匹配，如此实现寻路方式的多样化。另一方面，由于训练好的第二预测模型也能够基于第二智能体所处的不同场景，控制第二智能体按照不同的寻路方式进行移动，也即第二智能体也能够在不同的场景下执行不同的寻路方式，如此能够强化进行对抗的第一智能体的在不同的场景下执行不同的寻路方式的能力。使得寻路方式、移动方式多样化，通过多样化的寻路方式提升待训练的第一预测模型的训练效果，能够提升待训练的第一预测模型的训练准确性。此外，通过辅助奖励系数，能够将待训练的第一预测模型朝着预设拟人化动作的方向训练。
在一些实施例中，在接收到游戏对局指令之后，会先确定出第一智能体的第一初始位置以及第二智能体的第二初始位置，并在第一初始位置生成第一智能体，在第二初始位置生成第二智能体。基于此，参见图12，在上述步骤S101之前还可执行以下步骤S001至步骤S009，下面具体说明。
在步骤S001中，响应于游戏对局开启指令，基于等概率原则，从游戏对局对应的虚拟场景所覆盖的各个子区域中确定选中区域。
在一些实施例中，游戏对局开启指令的形式可以为触控形式、手势形式、语音形式等。
在一些实施例中，可将各个子区域的概率权重设置成相同值，并将各个子区域关联一个数字，再利用随机数生成器生成一个随机数，将该随机数关联的子区域确定为选中区域。
在步骤S002中，确定选中区域的第一中心点。
在一些实施例中，可采用几何中心法、重心法、点集法、距离加权法等方式确定出选中区的第一中心点的坐标。
在步骤S003中，基于第一中心点和预设第一最大寻路距离，确定第一出生区域，并确定出第一出生区域中的第一可用点集合。
在一些实施例中，可以第一中心点为圆心，以预设第一最大寻路距离为半径，确定出一个圆形的第一出生区域；也可以第一中心点为中心，以预设第一最大寻路距离为边长，确定出一个正方形的第一出生区域。
在一些实施例中，第一可用点(Navigable Points)指的是第一智能体可以移动到的有效位置或区域。这些点通常是第一智能体在游戏对局中进行路径规划和导航时可以考虑的目标点。第一可用点的确定通常基于以下几个方面：
第一方面，地形特征：可用点通常位于游戏地图中的平坦、可通行的地形区域，如平地、道路、楼梯等。这些区域通常是第一智能体可以安全移动和操作的区域。
第二方面，障碍物避让：第一可用点通常需要避开游戏地图中的障碍物或障碍物组成的区域，如墙壁、建筑物、水域等。第一智能体在进行路径规划时需要考虑如何绕过这些障碍物，找到安全的路径。
第三方面，目标点：第一可用点通常也包括游戏中的目标点或目标区域，如任务目标、敌人位置、资源点等。第一智能体可以通过路径规划找到最佳路径来达到这些目标点。
第四方面，玩家交互点：第一可用点还可以包括玩家可以进行交互的点或区域，如商店、NPC位置、交通工具等。第一智能体可以根据玩家的需求和行为选择合适的路径和目标点。
通过确定游戏地图中的可用点，可以为第一智能体提供有效的移动目标和路径规划的参考，从而使其在游戏中表现出更智能和灵活的行为。
在本申请实施例中，所有第一可用点构成第一可用点集合。
在步骤S004中，从第一可用点集合中确定第一智能体的第一初始位置，并在第一初始位置生成第一智能体。
在一些实施例中，可随机将第一可用点集合中的任意第一可用点确定为第一初始位置，并在该第一初始位置生成第一智能体，也即第一初始位置为第一智能体的出生位置。
在步骤S005中，将第一初始位置确定为第二中心点。
在步骤S006中，获取第二智能体对应的预设第二最大寻路距离，以及第二智能体对应的预设第二最小寻路距离。
在一些实施例中，预设第二最大寻路距离和预设第二最小寻路距离可以为提前预设的值。
在步骤S007中，基于第二中心点、预设第二最大寻路距离和预设第二最小寻路距离，确定第二出生区域，并确定出第二出生区域中的第二可用点集合。
在一些实施例中，可讲第二中心点作为圆心，将预设第二最大寻路距离作为外圆半径，将预设第二最小寻路距离作为内圆半径，确定出一个圆环，并将该圆环确定为第二出生区域。
在一些实施例中，确定第二出生区域中的第二可用点集合的实现过程，与上述步骤S003中“确定出第一出生区域中的第一可用点集合”的实现过程相类似，因此，确定第二出生区域中的第二可用点集合的实现过程，可参考上述步骤S003中“确定出第一出生区域中的第一可用点集合”的实现过程。
在步骤S008中，从第二可用点集合中确定候选初始位置。
在一些实施例中，在一些实施例中，可随机将第二可用点集合中的任意第二可用点确定为候选初始位置。
在步骤S009中，当候选初始位置与第一初始位置之间存在遮挡物时，将候选初始位置确定为第二智能体的第二初始位置，并在第二初始位置生成第二智能体。
在本申请实施例中，当候选初始位置与第一初始位置之间存在遮挡物时，将候选初始位置确定为第二智能体的第二初始位置，并在第二初始位置生成第二智能体。
在一些实施例中，当候选初始位置与第一初始位置之间不存在遮挡物时，则返回上述步骤S008，重新确定候选初始位置。
通过上述步骤S001至步骤S009，一方面，可提升第一智能体与第二智能体生成的随机性，提高每个区域作为选中区域的概率，还能确保选中区域内第一智能体与第二智能体双方完全对称；另一方面，通过两个出生位置之间存在障碍物，能够保证样本的多样性，也避免了开局瞬间有一方被对方击杀，能够提高游戏对局时间，保证得到足够的训练样本，从而提升模型训练效率。
下面，将说明本申请实施例在一个实际的应用场景中的示例性应用。
本申请实施例提出一种模型训练方法，该模型训练方法是基于大规模强化学习训练框架，结合近端策略优化算法，在超大地图场景、复杂多样的三维地形和高维结构化动作空间下，通过原子移动和导航网格移动相结合的方式，提出一套规则增强的混合移动寻路方案，再利用神经网络强大的自适应学习探索能力，来实现可以投放在超大地图、复杂多样地形下PVE、PVP的陪玩AI训练方案，其中，陪玩AI对应其他实施例中的第一智能体。
本申请实施例在原始动作空间基础上，额外引入导航网格移动作为全局移动逻辑，并采用集中式训练、混合式执行的模型架构。在战斗场景下，让AI重点学习精妙走位，此时神经网络更侧重选择原子移动方式，以便学到极致的微小操作，做出类似人类玩家的操作，来提高AI的拟人性。而在非战斗场景下，神经网络会淡化寻路学习过程，更倾向于选择导航网格的全局移动方式，以便能更快地遇到目标敌人快速切换到战斗场景中，使得对局能够生产更多战斗样本，提高训练效率，其中，目标敌人对应其他实施例中的第二智能体。
在一些实施例中，还引入完备的辅助奖励体系来增强策略的拟人性，在双人零和博弈场景下，使用自博弈训练来不断提高策略强度。混合寻路强化学习训练方案可以充分发挥强化学习的决策探索优势，根据对战局势自适应灵活选择何种寻路方式，不仅能兼顾全图寻路移动和战斗博弈能力，还能够提高训练效率，真正做到在超大地图、复杂多样三维地形场景下，生产满足业务要求的高水平陪玩AI，帮助业务团队更高效进行内容生成。
通过本申请实施例提供的模型训练方法，能够解决如下四个目的：
第一个目的，通过大规模分布式强化学习方式，加快采集训练数据用于策略学习；第二个目的，通过规则增强的混合寻路方式，配合强化学习的集中式训练，规避陪玩AI角色在移动或执行动作时，由于地形设计，导致陪玩AI角色被困在某个位置或无法按照预期进行移动的问题；第三个目的，实现在超大地图范围和复杂多变的三维地形场景中，训练出兼顾全图导航和战斗博弈的陪玩AI，不仅提高模型效果上限，也大大加快训练效率；第四个目的，通过全面奖惩机制增强模型拟人性和风格化引导，进一步加速模型适配落地。其中，模型对应其他实施例中的待训练的第一预测模型。
接下来将从三个方面来阐述本申请实施例提供的模型训练方法，第一方面是业务目标，业务目标是本申请实施例所依托的任务背景，需要明确AI所承担的任务和具体的作用。第二方面是环境交互，环境交互模块具有通用性，采用状态同步方式的网络游戏均可以采用这套训练推理模式。第三方面是训练方案，本申请实施例提出了具有差异化的训练方案，混合寻路方式的引入加速训练效率，提升模型上限，高度融合的神经网络和完备的奖惩体系有利于引导模型拟人性风格，有助于提升模型拟人性。
第一方面，业务目标。
由于农场地图是玩家活跃度最高的地图，以应用场景为农场地图举例来说，相关技术已经投放了行为树版本普通AI，用于陪玩和填充对局数，但是由于现有行为树版本AI逻辑较容易被玩家识别，导致玩家失去对战乐趣。基于此，业务目标为通过强化学习技术生产出高强度强拟人性AI，用于投放在高端对局中陪玩，要求在整张地图中，可以进行全图移动，合理利用掩体战斗和拉扯，走位灵活自然，开火表现拟人，游戏淘汰回放可以对齐人类玩家视角，打法展现出较高拟人性和多样性，真正做到强拟人化陪玩目的，从而带动游戏活跃度和玩家参与度。
由于陪玩AI会投放在农场地图的任意区域，且任意区域都可能会与玩家战斗，所以对模型泛化能力提出很高要求，因此，需要在模型训练对战逻辑与场景开发上做细致优化，实现训练场景覆盖全地图。参见图13，将农场地图划定为八个关键战斗区，分别为1301、1302、1303、1304、1305、1306、1307和1308，每个游戏对局开启时会按照等概率选择其中的一个战斗区，以提高主事件发生概率。
在一些实施例中，每个战斗区内对战双方(也即第一智能体和第二智能体)完全对称，并按照特定逻辑选择双方出生点，为了避免开局瞬间有一方被另一方击杀，要求彼此出生点之间有障碍物遮挡，保证样本多样性。在训练开始时，先随机选择一个战斗区，对于选中的战斗区，进一步随机选择内部出生点组合，其中一方死亡或者对局超时，单局结束，来提升训练效率。图14A是本申请实施例提供的确定对战配置逻辑的一种流程示意图，包括以下步骤S1401至步骤S1410，下面具体说明。
在步骤S1401中，确定选中战斗区。
承接上面的示例，是按照等概率原则从上述八个关键战斗区中选取战斗区，得到选中战斗区。其中，选中战斗区对应其他实施例中的选中区域。
在步骤S1402中，确定选中战斗区的中心点坐标。
在一些实施例中，可采用几何中心法、重心法、点集法、距离加权法等方式确定出选中战斗区的中心点坐标。
在步骤S1403中，获取第一智能体的出生最大寻路距离。
在一些实施例中，在游戏设计和开发过程中，会设定出第一智能体的出生最大寻路距离，也即第一智能体的出生最大寻路距离是提前预设好的值。
在步骤S1404中，确定第一出生区域。
在本申请实施例中，是以中心点坐标为圆心，以第一智能体的出生最大寻路距离为半径，确定出第一出生区域。参见图14B，401为选中战斗区。
在步骤S1405中，在第一出生区域范围内采样导航网格可用点，得到第一智能体的出生坐标。
在本申请实施例中，第一智能体的出生坐标可记为第一个出生点。继续参见图14B，402为第一个出生点。
在一些实施例中，上述步骤S1405的实现过程，与上述步骤S003中的“确定出第一出生区域中的第一可用点集合”以及上述步骤S004中的“从第一可用点集合中确定第一智能体的第一初始位置”的实现过程相类似。因此，上述步骤S1405的实现过程，可参考上述步骤S003中的“确定出第一出生区域中的第一可用点集合”以及上述步骤S004中的“从第一可用点集合中确定第一智能体的第一初始位置”的实现过程。
在步骤S1406中，获取第二智能体的出生最小寻路距离和出生最大寻路距离。
在一些实施例中，第二智能体的出生最小寻路距离和出生最大寻路距离，也是提前预设好的值。
在步骤S1407中，确定第二出生区域。
在本申请实施例中，是以第一智能体的出生坐标作为圆心，以第二智能体的出生最小寻路距离作为内圆半径，以第二智能体的出生最大寻路距离作为外圆半径，确定出一个圆环，将该圆环确定为第二出生区域。继续参见图14B，403为第二出生区域。
在步骤S1408中，在第二出生区域范围内采样导航网格可用点，得到采样点。
在一些实施例中，上述步骤S1408的实现过程，与上述步骤S007中的“确定出第二出生区域中的第二可用点集合”以及上述步骤S008中的“从第二可用点集合中确定候选初始位置”的实现过程相类似。因此，上述步骤S1408的实现过程，可参考上述步骤S007中的“确定出第二出生区域中的第二可用点集合”以及上述步骤S008中的“从第二可用点集合中确定候选初始位置”的实现过程。
在步骤S1409中，判断采样点与第一智能体的出生坐标之间是否存在遮挡。
在本申请实施例中，当采样点与第一智能体的出生坐标之间存在遮挡时，则能够避免游戏对局开局瞬间有一方被秒掉的现象，表征采样点能够作为第二智能体的出生坐标，则进入步骤S1410。而当采样点与第一智能体的出生坐标之间不存在遮挡时，则会产生游戏对局开局瞬间有一方被秒掉的现象，表征采样点不能够作为第二智能体的出生坐标，则返回步骤S1408，继续进行采样。
在步骤S1410中，将采样点确定为第二智能体的出生坐标。
在本申请实施例中，第二智能体的出生坐标可记为第二个出生点。继续参见图14B，404为第二个出生点。
第二方面，环境交互。
图15是本申请实施例提供的模型训练阶段的总体结构示意图，其中，游戏客户端1501是指合入AI Service SDK进行接口调用的游戏环境，游戏客户端向AI服务器发送开局请求，由服务器侧根据训练需求返回对应的对战配置数据，游戏客户端获取数据后构建对局并生成强化学习智能体，之后的每次请求响应的游戏帧样本都会处理成状态数据1502发送给强化学习框架1503进行交互，AI训练服务对当前游戏的请求帧进行动作预测，网络动作拼接成可执行的动作指令1504，并将可执行的动作指令1504回传给客户端游戏引擎执行，然后AI训练服务继续等待下一次游戏帧样本的请求响应。模型训练阶段会定期向训练平台上报业务侧统计指标，用于掌握训练效果，每个游戏对局结束时候，游戏客户端会调用有关逻辑接口销毁本局智能体，并以相同配置自动开启下一轮游戏对局，如此反复采集数据提供给强化学习框架做训练。
在本申请实施例中，模型训练阶段采用同步推理模式，同步推理业务流如图16所示。训练阶段的每一局游戏里是没有人类玩家的，也没有各种业务行为树AI和无关游戏业务逻辑，游戏内核(DS)里只有两个训练对战AI在运行，游戏业务逻辑简洁、纯粹，无需消耗大量服务器性能去做碰撞和射线检测，而且AI从发送游戏当前帧响应到拿到动作回包的这段时间间隔内，游戏内核处于同步等待状态，其中，状态请求03是由游戏内核01向AI训练服务1602发送的，动作回包04是由AI训练服务1602返回至游戏内核01，AI训练服务1602对应其他实施例中待训练的第一预测模型。只有拿到动作回包后才会继续运行游戏内核逻辑，所以训练阶段的游戏内核是状态请求03、动作回包04的不断循环过程，每间隔n个游戏帧做一次动作请求，模型训练阶段的同步推理方式可以将游戏内核本身游戏业务逻辑和强化学习训练统一在同一条时间线下运行。其中，n的取值范围为2至10。
在本申请实施例中，模型上线阶段采用异步推理模式，异步推理业务流如图17所示。上线阶段的每一局游戏里是同时存在大量线上玩家的，玩家和玩家之间，玩家和各种AI之间都会产生游戏互动逻辑，如果上线阶段还采取同步推理方式，则游戏内核每次发送游戏帧请求响应时，都会因为构建状态数据而在游戏内核里做大量碰撞和射线检测，这会消耗掉大部分服务器性能，这种服务器资源消耗会随着上线后每一局内AI投放数量的增大成正比，而且更为严重的是，由于同步推理方式每次状态请求、动作回包的过程中，游戏内核处于同步等待的状态，这段时间内不会运行业务本身游戏逻辑，只有拿到动作回包后才继续运行游戏内核游戏业务逻辑，导致这段同步等待空窗期内(可能只有几十毫秒)玩家会感觉到游戏出现帧卡顿现象，而且这种帧卡顿现象会随着模型推理耗时以及因网络波动导致的回包延迟进一步放大，这对于一款主打高画质拟真硬核的FPS游戏而言是不可接受的。所以必须要隔离游戏内核01的游戏业务逻辑和推理服务1702的逻辑，确保推理服务1702不会卡住游戏内核01本身游戏业务逻辑，同时将原本只会在游戏内核01侧进行的大量碰撞和射线检测工作剥离出游戏内核环境，直接迁移到推理服务1702上开发，推理服务1702会预先保存一份庞大的静态碰撞数据，每次都会对游戏内核01发送过来的状态请求03做射线检测查询工作，并向游戏内核01返回动作回包04，从而避免每次在游戏内核01侧动态做射线检测工作导致大量消耗游戏内核01性能，如此，只需要关注推理服务1702的逻辑，以提升开发效率。其中，推理服务1702对应其他实施例中的训练好的第一预测模型。
基于此，模型上线阶段采用异步推理模式，异步推理存在两条时间线，游戏内核侧定时(每间隔4帧)发送基础性的状态请求数据给推理服务，推理服务侧按照先进先出原则，对发过来的基础状态数据进行静态碰撞和射线检测查询工作，再加工处理成网络状态数据做推理计算，然后继续按照先进先出原则依次返回动作指令包给游戏内核侧。由于是异步推理方式，游戏内核每隔4帧稳定地发送状态请求响应后无需等待动作回包，继续运行自身游戏业务逻辑，同时每帧检测是否有动作回包，如有则立即执行最新动作。此外，如果推理服务侧动作回包的速度受到推理计算耗时和网络波动延迟等因素的影响，导致回包速度可能无法及时跟上发包的速度，随着回包数量越堆越多，模型预测的动作时效性会因回包不及时而越来越差，导致推理服务失去意义，所以需要确保推理服务的回包速度大于等于发包速度，至少要达到收发包的动态平衡，基于此，则可以通过增大请求间隔帧数n来降低发包速度，或者优化模型结构降低推理耗时来加速发包速度。如果回包堆积数量过大则通过回滚成行为树AI进行兜底。异步推理能够防止数据阻塞主线程，保证神经网络推理耗时以及因网络波动导致的动作回包延迟等因素不影响其他游戏业务逻辑，还能降低业务侧服务器占用情况，确保模型推理的安全性，如果推理会因为网络波动等因素导致推理失败，可以通过业务逻辑切换成普通的行为树AI，确保游戏逻辑稳定正常运行，这种混合式强化学习接入逻辑具有易用灵活安全等特点。
第三方面，训练侧。
针对一款主打战术博弈的第一人称射击手游，通常情况下，游戏地图场景足够开放多样，玩法也各有不同，类似真实世界的三维场景里有大量静态、动态与环境感知相关的物体，而且细节刻画十分逼真，三维地形也足够复杂，游戏操作面板也包括众多能同时执行的摇杆动作。在这种超大地图，立体复杂多变的游戏环境下想要训练出能全图流畅移动、合理利用各种掩体战斗拉扯的高强度陪玩AI，对于AI的流畅跑图、全图移动能力提出很高要求。为此，本申请实施例采用规则增强的混合寻路方式，配合丰富完备的奖惩体系优化，在双人零和博弈场景下做自博弈迭代训练，最终收敛至纳什均衡策略，此时模型的强度、拟人性均能满足业务目标。
在整张超大且复杂的农场地图三维场景中，如果仅使用十六方向的移动方式，AI会很容易被各种奇怪地形卡住，导致大量的训练对局里，神经网络都把重点放在学习如何寻路导航上，从而忽视学习最重要的战斗博弈拉扯能力，其中，十六方向的移动方式为间隔22.5°的原子移动。此外，即便网络已经重点学习如何导航寻路，由于训练地图规模实在过于庞大、地形过于复杂多变，AI依然很难做到流畅不卡地全图自由移动，更严重的是，战斗交火可能在地图上的任意区域发生，很容易出现AI正在被奇怪地形卡住时就进入了交火战斗情况，最终发现AI学习出来的最优策略就是在出生点附近被动等待目标敌人主动来找自己，然后自己被动反击，依靠提前站位、蹲伏等先发优势战胜主动靠近的目标敌人，这种策略明显不符合业务目标。还会使得训练周期拉长，时间成本拉大。
为了提高训练效率，同时兼顾全图寻路移动效果和战斗博弈能力，本申请实施例采用的是混合寻路方案，使用了基于规则增强的集中式训练、混合式执行的寻路移动方案，直接接入UE4底层导航网格寻路规则作为全局移动使用，同时原先的原子移动保留作为局部移动，为了灵活选择何时采用全局移动何时采用局部移动，神经网络新增一个混合寻路动作头，该寻路动作输出头用于确定采用哪种寻路方式，混合寻路方案的优点在于：
第一，依靠神经网络强大的自适应学习能力，自由灵活选择是规则移动还是原子移动。在非战斗场景下，网络更多学习到了规则移动，淡化其寻路学习过程。而在战斗场景下，网络更多学习到使用原子移动来进行局部走位、微操，让网络重点学习战斗博弈过程。不同场景下AI都有针对适应性的移动策略，显得十分灵活可控，真正做到全图范围的流畅寻路移动，同时对模型强度的提升有所帮助。
第二，使用混合寻路移动方案后，AI不容易被奇怪地形卡住了，训练对局可以生成更多战斗样本而非寻路样本，有利于网络更多关注学习彼此博弈拉扯的战斗打法上，加速了训练效率，节约训练的时间成本。
本申请实施例提供的混合寻路方案训练流程包括：首先，采用三指手法建模方案，将代表人类操作上限的三指操作规则去剪枝(或者过滤)大量非法无效的并行动作，以此来获得训练效率和拟人性的双重提升。
在一些实施例中，网络动作输出头类别及含义如表1所示，表1中Path type动作头表示选择规则移动方式或是选择原子移动方式。其中，Path type动作头还包含“保持前一次移动方式”这一维动作含义，这是因为规则移动逻辑是：先确定寻路目标点，基于该寻路目标点UE4去查询移动到该目标点的最优路径；再借助UE4底层导航网格沿着查询到的最优路径移动到寻路目标点，其中，寻路目标点对应其他实施例中目标位置，最优路径对应其他实施例中移动路径。
表1网络动作输出头类别及含义

在一些实施例中，由于查询最优寻路移动路径操作会消耗服务器性能，如果网络动作头的预测频率较高，会连续预测规则移动，也会短时间里连续不断地密集查询最优路径，从而导致短时间内大量占用业务团队服务器性能开销，带来游戏本身业务逻辑卡顿，影响线上的玩家游戏体验。此外，从上帝视角来看，模型可能并不需要在短时间内反复预测借助导航网格的规则移动，因为借助规则移动主要是发生在非战斗场景下的寻路，此时目标敌人的位置可能不会有太多变化，意味着到达目标敌人的最优路径可能不会有太多变化，上一次查询的最优路径在接下来一段时间内依然具有时效性，模型仍然能继续沿用上一次查询的最优路径。并且即便目标敌人位置有所变化，在非战斗场景下，模型只需要能做到全图流畅寻路移动，不被奇怪地形卡住即可，对于模型的走位、微操方面并无太多严苛的要求，所以模型依然可以沿用前一次查询到的最优路径。
通过本申请实施例提供的混合寻路方案，在非战斗场景下，模型并不需要频繁更换最优路径，仍然可以沿用前一次查询返回的最优路径，单次查询还能避免反复查询最优路径带来的不必要性能消耗，所以神经网络并不需要在短时间内频繁预测规则移动指令，只需要预测一次规则移动，后面的时间内都保持上次预测规则移动。UE4查询到的最优路径，则可继续移动，基于规则移动时是使用导航网格移动，也就是说，表1中Path type动作索引2所表示的含义为，使用导航网格沿用上次查询的最优路径移动。
在一些实施例中，当触发特定逻辑后，比如智能体已经达到寻路目标点的位置；或者在沿着最近查询到的最优路径进行移动过的程中提前遭遇到目标敌人，并进入战斗场景；或者沿着最近查询到的最优路径已经行走到一个时间阈值，在上述三种情况下，会打断导航网格的规则移动方式，对应表1中Path type动作索引3，表示停止辅助移动。等到下次网络预测的时候，重新开始选择辅助规则移动或者原子移动。在本申请实施例中，每次借助导航网格移动时候，都有时间阈值约束，不管有没有达到寻路目标点，只要到了时间阈值约束，就会停止基于导航网格的规则移动，等到下次神经网络决策时候，重新进行寻路方式选择。在整个混合寻路方案过程中，只有在Path type预测动作指令1的时候，进行最优路径查询工作，才会消耗服务器性能，后续只需要持续预测动作指令2，来保持基于导航网格的辅助规则移动即可，做到只占用极小的性能开销就能完成全图范围的移动导航，也不会影响战斗场景下的智能体的各种走位博弈，真正能够最大效率发挥强化学习的决策探索优势，有针对性的自适应学习不同场景下的寻路策略和博弈策略。如此，可以把重点都放在如何博弈拉扯上面，淡化非战斗场景下的寻路过程，加速训练效率的同时也提升了模型上限效果。
在本申请实施例中，预测混合寻路动作头的网络结构如图18A所示，将输入信息1801输入至神经网络1802，利用神经网络1802预测得出寻路索引1803，当寻路索引为0时，进行原子移动，动作指数为(0-15)；当寻路索引为1时，进行基于规则移动，动作指数为16；当寻路索引为2时，保持基于规则移动，动作指数为16；当寻路索引为3时，停止基于规则移动，动作指数为(0-15)。其中，输入信息1801对应其他实施例中的状态数据。接着，当寻路索引为1时，还会确定寻路目标点1804以及最佳路径1805；而当寻路索引为2时，继续沿着最佳路径1805移动，直至移动至寻路目标点1804。在一些实施例中，如果输入信息1801表征处于非对战场景，则移动至寻路目标点1804；如果输入信息1801表征触发上述三种特定逻辑中的任意一种时，则停止基于规则移动。
本申请实施例的混合寻路方案步骤如下：
步骤一，游戏对局按照时间切成若干时间片，每个时间片T的起始时刻由网络决定何种寻路方式。
在本申请实施例中，时间片T对应其他实施例中的预测周期。参见图18B，在时间片1和时间片4中，起始时刻的寻路方式为原子移动方式。
步骤二，当每个时间片段T的起始点选择规则移动时候，基于寻路目标点，UE4查询最优路径，然后借助导航网格移动过去，后续在该时间片段内保持该最优路径，并利用导航网格移动过去。
参见图18B中的时间片2、时间片3和时间片4。如此，只需进行一次寻路路径寻优，单次最优路径的查询，有利于降低服务器性能消耗。
步骤三，单个时间片段内，达到寻路目标点、达到时间阈值约束、有一方被看见的情况发生至少之一时，立即结束规则移动，在该时间片的剩余时间，重新切换成原子移动方式。
参见图18B，时间片2在达到时间阈值时将移动方式由规则移动方式切换为原子移动；时间片3在有一方被看见时将移动方式由规则移动方式切换为原子移动；时间片4在达到寻路目标点时将移动方式由规则移动方式切换为原子移动。其中，有一方被看见对应其他实施例中的第一智能体由遮挡场景切换为非遮挡场景。
步骤四，寻路目标点的选择，可以由模型输出也可以由规则输出，示例性的，可将寻路目标点设置为最近一次目标敌人声纹感知点。
图18A展示了寻路方式(Path type)网络动作头的决策逻辑，输入信息1801包括多模态环境信息组成，Path type动作指令含义如表1所示，其中，寻路方式网络动作头对应其他实施例中的混合寻路动作头。参见图18B，将游戏对局划分成多个时间片，每个时间片的起点时刻由神经网络Path type动作头，根据当前游戏局势(也即AI所处场景)选择具体寻路方式。“0”表示整个时间片段T都是原子移动；“1+2”表示先做动作指令“1”，随后进行动作指令“2”，表征先查询最优路径，再借助导航网格沿着最优路径移动；3表示停止辅助规则移动。
在一些实施例中，图19是本申请实施例提供的混合寻路的一种流程示意图，下面具体说明。
在步骤S1901中，进入下一时间片。
在本申请实施例中，是指当前时间片结束，进入新的时间片。
在步骤S1902中，判断是否处于对战场景。
在本申请实施例中，如果处于非对战场景，则进入步骤S1903；如果处于对战场景，则进入步骤S1905A。
在步骤S1903中，判断是否满足上述三种特定逻辑中的任意一种。
在本申请实施例中，如果不满足上述三种特定逻辑中的任意一种，表征此时没有达到寻路目标点，此时也没有达到时间阈值约束，此时没有一方被看见，则进入步骤S1904；如果满足上述三种特定逻辑中的任意一种，表征此时达到寻路目标点，或者此时达到时间阈值，满足了时间阈值的约束，或者此时有一方被看见，则进入步骤S1905D。
在步骤S1904，获取寻路索引。
在本申请实施例中，当寻路索引为0时，进入步骤S1905A；当寻路索引为1时，进入步骤S1905B；当寻路索引为2时，进入步骤S1905C；当寻路索引为3时，进入步骤S1905D。
在步骤S1905A中，控制智能体进行原子移动，并结束流程。
在步骤S1905B中，控制智能体基于规则移动。
在步骤S1905C中，控制智能体保持基于规则移动，并进入步骤S1907。
在步骤S1905D中，控制智能体停止基于规则移动，并结束流程。
在本申请实施例中，上述步骤S1905A、步骤S1905B、步骤S1905C以及步骤S1905D是根据寻路索引择一执行的。
在步骤S1906中，确定最优路径。
在本申请实施例中，最优路径是基于当前位置和寻路目标点确定出来的。
在步骤S1907中，保持当前最优路径。
在本申请实施例中，表征无需更新最优路径，继续保持当前最优路径。
在步骤S1908中，移动至寻路目标点。
在本申请实施例中，是借助导航网格沿着最优路径移动至寻路目标点。
在一些实施例中，可对于非战斗场景下，神经网络输出一个寻路目标点，用强化学习训练智能体靠近这个目标点，仅当智能体到达这个寻路目标点后，才会重新设置新的寻路目标点，智能体没到达寻路目标点的时候，用奖励鼓励智能体靠近寻路目标点，这种方式也可以达到相同非战斗场景下的寻路目的，同时也能避免多次路径查询操作而带来的服务器性能消耗。
通过上述步骤S1901至步骤S1908，能够在对战场景下，进行原子移动，在非对战场景下，进行基于规则的移动方式，能够进行匹配场景的移动方式，提升模型训练效率。此外，在进行基于规则的移动方式时，仅需确定一次最优路径，能够节省计算资源，从而加快模型训练速度。
图20是本申请实施例提供的模型结构的一种结构示意图，参见图20，该模型结构包括感知部分2001和决策部分2002。其中，感知部分2001包括环境信息0011、标量信息0012、深度图信息0013、环形射线信息0014、第一编码器0015、第二编码器0016、第三编码器0017、第四编码器0018、特征提取单元0019和全局信息0020；决策部分2002包括动作遮罩0021、策略网络0022和价值网络0023。
在本申请实施例中，参见图20，第一编码器0015用于对环境信息0011进行编码，得到编码后的环境信息；第二编码器0016用于对标量信息0012进行编码，得到编码后的标量信息；第三编码器0017用于对深度图信息0013进行编码，得到编码后的深度图信息，其中，第三编码器0017可以为Conv2D；第四编码器0018用于对环形射线信息0014进行编码，得到编码后的环形射线信息，其中，第四编码器0018可以为Conv1D。
继续参见图20，拼接编码后的环境信息、编码后的标量信息、编码后的深度图信息和编码后的环形射线信息，得到综合编码信息0024。接着，利用特征提取单元0019对综合编码信息0024进行特征提取，得到状态特征，其中，特征提取单元0019可以为长短期记忆神经网络。接着，将状态特征输入至策略网络0022，策略网络0022结合动作遮罩0021预测出动作指令集合0025。
继续参见图20，游戏开局则开启训练，游戏内核获取运行时状态数据，并按照表2将状态数据划分成不同模态感知数据，不同模态感知数据各自独立由不同种类的神经网络做特征提取，这样能更准确精细地表征当前游戏局势状态。其中，不同模态感知数据为环境信息、标量信息、深度图信息和环形射线信息。
表2基于多模态环境感知的状态数据划分结果表
继续参见图20，价值网络0023输入除了特征提取单元0019输出的隐藏状态编码信息，还包括全局信息，全局信息是获取自身和目标敌人的全部真实信息，这种包括“作弊”信息的全局输入特征可以降低价值网络0023的值估计偏差和方差，有利于策略收敛稳定，这种方式只需在模型训练阶段用到，因为实际上线部署阶段只需要策略网络不需要价值网络。此外，本申请实施例还将自回归嵌入引入到模型设计中，继续参见图20，每层动作头采样动作都做独热编码处理成嵌入向量，然后再和下一层的输入向量叠加起来作为下一层动作头的输入，逐层影响形成一种前后因果依赖，使得后续动作头做出的决策更加合理自然，图20所示模型训练结构设计优点如下：
第一，高度复杂的神经网络融合了列表、图像、标量、布尔等多模态输入信息；
第二，通过自回归嵌入的网络动作头设计，解耦FPS游戏结构化动作空间之间的依赖关系；
第三，动作遮罩加速训练效率，同时也提升了策略拟人性；
第四，不同类型输入分开激活，提高网络对输入感知的精准度。
由于原有的奖惩设置会使得智能体表现得只顾以击杀玩家为目标，打法无脑激进，更多是靠提前站位开枪、超高命中率获胜，缺少周旋博弈的过程，智能体也不会利用掩体进行拉扯战斗，整体对战的拟人性较低，无法满足业务侧的上线标准。因此，本申请实施例在模型训练过程中，会基于原有主奖励奖惩体系，加入更多的辅助拟人化奖惩来引导智能体的拟人性的行为。
在一些实施例中，根据业务侧玩法，制定各项奖惩系数的初始比例，分为主导奖惩和辅助拟人化奖惩，训练初期，鼓励智能体以最终胜利为导向进行训练，初期可将主导奖惩的系数权重调高，拟人化奖惩占比调低，随着训练不断进行，模型强度不断提高，需要根据模型的具体表现和对应拟人化指标变化趋势来动态调整辅助奖励的系数占比。如果智能体没有使用掩体拉扯战斗，分析得出是主导击杀奖励权重过大，智能体只顾着“贪心”获得更大数值的击杀奖励而放弃学习基于辅助奖励引导的拟人性的行为，所以当智能体强度达到要求后，需要动态调整辅助奖励的权重，保证智能体在拥有一定强度基础上能进一步引导出拟人性的行为。
在一些实施例中，还可以通过业务侧体验反馈来进一步优化模型表现，训练过程中可定期进行模型体验，基于体验得到的强度或者打法的问题，再对相应的奖励系数进行调整，从而引导出拟人化多风格表现，FPS品类游戏需要在强度和拟人性之间进行合理平衡，有时候为了拟人性可以牺牲部分强度，最终通过不断的体验迭代训练，得到了既能保证强度，又具有不同拟人化风格表现的符合业务目标的模型。
在一些实施例中，奖惩体系设置如表3所示：
表3奖惩体系设置表
本申请实施例采用的是自博弈迭代流程，由于FPS游戏类似真实世界的三维立体场景，地图范围足够广阔，场景地形足够复杂多变，需要训练出能够像人类在真实世界的拟人性和多样化打法的表现，则需要构建一个双人零和博弈框架，并通过自博弈技术不断迭代，最终收敛到纳什均衡策略，整体训练流程包括以下步骤一至步骤八：
步骤一，将游戏客户端内置行为树作为战斗对手，训练第0轮模型。
在一些实施例中，考虑到产品线上用户较为活跃并有大规模的落地对战数据，可以尝试使用人类数据进行早期行为模式引导，模型初期训练阶段会有很多重复无效的探索，甚至很容易会陷入某些不可预测的特殊状态中，对策略学习造成不可逆的长期影响，策略陷入局部最优点中难以跳出来。若能够利用监督学习的方式让智能体初始时刻就站在一个人类玩家的能力起点，然后再用强化学习去进一步探索策略，这样可以大大提升探索效率，而且能够有效避免初期策略学习陷入局部最优中，后续的拟人性训练也能事半功倍。
步骤二，第0轮模型加入“评估模型池”进行评估，得到评估结果。
步骤三，基于评估结果，确定是否将第0轮模型存入“对手模型池”。
步骤四，从“对手模型池”按照一定规则采样出过往模型作为对手，训练第K轮模型，直至达到最佳响应。
在本申请实施例中，可按照均匀采样规则、精英采样规则等。
步骤五，第K轮模型加入“评估模型池”进行评估，得到第K评估结果。
步骤六，基于第K评估结果，确定是否将第K轮模型存入“对手模型池”。
步骤七，返回步骤四以训练第K+1轮模型。
步骤八，直至训练收敛至纳什均衡策略，则停止训练迭代。
本申请实施例采用基于混合寻路方案的自博弈强化学习训练，发挥强化学习的决策探索优势，神经网络集中式训练，底层移动的混合式执行，根据对战局势自适应灵活选择何种移动方式，不仅能兼顾全图寻路移动和战斗博弈能力，还大大提高训练效率。此外，配合完备细致的辅助拟人化奖惩体系，真正做到在超大地图、复杂多样三维地形场景下，生产满足业务要求的高水平陪玩AI，帮助业务团队更高效进行内容生成。
基于多轮次自博弈迭代训练，本申请实施例训练的强化学习陪玩AI强度和拟人性都达到了业务目标，强化学习陪玩AI具备的优势包括：
第一，对于复杂障碍物的识别更加灵活，对比行为树版本，强化学习AI不需要预先构建掩体点就能够识别出掩体位置并前往躲避，且能够利用各种非常规掩体用于躲避战斗。
第二，行为灵活多变，相同缠斗场景下也有不同的战斗表现，可以提升重复可玩性。
第三，行为空间完全覆盖玩家的行为空间，完成所有玩家可以执行的动作，从而去到行为树去不到的地方，完成行为树无法做出的操作。
第四，开火拟人性和行为拟人性上都做到了行为树做不到的程度，游戏淘汰回放中也可以对齐人类玩家视角，真正做到了陪玩AI的目的。
图21A是基于混合移动寻路方案训练出的智能体在地图中活动区域的分布，图21B是基于原子移动寻路方案训练出的AI在地图中活动区域的分布，通过图21A和图21B可知，当采用混合移动寻路方案后，智能体(也即模型)活动区域更为分散，全图覆盖范围也更广，突显出混合移动寻路方案对于AI全图移动导航能力的提升。
参见图22，曲线1是基于混合移动寻路方案训练出的AI对战行为树胜率，曲线2是基于原子移动寻路方案训练出的AI对战行为树胜率，通过曲线1和曲线2可知，基于混合寻路方案训练智能体时，在相同时间内可以快速达到更高胜率，表明混合方案训练的AI模型上限更高。
参见图23，曲线3是基于混合移动寻路方案训练出的AI对战行为树平局率，曲线4是基于原子移动寻路方案训练出的AI对战行为树平局率，通过曲线3和曲线4可知，基于原子移动方案训练AI平局率较高，说明策略最终收敛出越来越消极的风格，表现为智能体不愿意主动出击，只肯被动地等待敌人前来战斗，这不符合项目组对智能体的期望；而基于混合移动方案训练的AI，在平局率更低的情况下，智能体的胜率更高，结合上述图21A，进一步表明基于混合移动方案训练的智能体主动性增强，活动区域范围更广，以及寻路能力、导航能力、模型强度均有所提升。
可以理解的是，在本申请实施例中，涉及到状态数据、待训练的第一预测模型、主奖励系数和辅助奖励系数等相关的数据，该相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准。
下面继续说明本申请实施例提供的模型训练装置455的实施为软件模块的示例性结构，在一些实施例中，如图2所示，存储在存储器450的模型训练装置455中的软件模块可以包括：
第一获取模块4551，配置为获取第一智能体在虚拟场景中的状态数据和所述第一智能体对应的待训练的第一预测模型；所述待训练的第一预测模型用于基于所述第一智能体的状态数据，控制所述第一智能体进行移动；指令预测模块4552，配置为利用所述待训练的第一预测模型对所述状态数据进行预测，得到动作指令集合，所述动作指令集合包括寻路方式和多个动作信息；执行模块4553，配置为配置为控制所述第一智能体基于所述寻路方式进行移动，并基于各个所述动作信息执行对应的动作，得到执行结果；调整模块4554，配置为基于多个执行结果的奖励信息调整所述待训练的第一预测模型的参数，得到训练好的第一预测模型，其中，所述奖励信息是基于所述多个执行结果、主奖励系数和辅助奖励系数确定的。
在一些实施例中，所述第一预测模型包括特征提取模块、寻路预测模块和多个动作预测模块，所述指令预测模块4552还配置为：利用所述特征提取模块对所述状态数据进行特征提取，得到状态特征；针对每一动作预测模块，利用所述动作预测模块基于所述状态特征进行预测，得到所述第一智能体的动作信息；利用所述寻路预测模块基于所述状态特征进行预测，得到所述第一智能体的移动方式；基于所述移动方式和各个动作信息，构建所述动作指令集合。
在一些实施例中，所述状态数据包括环境信息、标量信息、深度图信息和环形射线信息，所述特征提取模块包括编码单元和特征提取单元，所述指令预测模块4552还配置为：利用所述编码单元，分别对所述环境信息、所述标量信息、所述深度图信息和所述环形射线信息进行编码处理，对应得到编码后的环境信息、编码后的标量信息、编码后的深度图信息和编码后的环形射线信息；拼接所述编码后的环境信息、所述编码后的标量信息、所述编码后的深度图信息和所述编码后的环形射线信息，得到综合编码信息；利用所述特征提取单元对所述综合编码信息进行特征提取，得到所述状态特征。
在一些实施例中，所述指令预测模块4552还配置为：获取预设的动作顺序表，所述动作顺序表用于表征动作之间的排列顺序；基于所述动作顺序表，确定位于寻路动作标识前序的第一动作标识；获取所述第一动作标识对应的第一动作信息，所述第一动作信息是利用所述第一动作标识对应的动作预测模块预测得到；对所述第一动作信息进行编码，得到第一编码后动作信息；利用所述寻路预测模块基于所述状态特征和所述第一编码后动作信息进行预测，得到所述第一智能体的移动方式。
在一些实施例中，所述指令预测模块4552还配置为：按照预测周期，利用所述寻路预测模块对所述状态特征和所述第一编码后动作信息进行预测，得到目标寻路索引；基于预设的索引映射表，确定所述目标寻路索引对应的目标移动方式；将所述目标移动方式确定为所述第一智能体的移动方式。
在一些实施例中，所述目标寻路索引为第一寻路索引和第二寻路索引中的一种，当所述目标寻路索引为所述第二寻路索引时，确定所述第一智能体的移动方式为按照规则的移动方式；所述执行模块4553还配置为：获取目标位置和所述第一智能体的当前位置；基于所述目标位置和所述当前位置，确定所述第一智能体的移动路径；基于所述移动路径，控制所述第一智能体按照规则的移动方式进行移动；并基于各个动作信息，控制所述第一智能体执行各个动作。
在一些实施例中，所述执行模块4553还配置为：在控制所述第一智能体按照规则的移动方式进行移动的过程中，当确定满足移动方式切换条件时，将所述目标寻路索引更新为所述第一寻路索引；将所述第一智能体的移动方式切换为按照原子移动的移动方式；
控制所述第一智能体进行原子移动。
在一些实施例中，所述执行模块4553还配置为：当确定所述第一智能体移动至所述目标位置时，确定满足移动方式切换条件；或者，当确定所述第一智能体由遮挡场景切换为非遮挡场景时，确定满足移动方式切换条件，其中，所述遮挡场景表征所述第一智能体与所述第二智能体之间存在遮挡物，所述非遮挡场景表征所述第一智能体与所述第二智能体之间不存在遮挡物；或者，获取所述第一智能体按照规则的移动方式进行移动的移动时长，当所述移动时长达到时长阈值时，确定满足移动方式切换条件，其中，所述时长阈值小于所述预测周期。
在一些实施例中，所述指令预测模块4552还配置为：获取所述动作预测模块对应的第二动作标识；基于所述动作顺序表，确定位于所述第二动作标识前序的第三动作标识；获取所述第三动作标识对应的第二动作信息，所述第二动作信息是利用所述第三动作标识对应的动作预测模块预测得到；
对所述第二动作信息进行编码，得到第二编码后动作信息；利用所述第二动作标识对应的动作预测模块基于所述状态特征和所述第二编码后动作信息进行预测，得到所述第一智能体的动作信息。
在一些实施例中，指令预测模块4552还配置为：获取预设的动作遮罩，所述动作遮罩与所述虚拟场景中的互动规则相匹配，所述动作遮罩用于限制所述第一智能体的动作选择空间；将所述移动方式和所述各个动作信息构建为候选指令集合；利用所述动作遮罩对所述候选指令集合进行遮罩处理，得到所述动作指令集合。
在一些实施例中，所述动作信息中包括用于表征是否执行动作的第一标志值，指令预测模块4552还配置为：针对所述候选指令集合中的每一所述动作信息，从所述动作遮罩中获取所述动作信息对应的掩码值；当所述动作信息对应的掩码值为第一预设值，且所述动作信息中的第一标志值为第二预设值时，将所述动作信息中的第一标志值更新为第一预设值，其中，当所述第一标志值为第一预设值时表征不执行所述动作信息对应的动作。
在一些实施例中，所述调整模块4554还配置为：基于所述主奖励系数和所述辅助奖励系数确定奖励函数，其中，所述辅助奖励系数用于表征当所述第一智能体执行预设拟人化动作时所赋予的奖励系数；利用所述奖励函数确定每一执行结果的奖励信息；累加所述每一执行结果的奖励信息，得到累积奖励，将所述累积奖励反向传播至所述待训练的第一预测模型，并基于所述累积奖励调整所述待训练的第一预测模型的参数，直至达到训练结束条件，得到所述训练好的第一预测模型。
在一些实施例中，所述模型训练装置455还包括：第一确定模块，配置为响应于游戏对局开启指令，基于等概率原则，从所述游戏对局对应的虚拟场景所覆盖的各个子区域中确定选中区域；第二确定模块，配置为确定所述选中区域的第一中心点；第三确定模块，配置为基于所述第一中心点和预设第一最大寻路距离，确定第一出生区域，并确定出所述第一出生区域中的第一可用点集合；
第一生成模块，配置为从所述第一可用点集合中确定所述第一智能体的第一初始位置，并在所述第一初始位置生成所述第一智能体。
在一些实施例中，所述模型训练装置455还包括：第四确定模块，配置为将所述第一初始位置确定为第二中心点；第二获取模块，配置为获取所述第二智能体对应的预设第二最大寻路距离，以及所述第二智能体对应的预设第二最小寻路距离；第五确定模块，配置为基于所述第二中心点、所述预设第二最大寻路距离和所述预设第二最小寻路距离，确定第二出生区域，并确定出所述第二出生区域中的第二可用点集合；第六确定模块，配置为从所述第二可用点集合中确定候选初始位置；第二生成模块，配置为当所述候选初始位置与所述第一初始位置之间存在遮挡物时，将所述候选初始位置确定为所述第二智能体的第二初始位置，并在所述第二初始位置生成所述第二智能体。
本申请实施例提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行本申请实施例上述的模型训练方法。
本申请实施例提供一种存储有可执行指令的计算机可读存储介质，其中存储有可执行指令，当可执行指令被处理器执行时，将引起处理器执行本申请实施例提供的模型训练方法，例如，如图3和图12示出的模型训练方法。
在一些实施例中，计算机可读存储介质可以是FRAM、ROM、PROM、EPROM、EEPROM、闪存、磁表面存储器、光盘、或CD-ROM等存储器；也可以是包括上述存储器之一或任意组合的各种设备。
在一些实施例中，可执行指令可以采用程序、软件、软件模块、脚本或代码的形式，按任意形式的编程语言(包括编译或解释语言，或者声明性或过程性语言)来编写，并且其可按任意形式部署，包括被部署为独立的程序或者被部署为模块、组件、子例程或者适合在计算环境中使用的其它单元。
作为示例，可执行指令可以但不一定对应于文件系统中的文件，可以可被存储在保存其它程序或数据的文件的一部分，例如，存储在超文本标记语言(Hyper Text Markup Language，HTML)文档中的一个或多个脚本中，存储在专用于所讨论的程序的单个文件中，或者，存储在多个协同文件(例如，存储一个或多个模块、子程序或代码部分的文件)中。
作为示例，可执行指令可被部署为在一个计算设备上执行，或者在位于一个地点的多个计算设备上执行，又或者，在分布在多个地点且通过通信网络互连的多个计算设备上执行。
以上所述，仅为本申请的实施例而已，并非用于限定本申请的保护范围。凡在本申请的精神和范围之内所作的任何修改、等同替换和改进等，均包含在本申请的保护范围之内。

Claims (18)

  1. 一种模型训练方法，应用于计算机设备，所述方法包括：
    获取第一智能体在虚拟场景中的状态数据和所述第一智能体对应的待训练的第一预测模型；所述待训练的第一预测模型用于基于所述第一智能体的状态数据，控制所述第一智能体进行移动；
    利用所述待训练的第一预测模型对所述状态数据进行预测，得到动作指令集合，所述动作指令集合包括寻路方式和多个动作信息；
    控制所述第一智能体基于所述寻路方式进行移动，并基于各个所述动作信息执行对应的动作，得到执行结果；
    基于多个所述执行结果的奖励信息调整所述待训练的第一预测模型的参数，得到训练好的第一预测模型，其中，所述奖励信息是基于所述多个执行结果、主奖励系数和辅助奖励系数确定的。
  2. 根据权利要求1所述的方法，其中，所述第一预测模型包括特征提取模块、寻路预测模块和多个动作预测模块，所述利用所述待训练的第一预测模型对所述状态数据进行预测，得到包含寻路方式的动作指令集合，包括：
    利用所述特征提取模块对所述状态数据进行特征提取，得到状态特征；
    针对每一动作预测模块，利用所述动作预测模块基于所述状态特征进行预测，得到所述第一智能体的动作信息；
    利用所述寻路预测模块基于所述状态特征进行预测，得到所述第一智能体的寻路方式；
    基于所述寻路方式和各个动作信息，构建所述动作指令集合。
  3. 根据权利要求1或2所述的方法，其中，所述状态数据包括环境信息、标量信息、深度图信息和环形射线信息，所述特征提取模块包括编码单元和特征提取单元，所述利用所述特征提取模块对所述状态数据进行特征提取，得到状态特征，包括：
    利用所述编码单元，分别对所述环境信息、所述标量信息、所述深度图信息和所述环形射线信息进行编码处理，对应得到编码后的环境信息、编码后的标量信息、编码后的深度图信息和编码后的环形射线信息；
    拼接所述编码后的环境信息、所述编码后的标量信息、所述编码后的深度图信息和所述编码后的环形射线信息，得到综合编码信息；
    利用所述特征提取单元对所述综合编码信息进行特征提取，得到所述状态特征。
  4. 根据权利要求1至3任一项所述的方法，其中，所述利用所述寻路预测模块基于所述状态特征进行预测，得到所述第一智能体的寻路方式，包括：
    获取预设的动作顺序表，所述动作顺序表用于表征动作之间的排列顺序；
    基于所述动作顺序表，确定位于寻路动作标识前序的第一动作标识；
    获取所述第一动作标识对应的第一动作信息，所述第一动作信息是利用所述第一动作标识对应的动作预测模块预测得到；
    对所述第一动作信息进行编码，得到第一编码后动作信息；
    利用所述寻路预测模块基于所述状态特征和所述第一编码后动作信息进行预测，得到所述第一智能体的寻路方式。
  5. 根据权利要求1至4任一项所述的方法，其中，所述利用所述寻路预测模块基于所述状态特征和所述第一编码后动作信息进行预测，得到所述第一智能体的寻路方式，包括：
    按照预测周期，利用所述寻路预测模块对所述状态特征和所述第一编码后动作信息进行预测，得到目标寻路索引；
    基于预设的索引映射表，确定所述目标寻路索引对应的目标寻路方式；
    将所述目标寻路方式确定为所述第一智能体的寻路方式。
  6. 根据权利要求1至5任一项所述的方法，其中，所述目标寻路索引为第一寻路索引和第二寻路索引中的一种，当所述目标寻路索引为所述第二寻路索引时，确定所述第一智能体的寻路方式为按照规则的寻路方式；所述控制所述第一智能体基于所述寻路方式进行移动，并基于各个所述动作信息执行对应的动作，包括：
    获取目标位置和所述第一智能体的当前位置；
    基于所述目标位置和所述当前位置，确定所述第一智能体的移动路径；
    基于所述移动路径，控制所述第一智能体按照规则的寻路方式进行移动，并基于各个动作信息，控制所述第一智能体执行各个动作。
  7. 根据权利要求1至6任一项所述的方法，其中，所述方法还包括：
    在控制所述第一智能体按照规则的寻路方式进行移动的过程中，当确定满足寻路方式切换条件时，将所述目标寻路索引更新为所述第一寻路索引；
    将所述第一智能体的寻路方式切换为按照原子移动的寻路方式；
    控制所述第一智能体进行原子移动。
  8. 根据权利要求1至7任一项所述的方法，其中，所述方法还包括：
    当确定所述第一智能体移动至所述目标位置时，确定满足寻路方式切换条件；或者，
    当确定所述第一智能体由遮挡场景切换为非遮挡场景时，确定满足寻路方式切换条件，其中，所述遮挡场景表征所述第一智能体与所述第二智能体之间存在遮挡物，所述非遮挡场景表征所述第一智能体与所述第二智能体之间不存在遮挡物；或者，
    获取所述第一智能体按照规则的寻路方式进行移动的移动时长，当所述移动时长达到时长阈值时，确定满足寻路方式切换条件，其中，所述时长阈值小于所述预测周期。
  9. 根据权利要求1至8任一项所述的方法，其中，所述利用所述动作预测模块基于所述状态特征进行预测，得到所述第一智能体的动作信息，包括：
    获取所述动作预测模块对应的第二动作标识；
    基于所述动作顺序表，确定位于所述第二动作标识前序的第三动作标识；
    获取所述第三动作标识对应的第二动作信息，所述第二动作信息是利用所述第三动作标识对应的动作预测模块预测得到；
    对所述第二动作信息进行编码，得到第二编码后动作信息；
    利用所述第二动作标识对应的动作预测模块基于所述状态特征和所述第二编码后动作信息进行预测，得到所述第一智能体的动作信息。
  10. 根据权利要求1至9任一项所述的方法，其中，所述基于所述寻路方式和各个动作信息，构建所述动作指令集合，包括：
    获取预设的动作遮罩，所述动作遮罩与所述虚拟场景中的互动规则相匹配，所述动作遮罩用于限制所述第一智能体的动作选择空间；
    将所述寻路方式和所述各个动作信息构建为候选指令集合；
    利用所述动作遮罩对所述候选指令集合进行遮罩处理，得到所述动作指令集合。
  11. 根据权利要求1至10任一项所述的方法，其中，所述动作信息中包括用于表征是否执行动作的第一标志值，所述利用所述动作遮罩对所述候选指令集合进行遮罩处理，包括：
    针对所述候选指令集合中的每一所述动作信息，从所述动作遮罩中获取所述动作信息对应的掩码值；
    当所述动作信息对应的掩码值为第一预设值，且所述动作信息中的第一标志值为第二预设值时，将所述动作信息中的第一标志值更新为第一预设值，其中，当所述第一标志值为第一预设值时表征不执行所述动作信息对应的动作。
  12. 根据权利要求1至11任一项所述的方法，其中，所述基于多个所述执行结果的奖励信息调整所述待训练的第一预测模型的参数，得到训练好的第一预测模型，包括：
    基于所述主奖励系数和所述辅助奖励系数确定奖励函数，其中，所述辅助奖励系数用于表征当所述第一智能体执行预设拟人化动作时所赋予的奖励系数；
    利用所述奖励函数确定每一执行结果的奖励信息；
    累加所述每一执行结果的奖励信息，得到累积奖励，将所述累积奖励反向传播至所述待训练的第一预测模型，并基于所述累积奖励调整所述待训练的第一预测模型的参数，直至达到训练结束条件，得到所述训练好的第一预测模型。
  13. 根据权利要求1至12任一项所述的方法，其中，所述方法还包括：
    响应于游戏对局开启指令，基于等概率原则，从所述游戏对局对应的虚拟场景所覆盖的各个子区域中确定选中区域；
    确定所述选中区域的第一中心点；
    基于所述第一中心点和预设第一最大寻路距离，确定第一出生区域，并确定出所述第一出生区域中的第一可用点集合；
    从所述第一可用点集合中确定所述第一智能体的第一初始位置，并在所述第一初始位置生成所述第一智能体。
  14. 根据权利要求1至13任一项所述的方法，其中，所述方法还包括：
    将所述第一初始位置确定为第二中心点；
    获取所述第二智能体对应的预设第二最大寻路距离，以及所述第二智能体对应的预设第二最小寻路距离；
    基于所述第二中心点、所述预设第二最大寻路距离和所述预设第二最小寻路距离，确定第二出生区域，并确定出所述第二出生区域中的第二可用点集合；
    从所述第二可用点集合中确定候选初始位置；
    当所述候选初始位置与所述第一初始位置之间存在遮挡物时，将所述候选初始位置确定为所述第二智能体的第二初始位置，并在所述第二初始位置生成所述第二智能体。
  15. 一种模型训练装置，所述装置包括：
    第一获取模块，配置为获取第一智能体在虚拟场景中的状态数据和所述第一智能体对应的待训练的第一预测模型；所述待训练的第一预测模型用于基于所述第一智能体的状态数据，控制所述第一智能体进行移动；
    指令预测模块，配置为利用所述待训练的第一预测模型对所述状态数据进行预测，得到动作指令集合，所述动作指令集合包括寻路方式和多个动作信息；
    执行模块，配置为控制所述第一智能体基于所述寻路方式进行移动，并基于各个所述动作信息执行对应的动作，得到执行结果；
    调整模块，配置为基于多个所述执行结果的奖励信息调整所述待训练的第一预测模型的参数，得到训练好的第一预测模型，其中，所述奖励信息是基于所述多个执行结果、主奖励系数和辅助奖励系数确定的。
  16. 一种计算机设备，所述计算机设备包括：
    存储器，用于存储计算机可执行指令；
    处理器，用于执行所述存储器中存储的计算机可执行指令时，实现权利要求1至14任一项所述的模型训练方法。
  17. 一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令被处理器执行时实现权利要求1至14任一项所述的模型训练方法。
  18. 一种计算机程序产品，包括计算机程序或计算机可执行指令，所述计算机程序或计算机可执行指令被处理器执行时实现权利要求1至14任一项所述的模型训练方法。
PCT/CN2025/084695 2024-04-30 2025-03-25 模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品 Pending WO2025228010A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410545713.X 2024-04-30
CN202410545713.XA CN120874948A (zh) 2024-04-30 2024-04-30 模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品

Publications (1)

Publication Number Publication Date
WO2025228010A1 true WO2025228010A1 (zh) 2025-11-06

Family

ID=97460043

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/084695 Pending WO2025228010A1 (zh) 2024-04-30 2025-03-25 模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品

Country Status (2)

Country Link
CN (1) CN120874948A (zh)
WO (1) WO2025228010A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121117622A (zh) * 2025-11-12 2025-12-12 阿里巴巴（中国）有限公司 智能体训练方法、数据处理方法及问答方法
CN121714926A (zh) * 2025-12-09 2026-03-24 北京奇妙王国科技有限公司 棋盘游戏的处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品
CN121884646A (zh) * 2026-03-18 2026-04-17 中国科学技术大学 基于多智能体交互的启发式答疑方法、系统、设备及介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115253296A (zh) * 2022-07-20 2022-11-01 网易（杭州）网络有限公司 虚拟角色的控制方法、装置、终端设备及存储介质
CN116048062A (zh) * 2022-04-28 2023-05-02 西安交通大学 多智能体超视距网联协同感知动态决策方法及相关装置
WO2023071221A1 (zh) * 2021-10-29 2023-05-04 上海商汤智能科技有限公司 一种游戏中的交互方法、装置、计算机设备、存储介质、计算机程序及计算机程序产品
CN117196169A (zh) * 2023-06-26 2023-12-08 电子科技大学 一种基于深度强化学习的机位调度方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023071221A1 (zh) * 2021-10-29 2023-05-04 上海商汤智能科技有限公司 一种游戏中的交互方法、装置、计算机设备、存储介质、计算机程序及计算机程序产品
CN116048062A (zh) * 2022-04-28 2023-05-02 西安交通大学 多智能体超视距网联协同感知动态决策方法及相关装置
CN115253296A (zh) * 2022-07-20 2022-11-01 网易（杭州）网络有限公司 虚拟角色的控制方法、装置、终端设备及存储介质
CN117196169A (zh) * 2023-06-26 2023-12-08 电子科技大学 一种基于深度强化学习的机位调度方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121117622A (zh) * 2025-11-12 2025-12-12 阿里巴巴（中国）有限公司 智能体训练方法、数据处理方法及问答方法
CN121714926A (zh) * 2025-12-09 2026-03-24 北京奇妙王国科技有限公司 棋盘游戏的处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品
CN121884646A (zh) * 2026-03-18 2026-04-17 中国科学技术大学 基于多智能体交互的启发式答疑方法、系统、设备及介质

Also Published As

Publication number Publication date
CN120874948A (zh) 2025-10-31

Similar Documents

Publication Publication Date Title
JP7597298B2 (ja) 仮想キャラクタの制御方法、装置、電子機器、コンピュータ読み取り可能な記憶媒体及びコンピュータプログラム
KR102794785B1 (ko) 가상 소품의 상호 작용 처리 방법, 장치, 전자 기기 및 판독 가능한 저장 매체
KR102523888B1 (ko) 가상 환경에서 가상 객체를 스케줄링하기 위한 방법, 장치 및 디바이스
CN114247141A (zh) 虚拟场景中任务引导方法、装置、设备、介质及程序产品
CN112807681A (zh) 游戏控制方法、装置、电子设备及存储介质
CN114404976B (zh) 决策模型的训练方法、装置、计算机设备及存储介质
CN118036694B (zh) 一种训练智能体的方法、装置和设备及计算机存储介质
CN113018862B (zh) 虚拟对象的控制方法、装置、电子设备及存储介质
US20240123348A1 (en) Game character control method and apparatus, storage medium and electronic device
CN104102522B (zh) 交互式游戏中智能非玩家角色的人工情感驱动方法
CN112619148B (zh) 虚拟角色控制方法、装置、计算设备及存储介质
CN120874948A (zh) 模型训练方法、装置、计算机设备、计算机可读存储介质及计算机程序产品
CN120860591A (zh) 虚拟对象的控制方法和装置、存储介质及电子设备
JP2024546415A (ja) 仮想アイテムの処理方法およびその装置、電子機器、並びにコンピュータプログラム
US20260034449A1 (en) Action decision-making method and apparatus for virtual character, device, and storage medium
WO2025055551A1 (zh) 虚拟角色的动作决策方法、装置、设备及存储介质
WO2023226569A9 (zh) 虚拟场景中的消息处理方法、装置、电子设备及计算机可读存储介质及计算机程序产品
CN112933600B (zh) 虚拟对象控制方法、装置、计算机设备和存储介质
KR20240137072A (ko) 가상 시나리오의 정보 처리 방법, 장치, 기기, 저장 매체 및 프로그램 제품
CN115565031A (zh) 一种ai对象的策略预测模型训练方法、装置及电子设备
CN116966573A (zh) 交互模型处理方法、装置、计算机设备和存储介质
WO2024021750A1 (zh) 虚拟场景中的交互方法、装置、电子设备、计算机可读存储介质及计算机程序产品
US20250375707A1 (en) Interaction processing for virtual scene
HK40092640B (zh) 一种虚拟角色的控制方法、装置、电子设备和存储介质
HK40092640A (zh) 一种虚拟角色的控制方法、装置、电子设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25797186

Country of ref document: EP

Kind code of ref document: A1