KR102473196B1 - System and method for training ai character type of npc for virtual training - Google Patents

Abstract

An NPC type AI character learning method for virtual training is provided. The method includes at least one action consisting of a pair of variables for state information-action information for a scenario for virtual training of an NPC (Non Player Character) type AI character in virtual training content (hereinafter, a virtual training scenario). generating a value table model defined based on the value function; Setting the state information, the action information and reward information, and the next state information as input data, and setting a reward corresponding to the rule process defined in the virtual training scenario as output data to update and reinforce the value table model step; and performing, for the NPC-type AI character, an optimal action determined based on the updated and reinforced-learned value table model as state information in the virtual training scenario is changed.

Description

NPC type AI character learning system and method for virtual training {SYSTEM AND METHOD FOR TRAINING AI CHARACTER TYPE OF NPC FOR VIRTUAL TRAINING}

The present invention relates to an NPC type AI character learning system and method for virtual training.

In the case of virtual training using virtual reality contents, there is an advantage that training is possible by coping with real situations. However, in the case of conventional virtual training, most of the training was conducted with only a fixed story, and users who practiced repetitive training easily became bored, and there was a problem that the training effect gradually decreased.

In addition, in the case of NPCs (Non Player Characters) used in existing games or virtual training, only designated actions were performed or only corresponding actions were performed when a specific input value occurred according to the rules defined by the developer. Accordingly, in the case of existing NPCs, it was impossible to cope with unspecified situations, which had a problem that brought many limitations to directing various training situations.

Embodiments of the present invention relate to an NPC-type AI character that performs various training by helping or collaborating with a user's behavior and training situation in virtual training, helping users participating in training or autonomously acting according to the situation Provides an NPC-type AI character learning system and method for virtual training that creates various training situations and cooperates with users to solve training situations.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the NPC type AI character learning system and method for virtual training according to the first aspect of the present invention is a virtual non player character (NPC) AI character in virtual training content. generating a value table model defined based on at least one action value function composed of each variable pair of state information and motion information for a scenario for training (hereinafter, a virtual training scenario); Setting the state information, the action information and reward information, and the next state information as input data, and setting a reward corresponding to the rule process defined in the virtual training scenario as output data to update and reinforce the value table model step; and performing, for the NPC-type AI character, an optimal action determined based on the updated and reinforced-learned value table model as state information in the virtual training scenario is changed.

In some embodiments of the present invention, the generating of the value table model includes generating a reward table model including the reward information set to correspond to at least one state information-action information pair for the virtual training scenario. step; calculating a maximum value of the action value function having next state information after the current state in the virtual training scenario and all motion information in the next state information as variables; and generating the value table model by assigning a weight to a maximum value of the action value function and adding the maximum value to the reward table model.

In some embodiments of the present invention, the reward information is provided when moving from a first rule process defined in the virtual training scenario to a second rule process, and the second rule process corresponds to a target point in the virtual training scenario. In this case, a second reward greater than the first reward may be granted.

In some embodiments of the present invention, in the updating and reinforcement learning of the value table model, whenever a rule process defined in the virtual training scenario is reached, a deep neural network (DNN) and a long short term memory (LSTM) are combined. Based on the algorithm, the value table model may be updated and reinforced.

In some embodiments of the present invention, performing the determined optimal action based on the updated and reinforcement-learned value table model may include reading a motion data set corresponding to the determined optimal action; and driving an animation for the NPC-type AI character in the virtual training content based on the motion data set.

In some embodiments of the present invention, the NPC-type AI character includes joint information for a manipulable body composition, and a joint according to the joint information is manipulated to correspond to the motion information, and the motion information includes at least one of the above It includes motion data, is defined according to the virtual training scenario, and can be defined for motions that can occur in the virtual training scenario.

In some embodiments of the present invention, the state information defines the location, direction, and location variables of the movable path of the NPC-type AI character in the virtual training map data for the virtual training scenario, and the virtual training It may be data defining the current environment situation variable according to the scenario, the state value of the object, the behavior value of the NPC-type AI character, and the user's behavior.

In addition, the NPC type AI character learning system for virtual training according to the second aspect of the present invention is a scenario for virtual training of NPC (Non Player Character) type AI characters in virtual training contents (hereinafter referred to as virtual training scenarios). Based on the above, a memory for storing a program for virtual training of the NPC-type AI character and a processor for executing the program stored in the memory. At this time, as the processor executes the program, the processor generates a value table model defined based on an action value function composed of at least one state information-action information pair for the virtual training scenario, and the generated value Updating and reinforcement learning a table model, and performing an optimal action determined based on the updated and reinforcement-learned value table model as state information in the virtual training scenario changes for the NPC-type AI character, and the processor sets the state information, the action information, the reward information, and the next state information as input data, and sets the reward corresponding to the rule process defined in the virtual training scenario as output data to update the value table model and to perform reinforcement learning. let it

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

Existing AI characters require direct interaction with users, and are limited to interactive services through voice recognition. It can perform a role to participate in training jointly with the user by performing an action.

This does not stop at simply assisting the training situation, but has the same effect as actual training based on the improvement of the complexity of training scenarios through the production of various training situations that can occur in reality, such as generating new training situations according to the behavioral results of NPC-type AI characters. can maximize

In addition, existing virtual training is a method in which one or multiple users simultaneously access and repeatedly perform a given learning goal based on a set scenario, which is helpful for various situations in coping with real situations due to the same training process and derivation of results. It doesn't work.

On the other hand, in one embodiment of the present invention, various virtual trainings such as multi-person training, cooperative training, and trainee response training are possible through NPC-type AI characters, and various intermediate processes occur according to the user's behavior or NPC's behavior, resulting in a real situation. It has the advantage that coping training such as

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a flowchart of a NPC-type AI character learning method for virtual training according to an embodiment of the present invention.
2 is a flowchart for explaining a process of generating a value table model.
3 is a diagram for explaining a value table model generated based on an action value function.
4 is a diagram for explaining a compensation table model.
5 is a diagram showing an example of updating and reinforcement learning of a value table model based on an algorithm combining DNN and LSTM.
6 is a block diagram of an NPC-type AI character learning system for virtual training according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

The present invention relates to a NPC type AI character learning system 100 and method for virtual training.

Most of the character-related technologies using existing AI technologies were for interactive and information-providing services. The main purpose of this was to learn about technology that provides information by providing appropriate answers to user's questions or executing related commands.

In contrast, the present invention enables autonomous behavior suitable for virtual training by learning optimal behavioral tips according to a given situation and user's behavior even without a user's command, and it is possible to implement an NPC-type AI character capable of actively coping according to the situation do.

In one embodiment of the present invention, reinforcement learning for recognizing virtual training scenario situations, user state values, etc. and determining action values according to the standard procedure of the corresponding training, rather than optimal dialogue creation learning through voice recognition in the prior art. It is characterized by being the focus.

Hereinafter, a method performed by the NPC-type AI character learning system 100 for virtual training according to an embodiment of the present invention will be described with reference to FIGS. 1 to 5.

1 is a flowchart of a NPC-type AI character learning method for virtual training according to an embodiment of the present invention.

First, at least one action value composed of a pair of variables for state information and motion information for a scenario for virtual training of an NPC (Non Player Character) type AI character in virtual training content (hereinafter, a virtual training scenario) A value table model defined based on the function is created (S110).

As an embodiment, the present invention may perform 3D modeling of a character required for virtual training for an NPC-type AI character.

For reference, an NPC (Non Player Character) is a character that cannot be directly controlled by a user or trainer in a game, and is a kind of helper character that provides various contents such as quests to the user.

NPC is a word derived from TRPG (Table Role Playing Game), and has the opposite meaning to PC (Player Character). Most NPCs stay in one place or area and serve as helpers for the smooth progress of the game. It serves as a background like an extra supporting character in a movie or drama, and also provides content such as a quest to be performed by the user or an item after completing the quest.

These NPCs are mainly used as characters directly controlled by service providers in online games, and often appear as objects to obtain points in PC games. In one embodiment of the present invention, it is characterized in that it is applied to virtual training that helps or cooperates with the user's training.

As an embodiment, the NPC-type AI character in the present invention includes born information for a body composition that can be manipulated, and the joints are manipulated according to the joint information to correspond to the motion information.

Here, motion information includes at least one piece of motion data. Motion information is defined according to the virtual training scenario, and is defined for motions that can occur in the virtual training scenario. As an example, the motion information refers to the behavior of an NPC-type AI character playing virtual training content, and the motion of the character corresponds to this.

Motion data included in the motion information is composed of a motion data set (Rigging data sat) so that it can be applied to NPC-type AI character modeling in real time, and the motion data can be synchronized in response to decision actions for each situation.

Meanwhile, the value table model (Quality Table Model) generated in step S110 is defined based on at least one action value function composed of each variable pair of state information and motion information for the virtual training scenario.

2 is a flowchart for explaining a process of generating a value table model. 3 is a diagram for explaining a value table model generated based on an action value function. 4 is a diagram for explaining a compensation table model.

As an embodiment, the present invention generates a reward table model including reward information set to correspond to at least one state information-action information pair for a virtual training scenario (S111).

That is, an embodiment of the present invention may store and manage compensation information in a designated virtual training scenario as a reward table.

Here, the reward information is a kind of score obtained while playing the virtual training content, and may be provided as the user moves from the first rule process defined in the virtual training scenario to the second rule process.

In this case, when the second rule process corresponds to the target point in the virtual training scenario, a second reward greater than the first reward may be granted. At this time, the rule process means a strategy for selecting motion information for the NPC-type AI character to obtain the maximum reward from the current state information in the virtual training content.

Referring to the example of FIG. 4 , a reward is given when moving from one rule process to the next rule process according to the reward table model, and 100 points are given as a second reward when moving to reach a target point. As these rewards accumulate, the NPC-type AI character can determine and perform the rule process flow with the largest reward score as the optimal action.

Then, the maximum value of the action value function (Max [Q (next state, all actions)]) is calculated (S113).

Then, a value table model may be generated according to Equation 1 below by assigning a weight to the maximum value of the action value function and summing it to the reward table model (S115).

[Equation 1]

Q(state, action)=R(state, action) +Gamma(weight)*Max[Q(next state, all actions)]

The value table model created in this way includes at least one action value function, and the action value function may be defined using state information and action information as variables.

Here, the state information defines the position and direction of the NPC-type AI character in the virtual training map data for the virtual training scenario, and the location variable of the movable path, and the current environment situation variable according to the virtual training scenario, the object It refers to data that defines state values, behavior values of NPC-type AI characters, and user behavior. For example, state information means the position, speed, and distance of each object in the virtual training content, and the wall.

Each action value function of the value table model calculates a value value (Q) using state information and action information as variables. Such a value table model may separately acquire information on an expected value of how much reward the corresponding operation or state information will bring in the future.

An embodiment of the present invention as described above is characterized in that reinforcement learning is applied to perform real-time interaction in response to a user's behavior and environmental change while performing a specified behavior according to motion information.

Referring back to FIG. 1 , next, the virtual table model generated in step S110 is updated and reinforced (S120).

As an embodiment, whenever a rule process defined in a virtual training scenario is reached, a value table model may be updated and reinforced based on an algorithm combining a deep neural network (DNN) and a long short term memory (LSTM).

5 is a diagram showing an example of updating and reinforcement learning of a value table model based on an algorithm combining DNN and LSTM.

Specifically, for reinforcement learning, state information, action information, reward information, and next state information are set as input data, and rewards corresponding to the rule process defined in the virtual training scenario are set as output data to update the value table model and Reinforcement learning can be done.

The process of updating the value table model may be performed in such a way that value values are updated as learning progresses, and weight values are applied to find an optimal method to reach a goal as value values that start with 0 are accumulated. .

Referring back to FIG. 1, next, as the state information in the virtual training scenario is changed, the optimal action determined based on the updated and reinforced-learned value table model is performed on the NPC-type AI character (S130).

In one embodiment, the present invention can determine the optimal action for the NPC-type AI character when each situation change occurs in the virtual training scenario through the learned value table model, and perform the corresponding action according to the determined result. A motion data set corresponding to the determined optimal behavior is read. In addition, based on the read motion data set, animation for the NPC type AI character in the virtual training content can be driven.

At this time, feedback from a corresponding virtual training expert may be provided for the driving result for the NPC-type AI character in the virtual training content, and the value table model may be updated by reflecting the feedback.

In addition, in an embodiment of the present invention, in order to learn optimal behavior for each situation in a virtual training scenario, standard training procedures for current state information and behaviors of characters and corresponding maximum expected values may be set.

In addition, in relation to optimal behavior, for virtual training autonomous behavior learning of NPC-type AI characters, behavior-specific activity modules for autonomous behavior and interactive behavior based on DQN (Deep Q Network) neural network learning for each training situation can be applied. .

In the foregoing description, steps S110 to S130 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Also, some steps may be omitted if necessary, and the order of steps may be changed. In addition, even if other omitted contents, the contents described in FIGS. 1 to 5 are also applied to the NPC type AI character learning system 100 for virtual training of FIG. 6.

Hereinafter, an NPC-type AI character learning system 100 for virtual training according to an embodiment of the present invention will be described with reference to FIG. 6 .

6 is a block diagram of an NPC-type AI character learning system 100 for virtual training according to an embodiment of the present invention.

NPC type AI character learning system 100 for virtual training according to an embodiment of the present invention includes a memory 110 and a processor 120.

The memory 110 stores programs for virtual training of NPC-type AI characters based on scenarios for virtual training of NPC-type AI characters within virtual training contents, and the processor 120 stores programs stored in the memory 110. run

Meanwhile, the memory 110 collectively refers to a non-volatile storage device and a volatile storage device that continuously maintain stored information even when power is not supplied. The memory 120 may be NAND memory such as a compact flash (CF) card, a secure digital (SD) card, a memory stick, a solid-state drive (SSD), and a micro SD card. It may include a magnetic computer storage device such as a NAND flash memory, a hard disk drive (HDD), or the like, and an optical disc drive such as a CD-ROM or DVD-ROM.

As the processor 120 executes the program stored in the memory 110, it generates a value table model defined based on an action value function composed of at least one state information-action information pair with respect to the virtual training scenario, The value table model is updated and reinforced, and the optimal action determined based on the updated and reinforced-learned value table model is performed for the NPC-type AI character as the state information in the virtual training scenario is changed.

At this time, the processor 120 sets state information, operation information, reward information, and next state information as input data, and sets a reward corresponding to a rule process defined in the virtual training scenario as output data to update the value table model and learn reinforcement.

The NPC-type AI character learning method for virtual training according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a server, which is hardware, and stored in a medium.

The aforementioned program is C, C++, JAVA, machine language, etc. It may include a code coded in a computer language of. These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. have. In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected through a network, and computer readable codes may be stored in a distributed manner.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: NPC type AI character learning system for virtual training
110: memory
120: processor

Claims (8)

As a method performed by a computer,
For scenarios for virtual training of NPC (Non Player Character) type AI characters in virtual training contents (hereinafter referred to as virtual training scenarios), at least one action value function composed of each variable pair for state information-action information generating a value table model defined based on;
Setting the state information, the action information and reward information, and the next state information as input data, and setting a reward corresponding to the rule process defined in the virtual training scenario as output data to update and reinforce the value table model step; and
For the NPC-type AI character, as the state information in the virtual training scenario changes, performing the optimal action determined based on the updated and reinforced-learned value table model,
The step of generating the value table model,
generating a reward table model including the reward information set to correspond to at least one state information-action information pair for the virtual training scenario;
calculating a maximum value of the action value function having next state information after the current state in the virtual training scenario and all motion information in the next state information as variables; and
Generating the value table model by assigning a weight to the maximum value of the action value function and adding it to the reward table model.
NPC type AI character learning method for virtual training.
delete According to claim 1,
The reward information is provided as one moves from a first rule process defined in the virtual training scenario to a second rule process, and when the second rule process corresponds to a target point in the virtual training scenario, a first reward greater than the first reward information is provided. 2 that the reward is granted,
NPC type AI character learning method for virtual training.
According to claim 3,
Updating and reinforcement learning the value table model,
Whenever a rule process defined in the virtual training scenario is reached, the value table model is updated and reinforced based on an algorithm combining Deep Neural Network (DNN) and Long Short Term Memory (LSTM),
NPC type AI character learning method for virtual training.
According to claim 1,
The step of performing the optimal action determined based on the updated and reinforcement-learned value table model,
reading a motion data set corresponding to the determined optimal behavior; and
Based on the motion data set, driving animation for the NPC-type AI character in virtual training content,
NPC type AI character learning method for virtual training.
According to claim 5,
The NPC-type AI character includes joint information for a manipulable body composition, and joints are manipulated according to the joint information to correspond to the motion information,
The motion information includes at least one motion data, is defined according to the virtual training scenario, and is defined for motions that can occur in the virtual training scenario.
NPC type AI character learning method for virtual training.
According to claim 1,
The state information defines the position, direction, and location variables of the NPC-type AI character in the virtual training map data for the virtual training scenario, and the position variable of the movable path, and the current environmental situation variable according to the virtual training scenario, Data defining the state value of the object, the behavior value of the NPC-type AI character, and the user's behavior,
NPC type AI character learning method for virtual training.
In the NPC type AI character learning system for virtual training,
A memory in which a program for virtual training of an NPC-type AI character is stored based on a scenario (hereinafter referred to as a virtual training scenario) for virtual training of a non-player character (NPC)-type AI character in virtual training contents; and
Including a processor that executes the program stored in the memory,
As the program is executed, the processor generates a value table model defined based on an action value function composed of at least one state information-action information pair with respect to the virtual training scenario, and the generated value table model Renewing and reinforcement learning, and for the NPC-type AI character, as the state information in the virtual training scenario changes, performing the optimal action determined based on the updated and reinforcement-learned value table model,
The processor sets the state information, the operation information, the reward information, and the next state information as input data, and sets a reward corresponding to a rule process defined in the virtual training scenario as output data to update the value table model, and learn reinforcement,
The processor generates a reward table model including the reward information set to correspond to at least one state information-action information pair for the virtual training scenario, next state information after the current state in the virtual training scenario, and the The value table model is generated by calculating the maximum value of the action value function having all action information in the next state information as variables, assigning a weight to the maximum value of the action value function, and adding them to the reward table model. to do,
NPC type AI character learning system for virtual training.

Images