KR20240016817A - System and method for growing non-player character avatar within metaverse based on metaverse behavior log of user - Google Patents

Abstract

Embodiments include receiving a user's personal information; Creating a virtual NPC avatar having as a primary personality a personality corresponding to the user's personal information among pre-stored personalities - the virtual NPC avatar has a different shape from the user's avatar; Receiving activity data and manipulation data of the user captured on an electronic device while the user is active in the metaverse, the activity data including one or more of the user's image, text, and voice data; determining the user's emotional state from the user's activity data using a pre-stored emotion recognition model; Creating a secondary personality of the virtual NPC avatar based on the activity data and emotional state, and updating the personality of the virtual NPC avatar with the secondary personality, based on the user's metaverse activity log. This relates to a system and method for growing a virtual NPC avatar in a bus.

Description

System and method for growing a virtual NPC avatar in the metaverse based on the user's metaverse activity log {System and method for growing non-player character avatar within metaverse based on metaverse behavior log of user}

Embodiments of the present application relate to a system and method for growing an avatar in the metaverse based on a user's metaverse activity log, including data received while the user is active in the metaverse.

The word avatar was mainly used as a "player in a game" from the early days of the Internet to the early and mid-2000s, but has since been replaced by the term "character."

Recently, as metaverse technology began to gain attention, the term avatar came to the forefront again. However, it is currently used with a different meaning from its previous meaning. Specifically, the avatar in the metaverse not only projects my various personalities (multi-persona) into the virtual world of the metaverse, but is also an agent who acts within the metaverse by being delegated responsibilities, duties, and rights from the real me. It was defined and began to be used. The reason is that the metaverse is not just a virtual entertainment space, but a world where daily life and economic activities are possible, so the actions of the avatar in the metaverse are treated as equivalent to my actual actions, so the avatar is treated as a virtual world equivalent to the real world. This is because it may involve global social obligations and responsibilities.

Based on this definition, avatar technology has developed, and in the current metaverse world, the existence of an avatar is treated as the best essential element along with the objects, environments, places, and systems that exist within the metaverse world.

Avatars in the metaverse are classified into user avatars and non-user avatars, and non-user avatars are generally referred to as NPC avatars.

Virtual NPC avatars in the metaverse can provide various information to users or guide users directly.

However, currently, virtual NPC avatars are generally implemented based on a chatbot system that provides a number of candidate chats within rules predefined by the designer and additionally provides the next chat related to the selected candidate chat, so limited communication with the user is possible. There are limits.

Republic of Korea Patent Registration No. 10-2199843 (202012.31.)

In order to solve the above-described problem, embodiments of the present application are based on the metaverse activity log of the user in the metaverse, the behavior pattern of the user active in the metaverse, the activity range, and the characteristics of the virtual NPC avatar, the user and the metaverse. We aim to provide a system and method for growing virtual NPC avatars in the metaverse by reflecting interactions between them.

According to one aspect of the present application, a method of growing a virtual NPC avatar in the metaverse based on a user's metaverse activity log performed by one or more computing devices includes:

Receiving the user's personal information;

Creating a virtual NPC avatar having as a primary personality a personality corresponding to the user's personal information among pre-stored personalities - the virtual NPC avatar has a different shape from the user's avatar;

Receiving activity data and manipulation data of the user captured on an electronic device while the user is active in the metaverse, the activity data including one or more of the user's image, text, and voice data;

determining the user's emotional state from the user's activity data using a pre-stored emotion recognition model;

Generating a secondary personality of the virtual NPC avatar based on the activity data and emotional state, and updating the personality of the virtual NPC avatar with the secondary personality,

The step of creating the secondary personality is,

Forming a first data set including a plurality of pairs of user activity data for each activity in the user's metaverse, obtained according to a time series order, and the user's emotional state determined from the activity data; and

It may also include modeling a personality model indicating the secondary personality of the virtual NPC avatar using the first data set.

In one embodiment, the method:

calculating user behavior pattern data based on the received activity data and manipulation data;

forming a second data set including the user's activity data, an emotional state corresponding to the activity data, and the user's behavior pattern data corresponding to the activity data;

using the second data set to create a tertiary personality of the virtual NPC avatar that performs actions that mimic the user's actions in the metaverse; and

The step of updating the personality of the virtual NPC avatar to the generated tertiary personality may be further included.

In one embodiment, determining the user's emotional state from the user's activity data includes:

A step of determining the user's first emotional state from the user's image data using a pre-trained image-based emotion recognition model,

The image-based emotion recognition model is

a face detection unit that detects a face area in the input image, a feature extraction unit that extracts image features of the user from the detected face area, and a face in the input image that is pointed out based on the features extracted from a plurality of emotional states pre-specified in the model. It is a machine learning model that includes an emotion judgment unit that classifies the user's emotional state.

The emotion judgment unit includes a first emotion judgment unit for determining the user's emotion type, and a second emotion judgment unit for determining the user's emotion type,

The image-based emotion recognition model may extract features from the face area to determine the user's emotion type as a first depth, and extract features from the face area to determine the user's emotion type as a second depth.

In one embodiment, determining the user's emotional state from the user's activity data includes:

determining the user's second emotional state from the user's text data using a pre-trained text-based emotion recognition model; It may further include one or more steps of determining the user's third emotional state from the user's voice data using the text-based emotion recognition model or the pre-trained voice-based emotion recognition model.

In one embodiment, determining the user's emotional state from the user's activity data includes:

When a plurality of emotional states are determined among the first to third emotional states of the user, determining the final emotional state of the user based on a probability value of being classified into each of the plurality of emotional states,

The step of determining the final emotional state is,

A higher first weight is assigned to the first emotional state determined from the image data, a lower second weight is assigned to the second emotional state determined from the text data, and the first weight is set to the value of the first emotional state. The final emotional state may be determined based on the calculation value applied to and the second weight is applied to the value of the second emotional state.

In one embodiment, the personality of the virtual NPC avatar includes internal factors and external factors,

The internal factor is related to the emotional state and the external factor is related to the physical response expressed under the emotional state,

The physical response of the virtual NPC avatar with the secondary personality includes implementing facial expressions corresponding to emotional states or outputting information through the user's text output method or voice output method,

Forming the first data set includes:

The first data set may be formed so that the type of emotional state includes only at least some of the remaining emotional states that do not have a negative value.

In one embodiment, the user's behavior pattern data is,

In the user's metaverse, one of the following is the movement line of the user's avatar, movement location, movement frequency, location frequency, selected content related to the activity, preparation time to decide on the activity, activity start time, activity frequency, activity review, activity content, and activity scope. Including the above,

If the activity has related content, the preparation time taken to start the activity represents the preparation time taken to select the content, the activity start time represents the content start time, the activity frequency represents the frequency of content use, and the activity review represents the content use. It may represent a review, the activity content may represent content content, and the activity scope may represent a content selection range.

In one embodiment, the step of updating the personality of the virtual NPC avatar 201 to the generated tertiary personality includes,

modeling a tertiary personality using the second data set and a behavior imitation algorithm to have behavioral pattern data in the second data set as output and the remaining data as input; and

It may also include the step of inverse reinforcement learning (IRL) of the tertiary personality modeled based on the second data set.

In one embodiment, the method:

Generating an auxiliary personality that recommends content suitable for a user whose virtual NPC avatar matches at least one of personality and behavior patterns based on activity log data within the user's metaverse; and

further comprising updating the virtual NPC avatar to additionally have the secondary personality,

The network structure of the auxiliary nature includes a first sub-structure and a second sub-structure,

The first sub-structure may be modeled to infer a linear correlation between input data and output data, and the second sub-structure may be modeled to infer a non-linear correlation between input data and output data.

In one embodiment, creating the secondary personality includes:

forming a third data set including content information related to the user's activities in the metaverse, activity data corresponding to the content, emotional state, and behavior pattern data;

Classifying data related to content and data not related to content among activity data, emotional state, and behavior pattern data in the third data set;

forming a first subset of data related to the content and the content information;

forming a second subset consisting of data unrelated to the content and the content information; and

It may include learning a first sub-structure within the auxiliary personality using the first subset, and learning a second sub-structure within the auxiliary personality using the second subset.

In one embodiment, the method may be performed when the virtual NPC avatar updated to have the auxiliary personality approaches the user's avatar at a certain distance within the metaverse, at least some of its activity data, emotional state, and behavior pattern under the approaching situation. It may further include predicting content suitable for the user based on and outputting the predicted content within the metaverse to provide it to the user.

In one embodiment, the method:

When a plurality of virtual NPC avatars customized and created and grown according to the activities of each of the plurality of users in the metaverse exist in the metaverse, filtering one or more virtual NPC avatars customized to a specific user; and

A step of providing metaverse information through a filtered virtual NPC avatar may be further included.

In one embodiment, the method may further include, after filtering, re-configuring the metaverse so that only the filtered virtual NPC avatars exist.

A computer-readable recording medium according to another aspect of the present application may record a program for performing a method of growing a virtual NPC avatar in the metaverse based on the user's metaverse activity log according to the above-described embodiments. .

According to another aspect of the present application, a system for growing a virtual NPC avatar in the metaverse based on a user's metaverse activity log includes a service server. The service server is,

a data collection module that receives the user's personal information, the user's activity data and manipulation data captured on an electronic device while the user is active in the metaverse;

An avatar creation module that generates a virtual NPC avatar with a personality corresponding to the user's personal information among pre-stored personalities as a primary personality;

an emotion analysis module that determines the user's emotional state from the user's activity data using a pre-stored emotion recognition model;

and an avatar growth module that generates a secondary personality of the virtual NPC avatar based on the activity data and emotional state, and updates the personality of the virtual NPC avatar to the secondary personality.

In one embodiment, the service server,

It further includes a behavior analysis module that calculates user behavior pattern data based on the received activity data and manipulation data,

The avatar growth module is,

Form a second data set including the user's activity data, emotional state corresponding to the activity data, and user's behavior pattern data corresponding to the activity data, and use the second data set to enter the user's metaverse. and generate a tertiary personality of the virtual NPC avatar that performs actions that mimic the behavior, and update the personality of the virtual NPC avatar with the generated tertiary personality.

According to one aspect of the present invention, the system for growing a virtual NPC avatar in the metaverse includes the user's metaverse, including the behavior pattern, activity range, interaction response between the user and the metaverse, etc. of the user active in the domain in any metaverse. Based on the bus activity log, a customized personality can be created for the user and the already defined initial personality of the virtual NPC avatar can be updated with the customized personality.

As a result, the system for growing virtual NPC avatars in the metaverse can grow virtual NPC avatars to be more differentiated and deviate slightly from the rules defined by metaverse developers. In particular, the virtual NPC avatar can be developed into a personality that allows for smoother communication with the user as the amount of the user's metaverse activity log increases.

Additionally, the system for growing a virtual NPC avatar in the metaverse can effectively inform recommended content suitable for the user's preferences or other metaverse information through a custom-grown virtual NPC avatar.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

In order to more clearly explain the technical solutions of the embodiments of the present invention or the prior art, drawings necessary in the description of the embodiments are briefly introduced below. It should be understood that the drawings below are for illustrative purposes only and not for limiting purposes of the present specification. Additionally, for clarity of explanation, some elements may be shown in the drawings below with various modifications, such as exaggeration or omission.
1 is a schematic diagram of a system for growing a virtual NPC avatar in the metaverse, according to one aspect of the present application.
Figure 2 is a schematic diagram of a service server, according to an embodiment of the present application.
FIG. 3A is a flowchart of a method for growing a virtual NPC avatar in the metaverse based on a user's metaverse activity log, according to another aspect of the present application.
FIG. 3B is a diagram schematically explaining steps in the flowchart of FIG. 3A.
Figure 4 is a flowchart of a process for determining a user's emotional state using an image-based emotion recognition model, according to an embodiment of the present application.
Figure 5 is a schematic diagram of the process in Figure 4.
Figure 6 is a schematic diagram of a process for determining a user's multiple emotional states, according to an embodiment of the present application.
Figure 7 is a detailed schematic diagram explaining the process of Figure 6 by depth.
Figure 8 is a flowchart of a process for determining a user's emotional state using a text-based emotion recognition model, according to an embodiment of the present application.
Figure 9 is a schematic diagram of a process for determining a user's emotional state using a voice-based emotion recognition model, according to an embodiment of the present application.
Figure 10 is a schematic diagram of a process for determining a user's emotional state from various activity data using an integrated emotion recognition model, according to an embodiment of the present application.
Figure 11 is a schematic diagram of the process of growing a virtual NPC avatar by customizing it to the user with more emphasis on the external aspect, according to an embodiment of the present application.
12 is a schematic diagram of an action tree, according to an embodiment of the present application.
Figure 13 is a network structure diagram of an auxiliary nature according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be examined in detail with reference to the drawings.

However, this disclosure is not intended to limit the disclosure to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives to the embodiments of the disclosure. . In connection with the description of the drawings, similar reference numbers may be used for similar components.

In this specification, expressions such as “have,” “may have,” “includes,” or “may include” refer to the corresponding features (e.g., numerical values, functions, operations, steps, parts, elements, and/or components). It refers to the presence of components such as etc.) and does not exclude the presence or addition of additional features.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Expressions such as “first,” “second,” “first,” or “second” used in various embodiments may modify various elements regardless of order and/or importance, and limit the elements. I never do that. The above expressions can be used to distinguish one component from another. For example, the first component and the second component may represent different components, regardless of order or importance.

Embodiments of singular expressions used in this specification also include embodiments of plural expressions, unless phrases related to the singular expression clearly indicate the contrary.

As used herein, the expression “configured to” may mean, for example, “suitable for,” “having the capacity to,” or “having the capacity to.” ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or set) to” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or executing one or more software programs stored on a memory device. By doing so, it may mean a general-purpose processor (eg, CPU or application processor) capable of performing the corresponding operations.

1 is a schematic diagram of a system for growing a virtual NPC avatar in the metaverse, according to one aspect of the present application.

The system 1 for growing a virtual NPC avatar in the metaverse according to embodiments may be entirely hardware, entirely software, or may have aspects that are partly hardware and partly software. For example, a system may collectively refer to hardware equipped with data processing capabilities and operating software for running it. In this specification, terms such as “unit,” “system,” and “device” are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing device that includes a Central Processing Unit (CPU), Graphics Processing Unit (GPU), or other processor. Additionally, software may refer to a running process, object, executable, thread of execution, program, etc.

Referring to FIG. 1, the system 1 for growing a virtual NPC avatar in the metaverse includes a user's electronic device 100 and a service server 200 that provides a virtual NPC avatar 201.

A user refers to a person who experiences various activities by interacting with the metaverse provided by the service server 200 through the electronic device 100.

Metaverse is a compound word of meta, meaning processing and abstraction, and universe, meaning the real world. It refers to a virtual space that delivers stimuli and implements responses that are the same or similar to the real world. Depending on the topic, the metaverse can be a game or a business system suited to a specific domain. Alternatively, the metaverse may be implemented in the form of a library for reading books, a shopping mall, or a museum with various tangible experiences, depending on its form.

The system 1 for growing a virtual NPC avatar in the metaverse generates a virtual NPC avatar 201 through the service server 200. The virtual NPC avatar 201 is an avatar created by a metaverse designer that is different from the user's avatar.

The virtual NPC avatar 201 is configured to guide the user indirectly by providing information or more directly through actions. The service server 200 provides a metaverse including a virtual NPC avatar 201 and a user avatar to the user through the electronic device 100.

However, the virtual NPC avatar 201 is different from the regular NPC avatar basically placed in the metaverse. Regular NPC avatars are also virtual avatars, so they may be similar in appearance. However, in terms of its essence, a general NPC avatar is an object created based on content preset as NPC data, while a virtual NPC avatar 201 behaves in a form that imitates a user, communicates more autonomously with users, and It is an empathetic avatar that shares information about other users it is imitating.

The electronic device 100 and the service server 200 are connected through a telecommunication network.

The telecommunication network provides a wired/wireless telecommunication path through which the electronic device 100 and the service server 200 can transmit and receive data with each other. A telecommunication network is not limited to a communication method according to a specific communication protocol, and an appropriate communication method may be used depending on the implementation example. For example, when configured as an Internet Protocol (IP)-based system, the telecommunication network may be implemented as a wired and/or wireless Internet network. Alternatively, when the electronic device 100 and the service server 200 are implemented as mobile communication terminals, the telecommunication network may be implemented as a wireless network such as a cellular network or a wireless local area network (WLAN) network.

The electronic device 100 is a client terminal device that communicates with the service server 200 and includes at least one processor capable of processing data, a memory for storing data, and a communication unit for transmitting/receiving data. This electronic device 100 may be implemented as, for example, a laptop computer, other computing device, tablet, cellular phone, smart phone, smart watch, smart glasses, head-mounted display (HMD), other mobile device, or other wearable device. there is.

In the system (1) for growing a virtual NPC avatar in the metaverse, users access the metaverse, interact with the metaverse, and experience various activities within the metaverse. Users can also interact with the metaverse by manipulating their avatar within the metaverse. Interaction between the user and the metaverse includes interacting with another user's avatar and the virtual NPC avatar 201 through the user's avatar. Input data, output data, and computational data during interaction can also be managed as activity details. The activity details may be recorded and stored as a metaverse activity log.

In certain embodiments, the electronic device 100 is configured to obtain personal information of a user accessing the metaverse through an input device and transmit it to the service server 200.

In addition, the electronic device 100 is configured to acquire the user's manipulation data, including user interface manipulation commands and content interaction manipulation commands, while the user is active in the metaverse through an input device and transmit it to the service server 200. .

In addition, the electronic device 100 is configured to acquire activity data including the user's image, text, and voice while the user is active in the metaverse through a camera, microphone, and input device and transmit it to the service server 200. .

The electronic device 100 may be configured to receive and display a virtual NPC avatar 201 with a user-customized personality that reflects the user's metaverse activity log data based on manipulation data and activity data.

Additionally, in some embodiments, the electronic device 100 may be configured to provide recommended content suitable for the user's preferences or other metaverse information through the virtual NPC avatar 201.

The operation of this electronic device 100 will be described in more detail with reference to FIG. 3 below.

The service server 200 is a plurality of computer systems or computer software implemented as network servers. Here, a network server is a computer system and computer that is connected to a sub-device that can communicate with other network servers through a computer network such as a private intranet or the Internet, receives a request to perform a task, performs the task, and provides a performance result. Refers to software (network server program). However, in addition to these network server programs, it should be understood as a broad concept that includes a series of application programs operating on a network server and, in some cases, various databases built within it. The service server 200 may be implemented as any type or combination of types of computing devices, such as a network server, web server, file server, supercomputer, desktop computer, etc. For this purpose, the service server 200 includes at least one processor capable of processing data, a memory for storing data, and a communication unit for transmitting/receiving data.

Figure 2 is a schematic diagram of a service server, according to an embodiment of the present application.

Referring to FIG. 2, the service server 200 includes a data collection module 210, an avatar creation module 220, an emotion analysis module 240, an avatar growth module 250, and a behavior analysis module 260. do. In some embodiments, the service server 200 may further include an avatar DB 221 and/or an activity history DB 230. The activity history DB 230 includes an emotion analysis DB 234 and a behavior analysis DB 236. Additionally, in some embodiments, the service server 200 may further include a recommendation module 280 and a metaverse management module 290.

The components in FIG. 2 refer to components functionally implemented from a software perspective according to the operation of hardware (processor, memory, etc.) of the service server 200. For example, the data collection module 210 may be implemented by a communication unit or a processor. Additionally, the remaining modules 220, 240, 250, 260, 280, and 290 may be implemented by a processor. Additionally, the avatar DB 221 and the activity history DB 230 may be implemented by memory.

The service server 200 is a server that operates a service that creates and provides a metaverse. The service server 200 generates and provides metaverse data for displaying metaverse spaces and objects on the connected user's electronic device 100.

The metaverse data is data created by rendering an object model in a two-dimensional or three-dimensional virtual space. The objects include objects related to places provided by the virtual space (eg, products, installations, etc.), a user's avatar, and a virtual NPC avatar 201.

The data collection module 210 is configured to receive the user's personal information. In addition, the data collection module 210 allows the service server 200 to provide metaverse data embodying the metaverse environment and the user's avatar to the electronic device 100 while the user is active in the metaverse environment. It may be configured to receive the user's manipulation data and activity data captured by the electronic device 100.

The avatar creation module 220 is configured to create a virtual NPC avatar 201 that will grow according to the user's activities in the metaverse. In some embodiments, the avatar creation module 220 may be configured to generate a virtual NPC avatar whose primary personality is a personality corresponding to the user's personal information among a plurality of personalities pre-specified in the system.

In this specification, personality is an individual's psychological system that defines the implementation of specific behaviors in relation to the external environment. The personality may be structured in such a way that emotions are generated depending on the external environment and responses are output under those emotions. The output of the response may include an action, text, voice, facial expression, etc.

In other words, the personality may be expressed as a combination of internal factors such as the expression of emotions and external factors such as the output of physical reactions.

The emotion analysis module 240 is configured to determine the user's emotional state based on the received user's activity data.

The avatar growth module 250 may generate a new personality for the virtual NPC avatar 201 and update the existing personality of the virtual NPC avatar 201 with the new personality as the user's activities in the metaverse progress.

The new personality is created based on activity data and behavior pattern data acquired during activities within the metaverse. The avatar growth module 250 may grow the virtual NPC avatar 201 to be customized to the user's activities in the metaverse in terms of internal factors (eg, emotions) and external factors (eg, reactions, behavior patterns).

In some embodiments, the avatar growth module 250 may grow the virtual NPC avatar 201 to be more customized to the user around external factors based on data on the user's behavior patterns while the user is active in the metaverse. there is.

Additionally, in some embodiments, the avatar growth module 260 may update the virtual NPC avatar 201 to have additional auxiliary personalities described below.

The behavior analysis module 260 is configured to calculate behavior pattern data by analyzing the user's activities within the metaverse in terms of behavior, based on the user's manipulation data and activity data.

The recommendation module 280 may be further configured to create an auxiliary personality that recommends content corresponding to the user's tastes based on the user's activity data, emotional state, and behavior pattern data in the user's metaverse activity log data.

The metaverse management module 290 may filter one or more virtual NPC avatars 201 that have been customized and grown for the user among NPC avatars in the metaverse. In addition, the metaverse management module 290 may re-configure the metaverse so that only the one or more filtered virtual NPC avatars 201 are provided in at least some space or a dedicated space in the metaverse provided to the user. .

The service server 200 may generate a user's metaverse activity log when activity within the user's metaverse occurs.

The metaverse activity log is data that describes the user's activity details in the metaverse environment. The metaverse activity log includes metaverse data provided to the electronic device 100 during activity, user manipulation data and activity data captured by the electronic device 100, emotional state determined therefrom, and analyzed behavior patterns. It may also include .

The service server 200 may store the metaverse activity log in time series order.

The operation of the service server 200 and the components shown in FIG. 2 will be described in more detail with reference to FIG. 3 below.

FIG. 3A is a flowchart of a method of growing a virtual NPC avatar in the metaverse based on a user's metaverse activity log, according to another aspect of the present application, and FIG. 3B schematically explains the steps in the flowchart of FIG. 3A. This is a drawing.

The method of growing a virtual NPC avatar in the metaverse based on the user's metaverse activity log of FIG. 3 may be performed by one or more computing devices, such as the service server 200 of FIG. 2.

Referring to FIG. 3, the method of growing a virtual NPC avatar in the metaverse includes receiving the user's personal information (S100); and a step (S200) of generating a virtual NPC avatar 201 having as its primary personality a personality corresponding to the user's personal information among a plurality of personalities pre-specified in the system.

In step S100, the data collection module 210 may receive the user's personal information.

The user's personal information represents information entered into the system to subscribe to the Metaverse service. The user's personal information may include, for example, the user's age, gender, geographic location (eg, address, region), and other personal information. The user's personal information may be supplied to the avatar creation module 220 and used to create the primary personality of the virtual NPC avatar 201.

In the step S200, the avatar creation module 220 may generate a virtual NPC avatar whose primary personality is a personality corresponding to the user's personal information among a plurality of personalities pre-specified in the system.

The virtual NPC avatar 201 with the primary personality may be created to have a different appearance from the user's avatar.

When creating the first character, the avatar creation module 220 begins combining external aspects. For this combination, there is a basic template, and the eyes, nose, mouth position and body shape, gender, and other external factors are first determined according to the user's information and initial personality. The determined information is stored in the avatar DB 221, and when accessed through a client, data is basically randomly received, a virtual NPC avatar 201 is created within the metaverse world, and then provided to the user.

The avatar creation module 220 is configured to create a virtual NPC avatar whose primary personality is the personality corresponding to the user's personal information among the plurality of personalities pre-specified in the system. A personality representing each of a plurality of groups formed according to items of personal information may be designated as the plurality of personalities.

For example, when personal information of a 30-year-old man is received, the avatar creation module 200 may generate a virtual NPC avatar 201 whose primary personality is representative of a group of 30-year-old men.

This primary personality is a personality that does not reflect the personality of the user who will develop the virtual NPC avatar 201.

The virtual NPC avatar 201 with a primary personality maintains a simple behavior pattern such as wandering around a key location.

The virtual NPC avatar 201 initially has a completely different personality and tastes from the user, but gradually reflects the user's activities in the metaverse and learns the user's personality and tastes so that the personality and tastes gradually become similar. It can also grow by.

In some embodiments, when the virtual NPC avatar 201 is created in step S200, information describing the virtual NPC avatar 201 (e.g., appearance, primary personality, etc.) and the virtual NPC avatar 201 are provided. Virtual NPC avatar information, including personal information of the growing user, may be stored in the avatar DB 221.

In addition, the method of growing a virtual NPC avatar in the metaverse is that the service server 200 provides metaverse data embodying the metaverse environment and the user's avatar, so that the electronic device can be used while the user is active in the metaverse environment. It includes receiving the user's manipulation data and reaction data captured by 100 from the electronic device 100 (S300).

In step S300, the user's activities within the metaverse include interaction between the metaverse and the user. A user's interaction with the metaverse includes controlling the user's avatar, interacting with other avatars, and interacting with content within the metaverse, such as watching or evaluating content within the metaverse. Interacting with other avatars includes text chat, voice chat, and/or video chat.

The user's manipulation data is data representing user interface manipulation commands and content interaction manipulation commands. The user's activities within the metaverse may be defined according to the manipulation data. The user's manipulation data includes, for example, data representing user interface manipulation commands for managing the user's avatar, user interface manipulation commands for moving the avatar, and content interaction manipulation commands for progressing or viewing content.

The user's operation data may define the user's activity information. For example, when the user's electronic device 100 selects and executes chat, the user's activity information in the metaverse is defined as chat. Alternatively, when the user's electronic device 100 performs an operation to move in the metaverse world, the activity information may be defined as movement in the metaverse world.

For example, the data collection module 210 may receive the following manipulation data.

action detail k Key press time (response speed) p Pointing time (hover) to target pl Click time after pointing m Time until next action w waiting time

The user's activity data includes various body information of the user expressed during the user's activity within the metaverse when the user interacts with the metaverse. The user's activity data may include one or more of the user's image, voice, and text data.

In this specification, the content is not limited to commonly referred to as media data such as documents and videos. The content includes a metaverse environment provided in text and/or graphics, and various functions provided by the metaverse that can be used by users, such as chatting, play, and games. Appreciating the metaverse environment or traveling the metaverse environment may also be referred to as content activities. Some or all of the user's activity data may include content information about the activity.

The user's image is an image taken of the user's face during the user's activities in the metaverse. The user's image may be taken in response to the user's activity in the metaverse, or may be taken during part or all of the total time of the user's activity in the metaverse after individual access to the metaverse.

The image may be an individual image within a series of images, such as a video, or a still image. For example, the user's image may be a still image. The still image may have been taken in response to viewing content within the metaverse. Alternatively, the user's image may be a frame image of a video taken during some or all of the total time of activity in the metaverse after individual access to the metaverse.

The user's voice includes voice related to the user's activities occurring within the metaverse, such as voice chatting, exclamations, and recitation of metaverse information.

Additionally, the user's text includes text related to the user's activities that occurred within the metaverse, such as text chat.

In some embodiments, the activity data may be classified into activity data used for activities in the user's metaverse and activity data not used for activities in the user's metaverse.

The activity data used for the user's activities within the metaverse is data for commands and the contents of the activity while the user's activities within the metaverse are progressing.

Whether the activity data is activity data used in the user's activities within the metaverse may vary depending on the activity.

For example, when the activity within the metaverse is text chatting, the activity data used for the user's activity within the metaverse may include text data that constitutes the content of the text chat.

If the activity within the metaverse is voice chatting, the activity data used in the user's activity within the metaverse may include voice data that constitutes the content of the voice chat.

If the activity within the metaverse is a text chat, activity data not used in the user's activity within the metaverse includes the user's image and voice data.

On the other hand, if the activity within the metaverse is a video chat, the activity data used for the user's activity within the metaverse may include the user's image and voice data.

Additionally, the method of growing a virtual NPC avatar in the metaverse includes a step (S400) of determining the user's emotional state based on the user's activity data received in step (S300).

In step S400, the emotion analysis module 240 may determine the user's emotional state from the user's activity data using a pre-stored emotion recognition model.

In certain embodiments, the emotion recognition model may be an image-based emotion recognition model, a text-based emotion recognition model, or an integrated emotion recognition model. The integrated emotion recognition model is configured to recognize the user's emotional state from two or more of the user's image data, text data, and voice data.

The step (S400) includes determining the user's first emotional state from the user's image data, determining the user's second emotional state from the user's text data, and determining the user's third emotional state from the user's voice data. It may include one or more steps among the steps for determining the status. Each of the first to third emotional states is merely for distinguishing the type of raw activity data used for determination, and both the first to third emotional states represent results of predicting the user's emotional state.

The emotion analysis module 240 may determine the user's emotional state by inputting activity data corresponding to the included emotion recognition model into an emotion recognition model.

FIG. 4 is a flowchart of a process for determining a user's emotional state using an image-based emotion recognition model, according to an embodiment of the present application, and FIG. 5 is a schematic diagram of the process of FIG. 4 .

Referring to FIGS. 4 and 5 , the emotion analysis module 240 may determine the user's first emotional state by inputting the user's image into a pre-trained image-based emotion recognition model. The user's emotional state is used to grow the personality of the virtual NPC avatar 201, which has already been created with a primary personality, into a secondary personality.

The image-based emotion recognition model includes a face detection unit that detects a face area in an input image, a feature extraction unit that extracts user image features from the detected face area, and features extracted from a plurality of emotional states pre-specified in the model. It is a machine learning model that includes an emotion judgment unit that classifies the user's emotional state indicated by the face in the input image based on it. The machine learning model has a neural network structure.

The face detection unit may detect the user's face area through various face area detection algorithms. The face area detection algorithm may include, for example, Haar, Neural Network (NN), Support Vector Machine (SVM), Gabor, or SIFT, but is not limited thereto and may be other area detection algorithms.

In some embodiments, the user's image features may include landmarks within the user's face. The landmarks are information related to facial anatomical features such as eyes, nose, mouth, ears, etc. The landmark has a constant relative position within the face, and there is little change in the geometric relationship depending on the facial pose. Here, the pose represents the facial expression, or the rotation direction or tilt angle of the face. Extracting a landmark includes identifying the landmark and calculating location information of the identified landmark.

Additionally, the user's image features include geometric features that can be expressed as corners and edges. The geometric feature may include edges connecting landmarks. The edge may be a corner point that has not been extracted as the landmark.

The feature extraction unit may extract user image features through various feature extraction algorithms. The feature extraction algorithm may include, for example, Active Contour Model (ACM), Active Shape Model (ASM), Active Appearance model (AAM), Supervised Descent Method (SDM), or a neural network configured to extract image features. However, it is not limited to this and may be other landmark extraction algorithms. The neural network configured to extract the image features may be a neural network including a convolutional layer or other low-dimensional filter layer.

The emotion determination unit has a neural network structure configured to determine the user's emotional state by inferring the correlation between the features extracted from the input image and the emotional state. The emotion determination unit calculates the probability that the user corresponding to the input feature is classified into each of a plurality of emotional states pre-specified in the model based on the input feature, and determines the user's emotional state based on the calculated probability. It may be possible. The emotional state with the highest calculated probability value may be determined to be the user's emotional state.

The neural network structure may be, for example, a fully connected layer or MLP (Multi-layer Perceptron), but is not limited thereto and may be other NN structures.

The emotion recognition model may be trained in advance using a training data set consisting of a plurality of training samples. Each training sample has training data and label data, respectively. The training data may be user sample image data. The label data may be the user's actual emotional state at the time of capturing the sample image.

When training data for each training sample is input, the emotion recognition model computes the training data with internal parameters and calculates a result value (i.e., predicted emotional state). The parameters of the emotion recognition model are learned so that the error between the calculation result value and the value of the label data (actual emotional state) is further reduced. The trained emotion recognition model has parameters that minimize the error.

In some embodiments, a pre-trained emotion recognition model may be one in which an emotion judgment unit has been learned. At least one of the face detection unit and the feature extraction unit may use an already published algorithm without additional learning.

In this case, the emotion judgment unit may be trained in advance using a training data set consisting of a plurality of training samples. The training data in the training sample may be sample feature data extracted from the face area in the input image. The label data may be the user's actual emotional state at the time of capturing the sample image from which the sample feature is extracted.

Since the learning process of the emotion judgment unit is similar to the learning process of the emotion recognition model, detailed description is omitted.

In this way, the emotional state that can be calculated by the emotion recognition model depends on the training data set.

FIG. 6 is a schematic diagram of a process for determining a user's multiple emotional states according to an embodiment of the present application, and FIG. 7 is a detailed schematic diagram explaining the process of FIG. 6 by depth.

Referring to FIGS. 6 and 7 , the emotion analysis module 240 may determine at least one of emotion type and emotion type as the user's emotional state. The emotion analysis module 240 detects a face area in the user's image (S411), extracts features from the face area, determines the user's emotion type as the first depth (S413), and extracts features from the face area. It may be configured to extract and determine the user's emotion type as the second depth (S415).

The emotion types include positive and negative. For example, the emotion type may include only positive and negative emotions, or may include positive, neutral, and negative emotions as shown in FIG. 7 .

The emotion type is a more detailed emotional state than the emotion type. For example, the emotion types may include 7 types as shown in FIG. 7. However, the number of emotion types in FIG. 7 is merely illustrative, and the types and number may change depending on embodiments.

To this end, the emotion recognition model may include a first emotion determination unit for determining the user's emotion type and a second emotion determination unit for determining the user's emotion type.

The first emotion judgment unit and the second emotion judgment unit may be learned using training data sets containing different label data. Each label data in the training data set of the first emotion judgment unit may indicate positive, negative, or neutral. Label data in the training data set of the second emotion judgment unit may represent one of seven types of emotions.

The process of determining the emotion type and/or emotion type as an emotional state by the emotion judgment unit learned using this training data set is similar to the process of FIGS. 4 and 5, and therefore detailed description will be omitted.

Using the emotion recognition model, the user's first emotional state is determined. For example, the emotion analysis module 240 determines the user's emotional state as positive when the probability value (e.g., 0.9) of the emotional state being classified as positive is the highest using the image-based emotion recognition model of FIG. 5. .

Referring again to FIG. 3, in step S400, the emotion analysis module 240 may determine the emotional state of the second and third users from the user's voice data and/or the user's test data.

Figure 8 is a flowchart of a process for determining a user's emotional state using a text-based emotion recognition model, according to an embodiment of the present application.

Referring to FIG. 8, the emotion analysis module 240 may determine the user's second emotional state by inputting the user's text data into a pre-trained text-based emotion recognition model (S400).

As shown in FIG. 8, the user's text data may be the user's text data within the activity data, or may be data converted from the user's voice data within the activity data into text format.

To this end, the emotion analysis module 240 may include a speech-to-text (STT) unit that converts input voice into text. The STT unit is configured to extract voice features from voice data and calculate text corresponding to the voice data based on the extracted voice features. The STT unit may be comprised of an artificial neural network. The parameters of the STT unit may have values for converting the language speech of the training data set into language text by the learning method of the machine learning model.

The text-based emotion recognition model tokenizes the input text into a plurality of tokens, calculates a text vector by embedding each of the plurality of tokens, and calculates the input text based on the text vector calculated from a plurality of emotional states pre-specified for the model. It is a machine-learned natural language processing model configured to classify the user's emotional state indicated by .

The natural language processing model, for example, BERT (Bidirectional Encoder Representations from Transformers), ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately), tokenizes and embeds the input text word by word, or tokenizes it. It may be configured to segment and embed words into larger units of text (e.g., phrases, sentences, paragraphs, etc.) and/or encode them in token order. However, the natural language processing model is not limited to the above-described model and may have various structures capable of calculating text vectors from input text.

The parameters of the natural language processing model may be designed in advance to have values for processing the target language. For example, if the system 1 is designed to process Korean language data, the preprocessing unit may include Colbert, Electra, or other natural language processing models designed to process Korean language.

Since the learning process of the natural language processing model is similar to the learning process of the image-based emotion recognition model described above with reference to FIG. 4, the differences will be mainly described.

The text vector corresponds to the user's image features in Figure 4. Therefore, the natural language processing model is learned using the text vector as training data.

Additionally, in some embodiments, the emotion analysis module 240 may further include a text pre-processing unit. The emotion analysis module 240 is the user's text data in the activity data, or data converted from the user's voice data in the activity data into text format by a text pre-processing unit before inputting it into the text-based emotion recognition model. You can also process it. Then, the pre-processed text is input into the text-based emotion recognition model.

The text pre-processing unit may be configured to extract and remove preset special symbols and stop word texts from a corpus of text to be pre-processed, and/or perform normalization processing, before tokenization. However, it is not limited to this and various other text preprocessing operations may be performed.

In one example, the text preprocessing unit may use reference texts pre-registered as special symbols and stop words to check whether the text corpus to be preprocessed includes texts corresponding to special symbols and stop words. . The text preprocessing unit may remove special symbols and stop words from the text corpus and perform a tokenization operation.

The normalization process includes a cleaning process and/or a regular expression process to remove noise data from the existing corpus.

The emotion analysis module 240 preprocesses the user's text data, inputs the preprocessed text into a pre-trained text-based emotion recognition model, processes the embedding of the input text, calculates a feature vector, and calculates a feature vector based on the calculated feature vector. After calculating the probability values for each feature vector to be classified, the user's second emotional state may be determined based on the calculated probability values.

Figure 9 is a schematic diagram of a process for determining a user's emotional state using a voice-based emotion recognition model, according to an embodiment of the present application.

Referring to FIG. 9, in some embodiments, the emotion analysis module 240 may be configured to determine the user's emotional state from a voice signal in the user's voice data.

In this case, the step of calculating the user's emotion from the voice data includes converting the user's voice data into text data and using the converted text data as a pre-learned text-based emotion recognition model of FIG. 8 to identify the user's third emotion. -1 Step of determining emotional state; and/or determining the 3-2 emotional state of the user from the user's voice data using the pre-trained voice-based emotion recognition model of FIG. 9; It may also include and/or.

The emotion analysis module 240 converts the user's voice data into text data, pre-processes the converted text data, inputs the pre-processed text into a pre-trained text-based emotion recognition model, and embeds the input text into a feature vector. After calculating and calculating each probability value that the feature vector will be classified based on the calculated feature vector, the user's 3-1 emotional state may be determined based on the calculated probability value.

The voice-based emotion recognition model is a machine learning model configured to extract voice features from an input signal and classify the user's emotional state indicated by the input signal based on the voice features extracted from a plurality of emotional states pre-specified in the model. The machine learning model has a neural network structure that extracts and classifies features from signals.

The voice-based emotion recognition model may have a Convolutional Neural Network (CNN) or other NN structure.

Since the voice-based emotion recognition model is similar to the image-based emotion recognition model, detailed description will be omitted.

In some embodiments, in some embodiments, determining the 3-2 emotional state of the user may include removing noise from an audio signal; And it may include inputting the noise-removed voice signal into a voice-based emotion recognition model.

The emotion analysis module 240 may include a voice pre-processing unit that removes noise from the voice signal before extracting voice features. Then, the pre-processed voice signal is input to the voice-based emotion recognition model.

Like Figure 7, the emotion recognition model of Figures 8 and 9 may also be configured to determine multiple emotional states according to a training data set. The multiple emotional states include emotional states of a first depth and emotional states of a second depth.

The emotion analysis module 240 will be configured to process the user's text and voice to determine the user's emotion type as a first depth, and to process the user's text and voice to determine the user's emotion type as a second depth. It may be possible.

If any one of the first to third emotional states is determined in step S400, the determined emotional state is used as the user's emotional state to create the secondary personality.

Figure 10 is a schematic diagram of a process for determining a user's emotional state from various activity data using an integrated emotion recognition model, according to an embodiment of the present application.

Referring to FIG. 10, the integrated emotion recognition model includes two or more models among the image-based emotion recognition model of FIG. 4, the text-based emotion recognition model of FIG. 8, and the voice-based emotion recognition model of FIG. 9 as sub-models. The integrated emotion recognition model may recognize the user's emotional state from two or more of the user's image data, text data, and voice data.

The emotion analysis module 240 inputs the user's image data in the activity data into a pre-learned image-based emotion recognition model to determine the user's first emotional state, and inputs the user's text data in the activity data into a pre-learned text-based emotion recognition model. The user's second emotional state is determined by inputting it into the emotion recognition model, and the user's voice data in the activity data is converted into text data and input into the pre-trained text-based emotion recognition model to determine the third-first emotional state. The 3-2 emotional state may be determined by inputting the user's voice data into a previously learned voice-based emotion recognition model. Here, image data, text data, and voice data are data captured from the same activity and are activity data that forms a single metaverse activity log.

The step (S400) is a probability value of being classified into each of the plurality of emotional states (e.g., being classified into the determined first emotional state) when a plurality of emotional states are determined among the user's first to third emotional states. and determining the final emotional state of the user based on a probability value, a probability value of being classified as the determined second emotional state, and a probability value of being classified as the determined third emotional state.

To this end, the emotion analysis module 240 may determine the user's final emotional state based on each probability value of being classified as an emotional state determined by the emotion recognition model.

In some embodiments, the emotion analysis module 240 may determine the user's emotional state more truthfully by assigning different weights according to activity data.

A person's emotional state appears more truthfully in aspects that are not exposed to others. In particular, people tend to hide their negative emotional states from others. Since the activity data used in the user's activities within the metaverse is exposed to other users, it may indicate a completely false emotional state or may indicate that the level of the emotional state is somewhat suppressed even if it is a true emotional state.

The emotion analysis module 240 assigns a lower weight to the emotional state determined from activity data used in the user's activities in the metaverse, and the emotional state determined from activity data not used in the user's activities in the metaverse. It may be configured to give a higher weight to .

For example, when the activity data is received through text chat activity, activity data not used in the user's activities in the metaverse is image data, and activity data used in the user's activities in the metaverse is text data. In this case, the emotion analysis module 240 assigns a higher first weight to the first emotional state determined from the image data and a lower second weight to the second emotional state determined from the text data. , the final emotional state may be determined based on a calculation value obtained by applying the first weight to the value of the first emotional state and the second weight to the value of the second emotional state. The emotional state with the closest value to the calculation value may be determined as the final emotional state.

Additionally, in some embodiments, the emotion analysis module 240 may include an ensemble unit (ensemble learning). The ensemble unit is configured to calculate the user's emotional state by comprehensively inferring the correlation between the judgment results of a plurality of individual emotion recognition models and the user's emotional state. The ensemble unit may be configured to use voting, bagging, boosting, or stacking methods.

In this case, the emotion analysis module 240 may determine the user's final emotional state by applying the calculation result of applying the value of the corresponding emotional state to the weight assigned according to the activity data to the ensemble unit.

In some embodiments, the emotion analysis module 240 may store emotion judgment results according to the user's activities in the emotion analysis DB 234.

Referring again to FIG. 3, the method of growing the virtual NPC avatar in the metaverse is to grow the virtual NPC avatar 201 based on a pair of the user's activity data and the user's emotional state determined from the activity data. It includes creating a personality and updating the personality of the virtual NPC avatar 201 to the secondary personality (S500).

The step (S500) includes a first data set including a plurality of pairs consisting of the user's activity data for each activity in the user's metaverse, obtained according to a time series order, and the user's emotional state determined from the activity data. It includes forming a personality model indicating the secondary personality of the virtual NPC avatar 201 using the first data set. Modeling here may include building a model structure and/or setting parameter values.

As described above, the personality may be expressed as a combination of internal factors such as expression of emotions and external factors such as output of reactions. Then, the personality may be modeled as a model with internal factors as input and external factors as output.

The activity data received in step S300 includes responses output according to the external environment. The emotional state determined in step S400 represents the emotional state occurring in the relevant external environment.

In the step (S500), the avatar growth module 250 learns for each user the relationship between internal and external factors, indicating output of an appropriate facial expression, voice, or text when a specific emotional state occurs, and the virtual NPC avatar (201) ) can also create a secondary personality.

Specifically, the system (1) for growing a virtual NPC avatar in the metaverse acquires activity data and emotional state judgment results for the corresponding activities according to the time series order when the user engages in multiple activities in the metaverse. do. In the step S500, the avatar growth module 250 may model a personality model indicating a secondary personality using the first data set obtained through a plurality of activities. The modeling process is a process of defining the correlation between the determined emotional state, which is treated as input, and the received activity data, which is treated as output. By defining the correlation between these inputs and outputs, a secondary personality is created that outputs a specific response when a specific emotional state occurs in a specific external environment.

The virtual NPC avatar 201 with the secondary personality shares the same or similar emotional state as the user in a specific metaverse environment, implements the facial expression expressed by the user under the emotional state, or uses the user's text output method, It is re-configured to output information in a voice output manner. The text and voice are the result of reflecting the user's text and voice characteristics. For example, a virtual NPC avatar 201 with a secondary personality may provide content text in the same or similar text style.

The text output method may include text style, conversation text pattern, etc.

The voice output method may include voice style, conversation voice pattern, etc.

The avatar growth module 250 may update the personality of the virtual NPC avatar 201 created in step S200 from the primary personality to the secondary personality. In some embodiments, the avatar growth module 250 may store secondary personality in the avatar DB 221.

Through the above steps (S100 to S500), the virtual NPC avatar 201 may grow by customizing both internal and external factors to the user.

In some embodiments, forming the first data set may include forming the first data set such that the type of the emotional state includes only at least some emotional states among the remaining emotional states that do not have a negative value.

When creating a secondary personality only with non-negative emotional states (positive or neutral), the virtual NPC avatar 201 learns negative conversation content in moderation as much as possible, and instead focuses on learning positive or empathic parts. You can. Through this, it is possible to build a metaverse world that eliminates as many negative effects on users as possible.

Additionally, in some embodiments, the mapping method may be configured to customize the virtual NPC avatar 201 to the user with a greater focus on external aspects, based on the activity data and manipulation data received in step S300. Calculating pattern data (S600), using a second data set including the user's activity data, emotional state corresponding to the activity data, and user's behavior pattern data corresponding to the activity data, the user's metaverse A step (S700) of generating a tertiary personality of the virtual NPC avatar 201 that performs actions that mimic my behavior and updating the personality of the virtual NPC avatar 201 with the generated tertiary personality may be further included. there is.

Behavior can also be treated as an external factor that is influenced by brain factors of personality. This is because behavioral patterns are expressed differently depending on personality. This is because if your personality is impatient, your movement speed may be fast, and if your personality is more relaxed, your movement speed may be slower.

Figure 11 is a schematic diagram of the process of growing a virtual NPC avatar by customizing it to the user with more emphasis on the external aspect, according to an embodiment of the present application.

In step S600, the behavior analysis module 260 may calculate the user's behavior pattern data based on the user's manipulation data and activity data.

In some embodiments, the user's behavior pattern data includes information describing the behavior patterns of the user or his avatar in activities using the metaverse world as content, and/or activities conducted by selecting detailed content provided by the metaverse. may include information describing the behavior patterns of the user or his avatar.

For example, the user's behavior pattern data includes the user's avatar's movement line, movement location, movement frequency, location frequency, activity-related selected content, preparation time to decide on the activity, activity start time, and activity of the user's avatar in the user's metaverse. It may include one or more of the following: frequency, activity reviews, activity content, and activity scope. If the activity has related content, the preparation time taken to start the activity represents the preparation time taken to select the content, the activity start time represents the content start time, the activity frequency represents the frequency of content use, and the activity review represents the content use. Reviews (eg, reviews) may be indicated, activity content may be indicated as content content, and activity scope may be indicated as content selection range.

The behavior analysis module 260 may calculate the activity frequency based on the number of content clicks. Additionally, the behavior analysis module 260 may calculate the preparation time to determine content based on the pointing time (p) pointing to the target and the pointing time (pl) pointing to the target.

In addition, the behavior analysis module 260 may extract content usage time, reviews based on content usage, or the content of the conversation.

In this way, the behavior pattern data calculated in step S600 is related to the user's activity data received in step S300 and the user's emotional state determined in step S400, so that the virtual NPC avatar 201 determines the user's behavior pattern. It is used for additional learning.

In some embodiments, the behavior analysis module 260 may analyze activity pattern data for each activity as the user continues various activities in the metaverse and store the data in the behavior analysis DB 236 in time series order.

The step (S700) forms a second data set including the behavior pattern data calculated in step (S600), the user's activity data received in step (S300), and the user's emotional state determined in step (S400). It may also include generating a tertiary personality of the virtual NPC avatar 201 that imitates a user's behavior pattern using the second data set.

The second data set is obtained by adding corresponding behavior pattern data to the first data set. The tertiary personality may be a machine learning model similar to the secondary personality. The tertiary personality model outputs the behavior pattern of the imitation target and is modeled to have as input the remaining data excluding the behavior pattern data from the second data set.

The avatar growth module 250 may learn the virtual NPC avatar 201 to imitate the user's behavior pattern through various behavior learning methods (S700).

In one example, generating the tertiary personality in step S700 includes modeling the tertiary personality through an imitation algorithm based on the second data set. The avatar growth module 250 may store in advance various imitation algorithms capable of learning behavior cloning. The avatar growth module 250 may build an input/output model from the correlation between the behavior pattern and the remaining data in the second data set using the imitation algorithm.

Additionally, the step of learning the virtual NPC avatar 201 may include the step of inverse reinforcement learning (IRL) of the tertiary personality modeled based on the second data set.

In the above example, the process of imitation learning and inverse reinforcement learning may be performed using a re-learning framework such as MLOps (Machine Learning Operations) as a learning tool.

12 is a schematic diagram of an action tree, according to an embodiment of the present application.

In another example, learning the virtual NPC avatar 201 in step S700 includes generating the behavior tree of FIG. 12 based on the second data set; and injecting the user's behavior into the virtual NPC avatar 201 based on the behavior tree.

An activity in the action tree of FIG. 12 may be at least part of a logical path from a root node to an action node. This is because behavior is an action implemented by intention.

The avatar growth module 250 may designate a selector node, sequence node, and decorator node using emotional state and activity data, and may designate a behavior node using behavior pattern data. Additionally, the avatar growth module 250 may form a logical connection structure between corresponding nodes based on the correlation between emotional states, activity states, and behavior patterns.

The behavior tree of FIG. 12 is created as a tertiary personality.

When behavioral learning is completed in step S700, the virtual NPC avatar 201 may implement similar behavior patterns centered on the main movement path and main location of the avatar of the user who became the role model. As a result, the virtual NPC avatar 201 may not have a monotonous behavior pattern, such as wandering around a single location designated by the designer. The virtual NPC avatar 201 may automatically share its behavior pattern with the role model user or the avatar of another user with the same or similar behavior pattern as this user.

That is, the virtual NPC avatar 201 may be grown from one role model to share at least partially the behavior pattern with multiple users.

As a result, the virtual NPC avatar 201 may approach and talk to multiple user avatars while operating within the metaverse with the same or similar behavior pattern as that of a specific role model user. Additionally, the virtual NPC avatar 201 may effectively carry out propaganda activities to the plurality of user avatars at a specific location. The plurality of user avatars may have partially matching behavior patterns with the avatars of role model users, or may be user avatars that normally have different behavior patterns but whose behavior coincidentally is the same or similar to the behavior patterns.

In this way, the virtual NPC avatar 201 grows through users or user avatars, and learns new content information, existing event information, etc. from users and learns about feedback. Because of this, it may grow to deliver more comprehensive and systematic information about it.

And in this respect, the virtual NPC avatar 201 can be classified into an ecosystem that is more than an object that operates organically within the metaverse as a whole and can be used automatically.

- The information provided will vary depending on each metaverse environment.

- What content was good, what content had this function, etc.

- The number of people using the content, the time most often used to use the content, etc.

- Your own impressions and feedback information on using content or events, etc.

- Unique information acquired while communicating with others, etc.

Additionally, as the number of role model users increases, a plurality of virtual NPC avatars 201 may be created. Since the virtual NPC avatar 201 can have various personalities, the user can communicate with the virtual NPC avatar 201 that suits the user and receive empathetic conversation content from the virtual NPC avatar 201 with a familiar speaking style. , If it is expanded into a chat system with NPCs, it will be possible to expand into the emotional empathy part, etc.

Referring again to FIG. 3, in some embodiments, the method of growing a virtual NPC avatar in the metaverse includes at least one of the virtual NPC avatar 201 and personality and behavior patterns based on the user's activity log data in the metaverse. Creating an auxiliary personality that recommends content suitable for a matching user and updating the virtual NPC avatar 201 to have the auxiliary personality additionally (S800); And/or it may further include a step (S900) of filtering the metaverse information according to the personality or behavior pattern updated in step (S500), step (S700), or step (S800) and providing it to the user.

In the step (S800), the recommendation module 280 determines the secondary personality using a third data set including content information related to the user's activities in the metaverse, activity data corresponding to the content, emotional state, and behavior pattern data. You can also learn. The assistant personality infers the relationship between activity data, emotional state and behavior pattern data and the content selected for the activity, so that when an input data set including activity data, emotional state and behavior pattern data is input, the user of the input data set It may be a machine learning model that is trained to predict content suitable for users.

Figure 13 is a network structure diagram of an auxiliary nature according to an embodiment of the present application.

Referring to FIG. 13, the auxiliary network structure may include a first sub-structure and a second sub-structure. The first sub-structure models the linear correlation between input data and output data. The second sub-structure is modeled to infer a non-linear correlation between input data and output data. The second sub-structure may have various NN structures such as a fully connected layer and MLP.

The step of creating an auxiliary personality in step 800 includes forming a third data set including content information related to the user's activities in the metaverse, activity data corresponding to the content, emotional state, and behavior pattern data; Classifying data related to content and data not related to content among activity data, emotional state, and behavior pattern data in the third data set, forming a first subset consisting of data related to the content and the content information. forming a second subset of data unrelated to the content and the content information, and using the first subset to learn a first sub-structure within the auxiliary personality, the second sub-set It may also include learning a second sub-structure within the auxiliary personality using the set.

In some embodiments, the data related to the content may be some or all of emotional state and behavior pattern data. For example, data related to content may include one or more of the following: emotional state, preparation time for selecting the content, content start time, content usage frequency, content usage reviews (e.g., reviews), content content, and content selection range. It may be possible.

The first sub-structure is a linear relationship in which the more positive the emotional state when selecting the content, the higher the probability of predicting the content as suitable for the user, and the shorter the preparation time for selecting the content, the probability of predicting the content as suitable content for the user. Various linear relationships can be learned, such as this increasing linear relationship, the linear relationship in which the probability of predicting content as suitable for the user increases as the frequency of content use increases, and the linear relationship in which the probability of predicting content as suitable for the user increases as the evaluation value of content usage reviews increases. It may be possible.

The second sub-structure learns to predict content suitable for the user when the remaining data is input by inferring the non-linear relationship between the remaining data, which is difficult to model a linear relationship, and the content selected in the activity corresponding to the remaining data as a correlation. do.

The auxiliary network structure may have a Neutral CF (Collaborative Filtering) or Wide & Deep Learning structure, but is not limited thereto.

In the step S800, the avatar growth module 250 may inject the auxiliary personality into the virtual NPC avatar 201. Then, the virtual NPC avatar 201 grows to have the same or similar taste status as the user.

When the virtual NPC avatar 201 implements a specific emotional state or specific behavior pattern, it may predict content suitable for the user. The virtual NPC avatar 201 may output predicted content within the metaverse and recommend it to the user.

In some embodiments, the method of growing a virtual NPC avatar in the metaverse further includes outputting content when the virtual NPC avatar updated to have the auxiliary personality approaches a certain distance in the metaverse from the user's avatar. It may also be included.

When the virtual NPC avatar 201 updated to have the auxiliary personality approaches the role model user at a certain distance, at least part of its activity data, emotional state, and behavior pattern corresponds to the third data set under the approaching situation. Form the input data set. The virtual NPC avatar 201 may input an input data set to the auxiliary personality to predict content suitable for the user, output the predicted content within the metaverse, and provide it to the user. This is because content is more likely to attract user interest if it is recommended under the access situation.

In this way, the system 1 for growing the virtual NPC avatar in the metaverse can also effectively recommend content to the user through the moving virtual NPC avatar 201.

Through the steps S100 to S800, the service server 200 may create and grow a plurality of virtual NPC avatars 201 using each of the plurality of users as a role model.

In the step S900, the metaverse management module 290 may filter one or more virtual NPC avatars 201 that have been customized and grown for the user among NPC avatars in the metaverse. In addition, the metaverse management module 290 may re-configure the metaverse so that only the one or more filtered virtual NPC avatars 201 are provided in at least some space or a dedicated space in the metaverse provided to the user. .

In this metaverse re-construction process, only the virtual NPC avatar 201 with an updated personality or behavior pattern in step S500, step S700, or step S800 may exist in the metaverse (S900).

Each virtual NPC avatar 201 may be configured to individually provide information corresponding to updated personality and behavior. The above information is similar to content recommendation and represents various information transmitted within the metaverse other than content recommendation.

In this way, when the virtual NPC avatar 201 is configured to individually provide different information, when metaverse information is provided through the filtered virtual NPC avatar 201, the user may have the same or similar taste state. Only metaverse information may be provided.

This creates only virtual NPC avatars (201) with the same personality within one's metaverse space, and only virtual NPC avatars (201) that not only provide the information mentioned above but also have similar speaking style, speaking style, and empathy ability are applied to the user's avatar. By communicating together, you can build an environment that suits you and implement hyper-personalization.

In this way, the system and method for growing a virtual NPC avatar in the metaverse based on the user's metaverse activity log is used to promote propaganda to other third-party users through the virtual NPC avatar 201 grown with information on the user's main actions. , Alternatively, during interaction, the user may recommend desired information to a third-party user or present content and information preferred by the user without the need to directly interact with the avatar of the third-party user.

Providing such information will provide convenience to users with similar tendencies and an opportunity to experience new experiences for users with different tastes.

It will be clear to those skilled in the art that the system 1 for growing a virtual NPC avatar in the metaverse may include other components. For example, the system 1 for growing a virtual NPC avatar in the metaverse may include other hardware required for the operations described herein, including an input device for data entry and an output device for printing or other data display. It may also contain elements. In addition, it may further include a network, network interface, and protocol connecting the system 1 for growing the virtual NPC avatar in the metaverse and an external device (eg, a user terminal, an external database, etc.).

When implementing embodiments of the present invention using hardware, application specific integrated circuits (ASICs) or digital signal processors (DSPs), digital signal processing devices (DSPDs), or programmable PLDs (PLDs) configured to perform the embodiments of the present application. logic devices), field programmable gate arrays (FPGAs), etc. may be included in the components of the present application.

The operations of the system and method for growing a virtual NPC avatar in the metaverse according to the embodiments of the present application described above may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. For example, implemented with a program product comprised of a computer-readable medium containing program code, which can be executed by a processor to perform any or all steps, operations, or processes described.

The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable codes may be stored and executed in a distributed manner. Additionally, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

The present invention discussed above has been described with reference to the embodiments shown in the drawings, but these are merely illustrative examples, and those skilled in the art will understand that various modifications and modifications of the embodiments are possible therefrom. However, such modifications should be considered within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached patent claims.

Claims (16)

In a method of growing a virtual NPC avatar in the metaverse based on a user's metaverse activity log performed by one or more computing devices,
Receiving the user's personal information;
Creating a virtual NPC avatar having as a primary personality a personality corresponding to the user's personal information among pre-stored personalities - the virtual NPC avatar has a different shape from the user's avatar;
Receiving activity data and manipulation data of the user captured on an electronic device while the user is active in the metaverse, the activity data including one or more of the user's image, text, and voice data;
determining the user's emotional state from the user's activity data using a pre-stored emotion recognition model;
Generating a secondary personality of the virtual NPC avatar based on the activity data and emotional state, and updating the personality of the virtual NPC avatar with the secondary personality,
The step of creating the secondary personality is,
Forming a first data set including a plurality of pairs of user activity data for each activity in the user's metaverse, obtained according to a time series order, and the user's emotional state determined from the activity data; and
Characterized in that it includes the step of modeling a personality model indicating the secondary personality of the virtual NPC avatar using the first data set,
method. The method of claim 1, wherein the method comprises:
calculating user behavior pattern data based on the received activity data and manipulation data;
forming a second data set including the user's activity data, an emotional state corresponding to the activity data, and the user's behavior pattern data corresponding to the activity data;
using the second data set to create a tertiary personality of the virtual NPC avatar that performs actions that mimic the user's actions in the metaverse; and
Characterized in that it further comprises the step of updating the personality of the virtual NPC avatar to the generated tertiary personality,
method. The method of claim 1, wherein determining the user's emotional state from the user's activity data comprises:
A step of determining the user's first emotional state from the user's image data using a pre-trained image-based emotion recognition model,
The image-based emotion recognition model is
a face detection unit that detects a face area in the input image, a feature extraction unit that extracts image features of the user from the detected face area, and a face in the input image that is pointed out based on the features extracted from a plurality of emotional states pre-specified in the model. It is a machine learning model that includes an emotion judgment unit that classifies the user's emotional state.
The emotion judgment unit includes a first emotion judgment unit for determining the user's emotion type, and a second emotion judgment unit for determining the user's emotion type,
The image-based emotion recognition model extracts features from the face area to determine the user's emotion type as a first depth, and extracts features from the face area to determine the user's emotion type as a second depth. ,
method. The method of claim 3, wherein the step of determining the user's emotional state from the user's activity data comprises:
determining the user's second emotional state from the user's text data using a pre-trained text-based emotion recognition model; and determining the user's third emotional state from the user's voice data using the text-based emotion recognition model or the pre-trained voice-based emotion recognition model, characterized in that it further comprises one or more steps.
method. The method of claim 4, wherein the step of determining the user's emotional state from the user's activity data comprises:
When a plurality of emotional states are determined among the first to third emotional states of the user, determining the final emotional state of the user based on a probability value of being classified into each of the plurality of emotional states,
The step of determining the final emotional state is,
A higher first weight is assigned to the first emotional state determined from the image data, a lower second weight is assigned to the second emotional state determined from the text data, and the first weight is set to the value of the first emotional state. Characterized in that the final emotional state is determined based on the calculation value applied to and the second weight is applied to the value of the second emotional state,
method. The method of claim 1,
The personality of the virtual NPC avatar includes internal and external factors,
The internal factor is related to the emotional state and the external factor is related to the physical response expressed under the emotional state,
The physical response of the virtual NPC avatar with the secondary personality includes implementing facial expressions corresponding to emotional states or outputting information through the user's text output method or voice output method,
Forming the first data set includes:
Characterized in that forming the first data set such that the type of the emotional state includes only at least some emotional states among the remaining emotional states that do not have a negative value,
method. The method of claim 2, wherein the user's behavior pattern data is,
In the user's metaverse, one of the following is the movement line of the user's avatar, movement location, movement frequency, location frequency, selected content related to the activity, preparation time to decide on the activity, activity start time, activity frequency, activity review, activity content, and activity scope. Including the above,
If the activity has related content, the preparation time taken to start the activity represents the preparation time taken to select the content, the activity start time represents the content start time, the activity frequency represents the frequency of content use, and the activity review represents the content use. Characterized in that the review is indicated, the activity content indicates content content, and the activity range is indicated by a content selection range,
method. The method of claim 2, wherein the step of updating the personality of the virtual NPC avatar to the generated tertiary personality comprises:
modeling a tertiary personality using the second data set and a behavior imitation algorithm to have behavioral pattern data in the second data set as output and the remaining data as input; and
Characterized in that it includes the step of inverse reinforcement learning (IRL) on the tertiary personality modeled based on the second data set.
method. The method of claim 2, wherein the method comprises:
Generating an auxiliary personality that recommends content suitable for a user whose virtual NPC avatar matches at least one of personality and behavior patterns based on activity log data within the user's metaverse; and
further comprising updating the virtual NPC avatar to additionally have the secondary personality,
The network structure of the auxiliary nature includes a first sub-structure and a second sub-structure,
The first sub-structure models a linear correlation between input data and output data, and the second sub-structure is modeled to infer a non-linear correlation between input data and output data,
method. The method of claim 9, wherein creating the auxiliary personality comprises:
forming a third data set including content information related to the user's activities in the metaverse, activity data corresponding to the content, emotional state, and behavior pattern data;
Classifying data related to content and data not related to content among activity data, emotional state, and behavior pattern data in the third data set;
forming a first subset of data related to the content and the content information;
forming a second subset consisting of data unrelated to the content and the content information; and
Characterized in that it comprises learning a first sub-structure in the auxiliary personality using the first subset, and learning a second sub-structure in the auxiliary personality using the second subset.
method. The method of claim 9,
When the virtual NPC avatar updated to have the auxiliary personality approaches the user's avatar at a certain distance within the metaverse, content suitable for the user is provided based on at least some of the user's activity data, emotional state, and behavior pattern under the access situation. Further comprising the step of predicting and outputting the predicted content within the metaverse and providing it to the user.
method. The method of claim 2, wherein the method comprises:
When a plurality of virtual NPC avatars customized and created and grown according to the activities of each of the plurality of users in the metaverse exist in the metaverse, filtering one or more virtual NPC avatars customized to a specific user; and
Characterized by further comprising providing metaverse information through a filtered virtual NPC avatar,
method. The method of claim 12,
After filtering, the step of re-constructing the metaverse so that only the filtered virtual NPC avatars exist,
method. A computer-readable recording medium recording a program for performing a method of growing a virtual NPC avatar in the metaverse based on the user's metaverse activity log according to any one of claims 1 to 13. In a system that grows a virtual NPC avatar in the metaverse based on the user's metaverse activity log,
The system includes a service server,
The service server is,
a data collection module that receives the user's personal information, the user's activity data and manipulation data captured on an electronic device while the user is active in the metaverse;
An avatar creation module that generates a virtual NPC avatar with a personality corresponding to the user's personal information among pre-stored personalities as a primary personality;
an emotion analysis module that determines the user's emotional state from the user's activity data using a pre-stored emotion recognition model;
Characterized in that it includes an avatar growth module that generates a secondary personality of the virtual NPC avatar based on the activity data and emotional state, and updates the personality of the virtual NPC avatar to the secondary personality,
system. The method of claim 15, wherein the service server:
It further includes a behavior analysis module that calculates user behavior pattern data based on the received activity data and manipulation data,
The avatar growth module is,
Form a second data set including the user's activity data, emotional state corresponding to the activity data, and user's behavior pattern data corresponding to the activity data, and use the second data set to enter the user's metaverse. generating a tertiary personality of the virtual NPC avatar that performs actions that mimic the behavior, and further configured to update the virtual NPC avatar's personality with the generated tertiary personality,
system.

Images