KR20230103671A - Method and apparatus for providing a metaverse virtual reality companion animal capable of forming a relationship based on a user's emotions in a communication system - Google Patents

Abstract

The present invention relates to a method and apparatus for providing a metaverse virtual reality companion animal capable of forming a relationship based on a user's emotion in a communication system. Specifically, the present invention generates a companion animal avatar in the virtual reality of the metaverse in a communication system, analyzes the user's reaction emotion information from the user's reaction face image to the operation screen of the companion animal avatar, and then the user's reaction emotion Based on the information, the companion animal avatar performs a mutual reaction action, and based on the user's reaction and emotion information, the user's feedback on the specific behavior pattern of the companion animal avatar is reflected to determine the behavior pattern and personality of the companion animal avatar. A method and apparatus for providing a companion animal avatar that can emotionally communicate with and form a relationship with a user by performing modification and reinforcement learning.

Description

Method and apparatus for providing a metaverse virtual reality companion animal capable of forming a relationship based on a user's emotion in a communication system IN A COMMUNICATION SYSTEM}

The present invention relates to a method and apparatus for providing a metaverse virtual reality companion animal capable of forming a relationship based on a user's emotion in a communication system. Specifically, the present invention generates a companion animal avatar in the virtual reality of the metaverse in a communication system, analyzes the user's reaction emotion information from the user's reaction face image to the operation screen of the companion animal avatar, and then the user's reaction emotion Based on the information, the companion animal avatar performs a mutual reaction action, and based on the user's reaction and emotion information, the user's feedback on the specific behavior pattern of the companion animal avatar is reflected to determine the behavior pattern and personality of the companion animal avatar. A method and apparatus for providing a companion animal avatar that can emotionally communicate with and form a relationship with a user by performing modification and reinforcement learning.

With the recent increase in single-person households, the demand for companion animals that can live with humans as a family is increasing. What people who have companion animals generally expect from their companion animals is to perform an emotional connection with them as beings with feelings. However, while people live with companion animals and want to exchange emotional sympathy, they feel difficulties in raising companion animals for realistic reasons such as burden of management responsibility, time, cost, and space problems.

Recently, the virtual reality-related technology of the metaverse is attracting attention. Metaverse is a compound word of meta, which means processing and abstraction, and universe, which means the real world, and is a term that means the virtual world. Due to the metaverse, the boundary between the real world and the virtual world is gradually blurring. This is because various actions performed in the real world can be performed identically or similarly in the virtual world.

After creating a companion animal avatar in the virtual reality of the metaverse, analyzing the user's reaction emotion information from the user's reaction face image to the motion screen of the companion animal avatar, and based on the user's reaction emotion information, the companion animal avatar interacts with each other. If the reaction action is performed, the user can have the experience of emotionally sympathizing with the user and the companion animal in the metaverse virtual reality. In addition, based on the user's reaction and emotion information, if the user's feedback on the specific behavior pattern of the companion animal avatar is reflected and the behavior pattern and personality of the companion animal avatar are modified and reinforced, the user's good and disliked reactions are rejected. Since animal avatars can recognize and reflect, users can form emotional relationships with companion animal avatars while raising them.

Therefore, there is a need for a method and apparatus for providing a metaverse virtual reality companion animal capable of forming a relationship based on a user's emotion in a communication system.

Republic of Korea Patent No. 10-2199843 (2020.12.31.) (Virtual companion animal provision system in augmented reality)

The present invention aims to solve the following problems in order to solve the above problems.

An object of the present invention is to provide a method and apparatus for providing a metaverse virtual reality companion animal capable of forming a relationship based on a user's emotion in a communication system.

The present invention generates a companion animal avatar in the virtual reality of the metaverse in a communication system, analyzes the user's reaction emotion information from the user's reaction face image to the operation screen of the companion animal avatar, and then based on the user's reaction emotion information Accordingly, an object of the present invention is to provide a method and device for a companion animal avatar to perform an interactive reaction operation.

The present invention provides a method and apparatus for modifying and reinforcing the behavior pattern and personality of a companion animal avatar by reflecting user feedback on a specific behavior pattern of a companion animal avatar based on the user's reaction and emotion information in a communication system. It aims to provide

An object of the present invention is to provide a method and apparatus for providing a companion animal avatar that can emotionally communicate with and form a relationship with a user in a communication system.

The problems of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

Various embodiments of the present disclosure provide a method of operating a user's terminal in a communication system. The terminal includes a transceiver, a memory, a processor, an input device, an output device, and a camera. The method includes: transmitting setting information of a companion animal input through the input device to the server; receiving first motion information about an appearance and motion of a companion animal avatar in a virtual space from the server; The process of outputting the motion of the companion animal avatar through the output device based on the first motion information, and the process of obtaining a face image of the user looking at the companion animal avatar of the output device through the camera; The process of transmitting the emotion information of the user generated based on the face image to the server, and the process of receiving second motion information of the companion animal avatar generated based on the emotion information from the server; and outputting the motion of the companion animal avatar in the virtual space through the output device based on the second motion information.

Various embodiments of the present invention provide a user's terminal in a communication system. The terminal includes a memory, a processor, an input device, an output device, and a camera, and the processor is configured to perform a method of operating a user terminal in a communication system according to various embodiments of the present disclosure.

Various embodiments of the present invention provide a computer program recorded on a computer readable storage medium. The computer program is configured to perform a method of operating a user's terminal in a communication system according to various embodiments of the present disclosure.

The present invention may provide a method and apparatus for providing a metaverse virtual reality companion animal capable of forming a relationship based on a user's emotion in a communication system.

The present invention generates a companion animal avatar in the virtual reality of the metaverse in a communication system, analyzes the user's reaction emotion information from the user's reaction face image to the operation screen of the companion animal avatar, and then based on the user's reaction emotion information Thus, a method and apparatus for performing a mutual reaction action by a companion animal avatar may be provided.

The present invention provides a method and apparatus for modifying and reinforcing the behavior pattern and personality of a companion animal avatar by reflecting user feedback on a specific behavior pattern of a companion animal avatar based on the user's reaction and emotion information in a communication system. can provide.

The present invention may provide a method and apparatus for providing a companion animal avatar that can emotionally communicate with and form a relationship with a user in a communication system.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 illustrates a communication system according to various embodiments of the present invention.
2 shows a block diagram of a configuration of a user terminal according to various embodiments of the present invention.
Figure 3 shows a block diagram of the configuration of a server according to various embodiments of the present invention.
4 illustrates an example of a process for a user terminal to perform an action instruction for a companion animal avatar in a metaverse virtual reality as a voice message according to various embodiments of the present disclosure.
5 illustrates the structure of a machine learning model according to various embodiments of the present invention.
6 illustrates an example of a biometric information acquisition process according to various embodiments of the present disclosure.
7 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.
8 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.
9 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.
10 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.
11 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.
12 illustrates an example of a process of recognizing a user's emotion or emotion based on a visual image according to various embodiments of the present disclosure.
13 illustrates an example of a process of recognizing a user's emotion based on machine learning from a user's video image according to various embodiments of the present disclosure.
FIG. 14 illustrates an example of a process of assigning or subtracting a degree of sympathy for each interaction class with emotion extracted from a user's image and voice according to various embodiments of the present disclosure.
15 illustrates an example of a process of determining the motion of a companion animal avatar by reflecting the user's emotion feedback according to the user's face recognition according to various embodiments of the present disclosure.
16 illustrates an example of a process of initially pre-selling companion animal avatars in metaverse virtual reality according to various embodiments of the present disclosure.
17 illustrates an example of a process of naming and recognizing companion animal avatars by voice according to various embodiments of the present disclosure.
18 illustrates an example of a process of reflecting a degree of rapport according to user emotion recognition through training with a companion animal avatar according to various embodiments of the present disclosure.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. The present invention may be embodied in many different forms and is not limited to the embodiments described herein.

1 illustrates a communication system according to various embodiments of the present invention.

Referring to FIG. 1 , a communication system according to various embodiments of the present invention includes a user terminal (100: 100-1, 100-2, ..., 100-n), a server 200, a wired/wireless communication network ( 300).

The user terminals 100 (100-1, 100-2, ..., 100-n) are terminals operated by companions, that is, users who raise companion animal avatars in virtual reality. The user terminal (100: 100-1, 100-2, ..., 100-n) transmits the characteristic information of the companion animal to the server 200 through the wired/wireless communication network 300, and the server 200 ) is an electronic device capable of receiving companion animal avatar information in virtual reality and virtual reality information to which the companion animal avatar belongs. User terminals 100: 100-1, 100-2, ..., 100-n are personal computers, cellular phones, smart phones, tablet computers, head mount display (HMD) devices for VR (virtual reality) capable of transmitting and receiving An input device capable of inputting information, an output device capable of outputting information, a memory capable of storing information, a camera capable of taking an image of an object in the foreground, and a device capable of transmitting and receiving information. It may be an electronic device including a transceiver and at least one processor capable of performing information calculation.

The server 200 is a server operated by an online service provider based on a metaverse virtual reality platform. The server 200 allows the user to search based on the companion animal's characteristic information provided by the user terminals 100 (100-1, 100-2, ..., 100-n) through the wired/wireless communication network 300. A virtual reality animal avatar with the same or similar appearance, behavioral pattern, and personality as the intended companion animal is created, and the behavioral pattern and personality of the virtual reality animal avatar are trained or modified according to the characteristic information or feedback information input from the user, and the creation An electronic device capable of providing the user terminal 100 with service information for performing audio-visual communication between the user and the virtual reality avatar based on the user's appearance, behavior pattern, and personality. The server 200 may be an electronic device including a memory capable of storing information, a transceiver capable of transmitting and receiving information, and at least one processor capable of performing information calculation.

The wired/wireless communication network 300 provides a communication path through which user terminals 100: 100-1, 100-2, ..., 100-n, and the server 200 can transmit and receive signals and data to each other. to provide. The wired/wireless communication network 300 is not limited to a communication method according to a specific communication protocol, and an appropriate communication method may be used according to an implementation example. For example, when configured as an Internet protocol (IP) based system, the wired/wireless communication network 300 may be implemented as a wired/wireless Internet network, and the user terminals 100: 100-1, 100-2, ..., 100-n), and when the server 200 is implemented as a mobile communication terminal, the wired/wireless communication network 300 may be implemented as a wireless network such as a cellular network or a wireless local area network (WLAN) network. there is.

2 shows a block diagram of a configuration of a user terminal according to various embodiments of the present invention.

Referring to FIG. 2 , a user terminal (100: 100-1, 100-2, ..., 100-n) according to various embodiments of the present invention includes a transceiver 110, a memory 120, a processor 130 , an input device 110 , an output device 120 , and a camera 160 .

The transceiver 110 is connected to the processor 130 and transmits and/or receives signals. All or part of the transceiver 110 may be referred to as a transmitter, a receiver, or a transceiver. The transceiver 110 is a wired access system and a wireless access system, such as an institute of electrical and electronics engineers (IEEE) 802.xx system, an IEEE Wi-Fi system, a 3rd generation partnership project (3GPP) system, and a 3GPP long term evolution (LTE) system. , 3GPP 5G new radio (NR) system, 3GPP2 system, at least one of various wireless communication standards such as Bluetooth may be supported.

The memory 120 is connected to the transceiver 110, the memory 120, the processor 130, the input device 140, and the output device 150, information input through the input device 140, the transceiver 110 ) It is possible to store information received from the server 200 through communication. Also, the memory 120 may store information on an image captured by the camera 160 . In addition, the memory 120 is connected to the processor 130 and may store data such as basic programs for operation of the processor 130, application programs, setting information, and information generated by operation of the processor 130. . The memory 120 may include volatile memory, non-volatile memory, or a combination of volatile and non-volatile memories. Also, the memory 120 may provide stored data according to a request of the processor 130 .

The processor 130 may be configured to implement the procedures and/or methods proposed in the present invention. The processor 130 controls overall operations of the user terminal 100 . For example, the processor 130 transmits or receives information or the like through the transceiver 110 . Also, the processor 130 writes data to and reads data from the memory 120 . Also, the processor 130 receives information through the input device 140 . Also, the processor 130 outputs information through the output device 140 . Also, the processor 130 captures an image through the camera 160 . The processor 130 may include at least one processor.

The input device 140 is connected to the processor 130 and may input information and the like. According to an embodiment, the input device 140 may input information received from another device connected to the wired/wireless communication network 300 through the transceiver 130 . The input device 140 may include a touch display, a keypad, and a keyboard.

The output device 150 is connected to the processor 130 and can output information in the form of video/audio. According to an embodiment, the output device 150 may output information received from another device connected to the wired/wireless communication network 300 through the transceiver 110 . The output device 150 may include a display, a speaker, and the like.

The camera 160 is connected to the processor 130 and may capture an image of a front object.

Figure 3 shows a block diagram of the configuration of a server according to various embodiments of the present invention.

Referring to FIG. 3 , a server 200 according to various embodiments of the present disclosure includes a transceiver 210 , a memory 220 and a processor 230 .

The transceiver 210 is connected to the processor 230 and transmits and/or receives signals. All or part of the transceiver 210 may be referred to as a transmitter, a receiver, or a transceiver. The transceiver 210 is a wired access system and a wireless access system, such as an institute of electrical and electronics engineers (IEEE) 802.xx system, an IEEE Wi-Fi system, a 3rd generation partnership project (3GPP) system, and a 3GPP long term evolution (LTE) system. , 3GPP 5G new radio (NR) system, 3GPP2 system, at least one of various wireless communication standards such as Bluetooth may be supported.

The memory 220 is connected to the transceiver 220, and through communication of the transceiver 220, a user terminal 100: 100-1, 100-2, ..., 100-n, or a service terminal 300: Information received from 300-1, 300-2, ..., 300-n) may be stored. In addition, the memory 220 is connected to the processor 230 and may store data such as basic programs for operation of the processor 230, application programs, setting information, and information generated by operation of the processor 230. . The memory 220 may include volatile memory, non-volatile memory, or a combination of volatile and non-volatile memories. Also, the memory 220 may provide stored data according to a request of the processor 230 .

The processor 230 may be configured to implement the procedures and/or methods proposed in the present invention. The processor 230 controls overall operations of the server 200 . For example, the processor 230 transmits or receives information or the like through the transceiver 210 . Also, the processor 230 writes data to and reads data from the memory 220 . The processor 230 may include at least one processor.

4 illustrates an example of a process for a user terminal to perform an action instruction for a companion animal avatar in a metaverse virtual reality as a voice message according to various embodiments of the present disclosure.

In step S401, the user may speak a voice message instructing the companion animal avatar in the metaverse virtual reality displayed on the output device in the user's terminal. The terminal may receive the user's voice through a microphone included in an input device of the terminal.

In step S402, the terminal converts the received voice into speech to text (STT), converts the voice message into a text message, and recognizes the newly received message of the user by comparing it with the existing conversation set.

In step S403, a voice message is transmitted from the terminal to the server, and the instruction value according to the user's message is connected to the action value of the companion animal avatar. That is, an operation of the companion animal avatar suitable for the user's instruction may be determined.

In step S404, the server transmits motion information corresponding to the finally determined action value of the companion animal avatar and corresponding user interface information to the terminal. The terminal may output the screen of the companion animal avatar operating in the output device based on motion information of the companion animal avatar received from the server. Through this, the user can feel that the companion animal avatar in virtual reality responds to the instructions given by voice.

5 to 13, based on an image of a user's face looking at a screen on which a companion animal avatar is displayed on an output device of a user terminal, the user's emotion information is recognized using a machine learning model stored in the terminal's memory. An example of the process is shown. The terminal may obtain an image image of the user's face looking at the screen of the terminal using a front camera in the terminal.

5 illustrates the structure of a machine learning model according to various embodiments of the present invention.

The structure of a multi-layer perceptron (MLP) for providing content based on machine learning analysis of biometric information according to various embodiments of the present invention is shown.

Deep learning, as one of the emerging technologies in the field of machine learning, is a neural network composed of a plurality of hidden layers and a plurality of hidden units included in them. When low-level features are input to a deep learning model, these basic features are transformed into high-level features that can better explain the problem to be predicted while passing through a plurality of hidden layers. In this process, since prior knowledge or intuition of an expert is not required, subjective factors in feature extraction can be removed, and a model with higher generalization ability can be developed. Furthermore, in the case of deep learning, since feature extraction and model construction are composed of one set, there is an advantage in that the final model can be formed through a simpler process compared to existing machine learning theories.

A multi-layer perceptron (MLP) is a type of artificial neural network (ANN) with multiple nodes based on deep learning. Each node uses a non-linear activation function with neurons similar to animal connection patterns. This nonlinear property makes it possible to linearly distinguish inseparable data.

Referring to FIG. 5 , the artificial neural network 500 of the MLP model according to various embodiments of the present invention includes one or more input layers 510, a plurality of hidden layers 530, and one or more outputs. It consists of an output layer (550).

Input data such as an RGB value of each pixel in at least one ultrasound image per unit time is input to a node of the input layer 510 . Here, the user's biometric information, eg, electrocardiogram information, concentration level, happiness emotion rate information, and adjusted content information, eg, content genre, content topic, and content channel information 511 are deep It corresponds to the basic characteristic (low level feature) of the learning model.

A node of the hidden layer 530 performs calculations based on input factors. The hidden layer 530 is a layer in which units defined by a plurality of nodes formed by integrating the user's biometric information and the information 511 of the adjusted content are stored. As shown in FIG. 5 , the hidden layer 530 may include a plurality of hidden layers.

For example, when the hidden layer 530 is composed of a first hidden layer 531 and a second hidden layer 533, the first hidden layer 531 includes user biometric information and adjusted content information 511 As a layer in which first units 532 defined by a plurality of nodes formed by consolidating are stored, the first unit 532 corresponds to a higher characteristic of the user's biometric information and the information 511 of the adjusted content. The second hidden layer 533 is a layer in which second units 534, defined as a plurality of nodes formed by consolidating the first units of the first hidden layer 531, are stored. Corresponds to the upper characteristics of 1 unit 532.

Nodes of the output layer 550 represent calculated prediction results. The output layer 550 may include a plurality of prediction result units 551 . Specifically, the plurality of prediction result units 551 may include two units of a true unit and a false unit. Specifically, the true unit is a prediction result unit that means that the concentration value and happiness emotion rate among the user's biometric information are highly likely to be equal to or greater than the critical concentration value and threshold happiness emotion rate after adjusting the content to the adjusted content, and the false unit is the adjusted content content It is a prediction result unit that means that it is unlikely that the concentration value and the happiness emotion rate among the user's biometric information are higher than the critical concentration value and the critical happiness emotion rate after adjusting the content of Rho.

Weights are assigned to connections between the prediction result units 551 and the second units 534 included in the second hidden layer 533, which is the last layer among the hidden layers 530. Based on these weights, it is predicted whether the concentration value and the happy emotion ratio among the biometric information of the user after adjusting the content to the adjusted content are greater than or equal to the critical concentration value and the threshold happiness emotion rate.

For example, if any one of the second units 534 predicts a concentration value and a happiness emotion rate among the user's biometric information to be greater than or equal to a threshold concentration value and a threshold happiness emotion rate after adjusting the content to the adjusted content, a true unit. and a false unit, respectively. A positive weight will be assigned to the connection to the true unit, and a negative weight will be assigned to the connection to the false unit. Conversely, if any one of the second units 534 predicts the concentration value and the happiness emotion rate of the user's biometric information to be less than the threshold concentration value and the threshold happiness emotion rate after content adjustment to the adjusted content, true unit and false unit Each unit is connected, and a negative weight is assigned to a connection to a true unit, and a positive weight is assigned to a connection to a false unit.

A plurality of connection lines may be formed between the plurality of second units 534 and the charm unit. When the total sum of the plurality of connection lines has a positive value, the information 511 of the user's biometric information and the adjusted content in the input layer 510 is the concentration value and happiness among the user's biometric information after adjusting the content to the adjusted content. The emotion rate will be predicted with factors that are greater than or equal to the critical concentration value and the critical happiness emotion rate. According to an embodiment, whether the concentration value and the happiness emotion rate of the user's biometric information are equal to or greater than the critical concentration value and the threshold happiness emotion rate after adjusting the content to the adjusted content is compared with the total sum of a plurality of connection lines and a preset value. can also be predicted.

The artificial neural network 500 of the MLP model learns by adjusting learning parameters. According to one embodiment, the learning parameters include at least one of a weight and a variance. The learning parameters are iteratively adjusted through an optimization algorithm called gradient descent. Each time a prediction result is computed from a given data sample (forward propagation), the performance of the network is evaluated through a loss function that measures the prediction error. Each learning parameter of the artificial neural network 500 is adjusted by gradually increasing in the direction of minimizing the value of the loss function, and this process is called back-propagation.

After adjusting the content to the adjusted content through the above model, it is possible to predict whether or not the concentration value and the happiness emotion rate among the user's biometric information are greater than or equal to the critical concentration value and the threshold happiness emotion rate, and appropriately adjusted content can be determined.

6 illustrates an example of a biometric information acquisition process according to various embodiments of the present disclosure.

Referring to FIG. 6, an example of a process of obtaining biometric information of a user from a picture image of the upper body including the user's face is shown.

Specifically, the terminal may obtain information on the user's emotion type and emotion rate by analyzing a pattern of specific emotion points in the face and a pattern of specific movement points in the upper body of the video image. More specifically, a standard for obtaining a facial image of a basic expression (non-expression) and an emotional expression according to a number of emotional states compared thereto from the user, and defining a reference vertex for each facial part in the facial image of the basic expression or emotional expression. After masking (matching) the model and mapping a plurality of vertices for each facial part to the user's face image, coordinate change values between vertices in the same facial part are extracted from the facial image of the basic expression and emotional expression, and the vertices are compared to the human face group by expression unit (Action Unit) defined in, and extract the facial movement information of the expression unit including the change value of the centroid of the vertices of each expression unit group, and extract the extracted facial movement information as data for each emotional state Information on the type of emotion and the emotion rate can be obtained by making it a base.

At this time, the facial movement information in each expression unit may include information on happiness, neutral, anger, fear, surprise, sadness, disgust, and the like. there is.

7 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.

In various embodiments of the present invention, emotion and emotion are used as interchangeable terms, that is, terms that refer to the same meaning of emotion.

According to various embodiments of the present disclosure, the terminal i) detects a face in a visual image, ii) detects a face point, iii) defines a facial muscle, iv) tracks a muscle movement, v) extracts an effective emotion parameter, vi) builds a rule base, vii) Emotion recognition can be performed.

i) For face detection, the terminal uses the Viola-Jones algorithm using the facial feature-based Haar model. Accordingly, compared to existing color-based models based on facial features, it overcomes the limit caused by lighting and shows lower computing cost than AAM-based methods, which are statistical approaches based on facial models, and can be applied to real-time processes.

ii) Regarding face point detection, the terminal can detect 68 face points for the eyes, nose, and mouth using a DLIB-based Regression Trees ensemble algorithm.

iii) Regarding facial muscle definition, the terminal may define 39 facial muscles sensitive to expressions according to facial point-based facial muscle definition according to the facial action coding system.

iv) Regarding muscle movement tracking, the terminal may track the movement of the facial muscles according to the image frame, and may track the XY coordinates of the facial muscles and the amount of change in area according to facial expressions.

v) Regarding effective emotion parameter extraction, the terminal may extract emotions such as happiness, sadness, surprise, anger, disgust, and fear by extracting effective emotion parameters based on facial muscle movement characteristics for six basic emotions.

v) Regarding rule base construction, the terminal builds a rule base for emotion recognition from actual facial expression data, overcomes the personalization problem using a universal neutral face model, and builds a rule base based on statistical analysis. Face for emotion recognition model can be built.

vi) Regarding emotion recognition, the terminal can use a fuzzy-based emotion recognition algorithm to calculate a probability for each awakening based on a rule, and minimize the sensitivity of emotion recognition through a probabilistic approach of each emotion.

8 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.

Specifically, FIG. 8 illustrates a process of extracting face image data from an image of the upper body including the user's face.

First, the terminal performs Haar feature extraction. Specifically, the terminal compares each part of the image pixel by pixel using a kernel corresponding to facial features such as eyes and nose, and calculates a probability that a face exists from an input video image. The terminal subtracts the brightness values corresponding to the black and white areas from the input image and finds something that is above the threshold value.

Next, the UE performs cascade classification. Specifically, the terminal detects a face by using a cascade classifier capable of recognizing whether or not a face is a face with a Haar feature. In the embodiment of FIG. 8 , a previously trained Frontalface_alt2 model was applied. After reducing candidates from certain Haar features such as eyes, the terminal detects the final face with Haar features of detailed face elements.

Next, the terminal detects the landmark. In the embodiment of FIG. 8, the UE applied the DLIB-based Regression Trees ensemble model trained with the iBUG 300-W dataset. For the frontal face detected from the Haar model, 68 face points for eyes, nose, mouth, eyebrows, and chin are detected. Each face point has x, y coordinate values in a two-dimensional space.

Next, the terminal performs coordinate system normalization. Specifically, the detected face points are located in a coordinate system with the upper left corner of the screen as the origin. This acts as noise because it can show different values depending on the position of the face even though it is the same expression. Therefore, this problem can be solved by normalizing to a relative coordinate system with the eyebrows as the origin.

9 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.

Specifically, FIG. 9 shows a face point index for each name of an action unit (AU).

Facial muscle definition based on facial points according to the Facial Action Coding System (FACS) is as follows. The Facial Action Coding System (FACS) is a standard system that defines the movements of facial muscles used to express emotions (Ekman P, 1978). To analyze the corresponding facial muscles, 39 facial muscles (action units) are defined based on 68 facial points. At this time, the anatomical factors of the face are taken into consideration so that the movements of the muscles defined in the FACS can be expressed appropriately.

10 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.

Specifically, FIG. 10 shows an example of a facial muscle in the form of a polygon.

According to various embodiments of the present disclosure, the terminal may extract facial muscle features. Specifically, the terminal calculates the area and center x, y coordinates of the facial muscles from the XY coordinates of the facial points, calculates the area by considering the polygon shape of the non-standard facial muscles, and calculates the area by considering the polygon shape of the center x, y coordinates as well. The center of gravity can be calculated taking into account

According to various embodiments of the present disclosure, a terminal may track movement of facial muscles. Specifically, the terminal extracts the area and center x, y coordinates of each muscle every frame to track the movement of the facial muscles, calculates the amount of change in muscle characteristics with the current face based on the normal face, and uses this to calculate the image It can track muscle movements that change according to the frame.

11 illustrates an example of a process of recognizing a user's emotion or emotion based on a video image according to various embodiments of the present disclosure.

Specifically, FIG. 11 illustrates a process of obtaining data of each emotion or sensitivity based on facial muscle movements.

According to various embodiments of the present disclosure, the terminal may determine basic emotions or facial muscle movement characteristics according to basic emotions. Define the movement characteristics of facial muscles according to the emotion defined by FACS. It considers all six basic emotions: happiness, sadness, surprise, anger, disgust, and fear.

12 illustrates an example of a process of recognizing a user's emotion or emotion based on a visual image according to various embodiments of the present disclosure.

Specifically, FIG. 12 shows a rule base for emotion recognition. The rule base of FIG. 12 is a result derived through statistical analysis and rule base construction.

Specifically, statistical analysis was performed as follows. In order to build a rule base for emotion recognition, statistical analysis was performed on muscle movement feature variables from actual facial expression data, and facial expressions of happiness, sadness, surprise, anger, disgust, and fear, six basic emotions for 62 people Each photo was acquired, and ANOVA was performed for comparison of feature variables for 7 groups including the normal photo. Since the number of samples for each emotion is the same, post-hoc analysis was performed using Tukey.

In addition, rule base construction was performed as follows. As a result of statistical analysis, muscle movement feature variables showing differences according to emotion were established as a rule base for emotion recognition. In addition, a universal neutral face model was constructed to overcome the limitations of the facial structure showing slight differences for each individual.

Finally, according to various embodiments of the present invention, the terminal calculates emotion using a fuzzy-based emotion recognition algorithm. Specifically, the terminal calculates the probability for each emotion for the current face based on a rule base built from actual data, and shows how high the probability of each emotion is compared to other emotions to recognize the final emotion. to calculate the final probability. For example, the terminal considers all of the following conditions in order to calculate the probability that the current face is happy. i) What is the probability that the current face is happy compared to normal? ii) What is the probability that the current face is happy compared to sad? iii) What is the probability that the current face is happy compared to surprise? iv) What is the probability that the current face is happy compared to the angry face? v) What is the probability that the current face is happy compared to disgust? vi) What is the probability that the current face is happy compared to fear? As a result of considering the above, the terminal finally recognizes the emotion with the highest probability as the current emotion.

13 illustrates an example of a process of recognizing a user's emotion based on machine learning from a user's video image according to various embodiments of the present disclosure.

Referring to FIG. 13 , a process in which a visual image of a user's face is input through INPUT, goes through a machine learning model, and finally outputs emotion information of the user is illustrated.

The terminal may analyze what emotion the user has in detail through a machine learning model. In addition, the user can calculate the level of concentration as a value between 0 and 100, for example, whether the user is in a normal state, a concentration state, or an immersion state through emotion analysis. there is. This can be related to the intensity of emotions.

FIG. 14 illustrates an example of a process of assigning or subtracting a degree of sympathy for each interaction class with emotion extracted from a user's image and voice according to various embodiments of the present disclosure.

Referring to FIG. 14 , it is classified according to each situation in which a user and a companion animal avatar in virtual reality interact with each other. For example, a situation of selling companion animal avatars, a situation of caring for them, a situation of training, and a situation of playing are classified.

In addition, specific detailed steps are classified for each situation in which the user and the companion animal avatar in virtual reality interact. For example, for pre-sale, avatar selection is completed, name is being set, etc. are classified. For example, with regard to care, feeding, pooping, peeing, bathing, nursing, and the like are classified. For example, with respect to training, name calling, sitting, getting up, waiting, individual skills, and the like are classified. For example, with respect to play, walking, traveling, and the like are classified.

Also, based on the user's face image or voice, the analysis source of the emotion type or concentration is classified.

In addition, the intensity of emotion or the intensity of concentration can be analyzed for each emotion type or concentration.

In addition, the addition or subtraction of the degree of rapport between the user and the companion animal avatar is determined based on the analyzed emotion type, emotion intensity, or concentration intensity of the user's companion animal avatar.

Addition or subtraction of the determined sympathy may affect various fields such as loyalty to the user of the companion animal avatar, training performance, and growth rate. The user can feel the emotional interaction with the companion animal avatar while watching how their emotions affect the companion animal avatar.

15 illustrates an example of a process of determining the motion of a companion animal avatar by reflecting the user's emotion feedback according to the user's face recognition according to various embodiments of the present disclosure.

Referring to FIG. 15 , when a user uses an application (app) of a companion animal avatar in a terminal, the companion animal avatar in the application may respond to a case in which an owner actually searches for a companion animal.

When the application operates, the terminal may recognize the user's face image. Thereafter, the terminal may determine the user's emotion for the companion animal avatar and the concentration state for the user's companion animal avatar from the user's face image or voice.

The terminal may analyze the user's emotion type, emotion intensity, and concentration from the user's face image or voice, determine whether the user's empathy is positive or negative, generate the user's emotion information, and transmit it to the server.

The server may determine the operation of the companion animal avatar corresponding to the user's emotional state based on the user's emotion information received from the terminal, and transmit the operation information of the companion animal avatar to the terminal. The determined motion of the companion animal avatar may be determined to be charming or show off a skill when the user has a positive emotion or concentrates on the companion animal avatar for a long time, and the user has a neutral or negative emotion. If it is determined to have it, it may be determined to comfort, and if it is determined that no emotion is detected from the user for a certain period of time due to the user leaving the terminal for a long time, or the user's concentration on the companion animal avatar decreases It may be decided to bark and call.

16 illustrates an example of a process of initially pre-selling companion animal avatars in metaverse virtual reality according to various embodiments of the present disclosure.

Referring to FIG. 16, as one of the situations in which a user interacts with a companion animal avatar in the metaverse virtual reality, a situation in which a companion animal avatar is first sold is shown.

Users can select a companion animal avatar they like among various companion animal avatars on the screen, pay goods such as currency and coins in virtual reality, and then receive the companion animal avatar.

17 illustrates an example of a process of naming and recognizing companion animal avatars by voice according to various embodiments of the present disclosure.

Referring to FIG. 17 , the user may name the first companion animal avatar, speak the name by voice, and be recognized by the terminal. The terminal checks the recognized name from the user through STT (speech to text) and analyzes the user's emotion in the voice message when recognizing the name by voice. In the case of positive emotion, the companion animal avatar's sympathy with the user increases can be set to do so.

18 illustrates an example of a process of reflecting a degree of rapport according to user emotion recognition through training with a companion animal avatar according to various embodiments of the present disclosure.

Referring to FIG. 18 , an example of determining a reaction of a companion animal avatar in a training situation as one of situations in which a user can communicate with a companion animal avatar is shown.

A user may input instructions for the companion animal avatar through a voice message or a button click. The terminal recognizing the user's instruction message transmits the user's instruction to the server. The server may determine an operation of the companion animal avatar corresponding to the user's instruction based on the received instruction. In addition, in the case of a voice message, the emotion of the user in the voice message may be analyzed instead of simply recognizing the instruction itself. Whether or not the user has a liking for the companion animal avatar may be analyzed through the analysis of the user's emotion information. Through this, when the user's liking for the companion animal avatar is analyzed, the server may determine to operate a skill skill in addition to instructions directed to the companion animal avatar.

When positive emotions are extracted, the degree of sympathy increases, when negative emotions are extracted, the degree of sympathy decreases, and when neutral emotions are extracted, the degree of sympathy may not change.

According to various embodiments of the present invention, a user can select a desired companion animal avatar and receive a sale, and then interact in a space called a training center in the metaverse. For example, the degree of sympathy between the user and the companion animal avatar may be increased through the following interaction method. For example, the user can increase the level of rapport by naming the companion animal avatar and recognizing it as the owner through menus such as calling the animal by voice and petting it. For example, the user can increase the level of sympathy by regularly performing caring actions directly related to survival, such as feeding, watering, and cleaning up urine and excrement to the companion animal avatar. For example, the user can increase the level of rapport through simple training such as sitting, getting up, lying down, turning over, and waiting while giving snacks to the companion animal avatar. For example, a user increases rapport with a companion animal avatar by taking a walk, playing with it, or traveling.

In the above-described exemplary interaction process, the result may be reflected in the degree of sympathy between the user and the companion animal avatar through emotion recognition based on user face recognition. For example, three types of attention of the user, that is, a normal state, a concentration state, and an immersive state may be measured. For example, seven types of emotions of the user, that is, pleasure, surprise, sadness, anger, fear, displeasure, and calmness may be measured. Each type of concentration, type of emotion, degree of concentration, and intensity of emotion can be measured together.

The higher the user's concentration at the moment of interaction with the companion animal avatar, the more weight can be assigned to the increase in sympathy. Also, in the case of emotions of pleasure or surprise that are classified as positive among the seven types of emotions of the user, a weight may be assigned to an increase in sympathy.

If the user's concentration and positive emotions are high, the user's emotions are transmitted to the companion animal avatar in the virtual world, and the companion animal avatar's growth rate, training speed and performance, and loyalty may increase.

In the case of a user with a high degree of sympathy and a companion animal avatar, the companion animal avatar may perform a certain action by first responding to the user's emotion based on face recognition. An operation such as licking the user for consolation when the user is sad, acting cute when the user is in a bad mood, or asking for an object the user likes may be performed first without the user's instruction.

According to various embodiments of the present invention, in the process of a user raising and interacting with a companion animal avatar in a virtual world, it is possible to utilize facial recognition-based emotion recognition as well as menu selection and voice instructions in a terminal screen. So, even if the user does not have to speak or express with gestures, the companion animal avatar can recognize the emotions the user is feeling with only facial expressions, and it can be actively used in interaction with the companion animal avatar in the virtual world.

In various embodiments of the present disclosure, the user's emotion index includes an emotion index and a concentration index. A brief explanation of each indicator is as follows.

The user's emotion index can include 7 types of emotions, that is, joy, surprise, sadness, anger, fear, displeasure, and calmness. It is expressed as a value between 0 and 100, and the higher the value, the more positive it is. .

The user's concentration index may include three levels of concentration states, that is, an index of a normal state, a concentration state, and an immersion state, and is expressed as a value between 0 and 100, and the higher the value, the higher the degree of concentration.

Through the naming of companion animal avatars and emotional recognition through voice in calling companion animal avatar names, the more positive emotions, the faster the sympathy rises.

In training companion animal avatars, the higher the concentration index of the user and the higher the pleasure and surprise of the emotional index, the faster the sympathy rises.

The higher the user's concentration index and the higher the enjoyment of the emotional index, the faster the rapport rises in walking and playing with the companion animal avatar.

Various embodiments of the present disclosure provide a method of operating a user's terminal in a communication system. The terminal includes a transceiver, a memory, a processor, an input device, an output device, and a camera. The method includes: transmitting setting information of a companion animal input through the input device to the server; receiving first motion information about an appearance and motion of a companion animal avatar in a virtual space from the server; The process of outputting the motion of the companion animal avatar through the output device based on the first motion information, and the process of obtaining a face image of the user looking at the companion animal avatar of the output device through the camera; The process of transmitting the emotion information of the user generated based on the face image to the server, and the process of receiving second motion information of the companion animal avatar generated based on the emotion information from the server; and outputting the motion of the companion animal avatar in the virtual space through the output device based on the second motion information.

According to an embodiment, the setting information of the companion animal may include at least one of an animal type of the companion animal, a detailed species of the companion animal among the animal types, and detailed external features of the companion animal. . The setting information of the companion animal may further include basic characteristic setting information including body characteristics, behavioral characteristics, and personality characteristics of the companion animal. Personality information of the companion animal avatar may be generated based on the basic characteristic setting information. The personality information may be related to at least one of the sympathy of the companion animal avatar with the user, the loyalty of the companion animal avatar with the user, the training speed of the companion animal avatar, and the growth speed of the companion animal avatar. .

According to an embodiment, the memory may store a machine learning model for analyzing the emotion information of the user based on the face image of the user. The emotion information of the user includes information of one or more emotion types detected at a threshold rate or higher among a plurality of predetermined emotion types, and concentration numerical values for the companion animal avatar of the user detected from the face image among a predetermined concentration scale. information may be included. Each of the plurality of predetermined emotion types may have a positive or negative emotional value in relation to the companion animal avatar. The emotion information may further include the emotion type information according to the one or more sensed emotions, the emotion numerical value, and the detected intensity numerical value.

According to an embodiment, the personality information of the companion animal avatar may be modified by repeating the process of transmitting the emotion information and the process of receiving the second motion information of the companion animal avatar generated based on the emotion information. can Based on the personality information, new second motion information of the companion animal avatar may be generated and received.

According to an embodiment, the method may further include receiving, from the server, arbitrary simulation information about a first situation of the companion animal avatar together with the first action information. The emotion information may include information on the emotion type of the user for a motion based on the first motion information of the companion animal avatar in the first situation, the emotional numerical value, and the concentration numerical value. When simulation information of a second situation similar to the first situation is received together with the second motion information, the companion animal avatar may operate according to the second motion information generated based on the emotion information.

According to one embodiment, the input device may include a microphone. The method may further include transmitting a voice message for the companion animal avatar of the user acquired through the microphone to the server. As the voice message includes the set name of the companion animal avatar, the emotional information including a higher positive emotion numerical value and higher concentration numerical value may be transmitted to the server. The personality information of the companion animal avatar may have a degree of sympathy with the user modified based on the emotion information. Based on the personality information, new second motion information of the companion animal avatar may be generated and received.

According to an embodiment, the personality information of the companion animal avatar may have a degree of sympathy with the user modified based on the emotion information. The degree to which the personality information of the companion animal avatar is modified in correspondence to the emotion type information, the emotion numerical value, and the concentration numerical value of the user is determined according to the type of the first situation in which the companion animal avatar is placed. can When the type of the first situation is one of a plurality of predetermined situations, and each corresponding emotion type predetermined for each of the plurality of predetermined situations is included in the information of the emotion type of the user; A degree of modification of the personality information may be high.

According to an embodiment, the plurality of predetermined emotion types may include one or more of joy, surprise, sadness, anger, fear, displeasure, and calmness. The plurality of predetermined situations may include training, walking, and play. When the type of the first situation is training, and when the information of the emotion type of the user is joy or surprise, the degree to which the personality information is modified may be high. When the type of the first situation is walking or playing, and when the information on the emotion type of the user is enjoyment, the degree to which the personality information is modified may be high.

Various embodiments of the present invention provide a user's terminal in a communication system. The terminal includes a memory, a processor, an input device, an output device, and a camera, and the processor is configured to perform a method of operating a user terminal in a communication system according to various embodiments of the present disclosure.

Various embodiments of the present invention provide a computer program recorded on a computer readable storage medium. The computer program is configured to perform a method of operating a user's terminal in a communication system according to various embodiments of the present disclosure.

Various embodiments of the present invention provide a user's terminal in a communication system. The terminal includes a memory, a processor, an input device, an output device, and a camera, and the processor is configured to perform a method of operating a user terminal in a communication system according to various embodiments of the present disclosure.

Various embodiments of the present invention provide a computer program recorded on a computer readable storage medium. The computer program is configured to perform a method of operating a user's terminal in a communication system according to various embodiments of the present disclosure.

In the case of implementing the embodiment of the present invention using hardware, ASICs (application specific integrated circuits) or DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices) configured to perform the present invention , FPGAs (field programmable gate arrays), etc. may be provided in the processor of the present invention.

Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable medium. In addition, the structure of data used in the above-described method may be recorded on a computer-readable storage medium through various means. Program storage devices, which may be used to describe a storage device containing executable computer code for performing various methods of the present invention, should not be construed as including transitory objects such as carrier waves or signals. do. The computer-readable storage media includes storage media such as magnetic storage media (eg, ROM, floppy disk, hard disk, etc.) and optical reading media (eg, CD-ROM, DVD, etc.).

The embodiments described above are those in which elements and features of the present invention are combined in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form not combined with other components or features. In addition, it is also possible to configure an embodiment of the present invention by combining some elements and/or features. The order of operations described in the embodiments of the invention may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment. It is obvious that claims that do not have an explicit citation relationship in the claims can be combined to form an embodiment or can be included as new claims by amendment after filing.

It will be clear to those skilled in the art that the present invention can be embodied in other forms without departing from the technical spirit and essential characteristics of the present invention. Accordingly, the above embodiments should be considered in all respects as illustrative rather than restrictive. The scope of the present invention should be determined by reasonable interpretation of the appended claims and all possible changes within the equivalent scope of the present invention.

100: user terminal 110: transceiver
120: memory 130: processor
140: input device 150: output device
160: camera 200: server
210: transceiver 220: memory
230: processor 300: wired / wireless communication network
500: artificial neural network 510: input layer
511: input information 530: hidden layer
531: first hidden layer 532: first unit
533: second hidden layer 534: second unit
550: output layer 551: prediction result unit

Claims (10)

A method of operating a terminal of a user in a communication system, wherein the terminal includes a transceiver, a memory, a processor, an input device, an output device, and a camera,
Transmitting the setting information of the companion animal input through the input device to the server;
Receiving first motion information about the appearance and motion of a companion animal avatar in a virtual space from the server;
outputting a motion of the companion animal avatar through the output device based on the first motion information;
obtaining a face image of the user looking at the companion animal avatar of the output device through the camera;
transmitting emotion information of the user generated based on the face image to the server;
Receiving second motion information of the companion animal avatar generated based on the emotional information from the server;
And outputting the motion of the companion animal avatar in the virtual space through the output device based on the second motion information.
method.
According to claim 1,
The setting information of the companion animal includes at least one of an animal type of the companion animal, a detailed species of the companion animal among the animal types, and detailed features of the companion animal's appearance;
The setting information of the companion animal further includes basic characteristic setting information including body characteristics, behavioral characteristics, and personality characteristics of the companion animal;
Characteristic information of the companion animal avatar is generated based on the basic characteristic setting information;
The personality information is related to at least one of the sympathy of the companion animal avatar with the user, the loyalty of the companion animal avatar with the user, the training speed of the companion animal avatar, and the growth rate of the companion animal avatar,
method.
According to claim 2,
The memory stores a machine learning model for analyzing the emotion information of the user based on the face image of the user,
The emotion information of the user includes information of one or more emotion types detected at a threshold rate or higher among a plurality of predetermined emotion types, and concentration numerical values for the companion animal avatar of the user detected from the face image among a predetermined concentration scale. contains information;
Each of the plurality of predetermined emotion types has a predetermined positive or negative emotion value in relation to the companion animal avatar,
The emotion information further includes the emotion type information, the emotional numerical value, and the detected concentration numerical value according to the one or more detected emotions.
method.
According to claim 3,
The personality information of the companion animal avatar is modified by repeating the process of transmitting the emotion information and the process of receiving the second motion information of the companion animal avatar generated based on the emotion information;
Based on the personality information, the new second motion information of the companion animal avatar is generated and received.
method.
According to claim 3,
Further comprising receiving, from the server, arbitrary simulation information about a first situation of the companion animal avatar together with the first motion information;
The emotion information includes information on the emotion type of the user for a motion based on the first motion information of the companion animal avatar in the first situation, the emotional numerical value, and the concentration numerical value;
When simulation information of a second situation similar to the first situation is received together with the second motion information, the companion animal avatar operates according to the second motion information generated based on the emotion information.
method.
According to claim 5,
The input device includes a microphone,
Transmitting a voice message for the companion animal avatar of the user obtained through the microphone to the server;
As the voice message includes the name of the set companion animal avatar, the emotional information including a higher positive emotion numerical value and a higher concentration numerical value is transmitted to the server,
The personality information of the companion animal avatar is modified based on the emotional information, the degree of sympathy with the user,
Based on the personality information, the new second motion information of the companion animal avatar is generated and received.
method.
According to claim 5,
The personality information of the companion animal avatar is modified based on the emotional information, the degree of sympathy with the user,
The degree to which the personality information of the companion animal avatar is modified is determined according to the type of the first situation in which the companion animal avatar is located, corresponding to the emotion type information, the emotion numerical value, and the concentration numerical value of the user. becomes,
When the type of the first situation is one of a plurality of predetermined situations, and each corresponding emotion type predetermined for each of the plurality of predetermined situations is included in the information of the emotion type of the user; The degree to which the personality information is modified is high,
The plurality of predetermined emotion types include one or more of joy, surprise, sadness, anger, fear, displeasure, and calmness;
The predetermined plurality of situations include training, walking, and play,
When the type of the first situation is training, and when the information of the emotion type of the user is enjoyment or surprise, the degree to which the personality information is modified is high;
When the type of the first situation is walking or playing, and when the information of the emotion type of the user is enjoyment, the degree to which the personality information is modified is high.
method.
According to claim 1,
The emotion information of the user includes information of one or more emotion types detected at a threshold rate or more among a plurality of predetermined emotion types;
The plurality of predetermined emotion types include one or more of joy, surprise, sadness, anger, fear, displeasure, and calmness;
When the information of the emotion type of the user is sadness, the second action information includes a comforting action;
When the information on the emotion type of the user is displeasure, the second motion information includes an action of flirting or a motion of the companion animal avatar that the user likes in advance.
method.
In a user's terminal in a communication system,
Including a transceiver, memory, processor, input device, output device, camera,
wherein the processor is configured to perform the method according to any one of claims 1 to 8;
server.
A computer program recorded on a computer readable storage medium, configured to perform the method according to any one of claims 1 to 8.

Images