KR20240005419A - Digital content providing method and server supporting counterpart custom character motion control - Google Patents

Abstract

A digital content provision server and method supporting an opponent's customized character movement control are provided. The server according to the embodiment is a server that provides digital content in which a first character corresponding to a first user and a second character corresponding to a second user interact in a virtual space, and first user movement information of the first user a user data collection unit that receives; a character database storing first character information of the first character; a motion information conversion unit that converts the first user motion information into the first character motion information; a content generator that implements movement of the first character in the virtual space based on the first character movement information and the first character information; a preference information collection unit that collects a second user's feedback data on the first digital content including the first character and generates the second user's preference movement information for the first character based on the collected feedback data; and a character motion recommendation unit that provides first character recommended motion information to the first user based on the first character preference motion information of the second user.

Description

Digital content providing method and server supporting counterpart custom character motion control}

The present invention relates to a method and server for providing digital content that supports control of character movements customized to the opponent. Specifically, the present invention is a method of providing digital content that supports customized character movement control that can arouse the interest and interest of the other user by supporting the user to naturally assume a preferred posture according to the preference information of the other user using digital content together. and servers.

With the Fourth Industrial Revolution, digital technology has developed further, allowing consumers to enjoy a wider variety of content through digital means. In the past, entertainment, games, and play content accounted for most of the digital content, but now, offline experience content is gradually moving online, such as watching cultural content such as performances online or conducting school classes online. It is increasing. This expansion of the realm of digital content is expected to further expand through three-dimensional virtual space, especially mixed reality environments such as meta-verse.

In the metaverse or a virtual space where digital content is provided, users interact with other characters through their characters. Interaction in virtual space can be performed in a more active and expanded form than offline. In other words, exaggerated poses and facial expressions that cannot be performed offline can be conveyed to the other party through a character in a virtual space, and by further arousing the other party's interest and interest, it is possible to induce the other party to focus more on the interaction.

Arousing the other party's interest and interest is a very important factor when conducting education between a caregiver (teacher or parent) and a child in a virtual space. Virtual space corresponds to a space with a higher degree of freedom than offline space. Therefore, when children are unable to concentrate on educational activities and their interest in educational activities is dispersed, a problem arises in which the effectiveness of education decreases.

Therefore, while interaction (education) is in progress, the caregiver character must be able to appropriately perform the child's preferred movements, facial expressions, etc. to maintain the child's interest and interest and encourage the child to continue to focus on educational activities. , a setting in which methods and devices that can support such caregivers (users) are required.

Public Patent Publication No. 10-2004-0096799

The object of the present invention is to control a character to correspond to the user's movements, and recommend the character's motion according to the preference information of the other user who also uses digital content, thereby supporting the user to take the other user's preferred character movements. The goal is to provide a digital content provision method and server that supports the opponent's customized character movement control.

In addition, another task of the present invention is a digital content that supports character movement control customized to the opponent, which can output the user's preference information and movement recommendation timing by analyzing user images taken of the user through an artificial intelligence-based emotion analysis model. The provision method and server are provided.

In addition, another object of the present invention is to provide a digital content providing method and server that supports control of character movements customized to the other user, which can provide a user environment that can support the user taking the actions of the other user's preferred character.

The objects of the present invention are not limited to the objects mentioned above, and other objects and advantages of the present invention that are not mentioned can be understood by the following description and will be more clearly understood by the examples of the present invention. Additionally, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof indicated in the patent claims.

In order to solve the above problem, a digital content providing server supporting customized character movement control according to some embodiments of the present invention provides a first character corresponding to the first user and a second character corresponding to the second user in a virtual space. A server that provides interactive digital content, comprising: a user data collection unit that receives first user movement information of the first user; a character database storing first character information of the first character; a motion information conversion unit that converts the first user motion information into the first character motion information; a content generator that implements movement of the first character in the virtual space based on the first character movement information and the first character information; a preference information collection unit that collects a second user's feedback data on the first digital content including the first character and generates the second user's preference movement information for the first character based on the collected feedback data; and a character motion recommendation unit that provides first character recommended motion information to the first user based on the first character preference motion information of the second user.

In addition, the character motion recommendation unit provides recommended motion information for the first character to the first user through a posture support auxiliary screen that is distinct from the motion screen for the first character, and the recommended motion information for the first character is provided to the second user. It may include guide movement information according to the first character preference movement information.

In addition, the character motion recommendation unit displays the guide motion information and the first user motion information together on the posture support auxiliary screen, and the character motion recommendation unit determines whether the first user motion information matches the guide motion information. And, when the first user movement information and the guide movement information match, providing the posture support auxiliary screen to the first user may be terminated.

In addition, when the second user's first character preference motion information consists of a plurality of pieces, the character motion recommendation unit provides a plurality of first character preference motion information of the second user with a different degree of motion change in the current first user's motion information. First character preference motion information that is relatively smaller than one character's preference motion information may be determined, and guide motion information corresponding to the determined first character preference motion information may be generated and provided as the first character recommendation motion information.

Additionally, the preference information collection unit may include a user image collection module that collects a second user image captured by a second user using the first digital content as the feedback data; a user emotion analysis module that generates emotion information of the second user from the second user image using a deep learning-based pre-trained emotion analysis model; an analysis image determination module that determines at least one analysis image from the emotional information of the second user; a character preference motion determination module that collects motion information of the first character from the first digital content corresponding to the output time of the analysis image and generates first character preference motion information of the second user; and a motion recommendation timing determination module that determines a motion recommendation timing based on the emotion information of the second user.

In addition, the emotion analysis model is a classification model based on a convolutional neural network learned to determine the emotion class of the second user for each frame image by analyzing the movement change of the second user in the second user image for each frame, An emotion class may correspond to a positive emotion or a negative emotion.

In addition, the second user's emotion information is composed of a plurality of frame images to which an emotion class is assigned for each frame image arranged according to time, and the motion recommendation timing determination module determines the second user's emotion information from the second user's emotion information. By identifying emotional changes, you can determine when to recommend movement.

In addition, the emotion analysis model is composed of a Yolo (You Only Look Once) network, and the emotion analysis model identifies at least one of the main part of the second user's face and the main part of the body in the second user image, Based on a change in at least one of the identified main parts of the face and the main part of the body, the second user's emotion class is determined as a positive emotion or a negative emotion, and a confidence value for the determined class is also output, and the analysis is performed. The image determination module may first extract a frame image whose class is determined to be a positive emotion, and extract a frame image for which a confidence value greater than a preset threshold is given from the extracted frame image to determine the at least one analysis image.

A method of providing digital content supporting customized character movement control according to some embodiments of the present invention provides digital content in which a first character corresponding to a first user and a second character corresponding to a second user interact in a virtual space. A method comprising: receiving first user movement information of the first user; converting the first user motion information into the first character motion information; implementing movement of the first character in the virtual space based on the first character movement information and the first character information; and providing first character recommended motion information to the first user based on the first character preference motion information of the second user, wherein the first character preference motion information of the second user is the first character preference motion information. The second user's feedback data for the first digital content including is collected, and is generated based on the collected feedback data.

A digital content providing system supporting a counterpart's customized character movement control according to some embodiments of the present invention includes a first user device that generates first user movement information of a first user; a second user device generating second user movement information of a second user; and a server that generates digital content in which a first character corresponding to the first user and a second character corresponding to the second user interact in a virtual space, wherein the server generates digital content that interacts with the first character of the first user. a user data collection unit that receives movement information; a character database storing first character information of the first character; a motion information conversion unit that converts the first user motion information into the first character motion information; a content generator that implements movement of the first character in the virtual space based on the first character movement information and the first character information; a preference information collection unit that collects a second user's feedback data on the first digital content including the first character and generates the second user's preference movement information for the first character based on the collected feedback data; and a character motion recommendation unit that provides first character recommended motion information to the first user based on the first character preference motion information of the second user.

A method of providing digital content that supports the other user's customized character movement control, and the server controls the character to respond to the user's movements, but recommends character motion according to the preference information of the other user using the digital content together, so that the user can match the other user's preference. It can support taking character actions. Accordingly, it is possible to support the other user to continue to focus on the interaction on the digital content, and to stimulate the interest and interest of the other user, thereby maximizing the effect of the interaction in the digital content.

In addition, the user's preference information and movement recommendation timing can be determined by analyzing the user's captured video, and the other user's preferred character's movements can be supported at an appropriate time.

In addition, the user environment according to the embodiment supports the user to take the other user's preferred character posture by following the guide motion, so that the user can more easily take the other user's preferred character posture.

In addition to the above-described content, specific effects of the present invention are described below while explaining specific details for carrying out the invention.

FIG. 1 is a conceptual diagram illustrating a digital content provision system that supports character movement control customized to another party according to some embodiments of the present invention.
Figure 2 exemplarily shows a situation in which a sensing device senses user information.
Figures 3 and 4 exemplarily show screens where information sensed through an image sensor is reflected in a character and output.
Figure 5 is an example diagram for explaining movement control of a first character of a first user.
Figure 6 shows the configuration of a server according to some embodiments of the present invention.
Figure 7 is a block diagram showing the main configuration of a preference information collection unit according to an embodiment.
FIG. 8 exemplarily shows a situation in which a second user image is generated by photographing a second user using the first digital content.
Figure 9 exemplarily shows input data and output data of an emotion analysis model.
Figure 10 exemplarily shows an analysis image determined according to emotional information of a second user.
Figure 11 is an example diagram for explaining the process of determining a motion recommendation point.
Figure 12 is an example screen in which a posture support auxiliary screen is output on the first user operation screen.
13 and 14 are exemplary screens in which first character recommended movement information is provided through a posture support auxiliary screen.
Figure 15 is a flowchart of a method of providing digital content that supports character movement control customized to the other party according to some embodiments of the present invention.

Terms or words used in this specification and patent claims should not be construed as limited to their general or dictionary meaning. According to the principle that the inventor can define terms or word concepts in order to explain his or her invention in the best way, it should be interpreted with a meaning and concept consistent with the technical idea of the present invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are only one embodiment of the present invention and do not completely represent the technical idea of the present invention, so they cannot be replaced at the time of filing the present application. It should be understood that there may be various equivalents, variations, and applicable examples.

Terms such as first, second, A, and B used in the present specification and claims may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term 'and/or' includes any of a plurality of related stated items or a combination of a plurality of related stated items.

The terms used in the specification and claims are merely used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "include" or "have" should be understood as not precluding the existence or addition possibility of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. .

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains.

Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless clearly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

Additionally, each configuration, process, process, or method included in each embodiment of the present invention may be shared within the scope of not being technically contradictory to each other.

Hereinafter, with reference to FIGS. 1 to 15, a digital content providing system supporting customized character movement control according to some embodiments of the present invention will be described.

FIG. 1 is a conceptual diagram illustrating a digital content provision system that supports character movement control customized to another party according to some embodiments of the present invention. Figure 2 exemplarily shows a situation in which a sensing device senses user information. Figures 3 and 4 exemplarily show screens where information sensed through an image sensor is reflected in a character and output. Figure 5 is an example diagram for explaining movement control of a first character of a first user.

The digital content provision system 10 can provide digital content to a plurality of users. Here, digital content refers to content including virtual characters and virtual backgrounds implemented in a three-dimensional virtual space. Multiple users can create their own characters, participate in digital content through the created characters, and interact with each other through the characters. Digital content may be content in which multiple users participate together to achieve a specific purpose in a virtual space. For example, digital content may be content in which educators and children participate together for children's learning and education in a virtual space. Additionally, digital content may be content in which a singer provides a performance to an audience in a virtual space and the audience enjoys the singer's performance. Digital content is structured in the form of a metaverse to provide multiple users with a three-dimensional virtual world in which characters created by two or more users can participate and perform social, economic, and cultural activities similar to the real world. It can be configured.

Referring to FIG. 1, a digital content providing system 10 according to some embodiments of the present invention includes a server 100 and a plurality of user devices 200.

The server 100 may provide a service environment in which digital content can be used to a plurality of user devices 200. The server 100 and the plurality of user devices 200 may be implemented as a server-client system. In an embodiment, the server 100 may provide a service environment to users through a service application installed on each of the plurality of user devices 200. Here, the service application may be a dedicated application for providing a service environment, or a web browsing application for providing through a web page.

The user device 200 includes a user terminal 210 capable of operating a service application in a wired or wireless communication environment. In an exemplary embodiment, the user terminal 210 may be a personal computer (PC), but the present invention is not limited thereto, and may be applied without limitation to a device capable of mounting a service application as described above. For example, the user terminal 210 may include various types of electronic devices, such as a smart phone, a laptop, a tablet, a mobile phone, a smart phone, or a wearable device (eg, a watch-type terminal).

The user terminal 210 includes a communication unit that receives digital content from the server 100, a storage unit that stores the received digital content, a display unit that displays the received digital content, an input unit that receives user input, and a control unit that controls the configuration. may include each.

The server 100 is a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, and It may be implemented as at least one of a RAID (redundant array of inexpensive disks, or redundant array of independent disks) system, but the present embodiment is not limited thereto.

Meanwhile, the communication network 300 serves to connect the server 100 and the user terminal 210. In other words, the communication network 300 refers to a network that provides a connection path so that user terminals 210 can transmit and receive data after connecting to the server 100. Networks may include networks based on wired Internet technology, wireless Internet technology, and short-distance communication technology. Wired Internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

Wireless Internet technologies include, for example, Wireless LAN (WLAN), DLNA (Digital Living Network Alliance), Wibro (Wireless Broadband), Wimax (World Interoperability for Microwave Access: Wimax), and HSDPA (High Speed Downlink Packet). Access), HSUPA (High Speed Uplink Packet Access), IEEE 802.16, Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced), Wireless Mobile Broadband Service (WMBS) and 5G NR (New Radio) technology. However, this embodiment is not limited to this.

Short-range communication technologies include, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, and Near Field Communication. At least one of NFC), Ultrasound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio) may include. However, this embodiment is not limited to this.

The server 100 and the user terminal 210 that communicate through the network can comply with technical standards and standard communication methods for mobile communication. For example, standard communication methods include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), and EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only). , at least one of Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTEA), and 5G New Radio (NR) may include. However, this embodiment is not limited to this.

In digital content, the character's movement can be implemented by reflecting the user's movement. The user device 200 may include a sensing device 220 that detects the user's movement. The sensing device 220 may generate user movement information. The sensing device 220 can detect changes in the user's facial expression and changes in the user's pose. The sensing device 220 may include an image sensor 220A, such as a camera that photographs at least a portion of the user's body in real time, and a position sensor 220B, such as a gyro sensor attached to at least a portion of the user's body.

Referring to FIG. 2, the image sensor 220A can generate facial expression change information that can collect movement or facial expression changes of the user's face. The image sensor 220A can photograph the user's face in real time, recognize the user's face area in the captured image, and map the recognized face area with a plurality of points. The changing point may change according to the movement or expression change of the user's face, and the image sensor 220A may track the point change and generate expression change information. That is, facial expression change information, which is time series information that can confirm changes in major parts of the face, such as eye blinking or mouth shape changes over time, can be collected through the image sensor 220A.

Additionally, as shown in FIG. 2, the position sensor 220B may be attached to a part of the user's body. That is, the position sensor 220B may be attached to the user's major body parts, arms, ankles, pelvis, chest, etc., and may recognize changes in the position of the user's body to generate user's pose change information. That is, pose change information, which is time series information that can confirm changes in the positions of major parts of the body over time, can be collected through the position sensor 220B.

The sensing device 220 and the user terminal 210 may be connected to enable data exchange. In an embodiment, the sensing device 220 and the user terminal 210 may be configured to exchange data through short-range wireless communication, but are not limited thereto. The facial expression change information and pose change information generated through the sensing device 220 may be provided to the server 100 through the user terminal 210 as motion information. The server 100 may generate the movement of the corresponding character through the movement information. That is, the character's facial expression may change based on the facial expression change information, and the character's pose may change based on the pose change information. Referring to Figures 3 and 4, it can be seen that the character's facial expression is implemented based on the user's movement information collected in real time, and the character's facial expression also changes according to the change in the user's motion information.

The server 100 according to an embodiment of the present invention can operate a character corresponding to the user according to the user's movements. Accordingly, interaction between a plurality of characters in virtual space is performed based on the movements of users in the real world.

Referring to Figure 5, a first character of a first user corresponding to an educator or a parent and a second character of a second user corresponding to a student or child may interact (e.g., learn) within digital content. there is. The first user and the second user can create a first character and a second character, respectively. The first user may select as the first character a character with whom a psychological mutual trust relationship (rapport) has been formed with the second user so that the second user may have more interest and interest in learning activities.

Here, the first character moves based on the movement of the first user. The server 100 may support a first user performing movements of a first character preferred by a second user. In other words, the server 100 can support the first user to perform the movements and facial expressions of the first character preferred by the second user, and induce the second user to become more immersed and focused on the first character, thereby creating a virtual space. It can support further maximizing the effect of interaction.

Hereinafter, the character movement control support process performed by the server 100 will be described in more detail. In addition, the following description is for recommending to the first user a movement according to the second user's preference information in the interaction between the first user corresponding to the educator (parent) and the second user corresponding to the student (child). Explain the process and the data processing process for it. However, this is only an example, and the data processing process according to the users, number of users, and interactions between users of the present invention is not limited to this.

Figure 6 shows the configuration of a server according to some embodiments of the present invention.

Referring to FIG. 6, the server 100 includes a user data collection unit 110, a character database 120, a motion information conversion unit 130, a character movement recommendation unit 140, a preference information collection unit 150, and content. Includes a generating unit 160.

The user data collection unit 110 may receive user data from each of the plurality of user devices 200 . In embodiments, user data may include movement information collected from each user device. Here, the motion information may include facial expression change information and pose change information of each user. In an embodiment, the user data collection unit 110 may receive first motion information from a first user device.

Here, the user data collection unit 110 may perform normalization of the received user data. Normalization can refer to the task of matching data collected from various sensor devices and transmitted in various formats into one consistent format.

The user data collection unit 110 collects and normalizes user movement information for each user and uses it to create a character corresponding to each user. That is, based on the first movement information, the first character of the first user is created and the movement is controlled.

The character database 120 may store information about characters, that is, character information. Digital content according to an embodiment may provide a plurality of characters. Each of the plurality of users may have created a character by determining detailed information about the character, such as gender, age, appearance, size, skin color, clothing, basic movement, and voice profile. Character information for each character corresponding to a plurality of users may be stored in the character database 120. That is, the first character information of the first user and the second character information of the second user may be respectively stored in the character database 120. Here, characters can be prepared in various types. By way of example, the character may be a character depicting a human, but is not limited thereto, and may be an animal character or a creative character with an interesting shape. Additionally, the character may be configured to have at least a body that can move in virtual space and a face that can display the character's emotions. In embodiments, changes in the user's facial expression and pose may be reflected in facial changes and body changes of the corresponding character.

The motion information conversion unit 130 may receive user motion information and character information for each user, and generate character motion information for the character corresponding to each user. Character movement information is data about the character's movement, and may be data converted from the user's movement information to reflect the character's characteristics. Character movement information may be generated for each plurality of users, that is, for each character corresponding to each of the plurality of users. As an example, the motion information conversion unit 130 may receive first user motion information and first character information, and generate first character motion information indicating the movement of the first character.

The character motion recommendation unit 140 may provide first character recommendation motion information to the first user based on the first character motion information and the first character preference motion information of the second user, which is the other user.

The second user's first character preference movement information may be generated in the preference information collection unit 150. The preference information collection unit 150 may collect preference movement information for a specific character for each user. The process of generating recommended movement information for the first character for the second user in the preference information collection unit 150 and providing the recommended movement information for the first character to the first user will be described in more detail later.

The content generator 160 may generate a character motion sequence based on character correction movement information and character information. Here, the character information corresponds to the character's gender, age, appearance, size, skin color, clothing, etc., and the character's motion sequence is created through this character information and character compensation movement information.

Additionally, the content creation unit 160 generates a character shading sequence by considering character information, character correction movement information, and background information. Background information refers to information for configuring the background of a virtual space in which a preset character moves. The content generator 160 may derive light source information through a predetermined calculation formula based on information about the virtual space (e.g., amount of sunlight, weather, amount of clouds, etc.) included in the background information, and the derived light source information. A shading sequence can be generated based on .

Here, shading refers to the operation of changing the illuminance of the surface of an object depending on the distance and angle between the object and the light source. In the shading sequence, the surface of each polygon that makes up the character is shaded according to the location, brightness, and color of the light source according to the background information, thereby expressing three-dimensionality and realism for the object.

As in the above-described process, the character on which graphic processing of the character's movements and appearance has been performed is configured to output a voice according to the character. The content creation unit 160 may create a character and generate digital content based on the character's motion sequence, character shading sequence, and character voice profile.

Digital content is output through the first user's user device and the second user's user device, respectively, and the first user and the second user interact through the first character and the second character. To the first user, the operation screen of the first character, in which digital content is output based on the first character, is output through the first user's user device, and to the second user, the digital content is output based on the second character. The character's action screen may be output through the second user's user device. Not only the first character but also the second character being interacted with may be displayed on the first character's action screen.

During this interaction process between the first user and the second user through digital content, the character movement recommendation unit 140 provides a recommendation based on the first character movement information and the first character preference movement information of the second user, the other user. 1 First character recommended movement information may be provided to the user. This first character preference movement information of the second user may be generated in the preference information collection unit 150.

Hereinafter, the process of collecting character preference information performed by the preference information collection unit 150 and the process of providing the first character recommendation movement information to the first user by the character movement recommendation unit 140 will be described in detail. In particular, the following description describes a process of collecting the first character preference information of the second user and providing the first character recommended movement information to the first user, but the embodiment of the present invention is not limited thereto, and other users Of course, the same can be applied to characters.

Figure 7 is a block diagram showing the main configuration of a preference information collection unit according to an embodiment. FIG. 8 exemplarily shows a situation in which a second user image is generated by photographing a second user using the first digital content. Figure 9 exemplarily shows input data and output data of an emotion analysis model. Figure 10 exemplarily shows an analysis image determined according to emotional information of a second user. Figure 11 is an example diagram for explaining the process of determining a motion recommendation point. Figure 12 is an example screen in which a posture support auxiliary screen is output on the first user operation screen. 13 and 14 are exemplary screens in which first character recommended movement information is provided through a posture support auxiliary screen.

Referring to FIG. 7, the preference information collection unit 150 includes a user image collection module 151, a user emotion analysis module 152, an analysis image determination module 153, a character preference movement determination module 154, and a movement recommendation timing. Includes a decision module 155.

The user image collection module 151 may collect user images captured by users using digital content. These user videos correspond to user feedback data for analyzing user preference information.

In an embodiment, in a situation where digital content is output, the user device 200 may monitor the user using the digital content in real time and may generate a user image by photographing the user through the image sensor 220A. However, it is not limited to this, and in some embodiments, the server 100 may include digital content for each character to collect the user's character preference information, and may provide the corresponding digital content to the user to collect the user's character preference information. A user image, which is the user's feedback data, can be generated. The user image collection module 151 may receive user images from the user devices 200 of a plurality of users.

Referring to FIG. 8, the second user can use the first digital content output through the display unit. Here, the first digital content corresponds to digital content including the first character, and may be content prepared to collect the second user's preference information for the first character. The second user may experience various emotional changes depending on the content displayed in the first digital content. Positive emotions, such as joy, love, confidence, respect, or security, and negative emotions, such as shame, fear, anger, hatred, sadness, or anxiety, may appear. This change in the second user's emotion is caused by a change in the first digital content, particularly the first character, and may be accompanied by a change in the second user's facial expression and pose. The second user image generated through the image sensor 220A of the user device 200 may include changes in the second user's facial expression and pose. That is, the user's preferred movement information can be determined through the second user's emotions confirmed through the second user image.

The user emotion analysis module 152 may determine the user's emotion class by analyzing changes in the user's emotion in the user's image using digital content. The user emotion analysis module 152 may include a pre-learned emotion analysis model and may analyze user images through the emotion analysis model.

Referring to FIG. 9, the emotion analysis model may determine the emotion class of the second user by analyzing the change in facial expression and pose of the second user for each frame image in the second user image. The second user image may be divided into a plurality of frame images according to the frame, and may be sequentially input to the emotion analysis model. The emotion class of the second user is determined in response to each sequentially input frame image. Frame images with determined emotion classes are collected to form emotional information.

Here, the emotion class corresponds to positive emotion or negative emotion. In embodiments, a positive emotion may mean that the second user is in a state of joy, joy, love, confidence, respect, or stability, and a negative emotion may mean that the second user is bored, annoyed, ashamed, fearful, angry, hateful, or sad. Or it may mean being in a state of anxiety. For each frame image, it is determined whether the emotion of the second user using the digital content is a positive emotion or a negative emotion.

In embodiments, the emotion analysis model may be built through a deep learning-based learning model. For example, sentiment analysis models include Deep Feedforward Network (DFN), Convolutional Neural Network (CNN), Graph Neural Network (GNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Support vector machine (SVM), Artificial Neural Network (ANN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), Deep Residual Network (DRN), Generative Adversarial Network (GAN), Graph Convolutional Network (GCN), and Spiking Neural Network (SNN). It may be constructed using at least one of the following, but is not limited to this.

In some embodiments, the emotion analysis model may be comprised of a convolutional neural network-based image analysis model. In particular, the emotion analysis model may be composed of a one-stage detection network suitable for object detection among machine learning models based on a convolutional neural network. Here, the one-stage detection network refers to a detection network in which object detection (Regional Proposal) and class classification are performed simultaneously. In other words, the task of finding the object boundary box and classifying the class can be performed simultaneously at the final output stage of the network.

Illustratively, the emotion analysis model may be composed of a one-stage detection network such as a Yolo (You Only Look Once) network or CenterNet, but is not limited thereto. In an embodiment, the emotion analysis model is configured with a Yolo network and is trained to detect at least one of the main body part and the main part of the user's face, and the emotion of the user based on changes in the detected main body part and the main part of the face. The class can be determined. As an example, the emotion analysis model is trained to detect at least one of eyes, mouth, entire face, head, hand, leg, shoulder, and entire body, and can classify the user's emotion class according to changes in the detected part. .

In an embodiment, the emotion analysis model identifies at least one of the main part of the face and the main part of the body of the second user in the second user image using YOLO (You only look once), a real-time object detection recognition model, and identifies Configured to determine the emotion class of the second user as a positive emotion or negative emotion based on a change in at least one of the main parts of the face and the main part of the body, and output the confidence value for the determined class together with the determined class. It can be. As shown in Figure 10, a positive emotion class and confidence value determined through emotion analysis of the second user may be assigned.

The user emotion analysis module 152 determines the emotion class of the second user for each frame image, and can configure the emotion information of the second user by arranging a plurality of frame images to which emotion classes are assigned for each frame image according to time. there is. In an embodiment, each of the plurality of frame images may be configured to further include a determined emotion class and a confidence value for the emotion class.

The analysis image determination module 153 may determine an analysis image from the user's emotional information. The user's emotional information may be composed of a plurality of frame images determined to be positive emotion classes and the remaining frame images determined to be negative emotion classes. As an example, the analysis image determination module 153 may select an analysis image from the emotional information of the second user. The analysis image determination module 153 may primarily extract a frame image whose class is determined to be a positive emotion from the second user's emotion information, and secondarily extract a frame image given a confidence value greater than a preset threshold. Thus, at least one analysis image can be determined. The second user included in the image of FIG. 10 has a confidence value of 95 for positive emotion, and the image may be determined to be an analysis image.

The character preference movement determination module 154 may determine movement information of the first character in the first digital content corresponding to the output time of the analysis image. The character preference movement determination module 154 may collect movement information of the first character from the first digital content corresponding to the output time of the analysis image and generate preference movement information of the first character of the second user. When the analysis image determination module 153 determines that there are multiple analysis images, the character preference movement determination module 154 collects movement information of the first character corresponding to each of the plurality of analysis images.

The motion recommendation timing determination module 155 may determine the motion recommendation timing based on the emotion information of the second user. The second user's emotion information may be composed of a plurality of frame images to which an emotion class is assigned to each frame image, arranged according to time. The motion recommendation time determination module 155 may determine a motion recommendation time by identifying a change in the user's emotion from the second user's emotional information.

The motion recommendation timing determination module 155 may determine the timing at which a frame image corresponding to a positive emotion class changes to a frame image corresponding to a negative emotion class as the motion recommendation timing. If appropriate stimulation is provided to the second user at the point when the second user becomes bored and loses concentration in interacting with digital content, the second user can continue to maintain concentration in the interaction and feel less bored. . The motion recommendation time determination module 155 may analyze a change in the emotion of the second user, in particular, a change in emotion from a positive emotion to a negative emotion, from the emotion information of the second user, and determine the time point as the motion recommendation time. . Referring to FIG. 11, it can be seen that the 8th and 9th frame images are classified as positive emotions, and the subsequent 10th and 11th frame images are classified as negative emotions. The motion recommendation timing determination module 155 may determine the timing at which the 10th frame image is output as the motion recommendation timing. The determined motion recommendation time point may correspond to at least one piece of time information elapsed from the time when the first digital content and the second user start interacting. For example, time information such as 15 minutes 10 seconds, 24 minutes 25 seconds, or 30 minutes 45 seconds from the start point may be determined as the movement recommendation point.

The character preference movement determination module 154 generates the collected movement information of the first character as the first character preference movement information of the second user, and the second user Movements of the preferred first character may be provided by the first user to assist the second user in remaining focused on interacting with the digital content.

The character motion recommendation unit 140 may provide first character recommendation motion information to the first user based on the first character preference motion information of the second user. The character movement recommendation unit 140 monitors the first user to implement the first character recommendation movement preferred by the second user while the first character and the second character are interacting in the digital content. You can support by providing information. The first character recommended movement information may be provided through a posture support auxiliary screen that is distinct from the first character's motion screen.

The character movement recommendation unit 140 may output the first character recommendation movement information to the first user through the posture support auxiliary screen. The posture support auxiliary screen may be provided as a screen that is separate from the first character's action screen displayed to the first user. The posture support auxiliary screen may be provided as PIP (Picture In Picture), as shown in FIG. 12, but is not limited thereto.

The character motion recommendation unit 140 may output first character recommendation motion information to the first user based on the motion recommendation time point. For example, if the motion recommendation point is determined by time information such as 15 minutes 10 seconds, 24 minutes 25 seconds, or 30 minutes 45 seconds, the character movement recommendation unit 140 recommends the interaction between the first user and the second user. The first character recommended movement information may be provided to the first user at 15 minutes 10 seconds, 24 minutes 25 seconds, and 30 minutes 45 seconds, respectively, calculated from the start time. The character movement recommendation unit 140 may perform a recommendation so that the first character movement preferred by the second user is provided to the first user at a time when the second user may feel bored.

Referring to FIG. 12, the posture support auxiliary screen may be activated along with a guidance comment at the time of movement recommendation to provide first character recommended movement information to the first user. The first character recommended movement information displayed on the posture support auxiliary screen may include guide movement information according to the second user's first character preferred movement information. The guide movement information corresponds to an image schematically shown to guide the first user's posture so that the first user can take the movement of the first character preferred by the second user. An image of a person assuming a specific posture may be provided through guide movement information to assist the first user in assuming the corresponding posture.

The guide movement information may be configured according to the first character preference movement information of the second user. When the second user's first character preference motion information consists of a plurality of pieces, the character motion recommendation unit 140 may determine a different degree of motion change in the current first user's motion information among the plurality of first character preference motion information of the second user. First character preference motion information that is relatively smaller than the first character preference motion information may be determined, and guide motion information corresponding to the determined first character preference motion information may be generated and provided as the first character recommendation motion information.

Here, the character movement recommendation unit 140 may display the first user movement information along with the guide movement information on the posture support auxiliary screen. Referring to FIG. 13, it can be seen that the first user movement information is further displayed along with the guide movement information on the posture support auxiliary screen. In the posture support auxiliary screen, guide movement information may be displayed as a dotted line and first user movement information may be displayed as a solid line, but the information is not limited thereto.

The first user can check his or her own movement and guide movement information through the posture support auxiliary screen, and can change his or her movement to assume a posture corresponding to the guide movement information. That is, the first user can change his or her posture to correspond to the guide motion information, and thus the first user's motion information and the guide motion information can match.

Referring to FIG. 14, it can be seen that the first user's motion information and the guide motion information are matched. As the motion information of the first user matches the guide motion information, the motion information of the first character corresponding to the motion information of the first user may change, and the first character may perform the movement or pose preferred by the second user. I get drunk. Accordingly, the second user can maintain concentration on interaction with the first character.

In an embodiment, the character motion recommendation unit 140 determines whether the first user motion information and the guide motion information match, and if the first user motion information and the guide motion information match, displays the posture support auxiliary screen to the first user. You may terminate providing it to . In other words, the character movement recommendation unit 140 may control the first user to provide only the first character's movement screen after the first user naturally takes the first character's movement preferred by the second user. .

The digital content providing server that supports the other party's customized character movement control according to an embodiment of the present invention controls the character to correspond to the user's movement, and recommends character motion according to the preference information of the other user who also uses the digital content so that the user can It can support the other user's preferred character actions, allowing the other user to continue to focus on the interaction on digital content, and can arouse the other user's interest and interest, thereby increasing interaction in digital content. The effect can be maximized. In addition, the user's preference information and movement recommendation timing can be determined by analyzing the user's captured video, and the other user's preferred character's movements can be supported at an appropriate time. In addition, the user environment according to the embodiment supports the user to take the other user's preferred character posture by following the guide motion, so that the user can more easily take the other user's preferred character posture.

Hereinafter, with reference to FIG. 15, a method of providing digital content that supports character movement control customized to the other party according to some embodiments of the present invention will be described. The method of providing digital content according to the embodiment is a method performed on the server of the system according to FIGS. 1 to 14, and parts that overlap with the above-described embodiment are simplified or omitted.

Figure 15 is a flowchart of a method of providing digital content that supports character movement control customized to the other party according to some embodiments of the present invention.

Referring to FIG. 15, a method of providing digital content supporting a counterpart's customized character movement control according to some embodiments of the present invention includes receiving first user movement information of a first user (S100), first user movement information Converting the first character movement information (S110), implementing the movement of the first character in the virtual space based on the first character movement information and the first character information (S120), and the second user's It includes providing first character recommended motion information to the first user based on first character preference motion information (S130).

In an embodiment, the method further includes collecting first character preference movement information of a second user, wherein the first character preference movement information of the second user is a second user information for the first digital content including the first character. The user's feedback data may be collected, and the second user's first character preference movement information may be generated based on the collected feedback data.

In addition, step S130 includes providing the first character recommended movement information to the first user through a posture support auxiliary screen that is distinct from the first character's action screen, and the first character recommended movement information is It may include guide movement information according to the second user's first character preference movement information.

In addition, step S130 displays the guide motion information and the first user motion information together on the posture support auxiliary screen, determines whether the first user motion information and the guide motion information match, and determines whether the first user motion information matches the guide motion information. If the user movement information matches the guide movement information, providing the posture support auxiliary screen to the first user may be terminated.

In addition, in step S130, when the first character preference motion information of the second user consists of a plurality of first character preference motion information, the degree of motion change in the current first user's motion information among the plurality of first character preference motion information of the second user is different. It may include determining first character preference motion information that is relatively smaller than the first character preference motion, generating guide motion information corresponding to the determined first character preference motion information, and providing the guide motion information as the first character recommendation motion information. there is.

In an embodiment, collecting first character preference movement information of a second user may include collecting a second user image captured by a second user using the first digital content as the feedback data; Generating emotional information of the second user from the second user image using a deep learning-based pre-trained emotional analysis model; determining at least one analysis image from the emotional information of the second user; collecting movement information of a first character from the first digital content corresponding to the output time of the analysis image and generating preferred movement information of the first character of the second user; and determining a motion recommendation point based on the emotional information of the second user.

In addition, the emotion analysis model is a classification model based on a convolutional neural network learned to determine the emotion class of the second user for each frame image by analyzing the movement change of the second user in the second user image for each frame, An emotion class may correspond to a positive emotion or a negative emotion.

In addition, the emotion information of the second user is composed of a plurality of frame images to which an emotion class is assigned for each frame image arranged according to time, and the step of determining the motion recommendation point is performed by selecting the emotion information of the second user. 2 By identifying changes in the user's emotions, it is possible to determine when to recommend movement.

The emotion analysis model consists of a Yolo (You Only Look Once) network, and the emotion analysis model identifies at least one of the main part of the second user's face and the main part of the body in the second user image, and the identified Based on a change in at least one of the main parts of the face and the main part of the body, the second user's emotion class can be determined as a positive emotion or a negative emotion, and a confidence value for the determined class can also be output. In addition, the step of determining the analysis image includes first extracting a frame image whose class is determined to be a positive emotion, extracting a frame image with a confidence value greater than a preset threshold from the extracted frame image, and analyzing the at least one analysis image. can be decided.

The method of providing digital content that supports control of the opponent's customized character movement according to the embodiment may also be implemented in the form of a computer-readable medium that stores commands and data executable by a computer. At this time, instructions and data can be stored in the form of program code, and when executed by a processor, they can generate a certain program module and perform a certain operation. Additionally, computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may be computer recording media, which are volatile and non-volatile implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. It can include both volatile, removable and non-removable media. For example, computer recording media may be magnetic storage media such as HDDs and SSDs, optical recording media such as CDs, DVDs, and Blu-ray discs, or memory included in servers accessible through a network.

Additionally, the method of providing digital content that supports the other party's customized character movement control according to the embodiment may be implemented as a computer program (or computer program product) including instructions executable by a computer. A computer program includes programmable machine instructions processed by a processor and may be implemented in a high-level programming language, object-oriented programming language, assembly language, or machine language. . Additionally, the computer program may be recorded on a tangible computer-readable recording medium (eg, memory, hard disk, magnetic/optical medium, or solid-state drive (SSD)).

Accordingly, the method of providing digital content supporting the opponent's customized character movement control according to the embodiment can be implemented by executing the above-described computer program by a computing device. The computing device may include at least some of a processor, memory, a storage device, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to a low-speed bus and a storage device. Each of these components is connected to one another using various buses and may be mounted on a common motherboard or in some other suitable manner.

Here, the processor can process instructions within the computing device, such as displaying graphical information to provide a graphic user interface (GUI) on an external input or output device, such as a display connected to a high-speed interface. These may include instructions stored in memory or a storage device. In other embodiments, multiple processors and/or multiple buses may be utilized along with multiple memories and memory types as appropriate. Additionally, the processor may be implemented as a chipset consisting of chips including multiple independent analog and/or digital processors.

Memory also stores information within a computing device. In one example, memory may be comprised of volatile memory units or sets thereof. As another example, memory may consist of non-volatile memory units or sets thereof. The memory may also be another type of computer-readable medium, such as a magnetic or optical disk.

And the storage device can provide a large amount of storage space to the computing device. A storage device may be a computer-readable medium or a configuration that includes such media, and may include, for example, devices or other components within a storage area network (SAN), such as a floppy disk device, a hard disk device, an optical disk device, Or it may be a tape device, flash memory, or other similar semiconductor memory device or device array.

The above description is merely an illustrative explanation of the technical idea of the present embodiment, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these examples. The scope of protection of this embodiment should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this embodiment.

Claims (10)

A server that provides digital content in which a first character corresponding to a first user and a second character corresponding to a second user interact in a virtual space, the server comprising:
a user data collection unit that receives first user movement information of the first user;
a character database storing first character information of the first character;
a motion information conversion unit that converts the first user motion information into the first character motion information;
a content generator that implements movement of the first character in the virtual space based on the first character movement information and the first character information;
a preference information collection unit that collects a second user's feedback data on the first digital content including the first character and generates the second user's preference movement information for the first character based on the collected feedback data; and
Comprising a character motion recommendation unit that provides first character recommendation motion information to the first user based on the first character preference motion information of the second user.
A digital content provision server that supports opponent-customized character movement control.
According to claim 1,
The character movement recommendation unit provides the first character recommendation movement information to the first user through a posture support auxiliary screen that is distinct from the movement screen of the first character,
The first character recommended movement information includes guide movement information according to the first character preferred movement information of the second user.
A digital content provision server that supports opponent-customized character movement control.
According to clause 2,
The character movement recommendation unit displays the guide movement information and the first user movement information together on the posture support auxiliary screen,
The character motion recommendation unit determines whether the first user motion information and the guide motion information match, and when the first user motion information and the guide motion information match, displays the posture support auxiliary screen to the first user. ending what is provided
A digital content provision server that supports opponent-customized character movement control.
According to clause 2,
The character movement recommendation unit,
When the first character preference movement information of the second user consists of a plurality of first character preference movement information, the degree of movement change in the current first user's movement information among the plurality of first character preference movement information of the second user is relatively greater than that of the other first character preference movements. The enemy determines the first character's preferred movement information,
Generating guide movement information corresponding to the determined first character preference movement information and providing it as the first character recommended movement information
A digital content provision server that supports opponent-customized character movement control.
According to claim 1,
The preference information collection department
a user image collection module that collects a second user image captured by a second user using the first digital content as the feedback data;
a user emotion analysis module that generates emotion information of the second user from the second user image using a deep learning-based pre-trained emotion analysis model;
an analysis image determination module that determines at least one analysis image from the emotional information of the second user;
a character preference motion determination module that collects motion information of the first character from the first digital content corresponding to the output time of the analysis image and generates first character preference motion information of the second user; and
Comprising a motion recommendation timing determination module that determines a movement recommendation timing based on the emotion information of the second user.
A digital content provision server that supports opponent-customized character movement control.
According to clause 5,
The emotion analysis model is a convolutional neural network-based classification model learned to determine the emotion class of the second user for each frame image by analyzing the change in movement of the second user in the second user image for each frame,
The emotion class corresponds to positive or negative emotions.
A digital content provision server that supports opponent-customized character movement control.
According to clause 6,
The second user's emotion information is composed of a plurality of frame images to which an emotion class is assigned to each frame image, arranged according to time,
The motion recommendation timing determination module determines a movement recommendation timing by identifying a change in the emotion of the second user from the emotional information of the second user.
A digital content provision server that supports opponent-customized character movement control.
According to clause 6,
The sentiment analysis model consists of the Yolo (You Only Look Once) network,
The emotion analysis model is,
Identifying at least one of the main part of the second user's face and the main part of the body in the second user image, and creating an emotion class of the second user based on a change in at least one of the identified main part of the face and the main part of the body is determined as a positive or negative emotion, and the confidence value for the determined class is output together.
The analysis image determination module first extracts a frame image whose class is determined to be a positive emotion, and determines the at least one analysis image by extracting a frame image with a confidence value greater than a preset threshold from the extracted frame image.
A digital content provision server that supports opponent-customized character movement control.
A method of providing digital content in which a first character corresponding to a first user and a second character corresponding to a second user interact in a virtual space, the method comprising:
Receiving first user movement information of the first user;
converting the first user motion information into the first character motion information;
implementing movement of the first character in the virtual space based on the first character movement information and the first character information; and
Providing first character recommended movement information to the first user based on the first character preferred movement information of the second user,
The first character preference movement information of the second user is generated by collecting feedback data of the second user about the first digital content including the first character and based on the collected feedback data.
A method of providing digital content that supports the opponent's customized character movement control.
a first user device generating first user movement information of a first user;
a second user device generating second user movement information of a second user; and
A server that generates digital content in which a first character corresponding to the first user and a second character corresponding to the second user interact in a virtual space,
The server is,
a user data collection unit that receives first user movement information of the first user;
a character database storing first character information of the first character;
a motion information conversion unit that converts the first user motion information into the first character motion information;
a content generator that implements movement of the first character in the virtual space based on the first character movement information and the first character information;
a preference information collection unit that collects a second user's feedback data on the first digital content including the first character and generates the second user's preference movement information for the first character based on the collected feedback data; and
Comprising a character motion recommendation unit that provides first character recommendation motion information to the first user based on the first character preference motion information of the second user.
A digital content provision system that supports opponent-customized character movement control.

Images