KR102565196B1 - Method and system for providing digital human in virtual space - Google Patents

Abstract

A method for providing a digital human in a virtual space according to the present invention comprises the steps of: specifying a space of interest in a virtual space including a plurality of spaces; acquiring character information of a person associated with the space of interest; generating, by an artificial neural network, a digital human corresponding to the person on the basis of the acquired character information; and providing, in response to a user accessing the virtual space, the generated digital human to the space of interest in the virtual space, wherein the artificial neural network learns a method for generating the digital human with extracted character characteristics by receiving character information of a person associated with a specific space and extracting character characteristics from the received character information. According to the present invention, a service associated with a virtual space can be provided through a digital human suitable for spatial characteristics.

Description

Method and system for providing digital human in virtual space {METHOD AND SYSTEM FOR PROVIDING DIGITAL HUMAN IN VIRTUAL SPACE}

The present invention relates to a method and system for providing a digital human in a virtual space. More specifically, the present invention relates to a method and system for providing a digital human having extracted character characteristics through a virtual space by extracting character characteristics from information on a specific person using an artificial neural network.

As technology develops, the spread of electronic devices (eg, smart phones, tablet PCs, automated devices, etc.) has become popular, and accordingly, dependence on electronic devices is gradually increasing in many parts of daily life.

In particular, with the development of display technology, users can view not only flat images but also 3D images and 3D spaces through various means. Accordingly, virtual reality and augmented reality services are being provided to provide users with a 3D visual user experience. Experiencing a three-dimensional space has become a new trend in user experience, and users can simulate in virtual space, receive visual and other sensory experiences, and interact with virtual images.

Furthermore, as a representative service that encompasses all of the above concepts and provides a mixed reality environment, there is a metaverse. This metaverse is a compound word of 'meta', which means processing and abstraction, and 'universe', which means the real world, and means the connection between the three-dimensional virtual space and the real space. The metaverse is a more advanced concept than the existing virtual reality environment, and provides the above-described mixed reality environment in which virtual worlds such as the web and the Internet are absorbed into the real world.

The metaverse can be infinitely expanded according to user participation, and characters, environments, and spaces can be freely selected. For example, a user may create a terrain and place desired natural objects and facilities on the terrain.

On the other hand, a digital human is a method of graphical representation of a user that other users in the virtual space can see and interact with, and a user in the real space uses the digital human to interact with the digital humans of other users in the virtual space, Various economic activities, cultural activities, educational activities, etc. can be performed in the corresponding virtual space.

The present invention obtains person information of a real person related to a specific space of interest in a virtual space, and a human neural network creates a digital human corresponding to the real person using the person information, thereby providing the created digital human through the virtual space. It relates to methods and systems.

In order to solve the problems described above, a method for providing a digital human according to the present invention includes the steps of specifying a space of interest in the virtual space including a plurality of spaces, obtaining character information of a person related to the space of interest, and artificial Generating, by a neural network, the digital human corresponding to the person based on the acquired person information, and in response to a user accessing the virtual space, the generated digital human is selected from the space of interest in the virtual space. A method for generating the digital human having the extracted character characteristics by receiving character information of a person related to a specific space and extracting character characteristics from the received character information. learning can be characterized.

Furthermore, the person information includes image information and voice information about the person, and the artificial neural network generates the digital human to have at least a part of an appearance extracted from the image information and a voice extracted from the voice information. It can be characterized by learning methods.

Furthermore, at least some of the person information may be obtained through an information obtaining unit disposed in the real space with respect to the person in the real space corresponding to the space of interest.

Furthermore, the person information further includes image information including motions of the person in the real space, and the artificial neural network controls the digital human operating according to a gesture extracted from the image information. It can be characterized by learning how to generate.

Furthermore, the person information includes work information of the person, and the obtaining of the person information including the image information may include performing a plurality of motions of the person included in the image information based on a predetermined criterion. , identifying an effective motion corresponding to the work information of the person, and acquiring image information including the valid motion as the person information, wherein the artificial neural network includes a gesture extracted from the valid motion It may be characterized in that learning a method of generating the digital human operating according to.

Furthermore, the method for providing a digital human further includes the step of identifying the speech content of the person from the voice information, wherein the artificial neural network extracts the speech style of the person from the speech content, thereby retrieving the speech content of the person in the virtual space. It may be characterized by learning a method of generating the digital human that speaks according to a speech style.

Furthermore, providing the digital human to the space of interest may include receiving a request of the user in the virtual space, and sending the digital human that responds to the request with the person characteristics to the interest. A step of providing the space may be further included.

Furthermore, the artificial neural network extracts the personality type of the person from at least one of the image information and the voice information, and generates the digital human having at least a part of the appearance and voice corresponding to the extracted personality type. It can be characterized by learning how to do it.

Meanwhile, a system for providing a digital human in a virtual space according to the present invention includes a control unit, and the control unit specifies a space of interest in the virtual space including a plurality of spaces based on a user input, and the space of interest Person information of a related person is acquired, and an artificial neural network generates the digital human corresponding to the person based on the acquired person information, and in response to the user accessing the virtual space, the generated A digital human is provided to the space of interest among the virtual spaces, and the artificial neural network receives character information of a person related to a specific space and extracts character characteristics from the received character information. It may be characterized as learning how to create a human.

Meanwhile, as a program that is executed by one or more processes in an electronic device and can be stored in a computer-readable medium, the program comprises: specifying a space of interest in the virtual space including a plurality of spaces; Acquiring person information of a person related to the person, generating, by an artificial neural network, the digital human corresponding to the person based on the acquired person information, and in response to a user accessing the virtual space, and instructions for performing a step of providing the generated digital human to the space of interest among the virtual spaces, wherein the artificial neural network receives character information of a person related to a specific space, and extracts character characteristics from the received character information. By doing so, it may be characterized in that a method of generating the digital human having the extracted character characteristics is learned.

According to various embodiments of the present invention, a digital human providing method and system generates and provides a digital human corresponding to a person related to a specific space of interest among virtual spaces including a plurality of spaces, and provides the digital human through a digital human suitable for spatial characteristics. Services related to virtual space can be provided.

In addition, a digital human providing method and system according to various embodiments of the present invention extracts character characteristics using character information of a person related to a space of interest, generates and provides a digital human to have the extracted character characteristics, It is possible to minimize a sense of difference with a person and provide a user experience similar to a real space.

Furthermore, according to various embodiments of the present invention, a method and system for providing a digital human identify only gestures related to work in a specific space in a virtual space among a plurality of gestures performed by a person in a real space and learn the artificial neural network, A digital human reflecting both spatial characteristics and character characteristics can be provided, and through this, a user experience similar to a real experience can be provided.

1 shows a digital human providing system according to the present invention.
2 is a flowchart illustrating a method for providing a digital human according to the present invention.
3 illustrates a configuration for specifying a space of interest among virtual spaces including a plurality of spaces according to an embodiment.
4 illustrates a configuration for obtaining character information of a specific person in a real space using an information acquisition unit according to the present invention.
5 illustrates an artificial neural network learning to create a digital human using person information acquired through an information acquisition unit according to the present invention.
6 illustrates a configuration for generating a digital human having an external appearance extracted from image information by using image information among person information according to the present invention.
7 illustrates a configuration for generating a digital human having a voice extracted from voice information by using voice information among person information according to the present invention.
8A and 8B illustrate a configuration for generating a digital human operating according to at least a part of gestures extracted from the image information using image information among person information according to the present invention.
9 illustrates a configuration for generating a digital human operating according to the extracted speech style and personality type by extracting a speech style and personality type from person information according to the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same reference numerals will be assigned to the same or similar components regardless of reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

1 shows a digital human providing system according to the present invention.

Referring to FIG. 1 , a system for providing a digital human according to an embodiment 100 obtains character information related to a specific space among virtual spaces, and the artificial neural network 150 creates digital data using the acquired character information. It may be configured to provide the human 10 to the user terminal 200 through the virtual space 30 .

At this time, the user terminal 200 may be referred to as various types of electronic devices. For example, the user terminal 200 may refer to at least one of a smart phone, a tablet PC, an e-book reader, a desktop PC, a laptop PC, a home appliance, or a wearable device, but is not limited thereto. However, for convenience of description, the user terminal 200 will be described with reference to a head mounted display (HMD) providing a virtual reality environment among wearable devices.

In addition, the digital human 10 may be understood as a virtual character (or avatar) provided in a specific space of interest 170 of the virtual space 30 . That is, of course, the digital human 10 may be named a character or an avatar.

More specifically, the digital human 10 may be referred to as a virtual object that provides a response 102 to a user request 101 by being provided in the space of interest 170 of the virtual space 30 .

For example, the digital human providing system 100 provides the digital human 10 to a specific interest space 170 of the virtual space 30, so that the digital human 10 is requested by a user within the virtual space 30. A guide or guide corresponding to (101) may be provided.

Accordingly, the digital human provision system 100 according to the present invention can generate a digital human and provide a virtual space including the created digital human to the user through the user terminal 200 .

To this end, the digital human providing system 100 according to an embodiment includes at least one of a communication unit 110, a storage unit 120, an information acquisition unit 130, a control unit 140, and an artificial neural network 150. can do.

Meanwhile, the digital human provision system 100 according to the present invention may be implemented as an application or software. The digital human providing system 100 implemented as an application in this way can be downloaded through a program (eg, Play Store, App Store) capable of downloading applications on an electronic device, or implemented through an initial installation program on an electronic device. can In this case, the communication unit 110, the storage unit 120, the information acquisition unit 130, and the control unit 140 according to the present invention may be utilized as components of an electronic device.

In addition, the digital human providing system 100 according to the present invention may be provided through a web page accessed through the user terminal 200 .

The communication unit 110 according to the present invention may be configured to communicate with at least one of the user terminal 200 and an external server (not shown) by wire or wirelessly. The communication unit 110 transmits a virtual space (or metaverse) service including the digital human 10 to the user terminal 200 using wired or wireless communication, or receives person information about a specific person from an external server. can do.

At this time, the communication unit 110 may support various communication methods according to a communication standard of at least one of the communicating user terminal 200 and an external server.

For example, the communication unit 110 may include Wireless LAN (WLAN), Wireless-Fidelity (Wi-Fi), Wireless Fidelity (Wi-Fi) Direct, Digital Living Network Alliance (DLNA), Wireless Broadband (WiBro), WiMAX ( World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), 5th Generation Mobile Telecommunication (5G) , Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal) It may be configured to communicate with at least one of the user terminal 200 and an external server using at least one of Serial Bus) technologies.

Also, the storage unit 120 according to the present invention may be configured to store person information acquired from the information acquisition unit 130 or an external server (not shown). The storage unit 120 according to an embodiment may be provided in the digital human providing system 100 itself. According to another embodiment, at least a portion of the storage unit 120 may be referred to as at least one of a cloud server and a database of external servers. That is, the storage unit 120 is referred to as a space in which person information for a specific person is stored according to an embodiment, and it may be understood that there is no restriction on physical space.

In this case, the person information may include at least one of i) image information of a specific person, ii) audio information, iii) video information including motions of the person, and iv) conversation information between the person and the digital human. .

Meanwhile, the information acquisition unit 130 according to the present invention may be understood as a means for obtaining person information on a specific person. More specifically, the controller 140 may obtain character information of a person related to the space of interest 170 in the virtual space 30 by using the information acquisition unit 130 .

The information acquisition unit 130 according to an embodiment may be understood as a means for obtaining person information of a specific person in a real space corresponding to the space of interest 170 . For example, the information acquisition unit 130 may be understood as a camera, camcorder, or CCTV (closed-circuit television) for acquiring image or video information of a person in a real space, but is not limited thereto, and images or It can be understood as various means capable of acquiring image information.

For another example, the information acquisition unit 130 may be understood as a recorder for acquiring voice information of a person in a real space corresponding to the space of interest 170, but is not limited thereto, and voice information is not limited thereto. It can be understood as various means for obtaining information.

The information acquisition unit 130 according to another embodiment is for obtaining person information input from a manager and may be a medium between the manager and the digital human providing system 100 . More specifically, the information acquisition unit 130 may refer to an input means for receiving person information on a specific person from a manager.

At this time, the information acquisition unit 130 may be referred to as various types of input members. The information acquisition unit 130 according to an embodiment may include a mechanical input means (or a mechanical key, for example, a mouse, a joy stick, a physical button, a dome switch, jog wheel, jog switch, etc.) and at least one of touch input means. According to another embodiment, the touch input means is referred to as a virtual key, soft key, or visual key displayed on the touch screen through software processing, or other than the touch screen. It may be referred to as a touch key disposed in a part of. On the other hand, the virtual key or visual key can be displayed on the touch screen while having various forms, for example, graphic (graphic), text (text), icon (icon), video (video) or these may be made of a combination of

Next, the controller 140 may control overall operations of the digital human providing system 100 . According to an embodiment, the controller 140 may process data related to person information input through the information acquisition unit 130 or received from an external server through the communication unit 110 . In addition, by inputting the obtained person information to the artificial neural network 150, the artificial neural network 150 can be controlled to generate the digital human 10 or learn how to generate the digital human 10. In addition, the control unit 140 may store acquired person information and/or data received from the user terminal 200 in the storage unit 120 .

In addition, the control unit 140 transmits data related to the digital human 10 and the virtual space 30 to the user terminal 200, thereby including the digital human 10 through a display provided in the user terminal 200. It is possible to perform control so that the virtual space to be output. At this time, the type of display provided in the user terminal 200 is not limited.

In addition, the artificial neural network 150 according to the present invention can be understood as a recognition model that imitates the computational capability of a biological system by using a plurality of artificial neurons connected by connection lines. The artificial neural network 150 uses artificial neurons that simplify the functions of biological neurons, and the artificial neurons may be interconnected through a connection line having a connection weight. The connection weight, which is a parameter of the artificial neural network 150, is a value of a connection line and may indicate connection strength. The artificial neural network 150 may perform a human cognitive function or learning process through artificial neurons. At this time, the artificial neurons of the artificial neural network 150 may be referred to as nodes.

The artificial neural network 150 may include a plurality of layers. For example, the artificial neural network 150 may include an input layer, one or more hidden layers, and an output layer. The input layer may receive input data for learning of the artificial neural network 150 and pass it to the hidden layer, and the output layer may generate output data of the artificial neural network 150 based on signals received from nodes of the hidden layer. can One or more hidden layers may be positioned between the input layer and the output layer, and may convert input data transmitted through the input layer into values that are easy to predict. Nodes included in the input layer and one or more hidden layers may be connected to each other through a connection line having a connection weight, and nodes included in the hidden layer and the output layer may also be connected to each other through a connection line having a connection weight. The hidden layer may represent a convolution filter or a fully connected layer in a convolution neural network (CNN), or various types of filters or layers grouped based on special functions or characteristics.

Among artificial neural networks, an artificial neural network including a plurality of hidden layers is referred to as a deep neural network, and learning a deep neural network is referred to as deep learning.

According to the present invention, the artificial neural network 150 can be understood as a model trained to extract character characteristics (or personas) from character information through such deep learning and generate a digital human 10 having the extracted character characteristics. can For example, the artificial neural network 150 may be understood as a machine learning model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN). Furthermore, the artificial neural network 150 may be referred to as a machine learning model or a deep learning model.

More specifically, the artificial neural network 150 receives character information of a person related to a specific space of interest 170 in the virtual space 30, extracts character characteristics from the input character information, and extracts at least some of the extracted character characteristics. It can be designed to learn how to create a digital human 10 with

At this time, the artificial neural network 150 may be implemented inside the digital human providing system 100 or implemented in an external device (or server) connected to the digital human providing system 100 through the communication unit 110. That is, a space in which the artificial neural network 150 is implemented may be understood as having no restrictions on a physical space.

Based on the configuration of the digital human providing system 100 reviewed above, the learned artificial neural network creates a digital human using person information, and provides a virtual space service including the created digital human to the user terminal 200. Let us explain in more detail how to provide it to the user through

2 is a flowchart illustrating a method for providing a digital human according to the present invention. 3 illustrates a configuration for specifying a space of interest among virtual spaces including a plurality of spaces according to an embodiment. 4 illustrates a configuration for obtaining character information of a specific person in a real space using an information acquisition unit according to the present invention. 5 illustrates an artificial neural network learning to create a digital human using person information acquired through an information acquisition unit according to the present invention. 6 illustrates a configuration for generating a digital human having an external appearance extracted from image information by using image information among person information according to the present invention. 7 illustrates a configuration for generating a digital human having a voice extracted from voice information by using voice information among person information according to the present invention. 8A and 8B illustrate a configuration for generating a digital human operating according to at least a part of gestures extracted from the image information using image information among person information according to the present invention. 9 illustrates a configuration for generating a digital human operating according to the extracted speech style and personality type by extracting a speech style and personality type from person information according to the present invention.

The digital human providing system 100 according to the present invention acquires person information related to a specific space among virtual spaces, and displays a digital human generated by an artificial neural network using the person information through the virtual space displayed on the user terminal 200. can be provided to the user. However, below, for convenience of description, operations for providing the digital human 10 will be described as being performed by the control unit 140 of the digital human providing system 100 .

The controller 140 according to the present invention may specify a space of interest 170 in the virtual space 30 having a plurality of spaces (S201).

More specifically, the controller 140 may specify a selected space as the space of interest 170 based on a user input for one of the plurality of spaces 301 to 304 included in the virtual space 30 .

As shown in FIG. 3 , the controller 140 according to the present invention may input (eg, touch input or voice speech) to at least one of a plurality of virtual spaces 301 to 304 displayed to the user through the user terminal 200. ), it is possible to specify the space of interest.

According to another embodiment, the controller 140 may specify a space of interest based on a user's physical input to at least one of the plurality of virtual spaces 301 to 304 through the display of the user terminal 200. .

For example, the controller 140 may specify “viewing” as a space of interest in response to a user input for a first virtual space 301 including “viewing” among a plurality of virtual spaces 301 to 304. can

According to another embodiment, the control unit 140 allows the user to enter the virtual space ( 30), one of the plurality of virtual spaces 301 to 304 may be specified as a space of interest.

Furthermore, the controller 140 may specify a person related to the specified space of interest. More specifically, when the space of interest is specified, the controller 140 may specify a person related to the space of interest based on information stored in association with the space of interest.

For example, in response to specifying “City hall” as a space of interest, the controller 140 may determine “mayor” based on information stored in the storage unit 120 in association with the “city hall” space. ” can be specified as a person related to the space of interest.

Person information corresponding to each of a plurality of virtual spaces included in the virtual space may be stored in the storage unit 120 . That is, information about a plurality of virtual spaces and a person corresponding to each of the plurality of virtual spaces may be matched with each other and exist in the storage unit 120 as matching information. In this case, the information on the person corresponding to each of the plurality of virtual spaces may correspond to a person living (ex: working or living) in a real space corresponding to the plurality of virtual spaces. On the other hand, when the person living in the real space is changed, for example, when the first person resigns from a specific real space, or when the term of office expires, a second person, not the first person, lives in the real space. can do. When information on such a person is updated, matching information in the storage unit 120 may also be linked and updated. Meanwhile, since information on a person can be updated through various routes, a detailed description thereof will be omitted.

On the other hand, when there are a plurality of persons related to the space of interest in response to the space of interest being specified, a plurality of person information (eg, photo information, name information, job title information, etc. for each of the plurality of persons) is transmitted to the user terminal. It can be displayed through (200). At this time, the controller 140 may specify a person related to the space of interest based on any one or a user input for at least one of the plurality of displayed person information.

For example, the controller 140 transmits information on persons related to "city hall", for example, "mayor", "deputy mayor", and "civil affairs officer" in response to specifying "city hall" as a space of interest. It is displayed through the user terminal 200, and based on a user input for “mayor” among the displayed information, “mayor” may be specified as a person related to the space of interest.

However, the method of specifying the space of interest and the person related to the space of interest is not limited to the above examples, and can be understood as various means for specifying one of a plurality of virtual spaces and a person related to the specified space. there is.

Furthermore, the controller 140 according to the present invention may acquire person information of a person related to the specified space of interest (S203).

More specifically, the controller 140 may acquire person information 501 of the person 20 specified as a person related to the space of interest 170 by controlling the information acquisition unit 130 .

As shown in FIG. 4 , the controller 140 uses the information acquisition unit 130 disposed in the real space 172 corresponding to the space of interest 170 in the virtual space 30 to obtain the real space 172. Person information 501 for a person 20 existing in may be acquired.

For example, the controller 140 transmits image information or image information of the person 20 captured through the information acquisition unit 130 (eg, a camera) disposed in the real space 172 to the person 20. It can be obtained as person information 501 . For another example, the control unit 140 converts the voice information of the person 20 recorded through the information acquisition unit 130 (eg, a recorder) disposed in the real space 172 to the person information about the person 20. It can be obtained as (501). Furthermore, it goes without saying that the controller 140 may acquire the person information 501 for the person 20 by being disposed in a space different from the actual space or by using other means.

For example, the controller 140 may obtain the person information 501 of the person 20 related to the space of interest 170 by receiving person information from an external server through the communication unit 110 . The controller 140 may receive an image file or a voice file for a specific person 20 from an external server through the communication unit 110 .

Furthermore, in response to the space of interest 170 and the person 20 related to the space of interest 170 being specified, the controller 140 stores information in the storage unit 120 in association with the space of interest 170 and the person 20 . It can be obtained by calling or loading the stored person information 501 . In this case, the person information 501 may already exist in the storage unit 120 by being matched with the person 20 .

Furthermore, the controller 140 according to the present invention may control the artificial neural network 150 to generate a digital human 10 corresponding to the person 20 using the person information 501 (S205).

As shown in FIG. 5 , the controller 140 inputs the acquired person information 501 to the artificial neural network 150, so that the artificial neural network 150 corresponds to the person 20 using the person information 501. A digital human 10 may be created.

More specifically, the controller 140 inputs the acquired person information 501 to the artificial neural network 150, so that the artificial neural network 150 extracts a person characteristic (or persona) from the person information 501 and extracts it. It is possible to generate a digital human 10 corresponding to a person 20 existing in a real space by having a person characteristic (or persona (eg, appearance, voice, gesture)) .

At this time, the artificial neural network 150 may be designed to receive the person information 501 and learn how to generate the digital human 10 based on the person information 501 . More specifically, the artificial neural network 150 receives character information 501, extracts character characteristics (or personas) from the received character information 501, and creates a digital human 10 having the extracted character characteristics. can be designed to learn how to create

For example, the artificial neural network 150 may be designed to learn how to generate a digital human 10 having an appearance extracted from image information and/or a voice extracted from voice information among the acquired person information 501. there is.

The control unit 140 according to the present invention can control the artificial neural network 150 to create a digital human 10 having an external appearance 602 corresponding to that of a person 20 existing in the real space 172. .

More specifically, as shown in FIG. 6 , the controller 140 inputs the image information 601 of the person information 501 to the artificial neural network 150 so that the artificial neural network 150 can obtain from the image information 601 The digital human 10 having the extracted appearance 602 may be created.

The controller 140 may generate a digital human 10 corresponding to a 2D or 3D image based on the image information 601 through the artificial neural network 150 .

At this time, the image information 601 may include at least one of a face image, a body shape image, and a gesture image corresponding to a plurality of gestures having different meanings (eg, a greeting gesture, a rejection gesture, etc.) of the person 20. can The artificial neural network 150 may generate the digital human 10 so that the digital human 10 takes various facial expressions and actions in the space of interest based on such various image information. Meanwhile, the artificial neural network 150 may generate the digital human 10 having various facial expressions of the person 20 by synthesizing the face image of the person 20 and various previously secured facial expression images. Furthermore, the artificial neural network 150 may generate a digital human 10 that takes various gestures in various situations by synthesizing a body shape image and a plurality of gesture images of the person 20 and various previously secured gesture images. .

At this time, the artificial neural network 150 may be understood as various types of artificial intelligence models that process the image information 601 . For example, the artificial neural network 150 uses at least one of facial action coding system (FACS), speech emotion recognition (SER), facial expression recognition (FER), gaze tracking, Lucas-Kanade Fitting Algorithm, face reenactment, and face synthesis. As an artificial intelligence model using technology, a method of extracting an external appearance 602 from image information 601 using at least one of the above-described technologies and generating a digital human 10 having the extracted external appearance 602 can learn

However, the technique or method of extracting the external appearance 602 from the image information 601 by the artificial neural network 150 is not limited to the above example, and the external appearance 602 from the image information 601 using a biological neuron system. ) can be understood as a variety of models for extracting.

In addition, the controller 140 according to the present invention controls the artificial neural network 150 to generate the digital human 10 having a voice 702 corresponding to that of the person 20 existing in the real space 172. can

As shown in FIG. 7 , the controller 140 inputs voice information 701 of person information 501 to the artificial neural network 150, so that the artificial neural network 150 extracts a voice 702 from the voice information 701. ).

More specifically, the controller 140 inputs the voice information 701 obtained according to the speech of the person 20 to the artificial neural network 150, so that the artificial neural network 150 extracts the voice 702 from the voice information 701. ).

At this time, the artificial neural network 150 may be understood as various types of artificial intelligence models that process the voice information 701 . For example, the artificial neural network 150 is an artificial intelligence model that utilizes at least one of Wavenet-based text to speech (TTS) synthesis technology, multi-speaker TTS, multi-language TTS, voice conversion, and self-attention technology, A method of extracting a voice 702 from the voice information 701 using at least one of the above-described techniques and generating a digital human 10 having the extracted voice 702 may be learned.

However, the technology or method of extracting the voice 702 from the voice information 701 by the artificial neural network 150 is not limited to the above example, and the voice 702 from the voice information 701 using a biological neuron system. ) can be understood as a variety of models for extracting.

Through the above-described configuration, the digital human providing system 100 generates and provides a digital human having the appearance and/or voice of a person related to the space of interest, thereby providing a virtual user experience (UX) similar to a real experience. It can be provided through space (or metaverse space).

Furthermore, the controller 140 according to the present invention can control the artificial neural network 150 to create a digital human 10 that operates according to a gesture corresponding to the motion of the person 20 existing in the real space 172. there is.

To this end, as shown in FIGS. 8A and 8B , the controller 140 inputs image information 801 including the motion of the person 20 related to the space of interest 170 to the artificial neural network 150, thereby The artificial neural network 150 may generate the digital human 10 operating according to the gesture 802 extracted from the image information 801 .

Referring to FIG. 8A , the controller 140 controls the information acquisition unit 130 (eg, camera) disposed in the real space 172 corresponding to the space of interest 170, so that the person in the real space 172 Image information 801 including the operation of (20) can be obtained.

More specifically, the controller 140 generates image information 801 including the motion of the person 20 based on information obtained through the plurality of information acquisition units 131 to 134 disposed in the real space 172. can be obtained.

However, according to another embodiment, the plurality of information acquisition units 131 to 134 disposed in the real space 172 may be understood as a configuration in which a single information acquisition unit (eg, camera) moves while changing its location. there is.

For example, the control unit 140 uses structure from motion (SfM) to obtain a plurality of images (image i to image i) obtained through the plurality of information acquisition units 131 to 134 disposed in the real space 172. +3), image information 801 including the motion of the person 20 may be obtained.

More specifically, the controller 140 determines the person 20 based on the plurality of images acquired by the plurality of information acquisition units 131 to 134 and the viewing angle between the plurality of images, based on the plurality of feature points included in the person 20 . Information related to the operation of (20) can be obtained.

Furthermore, as shown in FIG. 8B , the controller 140 inputs the image information 801 of the person information 501 to the artificial neural network 150, so that the artificial neural network 150 extracts the gesture from the image information 801. A digital human 10 operating according to 802 may be created.

At this time, the artificial neural network 150 may be understood as various types of artificial intelligence models that process the image information 801 . For example, the artificial neural network 150 is an artificial intelligence model using at least one of an active shape model (ASM), lipsync animation, a deep convolutional generative adversarial network (DCGAN) structure, and a 3D reconstruction technique, and includes at least one of the above techniques. A method of extracting the gesture 802 from the image information 801 and generating the digital human 10 operating according to the extracted gesture 802 can be learned using one.

However, the technique or method of extracting the gesture 802 from the image information 801 by the artificial neural network 150 is not limited to the above example, and the gesture 802 from the image information 801 using a biological neuron system. ) can be understood as a variety of models for extracting.

In addition, the controller 140 identifies an effective motion among a plurality of motions of the person 20 included in the image information 801, and inputs an image including the effective motion to the artificial neural network 150, so that the artificial neural network 150 ) can generate the digital human 10 operating according to the gesture extracted based on the effective motion.

Here, the valid motion is a meaningful motion to be generated as a gesture of the digital human 10 in the virtual space, and may be different depending on the purpose, nature, or type of the virtual space. In addition, a plurality of valid operations may be defined for each virtual space and may be stored in the storage unit 120 . That is, at least one piece of valid motion information matched for each virtual space may exist in the storage unit 120 . For example, in the case of a virtual space corresponding to "hospital", i) receptionist action, ii) wheelchair pushing action, iii) greeting action, etc. may be defined and exist as valid actions. Furthermore, they correspond to "viewing". In the case of a virtual space, i) a greeting operation, ii) an operation of guiding document preparation, and the like may be defined as valid operations.

Meanwhile, the definition of such an effective motion may be performed in various ways, and may be performed by defining the task of the person 20 from the image information 801 acquired in the real space 172 . For example, the controller 140 may obtain work information of the person 20 in the space of interest 170 or the real space 172 corresponding to the space of interest 170 as the person information 501 .

More specifically, the controller 140 may extract work information of the person 20 in the real space 172 from image information 801 of the person 20 in the real space 172 . For example, the controller 140 may extract information about guidance work of the mayor based on image information of the mayor in the actual viewing space. In addition, the control unit 140 may extract (or define) the extracted work information and motion information of the person 20 when performing a task corresponding to the work information as an effective motion. Then, the control unit 140 obtains the extracted valid motion from the image information 801, stores it in a storage unit, and can use it to create a digital human.

Alternatively, in response to the space of interest 170 being specified, the controller 140 may acquire work information stored in the storage 120 in association with information on the space of interest 170 . For example, the controller 140 may obtain information about a guide service stored in association with the viewing space in response to specifying a viewing space within the virtual space 30 as a space of interest. Such work information may be acquired from work-related manuals, work guidelines, work lists, etc. obtained through various channels. In addition, the control unit 140 may extract what motion the person 20 took when performing the task according to the task information from the image, and define the extracted motion information as an effective motion for the task information. there is. Accordingly, in the virtual space (virtual space of interest), it is possible to create a digital human performing a defined valid operation.

Furthermore, the controller 140 identifies at least one motion corresponding to the work information of the person 20 as a valid motion based on a predetermined criterion among the plurality of motions of the person 20 included in the image information 801. can do.

For example, the controller 140 corresponds to motions stored in association with work information of the person 20 in the space of interest 170, among a plurality of motions of the person 20 included in the image information 801. An action can be identified as a valid action.

For another example, the controller 140 may select at least one of the plurality of actions of the person 20 included in the image information 801 through an input received from a manager (eg, a person performing a task related to creating a digital human). The operation of can be identified as a valid operation.

For another example, the control unit 140 determines an operation corresponding to a general operation stored as an operation performed regardless of a job corresponding to work information from a plurality of operations of the person 20 included in the image information 801. At least one operation excepted may be identified as a valid operation.

Meanwhile, in the process of acquiring the person information 501, the controller 140 may obtain object information of an object disposed in the space of interest 170 or the real space 172 corresponding to the space of interest 170. In this case, a virtual object corresponding to the obtained object information may be disposed in the virtual space (or the space of interest) so as to be interactive. In this case, part or all of the object information may be matched with the virtual object, and through this, the user may interact with the virtual object corresponding to the real object through the digital human in the virtual space.

In this regard, the controller 140 obtains object information of an object used by the person 20 in the real space 172 or adjacent to the person from image information 801 of the person 20 in the real space 172. can be extracted. For example, the control unit 140 may extract information about an automated device that guides according to the mayor's guidance work, based on image information of a mayor in an actual viewing space.

According to another embodiment, the controller 140 may acquire object information stored in the storage 120 in association with information on the space of interest 170 in response to the space of interest 170 being specified. For example, the controller 140 may obtain information about an automated device stored in association with the viewing space in response to the viewing space within the virtual space 30 being specified as the space of interest.

Furthermore, the controller 140 recognizes the position of the object using object information in the image information 801, and among the plurality of movements of the person 20 included in the image information 801, the position of the recognized object and the person Only an operation performed when the position of (20) is within a preset range can be identified as a valid operation.

At this time, the control unit 140 may match the identified effective operation with object information of the recognized object. Here, object information to be matched to the valid operation may be the same as object information matched to the virtual object. Through this, the controller 140 may generate a digital human to perform an effective operation matched with the virtual object when the digital human interacts with the virtual object in the virtual space (or space of interest).

In this regard, the controller 140 acquires a pattern of features by applying a process such as feature point extraction and boundary determination to the image information 801, and compares the obtained pattern with pre-prepared object information to recognize an object. can For example, the controller 140 may recognize an object from the image information 801 using a conventional technique such as SVM (Support Vector Machine). In addition, the controller 140 may recognize an object from the image information 801 using an object recognition model learned through a separate deep learning algorithm.

Meanwhile, for the process of identifying a valid motion, for example, the controller 140 determines the space of interest 170 and A motion of the person 20 adjacent to objects stored in association with each other may be identified as a valid motion.

For another example, the controller 140 may identify, as a valid motion, a motion of the person 20 adjacent to at least one object selected through an input received from a manager among a plurality of objects included in the image information 801. there is.

For another example, the control unit 140 controls at least one of the plurality of objects included in the image information 801 except for an object corresponding to a general object stored as an object disposed regardless of the space of interest 170. The motion of the adjacent person 20 may be identified as a valid motion.

In this regard, the controller 140 recognizes the position of the object using object information in the image information 801, and among the plurality of movements of the person 20 included in the image information 801, the position of the recognized object extracts at least one motion performed when the position of the person 20 and the person 20 is within a preset range, and among the extracted at least one motion, at least one corresponding to the work information of the person 20 based on a predetermined criterion; An action may be identified as a valid action.

Furthermore, the controller 140 according to the present invention may obtain image information including only effective motions as person information 501 . More specifically, the controller 140 obtains an image portion including an effective motion among a plurality of motions of the person 20 included in the image information 801 as the person information 501 and inputs the obtained image to the artificial neural network 150. can do.

That is, the controller 140 according to the present invention inputs image information including only effective motions among a plurality of motions of the person 20 to the artificial neural network 150, so that the artificial neural network 150 extracts a gesture from the effective motions and It is possible to create (or learn how to create) a digital human that operates according to the extracted gesture.

In addition, the controller 140 inputs the image information 801 of the person 20 in the real space 172 to the artificial neural network 150, so that the artificial neural network 150 determines the situational behavior of the person 20. It is possible to extract and generate a digital human 10 performing a gesture according to the extracted behavioral pattern. More specifically, the control unit 140 inputs the image information 801 of the person 20 in the real space 172 to the artificial neural network 150, so that the artificial neural network 150 generates data generated in the real space 172. A situation and a behavior pattern of the person 20 corresponding to each situation may be extracted, and a digital human 10 performing a gesture according to the extracted behavior pattern in response to a situation in the space of interest 170 may be generated. .

Through this, the digital human provision system 100 according to the present invention generates and provides a digital human operating according to a gesture extracted from an effective motion by an artificial neural network learned through an effective motion related to work among motions of a person in a real space. By doing so, it is possible to provide a user experience in a virtual space that is more accurate and similar to a real experience.

Furthermore, as shown in FIG. 9 , the control unit 140 according to the present invention inputs the speech contents 902 of the person identified from the person information to the artificial neural network 150, so that the artificial neural network 150 converts the speech contents ( 902), a speech style can be extracted, and the digital human 10 uttered according to the extracted speech style can be generated.

In this case, the speech content 902 may be obtained from voice information 701 of the person 20 in the real space 172 .

According to the embodiment, the content of speech 902 may be obtained from audio information 701 created at the same time point as the valid motion is identified in the video information 801 . In this case, the controller 140 may match the valid motion with the utterance 902 .

At this time, the controller 140 sends a plurality of voice information 701 (eg, "Hello", "Hello", "Hello") to any one type of effective operation (eg, greeting operation) extracted from different video information 801. Nice to meet you") is matched, only voice information 701 that satisfies a preset standard among a plurality of matched voice information 701 may be used. For example, the control unit 140, when a plurality of voice information 701 is matched to any one type of valid operation extracted from different video information 801, appears with the highest frequency among the plurality of voice information 701 Voice information 701 is available.

Through this, the control unit 140 can generate the digital human 10 to utter the utterance 902 matched to the valid motion at the time of taking the effective motion.

According to another embodiment, the utterance content 902 may be obtained from a conversation history of a conversation between the user and the digital human 10 in the virtual space 30 . In addition, the utterance content 902 may be obtained from a conversation history of conversations with the digital human 10 by other users other than the current user within the space of interest 170 of the virtual space 30 .

In addition, the artificial neural network 150 may be understood as various types of artificial intelligence models that process the speech contents 902 . For example, the artificial neural network 150 is an artificial intelligence model that utilizes at least one of speech to text (STT), dialog management (DM), self-attention, natural language processing (NLP), and deep question answering (QA). , A method of extracting a speech pattern from the speech contents 902 using at least one of the above-described techniques and generating a digital human 10 that speaks according to the extracted speech style may be learned.

However, the technique or method of extracting the speech pattern from the speech contents 902 by the artificial neural network 150 is not limited to the above example, and extracting the speech pattern from the speech contents 902 using a biological neuron system It can be understood in various models.

Also, the controller 140 may identify the personality type 901 (“Persona”) of the person 20 from the person information 501 .

More specifically, the controller 140 inputs at least one of the image information 601 and the audio information 701 to the artificial neural network 150 so that the artificial neural network 150 converts the image information 601 and the audio information 701 The personality type 901 of the person 20 may be extracted based on at least one of the extracted personality types 901, and the digital human 10 operating according to the extracted personality type 901 may be generated.

Meanwhile, the controller 140 may create a knowledge base ("knowledge base") 910 including the identified utterance content 902 from the person information 501 and the personality type 901 of the person 20. .

The control unit 140 according to the present invention is a digital human (for which the artificial neural network 150 provides a response 102 according to the user request 101 using the speech content 902 and/or the personality type 901). 10) can be created.

More specifically, the controller 140 generates the knowledge base 910 including the utterance content 902 and the personality type 901 so that the artificial neural network 150 responds to the user request 101 to generate the knowledge base 910. ) and create a digital human 10 that provides a response 102 based on the utterances 902 and/or the personality type 901 included in the knowledge base 910.

Through this, the digital human provision system 100 according to the present invention is a digital human 10 having a speech style and personality type 901 corresponding to the speech style and personality type of the person 20 in the real space 172. ), a response 102 to the user request 101 may be provided. For example, in response to the user request 101 , the controller 140 may generate and provide a digital human 10 that provides a response 102 with an official speech style and a friendly personality.

Furthermore, the controller 140 according to the present invention may provide the digital human 10 through the space of interest 170 in response to the user accessing the virtual space 30 (or the space of interest 170). (S207). More specifically, in response to the user accessing the virtual space 30 or providing the user request 101 through the virtual space 30, the controller 140 displays the appearance of a real person in the space of interest 170, The digital human 10 having at least one of voice and gesture may be provided.

That is, in response to the user accessing the virtual space 30 or providing the user request 101 through the virtual space 30, the controller 140 displays the appearance, voice, and appearance of a real person in the space of interest 170. By providing the digital human 10 having at least one of the gestures, the digital human 10 may provide (or perform) a response 102 to the user request 101 .

For example, in response to the user accessing the virtual space 30, the control unit 140 provides a digital human 10 operating with the appearance, voice and gesture of the “market” in the virtual viewing service center, The digital human 10 may provide a response 102 to a user request 101 for a civil petition in the virtual space 30 (or the space of interest 170 ).

As described above, the digital human providing system 100 according to various embodiments of the present invention creates and provides a digital human to have character characteristics extracted from the character information of the person 20 related to the space of interest 170. , it is possible to minimize a sense of difference with a real person and provide a user experience similar to a real space.

On the other hand, the computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. there is

Furthermore, the computer-readable medium may be a server or cloud storage that includes storage and can be accessed by electronic devices through communication. In this case, the computer may download the program according to the present invention from a server or cloud storage through wired or wireless communication.

Furthermore, in the present invention, the above-described computer is an electronic device equipped with a processor, that is, a CPU (Central Processing Unit), and there is no particular limitation on its type.

On the other hand, the above detailed description should not be construed as limiting in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (10)

In the method of providing a digital human in a virtual space,
specifying a space of interest in the virtual space including a plurality of spaces;
acquiring person information of a person related to the space of interest;
generating, by an artificial neural network, the digital human corresponding to the person based on the acquired person information; and
In response to a user accessing the virtual space, providing the created digital human to the space of interest among the virtual spaces;
The artificial neural network learns a method of generating the digital human having the extracted character characteristics by receiving character information of a person related to a specific space and extracting character characteristics from the input character information,
The person information includes image information about the person and business information of the person,
Acquiring the person information,
identifying an effective motion corresponding to work information of the person, based on a predetermined criterion, among a plurality of motions of the person included in the image information in a real space; and
Acquiring image information including the effective motion as the person information; includes,
The method of providing a digital human, characterized in that the artificial neural network learns a method of generating the digital human operating according to the gesture extracted from the effective motion. According to claim 1,
The person information includes image information and voice information about the person,
The method of providing a digital human, characterized in that the artificial neural network learns a method of generating the digital human to have at least a part of an appearance extracted from the image information and a voice extracted from the voice information. According to claim 2,
Wherein at least some of the person information is obtained through an information acquisition unit disposed in the real space with respect to the person in the real space corresponding to the space of interest. According to claim 2,
Providing the digital human to the space of interest includes:
receiving the user's request within the virtual space; and
and providing the digital human having the character characteristics and responding to the request to the space of interest. According to claim 4,
The artificial neural network extracts the personality type of the person from at least one of the image information and the voice information, and generates the digital human having at least a part of the appearance and voice corresponding to the extracted personality type. Characterized in that learning, digital human provision method. In the method of providing a digital human in a virtual space,
specifying a space of interest in the virtual space including a plurality of spaces;
acquiring person information of a person related to the space of interest;
generating, by an artificial neural network, the digital human corresponding to the person based on the acquired person information; and
In response to a user accessing the virtual space, providing the created digital human to the space of interest among the virtual spaces;
The artificial neural network learns a method of generating the digital human having the extracted character characteristics by receiving character information of a person related to a specific space and extracting character characteristics from the input character information,
The person information includes image information about the person,
Acquiring the person information,
obtaining object information of an object disposed in the space of interest;
Recognizing the object from the image information using the object information;
identifying only an operation performed when the position of the recognized object and the position of the person are within a preset range among the plurality of motions of the person included in the image information as a valid motion; and
Acquiring image information including the effective motion as the person information;
The method of providing a digital human, characterized in that the artificial neural network learns a method of generating the digital human operating according to the gesture extracted from the effective motion. delete delete In the digital human providing system in virtual space,
The controller of the digital human providing system:
Based on a user input, specifying a space of interest in the virtual space including a plurality of spaces;
Obtaining person information of a person related to the space of interest;
An artificial neural network generates the digital human corresponding to the person based on the acquired person information;
In response to the user accessing the virtual space, providing the generated digital human to the space of interest in the virtual space;
The artificial neural network learns a method of generating the digital human having the extracted character characteristics by receiving character information of a person related to a specific space and extracting character characteristics from the input character information,
The person information includes image information about the person and business information of the person,
The control unit,
Among the plurality of motions of the person in the real space included in the image information, based on a predetermined criterion, an effective motion corresponding to the work information of the person is identified, and image information including the effective motion is Obtained as the person information,
Wherein the artificial neural network learns a method of generating the digital human operating according to the gesture extracted from the effective motion. As a program that is executed by one or more processes in an electronic device and can be stored in a computer-readable medium,
said program,
specifying a space of interest in a virtual space including a plurality of spaces;
acquiring person information of a person related to the space of interest;
generating, by an artificial neural network, a digital human corresponding to the person based on the acquired person information; and
In response to a user accessing the virtual space, instructions for performing a step of providing the generated digital human to the space of interest among the virtual spaces;
The artificial neural network learns a method of generating the digital human having the extracted character characteristics by receiving character information of a person related to a specific space and extracting character characteristics from the input character information,
The person information includes image information about the person and business information of the person,
Acquiring the person information,
identifying an effective motion corresponding to work information of the person, based on a predetermined criterion, among a plurality of motions of the person included in the image information in a real space; and
Acquiring image information including the effective motion as the person information; includes,
The artificial neural network learns a method of generating the digital human operating according to the gesture extracted from the effective motion.