KR102642001B1 - Metaverse-based video chat service system and method - Google Patents

Abstract

The metaverse-based video chat service system according to an embodiment of the present invention connects a plurality of user terminals to participate in video chatting through a network, and provides a metaverse-based video chat service between the connected user terminals, but in reality, It includes an integrated management server that provides video chatting services through the user's video captured in real time from the user terminal in a metaverse space implemented in a 3D stereoscopic virtual reality space similar to the present invention. According to the present invention, various types of video chat rooms are provided. can be implemented as a metaverse to provide more immersion, and has the advantage of allowing users to safely conduct video chatting online and non-face-to-face.

Description

Metaverse-based video chat service system and method {Metaverse-based video chat service system and method}

The present invention relates to a metaverse-based video chat service system and method, and more specifically, to a metaverse-based video chat service system and method that can provide a video chat service in a three-dimensional virtual reality metaverse space close to reality. will be.

In general, with the rapid development in the field of information and communication technology, email, chat, chat, etc. that use data communication are being widely used as new methods for communication.

In particular, the chat service provision system allows real-time mutual communication with unspecified people, including acquaintances, through text, voice, and video, and allows for not only simple conversations but also general meetings, international conferences, expert consultations, and conversations with foreigners. It is also used for other purposes.

In addition, the conventional video chat service provision system is implemented so that the user can sign up, register a predetermined ID, and chat with other users 1:1 or 1:many, and there is no separate registration process or user It is also implemented so that one can chat with other users with guaranteed anonymity by keeping one's personal information private.

Recently, as the spread of smartphones increases, various chatting services using smartphones are also being commercialized.

However, in the case of the conventional chat service provision system, as it provides anonymity between users, there are problems with various side effects such as promoting illegal companies, swearing, personal slander, exposing personal information, and providing false information.

Meanwhile, non-face-to-face online services using the metaverse (a virtual world that transcends reality), which has recently been in the spotlight, are receiving more attention as non-face-to-face life is prolonged due to the resurgence of COVID-19, coupled with the desire to expand reality into the virtual world.

Metaverse is a word that combines 'Meta', meaning transcendence, and 'Universe', meaning world.

As a specific example of using the metaverse, universities have held entrance information sessions, new student welcome parties, and university festivals using the metaverse, and are also recently building a library.

As another example, companies are holding new employee recruitment briefing sessions and social contribution programs on Metaverse, and in the political world, presidential candidates have set up campaign venues on Naver Z's Metaverse platform 'Zepetto' to capture the votes of the so-called 'MZ generation'. there is.

The metaverse commonly thought of is games using avatars or non-face-to-face meetings held in virtual space.

Therefore, in order to solve the above-mentioned problem, a metaverse-based metaverse that can provide a realistic and immersive video chat service online and non-face-to-face, while maintaining the openness of chat partners by utilizing avatars in the metaverse space of virtual reality, which is close to reality, is used. Research on video chat service systems and methods has become necessary.

Republic of Korea Patent Publication No. 10-2018-0003911 (published on January 10, 2018)

The purpose of the present invention is to provide a video chat service in the metaverse space of virtual reality, which is close to reality, and to restrict the chat service according to the emotional state of the speaker during video chat along with authentication upon membership registration, and to evaluate the chat partner after chat. The goal is to provide a metaverse-based video chat service system and method that calculates chat reliability and provides it for subsequent inquiry.

The metaverse-based video chat service system according to an embodiment of the present invention connects a plurality of user terminals to participate in video chatting through a network, and provides a metaverse-based video chat service between the connected user terminals, but in reality, It includes an integrated management server that provides video chatting services through the user's video captured in real time from the user terminal in a metaverse space implemented in a 3D virtual reality space similar to .

The integrated management server includes a communication protocol that allows user terminal access through the network, and includes a communication unit that transmits and receives information for providing a video chat service; a metaverse generator that creates a metaverse, an online virtual space for performing video chat services; A chat management unit capable of providing a video chat service to a connected user terminal using metaverse data, which is a virtual space corresponding to the video chat room stored in the metaverse creation unit; A recommendation learning unit that utilizes an artificial intelligence-based recommendation algorithm to recommend a customized meeting place that matches the meeting tendency or type, or automatically recommends a chat partner that matches the user's personality based on the user's personal information collected during membership registration when matching chat partners. ; The integrated management server further includes a database that stores and manages data necessary to provide a metaverse-based video chat service.

The chat management unit can recognize the emotional state of the speaker conducting a video chat through an object recognition-based emotion estimation algorithm during chatting, and if the recognized emotional state is unpleasant and persists for more than a certain period of time or a certain number of times, a warning and alarm are issued. A message is transmitted, and if the warning and alarm message transmission continues for more than a certain number of times, the screen can be blocked and a penalty point given to the speaker who caused unpleasant feelings in the other party.

The integrated management server applies a chroma key to only the background portion of the original video of each user received to create a natural screen when compositing video images taken from a user terminal on a virtual background during video chatting, and then controls each user terminal participating in the video chatting. By transmitting to the screen, the screen is provided to match the virtual background.

In the metaverse-based video chat service method using an integrated management server according to an embodiment of the present invention, register and log in to use the video chat service provided by the integrated management server using user terminals owned by multiple users. A step of connecting; After logging in, the user terminal selects an avatar using a virtual or own image to use the Metaverse-based video chat service, selects a chat partner or a Metaverse video chat room space, or uses the integrated management server's recommendation algorithm upon request. A step of automatically recommending a chat partner or a Metaverse video chat room space through; and performing, by the user terminal, a video chat service in a video chat room in the metaverse space selected by the user or by automatic recommendation.

During video chatting, the integrated management server uses the object recognition-based emotion estimation algorithm described above to recognize the state of each speaker through emotion recognition and determine whether to block the conversation; And after the video chat ends, the user terminal provides the evaluation results of the video chat service and the chat partner (speaker) to the integrated management server to calculate the trustworthiness of the chat partner, and when using the video chat service in the future, other users are informed of the chat partner. A step of allowing reliability to be checked is further included.

The integrated management server collecting online meeting chat evaluation results input by the user; The integrated management server determining whether the collected chat evaluation result is greater than or equal to a preset standard value; If the chat evaluation result is higher than the reference value, the integrated management server grants permission to hold an offline meeting after raising the level of the chat host; If the chat evaluation result is below a standard value, the integrated management server issues a warning, lowers the level, and then restricts authority; And the integrated management server further includes the step of storing and updating user information data including level information in the DB to complete setting differential authority according to level.

The user terminal accesses the integrated management server and requests membership registration by inputting user information, which is personal information required for the video chat service; The user terminal logs in with registered user information and receives additional information about interests, preferred occupations, and hobbies for customized video chatting; And the integrated management server further includes a step of completing the membership registration by finally accepting the membership registration after final confirmation of the user information entered at the time of membership registration.

The metaverse-based video chat service system and method of the present invention can provide a video chat service in a virtual reality metaverse space close to reality, providing a realistic and immersive video chat service, and various types of video chat rooms in the metaverse. It can provide more immersion by implementing it, and has the advantage of allowing users to safely conduct video chatting online and non-face-to-face.

Additionally, along with authentication upon signing up for membership, you can select to automatically block or give penalty points depending on the emotional state of the speaker during video chat, which has the advantage of creating a healthy chatting culture.

In addition, it has the advantage of providing user convenience by calculating the chat reliability by evaluating the chat partner after chatting and providing it for subsequent inquiry so that it can be referred to when selecting the next chat partner.

In addition, based on the recommendation algorithm, it automatically recommends a Metaverse video chat room that matches the type or tendency of the chat partner or meeting, increasing user convenience and allowing the user to select the desired chat partner or meeting video chat room.

In addition, by applying chroma key when compositing video images taken from the user terminal's built-in camera or webcam on the virtual metaverse background and sending and receiving them to the screen of each user terminal participating in the video chat, a natural screen can be created to match the virtual background. This has the advantage of further increasing the immersion of video chat users in the metaverse space.

Figure 1 is a block diagram showing the configuration of a metaverse-based video chat service system according to an embodiment of the present invention.
Figure 2 is a block diagram showing the internal configuration of the integrated management server of Figure 1 in detail.
Figure 3 is a flowchart explaining the entire process of the metaverse-based video chat service method according to an embodiment of the present invention.
FIG. 4 is a flowchart specifically explaining the membership authentication process of the metaverse-based video chat service method of FIG. 3.
FIG. 5 is a flowchart specifically explaining the service evaluation and user permission setting process of the metaverse-based video chat service method of FIG. 3.
Figure 6 is a flowchart explaining the process of arranging video images of video chat participants on the table virtual background when the video chat meeting place is a table conference according to an embodiment of the present invention.
Figure 7 is a diagram illustrating an exemplary chroma key processing process for natural presentation during video chatting.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the presented embodiments, and those skilled in the art who understand the spirit of the present invention may add, change, or delete other components within the scope of the same spirit, or create other degenerative inventions or this invention. Other embodiments that are included within the scope of the invention can be easily proposed, but this will also be said to be included within the scope of the invention of the present application. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

FIG. 1 is a block diagram showing the configuration of a metaverse-based video chat service system according to an embodiment of the present invention, and FIG. 2 is a block diagram showing the internal configuration of the integrated management server of FIG. 1 in detail.

In the present invention, video chatting or chatting refers not only to video chatting or video calling between individuals or groups (chatting and calling may occur simultaneously), but also to video-related chatting and calling services such as video meetings and video conferences. It can be interpreted as an all-inclusive meaning.

Referring to FIG. 1, the metaverse-based video chatting service system of the present invention connects a plurality of user terminals (user terminals 1 to N, 100) to participate in video chatting through a network 400, and the connected user terminal 100 ) includes an integrated management server 300 that provides a metaverse-based video chat service between the users.

Specifically, the integrated management server 300 creates a 3D stereoscopic virtual reality space similar to reality in real time from a built-in camera (not shown) of the user terminal 100 or a webcam (not shown) connected to the user terminal 100 in the metaverse space. It provides a video chat service through your own recorded video, and provides realistic video chat by implementing various meeting places. Customized meetings according to the tendency or type of meeting are made using an artificial intelligence-based recommendation algorithm. It can recommend locations or automatically recommend chat partners that match the user's personality based on the user's personal information collected during membership registration when matching chat partners.

In addition, the integrated management server 300 can recognize the emotional state of the chatting speaker through an object recognition-based emotion estimation algorithm during chatting, and based on this, if the emotional change is judged to be a difficult situation to chat, such as an unpleasant state or an angry state, It may include functions that send warnings and alarm messages, block the screen, and grant penalty points if it continues longer than a standard time.

In addition, you can report a chat partner directly from the user terminal 100. When reporting, the screen is captured or the recorded data is separately stored and provided to the integrated management server 300, so the integrated management server 300 records the data. You can check and take action such as assigning penalty points.

In addition, the integrated management server 300 directly receives user evaluations of the chat service and the chat partner (speaker) after the chat, measures chat reliability based on each collected evaluation result, and searches for reliability so that it can be viewed during future chats. Services may also be provided.

Referring to FIG. 2, the integrated management server 300 to specifically perform the above-described functions includes a communication unit 310, a metaverse creation unit 320, a chat management unit 330, a recommendation learning unit 340, and an information protection unit. It may further include a unit 350 and a database 360.

The communication unit 310 includes a communication protocol that allows the user terminal 100 to connect through the network 400, and can transmit and receive various information (personal information upon membership registration, chat evaluation information, etc.) for providing a video chat service. .

In addition, the video chat service can be provided not only on a web-based basis, but also on a mobile basis depending on the type of the user terminal 100, and the communication unit 310 is a mobile-based metaverse video chat service installed and executed on the user terminal 100. An API (Application programming interface) interface for linking with the app may be further included.

The metaverse creation unit 320 can create a metaverse, which is an online virtual space for performing a video chat service.

The form of the metaverse can be, for example, a virtual space for a meeting place suitable for chatting or a space where the avatar is located before chatting. In the case of the metaverse space, separate metaverse data is created for various meeting places and stored in a database (360). After deciding, it can be selectively implemented in a video chat room according to the appropriate meeting place.

Meeting places can be, for example, conference rooms, cafes, restaurants, beaches, meeting rooms, parks, game facilities, tourist attractions, amusement parks, etc. Metaverse data on various meeting places depending on the needs such as the nature of the meeting or type of meeting. can be created individually.

In addition, as the metaverse data is implemented as a 3D virtual space, the chat service can be provided by chroma key synthesis of the virtual background and video video for a natural production during video chat service, and by changing it to various angles or viewpoints to suit the purpose and environment of the chat. For example, the user can directly set and switch between first-person (frontal) and third-person perspectives (oblique and perspective perspectives), or within 360-degree left, right, up and down angles.

For a specific example, Figure 7 shows an example showing the chroma key application process for natural production during video chatting, and the original video image captured from the built-in camera or webcam of the user terminal 100 on the virtual metaverse background ( In order to produce a natural screen when compositing (source), for example, the shader function of the Unity engine is used to apply a chroma key and then transmitted to the screen of each user terminal participating in the video chat to match the virtual background. This allows you to create a natural screen.

In other words, chroma key synthesis technology recognizes the surrounding background in addition to the human shape (silhouette) in each user's original video image and makes the background area in the video image transparent using a green screen or blue screen. , During compositing, the corresponding area can be replaced with a virtual background, allowing a natural composite screen as shown in Figure 7 to be created.

In order to create a video conference screen similar to the actual scene, it can be done from a first-person (frontal) or third-person perspective (oblique or perspective perspective) or by using the 360-degree left, right, up and down angle adjustment function.

In addition, in cases where the meeting place during video chat is seated at a table, a virtual table is added to the chroma-keyed video so that when the avatar is seated at the virtual background table on the chat service screen, it is possible to create the appearance of the user sitting at the actual table. By recognizing an object in the area where the table and the image screen overlap and applying the chroma key function to the area in contact with the object to make it transparent, you can create a natural sitting situation.

In addition, when video chatting on a table, a function of sitting at an angle can be implemented depending on the sitting position, and when sitting at a table, in a side sitting situation, the tilt of the video image can be applied according to the angle placed on the side to create a natural sitting situation. .

In addition, FIG. 6 is a flowchart explaining a process of arranging video images of video chat participants on a table virtual background when the video chat meeting place is a table conference according to an embodiment of the present invention.

Specifically, the integrated management server 300 can collect video images (photo screens) of table conference participants (S400). At this time, the information may be collected in real-time in the form of multiple images or videos through a webcam connected to the user terminal 100.

Additionally, the integrated management server 300 can extract the face from the video image and adjust it to a balanced average size (S402).

Here, the integrated management server 300 can adjust not only the size of the participant's face but also the size of the upper body. For example, upper body images including faces are obtained from images of multiple attendees, their sizes are calculated and the average size is calculated, and then the corresponding face and upper body images are calculated for participants whose size is smaller or larger than the average size. By adjusting to an average size, it is possible to provide participants with a realistic feeling of being in the same space.

As another example, the average face size in the participants' video images may be calculated, the face size may be adjusted based on this, and the upper body size may be adjusted according to this adjustment ratio.

Additionally, an image (photo) including a face is placed on a table as a virtual background (S404).

When placing a table, the size can be adjusted to provide a sense of perspective, and the table size can be automatically adjusted to the desired shape for a natural look in a video chat room. Furthermore, when switching to a seated screen at the table (by the user's seated button interface action), the table size can be automatically adjusted to the desired shape. By adjusting the screen tilt according to the side angle, the chat screen can be provided to the user in a natural way to match the virtual background (S406, S408).

Here, the integrated management server 300 can adjust the participant's seating angle for perspective in the case of a straight arrangement facing the front, but empty space may be created on the table and image screen due to angle adjustment. In this case, the integrated management server 300 can adjust the size of the table to cover the empty space between the participant and the table and the empty space between the participant's video screen and the virtual table object image.

Furthermore, instead of a virtual avatar that exists in the metaverse space before video chatting, you can use the video chat service by using an avatar with facial data synthesized from your own video video. In this way, you can use a video chat service with an avatar with facial data synthesized from the video video, etc. Even if a chat service is provided, an additional neural network learning algorithm can be used for image restoration in cases where the user's video image implemented in the metaverse space is not clear due to limitations in camera performance or screen distortion.

Specifically, by using generative adversarial networks, new images can be created or regenerated, which can be used to restore damaged images.

Unlike existing deep learning networks, generative adversarial networks are composed of multiple deep neural networks. Although they require dozens of times more calculations than existing deep neural network models to generate high-resolution images, they can provide excellent performance in image restoration. there is.

In addition, the generative adversarial network is an unsupervised learning-based generative model that adversarially trains two networks with a generator and a discriminator. Input data is input to the generator to create fake images similar to real images. It can be learned to do so. The input data may be a noise value. Noise values may follow any probability distribution. For example, it may be data generated by Zero-Mean Gaussian.

The discriminator can learn to distinguish between real images and fake images generated by the generator. More specifically, it can learn to have a high probability when inputting a real image, and to have a low probability when inputting a fake image. In other words, the discriminator can gradually learn to distinguish between real and fake images.

In addition, by using the generative adversarial network (GAN) algorithm to create fake images, you can implement your own avatar instead of your real photo in the virtual world of Metaverse. Through learning, the fake image can be closer to the real image. You can also create it.

Furthermore, among GANs, DC-GAN, which is a deep convolutional generative adversarial network, can be used to implement a deepfake avatar, and the form of the deepfake can resemble one's own face. Deepfakes can be carried out within an acceptable range, subject to agreement between the parties.

DC-GAN needs to construct a Cycle-GAN, which consists of two generators and two discriminators, and during the training period, two different sets of numerous images are sent as input, respectively, two DCGANs, that is, these two domains, It is represented by Y.

For one of the DCGANs, the input image x is a member of domain x).

Domain Y is the target domain of this GAN, and similarly for other GANs, the input image is y belonging to domain Y, and the generator (denoted by F) is also an image (F(y), which is difficult to distinguish from the display image). Therefore, becomes the target domain.

The discriminator of a GAN with generator G can be expressed as follows.

Since the goal is to distinguish fake images G(x) from real images in domain Y, GANs that differ as DY can be denoted as DX.

Additionally, by introducing cycle consistency loss (Lcyc), when an image is converted from one domain to another domain, when returning to this domain, the image returns to where it was last located. This means that it makes sense to map inputs and outputs, where the cycle consistency loss consists of two parts:

One is the forward cycle consistency loss (forward cyclic consistency loss) that returns to domain X via F(G(x)) = x (transforms the input image x from domain cycle-consistency loss), and the other is called backward cycle-consistency loss corresponding to G(F(y)) = y.

The chat management unit 330 can provide a video chat service to the connected user terminal 100 by utilizing the metaverse data, which is a virtual space corresponding to the video chat room, stored in the metaverse creation unit 320, and the number of people in the video chat room. can be set by the host, and one-to-one and one-to-many video meetings can be held freely, and the metaverse space to be provided depending on the number of people in the meeting can be selected and changed by the host to suit the nature of the meeting.

It can include various functions and user interfaces for video chatting services. In particular, the emotional state of the chatting speaker can be recognized through an object recognition-based emotion estimation algorithm during video chatting, and based on this, emotional changes can cause unpleasant state, If it is determined that the emotion persists, such as in an angry state, which may make continuous video chatting difficult because the other person is uncomfortable, a warning and alarm message is sent, and if it continues for longer than a standard time, the user can choose to block the screen and give penalty points. Functions may be included.

Specifically, video images are provided so that changes in the facial expressions of video chat users can be identified, and real-time emotional states can be determined from the facial images in the video video. A separate object recognition-based emotion estimation algorithm can be used for this purpose.

At this time, the object recognition-based emotion estimation algorithm first assumes that the video chat partner's image is provided in the form of real-time video to detect real-time or periodic changes depending on the emotional state, and the video chat partner's video image (at least the emotional state) The emotional state can be periodically determined using an object recognition-based emotion estimation algorithm through whole body images (including facial images).

In addition, the emotional state here is largely divided into an unpleasant state (fear, anger, fear, dissatisfaction, stiffness, sadness, helplessness, despair, etc.) and a pleasant state, and in situations where it is difficult to continue chatting (in case of an unpleasant state, a warning and alarm message is sent) It can be judged as a situation in which the user can choose to block the screen and give penalty points if it is transmitted and continues for longer than the standard time.

Here, if the speaker's emotion persists for more than a certain period of time or is repeated more than a certain number of times in one of the unpleasant states above, the speaker's current state may be judged to be unpleasant, and a warning and alarm message notifying this may be sent to the participant (who felt the unpleasant state). It can be sent to the participant, the participant who caused the other party an unpleasant state, and all participants attending the video conversation (meeting).

Afterwards, if the transmission of warning and alarm messages continues more than a certain number of times, the speaker who caused unpleasant feelings in the other party may be subject to screen blocking of the video conversation (conference) and a penalty (penalty point) may be given. This process can be performed when the consent of the participant who feels an unpleasant state is confirmed, that is, whether a specific other party is causing unpleasant emotions or whether a certain penalty is desired to be imposed on the other party.

If the chat is implemented only as a virtual avatar without an image of the chat partner, a separate video of oneself must be transmitted to the integrated management server 300 to start chatting, and the integrated management server 300 will periodically monitor the video and evaluate it. You can check your status, and if you do not receive or cannot receive a video video separately, you can determine your emotional state by judging preset unpleasant words and sentences based on chat text information. You can also check the emotional state of the avatar that contains the video video described above. In parallel with the information, you can more accurately identify the emotional state and determine that it is difficult to chat.

Object recognition within an image avatar can utilize an object recognition algorithm. For example, an object can be recognized by applying a CNN algorithm, an object recognition algorithm, from an avatar face image.

The CNN algorithm is a neural network algorithm that recognizes objects through convolution operations. In the present invention, the R-CNN (Region-based Convolutional Neural Networks) algorithm, which uses a deep learning regression method to specifically detect object locations, is used to identify objects. It can be recognized.

The R-CNN algorithm is a neural network algorithm that first creates a candidate region and then learns a CNN based on this to find the location of the object in the image. The object recognition process is a process of generating a candidate region using selective search in the input image; It includes the process of converting each candidate region to the same size, extracting features through CNN, and classifying objects in the candidate region using SVM (Support Vector Machine) using the extracted features. Because the position of the candidate area is not accurate, the object area box position can be accurately corrected through regression learning.

In addition, other neural network algorithms include Fast R-CNN, R-FCN, YOLO (You only Look Once), TensorFlow, and SSD (Single Shot MultiBox Detector). The neural network algorithms described above can be additionally applied or R- It can be applied as a replacement for CNN, and the object recognition speed can be improved by recognizing multiple objects simultaneously by applying the above-described neural network algorithm.

Furthermore, algorithms such as AdaBoost, Support Vector Machine (SVM), Linear Disciminant Analysis (LDA), and Principal Component Analysis (PCA) were added to increase the object recognition rate. All of these algorithmic techniques identify the recognition target area based on its appearance, detect the area around the object using a model trained by a set of video images to be used for training, and set various surrounding constraints. Because conditions are overcome through training, object recognition accuracy and reliability can be improved as a result.

In addition, after the chat, user evaluations of the chat service and chat partner (speaker) are directly entered and provided. Chat reliability is measured based on each collected evaluation result, and a reliability inquiry service can be provided so that it can be viewed during future chats. An evaluation screen may be provided on the user terminal 100 to measure the evaluation after the video chat ends.

The recommendation learning unit 340 utilizes an artificial intelligence-based recommendation algorithm to recommend a customized meeting location that matches the meeting tendency or type, or automatically selects a chat partner that matches the user's tendency based on the user's personal information collected during membership registration when matching chat partners. Features that can be recommended may be included.

Furthermore, if there is no information to recommend based on the user information after initially collecting user information at the time of membership registration, the integrated management server 300 uses the cold start of the recommendation algorithm to provide similar conditions to the user. User information can be searched and recommended meeting places or chat partners provided to users with the most similar conditions.

At this time, the recommended conditions are performed based on user information, and the user information collected for this purpose may include name, age, gender, personality, hobbies, interests, etc. Here, the virtually similar condition may be, for example, a condition in which the number of matching or similar pieces of information included in the user information is the greatest.

Additionally, the recommendation algorithm method applied to similar conditions can utilize an artificial intelligence algorithm using similarity analysis.

Furthermore, in addition to the similarity calculation method, matrix factorization is used to reduce the dimension of the interaction matrix and approximate it with two or more small matrices with k latent components, and to predict the rating of each user's item by multiplying the corresponding rows and columns. ) method can also be used.

In addition, the state space of the possible frequent item set is explored by utilizing association rules or mining rules in which frequently consumed items are connected to the edges of the graph, and branches of the infrequent search space are removed. Frequently used item sets are used to create rules, and these rules can also generate recommendations.

For example, user information is initially collected to provide a recommended chat service (automatic selection of customized video chat room space or automatic selection of chat partner) from the integrated management server 300 to the user terminal 100, and the collected user information Based on this, the chat service most suitable for the age group can be recommended, or a related rule can be created by considering the user's age, gender, hobbies, interests, etc. included in the user's user information, and a chat service that matches them can be recommended.

Furthermore, the recommendation learning unit 340 can provide a more detailed and accurate customized recommendation service based on big data through learning as the collected data on user information of the recommendation algorithm is added and increased.

The information protection department 350 builds a blockchain network in conjunction with multiple blockchain servers, generates public keys and private keys through the established internal blockchain network, converts them into hash values, and stores them distributedly. User authentication can be performed based on the public key and user information, which is the user's personal information provided during membership registration.

Furthermore, the information protection unit 350 can receive individual unique user information along with a public key from a plurality of user terminals 100 and generate a user certificate containing a hash value for the user information, and each user certificate. The storage method for can be achieved by a Merkle tree structure.

For example, each user certificate (transaction) is stored with a hash value in the lowest child node, and the Merkle root (parent node), the highest level of the Merkle tree, stores a hash value in an intermediate node on the path leading to the lowest child node. It is hashed and stored for sharing.

When determining the authenticity of the stored user certificate through this, the user certificate copied to the individual's user terminal 100 is compared with the user certificate in the database 360, and only the hash value hashed along the path of the Merkle tree is compared. This will be done.

At this time, since the comparison operation is performed on the path of the Merkle tree, there is no need to perform a comparison operation on the blocks of all nodes, so authenticity can be determined with a relatively easy amount of calculation, and forgery and falsification of transactions can be easily and quickly detected. , transactions can be easily verified even on the user terminal 100 in the form of a small-capacity portable terminal.

The database 360 can store and manage the data necessary to provide a metaverse-based video chat service in the integrated management server 300, such as user information provided at the time of membership registration and a plurality of metaverse data to be used in a video chat room. It may be included by default, and may also include control information, history information, event information, etc. required for the video chat service.

Figure 3 is a flowchart explaining the entire process of the metaverse-based video chat service method according to an embodiment of the present invention.

First, after signing up to use the Metaverse platform service for video chatting in a virtual space provided by the integrated management server 300 using the user terminal 100 owned by multiple users on the Metaverse platform of the present invention, Log in and connect (S100).

Next, in order to use the Metaverse platform service, you can select an avatar using a virtual or your own image, select a desired chat partner or video chat room, or receive automatic recommendations through the recommendation algorithm described above (S102).

Afterwards, a video chat service based on the Metaverse platform is performed in a video chat room in the 3D Metaverse space selected by the user or by automatic recommendation (S104).

In addition, during video chatting, the object recognition-based emotion estimation algorithm described above is used to recognize the state of each speaker through emotion recognition and determine whether to block it (S106).

Finally, after the video chat ends, the user terminal 100 provides the evaluation results of the video chat service and the chatting partner (speaker) to the integrated management server 300 to calculate the reliability of the chatting partner and provide video chatting service at a later date. When using it, you can check the trustworthiness of the chatting partner to other users and help them select a highly trustworthy partner to conduct a video chat (S108).

FIG. 4 is a flowchart specifically explaining the membership authentication process of the metaverse-based video chat service method of FIG. 3.

The user terminal 100 connects to the integrated management server 300 and requests membership by entering user information, which is personal information required for the video chat service (S200).

When requesting membership registration, the integrated management server 300 requests a mobile phone number, performs identity authentication by linking with a telecommunication company server (not shown) using the mobile phone number, and accepts membership upon completion of identity verification (S202).

Afterwards, the user terminal 100 may log in with the registered user information and receive additional input of interests, preferred occupations, hobbies, etc. for recommending metaverse meetings customized to the user (S204).

Finally, the integrated management server 300 can complete the membership registration by finally accepting the membership registration after final confirmation of the user information entered at the time of membership registration (S206).

FIG. 5 is a flowchart specifically explaining the service evaluation and user permission setting process of the metaverse-based video chat service method of FIG. 3.

First, the integrated management server 300 can collect the online meeting chat evaluation results entered by the user (S300).

The integrated management server 300 may determine whether the collected chat evaluation results are greater than or equal to a preset standard value (S302).

Here, the standard value may be a score corresponding to the standard value obtained by scoring the chat evaluation results, and the integrated management server 300 sums up the evaluation results of all users and determines that the corresponding score is trustworthy if the score (or star rating) is higher than the standard value. Additional privileges can be granted to the video chat room host by increasing the given level.

In other words, users who use all chat services basically have a level, and can increase their level starting from the initial level 1, and can be granted differentiated rights depending on the level.

Specifically, if the chat evaluation result is higher than the above standard value, permission to hold an offline meeting can be granted after the chat host's level is raised (S304).

Additionally, if the chat evaluation result is below the standard value, permissions may be restricted after a warning and level reduction (S306).

In addition, the chat evaluation can be divided into an evaluation of the video chat meeting and an evaluation of the chat partner, and the specific contents include whether it is the same as the meeting introduction, the host's expertise, the meeting atmosphere, willingness to participate in the offline meeting, meeting satisfaction, Chat status, profile introduction, identity, conversation manners, matching satisfaction, etc. may be included.

The integrated management server 300 can complete setting of differential authority according to level by storing and updating user information data including level information in the DB (S308).

In this specification, ‘terminal’ may be a wireless communication system that guarantees portability and mobility, and may be any type of handheld-based wireless communication system such as a smartphone, tablet PC, or laptop, for example. Additionally, the ‘terminal’ may be a wired communication system such as a PC that can connect to other terminals or servers through a network. In addition, a network refers to a connection structure that allows information exchange between nodes such as terminals and servers, including a local area network (LAN), a wide area network (WAN), and the Internet (WWW). : World Wide Web), wired and wireless data communication networks, telephone networks, and wired and wireless television communication networks.

Examples of wireless data communication networks include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, and ultrasound. This includes, but is not limited to, communication, Visible Light Communication (VLC), LiFi, etc.

100: User terminal
300: Integrated management server
310: Department of Communications
320: Metaverse generation unit
330: Chat management department
340: Recommended learning department
350: Information Protection Department
360: Database
400: Network

Claims (8)

Multiple user terminals to participate in video chatting are connected through a network, and a metaverse-based video chatting service is provided between the connected user terminals,
It includes an integrated management server that provides video chatting services through user video captured in real time from the plurality of user terminals in a metaverse space implemented in a 3D virtual reality space similar to reality,
The integrated management server is,
a communication unit that includes a communication protocol that allows a user terminal to connect through the network and transmits and receives information for providing a video chat service;
a metaverse generator that creates a metaverse, an online virtual space for performing video chat services;
A chat management unit capable of providing a video chat service to a connected user terminal using metaverse data, which is a virtual space corresponding to the video chat room stored in the metaverse creation unit;
Recommendation that utilizes an artificial intelligence-based recommendation algorithm to recommend a customized meeting place that matches the tendency or type of meeting, or automatically recommends a chat partner that matches the user's interests and tendencies based on the user's personal information collected during membership registration when matching chat partners. Learning Department;
It includes a database that stores and manages data necessary to provide a metaverse-based video chat service in the integrated management server,
The metaverse generator,
In order to produce a natural screen when compositing video images taken from a user terminal on a virtual background during video chatting, only the background part is chroma keyed to the original video image of each user received and then transmitted to the screen of each user terminal participating in the video chatting. A screen is provided to match the virtual background,
The metaverse generator,
Obtain an upper body image including the face from the video image and adjust the face size and upper body size to match the average size.
A video image including a face is placed on the virtual background, the size of the virtual background is adjusted to provide a sense of perspective, and the screen tilt of the 3D virtual space is adjusted according to the side angle of the video image,
Using an avatar synthesized with facial data of the user video image existing in the metaverse space before the video chat, if the user video image is not clear, image restoration is performed using a neural network learning algorithm
Metaverse-based video chat service system. delete In claim 1,
The chat management unit can recognize the emotional state of the speaker conducting a video chat through an object recognition-based emotion estimation algorithm during chatting, and if the recognized emotional state is unpleasant and persists for more than a certain period of time or a certain number of times, a warning and alarm are issued. A metaverse-based video chat service system that transmits a message and, when the warning and alarm message transmission continues for more than a certain number of times, blocks the screen and assigns penalty points to the speaker who caused unpleasant feelings in the other party. delete In the metaverse-based video chat service method using the integrated management server of claim 1 or claim 3,
A plurality of user terminals registering as members and logging in to use the video chat service provided by the integrated management server;
After logging in, the plurality of user terminals each select an avatar using a virtual or user image to use the metaverse-based video chat service, select a chat partner or a metaverse video chat room space, or, upon request, use the integrated management server. A step of automatically recommending a chat partner or a Metaverse video chat room space through a recommendation algorithm; and
The user terminal performs a video chat service in a video chat room in the metaverse space selected by the user or by automatic recommendation.
A metaverse-based video chat service method including. In claim 5,
During video chatting, the integrated management server uses an emotion estimation algorithm that can recognize the emotional state of the speaker conducting the video chat based on object recognition, and recognizes the state of each speaker through emotion recognition and determines whether to block it; and
After the video chat ends, the user terminal provides the evaluation results of the video chat service and chat partner to the integrated management server to calculate the reliability of the chat partner, and when using the video chat service in the future, other users can inquire about the chat partner's reliability. steps to enable
A metaverse-based video chat service method further comprising: In claim 6,
The integrated management server collecting online meeting chat evaluation results input by the user;
The integrated management server determining whether the collected chat evaluation result is greater than or equal to a preset standard value;
If the chat evaluation result is higher than the reference value, the integrated management server grants permission to hold an offline meeting after raising the level of the chat host;
If the chat evaluation result is below a standard value, the integrated management server issues a warning, lowers the level, and then restricts authority; and
The integrated management server stores and updates user information data including level information in the database to complete setting differential authority according to level.
A metaverse-based video chat service method further comprising: In claim 5,
The user terminal accesses the integrated management server and requests membership registration by inputting user information, which is personal information required for the video chat service;
The user terminal logs in with registered user information and receives additional information about interests, preferred occupations, and hobbies for customized video chatting; and
The integrated management server finally accepts the membership registration after final confirmation of the user information entered at the time of membership registration and completes the membership registration.
A metaverse-based video chat service method further comprising: