TWI842650B - System, method and computer program product for assisting multiple users to choice activity in virtual world - Google Patents

Abstract

A system, a method and a computer program product for assisting multiple users to choice an activity in a virtual world are provided. Speeches and motions of a plurality of VR devices accessing a virtual world are detected. a semantic of a speech is recognized by using an AI semantic model, and a motion is recognized by using an AI motion model. A recognition result of the semantics, a recognition result of the motion, and an association between the recognition result of the semantics and the recognition result of the motion are analyzed to produce an analyzation result. A recommended activity is determined from an activity database according to the analyzation result. A virtual image assistant in the virtual world provides the recommended activity, thereby transferring the VR devices to a scene of the recommended activity in the virtual world.

Description

Systems, methods and computer program products for assisting multiple people in selecting activities in a virtual world

This case is related to the field of virtual reality, and more specifically, to a system, method and computer program product for assisting multiple people in selecting activities in a virtual world.

In the application of the Metaverse virtual world, when multiple people in the same group enter the virtual world (such as the Metaverse platform) with a virtual reality (VR) device, because the Metaverse provides too many virtual activities, and when multiple people want to have fun together at the same event venue, there are many opinions, so it is difficult to decide on an activity suitable for this group.

Although current technology can help recommend suitable activities, it usually provides canned answers or preset questions for users to choose from. In addition, because the virtual world of the Metaverse exists digitally in the cloud, both activities and scenes can be infinitely extended, and there will be many types of activities. Although there is technology that allows participants to ask questions by voice, it usually only captures defined keywords for search. This method cannot accurately, quickly or accurately recommend the most suitable activities.

Therefore, how to assist multiple people in discussing in the virtual world to accurately, quickly or accurately recommend suitable activities is a problem to be solved in this field.

To solve the above problems and other problems, this case discloses a system, method, computer program product, and computer-readable recording medium for assisting multiple people in selecting activities in a virtual world.

The system disclosed in this case for assisting multiple people in selecting activities in a virtual world includes: a virtual image assistant module for displaying a virtual image assistant in the virtual world; a voice and semantic detection module for detecting the voices of multiple virtual reality devices entering the virtual world and the semantics of the voices; a gesture and action detection module for detecting the gestures and actions of the multiple virtual reality devices entering the virtual world; an AI recognition module for recognizing the voice, the semantics, the gestures, and the actions to determine whether the recognition result of the voice is an assistive voice, and using an AI semantic model to determine whether the recognition result of the semantics is an assistive voice. For the question semantics, the AI action model is used to determine whether the recognition result of the gesture is to open the collaborative gesture, and the AI action model is used to determine whether the recognition result of the action is an approval action; the activity analysis module is used to analyze the recognition result of the semantics of the AI semantic model, the recognition result of the action of the AI action model, and the correlation between the recognition result of the semantics and the recognition result of the action; and the activity determination module is used to determine the recommended activity from the activity database according to the analysis result of the activity analysis module, so that the virtual image assistant displayed in the virtual world provides the recommended activity.

The method disclosed in this case for assisting multiple people in selecting activities in a virtual world includes: detecting the voices of multiple virtual reality devices entering the virtual world and the meaning of the voices; detecting the gestures and actions of the multiple virtual reality devices entering the virtual world; recognizing the voices and the meanings to determine whether the recognition result of the voices is an assist voice, and using an AI semantic model to determine whether the recognition result of the meanings is a question meaning; using an AI action model to recognize the gestures and actions, and using the AI action model to determine whether the recognition result of the meanings is a question meaning; Determine whether the recognition result of the gesture is a collaborative gesture, and use the AI action model to determine whether the recognition result of the action is an approval action; analyze the recognition result of the semantics of the AI semantic model, the recognition result of the action of the AI action model, and the correlation between the recognition result of the semantics and the recognition result of the action to generate an analysis result; and determine a recommended activity from the activity database based on the analysis result, so that the virtual image assistant provides the recommended activity in the virtual world.

The computer program product disclosed in this case assists multiple people in selecting activities in a virtual world, and the method disclosed in this case assists multiple people in selecting activities in a virtual world by loading the program into a computer.

The computer-readable recording medium disclosed in this case for assisting multiple people in selecting activities in a virtual world stores instructions, and can use a computing device or a computer to execute the computer-readable recording medium through a processor and/or a memory, so as to execute the method for assisting multiple people in selecting activities in a virtual world disclosed in this case when executing the computer-readable recording medium.

In one embodiment, the AI semantic model is trained using a question-and-answer set related to an activity as training data, so that when the semantic recognition result is the question semantics, the AI semantic model generates the corresponding answer semantics, and the virtual image assistant provides answers in the virtual world based on the answer semantics through voice, pictures, videos, or event viewing windows.

In one embodiment, the AI action model is trained using the action data of the virtual reality device as training data, so that when the virtual image assistant provides the answer, the AI action model recognizes the reaction action of each virtual reality device to the answer and the AI semantic model recognizes the reaction semantics of each virtual reality device to the answer, and the AI action model recognizes the reaction action of each virtual reality device to the answer. The recognition module records the recognition results of each reaction action, the number of virtual reality devices that provide each reaction action, the recognition results of each reaction semantics, the number of virtual reality devices that provide each reaction semantics, and the correlation between the recognition results of each reaction action and the recognition results of each reaction semantics, so as to provide the activity analysis module for analysis.

In one embodiment, the AI action model is trained using various gesture images as training data, so that when the gesture and action detection module detects the gestures of the multiple virtual reality devices and the AI action model recognizes the gesture as turning on the assist gesture, or when the voice and semantic detection module detects the semantics of the multiple virtual reality devices and the AI semantic model recognizes the semantics as turning on the assist semantic, the virtual image assistant appears in the virtual world.

In one embodiment, the method for assisting multiple people in selecting activities in a virtual world disclosed in this case further includes an activity control module, wherein if the recommended activity determined by the activity determination module is ongoing in the virtual world, the activity control module transfers the multiple virtual reality devices into the recommended activity in the virtual world, and if the recommended activity determined by the activity determination module is not ongoing in the virtual world, the activity control module adds the recommended activity to the calendar, and notifies the multiple virtual reality devices before the recommended activity starts.

In one embodiment, the method for assisting multiple people in selecting activities in a virtual world disclosed in this case further includes an activity preview module, which provides an activity live viewing window to play the recommended activity or other activities, so that when it is detected that the virtual reality device in the virtual world hits or touches the virtual reality device in the virtual world, the virtual reality device corresponding to the virtual reality device is allowed to transfer to the recommended activity or the other activity in the virtual world.

Through the system, method, computer program product, and computer-readable recording medium disclosed in this case for assisting multiple people in selecting activities in a virtual world, a virtualized intelligent assistant is provided to assist multiple participants in discussing activities, and the voice or actions of multiple participants in the activity discussion are recorded for calculation and analysis through VR devices, and then searched in the activity database to determine recommended activities, and the participants of the VR device are moved to the venue of the recommended activity, so that it is more convenient and comfortable to immerse in virtual world activities.

11, 12: VR device

20: System

21: Avatar Assistant Module

22: Voice and semantic detection module

23: Gesture and motion detection module

24: AI recognition module

241:AI semantic model

242: AI action model

25: Activity analysis module

26: Activity decision module

27: Activity control module

28: Event preview module

30:Activity database

S201~S207: Steps

S301~S303: Steps

S401~S405: Steps

S501~S504: Steps

S601~S605: Steps

Figure 1 is a schematic diagram of the architecture of the system that assists multiple people in selecting activities in the virtual world.

Figure 2 is a schematic diagram of the steps of the method for assisting multiple people in selecting activities in a virtual world.

FIG3 is a schematic diagram of the steps of enabling assistance in the method of assisting multiple people to select activities in a virtual world in this case.

Figure 4 is a schematic diagram of the steps of discussing activities in the method of assisting multiple people in selecting activities in a virtual world in this case.

FIG5 is a schematic diagram of the steps after the question and answer session is terminated in the method of assisting multiple people in selecting activities in a virtual world in this case.

FIG6 is a schematic diagram of the steps of transitioning to recommending activities in the method of assisting multiple people in selecting activities in a virtual world in this case.

The following specific examples are used to illustrate the implementation of this case. People familiar with this technology can easily understand the other advantages and effects of this case from the content disclosed in this article. The structures, ratios, sizes, etc. shown in the attached figures of this manual are only used to match the content disclosed in the manual for people familiar with this technology to understand and read, and are not used to limit the conditions under which this case can be implemented. Therefore, any modification, change or adjustment should still fall within the scope of the technical content disclosed in this case without affecting the effects and purposes that can be achieved by this case.

As used herein, the terms "include", "comprising", "having", "containing" or any other variations thereof are intended to cover a non-exclusive inclusion. Unless otherwise indicated, singular forms such as "a", "an", "the" may also be used in the plural, and "or", "and/or" and the like may be used interchangeably.

Please refer to Figure 1, which is a schematic diagram of the architecture of the system for assisting multiple people in selecting activities in the virtual world. The system 20 includes a virtual image assistant module 21, a voice and semantic detection module 22, a gesture and action detection module 23, an artificial intelligence (AI) recognition module 24, an activity analysis module 25, an activity decision module 26, an activity control module 27, and an activity preview module 28.

The system 20 is applied to a virtual reality device (i.e., VR device) 11, 12 that enters a virtual world and connects to an activity database 30 to search for a recommended activity. The VR devices 11, 12 herein may refer to a mobile phone with a VR APP or a device that can load a VR APP to present a VR image, a VR wearable device, a VR wearable device with a handheld controller, a VR wearable device with an internal positioning device, or a VR wearable device with an external positioning device, etc. In addition, the system 20 may be, for example, a server, in which each module or unit may be software, hardware or firmware; if it is hardware, it may be a processing unit, processor, or computer host with data processing and computing capabilities; if it is software or firmware, it may include instructions executable by a processing unit, processor, computer or computer host, and may be installed on the same hardware device or distributed on different multiple hardware devices.

The virtual image assistant module 21 is used to make the virtual image assistant appear in the virtual world, or to display the virtual image assistant in the virtual world. For example, when the virtual avatar of the VR device 11, 12 in the virtual world speaks, the virtual image assistant will turn its head toward the virtual avatar to present a listening posture, and when the virtual image assistant answers, it will also answer the question facing the virtual avatar that asked the question.

The voice and semantic detection module 22 is used to detect the voice and the semantics of the VR devices 11 and 12 entering the virtual world. In other words, when a virtual avatar in the virtual world is detected to speak, the voice and semantic detection module 22 can convert the detected voice into semantic text for processing by the AI recognition module 24. In one embodiment, the technology for converting speech into semantics is, for example, natural language analysis technology. Through deep learning semantic analysis technology, the user's diverse, colloquial long sentences can be understood. For example, the semantics related to assisting in participating in activities can be understood from the speech of VR devices 11 and 12, and then the semantic text is provided to the AI recognition module 24 to input it into the AI semantic model 241.

The gesture and action detection module 23 is used to detect the gestures and actions of the VR devices 11 and 12 entering the virtual world. In other words, the VR devices 11 and 12 can transmit images or data such as gestures and actions to the system 20, and the system 20 transmits them to the AI recognition module 24 for processing.

The AI recognition module 24 is used to recognize the voice and semantics from the voice and semantic detection module 22 and recognize the gestures and actions from the gesture and action detection module 23, so as to use the AI semantic model 241 to determine whether the recognition result of the semantics is a question semantics or a termination question and answer semantics, and use the AI action model 242 to determine whether the recognition result of the gesture is to open the collaborative gesture or to end the collaborative gesture, and use the AI action model 242 to determine whether the recognition result of the action is an approval action. In one embodiment, the AI recognition module 24 analyzes the activity-related keywords based on the semantics provided by the voice and semantic detection module 22 to become activity keywords for subsequent analysis by the activity analysis module 25.

In one embodiment, the AI semantic model 241 can be trained using a question-and-answer set related to an activity as training data. In one embodiment, the AI action model 242 can be trained using virtual reality device action data as training data. In one embodiment, the AI action model 242 can be trained using various gesture images as training data.

In detail, when the gesture and action detection module 23 detects the gesture of the VR devices 11 and 12 and the AI action model 242 recognizes the gesture as turning on the collaboration gesture, the virtual image assistant module 21 displays the virtual image assistant in the virtual world. When the collaboration gesture is recognized to end, the virtual image assistant is no longer displayed in the virtual world. In addition, when the voice and semantic detection module 22 detects the voice of the VR device 11, 12, and the AI recognition module 24 recognizes that the assisting voice is turned on, the virtual image assistant module 21 displays the virtual image assistant in the virtual world, and when the assisting voice is recognized to be ended, the virtual image assistant is no longer displayed in the virtual world. In other words, the virtual image assistant can be called by voice or gesture in the virtual world.

Then, when the semantics of the AI semantic model 241 is recognized as the question semantics, the AI semantic model 241 generates the corresponding answer semantics, and the virtual image assistant provides an answer in the virtual world based on the answer semantics through voice, pictures, videos, or an event live viewing window. In addition, when the virtual image assistant provides the answer, the AI action model 242 can recognize the reaction action of the VR devices 11 and 12 to the answer and the AI semantic model 241 can recognize the reaction semantics of the VR devices 11 and 12 to the answer, or when any VR device 11 and 12 asks a question, the AI action model 242 can recognize the reaction action of the other VR devices 11 and 12 to the question and the AI semantic model 241 can recognize the reaction semantics of the other VR devices 11 and 12 to the question. Next, the AI recognition module 24 records the recognition results of each reaction action, the number of VR devices 11, 12 providing each reaction action, the recognition results of each reaction semantics, the number of VR devices 11, 12 providing each reaction semantics, and the correlation between the recognition results of each reaction action and the recognition results of each reaction semantics. For example, the action or gesture with the recognition result of approval or recognition is associated with the voice (whether it is a question, answer, or discussion, etc.) appearing in the virtual world at the moment.

Furthermore, when the semantics recognition result of the AI semantics model 241 is the termination of the question-and-answer semantics, the AI model 24 transmits all the data recorded above, that is, the recognition results of all the semantics and actions (or gestures) discussed about the activity before the termination of the question-and-answer semantics, the correlation between each other, the time of providing the semantics and actions (or gestures), the VR device of the semantics and actions (or gestures), etc., to the activity analysis module 25. Here, the semantics and actions (or gestures) include various semantics and actions (or gestures) about questions, answers or reactions.

The activity analysis module 25 is used to analyze the semantic recognition results of the AI semantic model 241, the action recognition results of the AI action model 242, and the correlation between the semantic recognition results and the action recognition results. In detail, the activity analysis module 25 analyzes and calculates all the data recorded in the discussion of this activity, including the recognition results of semantics (such as question semantics, answer semantics) and actions or gestures (such as approval actions, hand gestures), the recognition results of semantics or actions or gestures reflected in the question semantics or answer semantics, the correlation between them, the time of providing semantics and actions or gestures, and the VR device of semantics and actions or gestures, to generate activity keywords and weighted data related to the activity, and then combine them into a weighted data screening formula. In one embodiment, the activity analysis module 25 records the semantics of all VR device users participating in the activity and selects activity-related words as activity keywords through AI semantic analysis technology. The activity keywords are the main input values for subsequent searches in the activity database 30. The activity analysis module 25 determines whether there are actions of approval or recognition of other people through VR devices when the activity keyword is recorded, and identifies the weighted score defined for each action for weighted calculation.

The activity decision module 26 is used to determine the recommended activity from the activity database 30 according to the analysis result of the activity analysis module 25, so that the virtual image assistant can provide the recommended activity in the virtual world. In detail, the activity decision module 26 converts the activity keywords and weighted data generated by the activity analysis module 25 into a search syntax to search the activity database 30. Finally, the activity with the highest average score obtained by the weighted average algorithm is the recommended activity for this activity discussion. In one embodiment, the activity analysis module 25 combines the activity keywords related to the activity discussion with the weighted data based on the number of occurrences of the activity keywords and the currently recorded like actions into a weighted data screening formula (for example, if the activity keyword "concert" is recorded five times (one point per time), and three of the five times other people have like actions recorded (like actions +2 points, so 2x3=6, plus 6 points), then the activities searched using "concert" are weighted 11 points), and the activity decision module 26 then converts it into a search syntax and searches the activity database 30, so as to finally search out the activity with the highest average score according to the weighted average algorithm, which is the recommended activity for this activity discussion.

In addition, if the recommended activity determined by the activity determination module 26 is ongoing in the virtual world, the activity control module 27 transfers the VR devices 11 and 12 to the recommended activity in the virtual world; if the recommended activity determined by the activity determination module 26 is not ongoing in the virtual world, the activity control module 27 adds the recommended activity to the calendar and notifies the VR devices 11 and 12 before the recommended activity starts. In one embodiment, the activity control module 27 may send a text message to the participants one day or 30 minutes before the recommended activity, and the text message content may include a link to the activity venue. The participants of the VR devices 11 and 12 can directly join the activity by clicking on the link to join the activity. After the participants click on the link to join the activity, the activity control module 27 will notify other participants who participated in the discussion and were already at the activity venue at that time.

The activity preview module 28 can provide an activity live viewing window to play an activity, and when it is detected that the virtual avatar of the VR device 11, 12 in the virtual world hits or touches the activity live viewing window, the VR device 11, 12 corresponding to the virtual avatar is allowed to transfer to the activity in the virtual world. In addition, in addition to using the event live viewing window to play the ongoing event, the event preview module 28 can also use another event live viewing window to play the other event when there are other events to be participated in or recommended at the same time. Participants can switch to the other event live viewing window through calling, and when the virtual avatar hits the event live viewing window, it can be instantly transferred to the scene of the other event.

According to one or more embodiments of the system for assisting multiple people in selecting activities in a virtual world shown in FIG. 1 , a group of people wearing VR devices 11 and 12 enter the virtual world of the metaverse and want to have fun together in the same scene. They find that there are many types of activities in the virtual world and there are many opinions because there are many people in the group. At this time, they can call a virtual image assistant to assist. Through the question-and-answer interaction with the virtual image assistant, the system 20 will also calculate the screening activity conditional formula based on the content of the questions and answers and the response voices and response actions of multiple people during the question-and-answer session, and finally calculate the screening activity conditional formula based on the recorded or stored data, and search the activity database 30 to generate a recommended activity and assist multiple people to participate in the recommended activity together.

Please refer to Figure 2, which is a schematic diagram of the steps of the method for assisting multiple people in selecting activities in a virtual world. It should be noted that the method shown in Figure 2 can be executed by a server or a computer device.

Step S201 and step S202 detect the voice and the meaning of the voice of multiple VR devices in the virtual world and detect the gestures and actions of multiple VR devices in the virtual world. Specifically, when multiple people wear VR devices to enter the virtual world, the server detects the voice, action and gestures of the VR device participants, wherein the voice can be received by the VR device and transmitted to the server, and the gestures and/or actions can be detected by the VR device and transmitted to the server in the form of signals.

Step S203 and step S204, use AI semantic model to identify semantics and use AI action model to identify gestures and/or actions. In one embodiment, the server converts into semantics through semantic analyzer, semantic cloud, or artificial intelligence model with similar functions to extract keywords therein, and then uses AI semantic model to identify semantics and keywords therein. In another embodiment, the recognition result of voice can be to open the assisting voice or to end the assisting voice, the recognition result of semantics can be, for example, question semantics or to end the question and answer semantics, the recognition result of gestures can be to open the assisting gesture or to end the assisting gesture, and the recognition result of actions can be an approving action. The identification steps are shown in steps S301, S302, S401, S402, S404, S501, and S502 described below.

Step S205, analyzing the semantic recognition results of the AI semantic model, the action recognition results of the AI action model, and the correlation between the semantic recognition results and the action recognition results to generate analysis results. In one embodiment, the correlation may be, for example: the time and VR device of providing voice/semantics, the time and VR device of providing actions, the reaction semantics or reaction actions of other VR devices when a VR device provides voice/semantics or actions, the reaction actions or semantics of each VR device and the time of reaction when the virtual image assistant provides answers, and the keywords that each VR device has responded to. In another embodiment, the analysis results are activity keywords and weighted data related to the activity. In detail, the system records the semantics of all VR device users participating in the event, and selects activity-related words as activity keywords through AI semantic analysis. Activity keywords are the main input values for subsequent searches in the activity database. Each activity keyword is weighted by the number of times it appears in the discussion and the actions of other people through VR devices when the activity keyword is recorded. It determines whether there is an action of approval or recognition, and identifies the weighted score defined for each action for weighted calculation.

The analysis steps are, for example, steps S405 and S503 described later.

Step S206, based on the analysis results, a recommended activity is determined from the activity database. In one embodiment, the activity keywords recorded during the discussion of the activity are weighted according to the number of occurrences of the activity keywords and the currently recorded like actions, and then combined into a weighted data screening formula (for example, the activity keyword "concert" is recorded and recorded five times (one point per time), and three of the five times other people's like actions are recorded (like actions +2 points) points, so 2x3=6, add 6 points), then use the "concert" search to weight the activities by 11 points and convert them into search syntax, and search the activity database to finally find the activity with the highest average score according to the weighted average algorithm, which is the recommended activity discussed in this event. The steps for determining the recommended activities are, for example, steps S504 and S601 described below.

Step S207, allowing the virtual image assistant to provide recommended activities in the virtual space. In one embodiment, the recommended activities can be displayed in the form of a viewing window, so that when the virtual avatar in the virtual world hits or touches the viewing window of the activity site, it can be transferred to the recommended activity site in the virtual world. The step of providing recommended activities is, for example, steps S602~605 described later.

Please refer to Figures 3 to 6, which are schematic diagrams of the steps of starting assistance, discussing activities, ending the Q&A session, and moving to recommending activities in the method of assisting multiple people in selecting activities in the virtual world.

First, steps S301 to S303 illustrate that after recognizing the voice and gestures for starting assistance, the virtual image assistant begins to assist the participants in the same group to discuss the activity.

Step S301, recognize the voice or gesture of the VR device; Step S302, the recognition result of the voice is to turn on the assistant meaning, or the recognition result of the gesture is to turn on the assistant gesture; Step S303, the virtual image assistant appears in the virtual space.

Secondly, steps S401 to S405 explain that all voice or meaning (whether it is the answer of the virtual image assistant, the question or reaction of the VR device participant), action or gesture (whether it is the question or reaction of the VR device participant), time point, keywords, etc. in the activity discussion will be recorded for subsequent analysis.

Step S401, identify the semantics of VR devices; Step S402, the semantics identification result is the question semantics; Step S403, generate the corresponding answer semantics, which can be answered by the virtual image assistant through voice, pictures, videos, or event viewing windows according to the answer semantics; Step S404, identify the reaction semantics and reaction actions (such as the semantics or actions of approval) of all VR devices to the question semantics or answer semantics; Step S405, record the identification results of each action, the number of VR devices providing each action, the identification results of each semantics, the number of VR devices providing each semantics, and the correlation between the identification results of each action and the identification results of each semantics for subsequent analysis.

Furthermore, steps S501 to S504 explain: analyzing all the questions, answers and responses discussed in this activity, calculating the weighted data after the analysis, and then searching the activity database for recommended activities.

Step S501, identify the semantics of the VR device; Step S502, the semantics identification result is the termination of the question and answer semantics; Step S503, analyze all the data recorded before the termination of the question and answer semantics; Step S504, determine the recommended activities from the activity database based on the analysis results.

In addition, steps S601 to S605 explain that when a recommended event is generated, the virtual image assistant can teleport all members of the same group to the venue of the recommended event, or assist participants in adding the event to the calendar, and notify VR device participants via SMS when the event starts, and can also directly join the event through a link.

Step S601, determine the recommended activity from the activity database according to the analysis result; Step S602, determine whether the recommended activity is in progress, if the recommended activity is in progress, proceed to step S603, otherwise, if the recommended activity is not in progress, proceed to step S604; Step S603, transfer multiple VR devices to the recommended activity in the virtual space; Step S604, add the recommended activity to the calendar; Step S605, notify multiple VR devices before the recommended activity starts.

In addition, an event viewing window can be provided to play a recommended event or other ongoing event, and when it is detected that the virtual avatar of the VR device in the virtual world hits or touches the event viewing window, the VR devices 11 and 12 are allowed to transfer to the recommended event or other ongoing event in the virtual world.

In addition to one or more of the above embodiments, this case provides a computer program product, which executes the above method after the program is loaded into a computer; in addition, the computer program (product) can be stored in a recording medium and can also be directly transmitted and provided on the Internet. The computer program (product) is a thing that carries a program that can be read by a computer and is not limited to an external form. The computer includes but is not limited to an electronic device with a processor, such as a mobile phone or a tablet; In addition, this case also provides a computer that can The readable recording medium is applied to a computing device or computer having a processor and/or memory, and the computer-readable recording medium stores instructions, and the computing device or computer can execute the computer-readable recording medium through the processor and/or memory to execute the above method and/or content when executing the computer-readable recording medium. The computer-readable recording medium (such as a hard disk, a floppy disk, an optical disk, a USB flash drive) stores the computer program (product).

Based on one or more embodiments of the system, method, computer program product, and computer-readable recording medium for assisting multiple people in selecting activities in a virtual world, an example of a specific embodiment is provided below.

Many people in the same group put on VR devices and entered the Metaverse. Many friends wanted to go to the same venue to participate in activities together, but after entering the virtual world, they found that there were too many activities to choose from, and many of them had their own favorites. At this time, Xiaohua said that we should ask the "Smart Assistant" to help us choose. At this time, the "Smart Assistant" appeared in the form of a non-player character (NPC) and began to help everyone choose activities suitable for everyone in the Metaverse.

Xiaohua first said that he wanted to participate in sports activities. Can you recommend similar activities? At this time, the "Smart Assistant" listed several sports activities and introduced them one by one with voice, pictures, and videos. One of the activities was still in progress, so everyone could see the actual images and sounds of the activity through the real-time window. At this time, when the smart assistant introduced the activities, it would detect the voices and actions of multiple people and store the correlation between voices and actions. Then Xiaoming said that the dodgeball activity seemed good. At this time, Xiaohua and Xiaozhang used VR devices to give thumbs-up actions. During these discussions, the semantics and VR actions related to the activity selection will be detected and the correlation data will be recorded.

Next, the "Activity Smart Assistant" stores the question and answer semantics and related reaction actions related to the activity in the discussion just now, and performs weighted calculations based on the semantics or VR feedback actions that others agree with at the time, and calculates a weighted average calculation formula. Based on this calculation formula, it screens the activity database and finally decides to recommend the ongoing "Dodgeball Activity".

After recommending the ongoing "dodgeball activity", the smart assistant immediately transported the group to the "dodgeball activity" venue and determined the best landing point for the group.

In summary, unlike the canned answers or preset questions for users to choose from in the past, this solution allows participants in the virtual world metaverse to discuss together and interact with the virtual image assistant, so that the assistant can help them discuss and provide the most suitable activities (i.e., decide on a recommended activity) to participate in the activity. In addition, in addition to interacting with the virtual image assistant, this solution uses the virtual image assistant to not only detect voice problems of the VR device, but also observe and record the voice and actions related to the interaction between the group participants, understand the voice as meaning, and then identify what kind of meaning it is, provide corresponding answers, and then weight them with gestures or actions to increase the accuracy of the recommended activities. In addition to providing the most suitable activities, this case can also teleport all participants of the same group to the event venue, and can also assist participants in arranging the event schedule.

The above embodiments are only illustrative of the effects of this case, and are not intended to limit this case. Anyone familiar with this technology can modify and change the above embodiments without violating the spirit and scope of this case. Therefore, the scope of protection of this case should be as listed in the scope of the patent application described below.

S201~S207: Steps

Claims (11)

A system for collaborative multi-person discussion in a virtual world, comprising: Virtual image assistant module, used to display the virtual image assistant in the virtual world; The voice and semantic detection module is used to detect the voice and semantics of multiple virtual reality devices entering the virtual world; A gesture and action detection module, used to detect the gestures and actions of the multiple virtual reality devices entering the virtual world; The AI recognition module is used to recognize the voice, the meaning, the gesture, and the action to determine whether the recognition result of the voice is the voice to turn on the assistance, use the AI semantic model to determine whether the recognition result of the meaning is the question meaning, use the AI action model to determine whether the recognition result of the gesture is the gesture to turn on the assistance, and use the AI action model to determine whether the recognition result of the action is an approval action; An activity analysis module is used to analyze the semantic recognition result of the AI semantic model, the action recognition result of the AI action model, and the correlation between the semantic recognition result and the action recognition result to generate an analysis result; and The activity determination module is used to determine the recommended activities from the activity database according to the analysis results of the activity analysis module, so that the virtual image assistant displayed in the virtual world can provide the recommended activities. The system as claimed in claim 1, wherein the AI semantic model is trained using a question-and-answer set related to an activity as training data, so that when the semantic recognition result is the question semantics, the AI semantic model generates a corresponding answer semantics, and the virtual image assistant provides an answer in the virtual world based on the answer semantics through voice, pictures, videos, or an event viewing window. The system as claimed in claim 2, wherein the AI action model is trained using virtual reality device action data as training data, so that when the virtual image assistant provides the answer, the AI action model recognizes the response action of each virtual reality device to the answer and the AI semantic model recognizes the response semantics of each virtual reality device to the answer, The AI recognition module records the recognition results of each reaction action, the number of virtual reality devices that provide each reaction action, the recognition results of each reaction semantics, the number of virtual reality devices that provide each reaction semantics, and the correlation between the recognition results of each reaction action and the recognition results of each reaction semantics, so as to provide the activity analysis module for analysis. The system as claimed in claim 1, wherein the AI action model is trained using various gesture images as training data, so that when the gesture and action detection module detects the gestures of the multiple virtual reality devices and the AI action model recognizes the gestures as turning on the assist gesture, or when the voice and semantic detection module detects the semantics of the multiple virtual reality devices and the AI semantic model recognizes the semantics as turning on the assist semantics, the virtual image assistant appears in the virtual world. The system as described in claim 1 further includes an activity control module, wherein if the recommended activity determined by the activity determination module is ongoing in the virtual world, the activity control module transfers the multiple virtual reality devices into the recommended activity in the virtual world, and if the recommended activity determined by the activity determination module is not ongoing in the virtual world, the activity control module adds the recommended activity to the calendar, and notifies the multiple virtual reality devices before the recommended activity starts. A method for collaborative discussion among multiple people in a virtual world, comprising: Detecting the voices of multiple virtual reality devices entering the virtual world and the meaning of the voices; Detecting the gestures and actions of the multiple virtual reality devices entering the virtual world; Identify the voice and the meaning to determine whether the recognition result of the voice is to turn on the assistive voice, and use the AI semantic model to determine whether the recognition result of the meaning is the question meaning; Recognize the gesture and action, and use the AI action model to determine whether the recognition result of the gesture is a gesture to turn on the collaboration, and use the AI action model to determine whether the recognition result of the action is an approval action; Analyze the semantic recognition result of the AI semantic model, the action recognition result of the AI action model, and the correlation between the semantic recognition result and the action recognition result to generate an analysis result; and According to the analysis result, a recommended activity is determined from the activity database, and the virtual image assistant provides the recommended activity in the virtual world. The method as described in claim 6, wherein the AI semantic model is trained using a question-and-answer set related to the activity as training data, so that when the semantic recognition result is the question semantics, the AI semantic model generates the corresponding answer semantics, and the virtual image assistant provides an answer in the virtual world based on the answer semantics through voice, pictures, videos, or event live viewing windows. The method as described in claim 7, wherein the AI action model is trained using virtual reality device action data as training data, so that when the virtual image assistant provides the answer, the AI action model identifies the reaction action of each virtual reality device to the answer and the AI semantic model identifies the reaction semantics of each virtual reality device to the answer, and then records the recognition results of each reaction action, the number of virtual reality devices providing each reaction action, the recognition results of each reaction semantics, the number of virtual reality devices providing each reaction semantics, and the correlation between the recognition results of each reaction action and the recognition results of each reaction semantics, so as to provide analysis. The method as described in claim 6, wherein the AI action model is trained using various gesture images as training data, so that when the gestures of the multiple virtual reality devices are detected and the AI action model recognizes the gesture as turning on the assist gesture, or when the semantics of the multiple virtual reality devices are detected and the AI semantic model recognizes the semantics as turning on the assist semantics, the virtual image assistant appears in the virtual world. The method as described in claim 6 further includes: if the recommended activity is ongoing in the virtual world, the multiple virtual reality devices are transferred to the recommended activity in the virtual world; if the recommended activity is not ongoing in the virtual world, the recommended activity is added to the calendar, and before the recommended activity starts, the multiple virtual reality devices are notified. A computer program product, which is loaded into a computer to execute the method described in any one of claim 6-10.

Images