JP2024035212A - Metric-driven improvements to virtual reality experiences - Google Patents

Abstract

A system and method for obtaining metrics related to an extended reality experience and using the obtained metrics to perform remedial actions such as managing a user's motion sickness, determining user performance related to a designed game difficulty, and performing home automation. Data from multiple users as they navigate between determined checkpoints is obtained and used to determine a metric such as a median, typical, or other representative data. A current user's navigation through the same checkpoints is monitored and compared to the metric. Results from the comparison are used to enhance the extended reality experience, including customizing the experience with respect to motion sickness, game difficulty level, and home automation. Optionally, FIG.

Description

Embodiments of the present disclosure improve virtual reality gaming experiences, such as modifying game difficulty, managing motion sickness, and implementing home automation, based on data related to gameplay by a user. Regarding things.

Virtual reality (VR) experiences are used in several fields such as gaming, education, and manufacturing, to name a few. The experiences range from the least immersive to the most immersive, with the goal of giving the user a sense of a real-world environment. Virtual reality is also an experimental medium in its infancy compared to many other technologies, with plenty of room to grow and address current demands. For example, storytelling using this medium is still in its infancy. Antoine Ribordy does an excellent job of explaining how VR, as a medium, differs from regular storytelling. Ribordy says, "In VR storytelling, the environment plays a central role. Special attention must be paid to this. In fact, the whole experience is an act of balancing the story and the environment, and what the player is looking for. It provides answers."

Players or observers in a VR experience (whether together or separately in time) see different assets based on their perspective. To build viewer engagement, content creators must observe patterns in viewer behavior and determine whether their intended designs match viewer observations and actions. A well-designed experience is one that viewers will spend time with, feel engaged with, and ultimately enjoy the experience and perhaps revisit it often.

To understand the impact of different decisions in VR and determine how their designed experiences are perceived by users, content developers may use their own judgment, conduct focus groups, or chat groups, Feedback may be collected from users through informal means, such as via Discord, user surveys, or directly in the gaming application. These methods of obtaining user feedback have several problems.

One such problem is delayed feedback from users. This may be due to the use of older methods such as email, chat, or surveys to determine user experience. Many users believe that such methods of collecting user feedback are not convenient for users and require users to actively spend time on other platforms outside of the game to provide feedback to game developers. may not provide any feedback at all.

Another such problem with these feedback methods is that the feedback can be inaccurate. When the user is not immersed in the VR experience, the user may forget what happened little by little during the VR game, or may only remember certain instances, and the feedback may not be exact timestamps in the virtual game. can be much more difficult to match.

On the other hand, if the feedback method queries the user during the VR game, such queries may be difficult to understand when the user is more interested in immersing himself in the game and such queries are usually ignored. It can have the same effect as an intervening pop-up commercial, which can be annoying to the user. Even if the user provides feedback to the system when a query is made during the game, the user's feedback would cause the user to rush back to the game instead of responding to the query. Therefore, it may be limited.

Some VR experiences display certain assets and a user is required to interact with the assets in order to score points in the game. In such experiences, VR experience creators may desire to understand how users interacted with the designed assets. However, as mentioned above, methods currently used to obtain such feedback are ineffective and result in inaccurate, incomplete, or delayed feedback. Therefore, the VR experience developer may not know exactly why the user did not interact with the designed assets in the VR experience or why the user interacted with the assets in the manner that the user did.

Finally, motion sickness can occur when a user is playing a virtual game, especially in either a game where there is a lot of motion in the virtual world or a game that requires motion in the real world from the user wearing the headset. , which is a big concern. Gathering accurate, complete, and fine-grained data is a challenge due to the lack of feedback mechanisms. Therefore, game developers are not able to fully address the issue of motion sickness or determine why and why a user may experience motion sickness at a specific timestamp of the game.

Therefore, there is a need for better systems and methods for obtaining user feedback related to a user's virtual experience within a virtual environment so that the data can be used to augment the user's experience. exist.
The present invention provides, for example, the following items.
(Item 1)
A method of enhancing a virtual reality experience, the method comprising:
generating a virtual game on a display of an electronic device, a predetermined difficulty level being determined for a portion of the virtual game;
collecting score data from a plurality of users who played a portion of the virtual game, the score data relating to the scores of the plurality of users based on the predetermined difficulty level;
Calculating a performance metric based on the performance data collected from the plurality of users;
comparing the current user's performance data with the calculated performance metric;
and performing an augmentation of a virtual game in response to determining that the current user performance data is not within a threshold of the performance metric.
(Item 2)
The method described in the above item, wherein the threshold value of the performance metric includes a lower limit and an upper limit.
(Item 3)
Any one of the above items, wherein the lower limit is associated with a minimum number of interactions with virtual objects in the virtual game, and the upper limit is associated with a maximum number of interactions with virtual objects in the virtual game. The method described in section.
(Item 4)
determining that the current user's performance data is below a lower threshold of the performance metric;
The method of any one of the preceding items, further comprising: modifying the predetermined difficulty level for the portion of the virtual game to a lower predetermined difficulty level in response to the determination.
(Item 5)
determining that the current user's performance data exceeds a higher limit of the performance metric threshold;
The method of any one of the preceding items, further comprising: modifying the predetermined difficulty level for the portion of the virtual game to a higher predetermined difficulty level in response to the determination.
(Item 6)
A method according to any one of the preceding items, wherein the augmentation of the virtual game is either an increase or a decrease in the predetermined difficulty level.
(Item 7)
A method according to any of the preceding items, wherein the augmentation of the virtual game is a modification of the path that may be taken by the current user within the virtual game.
(Item 8)
A method according to any of the preceding items, wherein the route modification is associated with disallowing navigation by the current user on a route within the virtual game that was previously permitted for navigation.
(Item 9)
The part of the virtual game includes a start checkpoint and an end checkpoint, and the start checkpoint and the end checkpoint are determined based on the type of the virtual game. The method described in.
(Item 10)
Any of the above items further comprising selecting the starting checkpoint and the ending checkpoint in a shorter time span for a faster paced virtual game and in a relatively longer time span for a slower paced virtual game. The method described in Section 1.
(Item 11)
monitoring the current user's path within the virtual game;
determining a total number of available interaction opportunities based on the path taken by the current user within the virtual game;
and determining the number of interaction opportunities utilized by the current user from the total number of available interaction opportunities.
(Item 12)
The current user's performance is based on the number of interaction opportunities performed by the current user out of the total number of interaction opportunities available to the current user, as described in any one of the above items. the method of.
(Item 13)
monitoring the current user's gaze as the user navigates a path within the virtual game;
determining objects within a field of view (FOV) of the current user based on the path navigated by the current user within the virtual game;
The method of any of the preceding items, further comprising: determining whether the user has interacted with an object within the FOV.
(Item 14)
any one of the preceding items, wherein in response to determining that the user has not interacted with the object in the FOV, prompting the user to interact with the object in the FOV; The method described in.
(Item 15)
A method according to any of the preceding items, wherein the prompt is either a visual alert or an audible prompt displayed on a display of the electronic device.
(Item 16)
A method according to any of the preceding items, wherein the object in the FOV is highlighted in response to determining that the user has not interacted with the object in the FOV.
(Item 17)
10. The method of any preceding item, wherein the performance metric is selected from the group consisting of median, representative value, mean, standard deviation, or variance.
(Item 18)
A method,
generating a virtual game on a display of an electronic device;
determining a level of difficulty between a starting checkpoint and an ending checkpoint for the portion of the virtual game;
monitoring the user's progress within said virtual game, and determining whether said user a) meets said determined level of difficulty; or b) is unable to achieve said level of difficulty because it is not possible to meet said level of difficulty; or c) determining whether the above level of difficulty is to be exceeded;
and maintaining, decreasing, or increasing the level of difficulty in response to the determination.
(Item 19)
Monitoring the progress of the above users is
determining coordinates of a route taken by the user within the virtual game;
determining an orientation of a headset worn by the current user to play the virtual game based on the coordinates;
determining an object within the field of view of the headset based on the determined orientation;
and determining whether the current user has interacted with the determined object.
(Item 20)
A system for enhancing a virtual reality experience, the system comprising:
communication circuitry configured to access the electronic device;
A control circuit network,
generating a virtual game on a display of the electronic device, a predetermined difficulty level being determined for a portion of the virtual game;
collecting score data from a plurality of users who have played a portion of the virtual game, the score data relating to the scores of the plurality of users based on the predetermined difficulty level;
Calculating a performance metric based on the performance data collected from the plurality of users;
comparing the current user's performance data with the calculated performance metric;
and control circuitry configured to perform virtual game augmentation in response to determining that the current user performance data is not within the performance metric threshold; system.
(Item 21)
The system according to any one of the above items, wherein the performance metric threshold includes a lower limit and an upper limit.
(Item 22)
Any one of the above items, wherein the lower limit is associated with a minimum number of interactions with virtual objects in the virtual game, and the upper limit is associated with a maximum number of interactions with virtual objects in the virtual game. The system described in Section.
(Item 23)
determining that the current user performance data is below a lower threshold of the performance metric;
The above item further comprising: the control circuitry configured to: modify the predetermined difficulty level for the portion of the virtual game to a lower predetermined difficulty level in response to the determination. The system according to any one of the above.
(Item 24)
determining that the current user's performance data exceeds a higher limit of the performance metric threshold;
The above item further comprising: the control circuitry configured to: modify the predetermined difficulty level for the portion of the virtual game to a higher predetermined difficulty level in response to the determination. The system according to any one of the above.
(Item 25)
A system according to any of the preceding items, wherein the enhancement of the virtual game is implemented by the control circuitry configured to either increase or decrease the predetermined difficulty level.
(Item 26)
A system according to any of the preceding items, wherein augmentation of the virtual game is implemented by the control circuitry configured by modifying the path that may be taken by the current user within the virtual game.
(Item 27)
The system of any of the preceding items, wherein the route modification is associated with disallowing navigation by the current user on a route within the virtual game that was previously permitted for navigation.
(Item 28)
The portion of the virtual game includes a start checkpoint and an end checkpoint, the start checkpoint and the end checkpoint being determined by the control circuitry based on the type of the virtual game. The system described in any one of the items.
(Item 29)
the control circuitry configured to select the start checkpoint and the end checkpoint in a shorter time span for faster paced virtual games and in a relatively longer time span for slower paced virtual games; The system according to any one of the above items, further comprising:
(Item 30)
monitoring the current path of the user within the virtual game;
determining a total number of available interaction opportunities based on the path taken by the current user within the virtual game;
Any one of the above items, further comprising: said control circuitry configured to: determine the number of interaction opportunities utilized by said current user from said total number of available interaction opportunities; The system described in Section.
(Item 31)
The current user's performance is based on the number of interaction opportunities performed by the current user out of the total number of interaction opportunities available to the current user, as described in any one of the above items. system.
(Item 32)
monitoring the current user's gaze as the user navigates a path within the virtual game;
determining objects within a field of view (FOV) of the current user based on the path navigated by the current user within the virtual game;
The system of any one of the preceding items, further comprising: determining whether the user has interacted with an object within the FOV.
(Item 33)
In response to determining that the user has not interacted with the object within the FOV, the control circuitry is configured to prompt the user to interact with the object within the FOV. The system according to any one of the above items, comprising:
(Item 34)
The system of any one of the preceding items, wherein the prompt is either a visual alert or an audible prompt displayed on a display of the electronic device.
(Item 35)
In response to determining that the user has not interacted with the object within the FOV, the control circuitry is configured to highlight any of the items within the FOV. or the system described in item 1.
(Item 36)
A system according to any of the preceding items, wherein the electronic device is selected from the group consisting of virtual reality, augmented reality, or mixed reality devices.
(Summary)
Obtaining metrics related to the extended reality experience and using the obtained metrics to manage motion sickness in users, determine user performance related to designed game difficulty, and implement home automation. Systems and methods are disclosed for implementing remedial actions such as. The method includes determining a starting checkpoint and an ending checkpoint in an extended reality experience. Data from multiple users as they navigate between the determined checkpoints is obtained and used to determine metrics, such as medians, representative values, or other representative data. The current user's navigation through the same checkpoints is monitored and compared to metrics. Results from the comparison are used to enhance the extended reality experience, including customizing the experience with respect to motion sickness, game difficulty level, and home automation.

Various objects and advantages of the present disclosure will become apparent upon consideration of the following detailed description, considered in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout.

FIG. 1 is a block diagram of a process for collecting game data and using it to perform actions or provide recommendations to improve a virtual reality gaming experience, according to some embodiments of the present disclosure. It is a diagram.

FIG. 2 is a block diagram depicting checkpoint selection based on virtual game type, according to some embodiments of the present disclosure.

FIG. 3 is a block diagram of data collected from multiple users as the users navigate through checkpoints within a virtual game, according to some embodiments of the present disclosure.

FIG. 4 is a block diagram of an example system for improving virtual reality gaming experiences, according to some embodiments of the present disclosure.

FIG. 5 is a block diagram of an electronic device used in a gaming environment, according to some embodiments of the present disclosure.

FIG. 6 is a block diagram of an extended reality headset used in a gaming environment, according to some embodiments of the present disclosure.

FIG. 7 is a flowchart of a process for collecting data related to motion sickness in a virtual gaming environment, according to some embodiments of the present disclosure.

FIG. 8 is an example chart depicting various user background factors that may be considered to determine motion sickness associated with playing a virtual game, according to some embodiments of the present disclosure.

FIG. 9 is a flowchart of a process for providing recommendations or activating home automation to manage motion sickness caused by a virtual gaming environment, according to some embodiments of the present disclosure. .

FIG. 10 is a block diagram of an example of comparing current user data to median data to manage motion sickness, according to some embodiments of the present disclosure.

FIG. 11 is a block diagram of categories of recommendations that may be made to manage motion sickness in a virtual gaming environment, according to some embodiments of the present disclosure.

FIG. 12 is a block diagram of categories of devices and machines that may be activated via an API to manage motion sickness in a virtual gaming environment, according to some embodiments of the present disclosure.

FIG. 13A illustrates collecting data related to a user's handling of a level of difficulty in a virtual game and using the data to enhance or manage a level of difficulty in a virtual game, according to some embodiments of the present disclosure. 1 is a flowchart of the process.

FIG. 13B is a flowchart of a process for comparing current user data to median user data to manage the level of difficulty in a virtual game, according to some embodiments of the present disclosure.

FIG. 14 is a navigation tree depicting routes taken by multiple users between predetermined checkpoints within a virtual game, according to some embodiments of the present disclosure.

FIG. 15 is a block diagram depicting a user's field of view and objects/assets encountered within the field of view, according to some embodiments of the present disclosure.

FIG. 16 is a block diagram of a room layout and multiple routes taken to reach a destination, according to some embodiments of the present disclosure.

FIG. 17 is a block diagram of layout recommendations, according to some embodiments of the present disclosure.

FIG. 18 is a table of user navigation coordinates of user interactions as the user navigates from checkpoint C1 to checkpoint C2, according to some embodiments of the present disclosure.

FIG. 19 is a table of user interactions and missed opportunities, according to some embodiments of the present disclosure.

FIG. 20 is a table of content improvement suggestions based on user navigation in an extended reality experience, according to some embodiments of the present disclosure.

FIG. 21 is a block diagram depicting locally rendered and cloud rendered extended reality experiences in accordance with some embodiments of the present disclosure.

DETAILED DESCRIPTION According to some embodiments disclosed herein, some of the limitations mentioned above are applicable to obtaining user metrics during an extended reality experience, such as a virtual game, from multiple users. , generating median data and comparing the median data with current data of the current user to determine whether the current user is experiencing or will experience similar outcomes in the extended reality experience. be overcome by this. Some of the limitations mentioned above also determine the start and end checkpoints for monitoring the current user and the user navigation data collected between the start and end checkpoints. be overcome by providing remedial actions in response. For example, such remediation actions may include managing motion sickness, managing game difficulty levels, and adjusting the difficulty level as needed, all based on the path taken by the user between the start and end checkpoints. This includes adjusting speed, blocking paths in virtual games that are deemed difficult, and implementing home automation to manage motion sickness in users.

In one embodiment, an extended reality experience is rendered to multiple users. The extended reality experience may be a virtual reality game, an augmented reality skill learning tool, or any other extended reality experience that interfaces with real-life education. A virtual reality game is used to illustrate the processes and embodiments herein, however, the extended reality experience may be any other extended reality experience as well.

In some embodiments, multiple checkpoints C1 and C2 are determined. Checkpoints are determined based on the type of extended reality experience used. In one embodiment, in an experience involving slow gaming for children, millisecond-by-millisecond data collection is not required, so even if checkpoints C1 and C2 that are far apart in time are selected, good. In other embodiments, in fast-paced games, such as battles or car racing games between multiple players, where each turn or every millisecond can affect the outcome of the game, then checkpoints that are closer together in time C1 and C2 may be chosen to accumulate more data than slower moving games.

In one embodiment, once a checkpoint is selected, data for each of a plurality of users as the user navigates from checkpoint 1 to checkpoint 2 is monitored and logged. Such data includes the route taken by each of the plurality of users, the amount of time spent on the route, the challenges and obstacles overcome by the user along the route, the route taken by the user from checkpoint 1 to checkpoint 2. includes the user's headset field of view when navigating to the destination, the assets the user interacted with along the route and the time they interacted with, the CPU/GPU usage along the route, and the user's motion sickness data along the route. But that's fine.

Once data from multiple users has been collected and logged, a median value is calculated. Although median data is described, embodiments are not so limited; representative values, mean values, standard deviations, variances, or some other calculation based on an equation, or another statistical measure may also be used. may be used. The calculated data may also be referred to herein as performance metrics. The median value is then used to compare current data for current users currently navigating the same route in an extended reality experience. The comparison allows the system to enhance and modify the extended reality experience. This also allows the system to predict motion sickness or to manage it through home automation techniques once this occurs. The comparison also allows the system to make adjustments to the game difficulty level, such as increasing or decreasing the difficulty, for a better user experience.

FIG. 1 is a block diagram of a process for collecting game data and using it to perform actions or provide recommendations to improve a VR gaming experience, according to some embodiments of the present disclosure. It is.

In some embodiments, at block 101, a gaming environment is displayed on a display device. The gaming environment may be an extended reality environment, including virtual reality, augmented reality, mixed reality, or any other virtual type of reality, including within the metaverse.

In some embodiments, systems described herein, such as the systems of FIGS. 4 and 5, provide extended reality environments that include virtual simulations of both real-world experiences and other experiences that are fictional. For example, the system simulates the virtual layout of a city, such as San Francisco or Paris, and allows a user to create a virtual layout such as walking through the streets of San Francisco or playing a game of car racing through the streets of Paris. may allow activities to be carried out within a simulated environment. In other embodiments, a virtual environment may be created that is fictional, such as outer space or some other imaginary environment.

In some embodiments, these systems utilize extended reality devices such as VR headsets, VR glasses, or mobile phones that can act as VR devices. The extended reality device may include a display screen for displaying the simulated virtual environment. The extended reality device is worn on the user's head such that the display of the extended reality device is in front of the user's eyes, and the user is exposed to an extended reality 3D simulated environment that is depicted on the display of the extended reality device. can be made visible. In some embodiments, when navigation along a route is referenced, the reference is associated with a virtual reality device, such as a virtual reality headset or glasses. In other embodiments, reference is made to an augmented reality embodiment in which a real-world view is used, through which the real-world environment and a virtual overlay on the real-world environment are visualized via a headset or glasses. associated with an augmented reality device, such as an augmented reality headset or glasses, which may be used as an augmented reality device.

In some embodiments, the extended reality device also includes an inwardly facing camera to track the user's gaze, speakers for sound effects, and a sense of sensing effects displayed in the virtual world, such as earthquakes. may include any one or more motion-generating components, such as a vibration module, for providing a user with a motion-generating component, such as a vibration module. It may also include accelerometers, gyroscopes, and proximity sensors. This may include a processor and memory, such as a system on a chip (SoC).

Extended reality devices connect to the Internet and can download or access various applications that provide the user with an extended reality experience, such as games, factory settings, medical surgeries, training to drive a vehicle, exercises, etc. It could be possible. In some embodiments, an extended reality device may include more than one connectivity option for communicating with other devices, body sensors, or electronic devices for downloading applications; Options include SoCs that feature connectivity via API, Wi-Fi, Bluetooth, and/or other radio frequency connectivity in addition to available USB connectivity (e.g., USB Type-C). may include connections that use

As mentioned above, extended reality devices may connect to the Internet to download gaming or other types of applications. These applications may move in three-dimensional (3D) space, for example, in three translational x, y, z axes and three rotational axes, commonly referred to as six degrees of freedom (6DOF). May include assets within the virtual world.

In some embodiments, using an extended reality device, the system creates a video for each eye of the user that allows the user to play a virtual game. In some cases, during a virtual game, an avatar of a portion of a user's body is used, the user's entire body represented as an avatar is used, or no avatar is used. It can be either. When the avatar is used, the system creates the illusion that the user is really within the virtual environment being displayed.

In some embodiments, an extended reality device (such as an augmented reality, virtual reality headset, or dual-functionality headset, or virtual glasses) uses head tracking technology to Track the movement of the user's head while wearing the device on their head. Such tracking captures the user's head movements as a means of operating a camera and viewing objects in the virtual world. For example, if a user orients his or her head to the left, objects or assets that should be on the left will be visible to the user. Therefore, the image changes with respect to the user according to the way the user orients his or her head, ie, the extended reality headset.

In some embodiments, in addition, some extended reality headsets, attachments, and accessories that are attached directly to or associated with and paired with the headset can enhance interactions within the virtual game. includes a gaming controller that can act as a hand for the purpose of A user may be able to use such a controller to perform functions within a virtual environment. For example, a user may be able to drive a car, hit another avatar, displace an object, perform a golf swing, etc.

In addition to the controller, in some embodiments the system may also include a wearable device that can be worn on the user's body to provide a full-body VR experience. For example, embodiments combine an extended reality headset with sensors placed on the user's body to simulate and simulate all body movements performed in the real world as movements on the avatar of the body used in the virtual world. May include imitation.

In some embodiments, the provided extended reality environment may be stationary; in other embodiments, the provided extended reality environment may dynamically change or change from one location to another. The user may be asked to navigate to. For example, a user may be asked to navigate from one virtual room to another within an extended reality environment, or drive from one location to another within an extended reality environment.

In a stationary extended reality environment, in some embodiments a user may be able to look around using his extended reality headset without the need to move within the virtual world. For example, stationary experiences may include sitting in a virtual vehicle or playing a game of chess, which may involve only limited movement of a sit-down experience.

In some embodiments, other experiences may require the user to perform a number of movements or transition from one location to another. For example, a user may need to look around in all 360 degrees to protect against being hit in a virtual environment where objects are thrown at the user from all directions. In such situations, head tracking is important, as it reduces any lag between the user movement in the real world and the image that should be relayed to the user in the virtual world based on the real world movement. must be implemented in real time. For example, if a user quickly makes a sharp 180° turn to fight a virtual animal that jumps at him from behind, the image in the virtual world will quickly reduce lag to provide a real-world-like effect. It should display an athletic animal pouncing on the user, without accompanying text. Therefore, such dynamically moving experiences are more immersive and require headset movement tracking with greater accuracy.

Following user movements and matching them with virtual images with little lag is also important in reducing motion sickness among users. This is because, among other factors, the brain senses movement by getting signals from the inner ear, eyes, muscles, and joints, and when movement is not accompanied by a visual image that matches the movement, this occurs. This is because it can induce motion sickness.

Referring again to block 101, in some embodiments, an extended reality environment of a room may be displayed. As depicted in block 101, the exemplary virtual room includes a plurality of bookshelves, a chair, a table, a lamp, and a chaise lounge to simulate a real room setting. VR games may provide users with challenges to score or advance to the next level of the game, which may include rooms with higher level challenges. Challenges may require the user to find hidden objects or rearrange the layout of a room, or this may be a game where the user chases avatars and catches them as they run from room to room. could be.

At block 102, in some embodiments, a portion of the virtual game is selected. The selected portion may include a starting checkpoint, also referred to as checkpoint 1 or "C1," and an ending checkpoint, also referred to as checkpoint 2 or "C2." A checkpoint, in some embodiments, is an experience or portion of a game state that is saved for rollback or restart. Typically, a checkpoint is before anything new in the experience or game can be developed. For example, in games with multiple levels, there are typically checkpoints after each level is passed. If a character dies in the game, it will come back to life from a previous checkpoint.

Data during checkpoints may be collected by the system of FIG. 1 and used to provide an enhanced extended reality experience. For example, since there may be multiple tasks and challenges between checkpoints, the data related to the time taken to complete a set of tasks or a sequence of tasks or challenges is important between checkpoints C1 and C2. The total time spent may be presented. When the user reaches checkpoint C2, a timestamped entry is registered to indicate the achievement. If the user exits the experience before reaching checkpoint C2, the absence of this entry in the log file indicates that the user did not complete the task to reach checkpoint C2. Aggregation of such data during checkpoints helps users and programmers evaluate and adjust the virtual game for an enhanced experience.

In some embodiments, checkpoints are selected based on the type of extended reality game and its environment. As depicted in FIG. 2, at block 210, in one embodiment, the extended reality game may be a slow game for children. The game may not have fast moving components or rapidly changing graphics. It may not have very many assets. This may not allow for too many degrees of rotation since it does not have an image to show for each orientation of the user's headset. For example, an extended reality game may be a simple game such as tic-tac-toe or an educational game that requires a child to identify shapes, etc.

In this embodiment, it may not be necessary to accumulate data at shorter intervals. This is because in slower moving games it may be possible to skip frames and accumulate data over longer intervals without missing important components of the game. Thus, starting checkpoint C1 and ending checkpoint C2 may be selected 300 seconds apart, and data may be captured at 15 or 30 second intervals within a total time of 300 seconds.

As depicted in FIG. 2, at block 220, in one embodiment, the extended reality game may be a medium speed virtual game. A game may have moderately fast moving components that require rendering at 1 or 2 second intervals. This may have more assets than slower moving games. This also allows for more degrees of rotation, and at each rotation may have the image in the virtual world match the headset movement. For example, an extended reality game may be a game like playing tennis, chess, or walking through a maze. Since games like tennis depict the same tennis court and the tracking of a tennis ball as it is hit from side to side, the pace at which images are rendered in the virtual world is moderate, e.g. every 1-2 seconds. A tennis ball may be tracked at certain intervals.

In this embodiment, collecting data at shorter intervals may be required to track where the tennis ball is located after each shot. Thus, the starting checkpoint C1 and the ending checkpoint C2 may be selected 90 seconds apart, and data may be captured at intervals of every 1-2 seconds within a total time of 90 seconds.

As depicted in FIG. 2, at blocks 230 and 240, in one embodiment, the extended reality game may be a fast-paced game or a game at a level that requires faster responses. good. Games may have fast moving objects or rapidly changing graphics. This may have several objects/assets, such as a car racing game depicting several other cars moving at different speeds within the field of view and several obstacles on the path. It also allows for all degrees of rotation and may have a rapidly changing image to show for each orientation of the user's headset. For example, an extended reality game may be a fast car race, a game where fighter jets chase each other through mountains, a fast roller coaster, or a shooting game where several enemies attack at the same time.

In this embodiment, it may be necessary to accumulate data at much shorter intervals. This can be critical to a game, such as a fighter jet crashing into a mountain if proper maneuvers are not performed in a fast-moving game, or a user being hit by a bullet in a game where the user is being attacked from all angles. This is because it may not be possible to skip frames without missing certain parts. Thus, starting checkpoint C1 and ending checkpoint C2 may be selected 30 seconds apart, and data may be captured every 1/2 or 1/5 second.

Although certain time intervals are described above in connection with blocks 210-240, embodiments are not so limited, and embodiments may also include other starting and ending checkpoints and data accumulation intervals within a checkpoint. It is also assumed.

Referring again to block 102, once the starting checkpoint and ending checkpoint are identified, the system collects data from multiple users who played the game and navigated from the starting checkpoint C1 to the ending checkpoint C2. May be collected. One example of such data in graphical format is depicted in FIG. Although a graph chart is used as an example, the data may be in any other format, such as a table. As depicted in FIG. 3, two users, namely User 1 and User 2, have navigated from a starting checkpoint C1 to an ending checkpoint C2. The data were captured in all 6 DOFs (only 3 DOFs are shown). Data is captured from time T=t ₀ to time T=t ₁ . As this can be seen, the route navigated by user 1 is different from the route navigated by user 2. The path and other data from User 1 and User 2 can be used to calculate a median path, such as the median path depicted in FIG. 14.

The data collected relates to different experiences encountered by different users from multiple users. For example, different users may spend different amounts of time and encounter different challenges between two checkpoints; for example, one user may choose an easier route than another; One can lose life, while the other can pass through it without losing life.

Different users may also have different experiences in discovering assets within a game. Some users may discover assets that are critical to completing a level more quickly or in fewer attempts than others. Some users may choose to stay inside longer to enjoy the experience more and explore their surroundings. All such user experiences and their data may be captured while the user navigates from starting checkpoint C1 to ending checkpoint C2. As depicted in block 102, three users, namely User 1, User 2, and User 3, were monitored by the system as the users navigated from checkpoint C1 to C2.

User involvement and interaction in the experience depends on numerous factors. Aggregate user behavior, measured as the time spent between two checkpoints, and the percentage of sessions in which the user exits without reaching the next checkpoint may be considered high-level indicators. The methods and systems for logging and analyzing user behavior using the metrics described shall suggest deep insights to content creators on how to improve engagement. Examples of such data related to behavior are further described at block 103.

Block 103 depicts some examples of the types of data that may be collected while users navigate through checkpoints C1 and C2. As explained above, the interval at which data is collected may vary based on the type of application, eg, slow, medium, or fast paced virtual gaming.

In one embodiment, data related to route coordinates may be collected as multiple users navigate through checkpoints C1 and C2. The path coordinates may be x, y, z coordinates and pitch, roll, and yaw orientations as depicted in FIG. In this embodiment, the system may collect, at predetermined intervals, each movement along the path selected by the user, including all coordinates associated with the reference frame. For example, the reference frame may be a location within the virtual world, such as the origin on an object within the virtual world, or it may be the origin of the headset prior to the start of the game.

In another embodiment, data related to route timing may be collected as multiple users navigate through checkpoints C1 and C2. The route timing may indicate the amount of time it took the user to proceed from checkpoint C1 to checkpoint C2 along the user-selected route. In this embodiment, the system may collect data related to the amount of time it took the user to traverse through certain locations along the selected route at predetermined intervals. For example, if the path is to go from the settee to a more distant ramp in block 101, the system will determine the total amount of time taken from the settee to the further ramp and from the settee if that is the route selected by the user. The amount of time may be determined for intermediate points, such as to the chair and then to a lamp further from the chair.

In another embodiment, data related to encountered assets may be collected as multiple users navigate through checkpoints C1 and C2. Assets encountered along a route may be determined by determining the route the user has selected and what is within the user's field of view as the user navigates on that route. In this embodiment, the system may collect data related to the user's field of view as the user navigates along the route at predetermined intervals. For example, if the chosen path is to walk along the left side of the room to the nearest lamp, and the user is facing straight toward the lamp, the system may may determine that the only asset in its field of view encountered is the nearest lamp.

In another embodiment, data related to interacted assets may be collected as multiple users navigate through checkpoints C1 and C2. Assets interacted with along a path may be determined by monitoring the user's actions in the virtual game as the user navigates along his or her path. For example, a user may encounter certain assets in his or her view, but the user may not have interacted with those assets and may have simply walked past them. An example of an interaction could be moving a chair, sitting on a chair, tripping on a chair, or placing something on a chair. At predetermined intervals, the system may collect data related to the assets within its field of view and determine whether the user has interacted with them, and if applicable, it records the type of interaction. You may.

In another embodiment, data related to CPU and GPU usage may be collected as multiple users navigate through checkpoints C1 and C2. User headset movement requires rendering and re-rendering of images in the virtual world, so the amount of rendering required based on the amount of movement, the pace of movement, and the amount of images and details depicted is dependent on the CPU and Running out of GPU memory. The system may collect data related to the amount of CPU and GPU used based on the path taken by the user and the amount and quality of rendering and re-rendering performed at predetermined intervals. Such data may be used to manage and predict CPU and GPU usage and implement resource management for future users.

In yet another embodiment, data related to the level of difficulty may be collected as multiple users navigate through checkpoints C1 and C2. The level of difficulty experienced in progressing from C1 to C2 and all intermediate points may be collected at predetermined intervals. For example, the programmer may have programmed a certain level of difficulty at each intermediate point between C1 and C2, preventing the user from easily completing that game level. The system monitors the user's route and navigation and ensures that the user experiences the same level of difficulty or is within a programmed difficulty threshold while navigating from C1 to C2 and intermediate points. Decide whether or not. For example, some users may encounter levels of higher difficulty, while other users may be able to complete certain levels more easily and faster.

In another embodiment, data related to motion sickness may be collected as multiple users navigate through checkpoints C1 and C2. Because visual effects depicted within virtual games, such as user headset movement or virtual roller coaster rides, can cause nausea and motion sickness, the system provides multiple visual effects as the user navigates through checkpoints C1 and C2. Monitor users and collect data such as heart rate, blood pressure, EKG, and others from sensors and other wearables attached to the user's body. The system collects such data when certain fast paced twists and turns occur in the game or when graphics are rendered at a fast pace. It may also collect such data at predetermined intervals. Such data may be used to manage motion sickness by providing suggestions to the user or programmer regarding remedial steps.

At block 104, a median route based on routes taken by multiple users may be calculated. Although a median path is described, embodiments are not so limited and may use representative values, means, standard deviations, variances, or some other calculation based on equations or other calculations to determine the path. Statistical measures may also be used. Median values of other data other than the route data collected may also be calculated. The median value may represent a representative value user from multiple users. As depicted, a graph of median users is presented at block 104, along with several tables of median data for some of the types of data described in block 103. Although some of the types of data collected are described in blocks 103 and 104, embodiments are not so limited and other types of data are also envisioned. For example, data such as how users react once they encounter their first difficulties, or data related to the most frequently visited assets or the amount of time spent interacting with an asset. may also be collected.

At block 105, an example of one type of collected data is depicted, including multiple subcategories of collected data. In this embodiment, motion sickness data is one of the types of data being collected. When collecting subcategories of data related to motion sickness data, the system accesses multiple devices and sensors attached to the user's body, associated with a wearable worn by the user, or a device that remotely monitors the user. You may. For example, devices such as heart rate monitors, blood pressure monitors, blood glucose monitors, ECGs, and EKGs may be used as a user navigates from one point to the next along a path between checkpoints C1 and C2. It may be accessed by the system to obtain readings of the user's vitals. As explained above, this data may be collected regarding multiple users who played the game and navigated between checkpoints.

At block 106, the system may obtain current data from the current user as the user begins playing the game and navigates from checkpoint C1 to checkpoint C2. Monitoring may be real-time or performed at predetermined intervals at intermediate locations between checkpoints C1 and C2. Current data obtained from current users may be compared to median data.

As depicted, the median data shows that at time T01, the median user's heart rate was 72 BPM, while the current user's heart rate is 68 BPM. The median data further shows that as the median user navigates from one point to the next along their path in the game, i.e. from time T01 to time T02, their heart rate increases from 72 BPM to 81 BPM. Indicates an increase. Although median data is mentioned, other types of data based on calculating representative values, mean values, standard deviations, variances, or some other formulas can also be used. Current data captured over the same time interval shows that the current user's heart rate jumped from 68 BPM to 92 BPM while navigating from T01 to time T02.

In one embodiment, the increase in heart rate for the current user from 68 BPM to 92 BPM may be outside the safety threshold range. Such threshold ranges may be predetermined by the system or obtained based on medical advice regarding a particular user, or generally regarding any person playing a virtual game.

At block 107, upon determining that there is cause for concern about the current user's rapid increase in BPM, the system may provide recommendations to the user. Some examples of recommendations include opening a window, turning on a fan, or pausing the game for 30 seconds. In another embodiment, instead of providing recommendations to the user, the system may automatically turn on a fan in the room in which the user is currently playing the game, for example. The system may do so by accessing the fan via an API.

Although motion sickness was used as an example of the type of data collected and remedial steps that are subsequently taken to manage motion sickness, embodiments are not so limited. For example, the collected data may be of any type, including some of the example data types described in block 103, and the collected data may relate to the current user's current May be compared with data. In response to the comparison, the system may determine whether the current data is within a threshold of data collected for the median user, and if not, remedial actions may be taken.

In addition to the above, both translation and rotation data based on the user's movements while wearing the extended reality headset may be collected. In one example, different users may have different rotation vector graphs in their experience. Such a vector graph may be determined using a general tree based on user navigation between checkpoints C1 and C2. The difference between two consecutive recordings may represent a state change in the user's position in the 6DOF experience. Additional details regarding the generated tree and vector graph are described in connection with FIG. 13.

FIG. 4 is a block diagram of an example system for improving a virtual reality gaming experience, according to some embodiments of the present disclosure, and FIG. 5 is a block diagram of an example system for use in a gaming environment, according to some embodiments of the present disclosure. FIG. 2 is a block diagram of an electronic device. 4 and 5 also illustrate example devices, systems, servers, and related hardware that may be used to implement the processes, functions, and functionality described in connection with FIGS. 1-3 and 6-21. Explain. Additionally, FIGS. 4 and 5 also display an extended reality experience, set difficulty levels in the extended reality experience, obtain user metrics during an extended reality experience such as a virtual game from multiple users, and obtain median data. and compare the median data with the current data of the current user to determine whether the current user is experiencing or will experience similar outcomes in the extended reality experience, and Determine starting and ending checkpoints to monitor, provide improvement actions based on user or multiple user performance, manage difficulty levels based on user interaction, and optionally may be used to adjust, manage motion sickness, perform home automation functions, run artificial intelligence or machine learning algorithms, manage difficulty levels, motion sickness, or home automation.

In some embodiments, one or more portions or the entire system 400 may be configured as a system implementing the various features, processes, functionality, and components of FIGS. 1-3 and 6-21. good. Although FIG. 4 depicts a certain number of components, in various embodiments the system 400 may include fewer than the number of components shown and/or one or more of the number of components shown. May include multiples of

System 400 is shown to include a computing device 418, a server 402, and a communication network 414. Although a single instance of a component may be shown and described with respect to FIG. 4, it should be understood that additional instances of components may also be employed. For example, server 402 may include or be embedded within more than one server. Similarly, communications network 414 may include or be incorporated within more than one communications network. Server 402 is shown communicatively coupled to computing device 418 through communication network 414 . Although not shown in FIG. 4, server 402 may be communicatively coupled directly to computing device 418, for example, in a system where communication network 414 is absent or bypassed.

Communication network 414 may comprise one or more network systems such as, but not limited to, the Internet, LAN, WIFI, or other network systems suitable for audio processing applications. In some embodiments, system 400 runs server 402 and functionality that would otherwise be implemented by server 402 is instead implemented by one or more components of communication network 414, etc. Implemented by other components of system 400. In yet other embodiments, server 402 cooperates with one or more components of communication network 414 to implement certain functionality described herein in a distributed or collaborative manner. Similarly, in some embodiments, system 400 runs computing device 418 and functionality that would otherwise be implemented by computing device 418 is instead implemented by communication network 414 or server 402. or by other components of system 400, such as one or more components in combination. In yet other embodiments, computing device 418 cooperates with one or more components of communication network 414 or server 402 to implement certain functionality described herein in a distributed or collaborative manner. do.

Computing device 418 includes control circuitry 428, display 434, and input circuitry 416. Control circuitry 428, in turn, includes transceiver circuitry 462, storage 438, and processing circuitry 440. In some embodiments, computing device 418 or control circuitry 428 may be configured as media device 500 of FIG. 5.

Server 402 includes control circuitry 420 and storage 424. Each of storage devices 424 and 438 may be an electronic storage device. As referred to herein, the phrase "electronic storage device" or "storage device" refers to random access memory, read-only memory, hard drive, optical drive, digital video disc (DVD) recorder, compact disc (CD) recorder, BLU-RAY® disc (BD) recorder, BLU-RAY® 4D disc recorder, digital video recorder (sometimes referred to as personal video recorder, or PVR, DVR), solid state device; Any device for storing electronic data, computer software, or firmware, such as a quantum storage device, gaming console, gaming media, or any other suitable fixed or removable storage device, and/or any combination thereof. should be understood as meaning. Each storage device 424, 438 may be used to store various types of content, metadata, and/or other types of data (e.g., they can be used to of statistics or group statistics, difficulty level values, user and multiple user performance data or user performance metrics, start and end of checkpoints, data related to home automation devices and their settings and any user preferences, recommendations and remedial actions. (can be used to store lists, and ML and AI algorithms). Non-volatile memory may also be used (eg, to invoke startup routines and other instructions). Cloud-based storage may be used to supplement or in place of storage 424, 438. Some embodiments display an extended reality experience, set difficulty levels in the extended reality experience, obtain user metrics during an extended reality experience, such as a virtual game, from multiple users, and generate median data. compare the median data with the current data of the current user to determine whether the current user is experiencing or will experience similar outcomes in the extended reality experience, and monitor the current user. Determine starting and ending checkpoints for tasks, provide improvement actions based on user or multiple user performance, manage difficulty levels based on user interaction, and adjust as needed. , managing motion sickness, performing home automation functions, running artificial intelligence or machine learning algorithms, managing difficulty levels, motion sickness, or data related to managing home automation, and as described herein. Data related to all other processes and features may be recorded and stored in one or more of storage devices 412, 438.

In some embodiments, control circuitry 420 and/or 428 execute instructions for applications stored in memory (eg, storage 424 and/or storage 438). Specifically, control circuitry 420 and/or 428 may be commanded by an application to perform the functions discussed herein. In some implementations, any actions performed by control circuitry 420 and/or 428 may be based on instructions received from an application. For example, an application may be implemented as software or a set of executable instructions stored in storage device 424 and/or storage device 438 and executed by control circuitry 420 and/or 428. In some embodiments, the application may be a client/server application, where only the client application resides on computing device 418 and the server application resides on server 402.

The application may be implemented using any suitable architecture. For example, it may be a standalone application implemented entirely on computing device 418. In such an approach, instructions for the application are stored locally (e.g., in storage 438) and data for use by the application is periodically retrieved (e.g., from an out-of-band feed, from an Internet resource, or (using another suitable approach). Control circuitry 428 may retrieve instructions for applications from storage 438, process the instructions, and perform the functionality described herein. Based on the processed instructions, control circuitry 428 may determine the type of action to take in response to input received from input circuitry 416 or from communication network 414. For example, a given difficulty level is set in such a way that the user expects the user to make at least five attempts before completing a portion of the game, and the current user completes this in one attempt. In response to determining that the current user performance exceeds the difficulty level for a portion of the game, such input increases the game difficulty, etc. by easily exceeding expectations, etc. , are used by control circuitry 428 to implement remedial actions. Other such processes are illustrated in at least flowcharts 1, 7, 9, and 13A, 13B depicted herein.

In client/server-based embodiments, control circuitry 428 may include communication circuitry suitable for communicating with an application server (eg, server 402) or other networks or servers. Instructions for performing the functionality described herein may be stored on an application server. The communication circuitry may include a cable modem, an Ethernet card, or a wireless modem for communication with other equipment, or any other suitable communication circuitry. Such communications may involve the Internet or any other suitable communications network or pathway (eg, communications network 414). In another example of a client/server-based application, control circuitry 428 launches a web browser that interprets web pages provided by a remote server (eg, server 402). For example, a remote server may store instructions regarding the application within a storage device. The remote server may use circuitry (eg, control circuitry 428) to process stored instructions and/or generate displays. Computing device 418 may receive displays generated by a remote server and may display content of the display locally via display 434. In this manner, processing of instructions is performed remotely (e.g., by server 402) while the resulting display, such as a display window described elsewhere herein, is performed on computing device 418. Locally provided. Computing device 418 may receive input from a user via input circuitry 416 and transmit those inputs to a remote server for processing and generating a corresponding display. Alternatively, computing device 418 may receive input from a user via input circuitry 416 and process and display the received input locally by control circuitry 428 and display 434, respectively.

Server 402 and computing device 418 may transmit and receive content and data, such as objects, frames, snippets of interest, and inputs, to and from primary and secondary devices, such as AR devices. Control circuitry 420, 428 may send and receive commands, requests, and other suitable data through communication network 414 using transceiver circuitry 460, 462, respectively. Control circuitry 420 , 428 may communicate directly with each other using transceiver circuits 460 , 462 , respectively, bypassing communication network 414 .

It should be understood that computing device 418 is not limited to the embodiments and methods shown and described herein. In non-limiting examples, computing device 418 may include a virtual, augmented, or mixed reality headset, smart glasses, or a device that may perform functions within the metaverse, a primary device, a personal computer (PC), a laptop computer, tablet computer, WebTV box, personal computer television (PC/TV), PC media server, PC media center, handheld computer, mobile phone, smartphone, or any other device, computing equipment, or wireless device, and/or primary It may be a combination of content and secondary content that can be suitably displayed.

Control circuitry 420 and/or 418 may be based on any suitable processing circuitry, such as processing circuitry 426 and/or 440, respectively. As referred to herein, processing circuitry may include one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. ), and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores). In some embodiments, the processing circuitry includes a plurality of separate processors, for example, a plurality of the same type of processors (e.g., two Intel Core i9 processors) or a plurality of different processors (e.g., an Intel Core i7 processor and Intel Core i9 processors). In some embodiments, the control circuitry 420 and/or the control circuitry 418 display the extended reality experience, set difficulty levels in the extended reality experience, and accept requests from multiple users for the extended reality experience, such as a virtual game. obtain user metrics between, generate median data, and compare the median data to the current user's current data, and the current user is experiencing or will experience similar outcomes in the extended reality experience. Determine starting and ending checkpoints to monitor current users, provide improvement actions based on the user or users' performance, and based on user interactions. Manage difficulty levels and adjust as needed, manage motion sickness, perform home automation functions, run artificial intelligence or machine learning algorithms, manage difficulty levels, motion sickness, or home automation , Flowcharts 1, 7, 9, and 13A, 13B.

User input 404 may be received from a virtual, augmented, or mixed reality headset, mobile data, smart glasses. Transmission of user input 404 to computing device 418 is accomplished using a wired connection, such as an audio cable, USB cable, Ethernet cable, or the like, attached to a corresponding input port on the local device. or using a wireless connection such as Bluetooth, WIFI, WiMAX, GSM, UTMS, CDMA, TDMA, 3G, 4G, 4G LTE, or any suitable wireless transmission protocol. May be carried out. Input circuitry 416 may include a physical connection such as a 3.5mm audio jack, RCA audio jack, USB port, Ethernet port, or any other suitable connection for receiving audio via a wired connection. May be equipped with an input port or receive data via Bluetooth®, WIFI, WiMAX, GSM®, UTMS, CDMA, TDMA, 3G, 4G, 4G LTE, or other wireless transmission protocols The wireless receiver may be configured to:

Processing circuitry 440 may receive input 404 from input circuitry 416 . Processing circuitry 440 may convert or convert received user input 404, which may be in the form of voice input to a microphone or movements or gestures, into a digital signal. In some embodiments, input circuit 416 performs conversion to a digital signal. In some embodiments, processing circuitry 440 (or in some cases, processing circuitry 426) performs the disclosed processes and methods. For example, processing circuitry 440 or processing circuitry 426 may implement processes such as those illustrated in flowcharts 1, 7, 9, and 13A, 13B and other figures described herein.

FIG. 5 illustrates a generalized embodiment of an electronics device 500, according to one embodiment. In some embodiments, equipment device 500 is the same as equipment device 402 of FIG. 4. Equipment device 500 may receive content and data via input/output (I/O) path 502. I/O path 502 may include audio content (e.g., broadcast programming, on-demand programming, Internet content, content available via a local area network (LAN) or wide area network (WAN), and/or other content). and data may be provided to control circuitry 504, which includes processing circuitry 506 and storage 508. Control circuitry 504 may be used to send and receive commands, requests, and other suitable data using I/O path 502. I/O path 502 may connect control circuitry 504 (and specifically processing circuitry 506) to one or more communication paths. Although I/O functionality may be provided by one or more of these communication paths, they are shown in FIG. 5 as a single path to avoid overcomplicating the drawing. shown.

Control circuitry 504 may be based on any suitable processing circuitry, such as processing circuitry 506. As referred to herein, processing circuitry may include one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. ), and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or a supercomputer. In some embodiments, the processing circuitry includes a plurality of separate processors or processing units, for example, a plurality of processing units of the same type (e.g., two Intel Core i7 processors) or a plurality of different processors (e.g., an Intel Core i7 processor). Core i5 processors and Intel Core i7 processors).

Communication between two separate user devices, such as an extended reality headset and a gaming module or receiving server to send messages, or send the FOV or coordinates of a head-mounted VR device to a server, an extended reality experience displaying a level of difficulty in an extended reality experience, obtaining user metrics from multiple users during an extended reality experience such as a virtual game, and generating median data; A starting checkpoint for monitoring current users, comparing current data of users of and determining exit checkpoints; providing remedial actions based on user or multiple user performance; managing and adjusting difficulty levels as necessary based on user interaction; motion sickness; to manage difficulty levels, motion sickness, or home automation, perform home automation functions, run artificial intelligence or machine learning algorithms, manage difficulty levels, motion sickness, or home automation, and all other described herein. Communication between two separate user devices, such as data related to processes and features, may be implemented, at least in part, using control circuitry 504. Processes as described herein may be implemented in or supported by any suitable software, hardware, or combinations thereof. They may also be implemented on the user equipment, on a remote server, or across both.

In a client/server-based embodiment, control circuitry 504 displays the extended reality experience, sets difficulty levels in the extended reality experience, and collects user metrics during the extended reality experience, such as a virtual game, from multiple users. obtain, generate median data, and compare the median data to the current data of the current user to determine whether the current user is experiencing or will experience similar outcomes in the extended reality experience. Determine and monitor current users, determine starting and ending checkpoints, provide improvement actions based on user or multiple user performance, and manage difficulty levels based on user interactions and adjust as necessary, manage motion sickness, perform home automation functions, perform artificial intelligence or machine learning algorithms, manage difficulty levels, motion sickness, or home automation, and manage difficulty levels, motion sickness, or home automation; It may also include communication circuitry suitable to enable communication between two separate user devices to perform all related functions and processes as described. Instructions for performing the functionality mentioned above may be stored on one or more servers. The communications network may include a cable modem, an integrated services digital network (ISDN) modem, a digital subscriber line (DSL) modem, a telephone modem, an Ethernet card, or a wireless modem for communication with other equipment, or Any other suitable communication circuitry may also be included. Such communications may involve the Internet or any other suitable communications network or pathway. Additionally, the communication circuitry may include circuitry that enables peer-to-peer communication of primary equipment devices or communication of primary equipment devices at locations remote from each other (described in more detail below).

Memory may be an electronic storage device, provided as storage 508 that is part of control circuitry 504. As referred to herein, the phrase "electronic storage device" or "storage device" refers to random access memory, read-only memory, hard drive, optical drive, digital video disc (DVD) recorder, compact disc (CD) recorder, BLU-RAY® disc (BD) recorder, BLU-RAY® 3D disc recorder, digital video recorder (sometimes referred to as personal video recorder, or PVR, DVR), solid state device; Any device for storing electronic data, computer software, or firmware, such as a quantum storage device, gaming console, gaming media, or any other suitable fixed or removable storage device, and/or any combination thereof. should be understood as meaning. Storage device 508 stores user or group statistics, difficulty level values, user and plurality user performance data or user performance metrics, start and end of checkpoints, home automation devices and their settings, and the like as depicted in FIG. It may be used to store data related to any user preferences, lists of recommendations and remedial actions, and ML and AI algorithms, and all functionality and processes discussed herein. Cloud-based storage, described in connection with FIG. 5, may be used to supplement or in place of storage 508.

Control circuitry 504 includes audio generation circuitry, such as one or more analog tuners, audio generation circuitry, filters, or any other suitable tuning or audio circuitry or combination of such circuits; It may also include a net. Control circuitry 504 may also include scaler circuitry to upconvert and downconvert content to a preferred output format of electronic device 500. Control circuitry 504 may also include digital-to-analog converter circuitry and analog-to-digital converter circuitry to convert between digital and analog signals. Tuning and encoding circuitry may be used by electronic device 500 to receive and display, play, or record content. The circuitry described herein, including, for example, tuning, audio generation, encoding, decoding, encryption, decoding, scaler, and analog/digital circuitry may be implemented on one or more general purpose or specialized processors. may be implemented using software running on the If storage 508 is provided as a separate device from electronic device 500, tuning and encoding circuitry (including multiple tuners) may be associated with storage 508.

A user may issue commands to control circuitry 504, which are received by microphone 516. Microphone 516 may be any microphone (or microphones) capable of detecting human speech. Microphone 516 is connected to processing circuitry 506 and transmits detected voice commands and other utterances thereto for processing. In some embodiments, a voice assistant (eg, Siri, Alexa, Google Home, and similar such voice assistants) receives and processes voice commands and other utterances.

Electronic device 500 may include an interface 510. Interface 510 may be any suitable user interface, such as a remote control, mouse, trackball, keypad, keyboard, touch screen, touch pad, stylus input, joystick, or other user input interface. Display 512 may be provided as a standalone device or integrated with other elements of electronic device 500. For example, display 512 may be a touch screen or touch-sensitive display. In such situations, interface 510 may be integrated with or combined with microphone 516. When interface 510 is configured with a screen, such screen may include one or more monitors, a television, a liquid crystal display (LCD) for a mobile device, an active matrix display, a cathode ray tube display, a light emitting diode, etc. It may be a display, an organic light emitting diode display, a quantum dot display, or any other suitable device for displaying visual images. In some embodiments, interface 510 may be HDTV compatible. In some embodiments, display 512 may be a 3D display. Speaker (or speakers) 514 may be provided as integrated with other elements of electronic device 500 or may be a stand-alone unit. In some embodiments, display 512 may be output through speaker 514.

Although the equipment device 500 of FIG. 5 may be implemented in the system 400 of FIG. 4 as the primary equipment device 402, any other type of user equipment may also implement machine learning (ML) and artificial intelligence (AI) algorithms. It is suitable for enabling communication between two separate user devices to carry out the functions associated with implementing and all the functionality discussed in connection with the figures referred to in this application.

Any other type of suitable user equipment electronic device 500 may also be used to implement the ML and AI algorithms and related functions and processes as described herein. For example, extended reality headsets or smart glasses or similar such devices may be used. An electronic device may be part of a network of devices. Various network configurations of devices may be implemented and are discussed in more detail below.

FIG. 6 is a block diagram of an extended reality headset used in a gaming environment, according to some embodiments of the present disclosure.

In some embodiments, extended reality devices such as headsets, glasses, mobile phones, or other wearable devices may be used to perform the processes described herein. Extended reality devices may be non-immersive, fully immersive, semi-immersive, or have some other combination of virtual, augmented, or mixed reality. For example, the extended reality device may be an Oculus Rift ^™ , a Valve ^™ , a Sandbox VR ^™ , or a Sony PlayStation VR ^™ . This may also be smart/virtual glasses such as the Iristic Z1 ^™ , Epson Moverio BT-200 ^™ , or Sony SmartEyeglass ^™ . A non-immersive experience may allow a user to experience a virtual game through a computer by allowing the user to control characters within the software without directly interacting with the virtual environment. A fully immersive experience is a virtual world where the user interacts with the virtual environment based on their movements, such as a combat game where others shoot at the user, or where other cars in a race interact with the way the user drives. It can provide a realistic experience within a responsive car racing game, etc. There may also be some semi-immersive experiences that have a combination of both.

In some embodiments, the headset may be a virtual reality headset, in which a purely virtual experience may be visualized. In other embodiments, the headset may be an augmented reality headset in which the real world environment may be visualized, including any virtual overlays. In yet other embodiments, the headset may include dual functionality, having both VR and AR capabilities and being able to switch between AR and VR.

In some embodiments, an extended reality device may include a complete system with the processor and components needed to provide a complete extended reality experience. In other embodiments, the extended reality device may rely on external devices, such as smartphones, computers, and servers, to perform all processing. For example, a headset may be a plastic, metal, or cardboard holding case that allows viewing, which is connected to a smartphone via wires or wirelessly or via an API, The screen may be used as a lens for visual recognition. The headset may also be connected to a gaming console, where the headset has its own display screen, which is powered by the computer or gaming console.

As depicted in FIG. 6, in one embodiment, the extended reality headset used includes 6 DOF. The headset works by immersing the user in a virtual environment with all orientations, so the extended reality head provides a full 6 DOF for a fully immersive experience where the user's entire field of vision is utilized, including peripheral vision. set is preferred (although extended reality headsets with 3DOF can also be used).

Having 6DOF allows the user to move in all directions and also experience objects and environments in the virtual world from all directions, e.g. the user can move objects in 3D space. Can be done. These 6DOFs are commonly referred to as pitch, yaw, and roll, with rotational movement around the x, y, and z axes, and lateral movement along any one direction x, y, or z. corresponds to a translation along those axes, such that Tracking all 6 DOF allows the system to capture user movements and the user's field of view in both translational and rotational directions, thereby providing a complete 360° view in all directions. provide.

Although some references are made to types of extended reality headsets, embodiments are not so limited; any other extended reality headsets available on the market may also be used herein. May be used in conjunction with the described embodiments.

FIG. 7 is a flowchart of a process for collecting data related to motion sickness in a virtual gaming environment, according to some embodiments of the present disclosure.

In some embodiments, at block 710, a gaming environment is displayed on an extended reality device. The gaming environment may be an extended reality environment displayed on a display screen associated with a VR headset, VR glasses, or another type of VR module. The gaming environment displays a virtual game with multiple challenges and assets and may include virtual simulations of both real-world and fictional surroundings. The extended reality device on which the virtual game is displayed is connected to cameras, lenses, display screens, speakers, motion generation components, and other equipment needed to provide the complete extended reality experience. Good too. Extended reality devices may be able to connect to external applications via APIs to download or access various games. In one embodiment, the displayed virtual game may have a layout as depicted in block 101 of FIG.

At block 720, a starting checkpoint and an ending checkpoint are selected. As explained above, the game may have several checkpoints. Starting and ending checkpoints within the game may be selected such that challenges or experiences are contained within the checkpoints. These challenges may be promotion to the next game level, scoring opportunities, or in-game milestones.

Checkpoints also act as reference positions in the game state that are saved for rollbacks or restarts, such as when a character dies, the game can be reloaded or restarted from the last checkpoint. In some embodiments, checkpoints C1 and C2 are selected based on the type of extended reality game and its environment. For example, a game may not have fast moving components or rapidly changing graphics, in which case checkpoints with longer durations between them may be selected and the data , may be acquired at longer intervals so that the amount of data does not over-occupy CPU and GPU space. For example, for a 10 minute game with four levels, where the game is a slower game, a 300 second checkpoint with data collection every 15-30 seconds may be appropriate.

On the other hand, for fast-paced games with several moving components or rapidly changing graphics, checkpoints with shorter durations between them may be selected, and the data does not matter at any time in the game. It may be taken at shorter intervals, such as every 1/2 second or every few milliseconds, such as 100 milliseconds, to ensure that no changes are missed.

The higher the value of the time interval for collecting data, the smaller the log file, but the greater the risk of missing useful information or phenomena such as user movement that may have occurred between data collection points. Become. The shorter the time interval value for collecting data, the larger the log file will be, which will also consume a large amount of memory on the CPU and GPU. Therefore, some level of balancing may be implemented to ensure that the correct amount of data is collected while not overloading the CPU/GPU. Such balancing attempts may take into account the type of use as explained above, for example whether this is a fast or slow moving game.

Blocks 730, 740, 750, and 760 are used to collect motion sickness data from multiple users, aggregate the data, and calculate a median value that may be used for current and future users. To accomplish collection, aggregation, and median collection, in one embodiment, at block 730, control circuitry monitors multiple users involved in playing the virtual game. User data is collected as the users progress through the path between C1 and C2. The data includes translation and orientation movements (eg, qw, qx, qy, and qz) of the user between checkpoints C1 and C2. For example, if a user moves their head to the left, thereby moving their wearable headset to the left, the field of view when the user looks to the left determines which assets were in the user's view. recorded to do so. In addition to what is depicted in block 740, all obstacles encountered, challenges solved, assets viewed, assets interacted with, user vital signs during user movement, CPU/GPU utilization, motion sickness experienced, time taken by the user to complete a level of difficulty, coordinates of the route taken by the user, type of route, time in each segment of the route, and more. gaming metrics are collected.

At block 730, data related to motion sickness is also collected, among other types of data and metrics collected. Such data may be from wearable devices, sensors, or other attachments associated with the extended reality device, such as smart watches that may detect heartbeats, body sensors, and other types of monitors that may collect the user's vital signs. May be obtained. A user may also input such data into a system associated with the extended reality environment. In yet another embodiment, the control circuitry may collect demographic user information, health profile information, or other information prior to or during the extended reality experience. It may also obtain such information through other means (such as a profile from a content distribution storefront). The control circuitry may also obtain information such as the time of day, the time of the last meal, and the user's blood sugar level. An additional example of data that may be collected is depicted in FIG. 8 below.

At block 750, a determination is made whether the user data, regardless of how it is collected, can be used to determine whether the user has experienced motion sickness. Such decisions may be made based on the user's medical profile or general medical information available on the Internet. The decision also depends on the user's behavior in the game, for example that the user is not as attentive as before, which may indicate motion sickness, or that the user himself inputs information into the system associated with the extended reality environment. may be based on indicating that the user is experiencing motion sickness. The system may also have an icon or selection available for the user to select whether the user is experiencing motion sickness.

If motion sickness is detected at block 750, the user's vitals and any one or more motion sickness related data previously collected at block 730 are aggregated at block 760. Such aggregated data gives content creators the ability to make inferences regarding when current users may or may not experience motion sickness. A median value of the aggregated data is calculated. Median data may include the user's median orientation (qw, qx, qy, qz) at each point on the median path and provide details regarding the exact location on the path where motion sickness was experienced. An example of a median path is depicted in FIG. The utilization of median data obtained from multiple users and its application to the current user is further explained in the description of FIG. 9.

FIG. 8 is an example chart depicting various user background factors that may be considered to determine motion sickness associated with playing a virtual game, according to some embodiments of the present disclosure.

In some embodiments, the user's investigated past susceptibility (various), such as the user's ethnicity, age, location, disease, pupillary distance, genetics, certain environments, causes of motion sickness due to lack of sleep, etc. stimulus) and any previous medical history or symptoms, severity of the user's motion sickness, and other types of stimuli that cause motion sickness. can be taken into account when deciding whether to engage.

Whether this is when collecting data for multiple users, such as in block 102 of FIG. 1, or collecting current data for a current user, the data may be distorted based on the user's background information. For example, during the data collection stage, such as block 102 of FIG. 1, one user from the plurality of users may be experiencing significant motion sickness at a particular point in the game, while another user from the plurality of users of users experience little or no motion sickness. Depending on accessing the user's profile and based on further investigation of the user's background, the user may have certain medical may have been discovered to have problems. In one embodiment, the system may discard such user's data and consider it an outlier. In another embodiment, the system may log data and apply it when current users of similar backgrounds play virtual games.

In some embodiments, ethnicity, age, location-related illness, or genetics may also play a role in motion sickness, and such background information may be considered when evaluating current users or previous May be taken into account when collecting data from users. In some embodiments, the system may group the data into groups based on the user's ethnicity. Such data may then be used if a determination is made that the current user is also of the same ethnicity. Similarly, age and other background data may also be grouped and used accordingly. The system may also generate multiple subgroups with combinations of background information, such as creating groups for ethnicity and subgroups for different age ranges within the same ethnicity.

In some embodiments, based on the collected data, the system may predict the likelihood and probability that the current user will experience motion sickness during a particular stage of the virtual game. For example, the system may determine (as part of determining the user's background) that 72% of the users who played the game have prescription glasses. If the system encounters the current user and determines, based on the current user's profile or through facial attribute detection, that the user is wearing prescription glasses of a similar power, the system: The probability that the current user will experience motion sickness may be determined. Therefore, the system may change the game screen, provide an alert to the user to take a different route in the game, or simply alert the user that the user is likely to experience motion sickness. Either to warn you.

In some embodiments, the user may also review past user motion sickness data prior to or during the virtual game to determine his or her likelihood of getting motion sickness. The system may also provide a manual or appearance of virtual game points at which the user is likely to experience motion sickness, and which the user may avoid if the user so chooses. good.

The user's background, such as the user's ethnicity, age, field dependence, medical conditions, pupillary distance, genetics, investigated past susceptibility (and various other stimuli) may determine or predict motion sickness. Although discussed above in conjunction, such factors can also be used to determine their application to other virtual gaming and enhancement options. For example, such factors may be applied to determine a user's likely response to a level of game difficulty or response to a certain situation within a virtual game. For example, users of a certain younger age are more likely, based on their own experience, to take a certain path within a virtual game or to perform poorly at a certain game level than others who are older. It can be high. Similarly, some users who are older may not be able to perform well in certain situations compared to others who are younger. In addition, other background factors such as amount of virtual gaming experience, education, or motor skills may also be used to determine whether a user is likely to perform better in a given virtual gaming situation. good.

FIG. 9 is a flowchart of a process for providing recommendations or activating home automation to manage motion sickness resulting from a virtual gaming environment, according to some embodiments of the present disclosure. As discussed above, FIG. 8 illustrates some embodiments of collecting data from multiple users, and FIG. We describe several embodiments of how such collected data may be applied to current users to provide.

At block 910, median data is calculated based on motion sickness data collected from multiple users as the users navigate from checkpoint C1 to C2. Although calculations of median data are described, embodiments are not so limited, calculations may also be used to determine representative values, mean values, standard deviations, variances, or some other calculations based on equations. may be carried out.

Once the median data is calculated, in some embodiments the system may follow the path of Trajectory 1, and in other embodiments it may follow the path of Trajectory 2. Trajectory 1 is associated with determining remedial actions after discovering that the current user is experiencing motion sickness, whereas Trajectory 2 is associated with anticipating motion sickness and determining remedial actions prior to the user experiencing motion sickness. associated with taking improvement actions.

In one embodiment, either Trajectory 1 or Trajectory 2 may be selected by the system. In another embodiment, the system may select either Trajectory 1 or Trajectory 2 based on user preferences, such as preferences contained within the user's profile. In another embodiment, the system may provide an option for user selection that may be considered to determine whether Trajectory 1 or Trajectory 2 should be selected. In yet another embodiment, the system may analyze the user's prior gaming data, which may be obtained based on execution of a machine learning algorithm, to determine the likelihood that the user will experience motion sickness. If a determination is made that the user is likely to experience motion sickness, the system will follow Trajectory 2, in which remedial actions are applied proactively and in advance of the user experiencing actual motion sickness. may be selected.

In some embodiments, in Trajectory 1, at block 920, the system may monitor the current user's motion sickness factor at various times. These times may be t=0, t=1,...t=n, and the monitoring time "t" may be a predetermined interval. The system may monitor the user's motion sickness factor either periodically or constantly during game play. In another embodiment, the system may monitor the user's motion sickness factor after a certain milestone or challenge within the virtual game, such as a steep drop on a virtual roller coaster within the virtual game.

At block 930, control circuitry 420 or 428 determines whether the user's current data or factor is within a median data threshold. Some examples of user current and median data are provided in FIG. 10.

In some embodiments, as depicted in FIG. 10, control circuitry 420 or 428 monitors user data at any one or more times during a virtual session, such as a virtual game or experience. You may. It may also monitor user data at in-game milestones or at various gaming levels. It may also monitor user data periodically or continuously throughout the virtual session. Factors monitored are heart rate, body temperature, sweating level, dizziness, loss of concentration, lack of responsiveness to gaming tasks, headache, nausea, yawning, sighing, increased salivation, blurred vision, drowsiness. , spatial disorientation, or any other similar factors typically associated with motion sickness. Surveillance may be through cameras and sensors associated with the extended reality device. Monitoring may also be through data obtained from wearable devices, sensors, or other attachments connected to the extended reality device via an API.

One example of monitoring a user's current data and comparing it to median data is depicted at 1010 in FIG. In this example, data regarding a virtual experience, such as a virtual game, virtual room, virtual education, or any other type of virtual experience, may be collected from multiple users. A median value may be calculated for data collected from multiple users. As mentioned above, instead of the median calculation, other calculations may also be used. One of the monitoring attributes may be monitoring acceleration depicted in a virtual game consisting of a roller coaster ride. Acceleration may be measured at curve 1, which may be a sharp curve during a virtual roller coaster ride. A system administrator or game developer may have selected the acceleration at curve 1 to be monitored. For example, a system administrator or game developer may choose to monitor curve 1 because the curve is designed in a virtual game to be a sharp curve where the user will experience some gravity. In some cases, a system administrator or game developer may anticipate potential motion sickness for some people in this curve. Whatever the reason for monitoring the acceleration in curve 1, the calculated median data may show that the median user from the multiple users had a median heart rate of 72 BPM. Current user data on the same curve 1 with the same acceleration may also be monitored and the results may show that the current user's BPM is 75. Control circuitry 420 or 428 may determine whether the current user's BPM is within a threshold range of +/−5 BPM of 72 median BPM.

Another example of monitoring a user's current data and comparing it to median data is depicted at 1020 in FIG. Block 1020 is similar to block 1010 except that data is monitored for acceleration on curve 2. The median data calculated in curve 2 shows that the median user from multiple users had a median heart rate of 77 BPM, while the current user data in the same curve 2 with the same acceleration had a median heart rate of 84 BPM. , i.e., higher than the acceptable threshold of +/-5 of the median BPM of 77.

In yet another example, data regarding the virtual experience may be collected from multiple users at time t=t1, as depicted at 1030 in FIG. In this example, at time t=t1, the virtual experience may comprise a rotation in the virtual room. The rotation may be a visual rotation that is visible in the user's headset, or it may be a rotation that the user physically rotates in the real world environment to carry out instructions provided in the virtual world. There may be. A median value may be calculated at time t=t1 for data collected from multiple users while the users experience a real-world or virtual rotation in a virtual room. A system administrator or game developer may have chosen a particular rotational acceleration based on its expected motion sickness potential. As depicted, median data calculated based on data collected from multiple users for rotation at time t=t1 indicates a median body temperature of 98°F. Control circuitry 420 or 428, while monitoring the user's current data, may determine that the user's body temperature is 100° F. while undergoing the same virtual experience with multiple users at time t=t1. Control circuitry 420 or 428 may also determine that the user's body temperature of 100°F is outside the threshold of +/−1°F from the median body temperature of 98°F.

In another example, as depicted at 1040 in FIG. 10, data regarding a virtual experience, which in this case may be a virtual game, may be collected after 30 minutes of game play. A system administrator or game developer may select the 30-minute mark to determine the amount of fatigue and its causal role in motion sickness that a game player may experience after playing a virtual game for 30 minutes. be. The 30-minute mark is also selected based on data obtained from machine learning or AI algorithms that indicate that users typically experience fatigue, eye strain, or other symptoms that cause users to experience motion sickness. There may be cases where Whatever the reason for monitoring user responsiveness after 30 minutes of gameplay, median data calculated based on data collected from multiple users shows that the median user has a scoring rate of 67%. can be shown. The level of responsiveness in this example may be measured based on the user's ability to score in a predetermined percentage. The level of responsiveness in this example may also be measured by determining the scoring rate that the user performs after 30 minutes of game play and whether it is within a threshold of the user's starting scoring rate. As depicted, control circuitry 420 or 428, while monitoring the user's current scoring percentage, determines that the current user's scoring percentage is 53%. Control circuitry 420 or 428 may determine that the user's scoring rate is outside +/−10% of the median scoring rate.

Although several examples of monitoring are described in connection with FIG. 10, embodiments are not so limited, and other types of monitoring are also envisioned. For example, the control circuitry may also monitor the user's facial expressions to determine whether the user is suffering from motion sickness. In this example, the control circuitry may compare the user's baseline facial expression to the facial expression when a point in the virtual experience is depicted, such as a fast curve, descent, or acceleration. The control circuitry may detect changes in facial expression and determine whether the user is suffering from motion sickness.

Referring again to block 930 of FIG. 9, if a determination is made at block 930 that the user data does not exceed the median data threshold, the process returns to block 920 and the user returns to checkpoint C2 or the end of the game. Iterates until reached.

If a determination is made at block 930 that the user data exceeds the median data threshold, ie, the tolerance threshold, and deviates from the median data, the process moves to block 940 where remediation actions are taken.

In one embodiment, the remedial action may be to make a recommendation to reduce motion sickness. These recommendations may be provided to users, system administrators, virtual experience creators, or any combination thereof. Some examples of motion sickness recommendations are provided below in connection with the discussion of FIG. 11.

In another embodiment, the improvement action may be to automatically implement home automation via an API. Some examples of the types of home automation motion devices that may be controlled are provided below in connection with the discussion of FIG. 12.

Referring again to block 910, another path that may be selected may be the path of trajectory 2, as described above. In one embodiment, at block 950, control circuitry 420 or 428 tracks the user's current route. This is the path taken by the user within checkpoints C1 and C2, such as checkpoints C1 and C2 depicted in FIG.

Tracking the user's current route may be accomplished by periodically or continuously determining the user's coordinates as the user navigates from checkpoint C1 to C2. For example, a user may select a route while navigating from a settee to bookshelf 1 as depicted in FIG. Such coordinates may be recorded or logged as shown in FIGS. 14 and 18.

At block 960, a determination is made whether the current path is aligned with the median path. In one embodiment, if the median user experiences motion sickness along the route and the current user's route is aligned with the median user, control circuitry 420 or 428 may cause the current user to also experience motion sickness along the route. You can expect to experience intoxication. On the other hand, if the median user does not experience motion sickness along the route and the current user's route is aligned with the median user, control circuitry 420 or 428 determines whether the current user experiences motion sickness. I can predict that it won't. Control circuitry 420 or 428 also applies current user background data, such as from FIG. 8, to adjust or deviate from the median path to determine whether the user experiences motion sickness based on the user's background. You may decide whether or not.

If a determination is made at block 960 that the current user's path does not match the median path, the process returns to block 950 and the current user's path is updated until it matches the median path or Continues to be tracked until point C2 or the end of the virtual experience.

If a determination is made at block 960 that the current user's path matches the median path, the process moves to block 940 where remediation actions are taken.

In one embodiment, the remedial action may be to make a recommendation to reduce motion sickness. These recommendations may be provided to users, system administrators, virtual experience creators, or any combination thereof. Some examples of motion sickness recommendations are provided below in connection with the discussion of FIG. 11.

In another embodiment, the improvement action may be to automatically implement home automation via an API. Some examples of the types of home automation motion devices that may be controlled are provided below in connection with the discussion of FIG. 12.

As depicted in block 1105 of FIG. 11, one type of recommendation made to the current user may be to reduce rotation. In this embodiment, control circuitry 420 or 428 may determine that the amount of rotation in the virtual experience is causing motion sickness to the user. Accordingly, control circuitry 420 or 428 may provide an alert to the user to reduce the number of turns in either the virtual or real environment to minimize the degree of motion sickness. If the prior rotation is not the cause for motion sickness, the control circuitry 420 or 428 still controls future rotations in the game, either in the virtual or real environment, to minimize the degree of motion sickness. An alert may be provided to the user to reduce the number of rotations.

In another embodiment, the control circuitry 420 or 428 recommends that the user change his or her current route, which may be causing motion sickness, to a new route that is less likely to cause motion sickness. may be provided. For example, control circuitry 420 or 428 may associate a path with a motion sickness value based on the amount of twist, turn, orientation, speed, acceleration, and other factors that can cause motion sickness. Control circuitry 420 or 428 may determine that another route with a lower motion sickness value is preferred for the current user, and therefore instructs the current user to change to the new route. Would recommend. In another embodiment, control circuitry 420 or 428 may close or block paths with high motion sickness values when such is required.

In another embodiment, control circuitry 420 or 428 may provide recommendations to a system administrator or virtual game creator to reduce rotation. Such a recommendation would require the system administrator or game developer to either provide an alternate screen in the game that has less rotation, or remove any instructions in the game that require the current user to perform rotation. may be possible.

As depicted at block 1110, another type of recommendation made to the current user may be to reduce the user's velocity or acceleration in the virtual experience. In this embodiment, control circuitry 420 or 428 may determine that the amount of velocity or acceleration in the virtual experience is a contributing factor to the user's motion sickness. Accordingly, control circuitry 420 or 428 may provide an alert to the user to reduce the amount of velocity or acceleration.

For example, a user may use his or her virtual headset to play a virtual car racing game in which the user controls the amount of speed and acceleration as the virtual car speeds across a roadway or makes sharp turns. May be playing. Control circuitry 420 or 428 may alert the user to reduce his or her speed or acceleration when the user is making his or her turns. The control circuitry 420 or 428 also allows the system administrator or game developer to limit either the amount of speed and acceleration or the curvature of the curves that the vehicle can make turns in order to minimize the degree of motion sickness. You may also suggest to system administrators or game developers what they should do.

As depicted at block 1115, the recommendations may be in the form of instructions provided to the user as a pop-up screen or by other visual or audible means. For example, control circuitry 420 or 428 may send an audio signal to the user's headphones instructing the user not to perform a 360° sudden turn.

As depicted in block 1120, one type of recommendation is to rearrange the layout of a virtual space, such as the layout of a virtual room or the layout of a virtual roller coaster or virtual car racing game, to reduce motion sickness. This may also be done by the system administrator or game developer. For example, based on the motion sickness experienced by the median user and the current user, circuitry 420 or 428 reduces the amount of ups and downs in the virtual roller coaster to minimize or reduce motion sickness. or provide recommendations to system administrators or game developers to change the number of turns or the angle of turns in a roller coaster or car racing game, or to change the layout of a virtual climbing experience with deep drops. good. Recommendations may be general or specific outlining challenges or obstacles to be removed or reduced in order to reduce motion sickness.

As depicted at block 1125, control circuitry 420 or 428 may predict motion sickness based on median user data. As described above, if the median user data indicates that the median user experiences motion sickness at a particular time or during a particular segment of the virtual experience, the control circuitry 420 or 428 may predict a similar degree of motion sickness for the current user if the current user follows a similar path to the median user.

As depicted at block 1130, circuitry 420 or 428 monitors the user's vitals as the user plays a virtual game or navigates from checkpoint C1 to C2 in another type of virtual experience. Good too. Based on monitoring the vitals, circuitry 420 or 428 may make recommendations to the system administrator or game developer to take certain remedial actions. For example, in some embodiments, a virtual game or experience may be created such that when the user comes to a certain point within the game, alternative paths may be available to the user. The circuitry 420 or 428 may be configured by a system administrator or 428 to eliminate one of the alternative paths over which motion sickness may be experienced and maintain another path that has a lower likelihood of causing motion sickness. Recommendations may be made to game creators. In another embodiment, other game modification options may also be suggested, such as changing graphics, orientation, rotation, speed, or acceleration, based on the user's vitals.

As depicted at block 1135, circuitry 420 or 428 may provide other types of alerts to the current user to minimize motion sickness. For example, a virtual alert with an arrow may point toward a path of less twisting and turning or less shift of the user's center of gravity due to movement in the virtual or real world.

FIG. 12 is a block diagram of categories of devices and machines that may be activated via an API to manage motion sickness in a virtual gaming environment, according to some embodiments of the present disclosure.

In one embodiment, as depicted at block 1205, physical home windows may be controlled via an API as an improvement measure to reduce motion sickness. In this embodiment, the windows may include an automatic mechanism that allows the windows to be electronically controlled so that they can be opened and closed based on the type of signal received. In this embodiment, the window is wirelessly connected to either the control circuitry 420 or 428 associated with the headset, or the gaming server, or other intermediate home hub that is connected to both the headset and the gaming server and the automated window. May be connected.

In one embodiment, if control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as in block 930 of FIG. It may allow light and fresh air to enter the user's location to transmit a signal and thereby reduce the user's motion sickness.

In another embodiment, if control circuitry 420 or 428 anticipates that the user will likely experience motion sickness, such as in block 970 of FIG. In anticipation of motion sickness, a signal may be sent to open the window to prevent such occurrence. Control circuitry 420 or 428 may also determine the weather outside the home and take it into account when opening windows. For example, if it is the middle of winter and it is cold outside, such as 32 degrees Fahrenheit, the control circuitry 420 or 428 may only open the window for a short duration, such as 2 to 5 minutes, to prevent the area from becoming too cold. may be opened.

In another embodiment where home automation is applied, as depicted in block 1210, the temperature of the room in which the current user is playing a virtual game or experiencing a virtual environment through his or her headset is , may be modified as an improvement measure to reduce motion sickness. In this embodiment, the temperature controller or thermostat has Wi-Fi or API capability that allows the controller to be electronically controlled such that the temperature can be increased or decreased based on the type of signal received. May include.

For example, thermostats such as the Google Nest ^™ , Honeywell ^™ thermostat, or Ecobee ^™ Home are types of thermostats that include Wi-Fi control capabilities. In this embodiment, such exemplary thermostats and other types of temperature controllers are connected to either the control circuitry 420 or 428 associated with the headset, or the gaming server, or the headset and gaming server and the Wi-Fi and It may also be wirelessly connected to other intermediate home hubs that are connected to both API-enabled thermostats.

In one embodiment, if the control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as at block 930 of FIG. 9, the control circuitry 420 or 428 sends a signal to the thermostat. It's okay. The signal changes the temperature, such as by turning on the air conditioner, or turning on the fan, or increasing its speed, thereby allowing air flow and a reduction in temperature, in order to reduce motion sickness in the user. It may also be for the purpose of reducing the

In another embodiment, if control circuitry 420 or 428 anticipates that the user will likely experience motion sickness, such as in block 970 of FIG. In anticipation of getting motion sickness, turn on the air conditioner, or turn on the fan, or increase its speed, thereby reducing the air flow and temperature to prevent such occurrence. A signal may be sent to reduce the temperature, such as by enabling. In some environments, control circuitry 420 or 428 also remembers the user's preferred temperature settings and ensures that the preferred temperature is reached either when motion sickness is experienced or when motion sickness is anticipated. The thermostat may be set automatically.

In another embodiment where home automation is applied, the lighting in the room where the current user is playing a virtual game or experiencing a virtual environment through his headset, as depicted in block 1215. may be modified as an improvement measure to reduce motion sickness. In this embodiment, the lighting module and controller are equipped with a Wi- May include Fi or API capabilities.

For example, smart home lighting controllers, switches, and modules, such as smart switches from Lutron ^™ , Insteon ^™ , Wemo ^™ , and Phillips ^™ , can include Wi-Fi control capabilities. In this embodiment, smart home devices with such exemplary smart light switches and lights include either the control circuitry 420 or 428 associated with the headset, or the gaming server, or the headset and the gaming server and the Wi- It may be wirelessly connected to other intermediate home hubs that are connected to both Fi and API-enabled thermostats.

In one embodiment, if the control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as in block 930 of FIG. 9, the control circuitry 420 or 428 sends a signal to the smart light switch. You can also send it. The signal turns on the lighting at the user's location, brightens the lighting, or in the case of movable lights, moves the lighting direction to face the user, thereby causing the user's brain to select brighter lights. The user's area may be illuminated in an attempt to reduce the user's motion sickness by allowing the user to actively react to the motion. Such lighting may help reduce motion sickness, as poor peripheral vision or lack of awareness of the surroundings may be a contributing factor regarding motion sickness.

In another embodiment, if the control circuitry 420 or 428 anticipates that the user will likely experience motion sickness, such as in block 970 of FIG. A signal may be sent to a light switch. The signal turns on the lighting at the user's location, brightens the lighting, or in the case of movable lights, moves the lighting direction to face the user, thereby causing the user's brain to select brighter lights. The user's area may be illuminated in an attempt to reduce the user's motion sickness by allowing the user to actively react to the motion.

In yet another embodiment, as depicted at block 1220, a device, such as a refrigerator, water cooler, or other food storage device, allows a user to store certain foods as a remedy for reducing motion sickness. It may be illuminated or generate an audible sound to suggest to the user that he should eat or drink something. In this embodiment, refrigerators that have a touch screen display panel on their exterior surface, have Wi-Fi connectivity capability, or are API-enabled have a touch screen display that illuminates or wakes up their touch screen. or may be electronically connected and controlled to illuminate their screens in a certain color. Such refrigerators also provide text alerts on the touch screen, such as a glass of orange juice, or an audio signal or audio signal alerting the user to eat food or drink liquid to reduce the user's motion sickness. An utterance may also be generated. Such remediation measures may be taken if the user is experiencing motion sickness or in anticipation that the user may experience this based on the path the user is currently taking in the virtual experience. good.

In another embodiment, as depicted in block 1225, the control circuitry 420 or 428 may operate a fan, music player, door, etc. to change the environment within the room as an improvement measure to reduce motion sickness. , some other Wi-Fi or API-enabled devices, such as a recliner, may be turned on automatically. In this embodiment, Wi-Fi or API-enabled devices may allow them to be controlled electronically by control circuitry 420 or 428.

In one embodiment, control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as in block 930 of FIG. 9, or that the user is experiencing motion sickness, such as in block 970. If the control circuitry 420 or 428 anticipates a likely possibility, it may send a signal to the Wi-Fi or API enabled device. The signal may be for implementing a combination of actions to change the environment to reduce motion sickness in the user. For example, if a user is playing a virtual game on a Wi-Fi enabled recliner, control circuitry 420 or 428 may cause the user to sit upright from a reclined position to change the user's environment. , or vice versa.

In another embodiment where home automation is applied, within the room where the current user is playing a virtual game or experiencing a virtual environment through his or her headset, as depicted at block 1230, Exercise equipment located at or near the exercise equipment may be turned on and the user may be alerted to use the exercise equipment that is turned on in an attempt to reduce motion sickness. In this embodiment, the exercise machine may include a controller that is Wi-Fi or API enabled. The exercise machine controller may be electronically controlled so that it can be turned on and its speed adjusted.

For example, exercise bicycles and machines such as Peloton ^™ , NordicTrack ^™ , or smart home gym machines such as Tonal ^™ , and other similar smart exercise equipment may include Wi-Fi control capabilities. In this embodiment, such smart exercise equipment is connected to either the control circuitry 420 or 428 associated with the headset, or the gaming server, or both the headset or gaming server and a Wi-Fi and API-enabled thermostat. may be wirelessly connected to other intermediate home hubs.

In one embodiment, if control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as at block 930 of FIG. 9, control circuitry 420 or 428 sends a signal to the smart exercise equipment. You can also send it. The signal may be for turning on smart exercise equipment. The signal may also be for illuminating a smart exercise device, such as one with a display screen, or for generating an audible alert for the user to start exercising to reduce motion sickness in the user. .

In another embodiment, if the control circuitry 420 or 428 anticipates that the user will likely experience motion sickness, such as in block 970 of FIG. A signal may be sent to illuminate a smart exercise device, such as one with a screen, or to generate an audible alert for the user to begin exercising to reduce the user's motion sickness. Because certain exercises may reduce motion sickness, the smart exercise system may also be automatically controlled to turn on programs associated with such motion sickness reduction exercises.

Although several examples of types of home automation and smart devices are described above, embodiments are not so limited, and other types of home automation devices are also envisioned. For example, in some embodiments, control circuitry 420 or 428 implements smart home automation functions to minimize or reduce motion sickness for the current user. The functionality is performed either after determining that the current user is experiencing motion sickness, or proactively before the current user experiences motion sickness.

Embodiments include control circuitry 420 or 428 that generates an extended reality experience for display on an extended reality device, such as a VR headset. 420 or 428 determines whether the current user is experiencing or will experience motion sickness based on the path taken by the current user in the extended reality experience. Because some routes have more motion sickness factors than other routes, or some points or locations along the same route have more motion sickness factors than other points and locations, Control circuitry 420 or 428 analyzes the routes and determines which is more suitable for the current user. Some examples of such elements are sharp turns or orientations of the user in either the virtual or real world, high speeds, accelerations into negative gravity, steep drops in roller coasters, center of gravity in the virtual or real world. deviation of the environment, and exposing the user to high altitudes.

In some embodiments, control circuitry 420 or 428 may determine that the current user is experiencing or will experience motion sickness based on the path taken in the extended reality experience. in response to activating a smart home device to manage the current user's motion sickness. In these embodiments, control circuitry 420 or 428 determines smart home devices that are in the vicinity of the user, ie, within a threshold distance. To that end, control circuitry 420 or 428 scans for smart home devices, such as by using Bluetooth, and pairs the discovered smart home devices with the extended reality device, such as via an API. .

A threshold distance between the extended reality device and the discovered smart home device is determined by control circuitry 420 or 428. If the distance is within a threshold set by control circuitry 420 or 428, the smart home device may be used by control circuitry 420 or 428. A device that is a certain distance away, such as outside the room, outside the current user's FOV, or 30 feet away, may be considered to be outside the threshold distance.

In some embodiments, control circuitry 420 or 428 and the smart home device are turned on once a determination is made that the current user is likely to experience motion sickness. To make such a determination, control circuitry 420 or 428 obtains machine learning data from multiple users who have taken the same path as the current user. Control circuitry 420 or 428 then compares the current user's path to a representative, average, or median path from multiple users, or another representative path. Control circuitry 420 or 428 determines a location on the route where multiple users experience motion sickness, and determines whether the current user is on the same route where a representative user from the multiple users experiences motion sickness. , or approaching the same location on the same route, the control circuitry 420 or 428 determines whether the current user is also likely to experience motion sickness on the same route or the same location along the route. is determined to be high.

Some examples of smart home devices activated by control circuitry 420 or 428 include fans, air conditioners, lights, refrigerators, exercise equipment, temperature controllers, window opening devices, and chair reclining devices.

In some embodiments, control circuitry 420 or 428 accesses the current user's user profile. Control circuitry 420 or 428 obtains the user's preferred settings for smart home devices from the user profile. For example, a current user may prefer the air conditioner to be at 70 degrees Fahrenheit, the fan to be at a third speed, or the lights to be at a certain brightness. Control circuitry 420 or 428 uses such preferences and turns on smart home devices to preferred user settings.

In another embodiment, control circuitry 420 or 428 determines whether the activated smart device has resulted in reducing motion sickness for the current user. This may be done so based on the user's current performance in the extended reality experience, such as the user not scoring at the same percentage. It may also receive input from the current user indicating that the user continues to experience motion sickness. This may also lead to a conclusion that the current user is experiencing motion sickness if the current user's vitals match those associated with motion sickness, such as sweating or increased body temperature. The control circuitry 420 or 428 may turn the air conditioner to a lower temperature and the electric fan to a higher temperature in response to determining that the activated smart device is not currently producing results that reduce motion sickness for the user. Settings on smart home devices may be increased to a higher level, such as speed, opening windows wider, etc. Control circuitry 420 or 428 may also activate more than one smart home device simultaneously to increase efforts to reduce motion sickness.

13A and 13B are a flowchart of a process for collecting data related to a user's handling of a predetermined level of difficulty for a virtual experience or game. In one embodiment, FIG. 13A manages the predetermined level of difficulty and provides a link between the current user's performance in the virtual experience or virtual game and the level of difficulty predetermined by the control circuitry or content creator. Relates to adjusting the level of difficulty based on the comparison. In other words, determine whether the current user handled the system-set difficulty level with ease, effort, or as expected. On the other hand, FIG. 13B compares the current user's performance data in the virtual experience or virtual game to the median, representative value, or representative user. Median user data is calculated based on data, also referred to as performance metrics, compiled by the control circuitry from multiple users who played the virtual game or experienced the virtual experience.

In particular, according to some embodiments, FIG. 13 collects data related to a user's handling of a level of difficulty in a virtual game, uses the data to enhance the level of difficulty, or 3 is a flowchart of a process for managing.

In some embodiments, at block 1310, control circuitry 420 or 428 may generate a virtual game environment with a predetermined level of difficulty between checkpoint 1 and checkpoint 2 of the virtual game. . In another environment, if this is not a virtual game, this may be some other virtual experience that may require the user to navigate from checkpoint C1 to checkpoint C2, with obstacles in between. .

In one embodiment, a predetermined level of difficulty may involve certain types of obstacles or challenges. Control circuitry 420 or 428 may anticipate that the user will require a certain amount of time or a certain amount of steps before solving these obstacles and challenges.

In one embodiment, a level of predetermined difficulty may involve clearing or passing a gaming level after a period of time. For example, in a virtual game that is at a higher or advanced level in the game, such as level 6 in an eight-level virtual game, the game designer may require the user to at least Level 6 may be designed to require 3 minutes and a minimum of 5 attempts. Accordingly, control circuitry 420 or 428 may be configured to determine whether the level of designed difficulty at level 6 has been successfully overcome by the user before the three minute threshold time.

In one embodiment, the predetermined level of difficulty may be accompanied by a certain speed rating designed by the game designer. For example, in a virtual car racing game, it is possible for a user to take a curve faster than 70 mph without colliding with a wall or losing control of the car, causing the car to spin in a sharp circle. There may be curves that are designed not to yield. In this embodiment, the control circuitry 420 or 428 allows a user to drive his or her virtual car in a virtual car racing game above a difficulty threshold of 70 mph without hitting a wall or spinning uncontrollably. may be configured to determine whether it is possible to do so.

In another embodiment, a predetermined level of difficulty may involve passing through a path within a threshold number of attempts. For example, in a virtual game, a path between checkpoint C1 and checkpoint C2 may be such that it is impossible for the user to succeed in reaching the end of the path in fewer than three attempts. can be designed to a degree level. In this embodiment, the control circuitry 420 or 428 may be configured to determine whether the user is able to reach the end of the path in fewer attempts, such as one or two attempts. good.

In yet another embodiment, the predetermined level of difficulty requires the user to discover a certain number of assets that are hidden from the user and to solve a certain task in order for the hidden assets to be exposed. It may be designed to require. The difficulty level in this embodiment may be designed such that the user cannot discover all the assets without being killed at least once or within some predetermined amount of time. In this embodiment, the control circuitry 420 or 428 is configured to determine whether the user is able to discover more than a threshold number of assets or pass the level in advance of the expected time. It's okay.

At block 1315, control circuitry 420 or 428 may monitor the user's navigation through checkpoints C1 and C2 of the virtual game. Monitoring may be performed by cameras, sensors, heart rate monitors, or any other wearable device connected to the extended reality headset via an API. For example, as mentioned in some of the embodiments above, the control circuitry 420 or 428 allows the user to complete the obstacle course in fewer attempts or in a shorter amount of time than is designed for the difficulty level. Or the user's navigation may be monitored to determine whether it is possible to successfully complete a task or reach the end of a level or path. Control circuitry 420 or 428 then compares the user's performance to the difficulty level set for that level of the game by control circuitry 420 or 428, allowing the user to easily beat the difficulty level. It may be determined whether the

At block 1325, one embodiment determines whether A) the user is able to successfully reach checkpoint C2, and whether the user is able to do so within a predetermined difficulty level threshold. A decision is made that there is. In other words, the user performs as expected or at the difficulty level for which the difficulty level was designed, e.g. When designed, it is possible to reach the end of the path in four tries. In another example, the user can complete the difficulty level in 3 minutes, and the level is designed such that it is not possible for the user to reach checkpoint C2 in less than 2 minutes. Ta. In response to such a determination, at block 1330, the system determines whether the user meets or exceeds the difficulty level set for the route between those two checkpoints. The user may continue to be monitored throughout the remainder of the game, such as through checkpoints C2 and C3.

At block 1325, in one embodiment, a determination is made that B) the user is able to successfully reach checkpoint 2 before reaching a predetermined difficulty level threshold. In other words, the user can easily overcome the difficulty level set for this stage, such as by beating the level in a shorter amount of time or fewer attempts than the game level was designed for, i.e. before reaching the threshold level. It is possible to exceed. In response to such a determination, at block 1335, the system automatically increases the level of game difficulty or changes the game's design to increase the level of difficulty for future users. It may also be possible to provide suggestions to other parties.

At block 1325, in one embodiment, a determination is made that C) the user is able to reach checkpoint C2 after a predetermined difficulty level threshold. In other words, the user is not able to easily surpass the difficulty level set for this stage, e.g. the user takes too much time or too many attempts to reach checkpoint C2. There is. In response to such a determination, at block 1340, the system may automatically decrease the level of game difficulty. It may also suggest to the game designer to decrease the level of difficulty and provide data on the time or number of attempts the current user has taken to complete the difficulty level.

Referring now to FIG. 13B, a diagram illustrates a process for comparing a current user's performance to a median user's performance based on a predetermined difficulty level in a virtual game, according to some embodiments of the present disclosure. Draw a flowchart for.

In some embodiments, control circuitry 420 or 428 determines the types of enhancements and modifications to implement for the virtual game or experience based on how the current user performs in the virtual game or experience. do. For purposes of explanation, reference will be made to a virtual game; however, the processes of FIGS. 13A and 13B apply to both virtual games and any other type of virtual reality or extended reality experience. Control circuitry 420 or 428 generates a virtual game on a display of an electronic device, such as an extended reality headset, and predetermines levels of difficulty for portions of the virtual game. Control circuitry 420 or 428 collects performance data from multiple users who have played portions of the virtual game. For example, performance data may include the number of points scored, levels completed, obstacles surmounted, objects interacted with, time taken to accomplish a certain task or game level, and how a player performs and how a player plays a game. may include other game-related metrics that may be used to determine whether the game performed well during the segment. This performance data relates to the performance of multiple users based on a predetermined difficulty level. For example, by the number of points scored by multiple users under a given difficulty level, by whether the user performed well, by scoring more than the content creator could have expected for that level of difficulty, etc. , whether the user exceeded the expected performance range, etc. Control circuitry 420 or 428 calculates a median grade based on the performance data collected from the plurality of users and compares it to the current user's performance data. Control circuitry 420 or 428 performs augmentation of the virtual game in response to determining that the current user's performance data is not within a threshold of median performance data. These enhancements may include maintaining the same level of difficulty, or increasing or decreasing the level of difficulty. To implement these embodiments, control circuitry 420 or 428 performs the process of FIG. 13B as described below.

In some embodiments, at block 1310, control circuitry 420 or 428 may generate a virtual game environment with a predetermined level of difficulty between checkpoint 1 and checkpoint 2 of the virtual game. . A predetermined level of difficulty relates to a certain type of obstacle or challenge that a player of the game must experience in order to score or reach the next level of the virtual game. The predetermined level of difficulty is determined by the content creator or control circuitry 420 or 428 such that the player experiences certain hardships to make the game challenging, e.g. In order to successfully pass a game level, you may need to spend a certain amount of time, take a certain amount of steps, lose a certain amount of lives, take a certain path that is not easily discovered, etc. may be designed.

One example of a given level of difficulty in a game is that the curves are designed to be sharp, such that the player must slow down on the curves in order to reach the goal line within a certain amount of time. This includes making it difficult to clear or pass a gaming level within a certain time period, such as in a car racing game. In yet another example, the game may require a certain number of attempts by the player to reach the end. In another example, a player may need to discover a certain number of assets/objects that are hidden from the player and that are exposed after the player successfully navigates to an area in the game. obtain.

Whatever challenges or obstacles may be designed into the game to cause a certain level of difficulty, the control circuitry 420 or 428 at block 1345 creates checkpoints 1 and 2 with a predetermined level of difficulty. Acquire performance data such as performance metrics from multiple users who played the game between them. By accumulating performance data, control circuitry 420 or 428 may determine whether multiple users have experienced a predetermined level of difficulty designed within the game to be experienced by the users. good.

In one embodiment, control circuitry 420 or 428 may include a threshold for a predetermined level of difficulty. For example, if the predetermined level of difficulty for a player in a car racing game is to finish the race at an average speed of 75 MPH, the threshold may be set to +/-15 MPH, which is greater than at 60 MPH. Resulting in a threshold or performance window for lower performance and higher performance at 90 MPH. In other embodiments, the threshold may be related to time or number of trials. For example, if the user should navigate from checkpoint 1 to 2 after 5 attempts, the threshold may be set to +/−2 attempts.

At block 1350, in one embodiment, control circuitry 420 or 428 calculates a median performance based on performance data, such as performance metrics, obtained for the plurality of users. Median performance data may be the representative value for all multiple users, the average value for all multiple users, the standard deviation for all multiple users, or some other equation or method for calculating median performance. It may also be a statistical measure.

At block 1355, the median user, which in one embodiment is a hypothetical user determined based on calculating the median or other type of statistical data from the plurality of users, reaches checkpoint C2. A determination is made whether it is possible to do so successfully and within a predetermined difficulty level threshold. In other words, is the median user able to perform at the difficulty level as expected for which the difficulty level was designed? From the example above, in a car racing game where the speed for a given difficulty level is set to have an average speed of 75 MPH throughout the game with a threshold of +/-15 MPH, the determination by the control circuitry is: To determine if the median user was within +/-15 MPH of 70 MPH, in other words between the lower performance at 60 MPH and the higher performance at 90 MPH.

If a determination is made that the median user has completed the task, i.e. reached checkpoint 2, or in the car racing example, reached the end of the race outside the threshold, then at block 1365 the difficulty The threshold may be increased or decreased based on performance. In one embodiment, if the median user's performance falls below the lower end of the threshold of 60 MPH, the control circuitry 420 or 428 determines that the current predetermined difficulty level is too high for the median user and that this is reduced. may decide that it needs to be done. In a car racing embodiment, the control circuitry 420 or 428 reduces the difficulty level of a 75 MPH average speed to 65 MPH and provides a +/-15 threshold or some other threshold, such as +/-20, to provide a difficulty level. may be adjusted, thereby making it easier for the median user to reach the game level. In other embodiments, reducing the difficulty level may be implemented by blocking certain paths in the virtual game that have a higher difficulty level. In another embodiment, if the median user's performance falls within 90 MPH, above the upper threshold, the control circuitry 420 or 428 determines that the current predetermined difficulty level is low for the median user and that it is increased. You may decide that you need to In a car racing embodiment, the control circuitry 420 or 428 increases the difficulty level for a 75 MPH average speed to 100 MPH and provides a +/-10 threshold or some other threshold, such as +/-5, to increase the difficulty level to 100 MPH. may be adjusted, thereby making it more difficult for the median user to reach the game level.

If a determination is made that the median user has completed the task, i.e. reached checkpoint 2, or in the car racing example, reached the end of the race within a predetermined threshold of 75 MPH, then at block 1365 , the difficulty threshold may be maintained. This is because control circuitry 420 or 428 may determine that the currently set predetermined threshold is the correct degree of difficulty for the median user.

At block 1370, the current user performance is compared to the updated level of difficulty at block 1365 or the maintained level of difficulty at block 1360 based on the determination made at block 1355. Blocks 1375, 1385, and 1380 follow a process similar to blocks 1355, 1365, and 1360 to determine whether to increase, decrease, or maintain the same level of difficulty, but A comparison with the updated level of difficulty at block 1365 or the maintained level of difficulty at block 1360, ie, based on how the median user was able to perform.

In one embodiment, as described through blocks 1345-1385, control circuitry 420 or 428 performs at least two iterations of the check for a predetermined level to increase, decrease, or maintain it. You may decide whether or not to do so. The first iteration may be a check based on the method performed by the median user based on a predetermined difficulty level, and if that level needed to be adjusted for the median user; The second iteration is for a particular current user to determine whether further adjustments from those made for the median user are needed to customize for the current user. It may be. In addition to the above, statistics that are specific to the current user, such as those depicted in FIG. 8, may also be considered in further adjusting the level of predetermined difficulty. Other factors may also be factored in to adjust the difficulty level, such as whether the current user has played similar games or is an expert in the field of such games. For example, if control circuitry 420 or 428 determines that the user is an expert in such a game, the difficulty level may be increased to make this more challenging for the current user. good.

In another embodiment, control circuitry 420 or 428 may manage motion sickness in an extended reality experience by generating the extended reality experience on a display of an electronic device, such as an extended reality headset. Control circuitry 420 or 428 may determine first and second checkpoints in the extended reality experience, such as the checkpoints depicted in FIG. Control circuitry 420 or 428 determines a median path (or some other representative path) based on data from the plurality of users navigated between the first checkpoint and the second checkpoint. You may. This may be done so by calculating a representative value, an average value, or some other calculation based on an equation or some other statistical measure to obtain median data. Control circuitry 420 or 428 may determine the location in the median route where a threshold number of users from the plurality of users experienced motion sickness. For example, a median path may include a number of location points along the path, such as locations 1-n as the user navigates from a starting checkpoint to an ending checkpoint. Control circuitry 420 or 428 may determine whether the path taken by the current user is aligned with the median path. For example, control circuitry 420 or 428 may determine whether the current route is along the same location points as routes selected by multiple users.

In response to determining that the current user's route is aligned with the median route, the control circuitry 420 or 428 then determines whether the user is on the median route on which a threshold number of users have experienced motion sickness. Improvement actions may be taken prior to reaching a location. For example, if a route has location points 1, 2, 3, 4, and 5, and the median user experiences motion sickness at location point 4, such as due to a descent on a roller coaster, then such Data may be logged by control circuitry 420 or 428.

Control circuitry 420 or 428 may determine that the current user is also taking the path taken by the median user. It may also determine the distance the median user has traveled, eg, the distance the median user could have traveled so far by crossing location points 1, 2, and 3. The control circuitry 420 or 428 determines that the current user is on the same route and location point 4 is the location where motion sickness is experienced, so that as a preventive action, motion sickness is minimized. or the remedial action may be proactively carried out prior to the user reaching location point 4 so that it can be avoided.

In some embodiments, control circuitry 420 or 428 determines a first set of median path coordinates and then associates the first set of coordinates with the current path taken by the user. By comparing with the current set of coordinates, it may be determined whether the route taken by the current user is aligned with the median route. If control circuitry 420 or 428 makes a determination that the current set of coordinates is within a threshold distance of the first set of coordinates, control circuitry 420 or 428 may take remedial action. The threshold distance may be set by a content producer in control circuitry 420 or 428.

FIG. 14 is a navigation tree depicting routes taken by multiple users between predetermined checkpoints within a virtual game, according to some embodiments of the present disclosure.

The median path followed by the user is represented in x, y, z coordinates. The median path may also include orientation in addition to translation of the x, y, and z coordinates; however, for simplicity and explanation, only the x, y, z coordinates are depicted.

In some embodiments, the user's path between checkpoints C1 and C2 is determined using a common tree. For example, between checkpoints C1 and C2, the difference between two consecutive recordings may represent a state change of the user's position in the 6DOF virtual experience. Starting from the root node initialized to x, y, and z coordinates of 0, 0, 0, i.e. the initial position or origin at which the virtual experience begins, the records are checked against the children of the root node. Ru. In some embodiments, if a child representing a state change in x, y, and z already exists, this is selected as the next starting point and represents the traversal of the edge between the root node and its children. A count is incremented. If a child representing a state change in x, y, and z does not exist, a new child node is created and the edge is attributed a count of 1. This process is executed for each new record until checkpoint C2 is reached.

In one embodiment, a route tree may be generated based on navigation between checkpoints C1 and C2 by 12 users. The leaf node with the highest traversal count presents the final state change of the user's median path. Traversing the tree from the root node to the leaf node with the highest traversal count determines the median path taken by the user in the experience.

The leaf node traversal count (-1,0,0) is highest at n=5. Therefore, the user's median path is modeled as a (-3,-2,0) state change followed by a (-1,0,0) state change. Therefore, the total state change is the sum of the two resulting in the movement to (-4,-2,0).

Although discrete changes in x, y, and z coordinates are depicted, in reality each discrete state change, i.e., a node in the general tree, has a range of movement in the x, y, and z directions. May contain errors. For example, errors of x±6x, y±6y, and z±6z may be experienced, effectively creating "zones" of user movement (like a chessboard).

FIG. 15 is a block diagram depicting a user's field of view and objects/assets encountered within the field of view, according to some embodiments of the present disclosure. In some embodiments, a field of view (FOV) from the user's eyes is depicted. In some embodiments, content creators must observe patterns in a user's FOV in order to build a virtual experience with a good level of user engagement. By determining and monitoring a user's FOV, content creators may obtain data regarding objects appearing within the user's FOV. Since a well-designed extended reality experience is one in which users spend time, feel engaged, and interact with objects within their FOV, such FOV data is Provides insight into whether your designs match user observations and actions.

In one embodiment, the user's FOV is represented in the diagram as including the entire projection plane from the left vanishing point to the right vanishing point. The vanishing point 1540 is where the user's field of view is the smallest, and the center of the field of view pyramid 1530 is where the field of view is largest and towards which the user's eyes can be directed. As depicted, in this example, the user's FOV includes object 1510. The control circuitry determines that the user's gaze is directed toward the object 1510 if the viewing pyramid, which is a narrower segment of the user's eye than the entire FOV, is directed toward the object. You may decide that. To make such a determination, in one embodiment, the control circuitry may utilize an inward-facing camera associated with the extended reality headset to monitor the user's eye movements. In another embodiment, the control circuitry determines the orientation and coordinates of the extended reality headset and then calculates the viewing pyramid, thereby determining whether the user's gaze is directed towards the object 1510. may be determined.

In another embodiment, in addition to oriented the user's vision pyramid 1530 in a particular direction, the control circuitry also determines whether the user is viewing the near vision 1545 or the far vision 1550. You may. Such data may enable the control circuitry to determine the depth of the user's view and whether the user is viewing objects closer or further away within the pyramid of view. Such types of gaze tracking may be used in conjunction with position and rotation vectors to reduce the set of assets in view, thus improving the information available to users and content creators. For example, to determine whether a content creator should remove clutter by removing objects in the background that are not related to the game or virtual experience and are causing confusion or lack of focus for the user. Gaze information may also be used.

In some embodiments, the FOV is determined based on headset coordinates. The control circuit may obtain coordinates of the user on the path and use them to determine orientation of the extended reality headset. Orientation determination may allow control circuitry to determine possible objects within the FOV. The control circuit may further determine whether the user has interacted with an object within the FOV.

FIG. 16 is a block diagram of a room layout and multiple routes taken to reach a destination, according to some embodiments of the present disclosure. In some embodiments, a user may use a virtual reality headset or glasses when navigating within a room to reach a destination. As depicted, the layout in one embodiment includes a chaise longue 1610, chairs 1620, 1625, 1630 and other chairs, a table 1635, a lamp 1640 and other objects such as another lamp, and multiple bookshelves. The diagram also depicts a route 1650 taken by a first user and a route 1660 taken by a second user. The diagram further depicts certain locations along the route, such as locations 1670, 1680, and 1690, through which the user passes. Checkpoint C1 may be the starting location of the route, and the goal of the virtual experience may be, in one embodiment, to reach checkpoint C2.

Because different users may take different routes from checkpoint C1 to reach checkpoint C2, the users may spend different amounts of time and encounter different challenges between the two checkpoints. For example, one user may choose an easier or shorter path than another, one user may lose a life while another may pass through it without losing a life, or One user may be able to encounter more assets and interact with them based on the chosen path than another user. Different users may also have different experiences in discovering assets in a game. Some users may discover assets that are critical to completing a level faster or in fewer attempts than others. Some users may choose to stay inside longer to enjoy the experience more and explore their surroundings. If the experience is a game, one user can interact with more assets based on the path chosen and the number of assets viewed and interacted with compared to a user who interacted with fewer assets. get points. Data collected per route and by different users can be used by content creators to enhance future virtual experiences, adjust difficulty levels, manage motion sickness, or determine home automation options. may be done.

In one embodiment, as depicted, the path navigated by user 1 is different from the path navigated by user 2. The path navigated by user 1 passes through location 1690 and reaches its final destination at checkpoint C2, where ramp 1640 is located. As user 1 navigates along path 1, at each location along the path, the user's FOV is determined based on the orientation of the extended reality headset that user 1 is wearing. , lamps, and bookshelves.

In one embodiment, at location 1690, User 1's extended reality headset may be oriented in the direction of the arrow. Based on the orientation, the user's FOV 1655 at location 1690 may allow User 1 to view table 1635 and chair 1625 through his headset. User 1 may also partially view chair 1630 within his FOV. All other assets, including chair 1620, may not be visible to User 1 at location 1690 when the headset orientation is toward the arrow. Such data may be captured by control circuitry 420 or 428 to enhance future virtual experiences, adjust difficulty levels, manage motion sickness, determine home automation options, and other use cases. It may be used for various purposes, such as for

Based on the orientation in the direction of the arrow at location 1690, User 1 may be able to interact with assets 1635, 1625, and 1630. If the extended reality experience is a virtual game, the user may score points based on the user's interaction with these assets. Data related to all user interactions with assets, including assets viewed and interacted with, assets viewed but not interacted with, and extended reality display coordinates, translations, and orientations are shown in FIG. and 19 embodiments may be captured by the control circuitry. Data related to the time spent at each asset and the time required to navigate from one location to another along Pathway 1 may also be captured by control circuitry 420 or 428.

The path navigated by user 2 passes through locations 1670 and 1680 and reaches its final destination at checkpoint C2, where ramp 1640 is located. Locations 1670, 1680, and 1690 are arbitrarily chosen along the paths of users 1 and 2 to describe the users' FOV and assets that can potentially be interacted with. Any other location along the route may also have been chosen instead for a similar description of the FOV.

As user 2 navigates along path 2, at each location along the path, the user's FOV is determined based on the orientation of the extended reality headset that user 2 is wearing. , lamps, and bookshelves.

In one embodiment, at location 1670, user 2's extended reality headset may be oriented in the direction of the arrow. Based on the orientation, the user's FOV 1665 at location 1670 allows user 2 to view bookshelf 2 and partially view bookshelf 3, thereby creating an opportunity for user 2 to interact with them. can be done. All other assets, including bookshelf 1, cannot be visible to user 2 at location 1670 when the headset orientation is toward the arrow.

As User 2 progresses along the path selected by User 2 in the Extended Reality experience, in one embodiment, at location 1680, User 2's Extended Reality headset points to Bookshelf 2 in User 2's FOV. may be oriented in the direction of the arrow, which may make it possible to see only the Thus, the user 2 in the book orientation may have the opportunity to interact only with the bookshelf 2 (if the user so chooses).

User 2's navigation data, including what User 2 has seen within his FOV at all locations and the opportunities that have been available to User 2 based on what User 2 has seen within his FOV, is transmitted to the control circuitry. network 420 or 428 for a variety of purposes, such as enhancing future virtual experiences, adjusting difficulty levels, managing motion sickness, determining home automation options, and for other use cases. may be used for

In some embodiments, User 2 may be taking a shorter route than User 1, however, User 1 may have the opportunity to interact with more assets than User 2. If asset interactions are captured solely based on selected location points 1670, 1680, and 1690, in one embodiment, User 1 interacts with more assets (three chairs and a table). User 2 may be interacting with only two items (Bookshelves 2 and 3). Depending on how the content creator designed the extended reality game, if equal points were distributed for each interacted asset, user 1 would receive more points than user 2 based on the number of his interactions. get points. However, if a greater number of points were allocated for the bookshelf than for other objects, in one embodiment, the user 2 who interacted with the bookshelf would be considered to have a higher number of points than the other objects. User 1 can obtain more points than User 1 who chose the route that was not taken. In another embodiment, if the path length or the time required to reach checkpoint C2 from checkpoint C1 is given a higher value, user 2 takes the shorter path and reaches the goal in a shorter amount of time. may earn more points than User 1 for reaching the ground. In another embodiment, each step along the path may have a different FOV, and the control circuitry obtains data related to assets within the FOV along each step, and the assets that were in view and The interacted assets may be determined and tables such as those shown in FIGS. 18 and 19 generated.

In one embodiment, user navigation and routes may be monitored by control circuitry 420 or 428 on a periodic or continuous basis. Monitoring may be performed with respect to any and all movements and orientations of the extended reality headset.

In another embodiment, the control circuitry 420 or 428 provides monitoring and logging time periods based on scene complexity or interaction opportunities (based on path, orientation, assets in view, interaction opportunities, etc.). may be modified. Control circuitry 420 or 428 may also log data more frequently when scene complexity is high and less frequently when it is low.

In yet another embodiment, route monitoring and data logging is performed periodically at a given location or after a time frame to manage CPU and GPU utilization so that they are not overloaded. may be carried out. The amount of data collected may also be capped by the amount of memory space to be used within the CPU or GPU.

In addition to route data, control circuitry 420 or 428 may also monitor and log data related to motion sickness, home automation opportunities, difficulty status, and the like. Control circuitry 420 or 428 may also log data such as heart rate, ECG, and other user vitals to gain more insight regarding the content. Control circuitry 420 or 428 may also track the user's gaze, along with position and rotation vectors, to determine (and in some cases reduce) the set of assets in view. Examples of tables of logged data are depicted in FIGS. 18 and 19.

Control circuitry 420 or 428 optimizes the content once the median path taken by the user, the median orientation per point along that path, assets in view, and CPU/GPU utilization are determined. The collected and logged data may be used to Often, if there are assets that are rendered at very high detail (e.g., millions of polygons) that are not within the user's FOV, then during a period of time the control circuitry 420 or 428 uses the peak GPU Observe utilization and take appropriate action to reduce the level of detail for high-detail assets or remove them altogether.

In some embodiments, control circuitry 420 or 428 may determine that CPU or GPU utilization is higher than expected or above a predetermined threshold. Having a utilization rate above such a threshold means that the GPU may not be able to deliver rendered frames at the rate required to maintain consistent frames per second in the experience, thereby , may be an indication that the GPU is skipping a frame, thus having consequences that negatively impact the user experience. In such situations, control circuitry 420 or 428 may reduce the complexity, detail, resolution, and/or number of polygons used to avoid over-utilizing the GPU.

In some embodiments, in response to the control circuitry 420 or 428 determining that the CPU or GPU utilization is higher than expected or above a predetermined threshold, the control circuitry 420 or 428 may partially or completely change the spatial arrangement of assets or layouts. For example, in FIG. 16, control circuitry 420 or 428 may remove additional numbers of chairs that were not viewed or interacted with, or may relocate them to another side of the virtual room. Control circuitry 420 or 428 may also reduce the amount of detail of the lamp on the left side of the room that was not seen or interacted with, or its The spatial layout may be moved closer to the median path. Some additional layout changes that may be implemented are described below in FIG. 17.

In some embodiments, in response to the control circuitry 420 or 428 determining that the CPU or GPU utilization is higher than expected or above a predetermined threshold, the control circuitry 420 or 428 Reduces the number of times data is collected and logged; collects and logs data periodically, at regular intervals, or less frequently when scenes change or milestones are achieved. You may attach it.

FIG. 17 is a block diagram of layout recommendations, according to some embodiments of the present disclosure.

In some embodiments, at block 1705, control circuitry 420 or 428 changes the layout in the extended reality experience by increasing the difficulty level. As mentioned above, one of the reasons for acquiring data from multiple users when a user navigates from checkpoint C1 to C2 is that multiple users are using extended reality experiences such as in VR games. This is to determine whether you have experienced the level of difficulty as designed and anticipated by the creator. Multiple users have overcome challenges and obstacles designed by the content creator and reached checkpoint C2 in less than a threshold time, etc., designed to experience a certain level of difficulty, or more than what is designed. If a determination is made by control circuitry 420 or 428 that more points have been scored, control circuitry 420 or 428 may increase the difficulty level. Allowing multiple users to overcome the difficulty of a game as designed may be an indication that the difficulty level is easy for the user to navigate, and therefore the control circuitry 420 or 428 may Difficulty levels may be increased for a challenging experience.

In some embodiments, at block 1710, control circuitry 420 or 428 changes the layout in the extended reality experience by decreasing the difficulty level. As mentioned above, one of the reasons why data such as performance metrics is obtained from multiple users when the user navigates from checkpoint C1 to C2 is that multiple users This is to determine whether the experience has experienced a level of difficulty as designed and anticipated by the creator of the Extended Reality Experience. multiple users had difficulty reaching checkpoint C2, such as over a threshold time designed to experience a certain level of difficulty, overcoming challenges and obstacles designed by the content creator, or If a decision is made by the control circuitry 420 or 428 that they have scored less than what was designed for, or if some users have abandoned the game due to the level of difficulty, the control circuitry 420 or 428 may reduce the difficulty level so that the user can reach checkpoint C2 with less difficulty.

In some embodiments, at block 1715, control circuitry 420 or 428 changes the layout in the extended reality experience by changing assets in the layout, such as by adding or removing assets. For example, control circuitry 420 or 428 may determine that an asset was not noticed by multiple users and therefore the users did not interact with it. In another example, control circuitry 420 or 428 may determine that the additional assets add more processing on the CPU or GPU and are not essential to the extended reality experience. In yet other embodiments, control circuitry 420 or 428 may determine that a higher level of difficulty is added by adding more assets. Whatever the reason for adding or removing assets, control circuitry 420 or 428 does so based on performance metric data collected from multiple users as they navigate from checkpoint C1 to C2. It's okay.

Similar to adding or removing assets, control circuitry may add or remove obstacles at block 1720. One of the reasons for adding or removing obstacles may be to increase or decrease the level of difficulty. For example, control circuitry 420 or 428 indicates that multiple users have solved the current task in less than a threshold time designed to solve such task or bypass such obstacle; If it is determined that the object has been bypassed, more obstacles may be added. In another example, the control circuitry 420 or 428 may indicate that the plurality of users have not solved the current task or circumvented the current obstacle; An obstacle may be removed if it determines that it has done so in an amount of time that exceeds a predetermined threshold time designed to bypass such an obstacle. Obstacles may also be removed if the user abandons the game and has not solved any challenges of the bypassed obstacles.

In some embodiments, at block 1725, control circuitry 420 or 428 provides translational or directional cues to the user instead of or in addition to changing the layout in the extended reality experience. You may. As depicted in FIG. 15, arrow 1520 directs the user to orient toward the object if the user is not able to figure out where the object is located within a threshold amount of time. Good too. Having a translation cue is also an option that control circuitry 420 or 428 may activate if it determines that the user is about to abandon the game or is heading in the wrong direction. good.

In some embodiments, at block 1730, the control circuitry 420 or 428 modifies the layout in the extended reality experience by modifying possible paths that the user may take to reach the ending checkpoint C2. do. For example, a determination may be made by control circuitry 420 or 428 that multiple users are proceeding on a path that is an incorrect path or a path that causes the users to exceed a threshold amount of time to reach checkpoint C2 therein. If so, the control circuitry 420 or 428 may block the erroneous route, create another detour within the route to direct the user back to the correct route, or provide an indicator that the route is the erroneous route. may be provided.

In some embodiments, at block 1735, the control circuitry 420 or 428 colors, highlights, animates, increases the size, or any other coloring that would draw attention to the object that is to be found by multiple users. Enhancements may be displayed, including other enhancements. Although some layout changes and route modifications and enhancements are discussed in FIG. 17, embodiments are not limited to such, and may be used to modify the layout or draw attention to particular areas in an extended reality experience. Other enhancements are also envisioned.

FIG. 18 is a table of user navigation coordinates of user interactions as the user navigates from checkpoint C1 to checkpoint C2, according to some embodiments of the present disclosure.

In one embodiment, control circuitry 420 or 428 may collect data based on the navigation of multiple users from checkpoint C1 to checkpoint C2. Such data is aggregated, an average value is calculated for the data, the average value is compared to the current user data, and determines the type of extended reality experience modification and enhancement that is required; May be used to enhance the current user's experience and make it more enjoyable.

In one embodiment, control circuitry 420 or 428 may use collected data based on multiple users' navigation from checkpoint C1 to checkpoint C2 to generate table 1800. Although several columns of categories are listed in table 1800, embodiments are not so limited; other columns reflecting data regarding other measurements and indications may also be populated in the table. It's okay. For example, data related to acceleration, motion sickness, etc. may also be populated in table 1800.

In some embodiments, the data is transmitted to system components, such as a head-mounted display (HMD) (which is an extended reality device used by a user), left and right hand controllers (such as a virtual reality or extended reality gaming controller) , the position and orientation of the assets in view (such as the table and chairs in Figure 16), and the history of user activity at each recorded point in the simulation. This also includes CPU and GPU utilization based on the path taken by the user and the amount of CPU and GPU usage required to render the image and process the user's steps. In other embodiments, instead of recording data at each point, which may consume excessive memory and CPU/GPU usage, control circuitry 420 or 428 reduces the number of times data is collected and logged; Data may be collected and logged periodically, at regular intervals, or less frequently when scenes change or milestones are achieved.

For example, in a virtual experience, a 3D scene containing a large number of boxes (some of which are "flying" boxes) may be displayed as part of a VR game. These flying boxes can appear transiently. Additionally, control circuitry 420 or 428 may provide the user with the ability to open or close any of the boxes using point-and-click actions on a virtual reality or extended reality controller. When a user performs an action related to a flying box, such as by pointing and clicking with his or her mouse, control circuitry 420 or 428 may extract a table from each log file, such as table 1800. .

Control circuitry 420 or 428 may then aggregate logs received from multiple users, thereby providing content creators with a comprehensive view of user behavior in the extended reality experience. As mentioned above, comprehensive views are used to manage a user's motion sickness, implement home automation, or measure a user's performance through the designed difficulty level of the experience. Good too.

On a granular basis, in one embodiment, control circuitry 420 or 428 may determine a user's median path using aggregated data that provides a comprehensive view. As mentioned above, such a median path is calculated based on representative or average values for data obtained from a plurality of users and the paths taken by the users.

In another embodiment, control circuitry 420 or 428 uses aggregated data to determine the user's median orientation (qw, qx, qy, qz) at each point (or points) on the median path. ) may be determined. Orientation allows control circuitry 420 or 428 to determine the direction in which the extended reality device (HMD) is facing and, based on the facing direction, determine the assets that are within the user's FOV. .

In yet another embodiment, control circuitry 420 or 428 may use the aggregated data to determine the assets viewed by the user and the duration of such views. For example, in some cases a user may only view an asset for a short duration and not interact with the object, while in other cases the user may eventually You can spend a lot of time on guiding assets. The control circuitry may determine a representative value based on multiple users and determine whether the asset is in the representative value of interest to the user.

In another embodiment, control circuitry 420 or 428 uses aggregated data to determine whether multiple users have taken advantage of the opportunity to interact with an asset that has been programmed to interact. It's okay. If assets are not being interacted with, control circuitry 420 or 428 may displace them to a different spatial location, remove them from the game, or reduce the amount of detail, such as polygons, about the assets.

In another embodiment, control circuitry 420 or 428 may use aggregated data to determine large changes in x, y, z (high translation) across successive time periods. Determining such large changes helps content providers understand whether large translational movements, which imply faster movement or jumping from one location to another, are leading to motion sickness in the user. It may be possible to do so. The content provider may use such data to determine whether any enhancements related to motion sickness are required. Similarly, large changes in qw, qx, qy, qz across successive time periods (high angular movements) may also be monitored, recorded and used to determine any actions required. good. In addition, data related to the time rate of position change movements and angular movements due to large movements can help content creators monitor events that occur in the user experience in order to control abnormal/high movement in games. Allows you to understand where reductions can be made.

In yet another embodiment, control circuitry 420 or 428 uses the aggregated data to determine the time required to complete a set of tasks or sequence of tasks or the time required between completing tasks. Good too. Determining such data related to task completion time may be used by the control circuitry to determine whether the user is accomplishing the task at a level of difficulty less than that for which the level was designed, or whether the level of difficulty is may allow the user to decide whether to spend more than a threshold amount of time or lead to abandoning the game.

In some embodiments, the time required to complete a set of tasks or a sequence of tasks may represent the total time spent between checkpoints C1 and C2. When the user reaches checkpoint C2, a timestamped entry may be registered to indicate the achievement. If the user exits the experience before reaching checkpoint C2, the absence of this entry in the log file may be noted; such an entry indicates that the user did not complete the task to reach checkpoint C2. It can be shown that Aggregation of such data may be used to evaluate and adjust the level of difficulty in the experience.

Similarly, data related to the time taken to complete two individual tasks from the interaction/user activity column may be captured and logged. In some embodiments, after performing the first task, i.e., "Open (Box-7)," the user is prompted to perform the second task, i.e., "Open (Box-11)." When receiving a cue to guide the user, the control circuitry may use such data to determine the number of user sessions out of the total that successfully interpreted the cue. Timestamps recorded regarding user actions may be used by control circuitry to determine the time between completing two tasks.

FIG. 19 is a table of user interactions and missed opportunities, according to some embodiments of the present disclosure.

In one embodiment, at time t=T, the assets in the user's FOV may be Box-1, Box-2, Box-3, Box-4, Flying Box-1. The interaction opportunities provided to the user in the game are Open (Box-1), Open (Box-2), Open (Box-3), Open (Box-4), and Open (Flying Box-1). obtain. However, as depicted in the third column "Interaction/User Activity", the user had zero opportunities in the first time interval.

In another embodiment, at time t=2T, the user takes advantage of one of five possible opportunities in the second time interval, and in another embodiment, at time t=3T. In , the user took advantage of 3 out of 5 opportunities in the third time interval. Such aggregated data may be used to make inferences regarding whether users are taking advantage of opportunities provided and, if applicable, opportunities that are and are not being taken advantage of.

FIG. 20 is a table of content improvement suggestions based on user navigation in an extended reality experience, according to some embodiments of the present disclosure. In this embodiment, control circuitry 420 or 428 provides content improvement suggestions and/or suggested actions based on metrics or data obtained from multiple users. Some example content improvement suggestions include control circuitry highlighting an object to redirect the user toward or interact with the user, or perform a desired action, or Including suggesting training to perform the behavior. Some example suggested actions are for the control circuitry to display arrows, highlight objects with thick/contrasting colored borders to orient the user to assets or other interaction opportunities, Including suggesting changes to the spatial layout/map or providing behavioral training text and 3D visualization.

FIG. 21 is a block diagram depicting locally rendered and cloud rendered extended reality experiences in accordance with some embodiments of the present disclosure.

Where data is logged and the mechanism for uploading data to content creators may depend on the system architecture. In some embodiments, in cloud-rendered VR experiences, data may be logged directly at a cloud server that receives position and orientation data to render frames for the extended reality headset. . In other embodiments, during a locally rendered extended reality experience, data may be logged within the local computer or extended reality device and later uploaded to a system, such as the system depicted in FIG. good. Attachment to a storage device or extended reality device on a remote server is also envisioned in embodiments.

It will be apparent to those skilled in the art that the methods involved in the embodiments described above can be embodied in a computer program product that includes a computer usable and/or readable medium. For example, such computer-usable media may include a read-only memory device, such as a CD-ROM disk or a conventional ROM device, or a random read-only memory device, such as a hard drive device or a computer diskette, having computer-readable program code stored thereon. It may also consist of access memory. It should also be understood that the methods, techniques, and processes involved in this disclosure may be implemented using processing circuitry.

The processes discussed above are intended to be illustrative and not limiting. The following claims are meant to set forth the boundaries as to what this invention includes. Furthermore, the features and limitations described in any one embodiment may apply to any other embodiments herein, and the flowcharts or examples relating to one embodiment may be used in any suitable manner. Note that it may be combined with other embodiments of, performed in a different order, or performed in parallel. Additionally, the systems and methods described herein may be implemented in real time. It is also noted that the systems and/or methods described above may be applied to or used in accordance with other systems and/or methods.

Claims (36)

A method of enhancing a virtual reality experience, the method comprising:
generating a virtual game on a display of an electronic device, a predetermined difficulty level being determined for a portion of the virtual game;
collecting performance data from a plurality of users who have played a portion of the virtual game, the performance data relating to the performance of the plurality of users based on the predetermined difficulty level;
calculating a performance metric based on the performance data collected from the plurality of users;
comparing current user performance data with the calculated performance metric;
and performing an augmentation of a virtual game in response to determining that the current user performance data is not within a threshold of the performance metric. 2. The method of claim 1, wherein the performance metric threshold includes a lower bound and an upper bound. 3. The lower limit is associated with a minimum number of interactions with virtual objects within the virtual game, and the upper limit is associated with a maximum number of interactions with virtual objects within the virtual game. Method. determining that the current user performance data is below a lower threshold of the performance metric;
3. The method of claim 2, further comprising: modifying the predetermined difficulty level for the portion of the virtual game to a lower predetermined difficulty level in response to the determination. determining that the current user performance data exceeds a higher limit of the performance metric threshold;
3. The method of claim 2, further comprising: modifying the predetermined difficulty level for the portion of the virtual game to a higher predetermined difficulty level in response to the determination. 2. The method of claim 1, wherein the virtual game enhancement is either an increase or a decrease in the predetermined difficulty level. 7. The method of claim 6, wherein the augmentation of the virtual game is a modification of the path that may be taken by the current user within the virtual game. 8. The method of claim 7, wherein modifying the route is associated with disallowing navigation by the current user on a route within the virtual game that was previously allowed for navigation. The method of claim 1, wherein the portion of the virtual game includes a starting checkpoint and an ending checkpoint, the starting checkpoint and the ending checkpoint being determined based on a type of the virtual game. . 10. The method of claim 9, further comprising selecting the starting checkpoint and the ending checkpoint in a shorter time span for faster paced virtual games and in a relatively longer time span for slower paced virtual games. the method of. monitoring the current user's path within the virtual game;
determining a total number of interaction opportunities available based on the path taken by the current user within the virtual game;
2. The method of claim 1, further comprising: determining a number of interaction opportunities utilized by the current user from the total number of available interaction opportunities. 12. The method of claim 11, wherein the current user's performance is based on the number of interaction opportunities performed by the current user out of the total number of interaction opportunities available to the current user. monitoring the current user's gaze as the user navigates a path within the virtual game;
determining objects within a field of view (FOV) of the current user based on the path navigated by the current user within the virtual game;
The method of claim 1, further comprising: determining whether the user interacted with an object within the FOV. 14. The method of claim 13, prompting the user to interact with the object in the FOV in response to determining that the user did not interact with the object in the FOV. . 15. The method of claim 14, wherein the prompt is either a visual alert or an audible prompt displayed on a display of the electronic device. 14. The method of claim 13, wherein the object in the FOV is highlighted in response to determining that the user has not interacted with the object in the FOV. 2. The method of claim 1, wherein the performance metric is selected from the group consisting of median, representative value, mean, standard deviation, or variance. A method,
generating a virtual game on a display of an electronic device;
determining a level of difficulty between a starting checkpoint and an ending checkpoint for the portion of the virtual game;
monitoring the progress of a user within the virtual game, and determining whether the user a) meets the determined level of difficulty; or b) fails to meet the level of difficulty due to not being possible; or c) determining whether said level of difficulty is to be exceeded;
and maintaining, decreasing, or increasing the level of difficulty in response to the determination. Monitoring the user's progress includes:
determining coordinates of a route taken by the user within the virtual game;
determining an orientation of a headset worn by the current user to play the virtual game based on the coordinates;
determining an object within the field of view of the headset based on the determined orientation;
and determining whether the current user has interacted with the determined object. A system for enhancing a virtual reality experience, the system comprising:
communication circuitry configured to access the electronic device;
A control circuit network,
generating a virtual game on a display of the electronic device, a predetermined difficulty level being determined for a portion of the virtual game;
collecting performance data from a plurality of users who played a portion of the virtual game, the performance data relating to the performance of the plurality of users based on the predetermined difficulty level;
calculating a performance metric based on the performance data collected from the plurality of users;
comparing current user performance data with the calculated performance metric;
and control circuitry configured to perform virtual game augmentation in response to determining that the current user performance data is not within the performance metric threshold; system. 21. The system of claim 20, wherein the performance metric threshold includes a lower bound and an upper bound. 22. The lower limit is associated with a minimum number of interactions with a virtual object within the virtual game, and the upper limit is associated with a maximum number of interactions with a virtual object within the virtual game. system. determining that the current user performance data is below a lower threshold of the performance metric;
5. The control circuitry further comprising: modifying the predetermined difficulty level for the portion of the virtual game to a lower predetermined difficulty level in response to the determination. The system described in 21. determining that the current user performance data exceeds a higher limit of the performance metric threshold;
10. The control circuitry further comprising: modifying the predetermined difficulty level for the portion of the virtual game to a higher predetermined difficulty level in response to the determination. The system described in 21. 21. The system of claim 20, wherein the virtual game enhancement is implemented by the control circuitry configured to either increase or decrease the predetermined difficulty level. 21. The system of claim 20, wherein augmentation of the virtual game is implemented by the control circuitry configured by modifying a path that may be taken by the current user within the virtual game. 27. The system of claim 26, wherein the route modification is associated with disallowing navigation by the current user on a route within the virtual game that was previously permitted for navigation. The portion of the virtual game includes a starting checkpoint and an ending checkpoint, the starting checkpoint and the ending checkpoint being determined by the control circuitry based on a type of the virtual game. The system according to paragraph 20. the control circuitry configured to select the starting checkpoint and the ending checkpoint in a shorter time span for faster paced virtual games and in a relatively longer time span for slower paced virtual games; 29. The system of claim 28, further comprising. monitoring the current user's path within the virtual game;
determining a total number of interaction opportunities available based on the path taken by the current user within the virtual game;
21. The control circuitry of claim 20, further comprising: determining a number of interaction opportunities utilized by the current user from the total number of available interaction opportunities. system. 31. The system of claim 30, wherein the current user's performance is based on the number of interaction opportunities performed by the current user out of the total number of interaction opportunities available to the current user. monitoring the current user's gaze as the user navigates a path within the virtual game;
determining objects within a field of view (FOV) of the current user based on the path navigated by the current user within the virtual game;
21. The system of claim 20, further comprising: the control circuitry configured to: determine whether the user has interacted with an object within the FOV. In response to determining that the user has not interacted with the object within the FOV, the control circuitry is configured to prompt the user to interact with the object within the FOV. 33. The system of claim 32, configured. 34. The system of claim 33, wherein the prompt is either a visual alert or an audible prompt displayed on a display of the electronic device. 33. In response to determining that the user has not interacted with the object within the FOV, the control circuitry is configured to highlight the object within the FOV. System described. 21. The system of claim 20, wherein the electronic device is selected from the group consisting of virtual reality, augmented reality, or mixed reality devices.