CN117677345A - Enhanced meditation experience based on biofeedback - Google Patents

Abstract

Various implementations disclosed herein include devices, systems, and methods that provide customized feedback content during meditation experiences. For example, an exemplary process may include: obtaining physiological data via one or more sensors; determining an attention state based on the physiological data; customizing feedback content based on user attributes to change the state of attention during the meditation mode; and providing customized feedback content during the meditation mode based on the user attributes after a delay time.

Description

Enhanced meditation experience based on biofeedback

Technical Field

The present disclosure relates generally to presenting content via an electronic device, and in particular, to systems, methods, and devices that determine respiratory status and attention status during and/or based on the presentation of electronic content.

Background

The electronic device may be used to assist the user in engaging in various experiences in which a particular user state is desired. For example, the electronic device may be used to present content that guides the user through a meditation experience, where the user desires to relax and/or concentrate on things such as his or her breath. Such content is generally not responsive or adaptable to the actual state of the user and, therefore, may not be as effective or efficient as desired.

Disclosure of Invention

Various implementations disclosed herein include devices, systems, and methods that present content and evaluate respiratory status (e.g., breath 7 times per minute, etc.) and attention status (e.g., whether concentrating or working) based on physiological data (e.g., heart rate, respiratory rate, body temperature, electrocardiogram, blood oxygen saturation, skin conductance, etc.) and modify (e.g., enhance) the content based on respiratory status and attention status. For example, visual notifications and/or audio notifications may be provided to the user to focus on. Additionally, breath and attention statistics analysis and summary information may be provided.

The breathing state and the attention state when viewing and/or listening to content on an electronic device may have a significant impact on the experience. For example, it may be required to keep concentration and participation to obtain meaningful experiences, such as meditation, watching educational or entertainment content, learning new skills, and so forth. Improved techniques for evaluating respiratory status and attention status of users viewing and interacting with content may enhance user enjoyment, understanding, and learning of content. Furthermore, the content may not be presented in a manner that is meaningful to the particular user. The content creator and system may be able to provide a better and more targeted user experience (e.g., a more meaningful meditation experience) that the user is more likely to enjoy, understand, and learn from based on the respiratory state and attention state information.

In one exemplary implementation, the apparatus, system, and method facilitate meditation by tracking a user's respiratory state and attention state based on the use of sensors that obtain physiological data. The meditation may (but need not) guide the user to his or her breath. The system may identify a baseline user state and a target user state and enhance the experience to achieve the target. In addition to respiration and attention, sensor data may be used to determine electroencephalogram (EEG), temperature (e.g., over the nose), heart rate (e.g., over the forehead), and so forth. In some implementations, content enhancements may be selected based on characteristics of the user's environment (e.g., real world physical environment, virtual environment, or a combination of each). The device (e.g., a handheld device, laptop, desktop, or head-mounted device (HMD)) provides content (e.g., a visual experience and/or an auditory experience) corresponding to a real-world physical environment, a virtual reality environment, or a combination of such environments to a user. The content (e.g., an augmented reality (XR) environment) may be enhanced visual content/audio content and guidance that provides a closed-loop meditation experience based on real-time biofeedback.

In some implementations, the device obtains physiological data (e.g., respiratory data, image data (face, body, etc.), EEG amplitude, pupil modulation, eye gaze glance, etc.) associated with the user via one or more sensors. Based on the obtained physiological data, some techniques described herein determine a respiration state and a concentration state (e.g., concentration, distraction, etc.) of a user during an experience (e.g., meditation experience). For example, some implementations may identify that the user's eye characteristics (e.g., blink frequency, steady gaze direction, glance amplitude/velocity, and/or pupil radius) correspond to the attention state "focus" rather than the attention state "distraction". Based on the physiological data and associated physiological responses, these techniques may provide feedback to the user that the current respiratory state and/or the attention state is different from the expected state of the experience, recommend similar content or similar portions of the experience, and/or adjust, enhance, or otherwise modify the content.

In some implementations, the content modification may be the beginning or ending of the meditation experience, or a change during the ongoing experience. Some implementations may provide for visualization of a current respiratory state or a desired/improved respiratory state. For example, the user may visualize breathing oscillations by displaying a reduced and enlarged flower (e.g., for each breath) or by blowing out a virtual candle through the user's breath. Some implementations may use subtle cues to change from a brighter to darker environment and/or ring, audio, or chime. Some implementations may be explicitly upgraded from subtle cues (e.g., small icons mixed with background) to direct instructions (e.g., "control your breath"). Some implementations may use spatialization audio to redirect attention (e.g., bird song at a particular location of content). Some implementations may provide simultaneous/combined feedback regarding current and desired respiratory states and/or attention states.

Some implementations improve respiratory state and attention state assessment accuracy, e.g., to improve assessment of a user's attention to a task (e.g., informing the user that they are straying and/or not breathing during a meditation experience). Some implementations improve the user experience by providing an attention/respiration assessment that minimizes or avoids disrupting or interfering with the user experience, e.g., without significantly interrupting the user's attention or ability to perform tasks.

In some implementations, the respiratory rate (e.g., respiratory tracking) may involve sensor fusion of two or more different sensor data. For example, head pose from an Inertial Measurement Unit (IMU), audio from microphones, images from one or more cameras (e.g., a jaw camera, a lower body camera, an eye camera for tissue surrounding the eye, etc.), body motion, and/or signals of the face modulated by respiration (e.g., remote photoplethysmogram (PPG)). Using this type of sensor fusion to track the user's breath (such as when wearing an HMD) may eliminate the need for the user to wear sensors that are worn around, for example, the user's diaphragm.

Physiological response data (such as EEG amplitude/frequency, pupil modulation, eye gaze glance, etc.) may depend on the individual's state of attention and the nature of the scene in front of him or her and the content enhancement presented therein. Physiological response data may be obtained when using a device with eye tracking technology when a user performs tasks requiring different levels of attention, such as attention to concentration of meditation or educational videos (e.g., cooking educational videos). In some implementations, other sensors (such as EEG sensors) may be used to obtain physiological response data. Observing repeated measurements of physiological response data to an experience may give insight regarding potential respiratory and attention states of the user over different time scales. These breath and attention metrics may be used to provide feedback during the learning experience.

Some implementations use scene analysis (e.g., creating an attention map based on object detection, facial recognition, etc.) that identifies relevant regions of content to determine what a person is looking at during presentation of the content. Such scene analysis and/or determination of observed objects may be used to improve the determination of the breathing state and the attention state of the user.

In some implementations, meditation (e.g., at a particular time, place, task, etc.) may be recommended based on the user's respiratory status and attentiveness status (e.g., concentration, distraction, etc.) by identifying the type or characteristics of recommended meditation based on various factors (e.g., physical environment context, scene understanding of what the user is looking at in an XR environment, etc.). For example, one type of meditation (e.g., positive meditation for distraction) may be recommended in one case, and a different type of meditation (e.g., mobile/physical meditation for distraction and anxiety situations) may be recommended in another case. If the user wants to engage in an attentive session of concentration (e.g. concentrating on a single task, such as watching a video) and if user distraction is detected, open monitoring meditation may be recommended. For example, open monitoring meditation may allow and/or encourage a user to pay attention to multiple sound/visual perceptions/ideas in the meditation environment, and may restore his or her ability to focus on a single item. Additionally or alternatively, if the user wants to multitask using various applications, and the system detects that the user should be overwhelmed, the system may recommend that he or she perform a focused attention meditation technique (e.g., by focusing on a breath monitor icon (such as a flower moving with each breath) or blowing out a candle to pay attention to the breath). Focused attention meditation techniques may allow a user to regain the ability to focus on a single event once. In an exemplary implementation, a meditation session may be initiated for the user, which may be contrary to the primary task that he or she is completing, so that he or she may relax/recover during meditation and return to the task at hand more effectively.

Experience other than meditation experience may utilize the techniques described herein with respect to evaluating respiratory status and attention status. For example, the educational experience may inform the student to focus on the educational tasks when he or she appears to be distracted. Another example may be a workplace experience informing of a worker who needs to focus on his or her current task. For example, a surgeon who may feel lightly tired during long surgery is provided feedback, reminded that a truck driver driving for a long period of time is losing his or her attention and may need to stop alongside to sleep, etc. The techniques described herein may be tailored to any user and experience that may require some type of content enhancement to enter or maintain one or more particular respiratory and attention states.

The disclosed techniques may collect and use data from various sources to provide feedback content during meditation. In some cases, the data may include personal information data that uniquely identifies or may be used to locate or contact a particular individual. The personal information data may include location-based data, demographic data, telephone numbers, email addresses, social media account names, home or work addresses, data or records associated with the user's health or fitness level (e.g., information associated with vital signs, medications, workouts, etc.), date of birth, or other personal or identifying information.

It has been recognized that in some cases, such personal information data may be used to benefit the user. For example, personal information data may be used to provide feedback content during meditation. Thus, the use of such personal information data enables planned control of the delivered content.

It is contemplated that collection, disclosure, delivery, analysis, storage, or other use of personal information data should comply with established privacy policies or practices. Privacy policies and practices generally considered to meet or exceed the industry or government requirements for maintaining personal information data private and secure should be implemented and consistently used. These policies should be easily accessible and updated as the collection or use of personal information data changes. Personal information data should be collected for legal and legitimate uses and not shared or sold outside of these legal uses. The collection or sharing should occur after receiving notification consent of the user. Additional steps for safeguarding and securing access to personal information data and ensuring that other persons having access to personal information data adhere to their privacy policies and procedures should be considered. An evaluation by a third party may be performed to prove compliance with established privacy policies and practices. Policies and practices should be tailored to the particular type of personal information data collected or accessed and applicable to applicable laws and standards, including jurisdiction-specific considerations. For example, the collection or access of certain health data in the united states may be governed by federal or state law, such as the health insurance policies and liabilities act (HIPAA); while collection or access to the same health data may be subject to other regulations and policies in other countries. Thus, different privacy practices should be implemented for different types of personal information data in each country.

It is contemplated that in some cases, a user may selectively block use or access to personal information data. Hardware or software features may be provided to prevent or block access to personal information data. For example, where feedback content is provided during meditation, the disclosed techniques may be configured to allow a user to "opt-in" or "opt-out" to participate in the collection of personal information data during registration or at any time thereafter. In another example, the user may choose not to provide physiological data for the targeted content delivery service during meditation. In yet another example, the user may choose to limit the length of time that feedback content is provided during the meditation session is maintained or to disable monitoring the user altogether. The present technology may also provide notifications related to access or use of personal information data. For example, a first notification may be provided in response to a user downloading an application that has access to the user's personal information data, and a second notification may be provided to alert the user just prior to the application accessing the personal information data.

Personal information data should be managed and processed to reduce the risk of inadvertent or unauthorized access or use. The risk can be reduced by limiting the collection of data and deleting data once it is no longer needed. When applicable, data de-identification may be used to protect the privacy of the user. For example, de-identification may be performed by removing a particular identifier, controlling the specificity or amount of data stored (e.g., collecting home location data at a city level rather than an address level), controlling the manner in which data is stored (e.g., aggregating data across multiple users), or providing for the use of other techniques.

Although the present technology may broadly involve the use of personal information data, the present technology may be implemented without accessing such personal information data. In other words, the present technology is not rendered inoperable by the lack of some or all of such personal information data. For example, meditation feedback content may be selected and delivered to the user by inferring preferences based on non-personal information data, reduced amounts of personal information data, or publicly available information.

In general, one innovative aspect of the subject matter described in this specification can be embodied in various methods including the acts of: at an electronic device having a processor: obtaining physiological data associated with a user via one or more sensors; determining an attentiveness state of distraction during meditation, the attentiveness state being determined based on the physiological data; customizing feedback content to direct the user to change the state of attention during the meditation, wherein characteristics are customized based on user attributes; and providing customized feedback content during the meditation based on the user attributes after a delay time.

These and other embodiments can each optionally include one or more of the following features.

In some aspects, the method further comprises: responsive to providing the customized feedback content, determining an attention level over a period of time based on the physiological data, the attention level corresponding to the user; and determining a feedback metric for the user based on the determined level of attention over the period of time.

In some aspects, the user attribute includes meditation experience level. In some aspects, the meditation experience level is determined based on accessing a user profile. In some aspects, the meditation experience level is determined based on an analysis of historical data associated with the user for previous meditation experiences. In some aspects, the meditation experience level is updated based on the state of attention during the meditation.

In some aspects, the method further comprises: presenting instructions to the user to notice breathing; and evaluating an attention level to respiration based on the respiration state and the attention state, wherein the customized feedback is determined based on the attention level.

In some aspects, customizing the characteristic of the feedback content includes: determining a baseline corresponding to the user based on the physiological data; determining a target for the user based on the baseline; and determining customized feedback content based on the baseline and the target.

In some aspects, the method further comprises: identifying a meditation state, the meditation state corresponding to a plurality of time periods; and presenting an indication of progress based on the meditation status. In some aspects, the method further comprises: a portion of the meditation is identified, the portion being associated with a particular attention state. In some aspects, the method further comprises: determining a context of the meditation based on sensor data of an environment of the meditation; and customizing the characteristic of the feedback content based on the context of the meditation.

In some aspects, determining the context of the experience includes generating a scene understanding of the environment based on the sensor data of the environment, the scene understanding including a visual or audible attribute of the environment, and determining the context of the experience based on the scene understanding of the environment. In some aspects, determining the context of the experience includes determining an activity of the user based on a user's schedule.

In some aspects, the customized feedback content includes: conclusions of current meditation; an activation of another meditation different from the current meditation; or a change of the current meditation. In some aspects, the customized feedback content includes: the volume of the audio signal modulated based on the physiological data. In some aspects, the customized feedback content includes: a visual or audible representation of the state of attention or a change of the state of attention. In some aspects, the customized feedback content includes: a cue configured to trigger a change in the attention state. In some aspects, the customized feedback content includes: a graphical indication logo or sound configured to change a first attention state to a second attention state. In some aspects, the customized feedback content includes: visual or audible indication of suggested time for the new meditation experience.

In some aspects, the state of attention is determined based on measuring gaze or body stability using the physiological data. In some aspects, the attention state is determined based on determining the attention level. In some aspects, the attention state is determined based on a respiratory state. In some aspects, the attention state is evaluated using statistical or machine learning based classification techniques.

In some aspects, the physiological data includes at least one of skin temperature, respiration, photoplethysmography (PPG), electrodermal activity (EDA), eye gaze tracking, and pupil movement associated with the user.

In some aspects, the meditation environment comprises an extended reality (XR) environment. In some aspects, the meditation is presented to a plurality of users during a communication session. In some aspects, the device is a Head Mounted Device (HMD).

According to some implementations, a non-transitory computer readable storage medium has stored therein instructions that are computer executable to perform or cause to be performed any of the methods described herein. According to some implementations, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing performance of any of the methods described herein.

Drawings

Accordingly, the present disclosure may be understood by those of ordinary skill in the art, and the more detailed description may reference aspects of some illustrative implementations, some of which are shown in the accompanying drawings.

FIG. 1 illustrates a device that presents a visual experience and/or an auditory experience and obtains physiological data from a user, according to some implementations.

Fig. 2 illustrates the pupil of the user of fig. 1, wherein the diameter of the pupil varies over time, according to some implementations.

Fig. 3A and 3B illustrate detecting a respiratory state and an attention state of a user viewing content based on physiological data according to some implementations.

FIG. 4 illustrates a user viewing content based on tracking a user's breathing state and attention state, according to some implementations.

Fig. 5A-5C illustrate exemplary views of an electronic device viewing an XR environment, according to some implementations.

Fig. 6 illustrates a system diagram for evaluating respiratory status and attention status of a user viewing content based on physiological data, according to some implementations.

FIG. 7 is a flow chart representation of a method for evaluating a respiratory state and an attention state of a user viewing content based on physiological data and providing content modification based on the respiratory state and the attention state of the user, according to some implementations.

Figure 8 is a flowchart representation of a method for providing customized feedback content during meditation based on physiological data according to some implementations.

Fig. 9 illustrates device components of an exemplary device according to some implementations.

Fig. 10 illustrates an example Head Mounted Device (HMD) according to some implementations.

The various features shown in the drawings may not be drawn to scale according to common practice. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some figures may not depict all of the components of a given system, method, or apparatus. Finally, like reference numerals may be used to refer to like features throughout the specification and drawings.

Detailed Description

Numerous details are described to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings illustrate only some example aspects of the disclosure and therefore should not be considered limiting. It will be apparent to one of ordinary skill in the art that other effective aspects or variations do not include all of the specific details set forth herein. Moreover, well-known systems, methods, components, devices, and circuits have not been described in detail so as not to obscure the more pertinent aspects of the exemplary implementations described herein.

Fig. 1 shows a real world physical environment 5 comprising a device 10 with a display 15. In some implementations, the device 10 displays the content 20 to the user 25, as well as visual characteristics 30 associated with the content 20. For example, the content 20 may be buttons, user interface icons, text boxes, graphics, and the like. In some implementations, visual characteristics 30 associated with content 20 include visual characteristics such as hue, saturation, size, shape, spatial frequency, motion, highlighting, and the like. For example, the content 20 may be displayed with a visual characteristic 30 that covers or surrounds the color highlighting of the content 20.

In some implementations, the content 20 may be a visual experience (e.g., meditation experience), and the visual characteristics 30 of the visual experience may change continuously during the visual experience. As used herein, the phrase "experience" refers to a period of time that a user uses an electronic device that uses a physiological data stream to measure one or more respiratory states and/or attention states. In one example, a user has an experience in which the user perceives a real-world environment while holding, wearing, or approaching an electronic device that includes one or more sensors that obtain physiological data indicative of the user's respiratory state and attention state. In another example, a user has an experience in which the user perceives content displayed by an electronic device while the same or another electronic device obtains physiological data (e.g., pupil data, EEG data, etc.) to evaluate the user's respiratory state and attention state. In another example, a user has an experience in which the user holds, wears, or is in proximity to an electronic device that provides a series of audible or visual instructions that guide the experience. For example, the instructions may instruct the user to maintain or attempt to maintain a particular respiratory state (e.g., breath Per Minute (BPM) 7 times) and an attention state (e.g., focus on a particular visual element and/or audio element) during a particular period of time of the experience. For example, the user is instructed to concentrate on his or her breath and to pay attention to a specific part of the meditation video, etc. During such experiences, the same or another electronic device may obtain physiological data from one or more sensors to evaluate the respiratory state and the attentive state of the user.

In some implementations, the visual characteristics 30 are visual cues or audio cues of a user that are specific to the enhancement or modification of content of the experience (e.g., about focusing on a particular task during the experience, such as during a particular portion of the meditation or education/learning experience). In some implementations, the visual experience (e.g., content 20) may occupy the entire display area of the display 15. For example, during the meditation experience, the content 20 may be a video or sequence of images that may include visual cues and/or audio cues as visual characteristics 30 presented to the user regarding attention to a particular meditation technique. Other visual experiences that may be displayed for content 20 and visual and/or audio cues regarding visual characteristics 30 will be discussed further herein.

The device 10 obtains physiological data (e.g., EEG amplitude/frequency, pupil modulation, eye gaze glance, etc.) from the user 25 via the sensor 35. For example, the device 10 obtains pupil data 40 (e.g., eye gaze characteristic data). While this example and other examples discussed herein show a single device 10 in the real world environment 5, the techniques disclosed herein are applicable to multiple devices and multiple sensors, as well as other real world environments/experiences. For example, the functions of device 10 may be performed by a plurality of devices.

In some implementations, as shown in fig. 1, the device 10 is a handheld electronic device (e.g., a smart phone or tablet computer). In some implementations, the device 10 is a laptop computer or a desktop computer. In some implementations, the device 10 has a touch pad, and in some implementations, the device 10 has a touch sensitive display (also referred to as a "touch screen" or "touch screen display"). In some implementations, the device 10 is a wearable head mounted display ("HMD").

In some implementations, the device 10 includes an eye tracking system for detecting eye position and eye movement. For example, the eye tracking system may include one or more Infrared (IR) Light Emitting Diodes (LEDs), an eye tracking camera (e.g., a Near IR (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) to the eyes of the user 25. Further, the illumination source of the device 10 may emit NIR light to illuminate the eyes of the user 25, and the NIR camera may capture images of the eyes of the user 25. In some implementations, images captured by the eye tracking system may be analyzed to detect the position and movement of the eyes of user 25, or to detect other information about the eyes, such as pupil dilation or pupil diameter. Further, gaze points estimated from eye-tracked images may enable gaze-based interactions with content shown on a near-eye display of the device 10. Additional cameras may be included to capture other areas of the user (e.g., HMDs with jaw cameras for viewing the user's mouth, lower cameras for viewing the body, eye cameras for tissue surrounding the eyes, etc.). These cameras and other sensors may detect body motion and/or signals of the face modulated by the user's breath (e.g., remote PPG).

In some implementations, the device 10 has a Graphical User Interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing a plurality of functions. In some implementations, the user 25 interacts with the GUI through finger contacts and gestures on the touch-sensitive surface. In some implementations, these functions include image editing, drawing, rendering, word processing, web page creation, disk editing, spreadsheet making, game playing, phone calls, video conferencing, email sending and receiving, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playing, and/or digital video playing. Executable instructions for performing these functions may be included in a computer-readable storage medium or other computer program product configured for execution by one or more processors.

In some implementations, the apparatus 10 employs various sensors, detection or measurement systems. The detected physiological data may include, but is not limited to: EEG, electrocardiogram (ECG), electromyogram (EMG), functional near infrared spectrum signal (fNIRS), blood pressure, skin conductance or pupillary response. The device 10 is communicatively coupled to additional sensors. For example, the sensor 17 (e.g., EDA sensor) may be communicatively coupled to the device 10 via a wired or wireless connection, and the sensor 17 may be located on the skin of the user 25 (e.g., on an arm as shown, or placed on a user's hand/finger). For example, the sensor 17 may be used to detect EDA (e.g., skin conductance), heart rate, or other physiological data that utilizes contact with the user's skin. In addition, the device 10 (using one or more sensors) may detect multiple forms of physiological data simultaneously in order to benefit from the simultaneous acquisition of physiological data. Furthermore, in some implementations, the physiological data represents involuntary data, i.e., responses that are not consciously controlled. For example, the pupillary response may be indicative of involuntary movement.

In some implementations, one or both eyes 45 of user 25 (including one or both pupils 50 of user 25) present physiological data (e.g., pupil data 40) in the form of a pupillary response. The pupillary response of user 25 causes a change in the size or diameter of pupil 50 via the optic nerve and the opthalmic cranial nerve. For example, the pupillary response may include a constrictive response (pupil constriction), i.e., pupil narrowing, or a dilated response (pupil dilation), i.e., pupil widening. In some implementations, the device 10 can detect a pattern of physiological data representing a time-varying pupil diameter.

In some implementations, the pupillary response may be responsive to audible feedback (e.g., an audio notification to the user) detected by one or both ears 60 of user 25. For example, device 10 may include a speaker 12 that projects sound via sound waves 14. The device 10 may include other audio sources such as a headphone jack for headphones, a wireless connection to an external speaker, and so forth.

Fig. 2 shows the pupil 50 of the user 25 of fig. 1, wherein the diameter of the pupil 50 varies over time. Pupil diameter tracking may potentially indicate the physiological state of the user. As shown in fig. 2, the current physiological state (e.g., current pupil diameter 55) may change as compared to the past physiological state (e.g., past pupil diameter 57). For example, the current physiological state may include a current pupil diameter and the past physiological state may include a past pupil diameter.

The physiological data may change over time, and the device 10 may use the physiological data to measure one or both of a physiological response of the user to the visual characteristics 30 or an intent of the user to interact with the content 20. For example, when content 20 such as a list of content experiences (e.g., meditation environments) is presented by the device 10, the user 25 may select the experience without the user 25 having to complete a physical button press. In some implementations, the physiological data may include a physiological response of visual or auditory stimuli of the radius of the pupil 50 after the user 25 glances at the content 20, measured via eye tracking techniques (e.g., via an HMD). In some implementations, the physiological data includes EEG amplitude/frequency data measured via EEG techniques or EMG data measured from an EMG sensor or motion sensor.

A person may sense or interact with a physical environment or world without using an electronic device. Physical features such as physical objects or surfaces may be included within a physical environment. For example, the physical environment may correspond to a physical city with physical buildings, roads, and vehicles. People can directly perceive or interact with the physical environment by various means such as smell, vision, taste, hearing and touch. This may be in contrast to an augmented reality (XR) environment, which may refer to a partially or fully simulated environment in which people may sense or interact using an electronic device. The XR environment may include Virtual Reality (VR) content, mixed Reality (MR) content, augmented Reality (AR) content, and the like. Using an XR system, a portion of a person's physical motion or representation thereof may be tracked, and in response, properties of virtual objects in an XR environment may be changed in a manner consistent with at least one natural law. For example, an XR system may detect head movements of a user and adjust the auditory and graphical content presented to the user in a manner that simulates how sounds and views will change in a physical environment. In other examples, the XR system may detect movement of an electronic device (e.g., laptop, tablet, mobile phone, etc.) that presents the XR environment. Thus, the XR system may adjust the auditory and graphical content presented to the user in a manner that simulates how sound and views will change in the physical environment. In some instances, other inputs such as a representation of body movement (e.g., voice commands) may cause the XR system to adjust properties of the graphical content.

Numerous types of electronic systems may allow a user to sense or interact with an XR environment. The incomplete list includes lenses with integrated display capabilities (e.g., contact lenses), heads-up displays (HUDs), projection-based systems, head-mountable systems, windows or windshields with integrated display technology, headphones/earphones, input systems with or without haptic feedback (e.g., hand-held or wearable controllers), smartphones, tablet computers, desktop/laptop computers, and speaker arrays placed on the eyes of the user. The head-mounted system may include an opaque display and one or more speakers. Other head-mounted systems may be configured to receive an opaque external display, such as an opaque external display of a smart phone. The head-mounted system may use one or more image sensors to capture images/video of the physical environment or one or more microphones to capture audio of the physical environment. Some head-mounted systems may include a transparent or translucent display instead of an opaque display. The transparent or translucent display may direct light representing the image to the user's eye through a medium such as holographic medium, optical waveguide, optical combiner, optical reflector, other similar techniques, or combinations thereof. Various display technologies may be used, such as liquid crystal on silicon, LED, uLED, OLED, laser scanning light sources, digital light projection, or combinations thereof. In some examples, the transparent or translucent display may be selectively controlled to become opaque. Projection-based systems may utilize retinal projection techniques that project images onto the retina of a user, or may project virtual content into a physical environment, such as onto a physical surface or as a hologram.

Fig. 3A and 3B illustrate a respiratory state and an attentiveness state of a user evaluating viewing content based on obtained physiological data, in accordance with some implementations. In particular, fig. 3A and 3B illustrate presenting content 302 to a user (e.g., user 25 of fig. 1) in an environment 304 during content presentation (e.g., meditation experience) at a content presentation time 300A and later at a time of content presentation time 300B, respectively, wherein the user has a physiological response to the content (e.g., the user looks at a portion of the content as detected by the eye gaze characteristic data) via the obtained physiological data. For example, the user is presented with content 302 that includes visual content (e.g., meditation video), and physiological data of the user, such as eye gaze characteristic data 312, pupil data 314, respiration data 315, EDA data 316, and heart rate data 318, is continuously (or periodically) monitored. Physiological data may be initially obtained to determine baseline data for the user, and then during the experience (e.g., meditation session), the physiological data may be monitored and compared to the determined baseline to evaluate the respiratory state and the attention state of the user.

In the particular example of fig. 3A and 3B, at content presentation time 300A, the user's eye gaze characteristics are focused on content 302 such that attention scale 330 shows slider indication indicator 332 as being higher toward the "focused" portion and shows a lower respiratory rate (e.g., slider indication indicator 342 is shown at a lower frequency via respiratory scale 340). Then, at the content presentation time 300B of fig. 3B (e.g., during the distraction phase), the user's eye gaze characteristics 312 appear not to focus on the content 302, such that the attention scale 330 shows the slider indication marks 332 lower toward the "distraction" portion, and shows a higher respiratory rate on the respiratory scale 340 as compared to the content presentation time 300A.

In some implementations, the respiration state (e.g., via the respiration scale 340) is based on respiration data 315 acquired from a respiration sensor (e.g., a sensor worn on the user). Additionally or alternatively, the respiration data 315 may involve sensor fusion of different data acquired from the device 10 without the use of additional respiration sensors. For example, the different data acquired that may be fused may include head pose data from the IMU, audio from a microphone, camera images of the user's face and/or body (e.g., an HMD with a jaw camera, a lower camera for viewing the body, an eye camera for eye surrounding tissue, etc.), body motion, and/or a signal of the face modulated by breathing (e.g., remote PPG). Using this type of sensor fusion to track the user's breathing (such as when wearing an HMD) may eliminate the need for the user to wear sensors around, for example, the user's diaphragm, e.g., to track his or her breathing frequency.

In some implementations, the respiratory status and the attention status of the user viewing the content are evaluated based on the physiological data and the contextual data. For example, content 302 may be analyzed by a context analysis instruction set to determine context data for a user's experience (e.g., experience presented in a current physical environment when viewing video content on an electronic device such as an HMD). Determining context data of an experience may involve using computer vision to generate scene understanding of visual and/or auditory properties of a physical environment (e.g., environment 304), such as where a user is, what the user is doing, what objects are nearby. Additionally or alternatively, determining context data of an experience may involve determining scene understanding of visual and/or auditory properties of a content presentation (e.g., content 302, such as video). For example, the content 302 and environment 304 may include one or more persons, objects, or other background objects within a user's field of view that may be detected by an object detection algorithm, a face detection algorithm, or the like.

In some implementations, the user's experience level (e.g., meditation experience level) may be determined based on historical data, such as physiological data and contextual data from previous experiences. Additionally or alternatively, the user's experience level (e.g., meditation experience level) may be determined based on accessing a user profile (e.g., storing metrics in a database). For example, experience level scale 360 indicates that the user has a higher experience level at content presentation time 300B than at content presentation time 300A of the previous experience. For example, after a successful meditation experience (e.g., content presentation time 300A), the user's "experience" in a successful meditation has increased. The scale may be based on a measure of the amount of time spent in the attention state "focus" during the meditation experience or another type of experience, or may include additional factors and/or measures.

Fig. 4 illustrates evaluating the respiratory status and attention status of a user viewing content and modifying the content based on obtained physiological data according to some implementations. Content 302 including visual content (e.g., meditation video) is presented to a user at a content presentation time 400 (e.g., after content presentation time 300B), and physiological data of the user, such as eye gaze characteristic data 412, pupil data 314, respiratory data 315, EDA data 316, and heart rate data 318, are continuously or periodically monitored. In particular, fig. 4 illustrates presenting content 302 to a user (e.g., user 25 of fig. 1) in an environment 304 during content presentation (e.g., meditation experience) at a content presentation time 400, wherein the user has a physiological response to the content via the obtained physiological data. For example, the user looks at a portion of the content, particularly the content enhancement 403, as detected by the eye gaze characteristic data 412, wherein the content is modified based on the response. For example, after the content presentation time 300B, where the user is determined to be "tranquillizing", the content enhancement 403 is applied to the content 302. Thus, after analyzing the physiological data of the user (e.g., by the physiological data instruction set) and analyzing a period of time after the contextual data of the content 302 and/or the environment 304 (e.g., by the contextual instruction set), the content presentation time 400 and the content enhancement 403 are presented to the user because the respiratory state and/or the attention state assessment is that the user may have exhibited a higher respiratory state and/or an undesirable attention state (e.g., distraction). Fig. 4 also shows that the user is more "focused" based on the attention scale 330, where the attention slider indication identifier 432 is higher on the scale than the content presentation time 300B when the user is distracting. Additionally, the user's respiratory rate is shown on respiratory rate scale 340, with respiratory slide bar indication indicator 442 indicating a respiratory rate of about 7 BPM. Further, the user's experience level (e.g., meditation level) is shown on the experience level scale 460.

Fig. 4 also shows an exemplary view 450 of a physical environment (e.g., the real world environment 5 of fig. 1) provided by the electronic device 405 (e.g., the device 10). View 450 may be a real-time camera view of the physical environment, a view of the physical environment through a see-through display, or a view generated based on a 3D model corresponding to the physical environment. View 450 includes an application window (e.g., a representation of content 302) presented on a device of content 402. Presentation of content 402 includes content enhancement 403 (e.g., audio notification and/or visual notification). Additionally, view 450 includes visual respiration indication identifier 410 that provides an indication to the user of the user's respiration rate (e.g., as detected by obtained physiological data—respiration data 315). In particular, breath indicator 410a represents a user inhaling (e.g., making a deep breath in such a larger icon), while breath indicator 410b represents a user exhaling (e.g., making a deep breath in such a smaller icon). The visual meditation indication identifier 410 may be used for meditation experience to guide the user to use the appropriate breathing technique. Alternatively, other visual indication identifications may be used to visualize the breathing frequency (actual and desired) for the user. For example, a sine wave of the user's breathing frequency may be shown, as well as a desired sine wave that the user may try and simulate (e.g., slow down the user's breathing frequency for a particular meditation experience).

For example, the user exhibits a high respiratory state at work, the content augmentation 403 may indicate the high respiratory state (e.g., audio notification and/or visual notification) to the user, and may provide some alternative actions for calm to the user (e.g., meditation music, relaxed XR environment, etc.). As shown, the breath scale 340 and the attention state scale 330 provide possible uses for comparing detected levels of attention and respiratory frequency of the user with performance levels associated with those states. For example, for the above example exhibiting an abnormal breathing state and an attentive state (e.g., a level of concentration or a level of distraction), then content enhancement 403 can alert the user to the abnormal state to try and indicate that he or she reaches a level within the threshold of the current experience (e.g., controlling the user to a controlled breathing rate, such as 7 BPM). Additionally, the user's respiratory status and attention status ratings may be continuously monitored throughout the presentation of content 402.

Content enhancement 403 may include visual presentation. For example, an icon may appear, or a text box may appear that indicates that the user is focusing on. In some implementations, the content enhancement 403 may include auditory stimuli. For example, the spatialization audio may be presented to redirect the user's attention to a particular region of the content presentation (e.g., if it is determined that the user is exhibiting abnormal respiration levels, the user's attention may be diverted to some of the relaxed content).

In some implementations, the content enhancement 403 may include an entire display of visual content (e.g., a relaxed video over the entire display of the device). The content (and/or content enhancement 403) may include or provide a view of the 3D environment. Alternatively, the content augmentation 403 may include visual content of a frame surrounding the display of the device (e.g., on a mobile device, a virtual frame of the display is created to acquire the user's attention from a particular respiratory state and attention state). In some implementations, the content enhancement 403 can include a combination of visual content (e.g., a notification window, icon, or other visual content described herein) and/or auditory stimuli. For example, a notification window or arrow may guide the user to a particular content area and may present an audio signal that guides the user. These visual cues and/or auditory cues may help guide the user to specific content enhancements that may help the user to cope with different respiratory and attention states to increase his or her performance level (for a work experience), or simply to view content 302 comfortably (e.g., provide meditation if it is determined that the user is in a stressful environment or situation).

In some implementations, content augmentation may be referred to herein as "customized feedback content" for meditation experience such as meditation sessions. In some implementations, the customized feedback content includes: conclusions of current meditation; an activation of another meditation different from the current meditation; and/or a change of the current meditation. In some implementations, the customized feedback content includes: the volume of the audio signal modulated based on the physiological data. In some implementations, the customized feedback content includes: a visual or audible representation of the state of attention or a change of the state of attention. In some implementations, the customized feedback content includes: a cue configured to trigger a change in the attention state. In some implementations, the customized feedback content includes: a graphical indication logo or sound configured to change a first attention state to a second attention state. In some implementations, the customized feedback content includes: visual or audible indication of suggested time for the new meditation experience.

In some implementations, the content 402 of the device 10 is used only for an auditory experience. For example, for a driving experience (e.g., truck driver), the physiological data and/or contextual data of the environment may be analyzed by one or more sensors, and the customized feedback content may provide one or more audible signals for the content augmentation 403 to indicate when the user is out of focus while driving and needs to be alerted to a potentially dangerous situation, such as falling asleep while driving. Thus, content augmentation 403 may include a voice (e.g., "wake |") or a loud alert sound notification that alerts the driver. In some implementations, content 402 may be displayed to the user on the windshield of the vehicle or with the user wearing an HMD (such as through-the-eye glasses), and virtual notifications may be presented to the user as content enhancements 403. For example, an AR alert may be displayed to the truck driver to alert them to focus on driving.

Fig. 5A-5C illustrate exemplary views of an electronic device viewing an XR environment, according to some implementations. In particular, fig. 5A-5C illustrate different levels of immersion or displaying different levels of virtual content for a user viewing an XR environment. For example, fig. 5A-5C illustrate an exemplary electronic device 10 that provides a view 515A of a 3D environment 512A, a view 515B of a 3D environment 512B, and a view 515C of a 3D environment 512C, respectively, that operates in a physical environment 500 during viewing content (e.g., a mixed reality meditation experience when a city park walks). For example, fig. 5A-5C may represent viewing content at three different time periods while user 25 views content on a display of device 10 and may view at least a portion of physical environment 500. In these examples of fig. 5A-5C, physical environment 500 is a city park having cities (buildings) in a background including bystanders 520 and 522 and birds 524. In particular, fig. 5A-5C each show a user 25 (e.g., a viewer during a meditation experience) viewing content on the device 10 during sunrise in a city park with a city landscape in the background (e.g., the morning).

The electronic device 10 includes one or more cameras, microphones, depth sensors, or other sensors that may be used to capture information about the physical environment 500 and objects therein and information about the user 25 of the electronic device 10 (e.g., location data of the user 25) as well as to evaluate the physical environment and objects therein and the information. Information about the physical environment 500 and/or the user 25 may be used to provide visual and audio content during the meditation session. For example, the meditation session may provide views (e.g., views 515A, 515B, and 515C) of a 3D environment (e.g., 3D environments 512A, 512B, and 512C) generated based on camera images and/or depth camera images of the physical environment 100, and optionally include virtual content as part of the meditation experience to simulate natural walks in the city park, but supplement some (or all) of the physical content with virtual content.

In the example shown in fig. 5A, the electronic device 10 provides a view 515A that includes a representation 530 of the spectator 520, a representation 532 of the spectator 522, and a representation 534 of the bird 524. View 515A also includes experience level indication identifier 550, which may be displayed to user 25 during the experience and may allow the user to change one or more attributes associated with view 515A to improve meditation levels such as immersion levels (e.g., block potential distraction, change weather, time of day (sunrise/sunset/moon), etc.). Alternatively, the experience level indication identifier is not displayed to the user. Fig. 5A provides a view 515A of the simulated environment 500, i.e., a view that illustrates a physical environment in which there is no supplemental virtual content for the meditation experience (e.g., representations 530, 532, and 534 are shown). Similarly, in the example shown in FIG. 5B, the electronic device 10 provides a view 515B that includes a representation 540 of the spectator 522, a representation 542 of the spectator 520, a representation 544 of the bird 524, and a experience-level indication identifier 550. Fig. 5B provides a view 515B that has changed the appearance of representations 540, 542, 544 of bystanders 520, 522 and bird 524, respectively. For example, based on experience levels (e.g., meditation experience levels or immersion levels, etc.), the techniques described herein may determine to alter content feedback displayed to a user when meditation occurs in an urban park based on the context of the environment (e.g., bystanders and animals within a distracted field of view may be provided during meditation). In some implementations, the user may select any detected objects that may interfere with the user's meditation session, or the system may automatically fuzzy the objects, or may replace the objects with virtual objects. For example, if the distance of a person from user 25 is greater than a particular threshold (e.g., 10 feet or more), the system may change the views of representations 540, 542, 544 of bystanders 520, 522 and bird 524, respectively, as shown (e.g., an opaque ellipse instead of a person). Similarly, in the example shown in fig. 5C, the electronic device 10 provides a view 515C that includes a representation 560 of a moon, a representation 562 of a calm river, feedback content elements 564, and interactive indication identifiers 550. Fig. 5C provides a view 515C that has changed the time of day (e.g., now nighttime) and has removed some or most of the representations of the view of the physical environment (e.g., representations 540, 542, 544 and city background). For example, when meditation is in an urban park, the user 25 may be distracted by portions of the view of the physical environment 500 when attempting meditation (e.g., based on meditation experience levels). Accordingly, view 515C displays only the portion of the physical environment 500 that may be determined to be a natural landscape (e.g., tree), and may provide feedback content elements 564 to inform the user to focus on his or her breath (e.g., "deep breath"). For example, if the user is in an urban park with lakes and buildings in the background, the only part of the physical environment that might be shown to the user would be a lake, and the buildings could be virtually replaced with a background that is more suitable for meditation, such as a mountain in the background.

Additionally, the user may select any detected objects that may interfere with the user's meditation session, or the system may automatically blur the processing or removal of these objects, or may replace these objects with virtual objects. For example, in the example of view 515C, if the person is more than a particular threshold (e.g., 10 feet or more) from user 25, the system may change view 515C and remove representations 540, 542 of bystanders 520, 522, respectively. Thus, if spectators 520 and/or 522 walk closer to the user (e.g., less than 20 feet), or birds 524 fly closer to user 25, representations 540, 542, respectively, may begin to fade slowly as they walk closer to user 25 until they reach another threshold (e.g., less than 10 feet), and representations 540, 542 may then be fully shown, or the actual view of spectators 520, 522 may "break through" view 515, so that user 25 may clearly see the object approaching them. For example, representation 534 of bird 524 may initially be obscured due to the greater distance; however, if the bird (or any animal) is close to the user, the system may automatically adjust the view of representation 534 of bird 524 so that user 25 knows that the animal may be too close (e.g., a puppy running toward the user).

Fig. 6 is a system flow diagram of an exemplary environment 600 in which a respiration and attention assessment system can assess a user's respiration state and attention state based on physiological data and provide content augmentation within the presentation of content, according to some implementations. In some implementations, the system flow of the exemplary environment 600 is performed on a device (e.g., the device 10 of fig. 1), such as a mobile device, a desktop computer, a laptop computer, or a server device. The content of the exemplary environment 600 may be displayed on a device (such as an HMD) having a screen for displaying images (e.g., display 15) and/or a screen for viewing stereoscopic images (e.g., device 10 of fig. 1). In some implementations, the system flow of exemplary environment 600 is performed on processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 600 is performed on a processor executing code stored in a non-transitory computer readable medium (e.g., memory).

The system flow of routine environment 600 is illustrated: capturing content (e.g., video content or a series of image data) and presenting it to a user (e.g., meditation experience); obtaining physiological data associated with a user during presentation of content; evaluating a respiratory state and an attention state of the user based on the physiological data of the user; and providing content augmentation based on the respiratory status and the attention status (e.g., notification/alert based on the distraction threshold). In some implementations, the example environment 600 also analyzes content and/or environment for context data and provides content augmentation based on respiratory and attention states and the context data. For example, the respiratory state and attention state assessment techniques described herein determine respiratory state and attention state of a user during an experience (e.g., watching meditation video) based on obtained physiological data by providing content enhancements based on the respiratory state and attention state of the user (e.g., notifications, audible signals, alerts, icons, etc., that alert the user that they may be in a particular respiratory state and attention state during presentation of the content).

The exemplary environment 600 includes a content instruction set 610 configured with instructions executable by a processor to provide and/or track content 602 to be displayed on a device (e.g., device 10 of fig. 1). For example, when a user is within the physical environment 604 (e.g., room, outside, etc.), the content instruction set 610 provides the content presentation time 612 including the content 602 to the user 25. For example, the content 602 may include background image and sound data (e.g., video). The content presentation time 612 may be an XR experience (e.g., meditation experience) that includes some virtual content and some images of the physical environment (e.g., meditation experience when viewing natural scenes). Alternatively, the user may wear the HMD and look to the real physical environment via a real-time camera view, or the HMD allows the user to view a display, such as wearing smart glasses through which the user can view, but still present visual cues and/or audio cues. During the experience, while user 25 is viewing content 602, the user's respiratory rate (e.g., respiratory data 615) and pupil data 614 (e.g., pupil data 40 such as eye gaze characteristic data) are tracked and transmitted as physiological data 617. Additionally, other physiological data may be monitored and sent as physiological data 617 (such as EDA data 616 and heart rate data 618).

The environment 600 also includes a physiological tracking instruction set 630 to track a physiological attribute of a user as physiological tracking data 632 using one or more of the techniques discussed herein or other techniques that may be appropriate. For example, the physiological tracking instruction set 630 may obtain physiological data 617 (e.g., pupil data 614 and respiratory data 615) from the user 25 viewing the content 602. Additionally or alternatively, the user 25 may wear a sensor 625 (e.g., sensor 17 of fig. 1, such as an EEG sensor, EDA sensor, heart rate sensor, etc.) that generates sensor data 626 (e.g., EEG data, respiration data 615, EDA data 616, heart rate data 618) as additional physiological data. Thus, when the content 602 is presented to the user as the content presentation time 612, the physiological data 617 (e.g., pupil data 614 and breath data 615) and/or sensor data 626 are sent to the physiological tracking instruction set 630 to track the physiological properties of the user as physiological tracking data 632 using one or more of the techniques discussed herein or other techniques that may be appropriate. Alternatively, the physiological tracking instruction set 630 obtains physiological data associated with the user 25 from the physiological database 635 (e.g., if the physiological data 617 was previously analyzed by the physiological tracking instruction set, such as during a video of a previous viewing/analysis).

In one exemplary implementation, environment 600 further includes a context instruction set 640 configured with instructions that can be executed by the processor to obtain experience data (e.g., content 602) and other sensor data (e.g., image data of environment 604, image data of the face and/or eyes of user 25, etc.) presented to the user and generate context data 642 (e.g., identifying the content 602 and the person, object, etc. of environment 604). For example, the context instruction set 640 obtains the content 602 and sensor data 621 (e.g., image data) from the sensor 620 (e.g., RGB camera, depth camera, etc.), and determines the context data 642 based on identifying an area of the content when the user is viewing the presentation of the content 602 (e.g., content/video being viewed for the first time). Sensors 620 and 625 are shown as separate blocks (sensors); however, in some implementations, sensor 620 and sensor 625 are the same sensor.

Alternatively, the context instruction set 640 selects context data associated with the content 602 from the context database 645 (e.g., if the content 602 was previously analyzed by the context instruction set, such as during a previously viewed/analyzed video). In some implementations, the context instruction set 640 generates scene understands associated with the content 602 and/or the environment 604 as context data 642. For example, scene understanding may be utilized to track what a user may focus on during presentation of content 602, or where the user is, what the user is doing, what physical objects or people are in the vicinity of the user about environment 604.

In one exemplary implementation, environment 600 further includes a content enhancement instruction set 650 configured with instructions executable by the processor to evaluate the user's respiratory state and attention state based on physiological responses (e.g., eye gaze responses, respiratory frequency, etc.) using one or more of the techniques discussed herein or other techniques that may be appropriate. For example, the respiratory state and the attention state are evaluated by determining the position of the respiratory state and the attention state of the user relative to the indication marks (such as respiratory scale 340 and attention scale 330 of fig. 3 and 4). In particular, the content enhancement instruction set 650 obtains the physiological tracking data 632 from the physiological tracking instruction set 630 and the context data 642 (e.g., scene understanding data) from the context instruction set 640, and determines the breathing state and the attention state of the user 25 during presentation of the content 602 and based on the attributes of the physical environment 604 in which the user is viewing the content 602. For example, the context data 642 may provide a scene analysis that may be used by the content enhancement instruction set 650 to understand what a person is looking at, where they are, etc., and to improve the determination of respiratory and attention states. In some implementations, the content enhancement instruction set 650 can then provide content enhancement data 652 (e.g., visual cues and/or audible cues) to the content instruction set 610 based on the respiratory state and the attention state assessment. For example, discovering defined high/low level signs of attention and respiration (e.g., visual respiration indication markers 410) and providing performance feedback during the meditation experience may enhance the meditation experience of the user, provide additional benefits from meditation sessions, and provide guided and supported teaching methods (e.g., cradle teaching methods) to enable the user to practice through his meditation.

In some implementations, content enhancement data 652 may be utilized by content instruction set 610 to present audio and/or visual feedback cues or mechanisms to user 25 to relax and focus on breathing during high level stress situations (e.g., excessive anxiety about upcoming tests). In an educational experience, based on an evaluation from content enhancement instruction set 650 that user 25 is being distracted (e.g., low level attention indication) by user 25 boredom, feedback cues to the user may be mild alerts (e.g., a soothing or calm visual alert and/or audio alert) to resume a learning task.

In some implementations, user experience database 660 may be used to store information associated with user 25. For example, historical data may be stored. For example, after each meditation experience, the physiological data 632, the contextual data 642, meditation experience levels, and the like may be monitored and stored in the user experience database 660 before, during, and/or after the customized feedback content (e.g., content enhancement data 652) is presented to the user. In some implementations, the determination of meditation experience level may be based on an analysis of historical data associated with the user 25 for previous meditation experiences (e.g., stored in the user experience database 660). Alternatively, in some implementations, the meditation experience level may be determined based on accessing a user profile stored in the user experience database 660.

Fig. 7 is a flow chart illustrating an exemplary method 700. In some implementations, a device, such as device 10 (fig. 1), performs the techniques of method 700 to evaluate a respiratory state and an attention state of a user viewing content based on physiological data, and to provide modifications to the content based on the detected respiratory state and attention state. In some implementations, the techniques of method 700 are performed on a mobile device, desktop computer, laptop computer, HMD, or server device. In some implementations, the method 700 is performed on processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the method 700 is performed on a processor executing code stored in a non-transitory computer readable medium (e.g., memory).

At block 702, the method 700 presents content to a user and obtains physiological data associated with the user via one or more sensors. The physiological data may include EEG amplitude/frequency, image data of the user's face, pupil modulation, eye gaze saccades, EDA, heart rate, etc. For example, obtaining the physiological data may involve obtaining, via sensors on the watch, images of the eyes or EOG data from which gaze direction/movement, galvanic skin activity/skin conductance, heart rate may be determined. In addition, facial recognition via the HMD may be included as physiological data (e.g., reconstruction of the user's face).

In some implementations, obtaining physiological data associated with the physiological response of the user includes monitoring for a response or lack of response that occurs within a predetermined time after presentation of the content or the user performs the task. For example, the system may wait up to five seconds after an event within the video to see if the user is looking in a particular direction (e.g., physiological response).

In some implementations, obtaining physiological data (e.g., pupil data 40) is associated with a gaze of a user that may involve obtaining images of eyes or electrooculogram signal (EOG) data from which gaze direction and/or movement may be determined. In some implementations, the physiological data includes at least one of skin temperature, respiration, photoplethysmography (PPG), electrodermal activity (EDA), eye gaze tracking, and pupil movement associated with the user.

Some implementations obtain physiological data and other user information to help improve the user experience. In such a process, user preferences and privacy should be respected by ensuring that the user understands and agrees to the use of the user data, understands what type of user data is used, controls the collection and use of the user data, and limits the distribution of the user data (e.g., by ensuring that the user data is handled locally on the user's device), as examples. The user should have the option of selecting to join or select to exit as to whether to obtain or use his user data or otherwise turn on and off any features that obtain or use user information. Furthermore, each user will have the ability to access and otherwise find anything about him or her that the system has collected or determined. User data is securely stored on the user's device. User data used as input to the machine learning model is securely stored on the user's device, for example, to ensure privacy of the user. The user's device may have a secure storage area, e.g., a secure compartment, for protecting certain user information, such as data from image sensors and other sensors for facial recognition, or biometric recognition. User data associated with the user's body and/or attention state may be stored in such a secure compartment, thereby restricting access to the user data and restricting transmission of the user data to other devices to ensure that the user data remains securely on the user's device. User data may be prohibited from leaving the user device and may only be used in the machine learning model and other processes on the user device.

In some implementations, the presented content includes an XR experience (e.g., meditation session).

At block 704, the method 700 determines a respiratory state of the user based on the physiological data. For example, a machine learning model may be used to determine respiratory status based on physiological data and audio/visual content of the experience and/or environment. For example, one or more physiological characteristics may be determined, aggregated, and used to classify a user's respiratory state using statistical or machine learning techniques. In some implementations, the response may be compared to a previous respiratory response of the user himself or a typical respiratory response of the user to similar content of similar experience and/or similar environmental attributes.

The determined respiration state may be about 7 breaths per minute, as shown by the respiration scale 340 of fig. 3. In some implementations, determining the respiratory status may involve sensor fusion of the different acquired data without using additional respiratory sensors. For example, the different data acquired that may be fused may include head pose data from the IMU, audio from a microphone, camera images of the user's face and/or body (e.g., an HMD with a jaw camera, a lower camera, an eye camera for eye surrounding tissue, etc.), body motion, and/or a signal of the face modulated by breathing (e.g., remote PPG). Using this type of sensor fusion to track the user's breathing (such as when wearing an HMD) may eliminate the need for the user to wear sensors around, for example, the user's diaphragm, e.g., to track his or her breathing frequency.

At block 706, the method 700 determines an attention state of the user based on the obtained physiological data and the context of the experience. For example, a machine learning model may be used to determine respiratory status based on eye tracking and other physiological data, as well as audio/visual content of the experience and/or environment. For example, one or more physiological characteristics may be determined, aggregated, and used to classify a user's attention state using statistical or machine learning techniques. In some implementations, the response may be compared to the user's own previous response or the user's typical response to similar content of similar experience and/or similar environmental attributes. In some implementations, the state of attention is determined based on measuring gaze or body stability using the physiological data. In some implementations, the attention state is determined based on determining the attention level. In some implementations, the attention state is determined based on the respiratory state (e.g., a particular range of respiratory frequencies may indicate that the user is focused on a task).

In some implementations, determining that the user has a particular attention threshold (e.g., high, low, etc.) includes determining the level of attention as a sliding scale. For example, the system may determine the level of attention as an attention index that may be customized based on the type of content shown during the user experience. If there is a high level of attention, in the case of education, the content developer may design an environment for the experience that will provide the user with a "best" environment for learning the experience. For example, the ambience lighting is tuned so that the user can be at an optimal level to learn during the experience.

In some implementations, statistical or machine-learning based classification techniques may be used to determine respiratory status and attention status. For example, determining that the user has a respiratory state and an attention state includes using a machine learning model that is trained using baseline truth data that includes a self-assessment, wherein the user marks portions of the experience with respiratory state and attention state labels. For example, to determine baseline truth data including self-assessment, a group of subjects may be prompted at different time intervals (e.g., every 30 seconds) while viewing meditation video. Alternatively or additionally, the benchmark truth data comprising self-assessment while viewing video comprises different examples of meditation events. For example, after each "meditation event", each subject may be prompted to enter his or her breathing state and attention state at or after a particular meditation event in the video content.

In some implementations, the method 700 further includes: identifying an emotional state of the user, the emotional state corresponding to a plurality of time periods; and presenting an indication of progress based on the emotional state. For example, identifying the emotional state of the user may be based on user input or feedback during presentation of the content (e.g., an emotional log during the meditation experience), and/or identifying the emotional state of the user may be based on the obtained physiological data.

In some implementations, the content is presented to a plurality of users during a communication session. For example, a pair or group of people (e.g., 2 or more) may share an experience together in an XR environment (e.g., a video game that monitors physiological data of the people). The shared experience may include a teacher and one or more students, where the teacher (or any other person) may teach the user how to improve his or her experience during presentation of the content (e.g., focus on particular visual content and/or audio content, such as particular content in a game).

In some implementations, one or more pupil or EEG characteristics may be determined, aggregated, and used to classify the user's respiratory and attention states using statistical or machine learning techniques. In some implementations, the physiological data is classified based on comparing variability of the physiological data to a threshold. For example, if a baseline of EEG data of a user is determined during an initial period (e.g., 30 seconds to 60 seconds) and the EEG data deviates by more than +/-10% from the EEG baseline during a subsequent period (e.g., 5 seconds) after auditory stimulation, the techniques described herein may classify the user as transitioning from a high respiratory state and an attentive state and into a second low respiratory state and attentive state. Similarly, heart rate data and/or EDA data are classified based on their variability compared to a particular threshold.

In some implementations, the machine learning model is a neural network (e.g., an artificial neural network), a decision tree, a support vector machine, a bayesian network, and the like. These tags may be collected from the user in advance or from a population of people in advance and later fine tuned for individual users. Creating this tagged data may require many users to experience an experience (e.g., meditation experience) in which the user can listen to natural sounds (e.g., auditory stimuli) with a hybrid natural probe, and then randomly ask the user how to concentrate or relax (e.g., breathing state and attention state) soon after the probe is presented. Answers to these questions may generate labels at a time prior to the questions, and a deep neural network or a deep Long Short Term Memory (LSTM) network may learn a combination of features specific to the user or task given those labels (e.g., low respiratory and attentive states, high respiratory and attentive states, etc.).

At block 708, the method 700 determines modifications to the content based on the user's respiratory state and attention state, and at block 710, the method 700 presents the modified content to the user. For example, the determined respiratory status and attention status may be used to provide feedback to the user via content enhancements that may help the user, provide statistics to the user, and/or help the content creator improve the content of the experience. In some aspects, the content enhancement may be the beginning or ending of the meditation experience, or a change during the ongoing experience. In some implementations, the modification of the content includes: a graphical indication logo or sound configured to change a first attention state to a second attention state. In some implementations, the modification of the content includes: visual or audible indication of the proposed time of experience. In some aspects, content augmentation may use shrink and zoom icons such as hearts or flowers (e.g., visual respiration indication markers 410) to provide visualization of current respiration states or desired/improved respiration states (such as oscillations). Alternatively, interactive icons may be used to encourage the user to breathe harder/deeper, such as blowing off a virtual candle. In some aspects, content augmentation may use subtle cues to change from lighter to darker environments and/or ringtones, audio, chimes.

In some implementations, the method 700 further includes: presenting instructions to the user to notice breathing; and evaluating an attention level to respiration based on the respiration state and the attention state, wherein the modification is determined based on the attention level. For example, content augmentation may be explicitly upgraded from subtle cues to direct instructions (e.g., "focus on breathing"). In some aspects, content enhancement may use spatialization audio to redirect attention. In some implementations, the modifying includes: a visual or audible representation of the breathing state; and a visual or audible representation of the state of attention. In some implementations, the modifying includes: a cue configured to trigger a change in the respiratory state or the attention state.

In some embodiments, the method 700 further comprises: determining a baseline corresponding to the user based on the physiological data; determining a target for the user based on the baseline; and determining the modification based on the baseline and the target. For example, in some aspects, content augmentation may provide simultaneous/combined feedback regarding the respiratory state and the attention state. For example, a sine wave of the user's breathing frequency may be shown, as well as a desired sine wave that the user may try and simulate (e.g., slow down the user's breathing frequency for a particular meditation experience). In some implementations, during the meditation experience, the modification includes: initiation of new meditation; conclusions of meditation; or an ongoing meditation change.

Fig. 8 is a flow chart illustrating an exemplary method 800. In some implementations, a device, such as device 10 (fig. 1), performs the techniques of method 800 to provide customized feedback content during meditation based on physiological data. In some implementations, the techniques of method 800 are performed on a mobile device, desktop computer, laptop computer, HMD, or server device. In some implementations, the method 800 is performed on processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the method 800 is performed on a processor executing code stored in a non-transitory computer readable medium (e.g., memory).

At block 802, the method 800 obtains physiological data via one or more sensors. The physiological data may include EEG amplitude/frequency, image data of the user's face, pupil modulation, eye gaze saccades, EDA, heart rate, etc. For example, obtaining the physiological data may involve obtaining, via sensors on the watch, images of the eyes or EOG data from which gaze direction/movement, galvanic skin activity/skin conductance, heart rate may be determined. In addition, facial recognition via the HMD may be included as physiological data (e.g., reconstruction of the user's face).

At block 804, the method 800 determines an attention state during meditation mode based on the physiological data. For example, measured gaze (e.g., eye gaze stability) may be used to determine that the user is straying because, for example, in the context of breath tracking, straying tends to result in greater eye movement. In some implementations, the level of attention as well as systemic stability may be monitored.

In some implementations, a machine learning model may be used to determine respiratory status based on eye tracking and other physiological data as well as audio/visual content of the experience and/or environment. For example, one or more physiological characteristics may be determined, aggregated, and used to classify a user's attention state using statistical or machine learning techniques. In some implementations, the response may be compared to the user's own previous response or the user's typical response to similar content of similar experience and/or similar environmental attributes. In some implementations, the state of attention is determined based on measuring gaze or body stability using the physiological data. In some implementations, the attention state is determined based on determining the attention level. In some implementations, the attention state is determined based on the respiratory state (e.g., a particular range of respiratory frequencies may indicate that the user is focused on a task).

At block 806, the method 800 customizes feedback content for guiding the change in the state of attention during the meditation mode based on the user attributes. For example, based on visual/auditory scene understanding, the user attribute may be meditation experience level, mind level, user's schedule, where the user is, what the user is doing, what objects are in the vicinity of the user, etc. Additionally, feedback may be tailored with respect to how long the delay is, what type of feedback is provided, and so forth.

In some implementations, heuristic techniques are utilized to determine the appropriate amount of delay before attempting to assist the user in re-focusing (e.g., providing them with enough time to perform this operation before intervention). In some implementations, the amount of delay before providing the customized feedback content may vary based on the user's ability (e.g., based on meditation experience level).

In some implementations, the determined attention state may be used to provide feedback to the user via customized feedback content that may assist the user, provide statistics to the user, and/or assist the content creator in improving the content of the experience. In some aspects, the customized feedback content may be the beginning or end of the meditation experience. In some implementations, the modification of the content includes: a graphical indication logo or sound configured to change a first attention state to a second attention state. In some implementations, the modification of the content includes: visual or audible indication of the proposed time of experience. In some aspects, content augmentation may use shrink and zoom icons such as hearts or flowers (e.g., visual respiration indication markers 410) to provide visualization of current respiration states or desired/improved respiration states (such as oscillations). Alternatively, interactive icons may be used to encourage the user to breathe harder/deeper, such as blowing off a virtual candle. In some aspects, content augmentation may use subtle cues to change from lighter to darker environments and/or ringtones, audio, chimes.

At block 808, the method 800 provides customized feedback content during the meditation mode based on the user attributes after a delay time. In some implementations, the method 800 customizes feedback that encourages the user to re-focus on resolving the distraction detected during meditation, such as when feedback is provided, how long to delay after detecting distraction before providing feedback, the type of feedback provided, and the like. The feedback may be customized based on user attributes such as the user's ability or previous experience (e.g., a new user or an experienced meditation that has been historically shown may be re-focused based on the level of distraction). For example, as shown in fig. 5A-5C, the techniques described herein may tailor feedback content in the view of the device 10 to improve the user's meditation experience based on the user's meditation experience level and/or other potential surrounding objects (e.g., people, animals, etc.) that may be detrimental to the meditation experience.

In some implementations, the user's experience may be tracked after the customized feedback content is presented during the meditation session. In one exemplary embodiment, the method 800 further comprises: responsive to providing the customized feedback content, determining an attention level over a period of time based on the physiological data, the attention level corresponding to the user; and determining a feedback metric for the user based on the determined level of attention over the period of time. For example, after each meditation experience, physiological data, contextual information, meditation experience levels, etc. may be monitored and stored (e.g., user experience database 660 in fig. 6) before, during, and after the customized feedback content is presented to the user. In some implementations, the attribute is meditation experience level. In some implementations, the meditation experience level may be determined based on an analysis of historical data associated with the user for previous meditation experiences (e.g., stored in the user experience database 660). Alternatively, in some implementations, the meditation experience level may be determined based on accessing a user profile (e.g., stored in the user experience database 660). In some implementations, the meditation experience level may be updated based on the state of attention during the meditation experience.

In some implementations, the feedback content may be customized based on tracking respiration during the meditation session. In one exemplary embodiment, the method 800 further comprises: presenting instructions to the user to notice breathing; and evaluating an attention level to respiration based on the respiration state and the attention state, wherein the customized feedback is determined based on the attention level.

In some implementations, customizing the characteristic of the feedback content includes: determining a baseline corresponding to the user based on the physiological data; determining a target for the user based on the baseline; and determining customized feedback content based on the baseline and the target. For example, if a baseline of the user's respiratory state is determined during an initial period (e.g., 30 seconds to 60 seconds of the user's concentration in the meditation session), and during a subsequent period (e.g., 5 seconds) following certain feedback content such as auditory stimuli, if the respiratory data deviates more than +/-10% from the respiratory baseline during the subsequent period, the techniques described herein may classify the user as transitioning from the baseline respiratory state (e.g., lower quality meditation experience).

In some implementations, meditation experience logs may be utilized. In one exemplary embodiment, the method 800 further comprises: identifying a meditation state, the meditation state corresponding to a plurality of time periods; and presenting an indication of progress based on the meditation status. For example, meditation states such as emotional states, respiratory levels, etc. may be identified during the experience based on user input and/or physiological data.

In some implementations, the method 800 further includes: a portion of the meditation is identified, the portion being associated with a particular attention state. For example, the techniques described herein may recommend or not recommend similar content or portions of the content or help content developers to the user to improve the content for future users (e.g., provide helpful/therapeutic content that works for similar experiences of other users for meditation).

In some implementations, the meditation session can be customized based on the context of the environment. In one exemplary embodiment, the method 800 further comprises: determining a context of the meditation based on sensor data of an environment of the meditation; and customizing the characteristic of the feedback content based on the context of the meditation. For example, the customization may be based on generating scene understanding of visual and/or auditory properties of the environment (e.g., where the user is, what the user is doing, what objects are nearby) using computer vision. Additionally or alternatively, the customization may be based on video content presented to the user, or what he or she is doing in the environment. In some implementations, determining the context of the experience includes generating a scene understanding of the environment based on the sensor data of the environment, the scene understanding including a visual or audible attribute of the environment, and determining the context of the experience based on the scene understanding of the environment. In some implementations, determining the context of the experience includes determining an activity of the user based on a schedule of the user. For example, by accessing a user's calendar, the system may determine that there is a scheduled treatment session.

In some implementations, the customized feedback content includes: conclusions of current meditation; an activation of another meditation different from the current meditation; and/or a change of the current meditation. In some implementations, the customized feedback content includes: the volume of the audio signal modulated based on the physiological data. In some implementations, the customized feedback content includes: a visual or audible representation of the state of attention or a change of the state of attention. In some implementations, the customized feedback content includes: a cue configured to trigger a change in the attention state. In some implementations, the customized feedback content includes: a graphical indication logo or sound configured to change a first attention state to a second attention state. In some implementations, the customized feedback content includes: visual or audible indication of suggested time for the new meditation experience.

In some implementations, the meditation is presented to a plurality of users during a communication session. For example, a pair or group of people (e.g., 2 or more) may share meditation experiences together in an XR environment. The shared experience may include a tutor and a patient, where the tutor (or any other person) may teach the user how to better meditation (e.g., focus on particular visual and/or audio content, such as a bird song or waterfall) during presentation of the content.

The following embodiments or implementations illustrate aspects that may be utilized by the method 700, the method 800, or any other process described herein.

In some implementations, predicting the type of physiological state, such as meditation state (e.g., physical, cognitive, social) may be utilized by the processes described herein. In one exemplary implementation, determining the breathing state and the attention state of the user during a portion of the experience further comprises determining a meditation type of the user based on the sensor data, and providing content enhancement during the experience is further based on the meditation type. For example, if the user is experiencing a low level meditation experience (e.g., is working and is not focusing on breathing), the content augmentation may include video content and/or auditory content (e.g., a notification of "deep breath," or adding relaxation music) that may help the user obtain a target breathing state and/or attention state to find a better level meditation experience.

In some implementations, feedback may be provided to the user based on a determination that the respiratory state and the attention state (e.g., playing a vigorous video game) are different from the expected respiratory state and the attention state of the experience (e.g., the respiratory state and the attention state of a particular portion of the video game that the content developer wants to increase). In some implementations, the method 700 or the method 800 may further include: feedback (e.g., audio feedback such as "control your breath", visual feedback, etc.) is presented during the experience in response to determining that the breathing state and the attention state are different from the second breathing state and the attention state that are expected by the experience. In one example, during a portion of an educational experience in which a user is learning for a difficult test, the method determines to present feedback to indicate that the user is focused on breathing based on detecting that the user is conversely in a high breathing state and an attentive state while learning.

In some implementations, the respiratory state and the attention state are a first respiratory state and a first attention state, and the method further comprises: the method includes obtaining first physiological data (e.g., EEG amplitude, pupil movement, etc.) associated with a user's physiological response (or lack of response) to the content enhancement using a sensor, and determining a second respiratory state and a second attention state of the user based on the user's physiological response to the content enhancement. In some implementations, the method further includes: evaluating a second respiratory state and a second attention state of the user based on the user's physiological response to the content enhancement; and determining whether the content enhancement reduces the stress of the user by comparing the second respiratory state and the second attention state to the first respiratory state and the first attention state. For example, the respiration state and the attention state may be compared with the user's own previous response or a typical response of the user to a similar stimulus. Statistical or machine-learning based classification techniques may be used to determine respiratory status and attention status. Additionally, the determined respiratory status and attention status may be used to provide feedback/redirect the user to the user, provide statistics to the user, or help the content creator create a more efficient meditation experience, learning experience, respiration, workday, etc.

In some implementations, providing the content enhancement includes providing a graphical indication identifier or sound configured to change the respiratory state and the attention state to a second respiratory state and a second attention state that correspond to physiological data exhibited by the user during the portion of the experience in the task. In some implementations, providing content augmentation includes providing a mechanism for rewinding from content associated with a task or providing an interrupt (e.g., rewinding to replay a previous step during cooking of a video, or pausing an educational course for a rest between classes). In some implementations, providing content augmentation includes suggesting a time for another experience based on a respiratory state and an attention state.

In some implementations, a contextual analysis may be obtained or generated to determine what content the user is focusing on, which content is creating an increase (or decrease) in respiratory state and attention state, which may include a scene understanding of the content and/or physical environment. In one exemplary implementation, the method 700 or 800 may further comprise: a portion of the experience associated with the respiratory state and the attention profile is identified. For example, identifying a portion of the experience associated with a particularly high respiratory state and/or a particularly low attentive state (e.g., exceeding a threshold), the data may provide a recommendation (or counsel) to the user of similar content or portions of the content or help the content developer to refine the content for future users. For example, the goal of a content developer may be to increase stress in a video game, decrease stress on the meditation experience, or increase stress in the case of a user "boring" while learning or working (e.g., to improve attention/respiration performance levels).

In some implementations, the method 700 or the method 800 further includes: content corresponding to the experience is adjusted based on the respiratory status and the attention status (e.g., customized for the respiratory status and the attention status of the user). For example, content recommendations for a content developer may be provided based on determining respiratory and attention states during a presented experience and changes in the experience or content presented therein. For example, when providing a specific type of content, a user may concentrate on and perform a good meditation. In some implementations, the method 700 or the method 800 may further include: identifying content based on similarity of the content to the experience; and providing content recommendations to the user based on determining that the user has a breathing state and a attentiveness state (e.g., distraction) during the experience. In some implementations, the method 700 or the method 800 may further include: content included in the experience is customized based on the user's breathing state and attention state (e.g., the content is divided into smaller pieces).

In some implementations, the content of the experience may be adjusted corresponding to the experience based on a respiratory state and an attention state that are different from an expected respiratory state and an attention state of the experience. For example, content may be adapted by experienced developers to improve recorded content for subsequent use by a user or other users. In some implementations, the method 700 or the method 800 may further include: the content corresponding to the experience is adjusted in response to determining that the respiratory state and the attention state are different from a second respiratory state and an attention state expected by the experience.

In some implementations, the method 700 or method 800 may determine a context of an experience based on sensor data of an environment. For example, determining the context may involve using computer vision to generate scene understanding of visual and/or auditory properties of the environment-where the user is, what the user is doing, what objects are nearby. Additionally, a scene understanding of the content presented to the user may be generated, the scene understanding including visual and/or auditory properties of the content being viewed by the user.

In some aspects, different contexts of the presented content and environment are analyzed to determine where the user is, what the user is doing, what objects or people in the environment or within the content are nearby, what the user did earlier (e.g., meditation in the morning). Additionally, the contextual analysis may include image analysis (semantic segmentation), audio analysis (vibration sounds), position sensors (where the user is), motion sensors (fast moving vehicles), and even access other user data (e.g., the user's calendar). In one exemplary implementation, the method 700 or 800 may further comprise: determining a context of the experience by generating a scene understanding of the environment based on the sensor data of the environment, the scene understanding including a visual or audible attribute of the environment; and determining a context of the experience based on the scene understanding of the environment.

In some implementations, the sensor data includes image data, and generating the scene understanding is based at least on performing semantic segmentation of the image data and detecting one or more objects within the environment based on the semantic segmentation. In some implementations, determining the context of the experience includes determining an activity of the user based on a scene understanding of the environment. In some implementations, the sensor data includes location data of the user, and determining the context of the experience includes determining a location of the user within the environment based on the location data.

In some implementations, determining the context of the experience may involve identifying an object or person with which the user is interacting. Determining the context of the experience may involve determining that the user is talking to another person. Determining the context of the experience may involve determining that an interaction or session with another person may (or may not) cause a state of tension for the user. Evaluating whether an individual is more or less likely to elicit a stress response to the user may involve identifying the individual and classifying the individual based on appearance of the individual, based on actions of the individual, and/or based on activities in which the individual participates. For example, if other individuals are identified at work as a boss of the user, the boss may be identified via facial recognition or classified as a colleague. When interacting with a person classified as his or her boss, the user's pressure may then be tracked based on his or her respiratory status and attention status. When evaluating pressure therapy techniques to better address high pressure situations, it may be useful to provide feedback to the user (or his or her coaching staff) regarding the higher respiratory status and attention of the user when interacting with his or her boss. Additionally, the shared experience may include a group of users sharing common interests as meditation of a group, where the XR environment will enhance the collective experience of the group and/or the individual experience.

In some implementations, determining the context of an experience may involve determining a scene understanding or scene knowledge of a particular location of the user experience (e.g., a particular room, building, etc.) that is more or less likely to result in a stress state (e.g., based on past stress experiences occurring there). Determining scene understanding or scene knowledge of an experience may involve monitoring low-level characteristics of the scene that may cause stress. For example, as part of scene understanding or scene knowledge, loud noise, subtle sounds, bright light flashes, sirens, rumble, and the like may be monitored and analyzed. In addition, scene knowledge may provide information that a particular activity or content may be cumbersome or stressful. For example, the scene knowledge may include experiences or events that the user is currently participating in, such as attending interviews, reading distracted news stories, watching horror movies, playing violent video games, and so forth. Understanding scene knowledge may involve other strenuous experiences such as threat stimuli (e.g., offensive dogs), injuries to loved ones, perceived physical hazards to users (e.g., oncoming cars), network spoofing, personally-exclusive responsibility, and the like.

In some implementations, determining the context of the experience includes determining an activity of the user based on a schedule of the user. For example, the system may access a user's calendar to determine whether a particular event is occurring when a particular respiratory state and attention state are evaluated (e.g., a planned meditation session, the user attending an important meeting or curriculum late, or is scheduled to speak in the near future before a group).

In some implementations, the techniques described herein obtain physiological data (e.g., pupil data 40, EEG amplitude/frequency data, pupil modulation, eye gaze saccades, heart rate data, EDA data, etc.) from a user based on identifying typical interactions of the user with the experience. For example, the techniques may determine that variability in the eye gaze characteristics of a user is related to interactions with an experience. Additionally, the techniques described herein may then adjust visual characteristics of the experience, or adjust/alter sounds associated with the content enhancement, to enhance and physiologically responsive data associated with the experience and/or future interactions with the content enhancement presented within the experience. Furthermore, in some implementations, changing the content enhancement after the user interacts with the experience informs of the user's physiological response in subsequent interactions with the experience or a particular segment of the experience. For example, the user may present an expected physiological response associated with the change in experience. Thus, in some implementations, the techniques identify intent of a user to interact with the experience based on an expected physiological response. For example, the techniques may adapt or train the instruction set by capturing or storing physiological data of the user based on the user's interactions with the experience, and may detect future intent of the user to interact with the experience by identifying the physiological response of the user in the presentation of the expected enhanced/updated experience.

In some implementations, an estimator or statistical learning method is used to better understand or predict physiological data (e.g., pupil data characteristics, EEG data, EDA data, heart rate data, etc.). For example, statistics of EEG data may be estimated by sampling the data set with replacement data (e.g., self-help).

In some implementations, the technique may be trained on multiple sets of user physiological data and then adapted to each user individually. For example, the content creator may customize the educational experience (e.g., a culinary teaching video) based on user physiological data, such as the user may require background music, different ambience lighting to learn, or more or less audio or visual cues to continue to maintain meditation.

In some implementations, customization of the experience may be controlled by the user. For example, the user may select the experience he or she wants, such as he or she may select the surrounding environment, background scene, music, etc. Additionally, the user may alter the threshold at which content enhancement is provided. For example, the user may customize the sensitivity of triggering content enhancement based on the previous experience of the session. For example, the user may desire to feedback notifications less often and allow some degree of distraction (e.g., eye position deviation) before triggering the notification. Thus, when higher criteria are met, a particular experience may be customized upon triggering a threshold. For example, in some experiences (such as educational experiences), a user may not want to be disturbed during a learning session, even if he or she is briefly staring at a task or distraction by briefly looking to a different area (e.g., less than 30 seconds) to think about what he or she just read. However, the student/reader will want to be notified if he or she is distracted for a longer period of time (e.g., longer than or equal to 30 seconds) by providing content enhancements such as audible notification (e.g., "wake up").

In some implementations, the techniques described herein may interpret the real world environment 5 (e.g., visual quality such as brightness, contrast, semantic context) of the user 25 when evaluating how much content or content enhancement to be presented adjusts or adjusts to enhance the physiological response (e.g., pupillary response) of the user 25 to the visual characteristics 30 (e.g., content enhancement).

In some implementations, the physiological data (e.g., pupil data 40) may change over time, and the techniques described herein may use the physiological data to detect patterns. In some implementations, the pattern is a change in physiological data from one time to another, and in some other implementations, the pattern is a series of changes in physiological data over a period of time. Based on detecting the pattern, the techniques described herein may identify changes in the breathing state and the attention state of the user, and then may provide content enhancements (e.g., visual cues or audible cues about focusing on breathing) to the user 25 during the experience to return to the desired state (e.g., lower breathing state and attention state). For example, the breathing state and attention state of the user 25 may be identified by detecting patterns in the user's gaze characteristics, heart rate, and/or PDA data, visual cues or auditory cues associated with the experience may be adjusted (e.g., content enhancement of speech that indicates "focus on breathing" may also include visual cues or changes in the surrounding environment of the scene), and the user's gaze characteristics, heart rate, and/or PDA data compared to the adjusted experience may be used to confirm the user's breathing state and attention state.

In some implementations, the techniques described herein may utilize training or calibration sequences to adapt to particular physiological characteristics of a particular user 25. In some implementations, the technique presents a training scenario to the user 25 in which the user 25 is instructed to interact with screen items (e.g., feedback objects). By providing a known intent or region of interest to the user 25 (e.g., via instructions), the technique can record the user's physiological data (e.g., pupil data 40) and identify patterns associated with the user's physiological data. In some implementations, the techniques may alter the visual characteristics 30 associated with the content 20 (e.g., content enhancement) in order to further accommodate the unique physiological characteristics of the user 25. For example, the technique may instruct the user to subjectively select a button in the center of the screen associated with the identified region when counted in three and record physiological data of the user (e.g., pupil data 40) to identify a pattern associated with the breathing state and the attention state of the user. Further, the techniques may alter or alter visual characteristics associated with the content enhancement in order to identify patterns associated with physiological responses of the user to the altered visual characteristics. In some implementations, the pattern associated with the physiological response of the user 25 is stored in a user profile associated with the user, and the user profile can be updated or recalibrated at any time in the future. For example, the user profile may be automatically modified over time during the user experience to provide a more personalized user experience (e.g., a personal educational experience for the best learning experience while learning).

In some implementations, a machine learning model (e.g., a trained neural network) is applied to identify patterns in physiological data, including identifying physiological responses to presentation of content (e.g., content 20 of fig. 1) during a particular experience (e.g., education, meditation, teaching, etc.). Further, the machine learning model may be used to match these patterns with learning patterns corresponding to indications of interests or intentions of the user 25 interacting with the experience. In some implementations, the techniques described herein may learn patterns specific to a particular user 25. For example, the technique may begin learning from determining that the peak pattern represents an indication of the user's 25 interest or intent in response to a particular visual characteristic 30 within the content, and use that information to subsequently identify a similar peak pattern as another indication of the user's 25 interest or intent. Such learning may allow for relative interactions of the user with the plurality of visual characteristics 30 to further adjust the visual characteristics 30 and enhance the user's physiological response to the experience and presented content (e.g., focusing on a particular area of the content but not other distracted areas).

In some implementations, the position and features (e.g., edges of eyes, nose, or nostrils) of the head 27 of the user 25 are extracted by the device 10 and used to find coarse position coordinates of the eyes 45 of the user 25, thereby simplifying the determination of accurate eye 45 features (e.g., position, gaze direction, etc.) and making gaze characteristic measurements more reliable and robust. Furthermore, device 10 may easily combine the position of the 3D component of head 27 with gaze angle information obtained by eye component image analysis in order to identify a given screen object that user 25 views at any given time. In some implementations, the use of 3D mapping in combination with gaze tracking allows the user 25 to freely move his or her head 27 and eyes 45 while reducing or eliminating the need to actively track the head 27 using sensors or transmitters on the head 27.

By tracking the eyes 45, some implementations reduce the need to recalibrate the user 25 after the user 25 moves his or her head 27. In some implementations, the device 10 uses depth information to track movement of the pupil 50, thereby enabling calculation of a reliably presented pupil diameter 55 based on a single calibration by the user 25. Using techniques such as Pupil Center Cornea Reflection (PCCR), pupil tracking, and pupil shape, device 10 may calculate pupil diameter 55 and the gaze angle of eye 45 from the points of head 27, and use the positional information of head 27 to recalculate the gaze angle and other gaze characteristic measurements. In addition to reduced recalibration, further benefits of tracking head 27 may include reducing the number of light projection sources and reducing the number of cameras used to track eye 45.

In some implementations, the techniques described herein may identify a particular object within content presented on the display 15 of the device 10 at a location in the direction of user gaze. Further, the technique may change the state of visual characteristics 30 associated with a particular object or overall content experience in response to verbal commands received from the user 25 and the identified respiratory and attention states of the user 25. For example, the particular object within the content may be an icon associated with a software application, and the user 25 may look at the icon, speak the word "select" to select the application, and may apply a highlighting effect to the icon. The technique may then use additional physiological data (e.g., pupil data 40) in response to visual characteristics 30 (e.g., content enhancement) to further identify the respiratory state and the attention state of user 25 as confirmation of the user's verbal command. In some implementations, the technique can identify a given interactive item in response to a direction of a user's gaze and manipulate the given interactive item in response to physiological data (e.g., variability in gaze characteristics). The technique may then confirm the direction of the user's gaze based on the physiological data used to further identify the user's breathing state and attention state in response to interactions with the experience (e.g., interactions within a violent video game). In some implementations, the technique may remove interactive items or objects based on the identified interests or intents. In other implementations, the techniques may automatically capture an image of the content upon determining the interests or intentions of the user 25.

Fig. 9 is a block diagram of an exemplary device 900. Device 900 illustrates an exemplary device configuration of device 10. While certain specific features are shown, those of ordinary skill in the art will appreciate from the disclosure that various other features are not shown for brevity and so as not to obscure more pertinent aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some implementations, the device 10 includes one or more processing units 902 (e.g., microprocessors, ASIC, FPGA, GPU, CPU, processing cores, and the like), one or more input/output (I/O) devices and sensors 906, one or more communication interfaces 908 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I C, and/or the like), one or more programming (e.g., I/O) interfaces 910, one or more displays 912, one or more inwardly and/or outwardly facing image sensors 914, a memory 920, and one or more communication buses 904 for interconnecting these components and various other components.

In some implementations, the one or more communication buses 904 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors 906 include at least one of: an Inertial Measurement Unit (IMU), accelerometer, magnetometer, gyroscope, thermometer, one or more sensors to obtain physiological data (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), and so forth.

In some implementations, the one or more displays 912 are configured to present a view of the physical environment or the graphical environment to a user. In some implementations, the one or more displays 912 correspond to holographic, digital Light Processing (DLP), liquid Crystal Displays (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistors (OLET), organic Light Emitting Diodes (OLED), surface conduction electron emitter displays (SED), field Emission Displays (FED), quantum dot light emitting diodes (QD-LED), microelectromechanical systems (MEMS), and/or similar display types. In some implementations, the one or more displays 912 correspond to diffractive, reflective, polarizing, holographic, etc. waveguide displays. For example, the device 10 includes a single display. As another example, the device 10 includes a display for each eye of the user.

In some implementations, the one or more image sensor systems 914 are configured to obtain image data corresponding to at least a portion of the physical environment 5. For example, the one or more image sensor systems 914 include one or more RGB cameras (e.g., with Complementary Metal Oxide Semiconductor (CMOS) image sensors or Charge Coupled Device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, and the like. In various implementations, the one or more image sensor systems 914 also include an illumination source, such as a flash, that emits light. In various implementations, the one or more image sensor systems 914 also include an on-camera Image Signal Processor (ISP) configured to perform a plurality of processing operations on the image data.

Memory 920 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices. In some implementations, the memory 920 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 920 optionally includes one or more storage devices remotely located from the one or more processing units 902. Memory 920 includes a non-transitory computer-readable storage medium.

In some implementations, memory 920 or a non-transitory computer-readable storage medium of memory 920 stores an optional operating system 930 and one or more instruction sets 940. Operating system 930 includes procedures for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 940 includes executable software defined by binary information stored in the form of electrical charges. In some implementations, the instruction set 940 is software that is executable by the one or more processing units 902 to implement one or more of the techniques described herein.

The instruction set 940 includes a content instruction set 942, a physiological tracking instruction set 944, a context instruction set 946, and a respiratory status and content enhancement instruction set 948. The instruction set 940 may be embodied as a single software executable or as a plurality of software executable files.

In some implementations, the content instruction set 942 may be executed by the processing unit 902 to provide and/or track content for display on a device. The content instruction set 942 may be configured to monitor and track content over time (e.g., during an experience such as an educational session) and/or identify changing events that occur within the content. In some implementations, the content instruction set 942 may be configured to add change events to content (e.g., content enhancement) using one or more of the techniques discussed herein or other techniques as may be appropriate. For these purposes, in various implementations, the instructions include instructions and/or logic for the instructions as well as heuristics and metadata for the heuristics.

In some implementations, the physiological tracking instruction set 944 is executable by the processing unit 902 to track physiological attributes of the user (e.g., EEG amplitude/frequency, pupil modulation, eye gaze saccades, heart rate, EDA data, etc.) using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various implementations, the instructions include instructions and/or logic for the instructions as well as heuristics and metadata for the heuristics.

In some implementations, the contextual instruction set 946 may be executed by the processing unit 902 to determine the context of the experience and/or environment (e.g., create a scene understanding to determine objects or people in the content or in the environment, where the user is, what the user is looking at, etc.) using one or more of the techniques discussed herein (e.g., object detection, facial recognition, etc.) or other techniques that may be appropriate. For these purposes, in various implementations, the instructions include instructions and/or logic for the instructions as well as heuristics and metadata for the heuristics.

In some implementations, the respiration state and content enhancement instruction set 948 can be executed by the processing unit 902 to evaluate the respiration state and attention state (e.g., concentration, distraction, etc.) of the user based on the physiological data (e.g., eye gaze response) and contextual data of the content and/or environment using one or more of the techniques discussed herein or other techniques as may be appropriate. For these purposes, in various implementations, the instructions include instructions and/or logic for the instructions as well as heuristics and metadata for the heuristics.

While the instruction set 940 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may reside on separate computing devices. In addition, FIG. 9 is used more as a functional description of various features present in a particular implementation, as opposed to a schematic of the implementations described herein. As will be appreciated by one of ordinary skill in the art, the individually displayed items may be combined and some items may be separated. The actual number of instruction sets, and how features are distributed among them, will vary depending upon the particular implementation, and may depend in part on the particular combination of hardware, software, and/or firmware selected for the particular implementation.

Fig. 10 illustrates a block diagram of an exemplary head mounted device 1000, according to some implementations. The headset 1000 includes a housing 1001 (or shell) that houses the various components of the headset 1000. The housing 1001 includes (or is coupled to) an eye pad (not shown) disposed at a proximal (user 25) end of the housing 1001. In various implementations, the eye pad is a plastic or rubber piece that comfortably and snugly holds the headset 1000 in place on the face of the user 25 (e.g., around the eyes of the user 25).

The housing 1001 houses a display 1010 that displays images, emits light toward or onto the eyes of a user 25. In various implementations, the display 1010 emits light through an eyepiece having one or more optical elements 1005 that refract the light emitted by the display 1010, causing the display to appear to the user 25 as a virtual distance greater than the actual distance from the eye to the display 1010. For example, the optical element 1005 may include one or more lenses, waveguides, other Diffractive Optical Elements (DOEs), and the like. To enable the user 25 to focus on the display 1010, in various implementations, the virtual distance is at least greater than a minimum focal length of the eye (e.g., 6 cm). Furthermore, in order to provide a better user experience, in various implementations, the virtual distance is greater than 1 meter.

The housing 1001 also houses a tracking system that includes one or more light sources 1022, a camera 1024, and a controller 1080. One or more light sources 1022 emit light onto the eyes of user 25 that is reflected as a pattern of light (e.g., a flash) that is detectable by camera 1024. Based on the light pattern, controller 1080 may determine an eye-tracking characteristic of user 25. For example, controller 1080 may determine a gaze direction and/or a blink status (open or closed) of user 25. As another example, controller 1080 may determine a pupil center, pupil size, or point of interest. Thus, in various implementations, light is emitted by the one or more light sources 1022, reflected from the eyes of the user 25, and detected by the camera 1024. In various implementations, light from the eyes of user 25 is reflected from a hot mirror or passed through an eyepiece before reaching camera 1024.

The display 1010 emits light in a first wavelength range and the one or more light sources 1022 emit light in a second wavelength range. Similarly, the camera 1024 detects light in the second wavelength range. In various implementations, the first wavelength range is a visible wavelength range (e.g., a wavelength range of approximately 400nm to 800nm in the visible spectrum) and the second wavelength range is a near infrared wavelength range (e.g., a wavelength range of approximately 700nm to 1400nm in the near infrared spectrum).

In various implementations, eye tracking (or in particular, a determined gaze direction) is used to enable a user to interact (e.g., user 25 selects it by viewing an option on display 1010), provide a rendering of holes (e.g., presenting higher resolution in the area of display 1010 that user 25 is viewing and lower resolution elsewhere on display 1010), or correct distortion (e.g., for images to be provided on display 1010).

In various implementations, one or more light sources 1022 emit light toward the eyes of user 25 that is reflected in the form of multiple flashes.

In various implementations, the camera 1024 is a frame/shutter based camera that generates images of the eyes of the user 25 at a particular point in time or points in time at a frame rate. Each image comprises a matrix of pixel values corresponding to pixels of the image, which pixels correspond to the positions of the photo sensor matrix of the camera. In implementations, each image is used to measure or track pupil dilation by measuring changes in pixel intensities associated with one or both of the user's pupils.

In various implementations, the camera 1024 is an event camera that includes a plurality of light sensors (e.g., a matrix of light sensors) at a plurality of respective locations that generates an event message indicating a particular location of a particular light sensor in response to the particular light sensor detecting a change in light intensity.

It should be understood that the implementations described above are cited by way of example, and that the present disclosure is not limited to what has been particularly shown and described hereinabove. Rather, the scope includes both combinations and subcombinations of the various features described hereinabove as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

As described above, one aspect of the present technology is to collect and use physiological data to improve the user's electronic device experience in interacting with electronic content. The present disclosure contemplates that in some cases, the collected data may include personal information data that uniquely identifies a particular person or that may be used to identify an interest, characteristic, or predisposition of a particular person. Such personal information data may include physiological data, demographic data, location-based data, telephone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

The present disclosure recognizes that the use of such personal information data in the present technology may be used to benefit users. For example, personal information data may be used to improve the interaction and control capabilities of the electronic device. Thus, the use of such personal information data enables planned control of the electronic device. In addition, the present disclosure contemplates other uses for personal information data that are beneficial to the user.

The present disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information and/or physiological data will adhere to established privacy policies and/or privacy practices. In particular, such entities should exercise and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. For example, personal information from a user should be collected for legal and legitimate uses of an entity and not shared or sold outside of those legal uses. In addition, such collection should be done only after the user's informed consent. In addition, such entities should take any required steps to secure and protect access to such personal information data and to ensure that other people who are able to access the personal information data adhere to their privacy policies and procedures. In addition, such entities may subject themselves to third party evaluations to prove compliance with widely accepted privacy policies and practices.

Regardless of the foregoing, the present disclosure also contemplates implementations in which a user selectively prevents use or access to personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to content delivery services customized for a user, the techniques of the present invention may be configured to allow the user to choose to "join" or "leave" to participate in the collection of personal information data during the registration service. In another example, the user may choose not to provide personal information data for the targeted content delivery service. In yet another example, the user may choose not to provide personal information, but allow anonymous information to be transmitted for improved functionality of the device.

Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that the various embodiments may be implemented without accessing such personal information data. That is, various embodiments of the present technology do not fail to function properly due to the lack of all or a portion of such personal information data. For example, the content may be selected and delivered to the user by inferring preferences or settings based on non-personal information data or absolute minimum personal information such as content requested by a device associated with the user, other non-personal information available to the content delivery service, or publicly available information.

In some embodiments, the data is stored using a public/private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, the data may be stored anonymously (e.g., without identifying and/or personal information about the user, such as legal name, user name, time and location data, etc.). Thus, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user may access stored data from a user device other than the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access their stored data.

In general, one innovative exemplary embodiment of the subject matter described in this specification can be embodied in methods of: at a device comprising a processor, the methods include acts of: presenting the content to a user; obtaining physiological data associated with the user via one or more sensors; determining a respiration state of the user based on the physiological data; determining an attention state of the user based on the physiological data; determining a modification to the content based on the breathing state and the attention state of the user; and presenting the modified content to the user.

These exemplary embodiments and other exemplary embodiments can each optionally include one or more of the following features.

In some embodiments, the method further comprises: presenting instructions to the user to notice breathing; and evaluating an attention level to respiration based on the respiration state and the attention state, wherein the modification is determined based on the attention level.

In some embodiments, determining the modification comprises: determining a baseline corresponding to the user based on the respiratory state and the attention state; determining a target for the user based on the baseline; and determining the modification based on the baseline and the target.

In some embodiments, the modification comprises: initiation of new meditation; conclusions of meditation; or an ongoing meditation change. In some embodiments, the modification comprises: a visual or audible representation of the respiratory state or a change in the respiratory state. In some exemplary embodiments, the modification comprises: a visual or audible representation of the breathing state; and a visual or audible representation of the state of attention. In some exemplary embodiments, the modification comprises: a cue configured to trigger a change in the respiratory state or the attention state. In some embodiments, the modification comprises: a graphical indication logo or sound configured to change the attention state to a different attention state.

In some embodiments, the modification comprises: visual or audible indication of the proposed time of experience.

In some embodiments, the respiratory state is determined based on using physiological data to determine head pose, sound, jaw motion, cheek motion, nose motion, motion of tissue surrounding the eyes, or facial signals modulated by respiration.

In some embodiments, the state of attention is determined based on measuring gaze or body stability using the physiological data. In some aspects, the attention state is determined based on determining the attention level. In some aspects, the attention state is determined based on one or more physiological data streams via the output of one or more sensors.

In some embodiments, the method further comprises: identifying an emotional state of the user, the emotional state corresponding to a plurality of time periods; and presenting an indication of progress based on the emotional state.

In some embodiments, the content includes an augmented reality (XR) experience. In some embodiments, the content is presented to a plurality of users during a communication session. In some embodiments, the device is or includes a Head Mounted Device (HMD).

Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, it will be understood by those skilled in the art that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known by those of ordinary skill have not been described in detail so as not to obscure the claimed subject matter.

Unless specifically stated otherwise, it is appreciated that throughout the description, discussions utilizing terms such as "processing," "computing," "calculating," "determining," or "identifying" or the like, refer to the action or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical, electronic, or magnetic quantities within the computing platform's memory, registers, or other information storage device, transmission device, or display device.

The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. The computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include a multi-purpose microprocessor-based computer system that accesses stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more implementations of the subject invention. The teachings contained herein may be implemented in software for programming or configuring a computing device using any suitable programming, scripting, or other type of language or combination of languages.

Implementations of the methods disclosed herein may be performed in the operation of such computing devices. The order of the blocks presented in the above examples may be varied, e.g., the blocks may be reordered, combined, or divided into sub-blocks. Some blocks or processes may be performed in parallel.

The use of "adapted" or "configured to" herein is meant to be an open and inclusive language that does not exclude devices adapted or configured to perform additional tasks or steps. In addition, the use of "based on" is intended to be open and inclusive in that a process, step, calculation, or other action "based on" one or more of the stated conditions or values may be based on additional conditions or beyond the stated values in practice. Headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

It will also be understood that, although the terms "first," "second," etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, so long as all occurrences of "first node" are renamed consistently and all occurrences of "second node" are renamed consistently. The first node and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the claims. As used in the description of this specification and the appended claims, the singular forms "a," "an," and "the" are intended to cover the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof.

As used herein, the term "if" may be interpreted to mean "when the prerequisite is true" or "in response to a determination" or "upon a determination" or "in response to detecting" that the prerequisite is true, depending on the context. Similarly, the phrase "if it is determined that the prerequisite is true" or "if it is true" or "when it is true" is interpreted to mean "when it is determined that the prerequisite is true" or "in response to a determination" or "upon determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection that the prerequisite is true", depending on the context.

The foregoing description and summary of the invention should be understood to be in every respect illustrative and exemplary, but not limiting, and the scope of the invention disclosed herein is to be determined not by the detailed description of illustrative implementations, but by the full breadth permitted by the patent laws. It is to be understood that the specific implementations shown and described herein are merely illustrative of the principles of this invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims (55)

1. A method, the method comprising:

at a device comprising a processor:

obtaining physiological data via one or more sensors;

determining an attention state during meditation mode based on the physiological data;

customizing feedback content for guiding a change of the attention state during the meditation mode, wherein the feedback content is customized based on user attributes; and

after a delay time, customized feedback content is provided during the meditation mode based on the user attributes.

2. The method of claim 1, the method further comprising:

determining an attention level over a period of time based on the physiological data in response to providing the customized feedback content; and

A feedback metric is determined based on the determined level of attention over the period of time.

3. The method according to any one of claims 1 or 2, wherein the user attribute comprises meditation experience level.

4. A method according to claim 3, wherein the meditation experience level is determined based on accessing a user profile.

5. A method according to claim 3, wherein the meditation experience level is determined based on an analysis of previous meditation experiences.

6. The method according to any one of claims 3 to 5, wherein the meditation experience level is updated based on the attention state during the meditation mode.

7. The method of any one of claims 1 to 6, further comprising:

presenting instructions to notice breathing; and

the level of attention to breathing is evaluated based on the breathing state and the attention state,

wherein the customized feedback is determined based on the level of attention.

8. The method of any of claims 1-7, wherein customizing the feedback content comprises:

determining a baseline based on the physiological data;

determining a target based on the baseline; and

customized feedback content is determined based on the baseline and the target.

9. The method of any one of claims 1 to 8, the method further comprising:

identifying a meditation state, the meditation state corresponding to a plurality of time periods; and

an indication of progress is presented based on the meditation status.

10. The method of any one of claims 1 to 9, the method further comprising:

a portion of the meditation patterns is identified, the portion being associated with a particular attention state.

11. The method of any one of claims 1 to 10, the method further comprising:

determining a context of the meditation pattern based on sensor data of an environment of the meditation pattern; and

the feedback content is customized based on the context of the meditation patterns.

12. The method of claim 11, wherein determining the context of the experience comprises:

generating a scene understanding of the environment based on the sensor data of the environment, the scene understanding including a visual attribute or an auditory attribute of the environment; and

the context of the experience is determined based on the scene understanding of the environment.

13. The method of any of claims 11 or 12, wherein determining the context of the experience comprises: the activity is determined based on a schedule.

14. The method of any of claims 1 to 13, wherein the customized feedback content comprises:

conclusion of current meditation pattern;

the initiation of another meditation pattern than the current meditation pattern; or (b)

A change of the current meditation pattern.

15. The method of any of claims 1 to 14, wherein the customized feedback content comprises: volume of the audio signal modulated based on the physiological data.

16. The method of any of claims 1 to 15, wherein the customized feedback content comprises: a visual or audible representation of the state of attention, or a change in the state of attention.

17. The method of any of claims 1 to 16, wherein the customized feedback content comprises: a cue configured to trigger a change in the attention state.

18. The method of any of claims 1 to 17, wherein the customized feedback content comprises: a graphical indication logo or sound configured to change a first attention state to a second attention state.

19. The method of any of claims 1 to 18, wherein the customized feedback content comprises: visual or audible indication of suggested time for the new meditation experience.

20. The method of any one of claims 1 to 19, wherein the attention state is determined based on measuring gaze or body stability using the physiological data.

21. The method of any one of claims 1 to 20, wherein the attention state is determined based on determining an attention level.

22. The method of any one of claims 1 to 21, wherein the attention state is determined based on a respiratory state.

23. The method of any one of claims 1 to 22, wherein the attention state is assessed using statistical or machine learning based classification techniques.

24. The method of any one of claims 1 to 23, wherein the physiological data comprises at least one of skin temperature, respiration, photoplethysmography (PPG), electrodermal activity (EDA), eye gaze tracking, and pupil movement.

25. The method of any one of claims 1 to 24, wherein the meditation-mode environment comprises an augmented reality (XR) environment.

26. The method according to any one of claims 1 to 25, wherein the meditation pattern is presented in a plurality of instances during a communication session.

27. The method of any one of claims 1 to 26, wherein the device is a Head Mounted Device (HMD).

28. An apparatus, the apparatus comprising:

a non-transitory computer readable storage medium; and

one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:

obtaining physiological data via one or more sensors;

determining an attention state during meditation mode based on the physiological data;

customizing feedback content for guiding a change of the attention state during the meditation mode, wherein the feedback content is customized based on user attributes; and

after a delay time, customized feedback content is provided during the meditation mode based on the user attributes.

29. The apparatus of claim 28, the apparatus further comprising:

determining an attention level over a period of time based on the physiological data in response to providing the customized feedback content; and

a feedback metric is determined based on the determined level of attention over the period of time.

30. The apparatus according to any one of claims 28 or 29, wherein the user attribute comprises meditation experience level.

31. The apparatus of claim 30, wherein the meditation experience level is determined based on accessing a user profile.

32. The apparatus of claim 30, wherein the meditation experience level is determined based on an analysis of previous meditation experiences.

33. The apparatus according to any one of claims 28 to 32, wherein the meditation experience level is updated based on the attention state during the meditation mode.

34. The apparatus of any of claims 28-33, wherein the non-transitory computer-readable storage medium comprises additional program instructions that, when executed on the one or more processors, further cause the system to perform operations comprising:

presenting instructions to notice breathing; and

the level of attention to breathing is evaluated based on the breathing state and the attention state,

wherein the customized feedback is determined based on the level of attention.

35. The apparatus of any of claims 28 to 34, wherein customizing the feedback content comprises:

Determining a baseline based on the physiological data;

determining a target based on the baseline; and

customized feedback content is determined based on the baseline and the target.

36. The apparatus of any of claims 28-35, wherein the non-transitory computer-readable storage medium comprises additional program instructions that, when executed on the one or more processors, further cause the system to perform operations comprising:

identifying a meditation state, the meditation state corresponding to a plurality of time periods; and

an indication of progress is presented based on the meditation status.

37. The apparatus of any of claims 28-36, wherein the non-transitory computer-readable storage medium comprises additional program instructions that, when executed on the one or more processors, further cause the system to perform operations comprising:

a portion of the meditation patterns is identified, the portion being associated with a particular attention state.

38. The apparatus of any of claims 28-37, wherein the non-transitory computer-readable storage medium comprises additional program instructions that, when executed on the one or more processors, further cause the system to perform operations comprising:

Determining a context of the meditation pattern based on sensor data of an environment of the meditation pattern; and

the feedback content is customized based on the context of the meditation patterns.

39. The apparatus of claim 38, wherein determining the context of the experience comprises:

generating a scene understanding of the environment based on the sensor data of the environment, the scene understanding including a visual attribute or an auditory attribute of the environment; and

the context of the experience is determined based on the scene understanding of the environment.

40. The apparatus of any of claims 38 or 39, wherein determining the context of the experience comprises: the activity is determined based on a schedule.

41. The device of any of claims 28 to 40, wherein the customized feedback content comprises:

conclusion of current meditation pattern;

the initiation of another meditation pattern than the current meditation pattern; or (b)

A change of the current meditation pattern.

42. The apparatus of any of claims 28 to 41, wherein the customized feedback content comprises: volume of the audio signal modulated based on the physiological data.

43. The apparatus of any of claims 28 to 42, wherein the customized feedback content comprises: a visual or audible representation of the state of attention, or a change in the state of attention.

44. The apparatus of any of claims 28 to 43, wherein the customized feedback content comprises: a cue configured to trigger a change in the attention state.

45. The apparatus of any of claims 28 to 44, wherein the customized feedback content comprises: a graphical indication logo or sound configured to change a first attention state to a second attention state.

46. The device of any of claims 28-45, wherein the customized feedback content comprises: visual or audible indication of suggested time for the new meditation experience.

47. The device of any of claims 28 to 46, wherein the attention state is determined based on measuring gaze or physical stability using the physiological data.

48. The device of any one of claims 28 to 47, wherein the attention state is determined based on determining an attention level.

49. The device of any of claims 28-48, wherein the attention state is determined based on a respiratory state.

50. The apparatus of any one of claims 28 to 49, wherein the attention state is assessed using statistical or machine learning based classification techniques.

51. The device of any one of claims 28 to 50, wherein the physiological data comprises at least one of skin temperature, respiration, photoplethysmography (PPG), electrodermal activity (EDA), eye gaze tracking, and pupil movement.

52. The apparatus of any one of claims 28 to 51, wherein the meditation mode environment comprises an augmented reality (XR) environment.

53. The apparatus according to any one of claims 28 to 52, wherein the meditation pattern is presented in a plurality of instances during a communication session.

54. The device of any one of claims 28-53, wherein the device is a Head Mounted Device (HMD).

55. A non-transitory computer readable storage medium storing program instructions executable on a device to perform any one of the methods of claims 1-27.