TW202332408A - In-ear motion sensors for ar/vr applications and devices - Google Patents

Abstract

An in-ear device for immersive reality applications is provided. The device includes an in-ear fixture configured to fit in an ear canal of a user, a motion sensor mounted on the in-ear fixture and configured to provide a motion signal indicative of an inner body motion or a bulk body motion of the user, a speaker coupled to provide an audio signal to the user, and a processor that is coupled to an augmented reality headset, the processor configured to identify a health condition of the user based on the motion signal. A method for using the above device, a memory and a processor for storing and executing instructions to perform the method are also provided.

Description

In-ear motion sensors for AR/VR applications and devices

This disclosure relates to in-ear motion sensors for use in virtual reality and augmented reality environments and devices. More specifically, the present disclosure relates to motion sensors configured to monitor motion from within the body of a user of an in-ear device for immersive reality applications. Cross-references to related applications

This disclosure relates to and is claimed under 35 U.S.C. § 119(e), U.S. Provisional Application No. 63, filed on February 2, 2022, entitled In-Ear Biosensing for AR/VR Applications and Devices /305,932, and all of the following are U.S. Provisional Application No. 63/356,851, titled In-ear Electrodes for AR/VR Applications and Devices, filed on June 29, 2022, titled In-ear Electrodes for AR/VR Applications U.S. Provisional Application No. 63/356,860 for an in-ear optical sensor for AR/VR applications and devices, U.S. Provisional Application No. 63/356,864 for an in-ear motion sensor for AR/VR applications and devices, U.S. Provisional Application No. 63/356,872 for In-Ear Temperature Sensors for AR/VR Applications and Devices, U.S. Provisional Application No. 63/356,877 for In-Ear Microphones for AR/VR Applications and Devices, Priority is granted to U.S. Provisional Application No. 63/356,883 entitled In-Ear Sensors for AR/VR Applications and Devices and Methods of Using the Same, all inventions of Morteza KHALEGHIMEYBODI et al. This disclosure is also related to and claims priority to U.S. Non-Provisional Application No. 18/069,065, filed on December 20, 2022. The contents of the above application are hereby incorporated by reference in their entirety for all purposes.

In-ear devices currently used for mobile and immersive applications (eg, hearing aids, smart headphones, headphones, earphones, and the like) are often bulky and uncomfortable for users. Adding health-sensing capabilities to in-ear devices is hampered by the small form factor expected in such devices and the complex data processing and analysis involved.

In a first embodiment, a device includes an in-ear fixture configured to fit within an ear canal of a user, a motion sensor mounted on the in-ear fixture, and configured to provide a motion signal indicative of an internal movement or a whole body movement of the user, a speaker coupled to provide an audio signal to the user, and a processor coupled to a In an augmented reality head-mounted device, the processor is configured to identify a health condition of the user based on the motion signal.

In a second embodiment, a computer-implemented method includes receiving, from a motion sensor in an in-ear device, a motion indicative of an internal motion or a whole-body motion of a user of the in-ear device. signal, utilizing the motion signal to form a waveform, and identifying a health condition of the user based on the waveform.

In a third embodiment, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause a computer to perform a method. The method includes receiving a motion signal from a motion sensor in an in-ear device indicative of an internal motion or a whole-body motion of a user of the in-ear device, and utilizing the motion signal to form a waveform. , and identifying a health condition of the user according to the waveform.

In still other embodiments, a system includes a first device for storing instructions and a second device for executing the instructions and causing the system to perform a method. The method includes receiving a motion signal from a motion sensor in an in-ear device indicative of an internal motion or a whole-body motion of a user of the in-ear device, and utilizing the motion signal to form a waveform. , and identifying a health condition of the user according to the waveform.

These and other embodiments will become apparent to one of ordinary skill in the art upon consideration of the following.

In the following detailed description, numerous specific details are set forth to provide a complete understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that embodiments of the present disclosure may be practiced without some of these specific details. In other instances, well-known structures and techniques have not been shown in detail so as not to obscure the present disclosure. General overview

Head-mounted devices (eg, devices worn on the head, including but not limited to smart headphones, smart glasses, AR/VR head-mounted devices, smart glasses, etc.) provide opportunities to obtain valuable health information.

The ears (e.g., the ear canal and concha and pinna) are in close proximity to the brain, body chemistry, and blood vessels that dictate brain and cardiopulmonary activity and body temperature. More specifically, sensors can be placed in or around the ear canal (in the case of AR/VR headsets or smart glasses) to sense brain, heart, and eye electrophysiology. Activity (e.g., electroencephalogram EEG, electrocardiogram ECG, electrooculogram EOG, electrodermal activity EDA, and the like); or sensing vital signs (heart rate, respiratory rate, blood pressure, body temperature, and the like); or Sensing body chemistry (e.g., blood alcohol concentration, blood glucose estimates, and the like).

Motion sensors, optical sensors, microphones, thermometers, and electrodes as in the embodiments disclosed herein can be used in EOG, ECG, or EEG measurements, for example, for determining auditory attention; heart rate estimation , respiratory rate and similar ones, auditory steady-state response ASSR, or auditory brainstem response ABR. In certain embodiments, in-ear electrodes such as those disclosed herein can help measure resting state electrical oscillations (alpha waves in EEG), which can track relaxation/activity. In combination with other measurements, such as photoplethysmography PPG and/or electrocardiography ECG, it opens a new branch of diagnostic possibilities. In-ear EEG measurements can be used to track user attention (e.g., distinguish focus of attention and eye gaze direction).

The methods and devices disclosed herein include motion, optical and acoustic sensors, chemical sensors, and temperature sensors in and around the ears of an AR/VR headset user, combined with the above sensing Software correlation of signals provided by the device to produce a comprehensive diagnosis and health assessment of the user.

Some of the features disclosed here include in-ear or head-mounted body temperature sensing that utilizes infrared sensing and spectroscopy technology. In some embodiments, the contact area for a sensor as disclosed herein includes the inlet canal (such as an in-ear earbud) and within the auricular basin (within the pinna of the human ear), the top of the human ear area (where the glasses sit), and the nose pad area of the headset or smart glasses (where the glasses sit on the nose). Some measurements may include blood sugar levels, alcohol sensing, body temperature, blood pressure, and the like in or around the ears. Certain embodiments include optically based pulse transit time (PTT) methods for estimation using a combination of optical and electrical signals (e.g., PPG+ECG sensors, respectively) for glasses/head-mounted devices. blood pressure. Some embodiments utilize an optical sensing technology (PPG) combined with a deep neural network to train a network to obtain the user's blood pressure based on both PPG information and a corresponding ground truth blood pressure information. Certain embodiments include motion-based pulse transit time (PTT) for glasses/head-mounted devices that utilize a combination of motion sensors and electrical signals (e.g., IMU+ECG sensors, respectively) Methods to estimate blood pressure. In certain embodiments, PPG signals collected in IEM devices such as those disclosed herein may be capable of estimating the cognitive load on the user utilizing oxygenated and deoxygenated blood flow to the brain (oxygenated and deoxygenated blood flow). Heme) analysis. Some embodiments include sensing alcohol concentration through dispersion around the ear. Certain embodiments incorporate chemical sensing inlets around the contact points of the ear. In certain embodiments, IEM devices can perform alcohol monitoring and fat burning during exercise of the user. Example system architecture

Figure 1 depicts an AR headset 110-1 and an in-ear monitor (IEM) 100 in an architecture 10 configured to assess the health of a user 101, according to certain embodiments. The IEM 100 is inserted into the ear 170 of the user 101 to reach the ear canal 161 . AR head-mounted device 110-1 may include smart glasses having a memory circuit 120 that stores instructions and a processor circuit 112 that is configured to execute the instructions to perform steps in the methods disclosed herein. The AR head-mounted device 110-1 (or smart glasses) may also include a communication module 118 configured to communicate between the AR head-mounted device 110-1 (and/or the in-ear device 100, and/or the smart glasses). watch, or a combination of the above) and the mobile device 110-2 with the user (the AR head-mounted device 110-1 and the mobile device 110-2 will be collectively referred to as the "client device 110" below) wirelessly transmit information between the The communication module 118 may be configured to interface with a network 150 to transmit and receive information such as data group 103-1, data group 103-2, and data group 103-3, requests, responses, and commands to the network. Other devices on 150. In some embodiments, the communication module 118 may include a modem or an Ethernet card. Client device 110 can then be communicatively coupled to a remote server 130 and a database 152 via network 150 and send/share information, files, and the like (e.g., data set 103-2 and data) with each other. Group 103-3). Data sets 103-1, 103-2, and 103-3 will collectively be referred to as "data set 103" below. Network 150 may include, for example, any one or more of a local area network (LAN), a wide area network (WAN), the Internet, and the like. Furthermore, the network may include, but is not limited to, any one or more of the following network topologies, including bus network, star network, ring network, mesh network, and star network. Bus networks, tree or hierarchical networks, and the like.

In some embodiments, at least one of the steps of a method as disclosed herein is performed by processor 112, which provides data set 103-1 to mobile device 110-2. Mobile device 110-2 can further process the signal and provide data set 103-2 to database 152 via network 150. The remote server 130 may collect the data set 103-2 in the form from multiple AR head-mounted devices 110-1 and mobile devices 110-2 and perform further calculations. Additionally, by aggregating data from a group of individuals, the remote server can perform meaningful statistics. This data loop can be established if each user involved agrees to the use of non-personalized or anonymous data. In some embodiments, the remote server 130 and database 152 may be connected via a healthcare network, or a healthcare facility or institution (e.g., hospital, university, government agency, clinic, health insurance network, and similar) to manage. The mobile device 110-2, the AR headset 110-1, the in-ear device 100, and the applications therein may be provided by a different service provider (e.g., network operator, application developer, and the like). ) to manage. Furthermore, the AR head-mounted device 110-1 and the mobile device 110-2 may be from different manufacturers. User 101 is ultimately the sole owner of data set 103-1 and all data derived therefrom (e.g., data set 103). Therefore, all data streams (e.g., data set 103) are provided and processed by different entities. , or moderation, but authorized by user 101 and protected by network 150, server 130, database 152, and mobile device 110-2 for privacy and security.

Figure 2 depicts an augmented reality ecosystem 200 including a wearable device 205-1 (eg, an IEM) in the ear, a wearable device 205-2 on the wrist, a wearable device on the chest, in accordance with certain embodiments. The wearable device 205-3 and the smart glasses sensor 205-4 are used to assess the health of the user 201. In some embodiments, IEM 205-1 further includes an optical sensor configured to provide an optical signal 220-1 to a processor in a computer 240 via a data acquisition module (DAQ) 230 . IEM 205-1 may further include one or more contact electrodes configured to provide an electrical signal to a processor in a computer 240 via a data acquisition module (DAQ) 230. The computer 240 is configured to identify the cardiovascular condition of the user 201 based on a first electronic signal from the IEM 205-1 and the optical signal 220-1. In some embodiments, IEM 205-1 further includes a motion sensor (eg, accelerometer, contact microphone, or IMU) configured to provide a motion-based signal to the computer via DAQ 230 240. In some embodiments, a pair of IEMs 205 will be placed in each ear and have different optical, electrical (electrodes), sonic (microphones), or motion sensors (accelerometers, IMUs, contacts). microphones, etc.) may be placed in both sides; or in some cases, some sensors may be placed on one side (e.g., the right side) and some other sensors may be placed Place it on the other side (for example, the left side). Computer 240 is configured to identify the cardiovascular condition of the user based on a first electronic signal from IEM 205-1 and the motion signal. The optical sensor may be a photoplethysmography (PPG) sensor, and the optical signal 220 - 1 may include a digital or analog signal indicating the activity of blood vessels in the ear of the user 201 . The chest sensor 205-3 and the smart glasses sensor 205-4 may include ECG sensors to provide information from one or more areas around the chest and face of the user 201 (eg, at the ears, cheeks, and A dispersed signal 220-3 and 220-4 (or an ECG) from the outside of the nose) may be from certain electrodes placed on the area of the head, or from the IEM 205-1 electrodes, or collected by electrodes placed on the wrist device 205-2), and a wrist PPG sensor in the device 205-2 can target the activity of blood vessels around the wrist of the user 201 Provide a separate signal 220-2. IEM 205-1, wrist sensor 205-2, chest sensor 205-3, and smart glasses sensor 205-4 will be collectively referred to as "wearable devices (and sensors)" below. 205". Blood pressure (BP) measurement can be obtained using a tourniquet or pressure-free sphygmomanometer 210, and can also be determined by comparing PPG signals 220-1 and 220-2. Signals 220-1, 220-2, 220-3, and 220-4 (hereinafter collectively referred to as "signals 220") may be collected by DAQ 230 in computer 240 and digitized for processing. In some embodiments, signal 220 and other signals may be wired or wireless. In some embodiments, it may be preferable to have wireless signal communication between different wearable devices 205 of the user 201 . In some embodiments, wearable devices and sensors 205 may include one or more motion sensors, and motion-based information collected from the smart glasses, the IEM, chest, or wrist may be combined to produce more meaningful information.

3A-3C depict different embodiments of an in-ear monitor (IEM) 300A, 300B, and 300C (hereinafter collectively referred to as "IEM 300") in accordance with certain embodiments. IEM 300 may include a front end 301-1 that includes sensors and opens to the ear canal 361 and eardrum 362, and a back end 301-2 that includes a processor 312. The IEM 300 may include sensors, such as: an electrode 305 for sensing electrical signals, acoustic wave sensors 325-1 and 325-2 (for example, collectively referred to as "microphone 325" below), motion sensors 327 (e.g., accelerometer, contact microphone, inertial motion unit IMU, and the like), temperature sensor 329, and optical sensor including an emitter 321 and a detector 323 (e.g., based on Fourier transform, spectrum-based PPG sensors, LEDs and PDs in functional near-infrared fNIR sensors). Electrodes 305 may include biopotential electrodes for applications such as EEG, ECG, EOG, and EDA. Additionally, the processor 312 may process components and sensors 321, 323, 324 (a speaker), 325-1 (an internal At least some of the signal acquisition and control operations of microphones), 325-2 (external microphone, collectively referred to as "microphone 325" below), 327, and 329. The processor 312 may include a feedforward stage 311ff and a feedback stage 311fb that cooperate to process the signal from the sensor: noise reduction, balancing, filtering, and amplification.

In some embodiments, electrode 305 includes a contact electrode configured to send an electrical current from the skin within the user's ear canal. In some embodiments, an electrode 305 is coated with at least one of a gold layer, a silver layer, a silver chloride layer, or a combination thereof. In some embodiments, electrode 305 includes a capacitively coupled electrode that is positioned close enough to the user's skin, but does not touch. In some embodiments, the IEM 300 further includes at least one second electrode 305 mounted on the in-ear fixture 340 , the second electrode 305 being configured to receive a second electrical signal from the skin within the ear canal 361 . In some cases, the in-ear fixture 340 (also known as an "earbud") may be manufactured entirely from soft conductive materials (eg, conductive polymers, conductive adhesives, conductive paint, etc.). The entire earplug will then be conductive and will therefore act as a soft electrode to collect electrical signals from the skin of the ear canal. In some embodiments, the processor 312 is configured to select the first electronic signal when a quality of the first electronic signal is above a preselected threshold. In some embodiments, processor 312 is configured to utilize the second electronic signal to reduce a noise background from the first electronic signal. In some embodiments, the processor 312 is configured to determine the user's heart rate from the first electronic signal. In some embodiments, the processor 312 is configured to determine brain activity from the first electronic signal corresponding to the acoustic stimulation received in the external microphone.

In some embodiments, emitter 321 and detector 323 may be part of a self-mixing interferometer (SMI). SMIs are small, low-power, inexpensive, and sensitive optical elements configured to measure reflectance, including backscatter, and displacement of the skin. In some embodiments, skin displacements obtained using SMI are combined with heart rate measurements (eg, from PPG sensors, motion sensors, or ECG electrodes) to measure blood pressure and heart rate, or even eardrum vibrations. Also functions as an internal microphone.

The IEM 300 in an AR headset or smart glasses may include an in-ear fixture 340 configured to hermetically seal an ear canal of a user, a first electrode 305 mounted on the in-ear Fixation device 340 is mounted on and configured to receive a first electrical signal from the skin within ear canal 361 , and an internal microphone 325 - 1 is coupled to receive an internal acoustic signal propagating through ear canal 361 . The acoustic front-end includes an internal microphone 325-1 configured to detect sound waves (x _BC (t)) that propagate through the ear canal 361 and are generated internally by the body (e.g., heart rate is at approximately <100 Hz, respiratory rate is at About 50-1000Hz, and other sounds in the throat cavity). An external microphone 325-2 is coupled to receive an external acoustic signal x(t) propagated through the user's environment. In some embodiments, the internal signal x _BC (t) combined with the external signal x (t) can be used in acoustic applications, such as audio streaming, external sound, and active noise cancellation (ANC). , auditory correction, virtual presence and spatial audio, calling services, and the like. In some embodiments, at least some of the above procedures are performed jointly between the left and right ear IEM monitors 300. In some embodiments, the internal signal x _BC (t) may be captured using a motion sensor placed within the in-ear device. In certain other embodiments, a combination of motion sensor 327 and acoustic wave sensors or microphones 325-1 and 325-2 may be utilized to collect bimodal mechanical and acoustic turbulence driven by the heart beat.

The IEM 300B includes a sealing gasket 341 that separates the interior portion of the ear canal 361 from the environment, leaving a return port containing an acoustic barrier 344 for a pressure equalization (PEQ) tube 342 to discharge to the acoustic barrier. 344 (also shown in IEM 300C). The sealed cavity may enable monitoring of breathing rate and heart rate at low power use and in a small form factor (eg, isolating the signal from the internal acoustic microphone 325-1).

IEM 300C depicts processor circuit 312 to identify the user's cardiovascular condition or condition based on at least one of a first electronic signal, an internal acoustic signal, and an external acoustic signal (eg, from microphone 325). Neurological conditions. Some embodiments may include a down cable 345 to electrically couple the IEM and the VR headset or smart glasses, which includes a strain relief 343 .

4 is a diagram illustrating an ECG provided by an in-ear electrode or any electrode disposed on a wearable device (eg, a smart watch, or wristband, or the like) in accordance with certain embodiments. Signal 415 overlaps a motion waveform 410. ECG signal 415 provides a reference time for the onset of a heart pulse from which a systolic portion 405 and a diastolic portion 407 of motion waveform 410 can be identified. Thus, a length of time 417 between the initial electronic pulse 415 and the diastolic portion 407 may be indicative of, or have a direct correlation 420 with, the user's blood pressure 401 . Other correlation factors used to identify the user's blood pressure may include the ratio 402 of the amplitude of the systolic portion 405 relative to the diastolic portion 407 .

In some embodiments, the spectral signature of the systolic portion 405 and/or the diastolic portion 407 may also indicate the user's vital signs. It is generally observed that the contractile portion 405 contains a narrower bandwidth, while the diastolic portion 407 has a wider bandwidth.

Figure 5 is a diagram 500 depicting a motion waveform 510 obtained using a motion sensor in an IEM to determine a user's heart rate, according to certain embodiments. The chart 500 includes a time abscissa 501 and a signal value 502 . Motion waveform 510 may include raw signals from the sensor (e.g., when a ID accelerometer or contact microphone is used); or from a processing pipeline that combines motion from multiple axes (e.g., 3D axis acceleration (or the raw signal when a 6-DOF or 9-DOF IMU is used). A baseline truth ECG 515 is one that contains R-peak 517 .

In addition to heart rate and breathing rate, other measurements available from motion waveform 510 may include, but are not limited to, step count, posture estimation, and fall detection. Furthermore, some embodiments utilize the pulse transit time (PTT) technique to combine motion waveform 510 with an ECG signal 515 to enable blood pressure estimation.

In some embodiments, the motion waveform 510 can capture body-borne infrasound and low-frequency vibrations related to the user's vital signs. Signal processing techniques combined with artificial intelligence (AI) processing may be utilized to extract the user's vital signs (eg, heart rate, heart rate variability, respiratory rate, and blood pressure) from the motion waveform 510.

6 is a flowchart depicting steps in a method 600 of utilizing electrodes in an in-ear monitor to assess the health of a user of a head-mounted device or smart glasses, in accordance with certain embodiments. In some embodiments, at least one or more of the steps in method 600 may be performed by a processor executing on a user's body part (eg, ear, head, arm, wrist, foot). , ankles, fingers, toes, knees, shoulders, chest, back, and the like) in smart glasses or other wearable devices. In some embodiments, at least one or more of the steps in method 600 may be performed by a processor executing instructions stored in a memory, wherein the processor or the memory Or both are part of a mobile device of the user, a remote server or a database, communicatively coupled to each other through a network (see, processors 112, 312, and memory 120, client device 110, server 130, database 152, and network 150). Furthermore, the mobile device, the smart glasses, and the wearable device can communicate via a wireless communication system and protocol (for example, communication module 118, radio, Wi-Fi, Bluetooth, near field communication (NFC) , and the like) to be communicatively coupled to each other. In some embodiments, a method consistent with the present disclosure may include one or more steps from method 600 being performed in any order, simultaneously, quasi-simultaneously, or with temporal overlap.

Step 602 includes receiving a motion signal from a motion sensor indicative of an integral internal motion or a whole-body motion of a user of the in-ear device. In some embodiments, the motion sensor is an in-ear device or attached to a different part of the user's body (e.g., wrist, shoulder, arm, foot, or ankle, or a combination thereof). any combination) of a wearable device. In some embodiments, step 602 includes receiving a second motion signal indicative of the in-body motion or the whole-body motion from a second motion sensor in a second in-ear device. The second in-ear device can be placed on the opposite side, enabling information to be captured from both ears in the left and right ear canals. In some embodiments, step 602 includes receiving an electronic signal from an electrode indicative of a cardiovascular activity of the user.

Step 604 includes utilizing the motion signal to form a waveform. In some embodiments, step 604 includes utilizing the second motion signal to remove at least one of a noise component or an interference from the motion signal. A motion sensor in the in-ear device is configured to provide a motion-based signal to the processor, where the motion waveform may be a raw signal directly from the sensor (e.g., when 1D accelerometers or contact microphones are used), or raw signals from a processing pipeline that combines motion from multiple axes (e.g., 3D axis accelerometers or when 6 degrees of freedom (DOF) or 9 DOF of IMU is used).

Step 606 includes identifying a health condition of the user based on the waveform. In some embodiments, the motion signal is indicative of intrabody motion, and step 606 includes determining at least one of a heart rate and a breathing rate of the user. In some embodiments, the motion signal is indicative of a full-body motion of the user, and step 606 includes detecting a fall of the user, a cough of the user, a sneeze of the user, or It is to determine at least one of the number of steps of the user. In some embodiments, step 606 further includes identifying a systolic heart pulse and a diastolic heart pulse from the waveform, and identifying features in the systolic heart pulse and the diastolic heart pulse. In some embodiments, step 606 includes identifying a systolic heart pulse and a diastolic heart pulse from the waveform, and determining the user based on an amplitude of the diastolic heart pulse and an amplitude of the systolic heart pulse. of a blood pressure. In some embodiments, step 606 includes identifying a systolic heart pulse and a diastolic heart pulse from the waveform, and determining the user's condition based on a time delay of the diastolic heart pulse relative to the systolic heart pulse. One blood pressure. In some embodiments, step 606 includes using the waveform to filter one of a noise component and an interference from an audio signal transmitted to a speaker mounted in the in-ear device, and utilizing the The speaker is used to provide the audio signal to the user. In some embodiments, step 606 includes identifying a correlation between the electronic signal and the waveform, and determining one of a heart rate, a breathing rate, or a blood pressure of the user based on the correlation. . In some embodiments, step 606 further includes performing a spectral analysis on the waveform to identify a p wave, a QRS complex, and a T wave complex in an electrocardiogram. Hardware overview

7 is a block diagram depicting an example computer system 700 with which head mounted devices and other client devices 110 and method 600 may be implemented in accordance with certain embodiments. In some features, computer system 700 may utilize hardware or a combination of software and hardware, be implemented on a dedicated server, be integrated into another entity, or be distributed across multiple entity. The computer system 700 may include a desktop computer, a laptop computer, a tablet computer, a tablet phone, a smart phone, a feature phone, a server computer, or others. A server computer can be remotely located in a data center, or stored locally.

Computer system 700 includes a bus 708 or other communication mechanism for communicating information, and a processor 702 (eg, processor 112) coupled to bus 708 for processing information. For example, the computer system 700 may be implemented using one or more processors 702. Processor 702 may be a general purpose microprocessor, a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable A programmable logic device (PLD), a controller, a state machine, gated logic, discrete hardware components, or any other suitable entity that can perform computations or other processing of information.

In addition to hardware, computer system 700 may include code that generates an execution environment for the computer programs in question, such as those that constitute processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof. One or more combined codes, which are stored in an embedded memory 704 (e.g., memory 120), such as a random access memory (RAM), a flash memory, a unique Read memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), a temporary register, a hard disk, a removable disk, a CD-ROM, a A DVD, or any other suitable storage device, is coupled to bus 708 for storing information and instructions for execution by processor 702 . The processor 702 and the memory 704 may be supplemented by special purpose logic circuits or incorporated into special purpose logic circuits.

The instructions may be stored in the memory 704 and implemented in one or more computer program products, such as one or more modules of a computer program encoded on a computer-readable medium. Instructions for executing by the computer system 700 or controlling the operation of the computer system 700 according to any method well known to those skilled in the art, including but not limited to computer languages, such as Data-oriented languages (e.g., SQL, dBase), systems languages (e.g., C, Objective-C, C++, assembly languages), architectural languages (e.g., Java, .NET), and application languages (e.g., PHP, Ruby , Perl, Python). Instructions can also be used in, for example, array languages, aspect-oriented languages, assembly languages, editing languages, command line interface languages, compiled languages, parallel languages, curly bracket languages, data flow languages, data structure languages, declarative languages, esoteric languages languages, extended languages, fourth generation languages, functional languages, interactive pattern languages, interpretive languages, iterative languages, list-based languages, small languages, logic-based languages, machine languages, macro languages, metaprogramming Languages, multi-paradigm languages, numerical analysis, non-English-based languages, object-oriented class-based languages, object-oriented prototype-based languages, offside-rule languages, procedural languages, reflective languages, rule-based languages, scripting languages, stack-based languages, synchronization languages, syntax processing languages, visual languages, wirth languages, and xml-based languages. Memory 704 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 702.

A computer program as discussed here does not necessarily correspond to a file in a file system. A program may be stored as part of a file that contains other programs or data (for example, one or more scripts stored in a markup language file), in a single file that is specific to the program in question , or in multiple coordinated files (for example, files that store one or more modules, subroutines, or code portions). A computer program may be configured to execute on one computer or on multiple computers that are located at one location or dispersed across multiple locations and interconnected by a communications network . The programs and logic flows described in this specification may be executed by one or more programmable processors, which execute one or more computer programs to perform functions by operating on input data and producing output.

The computer system 700 further includes a data storage device 706, such as a magnetic disk or an optical disk, coupled to the bus 708 for storing information and instructions. Computer system 700 may be coupled to various devices via input/output modules 710 . The input/output module 710 can be any input/output module. An example input/output module 710 includes a data port such as a USB port. The input/output module 710 is configured to connect to a communication module 712 . An example communication module 712 includes a networking interface card, such as an Ethernet card and a modem. In some features, the input/output module 710 is configured to connect to a plurality of devices, such as an input device 714 and/or an output device 716 . Example input devices 714 include a keyboard and a pointing device (eg, a mouse or a trackball) through which a consumer can provide input to the computer system 700 . Other types of input devices 714 may also be utilized to provide interaction with a consumer, such as a tactile input device, a visual input device, an audio input device, or a human-machine interface device. For example, the feedback provided to the consumer may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the consumer may be Receive in any form, including sonic, speech, tactile, or brainwave input. An example output device 716 includes a display device, such as an LCD (liquid crystal display) screen, for displaying information to the consumer.

According to one aspect of the present disclosure, the headset and client device 110 may utilize, at least in part, a computer system 700 in response to the processor 702 executing one or more sequences of one or more instructions contained in the memory 704 to implement. Such instructions may be read into memory 704 from another machine-readable medium, such as data storage device 706 . Execution of the sequence of instructions contained in primary memory 704 causes processor 702 to perform the program steps described herein. One or more processors in a multi-processing configuration may also be employed to execute sequences of instructions contained in main memory 704 . In alternative features, hardwired circuitry may be used in place of or in conjunction with software instructions to implement the various features of the present disclosure. Accordingly, features of the present disclosure are not limited to any specific combination of hardware circuitry and software.

Various features of the subject matter described in this specification may be implemented in a computing system that includes a back-end component such as a data server, or that includes a middleware component such as an application server, or Includes a front-end component, such as a client computer with a graphical consumer interface or a web browser, through which consumers can interact with implementations of the subject matter described in this specification, or one or more such Any combination of backend, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communications network. The communication network may include, for example, any one or more of a LAN, a WAN, the Internet, and the like. Furthermore, the communication network may include, but is not limited to, any one or more of the following network topologies, including bus network, star network, ring network, mesh network, star network. Bus network, tree or hierarchical network, or similar. The communication module may be a modem or an Ethernet card, for example.

Computer system 700 may include clients and servers. The client and server are generally remote from each other and usually interact through a communications network. The client and server relationship occurs by computer programs executing on said individual computers and thus having a client-server relationship with each other. Computer system 700 may be, for example and without limitation, a desktop computer, a laptop computer, or a tablet computer. The computer system 700 can also be embedded in another device, such as, but not limited to, a mobile phone, a PDA, a mobile audio player, a global positioning system (GPS) receiver, a video game console, and/or a television set-top box.

The terms "machine-readable storage medium" or "computer-readable medium" as used herein refer to any one or more media that participates in providing instructions to processor 702 for execution. Such media can take many forms, including, but not limited to, non-volatile media, volatile media, and delivery media. Non-volatile media include, for example, optical disks or magnetic disks, such as the data storage device 706 . Volatile media includes dynamic memory, such as memory 704. Transmission media includes coaxial cables, copper wires, and fiber optics, including the wires that form bus 708. Common forms of machine-readable media include floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes , any other physical media with a hole pattern, RAM, PROM, EPROM, flash EPROM, any other memory chip or cartridge, or any other computer-readable media. The machine-readable storage medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagated signal, or one of them. or a combination of multiple.

As used herein, the phrase "at least one of" following a list of items (with the terms "and" or "or" separating any of the items) modifies the list as a whole. , rather than listing each member of the table (e.g., each item). The phrase "at least one of" does not require a selection of at least one of the items; rather, the phrase allows for at least one including at least one of any of the items, and/or at least one of any combination of the items, and/or at least one of each of the items stated. For example, the term "at least one of A, B, and C" or "at least one of A, B, or C" means only A, only B, or only C; any of A, B, and C combination; and/or at least one of each of A, B and C.

The word "exemplary" is used herein to mean "as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. For example, a feature, said feature, another feature, certain features, one or more features, an embodiment, said embodiment, another embodiment, certain embodiments, one or more embodiments, an implementation Example, said embodiment, another embodiment, certain embodiments, one or more embodiments, a configuration, said configuration, another configuration, certain configurations, one or more configurations, subject technology, said The wording of the disclosure, this disclosure, other variations thereof, and the like is for convenience and does not imply that the disclosure with respect to such wording is important to the subject technology or that such disclosure is applicable All configurations of the subject technology described. Disclosures regarding such wording may apply to all configurations, or to one or more configurations. One or more examples may apply to the disclosure of such wording. Words such as a feature or certain features may refer to one or more features and vice versa, and the same applies similarly to other previous words.

Unless expressly stated otherwise, references to an element in the singular are not intended to mean "one and only one" but rather "one or more." Male pronouns (eg, his) are inclusive of the female and neuter genders (eg, hers and its), and vice versa. The term "certain" refers to one or more. Underlined and/or italicized headings and subheadings are used for convenience only, do not limit the subject technology, and are not intended to be relevant to the interpretation of the description of the subject technology. Relational terms such as first and second and the like may be used to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such distinction between such entities or actions. relationship or sequence. All known or hereafter known structural and functional equivalents of the various arrangements of elements described throughout this disclosure to those skilled in the art are expressly incorporated by reference and are intended to be Covered by the subject technology described. Furthermore, nothing disclosed here is intended to be a contribution to the general public, regardless of whether the content of such disclosure is explicitly stated in the above description. No element of a claim is to be construed under the sixth paragraph of 35 U.S.C. Section 112 unless the element is expressly set forth using the words "means for" or is a method claim. In the case of , the element is set forth using the phrase "the step of".

Although this specification contains many details, these should not be construed as limitations in what may be described, but rather as illustrations of specific embodiments of the subject matter described. In this specification, certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as operating in some combination, and even initially described as such, one or more features from a described combination may in some cases be deleted from the combination, Therefore, the combination may be a combination or a variation of a combination.

The subject matter of this specification has been described with respect to specific features, but other features may be implemented and are within the scope of the following claims. For example, although the operations in the drawings are depicted in a specific order, this should not be understood to require that such operations be performed in the specific order shown or to be performed in a sequential order, or that all depicted operations must be performed. Execution to achieve desired results. The actions set forth in the claim can be performed in a different order and still achieve the desired results. For example, the methods depicted in the accompanying drawings do not necessarily require the specific order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above features should not be understood to require such separation in all features, and it should be understood that the program components and systems generally can be integrated together in a single software product. , or packaged into multiple software products.

The names, background, figure brief descriptions, abstract, and figures are hereby incorporated into the present disclosure and are provided as illustrative examples of the present disclosure and not as limitations. It is understood that they will not be used to limit the scope or meaning of the claims. Furthermore, in the detailed description, it can be seen that the description provides illustrative examples and that various features are grouped together in various embodiments for the purpose of streamlining the present disclosure. The method of disclosure is not intended to be interpreted as reflecting an intention that the subject matter requires more features than are expressly set forth in each claim. Rather, as the claims reflect, inventive subject matter lies in less than all features of a single disclosed arrangement or operation. Said claims are hereby incorporated into the detailed description, with each claim standing on its own as a separately stated subject matter.

The claims are not intended to be limited to the characteristics described herein, but are to be accorded the full scope consistent with the claims in the stated language and to encompass all legal equivalents. However, no claims are intended to cover subject matter that fails to satisfy the requirements of applicable patent statutes, nor should they be construed in such a manner.

10: Architecture 100: In-ear monitor (IEM) 101:User 103, 103-1, 103-2, 103-3: Data group 110:Customer device 110-1: AR head-mounted device 110-2:Mobile device 112: Processor circuit 118:Communication module 120:Memory circuit 130:Remote server 150:Internet 152:Database 161: ear canal 170:Ears 200:Augmented Reality Ecosystem 201:User 205: Wearable devices and sensors 205-1: Wearable devices in the ear 205-2: Wrist wearable devices 205-3: Wearable device for chest 205-4:Smart Glasses Sensor 210: Sphygmomanometer 220:Signal 220-1: Optical signals 220-2:Signal 220-3, 220-4: scattered signals 230:Data acquisition module (DAQ) 240:Computer 300, 300A, 300B, 300C: In-Ear Monitor (IEM) 301-1:Front end 301-2:Backend 305:Electrode 311fb: feedback level 311ff: Feedforward stage 312: Processor 321:Transmitter 323:Detector 324: Speaker 325:Microphone 325-1: Acoustic sensor/internal microphone 325-2: Acoustic sensor/external microphone 327:Motion sensor 329:Temperature sensor 330: Digital to analog and/or analog to digital converter (DAC/ADC) 340: In-ear fixture 341:Sealing gasket 342: Voltage equalization (PEQ) tube/flexible printed circuit board (FPCB) 343:Strain relief parts 344: Acoustic resistance network 345:down cable 361: ear canal 362:Eardrum 401: blood pressure 402: Proportion 405: Shrinking part 407: Diastolic part 410: Motion waveform 415:ECG signal 417: length of time 420:Relevance 500: Chart 501: Time abscissa 502: signal value 510: Motion waveform 515:ECG signal 517:R peak 600:Method 602: Step 604: Step 606: Step 700:Computer system 702: Processor 704:Memory 706:Data storage 708:Bus 710: Input/output module 712: Communication module 714:Input device 716:Output device

[FIG. 1] Depicts an AR headset and an in-ear monitor (IEM) in an architecture configured to assess a user's health, in accordance with certain embodiments.

[Figure 2] depicts an augmented reality ecosystem that includes wearable devices in the ears and on the wrist to assess a user's health, according to certain embodiments.

[FIG. 3A]-[FIG. 3C] depict different embodiments of an in-ear monitor (IEM) according to certain embodiments.

[Figure 4] depicts a motion waveform overlaid with an electrocardiogram (ECG) signal provided by an in-ear electrode or any electrode provided on a wearable device, according to certain embodiments.

[Figure 5] depicts a motion waveform obtained using a motion sensor in an IEM to determine a heart rate of a user, according to certain embodiments.

[Figure 6] Depicts steps in a method for assessing the health of a user of a head-mounted device or smart glasses using a motion sensor in an in-ear monitor, according to certain embodiments flow chart.

[FIG. 7] Depicts a block diagram of an example computer system with which a head mounted device and other client devices, and with which the method of FIG. 6 is implemented, is depicted, in accordance with certain embodiments.

In the drawings, elements with the same or similar reference symbols are associated with the same or similar features and attributes, unless expressly stated otherwise.

10: Architecture

100: In-ear monitor (IEM)

101:User

103-1, 103-2, 103-3: Data group

110-1: AR head-mounted device

110-2:Mobile device

112: Processor circuit

118:Communication module

120:Memory circuit

130:Remote server

150:Internet

152:Database

161: ear canal

170:Ears

Claims (20)

A device comprising: an in-ear fixture configured to seal the user's ear canal; a motion sensor mounted on the in-ear fixture and configured to provide a motion signal indicative of in-body or whole-body motion of the user; a speaker coupled to provide an audio signal to the user; and A processor coupled to the augmented reality head-mounted device, the processor configured to identify the health status of the user based on the motion signal. The device of claim 1, wherein the motion sensor is a contact microphone, and the motion signal is sent to the motion sensor and the ear of the user through the user's body. in the contact points between channels. The device of claim 1, wherein the motion sensor is an inertial motion unit, and the motion signal indicates the whole body motion of the user, which includes one of sneezing, coughing, falling, and stepping. . The device of claim 1, wherein the motion sensor is a microelectronic motion system. The device of claim 1, wherein the motion signal is indicative of the internal motion of the user and includes one of a heart rate or a breathing rate. The device of claim 1, further comprising electrodes mounted on the in-ear fixture, the electrodes being configured to receive signals indicative of cardiovascular activity of the user. The device of claim 1, further comprising electrodes mounted on the in-ear fixture, the electrodes being configured to receive signals indicative of neural activity of the user. The device of claim 1, wherein the processor is configured to synchronize the waveform with the motion signal and the waveform with the electronic signal to determine the vital signs of the user. The device of claim 1, wherein the processor is configured to determine the heart rate of the user from the motion signal. The device of claim 1, wherein the processor is configured to remove one of a noise component or interference from the audio signal transmitted to the user based on the motion signal. A computer-implemented method comprising: receiving motion signals from a motion sensor in the in-ear device indicative of in-body motion or whole-body motion of the user of the in-ear device; utilizing the motion signal to form a waveform; and The user's health status is identified based on the waveform. The computer-implemented method of claim 11, further comprising receiving a second motion signal indicative of the in-body motion or the whole-body motion from a second motion sensor in a second in-ear device, and wherein forming the waveform Includes utilizing the second motion signal to remove at least one of a noise component or interference from the motion signal. The computer-implemented method of claim 11, wherein the motion signal is indicative of the in vivo motion, and identifying the health condition of the user includes determining at least one of a heart rate and a breathing rate of the user. The computer-implemented method of claim 11, wherein said motion signal indicates said whole body motion of said user, and identifying said health condition of said user includes detecting a fall of said user, said use At least one of the user's cough, the user's sneezing, or the number of steps of the user is determined. The computer-implemented method of claim 11, further comprising identifying a systolic heart pulse and a diastolic heart pulse from the waveform, wherein identifying the health condition of the user based on the waveform includes identifying the systolic heart pulse. and features in said diastolic heart pulse. The computer-implemented method of claim 11, further comprising identifying a systolic heart pulse and a diastolic heart pulse from the waveform, wherein identifying the health condition of the user based on the waveform includes based on the diastolic heart pulse. The amplitude and the amplitude of the systolic heart pulse are used to determine the user's blood pressure. The computer-implemented method of claim 11, further comprising identifying a systolic heart pulse and a diastolic heart pulse from the waveform, wherein identifying the health condition of the user based on the waveform includes relative to the diastolic heart pulse based on the waveform. The user's blood pressure is determined based on the time delay of the systolic heart pulse. The computer-implemented method of claim 11, further comprising using the waveform to filter one of noise components and interference from the audio signal transmitted to a speaker installed in the in-ear device, and using the A speaker is used to provide the audio signal to the user. The computer-implemented method of claim 11, further comprising: receiving an electronic signal from an electrode indicative of the user's cardiovascular activity, identifying a correlation between the electronic signal and the waveform, and based on the correlation or It is a deep neural network algorithm to determine one of the user's heart rate, breathing rate, or blood pressure. The computer-implemented method of claim 11, wherein identifying the health condition of the user further includes performing spectral analysis on the waveform to identify p-waves, QRS complexes, and T-wave complexes in an electrocardiogram. .

Images