US10126829B2 - Methods and systems for monitoring and influencing gesture-based behaviors - Google Patents

Methods and systems for monitoring and influencing gesture-based behaviors Download PDF

Info

Publication number
US10126829B2
US10126829B2 US15/603,246 US201715603246A US10126829B2 US 10126829 B2 US10126829 B2 US 10126829B2 US 201715603246 A US201715603246 A US 201715603246A US 10126829 B2 US10126829 B2 US 10126829B2
Authority
US
United States
Prior art keywords
user
sensor data
sensor
wearable device
gesture
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US15/603,246
Other languages
English (en)
Other versions
US20170262064A1 (en
Inventor
Eran Ofir
Uri Schatzberg
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Somatix Inc
Original Assignee
Somatix Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US15/603,246 priority Critical patent/US10126829B2/en
Application filed by Somatix Inc filed Critical Somatix Inc
Publication of US20170262064A1 publication Critical patent/US20170262064A1/en
Assigned to SOMATIX, INC. reassignment SOMATIX, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: OFIR, Eran
Priority to US16/184,851 priority patent/US10474244B2/en
Assigned to SOMATIX, INC. reassignment SOMATIX, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: OFIR, Eran, SCHATZBERG, URI
Application granted granted Critical
Publication of US10126829B2 publication Critical patent/US10126829B2/en
Assigned to MEIR AKKERMAN, MARS HEALTH FUND I, LLC, YNOR NERO, LLC, REMOT NERO, LLC reassignment MEIR AKKERMAN SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SOMATIX, INC.
Priority to US16/679,080 priority patent/US11112874B2/en
Priority to US17/388,921 priority patent/US11550400B2/en
Priority to US18/061,964 priority patent/US20230359282A1/en
Assigned to SOMATIX, INC. reassignment SOMATIX, INC. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: MARS HEALTH FUND I, LLC
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • A—HUMAN NECESSITIES
    • A24—TOBACCO; CIGARS; CIGARETTES; SIMULATED SMOKING DEVICES; SMOKERS' REQUISITES
    • A24F—SMOKERS' REQUISITES; MATCH BOXES; SIMULATED SMOKING DEVICES
    • A24F47/00—Smokers' requisites not otherwise provided for
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1123—Discriminating type of movement, e.g. walking or running
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
    • A61B5/6802—Sensor mounted on worn items
    • A61B5/681—Wristwatch-type devices
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
    • A61B5/6813—Specially adapted to be attached to a specific body part
    • A61B5/6824—Arm or wrist
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2562/00—Details of sensors; Constructional details of sensor housings or probes; Accessories for sensors
    • A61B2562/02—Details of sensors specially adapted for in-vivo measurements
    • A61B2562/0219—Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2562/00—Details of sensors; Constructional details of sensor housings or probes; Accessories for sensors
    • A61B2562/02—Details of sensors specially adapted for in-vivo measurements
    • A61B2562/0228—Microwave sensors
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1116—Determining posture transitions

Definitions

  • wearable devices such as smartwatches and wristbands in the consumer electronics market.
  • Wearable devices make computing technology pervasive by interweaving it into users' daily lives. These wearable devices generally allow users to track their fitness, activities, health and/or well-being, through the use of electronics, software, and sensors in those devices.
  • wearable devices are typically geared towards improvement of users' fitness and well-being. Additionally, the use of wearable devices can be extended to other areas such as healthcare monitoring. Although wearable devices are capable of collecting large volumes of data about users, there is presently a lack of systems and algorithms that can accurately and efficiently analyze large volumes of data in certain healthcare areas. Examples of those healthcare areas may include monitoring of smoking behavior (e.g., smoking cessation), monitoring of certain types of eating and/or drinking disorders, monitoring of certain types of obsessive compulsive disorders, or monitoring of certain types of neurological diseases that display symptoms associated with repetitive vibration or shaking of a person's hands. Each of the above behaviors may be characterized by different and frequent ‘hand-to-mouth’ gestures. Existing systems and algorithms often lack the capability to accurately detect and monitor those gestures in real-time.
  • a plurality of physical motion profile patterns may be stored in a library, and gesture recognition may be carried out by comparing the user's physical gesture (e.g., a shape of the gesture) against the plurality of motion profile patterns.
  • this form of gesture recognition has several shortcomings. For example, body movement is different for different people and depends on a large number of parameters (e.g., body structure and its physical ratios, height, weight, posture (standing/sitting/driving), habits etc.). The body movement for each person may also vary at different times depending on his/her mood and stress level, injuries, location (work/home/at a bar with friends), which hand is being used, time of day, etc.
  • a cigarette as used herein may refer to any type of tobacco products including, but not limited to, rolled cigarettes, cigarettes, e-cigarettes, cigars, and/or smoking pipes.
  • gesture recognition systems may not be able to detect whether a user is drinking a hot beverage or a cold beverage, or smoking with a left hand or a right hand.
  • Existing gesture recognition systems also lack adaptability, and are generally unable to capture and reflect changes in the user's gestures and/or behavior over time.
  • smoking involves unique and complex hand-to-mouth gestures that vary between smokers, depending on the type, size, and/or brand of cigarette, a person's smoking history, gender, day and time of day of smoking, and a plethora of other factors. All these factors make it difficult to track and filter out smoking gestures and patterns.
  • systems and algorithms that can accurately recognize hand-to-mouth gestures of a user and detect smoking lapses in real-time.
  • Such information may be personalized and dynamically provided in real-time to the user on a computing device. The information can help the user to make informed decisions about his/her overall well-being, and show the user the progress that has been made.
  • the systems and methods disclosed herein address at least the above needs.
  • a gesture recognition method may comprise: obtaining sensor data collected using at least one sensor located on a wearable device, wherein said wearable device is configured to be worn by a user; and analyzing the sensor data to determine a probability of the user performing a predefined gesture, wherein the probability is determined based in part on a magnitude of a motion vector in the sensor data, and without comparing the motion vector to one or more physical motion profiles.
  • a system for implementing gesture recognition may comprise a memory for storing sensor data collected using at least one sensor located on a wearable device, wherein the wearable device is configured to be worn by a user.
  • the system may further comprise one or more processors configured to execute the set of software instructions to: analyze the sensor data to determine a probability of the user performing a predefined gesture, wherein the probability is determined based in part on a magnitude of a motion vector in the sensor data, and without comparing the motion vector to one or more physical motion profiles.
  • a tangible computer readable medium storing instructions that, when executed by one or more processors, causes the one or more processors to perform a computer-implemented gesture recognition method may be provided.
  • the method may comprise: obtaining sensor data collected using at least one sensor located on a wearable device, wherein said wearable device is configured to be worn by a user; and analyzing the sensor data to determine a probability of the user performing a predefined gesture, wherein the probability is determined based in part on a magnitude of a motion vector in the sensor data, and without comparing the motion vector to one or more physical motion profiles.
  • the predefined gesture may be selected from a group of different gestures associated with different activities.
  • the gestures associated with the different activities may be differentiated from one another based at least on the magnitude of different motion vectors in the sensor data, and without comparing the motion vectors to the one or more physical motion profiles.
  • the at least one sensor may comprise an accelerometer and a gyroscope.
  • the magnitude of the motion vector may comprise: (1) a magnitude of the acceleration vector obtained from the accelerometer, and/or (2) a magnitude of an angular velocity vector obtained from the gyroscope.
  • the probability may be determined based in part on the magnitude of the acceleration vector and/or the magnitude of the angular velocity vector.
  • the probability may be determined based in part on the magnitude of the acceleration vector and/or the magnitude of the angular velocity vector within different temporal periods, and without comparing the acceleration vector and/or the angular velocity vector to the one or more physical motion profiles.
  • a pitch angle, a roll angle, and/or a yaw angle of the wearable device may be calculated based on the acceleration vector and/or the angular velocity vector.
  • the probability may be determined based on the pitch angle, the roll angle, and/or the yaw angle.
  • a correlation may be determined between the magnitude of the acceleration vector and the magnitude of the angular velocity vector within different temporal periods, so as to determine the probability of the user performing the predefined gesture.
  • At least one sensor may further comprise one or more of the following: a magnetometer, a heart rate monitor, a global positioning system (GPS) receiver, an external temperature sensor, a microphone, a skin temperature sensor, a capacitive sensor, and/or a sensor configured to detect a galvanic skin response.
  • GPS global positioning system
  • the sensor data may be analyzed without comparing the sensor data against the one or more physical motion profiles.
  • a shape of the one or more physical motion profiles may be substantially similar to a shape of one or more physical gestures of the user.
  • Analyzing the sensor data may further comprise calculating a multi-dimensional distribution function, wherein said multi-dimensional distribution function is a probability function of a plurality of features.
  • the plurality of features may be associated with various aspects of the predefined gesture.
  • the plurality of features may comprise two or more of the following features: (1) a time duration of a submotion during the gesture; (2) the magnitude of the acceleration vector; (3) the magnitude of the angular velocity vector; (4) the roll angle; and (5) the pitch angle.
  • the multi-dimensional distribution function may be associated with one or more characteristic of the predefined gesture.
  • the plurality of features may be encoded within the sensor data, and extracted from the sensor data. Two or more features may be correlated.
  • the multi-dimensional distribution function may be configured to return a single probability value between 0 and 1, and wherein the probability value represents a probability of each feature.
  • each feature may be represented by a discrete value. In other cases, each feature may be measurable along a continuum.
  • the multi-dimensional distribution function may be calculated by using Singular Value Decomposition (SVD) to de-correlate the two or more correlated features such that they are approximately orthogonal to each other. The use of the SVD may reduce a processing time required to compute the probability value for the multi-dimensional distribution function and may reduce the amount of sensor data needed to determine the probability of the user performing the predefined gesture.
  • SVD Singular Value Decomposition
  • the function f(p 1 ) may be a 1D probability density distribution of a first feature
  • the function f(p 2 ) may be a 1D probability density distribution of a second feature
  • the function f(p n ) may be a 1D probability density distribution of a n-th feature.
  • the 1D probability density distribution of each feature may be obtained from a sample size of each feature.
  • the sample size may be constant across all of the features. In other cases, the sample size may be variable between different features.
  • One or more of the plurality of features may be determined whether they are statistically insignificant.
  • the one or more statistically insignificant features may have low correlation with the predefined gesture.
  • the one or more statistically insignificant features may be removed from the multi-dimensional distribution function. Removing the one or more statistically insignificant features from the multi-dimensional distribution function may reduce a computing time and/or power required to calculate the probability value for the multi-dimensional distribution function.
  • Analyzing the sensor data may further comprise applying a filter to the sensor data.
  • the filter may be a higher order complex filter comprising a finite-impulse-response (FIR) filter and/or an infinite-impulse-response (IIR) filter.
  • the filter may be a Kalman filter or a Parks-McClellan filter.
  • the wearable device may be configured to transmit the sensor data to a user device and/or a server for the analysis of the sensor data.
  • the transmission of the sensor data may be via one or more wireless or wired communication channels.
  • the one or more wireless communication channels may comprise BLE (Bluetooth Low Energy), WiFi, 3G, and/or 4G networks.
  • the sensor data may be stored in a memory on the wearable device when the wearable device is not in operable communication with the user device and/or the server.
  • the sensor data may be transmitted from the wearable device to the user device and/or the server when operable communication between the wearable device and the user device and/or the server is re-established.
  • a data compression step may be applied to the sensor data.
  • the compression of the sensor data may reduce a bandwidth required to transmit the sensor data, and the compression of the sensor data may reduce a power consumption of the wearable device during the transmission of the sensor data.
  • the data compression step may comprise calculating a time-based difference between samples of the sensor data along different axes of measurement. The time-based difference may be transmitted from the wearable device to a user device and/or a server.
  • the sensor data may be compressed using a predefined number of bits.
  • the one or more sensors may be configured to collect the sensor data at a predetermined frequency.
  • the predetermined frequency may be configured to optimize and/or reduce a power consumption of the wearable device.
  • the predetermined frequency may range from about 10 Hz to about 20 Hz.
  • the one or more sensors may be configured to collect the sensor data at a first predetermined frequency when the probability that the user is performing the gesture is below a predefined threshold value.
  • the one or more sensors may be configured to collect the sensor data at a second predetermined frequency when the probability that the user is performing the gesture is above a predefined threshold value.
  • the second predetermined frequency may be higher than the first predetermined frequency.
  • the one or more sensors may be configured to collect the sensor data for a predetermined time duration.
  • the one or more sensors may be configured to collect the sensor data continuously in real-time when the wearable device is powered on.
  • the one or more sensors may comprise a first group of sensors and a second group of sensors.
  • the first group of sensors and the second group of sensors may be selectively activated to reduce power consumption of the wearable device.
  • the first group of sensors and the second group of sensors may be selectively activated to reduce an amount of the collected sensor data.
  • the reduction in the amount of sensor data may allow for faster analysis/processing of the sensor data, and reduce an amount of memory required to store the sensor data.
  • the first group of sensors may be activated when the wearable device is powered on.
  • the first group of sensors may be used to determine the probability of the user performing the predefined gesture.
  • the second group of sensors may be inactive when the probability that the user is performing the gesture is below a predefined threshold value.
  • the second group of sensors may be selectively activated when the wearable device is powered on and when the probability that the user is performing the gesture is above a predefined threshold value.
  • the second group of sensors may be selectively activated upon determining that the user is performing the predefined gesture.
  • the second group of sensors may be activated to collect additional sensor data, so as to confirm that the user is performing the predefined gesture, monitor the gesture, and collect additional sensor data relating to the gesture.
  • An accuracy of detection of the predefined gesture may be improved when the wearable device is in the accuracy mode, since more information (greater amount of sensor data) is available for analysis in the accuracy mode.
  • the sensor data may not be analyzed or transmitted when the wearable device is in an idle mode or a charging mode.
  • a method of detecting a smoking gesture may comprise: obtaining sensor data collected using one or more sensors, wherein said sensors comprise a multi-axis accelerometer that is located on a wearable device configured to be worn by a user; and analyzing the sensor data to determine a probability of the user smoking, wherein the probability is determined based in part on a magnitude of an acceleration vector in the sensor data, and without comparing the motion vector to one or more physical motion profiles.
  • a system for implementing gesture recognition may comprise a memory for storing sensor data collected using one or more sensors, wherein the sensors may comprise a multi-axis accelerometer that is located on a wearable device configured to be worn by a user.
  • the system may further comprise one or more processors configured to execute a set of software instructions to: analyze the sensor data to determine a probability of the user smoking, wherein the probability is determined based in part on a magnitude of an acceleration vector in the sensor data, and without comparing the motion vector to one or more physical motion profiles.
  • a tangible computer readable medium storing instructions that, when executed by one or more processors, causes the one or more processors to perform a computer-implemented gesture recognition method may be provided.
  • the method may comprise: obtaining sensor data collected using one or more sensors, wherein said sensors comprise a multi-axis accelerometer that is located on a wearable device configured to be worn by a user; and analyzing the sensor data to determine a probability of the user smoking, wherein the probability is determined based in part on a magnitude of an acceleration vector in the sensor data, and without comparing the motion vector to one or more physical motion profiles.
  • analyzing the sensor data may comprise analyzing one or more features in the sensor data to determine a probability of the user taking a cigarette puff.
  • the features may comprise at least one of the following: (1) a time duration that a potential cigarette is detected in the user's mouth; (2) a roll angle of the user's arm; (3) a pitch angle of the smoker's arm; (4) a time duration of a potential smoking puff; (5) a time duration between consecutive potential puffs; (6) number of potential puffs that the user takes to finish smoking a cigarette; (7) the magnitude of the acceleration vector; (8) a speed of the user's arm; (9) an inhale region corresponding to an arm-to-mouth gesture; and/or (10) an exhale region corresponding to an arm-down-from-mouth gesture.
  • the features may be extracted from the sensor data.
  • the probability of the user smoking may be adjusted based on one or more user inputs.
  • the user inputs may comprise: (1) an input signal indicating that the user did not smoke; (2) an input signal indicating that the user had smoked; and (3) an input signal indicating that the user had smoked but the smoking gesture was not recognized or detected.
  • a user configuration file (UCF) for the user may be generated based on the analyzed sensor data and the one or more user inputs.
  • the UCF may be general to a plurality of users.
  • the UCF may become unique to each user after a period of time.
  • the UCF may be configured to adapt and change over time depending on the user's behavior.
  • the UCF may comprise a list of user parameters associated with different activities besides smoking.
  • the different activities may comprise at least one of the following: standing, walking, sitting, driving, drinking, eating, and/or leaning while standing or sitting.
  • the UCF may be dynamically changed when no smoking of the user has been detected for a predetermined time period.
  • the UCF may be dynamically changed to verify that the user has not smoked for the predetermined time period.
  • the user may be determined whether to be smoking with a right hand or a left hand based on a roll angle, a pitch angle, and/or a yaw angle extracted from the sensor data.
  • the UCF may be updated with the left/right hand information of the user.
  • the probability may be determined using a multi-dimensional distribution function that is associated with one or more smoking characteristics.
  • the one or more smoking characteristics may comprise the user taking one or more cigarette puffs.
  • the multi-dimensional distribution function may be generated for each puff.
  • the probability of the user smoking may be determined based on: (1) a number of potential puffs; (2) the multi-dimensional distribution function for each potential puff; and (3) a time duration in which the number of potential puffs occur.
  • a sum of the multi-dimensional distribution functions for a number of potential puffs may be determined whether to be equal to or greater than a predetermined probability threshold.
  • the user may be determined to be smoking when the sum is equal to or greater than the predetermined probability threshold, and the user may be determined not to be smoking when the sum is less than the predetermined probability threshold.
  • the user may be determined to be smoking when a predetermined number of puffs have been detected within a predetermined time period.
  • the roll and pitch angles associated with the potential puffs may be analyzed, and the puffs whose roll and pitch angles fall outside of a predetermined roll/pitch threshold may be discarded.
  • a time duration between the potential puffs may be analyzed, and the puffs where the time duration falls outside of a predetermined time period may be discarded.
  • FIG. 1 illustrates a healthcare monitoring system in accordance with some embodiments
  • FIG. 2 illustrates exemplary components in a healthcare monitoring system, in accordance with some embodiments
  • FIG. 3 illustrates the determination of a pitch angle, a roll angle, and/or a yaw angle of a wearable device based on sensor data from a gyroscope and/or an accelerometer on the wearable device, in accordance with some embodiments;
  • FIG. 5 illustrates the correlation in magnitudes of the acceleration vector and the angular velocity vector as a user is eating, in accordance with some embodiments
  • FIG. 6 illustrates the correlation in magnitudes of the acceleration vector and the angular velocity vector as a user is brushing teeth, in accordance with some embodiments
  • FIG. 9 is graph of the probability that a user is smoking a cigarette as a function of number of smoking puffs, in accordance with some embodiments.
  • FIG. 13 illustrates an exemplary window depicting the number of cigarettes smoked during a day by a user, in accordance with some embodiments
  • FIG. 18 illustrates an exemplary window ranking a smoker's cessation success/performance against other smokers in a group, in accordance with some embodiments.
  • FIG. 19 illustrates an exemplary window showing a plurality of smoking metrics of a user, in accordance with some embodiments.
  • Wearable devices have become increasingly popular in recent years. Although wearable devices are capable of collecting large volumes of data about users, there is presently a lack of systems and algorithms that can accurately and efficiently analyze large volumes of data, particularly in certain healthcare areas.
  • the gesture analysis engine may be implemented anywhere within the healthcare monitoring system, and/or outside of the healthcare monitoring system. In some embodiments, the gesture analysis engine may be implemented on the server. In other embodiments, the gesture analysis engine may be implemented on the user device. Additionally, the gesture analysis engine may be implemented on the wearable device. In some further embodiments, a plurality of gesture analysis engines may be implemented on one or more servers, user devices, and/or wearable devices. Alternatively, the gesture analysis engine may be implemented in one or more databases. The gesture analysis engine may be implemented using software, hardware, or a combination of software and hardware in one or more of the above-mentioned components within the healthcare monitoring system.
  • User device 102 may include one or more processors that are capable of executing non-transitory computer readable media that may provide instructions for one or more operations consistent with the disclosed embodiments.
  • the user device may include one or more memory storage devices comprising non-transitory computer readable media including code, logic, or instructions for performing the one or more operations.
  • the user device may include software applications that allow the user device to communicate with and transfer data between wearable device 104 , server 106 , gesture analysis engine 108 , and/or database 110 .
  • the user device may include a communication unit, which may permit the communications with one or more other components in healthcare monitoring system 100 .
  • the communication unit may include a single communication module, or multiple communication modules.
  • the user device may be capable of interacting with one or more components in healthcare monitoring system 100 using a single communication link or multiple different types of communication links.
  • Wearable device 104 may further include one or more devices capable of emitting a signal into an environment.
  • the wearable device may include an emitter along an electromagnetic spectrum (e.g., visible light emitter, ultraviolet emitter, infrared emitter).
  • the wearable device may include a laser or any other type of electromagnetic emitter.
  • the wearable device may emit one or more vibrations, such as ultrasonic signals.
  • the wearable device may emit audible sounds (e.g., from a speaker).
  • the wearable device may emit wireless signals, such as radio signals or other types of signals.
  • the input may also indicate how the user is feeling (e.g., whether the user is feeling motivated or discouraged) during the course of a program aimed at mitigating certain behaviors (e.g., smoking).
  • the user's input may be indicative of the user's thoughts, feelings, moods, opinions, questions, and/or answers relating to smoking.
  • Computer-readable instructions can be stored on a tangible non-transitory computer-readable medium, such as a flexible disk, a hard disk, a CD-ROM (compact disk-read only memory), and MO (magneto-optical), a DVD-ROM (digital versatile disk-read only memory), a DVD RAM (digital versatile disk-random access memory), or a semiconductor memory.
  • a tangible non-transitory computer-readable medium such as a flexible disk, a hard disk, a CD-ROM (compact disk-read only memory), and MO (magneto-optical), a DVD-ROM (digital versatile disk-read only memory), a DVD RAM (digital versatile disk-random access memory), or a semiconductor memory.
  • the methods can be implemented in hardware components or combinations of hardware and software such as, for example, ASICs, special purpose computers, or general purpose computers.
  • any of the user device, wearable device, server, gesture analysis engine, and the database may, in some embodiments, be implemented as a computer system.
  • the network is shown in FIG. 1 as a “central” point for communications between components, the disclosed embodiments are not so limited.
  • one or more components of the network layout may be interconnected in a variety of ways, and may in some embodiments be directly connected to, co-located with, or remote from one another, as one of ordinary skill will appreciate.
  • the disclosed embodiments may be implemented on the server, the disclosed embodiments are not so limited.
  • other devices such as gesture analysis system(s) and/or database(s) may be configured to perform one or more of the processes and functionalities consistent with the disclosed embodiments, including embodiments described with respect to the server.
  • the gesture analysis engine(s) may be implemented as one or more computers storing instructions that, when executed by processor(s), analyze input data from a user device and/or a wearable device in order to detect and/or monitor a predetermined gesture, and to provide information (e.g., recommendations) to assist the user in managing behavior associated with the predetermined gesture.
  • the gesture analysis engine(s) may also be configured to store, search, retrieve, and/or analyze data and information stored in one or more databases.
  • the data and information may include raw data collected from accelerometers and gyroscopes on one or more wearable devices, as well as each user's historical behavioral pattern and social interactions relating to the type of behavior (e.g., smoking).
  • server 106 may be a computer in which the gesture analysis engine is implemented.
  • a server may access and execute gesture analysis engine(s) to perform one or more processes consistent with the disclosed embodiments.
  • the gesture analysis engine(s) may be software stored in memory accessible by a server (e.g., in memory local to the server or remote memory accessible over a communication link, such as the network).
  • the gesture analysis engine(s) may be implemented as one or more computers, as software stored on a memory device accessible by the server, or a combination thereof.
  • one gesture analysis engine(s) may be a computer executing one or more gesture recognition techniques
  • another gesture analysis engine(s) may be software that, when executed by a server, performs one or more gesture recognition techniques.
  • gesture analysis engine and its communication with the user device and wearable device, will be described in detail below with reference to FIG. 2 .
  • various embodiments are described herein using monitoring or cessation of smoking behavior as an example, it should be noted that the disclosure is not limited thereto, and can be used to monitor other types of behaviors and activities besides smoking.
  • wearable device 104 may comprise at least one sensor 105 .
  • the wearable device may comprise an accelerometer 105 - 1 and a gyroscope 105 - 2 .
  • One or more other types of sensors as described elsewhere herein may be incorporated into the wearable device.
  • an accelerometer may be disposed on the wearable device.
  • the accelerometer may be a multi-axis accelerometer such as an n-axis accelerometer, whereby n may be an integer that is equal to or greater than 2.
  • the accelerometer may be a 3-axis accelerometer.
  • the accelerometer may be able of measuring acceleration along an X-axis, a Y-axis, and a Z-axis in a local coordinate system that is defined relative to the wearable device.
  • the gesture analysis engine may be configured to determine the probability of the user performing the predefined gesture based in part on the magnitude of the accelerator vector. For example, the gesture analysis engine may be configured to determine the probability based in part on the magnitude of the acceleration vector within different temporal periods, and without comparing the acceleration vector (and/or each acceleration component in the acceleration vector) to one or more physical motion profiles.
  • a pitch angle and/or a roll angle of the wearable device may be calculated by the gesture analysis engine using the acceleration components along the X-axis, Y-axis, and Z-axis.
  • the pitch angle and the roll angle may be indicative of a rotational motion of a portion of the user's body (where the wearable device is worn) about the Y-axis and the X-axis, respectively.
  • the gesture analysis engine may be configured to determine the probability of the user performing the predefined gesture based in part of the pitch angle and/or the roll angle.
  • the gesture analysis engine may be configured to determine the probability of the user performing the predefined gesture based in part on the magnitude of the angular velocity vector, and without comparing the angular velocity vector to one or more physical motion profiles. For example, the gesture analysis engine may be configured to determine the probability based in part on the magnitude of the angular velocity vector within different temporal periods.
  • a pitch angle, a roll angle, and/or a yaw angle of the wearable device may be determined based on sensor data from the gyroscope and/or the accelerometer on the wearable device.
  • the pitch angle, the roll angle, and/or the yaw angle may be indicative of a rotational motion of a part of the user's body about an X-axis, a Y-axis, and a Z-axis in a local coordinate system that is defined on the wearable device.
  • FIGS. 4, 5, 6, and 7 illustrate the data collected by an accelerometer and a gyroscope on a wearable device as a user is performing different activities (gestures), in accordance with some embodiments.
  • parts A and B of FIG. 4 respectively illustrate the magnitudes of the angular velocity vector and the acceleration vector as a user is smoking.
  • the magnitudes of the angular velocity vector in part A and the acceleration vector in part B show a temporal correlation, as indicated by the circled regions.
  • the gesture analysis engine can determine a probability that the user's gesture corresponds to smoking.
  • the data may indicate that the user has taken four cigarette puffs (as indicated by the four circled regions in part A).
  • Parts A and B of FIG. 7 respectively illustrate the magnitudes of the acceleration vector and the angular velocity vector as a user is drinking a cold drink. As shown in FIG. 7 , there is also a correlation between the magnitudes of the acceleration vector and the angular velocity vector.
  • the user may first bring the cigarette to the mouth (hand-to-mouth gesture), inhale (take a puff), remove the cigarette from the mouth (mouth-to-hand gesture), and exhale.
  • the user's hand may be in a rest position.
  • the user may be bringing the cigarette to the mouth.
  • the user may be taking a puff (inhaling).
  • the user may be removing the cigarette from the mouth and exhaling.
  • the user's hand may be again in a rest position.
  • the magnitudes of the acceleration vector and the angular velocity vector may show a correlation for each submotion of the smoking gesture.
  • the gesture analysis engine may be configured to analyze the sensor data without comparing the sensor data against one or more physical motion profile patterns.
  • a physical motion profile pattern as used herein may refer to any pattern that has substantially a same profile as a corresponding physical gesture of a user.
  • a shape of the physical motion profile pattern may be substantially similar to a shape of the corresponding physical gesture of the user. For example, if a user physically makes an L-shaped gesture, a corresponding physical motion profile pattern may have substantially an L-shape.
  • the gesture analysis engine may be configured to calculate a multi-dimensional distribution function that is a probability function of a plurality of features in the sensor data.
  • the features may be extracted from the sensor data.
  • the plurality of features may comprise n number of features denoted by p 1 through p n , where n may be any integer greater than 1.
  • the multi-dimensional distribution function may be denoted by f(p 1 , p 2 , . . . , p n ).
  • the multi-dimensional distribution function may be configured to return a single probability value between 0 and 1, with the probability value representing a probability across a range of possible values for each feature.
  • Each feature may be represented by a discrete value. Additionally, each feature may be measurable along a continuum.
  • the plurality of features may be encoded within the sensor data, and extracted from the sensor data using the gesture analysis engine 108 .
  • the gesture analysis engine may be configured to calculate the multi-dimensional distribution function by using Singular Value Decomposition (SVD) to de-correlate the features such that they are approximately orthogonal to each other.
  • SVD Singular Value Decomposition
  • the use of SVD can reduce a processing time required to compute a probability value for the multi-dimensional distribution function, and can reduce the amount of data required by the gesture analysis engine to determine a high probability (statistically significant) that the user is performing the predefined gesture.
  • the function f(p 1 ) may be a 1D probability density distribution of a first feature
  • the function f(p 2 ) may be a 1D probability density distribution of a second feature
  • the function f(p 3 ) may be a 1D probability density distribution of a third feature
  • the function f(p n ) may be a 1D probability density distribution of a n-th feature.
  • the 1D probability density distribution of each feature may be obtained from a sample size of each feature. In some embodiments, the sample size may be constant across all of the features. In other embodiments, the sample size may be variable between different features.
  • the gesture analysis engine may be configured to determine whether one or more of the plurality of features are statistically insignificant. For example, one or more statistically insignificant features may have a low correlation with the predefined gesture. In some embodiments, the gesture analysis engine may be further configured to remove the one or more statistically insignificant features from the multi-dimensional distribution function. By removing the one or more statistically insignificant features from the multi-dimensional distribution function, a computing time and/or power required to calculate a probability value for the multi-dimensional distribution function can be reduced.
  • the gesture analysis engine may be configured to analyze the sensor data to determine a probability that a user smoking.
  • the probability may be determined based in part on a magnitude of an acceleration vector and/or an angular velocity vector in the sensor data, and without comparing the acceleration vector and/or the angular velocity vector to one or more physical motion profiles.
  • the gesture analysis engine may be configured to analyze one or more features in the sensor data to determine a probability of the user taking a cigarette puff
  • the features may comprise at least one of the following: (1) a time duration that a potential cigarette is detected in a mouth of the user; (2) a roll angle of the user's arm; (3) a pitch angle of the smoker's arm; (4) a time duration of a potential smoking puff; (5) a time duration between consecutive potential puffs; (6) number of potential puffs that the user takes to finish smoking a cigarette; (7) the magnitude of the acceleration vector; (8) a speed of the user's arm; (9) an inhale region corresponding to an arm-to-mouth gesture; and (10) an exhale region corresponding to an arm-down-from-mouth gesture.
  • the gesture analysis engine may extract the features from the sensor data and insert them into a mathematical function to obtain the confidence (0-100%) level for which these features match a smoking gesture. If the confidence level is high, the gesture analysis engine may determine that the user has smoked a cigarette.
  • the mathematical function represents the user statistics. Different users have different statistics and functions.
  • a mathematical function may be represented by its polynomial coefficients (a's). Accordingly, the function may be defined by a set of numbers (a's). For example, in the equation show below, P is the function, x is the feature inserted into the function, and a′s are the coefficients that represent the function.
  • the gesture analysis engine may be configured to calculate a multi-dimensional distribution function associated with one or more smoking characteristics.
  • the one or more smoking characteristics may be associated with a user taking a cigarette puff.
  • the gesture analysis engine may be configured to generate a multi-dimensional distribution function for each puff.
  • the gesture analysis engine may be configured to determine the probability of the user smoking based on: (1) a number of potential puffs; (2) the multi-dimensional distribution function for each potential puff; and (3) a time duration in which the number of potential puffs occur.
  • the gesture analysis engine may be configured to determine whether a sum of the multi-dimensional distribution functions for a number of potential puffs is equal to or greater than a predetermined probability threshold.
  • the gesture analysis engine may determine that the user is smoking when the sum is equal to or greater than the predetermined probability threshold, and that the user is not smoking when the sum is less than the predetermined probability threshold. In some embodiments, the gesture analysis engine may determine that the user is smoking when a predetermined number of puffs have been detected within a predetermined time period. For in some cases, the predetermined number of puffs may be at least three puffs, and the predetermined time period may be about five to six minutes.
  • the gesture analysis engine may be configured to analyze the roll and pitch angles associated with the potential puffs, and discard those puffs whose roll and pitch angles fall outside of a predetermined roll/pitch threshold. The gesture analysis engine may also be configured to analyze a time duration between the potential puffs, and discarding the puffs where the time duration falls outside of a predetermined time period.
  • the gesture analysis engine may implement an algorithm with a broad statistics that fit the average person (everyone) for a specific type of behavior or gesture.
  • the algorithm can be configured to adapt the statistics to a specific person over time. Each person may subsequently have a unique configuration file with his/her personal statistics, as described below.
  • the gesture analysis engine may be configured to generate a user configuration file (UCF) for the user based on the analyzed sensor data and the one or more user inputs.
  • UCF user configuration file
  • the gesture analysis engine may generate a general UCF.
  • the general UCF may be generic and non-specific to any user.
  • the general UCF may comprise a list of user parameters associated with smoking.
  • the general UCF may comprise a list of user parameters associated with different activities besides smoking. Examples of those activities may comprise at least one of the following: standing, walking, sitting, driving, drinking, eating, and leaning while either standing or sitting.
  • the leaning may be associated with the user's elbow. For example, the user may be sitting and leaning an elbow on an object.
  • the gesture analysis engine may be configured to generate a left hand UCF and/or right hand UCF for the user in addition to the general UCF.
  • the left hand UCF and/or right hand UCF may be incorporated in the general UCF.
  • the UCF may be configured to adapt and change over time depending on the user's behavior. Accordingly, after the gesture analysis engine has collected and analyzed historical behavioral data of the user for some time, the gesture analysis engine may generate a personal UCF that is unique to the user, based on changes to the general UCF and/or the left/right hand UCFs.
  • the gesture analysis engine may be configured to dynamically change the general UCF, left/right UCF, and/or personal UCF when the system detects that the user has not performed a predefined gesture for a predetermined time period. For example, one or more of the above UCFs may be dynamically changed when the system detects that the user has not smoked for a predetermined time period.
  • the system may send a question or prompt to the user (on user device and/or wearable device) requesting the user to verify that he/she has not smoked for the predetermined time period.
  • the gesture analysis engine may be configured to analyze the user's social network interaction using an application (e.g., a mobile application) provided by the gesture analysis engine.
  • the application may allow a user to pick a social group within the application and to compare his/her performance to other users in the social group.
  • the social group may be defined by the users.
  • the users in the social group may be seeking to manage or control a certain type of behavior or habit (e.g., smoking) using the application.
  • the user's performance may include the user's successes and/or failures in managing the type of behavior or habit, compared to other users in the group.
  • the gesture analysis engine can more accurately monitor the user's progress and provide personalized recommendations to the user.
  • the gesture analysis engine in addition to providing a user with the information that the user seeks and will most likely consume, can further provide personalized recommendations to influence the user's needs and behavior.
  • the user's needs and challenges may vary each day. For example, the user may suffer from anxiety, depression, low spirits, lack of energy, urge to smoke, etc. Furthermore, the user may be influenced by other events such as stress and peer pressure.
  • the gesture analysis engine can be configured to take into account the dynamic nature of the user's experiences during the smoking cessation program. For example, the gesture analysis engine can parametrize the user's behavior and body response characteristics at different timeframes. In some embodiments, the gesture analysis engine can be configured to determine the user's potential needs, and provide personalized recommendations based on those potential needs. Accordingly, in some embodiments, the gesture analysis engine may be capable of sentiment analysis, so as to more accurately construe and predict the user's needs and behavior.
  • the analyzed data may be provided by the gesture analysis engine to a healthcare organization, an insurance company, and/or government agency.
  • One or more of the above entities may use the data to tailor preventive behavioral programs that promote the health and well-being of the users.
  • Some or all of the sensors on the wearable sensor may be activated at any time. In some embodiments, a subset of the sensors may be activated to reduce power consumption of the wearable device.
  • the gesture analysis engine and/or user device detects that the user may be taking a first potential cigarette puff (e.g., with probability ⁇ 1)
  • the gesture analysis engine and/or user device may be configured to transmit signals to the wearable sensor to turn on the other sensors.
  • Some or all of the sensor data may be aggregated and sent in blocks from the wearable device to the gesture analysis engine in real-time (either directly or via the user device).
  • the gesture analysis engine may be configured to extract a set of pre-defined features from some or all of the sensor data (step 1004 ).
  • the gesture analysis engine may be configured to use the set of pre-defined features to detect a probability of the user smoking, by rating a potential cigarette puff and/or number of puffs (steps 1006 and 1008 ). This may be achieved by analyzing the magnitudes of the acceleration vector and/or angular velocity vector of the hand-to-mouth and mouth-to-hand gestures against certain smoking models.
  • the gesture analysis engine can detect whether the user is smoking or has smoked a cigarette (step 1010 ).
  • the gesture analysis engine may transmit and store smoking-related information into a database for further and/or future analysis (step 1012 ).
  • the smoking-related information may comprise a duration of a cigarette puff, cigarette type, personal information of the user, location of the user, time of smoking, etc.
  • the smoking-related information may be accumulated over time and used to generate smoking-behavioral trends of the user.
  • the smoking-related information may be displayed on a graphical display on the user device (step 1014 ).
  • the gesture analysis engine can use the smoking-behavioral trends to improve a confidence level of the statistical analysis, and to predict when/where a user is likely to smoke.
  • the gesture analysis engine may analyze the smoking-behavioral trends to detect hidden correlations between different parameters in the information. The hidden correlations may be used to predict user behavior and/or habits.
  • the user device determines if a time window has expired (step 1114 ). If the time window has not expired, the wearable device may continue to collect sensor data. If the time window has expired, the user device may determine whether an event count is greater than a threshold count (step 1116 ). If the event count is less than the threshold count, the time window and event count may be reset (step 1120 ) so that a new set of sensor data may be collected. If the event count is greater than the threshold count, some or all of the sensor data may be transmitted to the gesture analysis engine to detect a probability of the user smoking (step 1118 ). For example, when a sufficient regions have been detected in a pre-defined time window (e.g., 10 minutes), the user device may transmit some or all of the sensor data (including the sensor data it has already processed) to the gesture analysis engine.
  • a threshold count e.g. 10 minutes
  • the gesture analysis engine may be configured to evaluate each puff candidate by comparing it to pre-defined statistics and rate each puff. For example, the gesture analysis engine may extract information from the puff signal (e.g., length of time a signal is low, etc.) and compare each value with a pre-defined empirical statistical model. The model may be general (the same for all smokers), or specific for each smoker. The probabilities are then aggregated into a puff rating. In some embodiments, one or more features may be extracted from the candidate puff signal and processed using machine learning algorithms to produce a puff rating.
  • the machine learning may comprise supervised-learning, semi-supervised learning or unsupervised learning techniques.
  • the gesture analysis engine can then determine whether a cigarette was smoked. For example, the gesture analysis engine may count the puffs above a certain rating (e.g. 50%) and compare the number of puffs to a threshold (e.g. 4 puffs). If the counted number of puffs is greater than the threshold, the gesture analysis engine may determine that the user is smoking a cigarette. Conversely, if the counted number of puffs is less than the threshold, the gesture analysis engine may determine that the user is not smoking a cigarette, and may be performing some other gesture.
  • a certain rating e.g. 50%
  • a threshold e.g. 4 puffs
  • the gesture analysis engine may be configured to process a whole cigarette signal instead of individually analyzing single puffs (e.g., a 10-minute signal instead of an 8-second signal).
  • a whole cigarette signal may be illustrated in, for example FIG. 4 .
  • a single puff signal may be illustrated in, for example FIG. 8 .
  • the gesture analysis engine can analyze a pre-defined time window of accelerometer and/or gyroscope signals (e.g., the time it may take to smoke a cigarette may be about 10 min) and detect a user possibly smoking a cigarette based on the signals.
  • the gesture analysis engine may be configured to determine a total time that the signal variance is below a pre-defined threshold. Alternatively, the gesture analysis engine may be configured to determine a relationship between the time that the signal variance is below the threshold and the time that it is above the threshold.
  • the system may then determine that the user may be possibly smoking. Once a possible cigarette is detected, the entire signal can be transmitted to the gesture analysis engine.
  • the gesture analysis engine may analyze all of the signals (instead of processing each puff separately) and rate the possible cigarette. This can be done by transforming the signals into frequency domain and extracting features (e.g., energy in specific frequencies, etc.).
  • the gesture analysis engine can also process the signals, the signal power, and/or the signal derivative (rate of change) and extract features therefrom. The features can then be used to rate the possible cigarette. Once the cigarette is rated, the gesture analysis engine can determine whether the rating is greater than a pre-defined threshold (e.g. 50%).
  • a pre-defined threshold e.g. 50%
  • the gesture analysis engine may try to estimate other puffs based on the first puff sample.
  • the gesture analysis engine may be configured to extract features from puff candidates as well as from a whole cigarette, to determine whether the user has smoked a cigarette.
  • the gesture analysis engine may be configured to alert and inform the user of changes in behavior, patterns, goals matching and other consumption related alerts. For example, the gesture analysis engine may provide an alert when the user behavior diverges from the user's typical behavior or historical behavior. For example, the gesture analysis engine may detect that a user normally smokes 2 cigarettes in the morning and 2 cigarettes in the evening. When the system detects that the user started smoking 2 additional cigarettes at noon, the system may send an alert to the user so that the user may refrain from smoking the additional cigarettes.
  • FIG. 12 is a flowchart of a method 1200 of detecting a probability of a user smoking a cigarette, in accordance with some further embodiments.
  • the sensor data may be transmitted to an algorithm manager (ALGO Manager).
  • the ALGO Manager may be a module located on the wearable device, user device, server, and/or gesture analysis engine.
  • the ALGO Manager may be configured to extract a portion of the sensor data, and transmit the extracted portion to a filter module (Pre-Filter).
  • the Pre-Filter may be located on the wearable device, user device, server, and/or gesture analysis engine.
  • the Pre-Filter may apply a filter to the sensor data prior to the analysis of the sensor data.
  • analyzing the sensor data may further comprise applying a filter to the sensor data.
  • the filter may be applied to reduce noise in the sensor data.
  • the filter may be a higher order complex filter such as a finite-impulse-response (FIR) filter or an infinite-impulse-response (IIR) filter.
  • FIR finite-impulse-response
  • IIR infinite-impulse-response
  • the filter may be a Kalman filter or a Parks-McClellan filter.
  • the filter may be applied using one or more processors on the wearable device.
  • the filter may be applied using one or more processors on the user device.
  • the filter may be applied using one or more processors on the server.
  • the filter may be applied using one or more processors in the gesture analysis engine.
  • the filtered sensor data may be provided to the gesture analysis engine as buffered data of a predetermined time block (e.g., in 12 secs block).
  • the buffered data may be received at gesture analysis engine at a predetermined time interval (e.g., every 5 secs).
  • the gesture analysis engine may be configured to detect probabilities of puffs from the buffered blocks.
  • the gesture analysis engine may be configured to detect static areas in candidate puff signals.
  • the static areas may correspond to regions in the signals where one or both signals are beneath predefined respective thresholds. These regions may correspond to ‘suspected puff areas’.
  • the gesture analysis engine may be configured to extract features from the candidate puff signals, and to analyze the features using statistics (e.g., multi-dimensional distribution function) to produce a puff rating.
  • the candidate puffs with the respective puff ratings may be inserted into a puff queue.
  • the gesture analysis engine may be configured to determine the probability of the user smoking based on the puffs in the puff queue. Additionally, the method of FIG. 12 may incorporate one or more of steps previously described in FIGS. 10 and 11 .
  • the sensor data may be stored in a memory on the wearable device when the wearable device is not in operable communication with the user device and/or the server. In those embodiments, the sensor data may be transmitted from the wearable device to the user device when operable communication between the user device and the wearable device is re-established. Alternatively, the sensor data may be transmitted from the wearable device to the server when operable communication between the server and the wearable device is re-established.
  • a data compression step may be applied to the sensor data prior to data transmission.
  • the compression of the sensor data can reduce a bandwidth required to transmit the sensor data, and can also reduce a power consumption of the wearable device during transmission of the sensor data.
  • the data compression step may comprise calculating a difference between samples of the sensor data. The difference may be time-based (t) or spatial-based (X, Y, and Z). For example, if there is no difference in the acceleration magnitudes of a current data sample and previous data samples, the sensor data is not re-transmitted.
  • the sensor data may be compressed using a predefined number of bits (e.g., 16 bits). For example, 32-bit or 64-bit sensor data may be compressed to 16 bits.
  • the sensor data may be collected at a predetermined frequency.
  • the predetermined frequency may be configured to optimize and/or reduce a power consumption of the wearable device.
  • the predetermined frequency may range from about 10 Hz to about 20 Hz.
  • one or more sensors may be configured to collect the sensor data at a first predetermined frequency when the gesture analysis engine determines that the user is not smoking.
  • the one or more sensors may be configured to collect the sensor data at a second predetermined frequency when the gesture analysis engine determines a high probability that the user is smoking.
  • the second predetermined frequency may be higher than the first predetermined frequency.
  • the one or more sensors may be configured to collect the sensor data for a predetermined time duration.
  • the one or more sensors may be configured to collect the sensor data continuously in real-time when the wearable device is powered on.
  • a frequency of the sensor data collection may be adjusted based on the different times of the day and/or the different geographical locations. For example, the frequency of the sensor data collection may be increased at times of the day and/or at geographical locations where the probability of the user performing the predefined gesture is above a predetermined threshold value. Conversely, the frequency of the sensor data collection may be decreased at times of the day and/or at geographical locations where the probability of the user performing the predefined gesture is below a predetermined threshold value.
  • one or more sensors in the wearable device and/or the user device may be selectively activated based on the probability of the user performing the predefined gesture at different times of the day and/or at different geographical locations.
  • the one or more sensors may comprise a first group of sensors and a second group of sensors.
  • the first and second groups of sensors may be selectively activated to reduce power consumption of the wearable device, and to reduce an amount of the collected sensor data.
  • the reduction in the sensor data can allow faster analysis/processing of the sensor data, and reduce an amount of memory required to store the sensor data.
  • the first group of sensors may be activated when the wearable device is powered on.
  • the first group of sensors may be used to determine whether there is a high probability that the user is smoking.
  • the second group of sensors may be inactive prior to determining whether the user is smoking.
  • the second group of sensors may be selectively activated when the wearable device is powered on, depending on whether there is a high probability that the user is smoking. For example, the second group of sensors may be selectively activated upon determining that there is a high probability that the user is smoking.
  • the second group of sensors may be activated to collect additional sensor data, so as to confirm that the user is smoking, monitor the smoking, and collect additional smoking-related data.
  • the wearable device may be configured to operate in a plurality of energy and/or performance modes.
  • the modes may comprise a low power mode in which only some of the sensors are turned on.
  • the wearable device may have low power consumption when the wearable device is in the low power mode.
  • An accuracy of detection of the predefined gesture may be reduced when the wearable device is in the low power mode, since less information (less amount of sensor data) is available for analysis in the low power mode.
  • the modes may comprise an accuracy mode in which all of the sensors are turned on.
  • the wearable device may have high power consumption when the wearable device is in the accuracy mode.
  • An accuracy of detection of the predefined gesture may be improved when the wearable device is in the accuracy mode, since more information (greater amount of sensor data) is available for analysis in the accuracy mode.
  • the sensor data may not be analyzed when the wearable device and/or the user device is in an idle mode or a charging mode.
  • the sensor data may comprise one or more parameters.
  • the parameters may comprise at least one of the following: (1) a hand which the user smokes with; (2) a pulse pattern of the user; (3) a location of the user; (4) a wearable device identifier and a user device identifier (e.g., MSISDN or Android ID or Advertiser ID or IMEI+mac address); and (5) smoking statistics of the user.
  • the one or more parameters may be unique to the user, the wearable device, and/or the user device.
  • an identity of the user may be authenticated based on the one or more parameters. The identity of the user may need to be authenticated to prevent misuse of the wearable device and/or user device.
  • the gesture analysis engine can generate one or more graphical user interfaces (GUIs) comprising statistics of the user's behavior.
  • GUIs may be rendered on a display screen on a user device.
  • a GUI is a type of interface that allows users to interact with electronic devices through graphical icons and visual indicators such as secondary notation, as opposed to text-based interfaces, typed command labels or text navigation.
  • the actions in a GUI are usually performed through direct manipulation of the graphical elements.
  • GUIs can be found in hand-held devices such as MP3 players, portable media players, gaming devices and smaller household, office and industry equipment.
  • the GUIs may be provided in a software, a software application, a web browser, etc.
  • the GUIs may be displayed on a user device (e.g., user device 102 of FIG. 1 ).
  • the GUIs may be provided through a mobile application. Examples of such GUIs are illustrated in FIGS. 13 through 19 and described as follows.
  • Window 1300 of FIG. 13 may be generated after the user device is connected to the gesture analysis engine and data has been obtained from the gesture analysis engine.
  • Window 1300 may be an exemplary window depicting various smoking monitoring metrics.
  • window 1300 may correspond to a home landing page that a user will view first when opening the application or logging into the application.
  • Window 1300 may indicate the smoking metrics for the user by day.
  • window 1300 may display that the user had smoked 4 cigarettes for that day, with 0% improvement compared to the previous day, spent $1.30 on cigarettes, and potentially ‘wasted’ 44 minutes of his/her life by smoking the 4 cigarettes that day.
  • the amount of time ‘wasted’ may be indicative of a health impact from smoking a number of cigarettes.
  • a user may view his/her smoking metrics by week. For example, as shown in window 1500 , the barchart indicates that the user smoked 16 cigarettes on Sunday, 14 on Monday, 19 on Tuesday, 17 on Wednesday, 12 on Thursday, 15 on Friday, and 14 on Saturday. It may be observed that the user smoked the least on Thursday and smoked the most on Tuesday.
  • the piechart in window 1500 further illustrates that 38% of the smoking occurred on weekdays and 62% occurred on weekends.
  • a user may view his/her smoking metrics by month.
  • the barchart indicates that the user smoked 102 cigarettes in Week 1 , 115 in Week 2 , 98 in Week 3 , and 104 in Week 4 . It may be observed that the user smoked the least in Week 3 and smoked the most in Week 2 .
  • the piechart in window 1600 further illustrates that 12% of the smoking occurred in the morning, 45% occurred at noon, 26% occurred in the evening, and 17% occurred at night.
  • a user may set goals in the application. For example, as shown in window 1700 of FIG. 17 , the user may set a goal of limiting to 14 cigarettes within a day. This may require the user to spend $4.48 on the cigarettes. Additionally, smoking 14 cigarettes could potentially waste 154 mins of the user's life.
  • a user may view his smoking behavior compared to other users. For example, as shown in window 1800 of FIG. 18 , smoking an average of 14 cigarettes per day and an average of 98 cigarettes per week may place the user in the 6 th percentile within the group of users.
  • a user may view various metrics associated with his smoking patterns. For example, window 1900 of FIG. 19 illustrates that the user had smoked a total of 425 cigarettes, spent $136 on cigarettes, smoked an average of 17 cigarettes per day, and potentially ‘wasted’ 77 hours of his life by smoking. Additionally, window 1900 shows that 18% of the smoking occurred at home, 62% occurred at work, and 20% occurred at other locations.
  • different colors and shading may be used to differentiate the segments from each other.
  • the numbers and words for various metrics may be provided in different colors and shades to improve readability, and to distinguish the metrics from one another. Any color scheme or any other visual differentiation scheme may be contemplated.
  • the gesture analysis engine may be configured to receive puffs/cigarettes information from a plurality of wearable devices and/or user devices. Each wearable device and/or user device may serve as a data node that provides user consumption data to a database connected to the gesture analysis engine. The database may be updated in real-time with the user's smoking data.
  • the gesture analysis engine may be configured to generate consumption statistics and determine smoking-related social patterns. For example, the gesture analysis engine can generate a visual representation of aggregated consumption data (e.g., total number of cigarettes smoked by day/week/month). The consumption data may further include the market share of each cigarettes brand, consumption per user gender by cigarette brand, and consumer preferences.
  • Consumer preferences may include time of smoking by cigarette brand, location of smoking (home/work/driving/other), smoking frequency (per event, per time, per person), consumption per capita, and correlation of smoking with consumption of other products (such as coffee).
  • the gesture analysis engine can also analyze the consumption statistics to determine consumption patterns (correlation) for different brands, geography, and/or time periods.
  • the gesture analysis engine may be capable of cross-learning and recognizing the correlation/impact between different smokers, which can help evaluate the optimized paths for smoking cessation for the user as well as his/her social circle.
  • the gesture analysis engine may detect that user X is a leader in his social circle and cessation of smoking by user X may significantly influence others in his social circle to change their smoking behavior. Accordingly, the gesture analysis engine may provide additional incentives to user X to assist him in smoking cessation, so that the effect can be proliferated across the social circle.
  • the gesture analysis system is capable of differentiating between smoking patterns of the moving hand and other movements that are not smoking related.
  • the algorithms described herein may be based in part on statistical analysis, machine learning, signal processing, pattern recognition, and detection theory.
  • An algorithm may assume a certain smoking model and try to detect the smoking of a cigarette based on the model.
  • the algorithm may also estimate a different smoking model for each smoker and use the model to detect a specific smoker is smoking.
  • the gesture analysis system can analyze geographical, time-based and user attributes (e.g., age, gender, job vocation, etc.) cigarette consumption trends by aggregating data from a plurality of wearable devices worn by a plurality of users who smoke.
  • geographical, time-based and user attributes e.g., age, gender, job vocation, etc.
  • the gesture analysis system can be used to implement a smoking cessation program based in part on cognitive behavioral psychology, by using constant monitoring of near real time smoking and goal accomplishment in the program. Using the monitoring system, a user can be notified of each cigarette that he/she smoked, and receive instant notification regarding his/her smoking patterns and information on the progress in reaching his/her smoking reduction goals. Real-time generation of smoking alerts can be highly effective for smoking cessation.
  • various pattern recognition algorithms can be used to determine the required milestone/incentive to be offered to the user in order to effectively influence his/her smoking habits, which can help to change the user's smoking behavior and decrease the health risks caused by smoking.
  • a and/or B encompasses one or more of A or B, and combinations thereof such as A and B. It will be understood that although the terms “first,” “second,” “third” etc. may be used herein to describe various elements, components, regions and/or sections, these elements, components, regions and/or sections should not be limited by these terms. These terms are merely used to distinguish one element, component, region or section from another element, component, region or section. Thus, a first element, component, region or section discussed below could be termed a second element, component, region or section without departing from the teachings of the present disclosure.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Pathology (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Biomedical Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Dentistry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Physiology (AREA)
  • User Interface Of Digital Computer (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)
  • Medical Treatment And Welfare Office Work (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
US15/603,246 2014-12-16 2017-05-23 Methods and systems for monitoring and influencing gesture-based behaviors Active US10126829B2 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US15/603,246 US10126829B2 (en) 2014-12-16 2017-05-23 Methods and systems for monitoring and influencing gesture-based behaviors
US16/184,851 US10474244B2 (en) 2014-12-16 2018-11-08 Methods and systems for monitoring and influencing gesture-based behaviors
US16/679,080 US11112874B2 (en) 2014-12-16 2019-11-08 Methods and systems for monitoring and influencing gesture-based behaviors
US17/388,921 US11550400B2 (en) 2014-12-16 2021-07-29 Methods and systems for monitoring and influencing gesture-based behaviors
US18/061,964 US20230359282A1 (en) 2014-12-16 2022-12-05 Methods and systems for monitoring and influencing gesture-based behaviors

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201462092283P 2014-12-16 2014-12-16
PCT/US2015/065862 WO2016100368A1 (fr) 2014-12-16 2015-12-15 Procédés et systèmes pour surveiller et influencer des comportements basés sur un geste
US15/603,246 US10126829B2 (en) 2014-12-16 2017-05-23 Methods and systems for monitoring and influencing gesture-based behaviors

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2015/065862 Continuation WO2016100368A1 (fr) 2014-12-16 2015-12-15 Procédés et systèmes pour surveiller et influencer des comportements basés sur un geste

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/184,851 Continuation US10474244B2 (en) 2014-12-16 2018-11-08 Methods and systems for monitoring and influencing gesture-based behaviors

Publications (2)

Publication Number Publication Date
US20170262064A1 US20170262064A1 (en) 2017-09-14
US10126829B2 true US10126829B2 (en) 2018-11-13

Family

ID=56127477

Family Applications (5)

Application Number Title Priority Date Filing Date
US15/603,246 Active US10126829B2 (en) 2014-12-16 2017-05-23 Methods and systems for monitoring and influencing gesture-based behaviors
US16/184,851 Active US10474244B2 (en) 2014-12-16 2018-11-08 Methods and systems for monitoring and influencing gesture-based behaviors
US16/679,080 Active US11112874B2 (en) 2014-12-16 2019-11-08 Methods and systems for monitoring and influencing gesture-based behaviors
US17/388,921 Active US11550400B2 (en) 2014-12-16 2021-07-29 Methods and systems for monitoring and influencing gesture-based behaviors
US18/061,964 Abandoned US20230359282A1 (en) 2014-12-16 2022-12-05 Methods and systems for monitoring and influencing gesture-based behaviors

Family Applications After (4)

Application Number Title Priority Date Filing Date
US16/184,851 Active US10474244B2 (en) 2014-12-16 2018-11-08 Methods and systems for monitoring and influencing gesture-based behaviors
US16/679,080 Active US11112874B2 (en) 2014-12-16 2019-11-08 Methods and systems for monitoring and influencing gesture-based behaviors
US17/388,921 Active US11550400B2 (en) 2014-12-16 2021-07-29 Methods and systems for monitoring and influencing gesture-based behaviors
US18/061,964 Abandoned US20230359282A1 (en) 2014-12-16 2022-12-05 Methods and systems for monitoring and influencing gesture-based behaviors

Country Status (6)

Country Link
US (5) US10126829B2 (fr)
EP (1) EP3234731B1 (fr)
JP (2) JP2018506773A (fr)
CN (1) CN107250947B (fr)
HK (1) HK1245940A1 (fr)
WO (1) WO2016100368A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10474244B2 (en) 2014-12-16 2019-11-12 Somatix, Inc. Methods and systems for monitoring and influencing gesture-based behaviors
US11790698B2 (en) 2019-11-01 2023-10-17 Samsung Electronics Co., Ltd. Electronic device for recognizing gesture of user using a plurality of sensor signals

Families Citing this family (72)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11921471B2 (en) 2013-08-16 2024-03-05 Meta Platforms Technologies, Llc Systems, articles, and methods for wearable devices having secondary power sources in links of a band for providing secondary power in addition to a primary power source
US20150124566A1 (en) 2013-10-04 2015-05-07 Thalmic Labs Inc. Systems, articles and methods for wearable electronic devices employing contact sensors
US10188309B2 (en) 2013-11-27 2019-01-29 North Inc. Systems, articles, and methods for electromyography sensors
US9552070B2 (en) * 2014-09-23 2017-01-24 Microsoft Technology Licensing, Llc Tracking hand/body pose
USD980507S1 (en) 2015-04-22 2023-03-07 Altria Client Services Llc Electronic vaping device
US10064432B2 (en) 2015-04-22 2018-09-04 Altria Client Services Llc Pod assembly, dispensing body, and E-vapor apparatus including the same
US10104913B2 (en) 2015-04-22 2018-10-23 Altria Client Services Llc Pod assembly, dispensing body, and E-vapor apparatus including the same
USD874720S1 (en) 2015-04-22 2020-02-04 Altria Client Services, Llc Pod for an electronic vaping device
USD874059S1 (en) 2015-04-22 2020-01-28 Altria Client Servies Llc Electronic vaping device
UA128328C2 (uk) 2015-04-22 2024-06-12 Олтріа Клайєнт Сервісиз Ллк Капсула для електронного пристрою для вироблення пари і електронний пристрій для вироблення пари, що містить таку капсулу
US10671031B2 (en) 2015-04-22 2020-06-02 Altria Client Services Llc Body gesture control system for button-less vaping
US11103139B2 (en) * 2015-06-14 2021-08-31 Facense Ltd. Detecting fever from video images and a baseline
US11154203B2 (en) * 2015-06-14 2021-10-26 Facense Ltd. Detecting fever from images and temperatures
US10129348B2 (en) * 2015-12-04 2018-11-13 At&T Intellectual Property I, L.P. Method and apparatus for modifying behavior of a user via a network
CN105549408B (zh) * 2015-12-31 2018-12-18 歌尔股份有限公司 可穿戴设备、智能家居服务器及其控制方法和系统
GB2546737A (en) * 2016-01-26 2017-08-02 Univ Bristol Method and device for detecting a smoking gesture
CA3013053A1 (fr) 2016-01-28 2017-08-03 Savor Labs, Inc. Procede et appareil de suivi de prise alimentaire et d'autres comportements et de fourniture d'un retour d'informations pertinentes
US20230368046A9 (en) * 2016-01-28 2023-11-16 Medtronic Minimed, Inc. Activation of Ancillary Sensor Systems Based on Triggers from a Wearable Gesture Sensing Device
WO2017139736A1 (fr) * 2016-02-12 2017-08-17 Als Therapy Development Institute Mesure de progression de la sclérose latérale amyotrophique (als) sur la base de données cinétiques
CN110300542A (zh) 2016-07-25 2019-10-01 开创拉布斯公司 使用可穿戴的自动传感器预测肌肉骨骼位置信息的方法和装置
US10255738B1 (en) 2016-07-25 2019-04-09 United Services Automobile Association (Usaa) Authentication based on through-body signals detected in body area networks
US11216069B2 (en) 2018-05-08 2022-01-04 Facebook Technologies, Llc Systems and methods for improved speech recognition using neuromuscular information
US11635736B2 (en) 2017-10-19 2023-04-25 Meta Platforms Technologies, Llc Systems and methods for identifying biological structures associated with neuromuscular source signals
US20200073483A1 (en) * 2018-08-31 2020-03-05 Ctrl-Labs Corporation Camera-guided interpretation of neuromuscular signals
EP3507731A4 (fr) * 2016-09-02 2020-04-22 Lumme Inc Systèmes et procédés de surveillance sanitaire
US10007948B1 (en) * 2016-12-22 2018-06-26 Capital One Services, Llc Systems and methods for facilitating a transaction relating to newly identified items using augmented reality
WO2018125674A1 (fr) * 2016-12-27 2018-07-05 Altria Client Services Llc Système de commande par geste corporel pour vapotage sans bouton
WO2018156809A1 (fr) * 2017-02-24 2018-08-30 Masimo Corporation Système de réalité augmentée permettant d'afficher des données de patient
KR102567007B1 (ko) 2017-02-24 2023-08-16 마시모 코오퍼레이션 의료 모니터링 데이터 표시 시스템
US10551935B2 (en) 2017-04-07 2020-02-04 University Of South Carolina Wearable computing device featuring machine-learning-based smoking detection
EP3622529A1 (fr) 2017-05-08 2020-03-18 Masimo Corporation Système permettant d'apparier un système médical à un dispositif de commande de réseau à l'aide d'une clé électronique
EP3667564A4 (fr) * 2017-08-08 2021-04-07 Fang, Chao Système d'acquisition de geste
CN109426827B (zh) * 2017-08-25 2021-05-07 中国科学院计算技术研究所 一种多模态传感器协同感知方法及系统
US11039645B2 (en) 2017-09-19 2021-06-22 Rai Strategic Holdings, Inc. Differential pressure sensor for an aerosol delivery device
US10928905B2 (en) * 2017-10-09 2021-02-23 TacSense, Inc. Body motion and position sensing, recognition and analytics from an array of wearable pressure sensors
WO2019118917A1 (fr) * 2017-12-15 2019-06-20 Somatix, Inc. Systèmes et méthodes de surveillance du bien-être d'un utilisateur
US11493993B2 (en) 2019-09-04 2022-11-08 Meta Platforms Technologies, Llc Systems, methods, and interfaces for performing inputs based on neuromuscular control
US11150730B1 (en) 2019-04-30 2021-10-19 Facebook Technologies, Llc Devices, systems, and methods for controlling computing devices via neuromuscular signals of users
US11961494B1 (en) 2019-03-29 2024-04-16 Meta Platforms Technologies, Llc Electromagnetic interference reduction in extended reality environments
US11567573B2 (en) 2018-09-20 2023-01-31 Meta Platforms Technologies, Llc Neuromuscular text entry, writing and drawing in augmented reality systems
US11481030B2 (en) 2019-03-29 2022-10-25 Meta Platforms Technologies, Llc Methods and apparatus for gesture detection and classification
US10937414B2 (en) 2018-05-08 2021-03-02 Facebook Technologies, Llc Systems and methods for text input using neuromuscular information
US11907423B2 (en) 2019-11-25 2024-02-20 Meta Platforms Technologies, Llc Systems and methods for contextualized interactions with an environment
EP3785275A1 (fr) * 2018-04-26 2021-03-03 Philip Morris Products S.a.s. Détection de consommation automatique destinée à être utilisée dans la conversion de la consommation d'articles de génération d'aérosol
CN110411460B (zh) * 2018-04-27 2021-08-27 高树安弗(杭州)科技有限公司 用于自适应地控制跟踪设备的方法和系统
US10592001B2 (en) 2018-05-08 2020-03-17 Facebook Technologies, Llc Systems and methods for improved speech recognition using neuromuscular information
EP3836834A4 (fr) * 2018-08-17 2021-09-29 LVL Technologies, Inc. Système et procédé de détection contextuelle de consommation de boissons
US10831280B2 (en) * 2018-10-09 2020-11-10 International Business Machines Corporation Augmented reality system for efficient and intuitive document classification
US20200135320A1 (en) 2018-10-31 2020-04-30 Medtronic Minimed, Inc. Automated detection of a physical behavior event and corresponding adjustment of a medication dispensing system based on historical events
CN113423341A (zh) * 2018-11-27 2021-09-21 脸谱科技有限责任公司 用于可穿戴电极传感器系统的自动校准的方法和装置
CN109737957B (zh) * 2019-01-31 2020-11-06 济南大学 一种采用级联FIR滤波的INS/LiDAR组合导航方法及系统
US20200326765A1 (en) * 2019-04-12 2020-10-15 XRSpace CO., LTD. Head mounted display system capable of indicating a tracking unit to track a hand gesture or a hand movement of a user or not, related method and related non-transitory computer readable storage medium
CN110122929B (zh) * 2019-05-17 2021-07-30 南京大学 一种基于惯性传感器的抽烟事件监测系统及监测方法
JP7152357B2 (ja) * 2019-05-24 2022-10-12 株式会社日立製作所 正解データ作成支援システムおよび正解データ作成支援方法
CN110146657A (zh) * 2019-06-10 2019-08-20 中国烟草总公司郑州烟草研究院 一种评价卷烟落头倾向的方法
CN113924568A (zh) 2019-06-26 2022-01-11 谷歌有限责任公司 基于雷达的认证状态反馈
US11868537B2 (en) 2019-07-26 2024-01-09 Google Llc Robust radar-based gesture-recognition by user equipment
JP7316383B2 (ja) 2019-07-26 2023-07-27 グーグル エルエルシー Ｉｍｕおよびレーダーを介した認証管理
CN118192796A (zh) 2019-08-30 2024-06-14 谷歌有限责任公司 移动设备的输入方法
CN113892072A (zh) * 2019-08-30 2022-01-04 谷歌有限责任公司 用于暂停的雷达姿势的视觉指示器
US11032409B1 (en) * 2019-09-20 2021-06-08 Yellcast, Inc Methods for geographic gesturing using a mobile device for interactions with nearby other mobile devices
CN110779148B (zh) * 2019-11-14 2021-05-14 宁波奥克斯电气股份有限公司 一种基于毫米波的空调趣味控制方法、装置及空调器
IT202000001603A1 (it) * 2020-01-28 2021-07-28 St Microelectronics Srl Sistema e metodo di riconoscimento di un gesto di tocco
WO2021171280A1 (fr) * 2020-02-24 2021-09-02 Agt International Gmbh Suivi de la dynamique d'utilisateurs et d'objets à l'aide d'un dispositif informatisé
US11150737B2 (en) * 2020-03-16 2021-10-19 Facebook Technologies, Llc Apparatus, system, and method for wrist tracking and gesture detection via time of flight sensors
CN112216062B (zh) * 2020-09-11 2022-05-06 深圳市朗尼科智能股份有限公司 社区安全预警方法、装置、计算机设备以及系统
AU2021106501A4 (en) * 2020-10-23 2021-11-04 Elbaware Limited Device, system and method for determining face touching
US11868531B1 (en) 2021-04-08 2024-01-09 Meta Platforms Technologies, Llc Wearable device providing for thumb-to-finger-based input gestures detected based on neuromuscular signals, and systems and methods of use thereof
CN113288122B (zh) * 2021-05-21 2023-12-19 河南理工大学 一种可穿戴坐姿监测装置和坐姿监测方法
CN115240365B (zh) * 2022-07-19 2023-03-10 成都信息工程大学 一种融合多维信息的电力作业人员危险态势动态感知方法
CN116774597B (zh) * 2023-05-05 2024-07-05 杭州手趣科技有限公司 一种基于可穿戴设备的智能家居设备控制方法及系统
CN116649959B (zh) * 2023-05-31 2024-07-12 北京欧应科技有限公司 监测系统、判断佩戴装置定位的方法及存储介质

Citations (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020022774A1 (en) 2000-08-08 2002-02-21 Eddy Karnieli Method for monitoring food intake
US6605038B1 (en) 2000-06-16 2003-08-12 Bodymedia, Inc. System for monitoring health, wellness and fitness
US20070024581A1 (en) * 2005-08-01 2007-02-01 Samsung Electronics Co., Ltd. Apparatus and method for detecting motion with low power consumption in inertia sensor
US20080267444A1 (en) 2005-12-15 2008-10-30 Koninklijke Philips Electronics, N.V. Modifying a Person's Eating and Activity Habits
US20090012433A1 (en) 2007-06-18 2009-01-08 Fernstrom John D Method, apparatus and system for food intake and physical activity assessment
US20090023555A1 (en) 2007-04-26 2009-01-22 Heather Raymond Method and system for developing or tracking a program for medical treatment
US20090062696A1 (en) 2007-05-18 2009-03-05 Vaidhi Nathan Abnormal motion detector and monitor
US8073707B2 (en) 2000-06-16 2011-12-06 Bodymedia, Inc. System for detecting, monitoring, and reporting an individual's physiological or contextual status
US20120289244A1 (en) 2011-05-12 2012-11-15 Amit Goyal Contextually aware mobile device
US20130072765A1 (en) 2011-09-19 2013-03-21 Philippe Kahn Body-Worn Monitor
US8449471B2 (en) 2006-05-24 2013-05-28 Bao Tran Health monitoring appliance
US20130190903A1 (en) * 2012-01-19 2013-07-25 Nike, Inc. Action Detection and Activity Classification
US20130194066A1 (en) 2011-06-10 2013-08-01 Aliphcom Motion profile templates and movement languages for wearable devices
US20130229508A1 (en) * 2012-03-01 2013-09-05 Qualcomm Incorporated Gesture Detection Based on Information from Multiple Types of Sensors
US8531291B2 (en) 2005-10-16 2013-09-10 Bao Tran Personal emergency response (PER) system
US20140018686A1 (en) * 2011-03-29 2014-01-16 Pedro J. Medelius Data collection unit power and noise management
US20140139422A1 (en) 2012-11-20 2014-05-22 Samsung Electronics Company, Ltd. User Gesture Input to Wearable Electronic Device Involving Outward-Facing Sensor of Device
US20140149754A1 (en) 2012-11-29 2014-05-29 Amazon Technologies, Inc. Gesture detection management for an electronic device
US20140191955A1 (en) 2010-07-13 2014-07-10 Giuseppe Raffa Efficient gesture processing
US20140214448A1 (en) 2009-12-23 2014-07-31 Ai Cure Technologies, LLC Method and Apparatus for Verification of Medication Adherence
US20140335490A1 (en) 2011-12-07 2014-11-13 Access Business Group International Llc Behavior tracking and modification system
CA2838232A1 (fr) 2013-05-16 2014-11-16 Carepredict, Inc. Procedes et systemes pour determiner a distance des niveaux d'intervention en soins de sante
US20140347491A1 (en) 2013-05-23 2014-11-27 Robert A. Connor Smart Watch and Food-Imaging Member for Monitoring Food Consumption
US20140376773A1 (en) * 2013-06-21 2014-12-25 Leap Motion, Inc. Tunable operational parameters in motion-capture and touchless interface operation
US8968195B2 (en) 2006-05-12 2015-03-03 Bao Tran Health monitoring appliance
US20150078140A1 (en) 2008-12-07 2015-03-19 Apdm, Inc. Wearable Apparatus
US20150161516A1 (en) * 2013-12-06 2015-06-11 President And Fellows Of Harvard College Method and apparatus for detecting mode of motion with principal component analysis and hidden markov model
US20150164377A1 (en) 2013-03-13 2015-06-18 Vaidhi Nathan System and method of body motion analytics recognition and alerting
US20150254992A1 (en) * 2014-03-06 2015-09-10 Maneesh SETHI Memory-enhancing and habit-training methods and devices
US20160005299A1 (en) 2014-07-07 2016-01-07 Google Inc. Meal-based medication reminder system
WO2016100368A1 (fr) 2014-12-16 2016-06-23 Somatix, Inc. Procédés et systèmes pour surveiller et influencer des comportements basés sur un geste
US20160206921A1 (en) 2013-11-04 2016-07-21 Intel Corporartion Detection of biking, walking, and running
US20170220772A1 (en) 2016-01-28 2017-08-03 Savor Labs, Inc. Method and apparatus for tracking of food intake and other behaviors and providing relevant feedback

Family Cites Families (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004008573A (ja) * 2002-06-07 2004-01-15 Seiko Instruments Inc 遠隔健康診断装置、及びその診断方法、並びに遠隔健康診断プログラム
JP2004024551A (ja) * 2002-06-26 2004-01-29 Renesas Technology Corp センサシステム用半導体装置
JP2006026037A (ja) * 2004-07-15 2006-02-02 Hitachi Ltd 健康管理支援システム
CN101198277B (zh) * 2005-02-22 2011-06-15 海尔思-斯玛特有限公司 用于生理学和心理生理学监控的系统
TWI348639B (en) * 2005-12-16 2011-09-11 Ind Tech Res Inst Motion recognition system and method for controlling electronic device
US7725547B2 (en) * 2006-09-06 2010-05-25 International Business Machines Corporation Informing a user of gestures made by others out of the user's line of sight
US20090262074A1 (en) * 2007-01-05 2009-10-22 Invensense Inc. Controlling and accessing content using motion processing on mobile devices
JP2009140107A (ja) * 2007-12-04 2009-06-25 Sony Corp 入力装置及び制御システム
KR100927009B1 (ko) * 2008-02-04 2009-11-16 광주과학기술원 증강 현실에서의 햅틱 상호 작용 방법 및 그 시스템
WO2010137325A1 (fr) * 2009-05-27 2010-12-02 パナソニック株式会社 Dispositif de reconnaissance de comportement
US8760392B2 (en) * 2010-04-20 2014-06-24 Invensense, Inc. Wireless motion processing sensor systems suitable for mobile and battery operation
CN102377871B (zh) * 2010-08-24 2013-12-04 联想(北京)有限公司 一种信息处理设备及其控制方法
US8919848B2 (en) * 2011-11-16 2014-12-30 Flextronics Ap, Llc Universal console chassis for the car
JP5703090B2 (ja) * 2011-03-30 2015-04-15 株式会社日立製作所 健康管理システム
US20130077820A1 (en) * 2011-09-26 2013-03-28 Microsoft Corporation Machine learning gesture detection
JP5885292B2 (ja) * 2011-11-08 2016-03-15 公立大学法人首都大学東京 行動認識プログラムおよび行動認識用の処理装置
US10776103B2 (en) * 2011-12-19 2020-09-15 Majen Tech, LLC System, method, and computer program product for coordination among multiple devices
GB2502087A (en) * 2012-05-16 2013-11-20 St Microelectronics Res & Dev Gesture recognition
JP6011127B2 (ja) * 2012-08-06 2016-10-19 富士ゼロックス株式会社 行為検出プログラム及び行為検出装置
WO2014046204A1 (fr) * 2012-09-19 2014-03-27 株式会社ニコン Équipement électronique et programme associé
JP6159556B2 (ja) 2012-09-20 2017-07-05 シャープ株式会社 非水系二次電池、並びに非水系二次電池用難燃剤及び添加剤
US9092600B2 (en) * 2012-11-05 2015-07-28 Microsoft Technology Licensing, Llc User authentication on augmented reality display device
US9865176B2 (en) * 2012-12-07 2018-01-09 Koninklijke Philips N.V. Health monitoring system
KR102037416B1 (ko) * 2012-12-17 2019-10-28 삼성전자주식회사 외부 기기 관리 방법, 외부 기기의 동작 방법, 호스트 단말, 관리 서버 및 외부 기기
US9174084B2 (en) * 2013-03-05 2015-11-03 Microsoft Technology Licensing, Llc Automatic exercise segmentation and recognition
US9442570B2 (en) * 2013-03-13 2016-09-13 Google Technology Holdings LLC Method and system for gesture recognition
US20140288876A1 (en) * 2013-03-15 2014-09-25 Aliphcom Dynamic control of sampling rate of motion to modify power consumption
RU2639954C2 (ru) * 2013-04-30 2017-12-25 Ска Хайджин Продактс Аб Способ и устройство для сбора и хранения данных, относящихся к состоянию впитывающего изделия
JP2014238812A (ja) * 2013-05-10 2014-12-18 株式会社リコー 情報処理装置、動作特定方法及び動作特定プログラム
IL229369A (en) * 2013-11-11 2017-12-31 Vestigo Tech Ltd Low-bandwidth system and methods for wireless data monitoring for moving devices
JP5613314B1 (ja) * 2013-11-14 2014-10-22 Ｊｆｅシステムズ株式会社 ジェスチャー検出装置、ジェスチャー検出プログラム、ジェスチャー認識装置およびジェスチャー認識プログラム
EP2911089B1 (fr) * 2014-02-25 2018-04-04 Karlsruher Institut für Technologie Procédé et système de reconnaissance de geste et d'écriture manuscrite
US9740396B1 (en) * 2014-06-25 2017-08-22 Amazon Technologies, Inc. Adaptive gesture recognition
US9699859B1 (en) * 2014-08-01 2017-07-04 Moov Inc. Configuration of an automated personal fitness coaching device
JP6727921B2 (ja) * 2016-05-18 2020-07-22 ソニーモバイルコミュニケーションズ株式会社 情報処理装置、情報処理システム、情報処理方法

Patent Citations (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6605038B1 (en) 2000-06-16 2003-08-12 Bodymedia, Inc. System for monitoring health, wellness and fitness
US9033875B2 (en) 2000-06-16 2015-05-19 Bodymedia, Inc. Multi-sensor system, device, and method for deriving human status information
US8073707B2 (en) 2000-06-16 2011-12-06 Bodymedia, Inc. System for detecting, monitoring, and reporting an individual's physiological or contextual status
US20020022774A1 (en) 2000-08-08 2002-02-21 Eddy Karnieli Method for monitoring food intake
US20070024581A1 (en) * 2005-08-01 2007-02-01 Samsung Electronics Co., Ltd. Apparatus and method for detecting motion with low power consumption in inertia sensor
US8531291B2 (en) 2005-10-16 2013-09-10 Bao Tran Personal emergency response (PER) system
US20080267444A1 (en) 2005-12-15 2008-10-30 Koninklijke Philips Electronics, N.V. Modifying a Person's Eating and Activity Habits
US8968195B2 (en) 2006-05-12 2015-03-03 Bao Tran Health monitoring appliance
US8449471B2 (en) 2006-05-24 2013-05-28 Bao Tran Health monitoring appliance
US9107586B2 (en) 2006-05-24 2015-08-18 Empire Ip Llc Fitness monitoring
US20090023555A1 (en) 2007-04-26 2009-01-22 Heather Raymond Method and system for developing or tracking a program for medical treatment
US20150190085A1 (en) 2007-05-18 2015-07-09 Smart Monitor Corp. Abnormal motion detector and monitor
US20090062696A1 (en) 2007-05-18 2009-03-05 Vaidhi Nathan Abnormal motion detector and monitor
US20090012433A1 (en) 2007-06-18 2009-01-08 Fernstrom John D Method, apparatus and system for food intake and physical activity assessment
US20150078140A1 (en) 2008-12-07 2015-03-19 Apdm, Inc. Wearable Apparatus
US20140214448A1 (en) 2009-12-23 2014-07-31 Ai Cure Technologies, LLC Method and Apparatus for Verification of Medication Adherence
US20140191955A1 (en) 2010-07-13 2014-07-10 Giuseppe Raffa Efficient gesture processing
US20140018686A1 (en) * 2011-03-29 2014-01-16 Pedro J. Medelius Data collection unit power and noise management
US20120289244A1 (en) 2011-05-12 2012-11-15 Amit Goyal Contextually aware mobile device
US20130194066A1 (en) 2011-06-10 2013-08-01 Aliphcom Motion profile templates and movement languages for wearable devices
US20130072765A1 (en) 2011-09-19 2013-03-21 Philippe Kahn Body-Worn Monitor
US20140335490A1 (en) 2011-12-07 2014-11-13 Access Business Group International Llc Behavior tracking and modification system
US20130190903A1 (en) * 2012-01-19 2013-07-25 Nike, Inc. Action Detection and Activity Classification
US20130229508A1 (en) * 2012-03-01 2013-09-05 Qualcomm Incorporated Gesture Detection Based on Information from Multiple Types of Sensors
US20140139422A1 (en) 2012-11-20 2014-05-22 Samsung Electronics Company, Ltd. User Gesture Input to Wearable Electronic Device Involving Outward-Facing Sensor of Device
US20140149754A1 (en) 2012-11-29 2014-05-29 Amazon Technologies, Inc. Gesture detection management for an electronic device
US20150164377A1 (en) 2013-03-13 2015-06-18 Vaidhi Nathan System and method of body motion analytics recognition and alerting
US9734295B1 (en) 2013-05-16 2017-08-15 Carepredict, Inc. Methods and systems for remotely determining levels of healthcare interventions
FR3005767A1 (fr) 2013-05-16 2014-11-21 Carepredict Inc Procede et dispositif de determination a distance de besoins d'assistance medicale
KR20140135591A (ko) 2013-05-16 2014-11-26 케어프리딕트, 인코퍼레이티드 의료 중재 레벨을 원격으로 판단하기 위한 방법 및 시스템
CA2838232A1 (fr) 2013-05-16 2014-11-16 Carepredict, Inc. Procedes et systemes pour determiner a distance des niveaux d'intervention en soins de sante
US9798860B1 (en) 2013-05-16 2017-10-24 Carepredict, Inc. Methods and systems for remotely determining levels of healthcare interventions
US20170293729A1 (en) 2013-05-16 2017-10-12 Carepredict, Inc. Methods and Systems for Remotely Determining Levels of Healthcare Interventions
US20140347491A1 (en) 2013-05-23 2014-11-27 Robert A. Connor Smart Watch and Food-Imaging Member for Monitoring Food Consumption
US20140376773A1 (en) * 2013-06-21 2014-12-25 Leap Motion, Inc. Tunable operational parameters in motion-capture and touchless interface operation
US20160206921A1 (en) 2013-11-04 2016-07-21 Intel Corporartion Detection of biking, walking, and running
US20150161516A1 (en) * 2013-12-06 2015-06-11 President And Fellows Of Harvard College Method and apparatus for detecting mode of motion with principal component analysis and hidden markov model
US20150254992A1 (en) * 2014-03-06 2015-09-10 Maneesh SETHI Memory-enhancing and habit-training methods and devices
US20160005299A1 (en) 2014-07-07 2016-01-07 Google Inc. Meal-based medication reminder system
WO2016100368A1 (fr) 2014-12-16 2016-06-23 Somatix, Inc. Procédés et systèmes pour surveiller et influencer des comportements basés sur un geste
WO2017132690A1 (fr) 2016-01-28 2017-08-03 Savor Labs, Inc. Procédé et appareil de suivi de prise alimentaire et d'autres comportements et de fourniture d'un retour d'informations pertinentes
US20170220772A1 (en) 2016-01-28 2017-08-03 Savor Labs, Inc. Method and apparatus for tracking of food intake and other behaviors and providing relevant feedback

Non-Patent Citations (11)

* Cited by examiner, † Cited by third party
Title
Clinical Trial of Clickotine, Available at http://clicktherapeutics.com/publications/cns-summit.pdf, 1 page, Accessed on Jun. 19, 2017.
EP15870892.5 Extended Search Report dated Jun. 14, 2018.
Iacoviello, et al., Clickotine, A Personalized Smartphone App for Smoking Cessation: Initial Evaluation, JMIR Mhealth and Uhealth, 2017, 5(4):e56, 15 pages.
Iacoviello, et al., Six Month Outcomes, Available at http://clicktherapeutics.com/publications/society-for-research-on-nicotine-and-tobacco.pdf, 1 page, Accessed on Jun. 19, 2017.
International search report and written opinion dated Mar. 3, 2016 for PCT Application No. US2015/065862.
International search report dated Apr. 10, 2017 for PCT Application No. US2017-015682.
Novet, Google patents a wearable that tells you to take your medications when you eat, Available at https://venturebeat.com/2016/01/09/google-patents-a-wearable-that-tells-you-to -take-your-medications-when-you-eat/, 17 pages, Accessed on Jun. 19, 2017.
Pai, Researchers use mobile sensors to identify lapses in smoking cessation, stress, Oct. 14, 2015, MobiHealthNews, 2 pages.
Parate et al. RisQ: Recognizing Smoking Gestures with Inertial Sensors on a Wristband. MobiSys '14, Jun. 16-19, 2014, Bretton Woods, New Hampshire, USA. Copyright 2014 ACM. http://dx.doi.org/10.1145/2594368.2594379.
Saleem, et al., Self-Reported Depression, Available at http://clicktherapeutics.com/publications/saleem-2017-society-of-biological-psychiatry.pdf, 1 page, Accessed on Jun. 19, 2017.
Saleheen, et al., PuffMarker: A Multi-Sensor Approach for Pinpointing the Timing of First Lapse in Smoking Cessation, 12 pages.

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10474244B2 (en) 2014-12-16 2019-11-12 Somatix, Inc. Methods and systems for monitoring and influencing gesture-based behaviors
US11112874B2 (en) 2014-12-16 2021-09-07 Somatix, Inc. Methods and systems for monitoring and influencing gesture-based behaviors
US11550400B2 (en) 2014-12-16 2023-01-10 Somatix, Inc. Methods and systems for monitoring and influencing gesture-based behaviors
US11790698B2 (en) 2019-11-01 2023-10-17 Samsung Electronics Co., Ltd. Electronic device for recognizing gesture of user using a plurality of sensor signals

Also Published As

Publication number Publication date
JP2018506773A (ja) 2018-03-08
HK1245940A1 (zh) 2018-08-31
US10474244B2 (en) 2019-11-12
US20200225760A1 (en) 2020-07-16
JP7336549B2 (ja) 2023-08-31
US20170262064A1 (en) 2017-09-14
JP2022058756A (ja) 2022-04-12
CN107250947B (zh) 2020-07-28
WO2016100368A1 (fr) 2016-06-23
US11112874B2 (en) 2021-09-07
CN107250947A (zh) 2017-10-13
US11550400B2 (en) 2023-01-10
EP3234731B1 (fr) 2020-07-01
EP3234731A4 (fr) 2018-07-18
US20190079593A1 (en) 2019-03-14
US20230359282A1 (en) 2023-11-09
US20220155872A1 (en) 2022-05-19
EP3234731A1 (fr) 2017-10-25

Similar Documents

Publication Publication Date Title
US11550400B2 (en) Methods and systems for monitoring and influencing gesture-based behaviors
US20180056130A1 (en) Providing insights based on health-related information
US20170143246A1 (en) Systems and methods for estimating and predicting emotional states and affects and providing real time feedback
US20210059591A1 (en) Systems and methods for estimating and predicting emotional states and affects and providing real time feedback
US20150137994A1 (en) Data-capable band management in an autonomous advisory application and network communication data environment
US20180107943A1 (en) Periodic stress tracking
US20140085077A1 (en) Sedentary activity management method and apparatus using data from a data-capable band for managing health and wellness
US20140099614A1 (en) Method for delivering behavior change directives to a user
US20180107793A1 (en) Health activity monitoring and work scheduling
WO2017024161A1 (fr) Détection de schémas d'entraînement
US20120326873A1 (en) Activity attainment method and apparatus for a wellness application using data from a data-capable band
US20130002435A1 (en) Sleep management method and apparatus for a wellness application using data from a data-capable band
US20180060500A1 (en) Smart health activity scheduling
WO2017184393A1 (fr) Détection d'anomalie de niveau d'énergie d'utilisateur
US20210401338A1 (en) Systems and methods for estimating and predicting emotional states and affects and providing real time feedback
CN105996984B (zh) 利用可穿戴电子装置的久坐时间段检测
US20170100067A1 (en) Method and system for emotion measurement
CN108351846B (zh) 通信系统和通信控制方法
CA2933179A1 (fr) Gestion de bande a capacite d'utilisation de donnees

Legal Events

Date Code Title Description
AS Assignment

Owner name: SOMATIX, INC., NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:OFIR, ERAN;REEL/FRAME:046984/0514

Effective date: 20180926

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: SOMATIX, INC., NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:OFIR, ERAN;SCHATZBERG, URI;SIGNING DATES FROM 20180925 TO 20180926;REEL/FRAME:047478/0625

AS Assignment

Owner name: REMOT NERO, LLC, NEW YORK

Free format text: SECURITY INTEREST;ASSIGNOR:SOMATIX, INC.;REEL/FRAME:048111/0586

Effective date: 20190122

Owner name: MEIR AKKERMAN, ISRAEL

Free format text: SECURITY INTEREST;ASSIGNOR:SOMATIX, INC.;REEL/FRAME:048111/0586

Effective date: 20190122

Owner name: MARS HEALTH FUND I, LLC, NEW YORK

Free format text: SECURITY INTEREST;ASSIGNOR:SOMATIX, INC.;REEL/FRAME:048111/0586

Effective date: 20190122

Owner name: YNOR NERO, LLC, NEW YORK

Free format text: SECURITY INTEREST;ASSIGNOR:SOMATIX, INC.;REEL/FRAME:048111/0586

Effective date: 20190122

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

AS Assignment

Owner name: SOMATIX, INC., NEW YORK

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MARS HEALTH FUND I, LLC;REEL/FRAME:062630/0948

Effective date: 20221231