Classifications

• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/01—Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue
  • A61B5/015—By temperature mapping of body part
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/08—Detecting, measuring or recording devices for evaluating the respiratory organs
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
  • A61B5/163—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state by tracking eye movement, gaze, or pupil change
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
  • A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
  • A61B5/6802—Sensor mounted on worn items
  • A61B5/6803—Head-worn items, e.g. helmets, masks, headphones or goggles
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/74—Details of notification to user or communication with user or patient ; user input means
  • A61B5/746—Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
  • G01J5/0022—Radiation pyrometry, e.g. infrared or optical thermometry for sensing the radiation of moving bodies
  • G01J5/0025—Living bodies
• • G—PHYSICS
  • G02—OPTICS
  • G02C—SPECTACLES; SUNGLASSES OR GOGGLES INSOFAR AS THEY HAVE THE SAME FEATURES AS SPECTACLES; CONTACT LENSES
  • G02C11/00—Non-optical adjuncts; Attachment thereof
  • G02C11/10—Electronic devices other than hearing aids
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
  • A61B5/024—Detecting, measuring or recording pulse rate or heart rate
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/08—Detecting, measuring or recording devices for evaluating the respiratory organs
  • A61B5/0816—Measuring devices for examining respiratory frequency
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
  • A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/41—Detecting, measuring or recording for evaluating the immune or lymphatic systems
  • A61B5/411—Detecting or monitoring allergy or intolerance reactions to an allergenic agent or substance
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
  • G01J2005/0077—Imaging

Definitions

• This application relates to head-mounted systems to measure facial temperature and to capture images of the face.
• Manifestation of various physiological responses involves temperature changes at various regions of the human face; measuring temperatures and/or temperature changes at the various regions on the face may help determine the level of stress a person is feeling, an extent of an allergic reaction the person has, or how a user feels, e.g., whether the user is nervous, calm, or happy.
• manifestation of physiological responses can involve facial skin color changes (FSCC), such as slight color changes due to a cardiac pulse or expression of an emotional response.
• FSCC can be detected from visible -light images of various regions of the face.
• monitoring and analyzing measurements of the face can be useful for many health-related and life-logging related applications.
• collecting such data over time when people are going through their daily activities can be very difficult, typically involves utilizing cameras that need to be continually pointed at a person's face, and often involves performing various complex image analysis procedures, such as procedures involving image registration and face tracking.
• the measurements may be affected by various confounding factors such as thermal radiation directed at the face, touching the face, or consumption of certain substances (e.g., medication, alcohol, or caffeine).
• the measurements are to be collected over a long period of time, while the person performs various day- to-day activities in uncontrolled settings.
• Some aspects of this disclosure involve head-mounted systems that are utilized to take thermal measurements of a user's face to detect various physiological responses, such as an allergic reaction, stress, a headache, and more.
• these systems involve one or more head-mounted thermal cameras that may be physically coupled to a frame worn on the user's head and are utilized to take thermal measurements of one or more regions of interest (ROIs).
• ROIs regions of interest
• the thermal measurements can be analyzed to detect occurrences of one or more of the various physiological responses.
• the frame may belong to various head-mounted systems, ranging from eyeglasses to more sophisticated headsets, such as virtual reality systems, augmented reality systems, or mixed reality systems.
• systems described in this disclosure are intended for "real world", day-to-day, use.
• a confounding factor can be a cause of warming and/or cooling of certain ROIs the face, which is unrelated to a physiological response being detected, and as such, can reduce the accuracy of the detection of the physiological response.
• FSCC facial skin color changes
• Some examples of physiological responses whose manifestation involves FSCC include emotional responses (which at times may be hidden to the naked eye), and physiological signals such as a heart rate, heart rate variability, or a breathing rate.
• FIG. la and FIG. lb illustrate various inward-facing head-mounted cameras coupled to an eyeglasses frame
• FIG. 2 illustrates inward-facing head-mounted cameras coupled to an augmented reality device
• FIG. 3 illustrates head-mounted cameras coupled to a virtual reality device
• FIG. 4 illustrates a side view of head-mounted cameras coupled to an augmented reality device
• FIG. 5 illustrates a side view of head-mounted cameras coupled to a sunglasses frame
• FIG. 6 to FIG. 9 illustrate HMSs configured to measure various regions of interest (ROIs) relevant to some of the embodiments describes herein;
• ROIs regions of interest
• FIG. 10 to FIG. 13 illustrate various embodiments of systems that include inward-facing head- mounted cameras having multi-pixel sensors (FPA sensors);
• FPA sensors multi-pixel sensors
• FIG. 14a, FIG. 14b, and FIG. 14c illustrate embodiments of two right and left clip-on devices that re configured to attached/detached from an eyeglasses frame;
• FIG. 15a and FIG. 15b illustrate an embodiment of a clip-on device that includes inward-facing head-mounted cameras pointed at the lower part of the face and the forehead;
• FIG. 16a and FIG. 16b illustrate embodiments of right and left clip-on devices that are configured to be attached behind an eyeglasses frame
• FIG. 17a and FIG. 17b illustrate an embodiment of a single-unit clip-on device that is configured to be attached behind an eyeglasses frame;
• FIG. 18 illustrates embodiments of right and left clip-on devices, which are configured to be attached/detached from an eyeglasses frame, and have protruding arms to hold inward-facing head- mounted cameras;
• FIG. 19 illustrates a scenario in which an alert regarding a possible stroke is issued
• FIG. 20a and FIG. 20b illustrate the making of different detections of emotional response based on thermal measurements compared to the emotional response that is visible in a facial expression
• FIG. 21 illustrates an embodiment of a smartphone app that provides a user with feedback about how he/she looks to others
• FIG. 22 illustrates one embodiment of a tablet app that provides the user a feedback about how he/she felt during a certain period
• FIG. 23 illustrates an embodiment of the system configured to detect a physiological response based on facial skin color changes (FSCC);
• FSCC facial skin color changes
• FIG. 24a and FIG. 24b illustrate heating of a ROI for different reasons: sinusitis (which is detected), and acne (which is not detected as sinusitis);
• FIG. 25a and FIG. 25b illustrate an embodiment of a system that provides indications when the user touches his/her face
• FIG. 26a illustrates a first case where a user's hair does not occlude the forehead
• FIG. 26b illustrates a second case where a user's hair occludes the forehead and the system requests the user to move the hair in order to enable correct measurements of the forehead;
• FIG. 27a illustrates an embodiment of a system that detects a physiological response based on measurements taken by an inward-facing head-mounted thermal camera and an outward-facing head- mounted thermal camera;
• FIG. 27b illustrates a scenario in which a user receives an indication on a GUI that the user is not monitored in direct sunlight
• FIG. 28 illustrates that the effect of consuming alcohol on values of thermal measurements
• FIG. 29 illustrates an increase to thermal measurements due to smoking
• FIG. 30 illustrates a decrease to thermal measurements due to taking medication
• FIG. 31a and FIG. 31b are schematic illustrations of possible embodiments for computers.
• a "thermal camera” refers herein to a non-contact device that measures electromagnetic radiation having wavelengths longer than 2500 nanometer (nm) and does not touch its region of interest (ROI).
• a thermal camera may include one sensing element (pixel), or multiple sensing elements that are also referred to herein as “sensing pixels", “pixels”, and/or focal-plane array (FPA).
• a thermal camera may be based on an uncooled thermal sensor, such as a thermopile sensor, a microbolometer sensor (where microbolometer refers to any type of a bolometer sensor and its equivalents), a pyroelectric sensor, or a ferroelectric sensor.
• Sentences in the form of "thermal measurements of an ROI” refer to at least one of: (i) temperature measurements of the ROI (TR O I), such as when using thermopile or microbolometer sensors, and (ii) temperature change measurements of the ROI (ATR O I), such as when using a pyroelectric sensor or when deriving the temperature changes from temperature measurements taken at different times by a thermopile sensor or a microbolometer sensor.
• a device such as a thermal camera, may be positioned such that it occludes an ROI on the user's face, while in other embodiments, the device may be positioned such that it does not occlude the ROI
• Sentences in the form of "the system/camera does not occlude the ROI” indicate that the ROI can be observed by a third person located in front of the user and looking at the ROI, such as illustrated by all the ROIs in FIG. 7, FIG. 11 and FIG. 19.
• Sentences in the form of "the system/camera occludes the ROI” indicate that some of the ROIs cannot be observed directly by that third person, such as ROIs 19 and 37 that are occluded by the lenses in FIG. la, and ROIs 97 and 102 that are occluded by cameras 91 and 96, respectively, in FIG. 9.
• thermal cameras that do not occlude their ROIs on the face may provide one or more advantages to the user, to the HMS, and/or to the thermal cameras, which may relate to one or more of the following: esthetics, better ventilation of the face, reduced weight, simplicity to wear, and reduced likelihood to being tarnished.
• a "Visible-light camera” refers to a non-contact device designed to detect at least some of the visible spectrum, such as cameras with optical lenses and CMOS or CCD sensors.
• inward -facing head-mounted camera refers to a camera configured to be worn on a user's head and to remain pointed at its ROI, which is on the user's face, also when the user's head makes angular and lateral movements (such as movements with an angular velocity above 0.1 rad/sec, above 0.5 rad/sec, and/or above 1 rad/sec).
• a head-mounted camera (which may be inward-facing and/or outward-facing) may be physically coupled to a frame worn on the user's head, may be attached to eyeglass using a clip-on mechanism (configured to be attached to and detached from the eyeglasses), or may be mounted to the user's head using any other known device that keeps the camera in a fixed position relative to the user's head also when the head moves.
• Sentences in the form of "camera physically coupled to the frame” mean that the camera moves with the frame, such as when the camera is fixed to (or integrated into) the frame, or when the camera is fixed to (or integrated into) an element that is physically coupled to the frame.
• CAM denotes “inward-facing head-mounted thermal camera”
• CAM 0U t denotes “outward-facing head-mounted thermal camera”
• VCAM denotes “inward-facing head-mounted visible -light camera”
• VCAM 0Ut denotes "outward-facing head-mounted visible -light camera”.
• Sentences in the form of "a frame configured to be worn on a user's head” or "a frame worn on a user's head” refer to a mechanical structure that loads more than 50% of its weight on the user's head.
• an eyeglasses frame may include two temples connected to two rims connected by a bridge; the frame in Oculus RiftTM includes the foam placed on the user's face and the straps; and the frames in Google GlassTM and Spectacles by Snap Inc. are similar to eyeglasses frames.
• the frame may connect to, be affixed within, and/or be integrated with, a helmet (e.g., sports, motorcycle, bicycle, and/or combat helmets) and/or a brainwave-measuring headset.
• a helmet e.g., sports, motorcycle, bicycle, and/or combat helmets
• cameras are located close to a user's face, such as at most 2 cm, 5 cm, 10 cm, 1 cm, or 20 cm from the face (herein “cm” denotes to centimeters).
• cm denotes to centimeters.
• the head-mounted cameras used in various embodiments may be lightweight, such that each camera weighs below 10 g, 5 g, 1 g, and/or 0.5 g (herein "g” denotes to grams).
• FIG. la illustrates various inward-facing head-mounted cameras coupled to an eyeglasses frame 15.
• Cameras 10 and 12 measure regions 11 and 13 on the forehead, respectively.
• Cameras 18 and 36 measure regions on the periorbital areas 19 and 37, respectively.
• the HMS further includes an optional computer 16, which may include a processor, memory, a battery and/or a communication module.
• FIG. lb illustrates a similar HMS in which inward-facing head-mounted cameras 48 and 49 measure regions 41 and 41, respectively.
• Cameras 22 and 24 measure regions 23 and 25, respectively.
• Camera 28 measures region 29.
• cameras 26 and 43 measure regions 38 and 39, respectively.
• FIG. 2 illustrates inward-facing head-mounted cameras coupled to an augmented reality device such as Microsoft HoloLensTM.
• FIG. 3 illustrates head-mounted cameras coupled to a virtual reality device such as Facebook's Oculus RiftTM.
• FIG. 4 is a side view illustration of head-mounted cameras coupled to an augmented reality device such as Google GlassTM.
• FIG. 5 is another side view illustration of head-mounted cameras coupled to a sunglasses frame.
• FIG. 6 to FIG. 9 illustrate HMSs configured to measure various ROIs relevant to some of the embodiments describes herein.
• FIG. 6 illustrates a frame 35 that mounts inward-facing head-mounted cameras 30 and 31 that measure regions 32 and 33 on the forehead, respectively.
• FIG. 7 illustrates a frame 75 that mounts inward-facing head-mounted cameras 70 and 71 that measure regions 72 and 73 on the forehead, respectively, and inward-facing head-mounted cameras 76 and 77 that measure regions 78 and 79 on the upper lip, respectively.
• FIG. 8 illustrates a frame 84 that mounts inward- facing head-mounted cameras 80 and 81 that measure regions 82 and 83 on the sides of the nose, respectively.
• FIG. 9 illustrates a frame 90 that includes (i) inward-facing head-mounted cameras 91 and 92 that are mounted to protruding arms and measure regions 97 and 98 on the forehead, respectively, (ii) inward-facing head-mounted cameras 95 and 96, which are also mounted to protruding arms, which measure regions 101 and 102 on the lower part of the face, respectively, and (iii) head-mounted cameras 93 and 94 that measure regions on the periorbital areas 99 and 100, respectively.
• FIG. 10 to FIG. 13 illustrate various inward-facing head-mounted cameras having multi-pixel sensors (FPA sensors), configured to measure various ROIs relevant to some of the embodiments describes herein.
• FIG. 10 illustrates head-mounted cameras 120 and 122 that measure regions 121 and 123 on the forehead, respectively, and mounts head-mounted camera 124 that measure region 125 on the nose.
• FIG. 11 illustrates head-mounted cameras 126 and 128 that measure regions 127 and 129 on the upper lip, respectively, in addition to the head-mounted cameras already described in FIG. 10.
• FIG. 12 illustrates head-mounted cameras 130 and 132 that measure larger regions 131 and 133 on the upper lip and the sides of the nose, respectively.
• FIG. 13 illustrates head -mounted cameras 134 and 137 that measure regions 135 and 138 on the right and left cheeks and right and left sides of the mouth, respectively, in addition to the head-mounted cameras already described in FIG. 12.
• the head-mounted cameras may be physically coupled to the frame using a clip-on device configured to be attached/detached from a pair of eyeglasses in order to secure/release the device to/from the eyeglasses, multiple times.
• the clip-on device holds at least an inward-facing camera, a processor, a battery, and a wireless communication module.
• Most of the clip- on device may be located in front of the frame (as illustrated in FIG. 14b, FIG. 15b, and FIG. 18), or alternatively, most of the clip-on device may be located behind the frame, as illustrated in FIG. 16b and FIG. 17b.
• FIG. 14a, FIG. 14b, and FIG. 14c illustrate two right and left clip-on devices 141 and 142, respectively, configured to attached/detached from an eyeglasses frame 140.
• the clip-on device 142 includes an inward-facing head-mounted camera 143 pointed at a region on the lower part of the face (such as the upper lip, mouth, nose, and/or cheek), an inward-facing head-mounted camera 144 pointed at the forehead, and other electronics 145 (such as a processor, a battery, and/or a wireless communication module).
• the clip-on devices 141 and 142 may include additional cameras illustrated in the drawings as black circles.
• FIG. 15a and FIG. 15b illustrate a clip-on device 147 that includes an inward-facing head- mounted camera 148 pointed at a region on the lower part of the face (such as the nose), and an inward-facing head-mounted camera 149 pointed at the forehead.
• the other electronics such as a processor, a battery, and/or a wireless communication module is located inside the box 150, which also holds the cameras 148 and 149.
• FIG. 16a and FIG. 16b illustrate two right and left clip-on devices 160 and 161, respectively, configured to be attached behind an eyeglasses frame 165.
• the clip-on device 160 includes an inward-facing head-mounted camera 162 pointed at a region on the lower part of the face (such as the upper lip, mouth, nose, and/or cheek), an inward-facing head-mounted camera 163 pointed at the forehead, and other electronics 164 (such as a processor, a battery, and/or a wireless communication module).
• the clip-on devices 160 and 161 may include additional cameras illustrated in the drawings as black circles.
• FIG. 17a and FIG. 17b illustrate a single -unit clip-on device 170, configured to be attached behind an eyeglasses frame 176.
• the single -unit clip-on device 170 includes inward-facing head- mounted cameras 171 and 172 pointed at regions on the lower part of the face (such as the upper lip, mouth, nose, and/or cheek), inward-facing head-mounted cameras 173 and 174 pointed at the forehead, a spring 175 configured to apply force that holds the clip-on device 170 to the frame 176, and other electronics 177 (such as a processor, a battery, and/or a wireless communication module).
• the clip-on device 170 may include additional cameras illustrated in the drawings as black circles.
• FIG. 18 illustrates two right and left clip-on devices 153 and 154, respectively, configured to attached/detached from an eyeglasses frame, and having protruding arms to hold the inward-facing head-mounted cameras.
• Head-mounted camera 155 measures a region on the lower part of the face
• head-mounted camera 156 measures regions on the forehead
• the left clip-on device 154 further includes other electronics 157 (such as a processor, a battery, and/or a wireless communication module).
• the clip-on devices 153 and 154 may include additional cameras illustrated in the drawings as black circles.
• the elliptic and other shapes of the ROIs in some of the drawings are just for illustration purposes, and the actual shapes of the ROIs are usually not as illustrated. It is possible to calculate the accurate shape of an ROI using various methods, such as a computerized simulation using a 3D model of the face and a model of a head-mounted system (HMS) to which a thermal camera is physically coupled, or by placing a LED instead of the sensor (while maintaining the same field of view) and observing the illumination pattern on the face. Furthermore, illustrations and discussions of a camera represent one or more cameras, where each camera may have the same FOV and/or different FOVs.
• HMS head-mounted system
• the cameras may include one or more sensing elements (pixels), even when multiple sensing elements do not explicitly appear in the figures; when a camera includes multiple sensing elements then the illustrated ROI usually refers to the total ROI captured by the camera, which is made of multiple regions that are respectively captured by the different sensing elements.
• the positions of the cameras in the figures are just for illustration, and the cameras may be placed at other positions on the HMS.
• the ROI may cover another area (in addition to the area).
• a sentence in the form of "an ROI on the nose” may refer to either: 100% of the ROI is on the nose, or some of the ROI is on the nose and some of the ROI is on the upper lip.
• physiological responses include stress, an allergic reaction, an asthma attack, a stroke, dehydration, intoxication, or a headache (which includes a migraine).
• physiological responses include manifestations of fear, startle, sexual arousal, anxiety, joy, pain or guilt.
• physiological responses include physiological signals such as a heart rate or a value of a respiratory parameter of the user.
• detecting a physiological response may involve one or more of the following: determining whether the user has/had the physiological response, identifying an imminent attack associated with the physiological response, and/or calculating the extent of the physiological response.
• detection of the physiological response is done by processing thermal measurements that fall within a certain window of time that characterizes the physiological response.
• the window may be five seconds long, thirty seconds long, two minutes long, five minutes long, fifteen minutes long, or one hour long.
• Detecting the physiological response may involve analysis of thermal measurements taken during multiple of the above -described windows, such as measurements taken during different days.
• a computer may receive a stream of thermal measurements, taken while the user wears an HMS with coupled thermal cameras during the day, and periodically evaluate measurements that fall within a sliding window of a certain size.
• models are generated based on measurements taken over long periods.
• Sentences of the form of "measurements taken during different days” or “measurements taken over more than a week” are not limited to continuous measurements spanning the different days or over the week, respectively.
• “measurements taken over more than a week” may be taken by eyeglasses equipped with thermal cameras, which are worn for more than a week, 8 hours a day. In this example, the user is not required to wear the eyeglasses while sleeping in order to take measurements over more than a week.
• sentences of the form of "measurements taken over more than 5 days, at least 2 hours a day” refer to a set comprising at least 10 measurements taken over 5 different days, where at least two measurements are taken each day at times separated by at least two hours.
• Utilizing measurements taken of a long period may have an advantage, in some embodiments, of contributing to the generalizability of a trained model.
• Measurements taken over the long period likely include measurements taken in different environments and/or measurements taken while the measured user was in various physiological and/or mental states (e.g., before/after meals and/or while the measured user was sleepy/energetic/happy/depressed, etc.). Training a model on such data can improve the performance of systems that utilize the model in the diverse settings often encountered in real-world use (as opposed to controlled laboratory-like settings). Additionally, taking the measurements over the long period may have the advantage of enabling collection of a large amount of training data that is required for some machine learning approaches (e.g., "deep learning").
• Detecting the physiological response may involve performing various types of calculations by a computer.
• detecting the physiological response may involve performing one or more of the following operations: comparing thermal measurements to a threshold (when the threshold is reached that may be indicative of an occurrence of the physiological response), comparing thermal measurements to a reference time series, and/or by performing calculations that involve a model trained using machine learning methods.
• the thermal measurements upon which the one or more operations are performed are taken during a window of time of a certain length, which may optionally depend on the type of physiological response being detected.
• the window may be shorter than one or more of the following durations: five seconds, fifteen seconds, one minute, five minutes, thirty minute, one hour, four hours, one day, or one week.
• the window may be longer than one or more of the aforementioned durations.
• detection of the physiological response at a certain time may be done based on a subset of the measurements that falls within a certain window near the certain time; the detection at the certain time does not necessarily involve utilizing all values collected throughout the long period.
• detecting the physiological response of a user may involve utilizing baseline thermal measurement values, most of which were taken when the user was not experiencing the physiological response.
• detecting the physiological response may rely on observing a change to typical temperatures at one or more ROIs (the baseline), where different users might have different typical temperatures at the ROIs (i.e., different baselines).
• detecting the physiological response may rely on observing a change to a baseline level, which is determined based on previous measurements taken during the preceding minutes and/or hours.
• detecting a physiological response involves determining the extent of the physiological response, which may be expressed in various ways that are indicative of the extent of the physiological response, such as: (i) a binary value indicative of whether the user experienced, and/or is experiencing, the physiological response, (ii) a numerical value indicative of the magnitude of the physiological response, (iii) a catel value indicative of the severity/extent of the physiological response, (iv) an expected change in thermal measurements of an ROI (denoted THROI or some variation thereof), and/or (v) rate of change in THR O I.
• a physiological signal e.g., a heart rate, a breathing rate, and an extent of frontal lobe brain activity
• the extent of the physiological response may be interpreted as the value of the physiological signal.
• machine learning methods refers to learning from examples using one or more approaches.
• the approaches may be considered supervised, semi-supervised, and/or unsupervised methods.
• machine learning approaches include: decision tree learning, association rule learning, regression models, nearest neighbors classifiers, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, genetic algorithms, rule-based machine learning, and/or learning classifier systems.
• a "machine learning-based model” is a model trained using machine learning methods.
• a “machine learning-based model” may simply be called a “model”.
• a model refers to a model as being “machine learning-based” is intended to indicate that the model is trained using machine learning methods (otherwise, “model” may also refer to a model generated by methods other than machine learning).
• a computer is configured to detect the physiological response by generating feature values based on the thermal measurements (and possibly other values), and/or based on values derived therefrom (e.g., statistics of the measurements). The computer then utilizes the machine learning-based model to calculate, based on the feature values, a value that is indicative of whether, and/or to what extent, the user is experiencing (and/or is about to experience) the physiological response.
• calculating said value is considered "detecting the physiological response”.
• the value calculated by the computer is indicative of the probability that the user has/had the physiological response.
• feature values may be considered input to a computer that utilizes a model to perform the calculation of a value, such as the value indicative of the extent of the physiological response mentioned above.
• a “feature” typically refers to a certain type of value, and represents a property
• “feature value” is the value of the property with a certain instance (sample).
• a feature may be temperature at a certain ROI, while the feature value corresponding to that feature may be 36.9 0 C in one instance and 37.3 0 C in another instance.
• a machine learning-based model used to detect a physiological response is trained based on data that includes samples.
• Each sample includes feature values and a label.
• the feature values may include various types of values. At least some of the feature values of a sample are generated based on measurements of a user taken during a certain period of time (e.g., thermal measurements taken during the certain period of time). Optionally, some of the feature values may be based on various other sources of information described herein.
• the label is indicative of a physiological response of the user corresponding to the certain period of time.
• the label may be indicative of whether the physiological response occurred during the certain period and/or the extent of the physiological response during the certain period. Additionally or alternatively, the label may be indicative of how long the physiological response lasted.
• Labels of samples may be generated using various approaches, such as self -report by users, annotation by experts that analyze the training data, automatic annotation by a computer that analyzes the training data and/or analyzes additional data related to the training data, and/or utilizing additional sensors that provide data useful for generating the labels.
• a model is trained based on certain measurements (e.g., "a model trained based on TH RO i taken on different days"), it means that the model was trained on samples comprising feature values generated based on the certain measurements and labels corresponding to the certain measurements.
• a label corresponding to a measurement is indicative of the physiological response at the time the measurement was taken.
• Various types of feature values may be generated based on thermal measurements.
• some feature values are indicative of temperatures at certain ROIs.
• other feature values may represent a temperature change at certain ROIs. The temperature changes may be with respect to a certain time and/or with respect to a different ROI.
• some feature values may describe temperatures (or temperature changes) at a certain ROI at different points of time.
• these feature values may include various functions and/or statistics of the thermal measurements such as minimum/maximum measurement values and/or average values during certain windows of time.
• feature values are generated based on data comprising multiple sources, it means that for each source, there is at least one feature value that is generated based on that source (and possibly other data).
• stating that feature values are generated from thermal measurements of first and second ROIs means that the feature values may include a first feature value generated based on THROII and a second feature value generated based on TH OO.
• a sample is considered generated based on measurements of a user (e.g., measurements comprising THROII and THROU) when it includes feature values generated based on the measurements of the user.
• At least some feature values utilized by a computer may be generated based on additional sources of data that may affect temperatures measured at various facial ROIs.
• the additional sources include: (i) measurements of the environment such as temperature, humidity level, noise level, elevation, air quality, a wind speed, precipitation, and infrared radiation; (ii) contextual information such as the time of day (e.g., to account for effects of the circadian rhythm), day of month (e.g., to account for effects of the lunar rhythm), day in the year (e.g., to account for seasonal effects), and/or stage in a menstrual cycle; (iii) information about the user being measured such as sex, age, weight, height, and/or body build.
• measurements of the environment such as temperature, humidity level, noise level, elevation, air quality, a wind speed, precipitation, and infrared radiation
• contextual information such as the time of day (e.g., to account for effects of the circadian rhythm), day of month (e.g., to account for effects of the lunar rhythm), day in the year (e.g., to account for seasonal effects), and/or stage in a menstrual cycle
• At least some feature values may be generated based on physiological signals of the user obtained by sensors that are not thermal cameras, such as a visible -light camera, a photoplethysmogram (PPG) sensor, an electrocardiogram (ECG) sensor, an electroencephalography (EEG) sensor, a galvanic skin response (GSR) sensor, or a thermistor.
• sensors that are not thermal cameras, such as a visible -light camera, a photoplethysmogram (PPG) sensor, an electrocardiogram (ECG) sensor, an electroencephalography (EEG) sensor, a galvanic skin response (GSR) sensor, or a thermistor.
• PPG photoplethysmogram
• ECG electrocardiogram
• EEG electroencephalography
• GSR galvanic skin response
• the machine learning-based model used to detect a physiological response may be trained, in some embodiments, based on data collected in day-to-day, real world scenarios. As such, the data may be collected at different times of the day, while users perform various activities, and in various environmental conditions. Utilizing such diverse training data may enable a trained model to be more resilient to the various effects different conditions can have on the values of thermal measurements, and consequently, be able to achieve better detection of the physiological response in real world day- to-day scenarios.
• a confounding factor can be a cause of warming and/or cooling of certain regions of the face, which is unrelated to a physiological response being detected, and as such, may reduce the accuracy of the detection of the physiological response.
• confounding factors include: (i) environmental phenomena such as direct sunlight, air conditioning, and/or wind; (ii) things that are on the user's face, which are not typically there and/or do not characterize the faces of most users (e.g., cosmetics, ointments, sweat, hair, facial hair, skin blemishes, acne, inflammation, piercings, body paint, and food leftovers); (iii) physical activity that may affect the user's heart rate, blood circulation, and/or blood distribution (e.g., walking, running, jumping, and/or bending over); (iv) consumption of substances to which the body has a physiological response that may involve changes to temperatures at various facial ROIs, such as various medications, alcohol, caffeine, tobacco, and/or certain types of food; and/or (v) disruptive facial movements (e.g., frowning, talking, eating, drinking, sneezing, and coughing).
• environmental phenomena such as direct sunlight, air conditioning, and/or wind
• systems may incorporate measures designed to accommodate for the confounding factors.
• these measures may involve generating feature values that are based on additional sensors, other than the thermal cameras.
• these measures may involve refraining from detecting the physiological response, which should be interpreted as refraining from providing an indication that the user has the physiological response. For example, if an occurrence of a certain confounding factor is identified, such as strong directional sunlight that heats one side of the face, the system may refrain from detecting that the user had a stroke. In this example, the user may not be alerted even though a temperature difference between symmetric ROIs on both sides of the face reaches a threshold that, under other circumstances, would warrant alerting the user.
• Training data used to train a model for detecting a physiological response may include, in some embodiments, a diverse set of samples corresponding to various conditions, some of which involve occurrence of confounding factors (when there is no physiological response and/or when there is a physiological response). Having samples in which a confounding factor occurs (e.g., the user is in direct sunlight or touches the face) can lead to a model that is less susceptible to wrongfully detect the physiological response (which may be considered an occurrence of a false positive) in real world situations.
• the model may be provided for use by a system that detects the physiological response.
• Providing the model may involve performing different operations, such as forwarding the model to the system via a computer network and/or a shared computer storage medium, storing the model in a location from which the system can retrieve the model (such as a database and/or cloud-based storage), and/or notifying the system regarding the existence of the model and/or regarding an update to the model.
• a model for detecting a physiological response may include different types of parameters.
• the model comprises parameters of a decision tree.
• the computer simulates a traversal along a path in the decision tree, determining which branches to take based on the feature values.
• a value indicative of the physiological response may be obtained at the leaf node and/or based on calculations involving values on nodes and/or edges along the path;
• the model comprises parameters of a regression model (e.g., regression coefficients in a linear regression model or a logistic regression model).
• the computer multiplies the feature values (which may be considered a regressor) with the parameters of the regression model in order to obtain the value indicative of the physiological response; and/or (c) the model comprises parameters of a neural network.
• the parameters may include values defining at least the following: (i) an interconnection pattern between different layers of neurons, (ii) weights of the interconnections, and (iii) activation functions that convert each neuron's weighted input to its output activation.
• the computer provides the feature values as inputs to the neural network, computes the values of the various activation functions and propagates values between layers, and obtains an output from the network, which is the value indicative of the physiological response.
• a user interface may be utilized, in some embodiments, to notify the user and/or some other entity, such as a caregiver, about the physiological response and/or present an alert responsive to an indication that the extent of the physiological response reaches a threshold.
• the UI may include a screen to display the notification and/or alert, a speaker to play an audio notification, a tactile UI, and/or a vibrating UI.
• "alerting" about a physiological response of a user refers to informing about one or more of the following: the occurrence of a physiological response that the user does not usually have (e.g., a stroke, intoxication, and/or dehydration), an imminent physiological response (e.g., an allergic reaction, an epilepsy attack, and/or a migraine), and an extent of the physiological response reaching a threshold (e.g., stress and/or anger reaching a predetermined level).
• a physiological response that the user does not usually have e.g., a stroke, intoxication, and/or dehydration
• an imminent physiological response e.g., an allergic reaction, an epilepsy attack, and/or a migraine
• an extent of the physiological response reaching a threshold e.g., stress and/or anger reaching a predetermined level
• Thermal measurements of a user's face can be useful for various applications such as detection of physiological responses that may manifest through temperature changes to various regions on the face.
• physiological responses include manifestation of emotional responses (e.g., fear and anxiety) or physiological signals (e.g., a heart rate, a breathing rate, and brain activity).
• emotional responses e.g., fear and anxiety
• physiological signals e.g., a heart rate, a breathing rate, and brain activity.
• collecting such data over time when people are going through their daily activities can be very difficult.
• collection of such data involves utilizing thermal cameras that are bulky, expensive and need to be continually pointed at a person's face.
• collecting the required measurements often involves performing various complex image analysis procedures, such as procedures involving image registration and face tracking.
• Another challenge involve in detecting physiological responses based on thermal measurements of regions on the face involves influence of various confounding factors such as facial movements (e.g., facial expressions, talking, or eating) and the presence of various substances on the face (e.g., makeup, facial hair, or sweat). These confounding factors can change the thermal measurements and lead to errors in the detection of the physiological responses. These confounding factors are often not easily identified from the thermal measurements.
• facial movements e.g., facial expressions, talking, or eating
• substances on the face e.g., makeup, facial hair, or sweat
• the measurements are to be collected over a long period of time, while the person performs various day-to-day activities. Furthermore, it may be beneficial to account for confounding factors in the process of detection of physiological responses based on thermal measurements.
• a system configured to detect a physiological response includes an inward-facing head-mounted thermal camera (CAM), an inward-facing head- mounted visible-light camera (VCAM), and a computer.
• CAM head-mounted thermal camera
• VCAM visible-light camera
• the system may optionally include additional elements such as a frame and additional cameras.
• CAM is worn on a user's head and takes thermal measurements of a first ROI (THROII) on the user's face.
• THROII first ROI
• CAM weighs below 10 g.
• CAM is located less than 15 cm from the user's face.
• CAM utilizes a microbolometer or a thermopile sensor.
• CAM includes a focal-plane array (FPA) sensor and an infrared lens, and the FPA plane is tilted by more than 2° relative to the infrared lens plane according to the Scheimpfiug principle in order to improve the sharpness of the image of ROIi (where the lens plane refers to a plane that is perpendicular to the optical axis of the lens, which may include one or more lenses).
• FPA focal-plane array
• VCAM is worn on the user's head and takes images of a second ROI (IMROO) on the user's face.
• IMROO a second ROI
• VCAM weighs below 10 g and is located less than 15 cm from the face.
• ROIi and ROI 2 overlap (which means extend over so as to cover at least partly).
• ROI 2 may cover at least half of the area covered by ROIi.
• VCAM includes a multi-pixel sensor and a lens, and the sensor plane is tilted by more than 2° relative to the lens plane according to the Scheimpflug principle in order to improve the sharpness of the image of ROI 2 .
• the system may be constructed in a way that none of the system's components (including the frame and cameras) occludes ROIi and/or ROI 2 .
• the system may be constructed in a way that at least some of the system components (e.g., the frame and/or CAM) may occlude ROIi and/or ROI 2 .
• the computer detects the physiological response based on THROII, IMROO, and a model.
• the model includes one or more thresholds to which THROII and/or IMROO may be compared in order to detect the physiological response.
• the model includes one or more reference time series to which THROII and/or IMROO may be compared in order to detect the physiological response.
• the computer detects the physiological response by generating feature values based on THROII and IMROO, and utilizing the model to calculate, based on the feature values, a value indicative of the extent of the physiological response.
• the model may be referred to as a "machine learning-based model".
• At least some of the feature values, which are generated based on IM ROO may be used to identify, and/or account for, various confounding factors that can alter TH ROII without being directly related to the physiological response.
• detections of the physiological responses based on THROII and IMROO are more accurate than detections of the physiological responses based on THROII without IMROO.
• the physiological response is indicative of an occurrence of at least one of the following emotional states of the user: joy, fear, sadness, and anger.
• the physiological response is indicative of an occurrence of one or more of the following: stress, mental workload, an allergic reaction, a headache, dehydration, intoxication, and a stroke.
• the physiological response may be a physiological signal of the user.
• the physiological response is a heart rate of the user, and in this example, ROIi is on the skin above at least one of the superficial temporal artery and the frontal superficial temporal artery.
• the physiological response is frontal lobe brain activity of the user, and in this example, ROIi is on the forehead.
• the physiological signal is a breathing rate of the user, and ROIi is on the nasal area.
• a machine learning-based model used to detect a physiological response is typically trained on samples, where each sample includes feature values generated based on THROII and IM OO taken during a certain period, and a label indicative of the physiological response of the user during the certain period.
• the model is trained on samples generated based on measurements of the user (in which case the model may be considered a personalized model of the user).
• the model is trained on samples generated based on measurements of one or more other users.
• the samples are generated based on measurements taken while the user being measured was in different situations.
• the samples are generated based on measurements taken on different days.
• images such as IMR O E may be utilized to generate various types of feature values, which may be utilized to detect the physiological response and/or detect an occurrence of a confounding factor.
• Some of the feature values generated based on images may include high- level facial -related feature values and their derivatives, such as location and dimensions of facial features and/or landmarks, identification of action units (AUs) in sequences of images, and/or blendshape weights.
• features include various low-level features such as features generated using Gabor filters, local binary patterns (LBP) and their derivatives, algorithms such as SIFT and/or SURF (and their derivatives), image keypoints, histograms of oriented gradients (HOG) descriptors, and statistical procedures such independent component analysis (ICA), principal component analysis (PCA), or linear discriminant analysis (LDA).
• feature values may include features derived from multiple images taken at different times, such as volume local binary patterns (VLBP), cuboids, and/or optical strain-based features. Additionally, some of the feature values may be based on other data, such as feature values generated based audio processing of data received from a head-mounted microphone. The audio processing may detect noises associated with talking, eating, and drinking, and convert it to feature values to be provided to the machine learning-based model.
• THROII and IMROU may confer some advantages in some embodiments. For example, there may be times when THROII and IMROEZ provide complementing signals of a physiological response (e.g., due to their ability to measure manifestations of different physiological processes related to the physiological response). This can increase the accuracy of the detections.
• the computer may identify facial expressions from IMROI2, and detect the emotional response of the user based on THROII and the identified facial expressions. For example, at least some of the feature values generated based on IMROI2, which are used to detect the emotional response, are indicative of the facial expressions.
• detections of emotional responses based on both THROII and the identified facial expressions are more accurate than detections of the emotional responses based on either THROII or the identified facial expressions.
• ROIi and ROI2 are on the mouth
• IMROI2 are indicative of a change in a facial expression during a certain period that involves a transition from a facial expression in which the lips are in contact to a facial expression with an open mouth.
• the computer may be able attribute a change in TH ROII to opening the mouth rather than a change in the temperature of the lips.
• ROIi and ROI 2 are on the nose and upper lip
• IM OO are indicative of a change in a facial expression during a certain period that involves a transition from a neutral facial expression to a facial expression of disgust.
• the computer may be able attribute a change in TH ROII to a raised upper lip and wrinkled nose instead of a change in the temperature of the nose and upper lip.
• ROIi and ROI2 are on the user's forehead located about 1 cm above at least one of the user's eyebrows, and IM ROO are indicative of a change in a facial expression during a certain period that involves a transition from a neutral expression to a facial expression involving raised eyebrows.
• the computer may be able attribute a change in TH ROII to raising the eyebrows instead of a change in the temperature of forehead.
• Zeng, Zhihong, et al. A survey of affect recognition methods: Audio, visual, and spontaneous expressions.” IEEE transactions on pattern analysis and machine intelligence 31.1 (2009): 39-58, describes some of the algorithmic approaches that may be used for this task.
• THROII and IMROE may provide different and even possibly contradicting indications regarding the physiological response.
• facial expressions may not always express how a user truly feels. For example, when in company of other people, a user may conceal his or her true feelings by making non-genuine facial expressions. However, at the same time, thermal measurements of the user's face may reveal the user's true emotions. Thus, a system that relies only on IMROE to determine the user's emotional response may be mistaken at times, and using THROII can help make detections more accurate.
• the computer responsive to receiving a first set of THROII and IMROE taken during a first period in which the user expressed a certain facial expression, the computer detects a first emotional response of the user. Additionally, responsive to receiving a second set of THROII and IMROE taken during a second period in which the user expressed again the certain facial expression, the computer detects a second emotional response of the user, which is not the same as the first emotional response.
• the computer detected different emotional responses in this example because THR O II of the first set are indicative of a first physiological response that reaches a threshold, while THR O II of the second set are indicative of a second physiological response that does not reach the threshold. Following are some more detailed examples of situations in which this may occur.
• the first set includes IMR O E indicative of a facial expression that is a smile and THR O II indicative of stress below a certain threshold, and the first emotional response detected by the computer is happiness.
• the second set in this example includes IMR O E indicative of a facial expression that is a smile and THR O II indicative of stress above the certain threshold, and the second emotional response detected by the computer is discomfort.
• the first set includes IMR O E indicative of a facial expression that is a neutral expression and THR O II indicative of stress below a certain threshold, and the first emotional response detected by the computer is comfort.
• the second set includes IMR O E indicative of a facial expression that is neutral and THR O II indicative of stress above the certain threshold, and the second emotional response detected by the computer is concealment.
• the first set includes IMR O E indicative of a facial expression that is an expression of anger and THR O II indicative of stress above a certain threshold, and the first emotional response detected by the computer is anger.
• the second set includes IMR O E indicative of a facial expression that is an expression of anger and THR O II indicative of stress below the certain threshold, and the second emotional response detected by the computer is indicative of pretending to be angry.
• FIG. 20a and FIG. 20b The phenomenon of making different detections based on thermal measurements compared to the emotional response that is visible in a facial expression is illustrated in FIG. 20a and FIG. 20b.
• the illustrated figures include an HMS with CAM 514 and VCAM 515 that may cover portions of a cheek, mouth and/or nose.
• FIG. 20a illustrates a case in which the user's smiling face may be mistaken for happiness; however, the cold nose indicates that the user is in fact stressed.
• FIG. 20b illustrates a case in which the facial expression indicates that the user is in a neutral state; however, the warm nose indicates that the user is excited.
• FIG. 20a and FIG. 20b also illustrate a second CAM 516 and a second VCAM 517, which may be utilized in some embodiments, as described herein.
• FIG. 21 illustrates one embodiment of a smartphone app that provides the user with feedback about how he/she looks to others.
• the illustrated app shows that the user was happy 96 time and angry 20 times. Because the purpose of this app is to measure how the user looks to others, the computer counts the facial expressions based on IMROD without correcting the facial expressions according THROII.
• FIG. 22 illustrates one embodiment of a tablet app that provides the user a feedback about how he/she felt during a certain period (e.g., during the day, the week, or while being at a certain location).
• the illustrated app shows that the user felt sad 56 minutes and happy 135 minutes. Because the purpose of this app is to measure how the user feels (and not just how the user looks to others), the computer determines the user's emotional state based on a combined analysis of IMROE and THROII , as exemplified above.
• the system may include a second inward-facing head-mounted thermal camera (CAM2) that takes thermal measurements of a third ROI (THR O D) on the face.
• CAM2 weighs below 10 g and is physically coupled to the frame.
• the center of ROIi is to the right of the center of the third region of interest (RO3 ⁇ 4), and the symmetric overlapping between ROIi and ROI 3 is above 50%.
• the computer accounts for facial thermal asymmetry, based on a difference between THROII and THROD-
• the symmetric overlapping is considered with respect to the vertical symmetry axis that divides the face to the right and left portions.
• the symmetric overlapping between ROIi and ROI 3 may be observed by comparing the overlap between ROIi and a mirror image of ROI 3 , where the mirror image is with respect to a mirror that is perpendicular to the front of the face and whose intersection with the face is along the vertical symmetry axis (which goes through the middle of the forehead and the middle of the nose).
• Some examples of calculations that may be performed by the computer to account for thermal asymmetry include: (i) utilizing different thresholds to which THROII and THROD are compared; (ii) utilizing different reference time series to which THR O II and THR O D are compared; (iii) utilizing a machine learning-based model that provides different results for first and second events that involve the same average change in THROII and THROD with different extents of asymmetry in THROII and THR O D; and (iv) utilizing the asymmetry for differentiating between (a) temperature changes in THR O II and THR O D that are related to the physiological response and (b) temperature changes in THROII and THROD that are unrelated to the physiological response.
• the system may include a second inward-facing head-mounted visible-light camera (VCAM2) that takes images of a third ROI (IMROD) on the face.
• VCAM2 weighs below 10 g and is physically coupled to the frame.
• VCAM and VCAM2 are located at least 0.5 cm to the right and to the left of the vertical symmetry axis that divides the face, respectively, and the symmetric overlapping between ROI 2 and ROI 3 is above 50%.
• the computer detects the physiological response also based on IMROD- For example, the computer may generate some feature values based on IMROD, which may be similar to feature values generated based on IMROD, and utilizes the some feature values in the detection of the physiological response. In another example, the computer detects the physiological response based on the extent of symmetry between symmetric facial elements extracted from IMROEZ and IMROB.
• IMROU may include recognizable facial skin color changes (FSCC).
• FSCC facial skin color changes
• FSCC are typically a result of changes in the concentration levels of hemoglobin and blood oxygenation under a user's facial skin, and are discussed in more detail elsewhere in this disclosure.
• the computer calculates, based on IMROO, a value indicative of FSCC, and detects an emotional state of the user based on the calculated value.
• detections of the physiological response based on both THROII and FSCC are more accurate than detections of the physiological response based on either THR O II or FSCC.
• the computer generates feature values that are indicative of FSCC in IMR O E, and utilizes a model to detect the physiological response based on the feature values.
• the model was trained based on samples, with each sample including feature values generated based on corresponding measurements of the user and a label indicative of the physiological response.
• the label may be derived, for example, from analysis of the user's speech/writing, facial expression analysis, speech emotion analysis, and/or emotion extraction from analyzing galvanic skin response (GSR) and heart rate variability (HRV).
• IMR O I 2 may be utilized, in some embodiments, to detect occurrences of confounding factors that can affect the temperature on the face, but are unrelated to the physiological response being detected. Thus, occurrences of confounding factors can reduce the accuracy of detections of the physiological response based on thermal measurements (such as based on THR O II). Detecting occurrences of the confounding factors described below (cosmetics, sweat, hair, inflammation and touching) may be done utilizing various image -processing and/or image-analysis techniques known in the art.
• detecting occurrences of at least some of the confounding factors described below may involve a machine learning algorithm trained to detect the confounding factors, and/or comparing IMR O I 2 to reference images that involve and do not involve the confounding factor (e.g., a first set of reference IMROK in which makeup was applied to the face and a second set of reference IMROO in which the face was bare of makeup).
• a machine learning algorithm trained to detect the confounding factors, and/or comparing IMR O I 2 to reference images that involve and do not involve the confounding factor (e.g., a first set of reference IMROK in which makeup was applied to the face and a second set of reference IMROO in which the face was bare of makeup).
• the computer may utilize detection of confounding factors in various ways in order to improve the detection of the physiological response based on THROII.
• the computer may refrain from making a detection of the physiological response responsive to identifying that the extent of a certain confounding factor reaches a threshold. For example, certain physiological responses may not be detected if there is extensive facial hair on the face or extensive skin inflammation.
• the model used to detect the physiological response may include a certain feature that corresponds to a certain confounding factor, and the computer may generate a certain feature value indicative of the extent of the certain confounding factor.
• the model in this case may be trained on samples in which the certain feature has different values, such as some of the samples used to train the model are generated based on measurements taken while the certain confounding factor occurred, and other samples used to train the model were generated based on measurements taken while the certain confounding factor did not occur.
• the computer may weight measurements based on the occurrence of confounding factors, such that measurements taken while certain confounding factors occurred, may be given lower weights than measurements taken while the certain confounding factor did not occur.
• lower weights for measurements mean that they have a smaller influence on the detection of the physiological response than measurements with higher weights.
• confounding factors that may be detected, in some embodiments, based on IMROK-
• Some types of cosmetics may mask an ROI, affect the ROFs emissivity, and/or affect the ROFs temperature.
• cosmetics may mask an ROI, affect the ROFs emissivity, and/or affect the ROFs temperature.
• the model was trained on: samples generated based on a first set of THROII and IMROE taken after cosmetics were applied to a portion of the overlapping region between ROIi and ROI 2 , and other samples generated based on a second set of THROII and IMROE taken while the overlapping region was bare of cosmetics.
• utilizing this model may enable the computer to account for presence of cosmetics on a portion of ROI 2 .
• Sweating may affect the ROFs emissivity.
• the model was trained on: samples generated from a first set of THROII and IMROE taken while sweat was detectable on a portion of the overlapping region between ROII and ROI2, and additional samples generated from a second set of THROII and IMROE taken while sweat was not detectable on the overlapping region.
• utilizing this model may enable the computer to account for sweat on the overlapping region.
• Dense hair may affect the ROFs emissivity, which may make the ROI appear, in thermal imaging, colder than it really is.
• hair density both referred to as hair density
• the model was trained on: samples generated from a first set of THROII and IMROO taken while hair density on a portion of the overlapping region between ROIi and ROF was at a first level, and additional samples generated from a second set of THROII and IMROU taken while hair density on the portion of the overlapping region between ROIi and ROI 2 was at a second level higher than the first level.
• utilizing a model trained so may enable the computer to account for hair on the overlapping region.
• the system may request the user to move her hair in order to enable the thermal cameras to take correct measurements.
• FIG. 26a illustrates a first case where the user's hair does not occlude the forehead.
• FIG. 26b illustrates a second case where the user's hair does occlude the forehead and thus the system requests the user to move the hair in order to enable correct measurements of the forehead.
• FIG. 24a illustrates heating of the ROI because of sinusitis, for which the system detects the physiological response (sinusitis).
• FIG. 24b illustrates heating of the same ROI because of acne, for which the system does detect sinusitis.
• the model was trained on: samples generated from a first set of TH OII and IMROO taken while skin inflammation was detectable on a portion of the overlapping region between ROIi and ROI2, and additional samples generated from a second set of THROII and IM ROE taken while skin inflammation was not detectable on the overlapping region.
• utilizing a model trained so may enable the computer to account for skin inflammation on the overlapping region.
• Touching the ROI may affect TH ROI by increasing or decreasing the temperature at the touched region.
• touching the ROI may be considered a confounding factor that can make detections of the physiological response less accurate.
• the model was trained on: samples generated from a first set of THROII and IMROE taken while detecting that the user touches a portion of the overlapping region between ROIi and ROI 2 , and additional samples generated from a second set of THROII and IMROE taken while detecting that the user does not touch the overlapping region.
• utilizing a model trained so may enables the computer to account for touching the overlapping region.
• FIG. 25a and FIG. 25b illustrate one embodiment of a system that provides indications when the user touches his/her face.
• the system includes a frame 533, head-mounted sensors (530, 531 , 532) able to detect touching the face, and head-mounted thermal cameras (534, 535, 536, 537).
• the head-mounted sensors are visible-light cameras that take images of the ROIs.
• Head-mounted sensor 530 captures an ROI above the frame
• head-mounted sensors 531 and 532 capture ROIs below the frame.
• Hot spot 538 which is measured by the thermal camera 534, was caused by touching the forehead and is unrelated to the physiological response being detected.
• the computer may use the associated thermal measurements differently than it would use had the touching not been detected.
• a user interface may provide an indication that touching the ROI hinders the detection of the physiological response.
• the computer may identify disruptive activities, such as talking, eating, and drinking, and utilize the identified disruptive activities in order to more accurately detect the physiological response.
• the computer utilizes a machine learning-based approach to handle the disruptive activities.
• This approach may include (i) identifying, based on IMROO, occurrences of one or more of the disruptive activities, (ii) generating feature values based on the identified disruptive activities, and (iii) utilizing a machine learning-based model to detect the physiological response based on the feature values and feature values generated from THROII .
• the computer may utilize IMROU to generate an avatar of the user (e.g., in order to represent the user in a virtual environment).
• the avatar may express emotional responses of the user, which are detected based on IMROI2-
• the computer may modify the avatar to show synthesized facial expressions that are not manifested in the user's actual facial expressions, but the synthesized facial expressions correspond to emotional responses detected based on THR O II-
• Thermal measurements of the face can be utilized for various applications, which include detection of various physiological responses and/or medical conditions.
• thermal measurements can be influenced by various intrinsic and extrinsic factors.
• Some examples of such factors include infrared radiation directed at the face (e.g., sunlight), wind, physical contact with the face (e.g., touching the face with the hand), and consumption of substances such as various medications, alcohol, caffeine, and nicotine.
• these factors may be unrelated to a physiological response and/or medical condition being detected, they may nonetheless affect the temperatures at various regions of interest on the face. Therefore, these factors may be considered confounding factors that hinder the detection of the physiological response and/or medical condition.
• thermal measurements collected in real-life scenarios to account for various intrinsic and/or extrinsic factors that may alter the thermal measurements, in order to avoid the detrimental effect that such factors may have on the utility of the thermal measurements for various applications.
• CAM inward-facing head-mounted thermal camera
• CAM 0Ut outward-facing head-mounted thermal camera
• CAM out measures the environment and generates data indicative of confounding factors, such as direct sunlight or air conditioning. Accounting for confounding factors enables the system to more accurately detect the physiological response compared to a system that does not account for these confounding factors.
• CAMm and/or CAM ou t are physically coupled to a frame worn on a user's head, such as a frame of a pair of eyeglasses or an augmented reality device.
• each of CAMikos and CAM 0U t weighs below 5 g and is located less than 15 cm from the user's face.
• CAMin takes thermal measurements of an ROI (THROI) on the user's face.
• ROI ROI
• CAMi does not occlude the ROI.
• the ROI includes a region on the forehead and the physiological response involves stress, a headache, and/or a stroke.
• the ROI includes a region on the nose and the physiological response is an allergic reaction.
• CAMout takes thermal measurements of the environment (THENV).
• CAM 0 ut does not occlude the ROI.
• the angle between the optical axes of CAM m and CAMout is at least 45°, 90°, 130°, 170°, or 180°.
• the field of view (FOV) of CAM m is larger than the FOV of CAMout and/or the noise equivalent differential temperature (NEDT) of CAMm is lower than NEDT of CAMout.
• FOV field of view
• NEDT noise equivalent differential temperature
• CAMi Vietnamese has a FOV smaller than 80° and CAM out has a FOV larger than 80° .
• CAM m has more sensing elements than CAM 0Ut (e.g., CAM m has at least double the number of pixels as CAM 0U t).
• CAMi n and CAM are based on sensors of the same type with similar operating parameters.
• CAMi n and CAM 0Ut are located less than 5 cm or 1 cm apart. Having sensors of the same type, which are located near each other, may have an advantage of having both CAMi n and CAM rat be subject to similar inaccuracies resulting from heat conductance and package temperature.
• CAM m and CAM 0Ut may be based on sensors of different types, with different operating parameters. For example, CAM m may be based on a microbolometer FPA while CAM may be based on a thermopile (that may be significantly less expensive than the microbolometer).
• FIG. 27a illustrates one embodiment of the system that includes inward-facing and outward- facing head-mounted thermal cameras on both sides of the frame.
• CAM m is the inward-facing thermal camera 12, which takes thermal measurements of ROI 13
• CAM 0Ut is the outward-facing thermal camera 62.
• Arc 64 illustrates the larger FOV of CAM out 62, compared to the FOV of CAMi n that covers ROI 13.
• the illustrated embodiment includes a second head-mounted thermal camera 10 (CAM; complicat2) on the right side of the frame, which takes thermal measurements of ROI 11, and a second outward-facing head-mounted thermal camera 63 (CAM ou t2)-
• FIG. 27b illustrates receiving an indication on a GUI (on the illustrated laptop) that the user is not monitored in direct sunlight.
• Cameras 520 and 521 are the outward-facing head-mounted thermal cameras.
• the computer detects a physiological response based on THROI and THENV-
• THENV are utilized to account for at least some of the effect of heat transferred from the environment to the ROI (and not due to the user's physiological response).
• detections of the physiological response based on TH ROI and TH ENV may be more accurate than detections of the physiological response based on THROI without THENV.
• the computer may utilize THENV to increase the accuracy of detecting the physiological response.
• the computer generates feature values based on a set of THROI and THENV, and utilizes a machine learning-based model to detect, based on the feature values, the physiological response.
• the computer may make different detections of the physiological response based on similar TH ROI that are taken in dissimilar environments. For example, responsive to receiving a first set of measurements in which TH ROI reaches a first threshold while TH ENV does not reach a second threshold, the computer detects the physiological response.
• the computer does not detect the physiological response.
• THENV reaching the second threshold indicates that the user was exposed to high infrared radiation that is expected to interfere with the detection.
• the computer may utilize THENV for the selection of values that are appropriate for the detection of the physiological response.
• the computer may select different thresholds (to which THROI are compared) for detecting the physiological response.
• different THENV may cause the computer to use different thresholds.
• the computer may utilize THE NV to select an appropriate reference time series (to which THR O I may be compared) for detecting the physiological response.
• the computer may utilize THE NV to select an appropriate model to utilize to detect the physiological response based on the feature values generated based on THROI-
• the computer may normalize THROI based on THENV-
• the normalization may involve subtracting a value proportional to THE NV from THR O I, such that the value of the temperature at the ROI is adjusted based on the temperature of the environment at that time and/or in temporal proximity to that time (e.g., using an average of the environment temperature during the preceding minute).
• the computer may adjust weights associated with at least some THR O I based on THE NV , such that the weight of measurements from among THR O I that were taken during times the measurements of the environment indicated extreme environmental temperatures is reduced.
• the computer may refrain from performing detection of the physiological response. This way, the computer can avoid making a prediction that is at high risk of being wrong due to the influence of the extreme environmental temperatures.
• the computer may determine that the difference between THROI and THENV are not in an acceptable range (e.g., there is a difference of more than 15 0 C between the two), and refrain from making a detection of the physiological response in that event.
• the system may include a second outward-facing head-mounted thermal camera (CAM 0U t2), which takes thermal measurements of the environment (THENV2)-
• CAM 0U t2 second outward-facing head-mounted thermal camera
• THENV2 thermal measurements of the environment
• Utilizing two or more outward-facing head-mounted thermal cameras such as CAM 0U t and CAM 0U t2 can help identify cases in which there is a directional environmental interference (e.g., sunlight coming from a certain direction). In some cases, such a directional interference can lead to refraining from making a detection of the physiological response.
• a directional environmental interference e.g., sunlight coming from a certain direction
• the computer responsive to receiving a first set of measurements in which THROI reach a first threshold while the difference between THENV and THENV2 does not reach a second threshold, the computer detects the physiological response. However, responsive to receiving a second set of measurements in which THROI reach the first threshold while the difference between THENV and THENV2 reaches the second threshold, the computer does not detect the physiological response.
• the computer detects the physiological response based on a difference between THROI, THENV, and THENV2, while taking into account the angle between the optical axes of CAM 0U t and CAM 0U t2 and a graph of responsivity as function of the angle from the optical axes of each of CAM 0U t and CAM 0U t2-
• CAMin and CAM 0Ut are located to the right of the vertical symmetry axis that divides the user's face, and the ROI is on the right side of the face.
• the system includes a second inward-facing head-mounted thermal camera (CAMM) and a second outward- facing head-mounted thermal camera (CAM 0Ut2 ) located to the left of the vertical symmetry axis.
• CAMin2 takes thermal measurements of a second ROI (THROK) on the left side of the face, and does not occlude the second ROI (ROI2).
• CAM 0U t2 takes thermal measurements of the environment (THENV2) that is more to the left relative to THENV-
• the computer detects the physiological response also based on THROO and THENV2-
• the optical axes of CAMi n and CAM out are above the Frankfort horizontal plane
• the system further includes a second inward-facing head-mounted thermal camera (CAMM) and a second outward-facing head-mounted thermal camera (CAM 0Ut2 ), located such that their optical axes are below the Frankfort horizontal plane, which take thermal measurements THROE and T3 ⁇ 4NV2, respectively.
• the computer detects the physiological response also based on TH OO and THENV2-
• the computer detects the physiological response by performing at least one of the following calculations: (i) when the difference between THENV and THENV2 reaches a threshold, the computer normalizes THR O I and THR OO differently against thermal interference from the environment, (ii) when THE NV does not reach a predetermined threshold for thermal environmental interference, while THENV2 reaches the predetermined threshold, the computer assigns THROI a higher weight than THROO for detecting the physiological response, and (iii) the computer generates feature values based on THROI, THENV, THENV2 and optionally THROE? and utilizes a model to detect, based on the feature values, the physiological response.
• the model was trained based on a first set of THROI, THROO, THENV and THENV2 of one or more users taken while the one or more users had the physiological response, and a second set of THROI, THROE, THENV and THENV2 of the one or more users taken while the one or more users did not have the physiological response.
• some embodiments may include a sensor that may be used to address various other confounding factors, such as user movements and wind, which are discussed below.
• the sensor is coupled to a frame worn on the user's head.
• An example of such a sensor is sensor 68 in FIG. 27a.
• the senor takes measurements (denoted m CO nf) that are indicative of an extent of the user's activity, an orientation of the user's head, and/or a change in a position of the user's body.
• the sensor may be (i) a movement sensor that is physically coupled to a frame worn on the user's head, or coupled to a wearable device worn by the user, (ii) a visible-light camera that takes images of the user, and/or (iii) an active 3D tracking device that emits electromagnetic waves and generates 3D images based on received reflections of the emitted electromagnetic waves.
• the computer detects the physiological response also based on iriconf-
• the computer may refrain from detecting the physiological response if m con f reaches a threshold (which may indicate the user was very active which causes an increase in body temperature).
• the computer generates feature values based on TH ROI , TH ENV , and mconf and utilizes a model to detect the physiological response based on the feature values.
• the model was trained based on previous THROI, THENV, and mconf taken while the user had different activity levels.
• the model may be trained based on: a first set of previous TH ROI , TH ENV , and rriconf taken while the user was walking or running, and a second set of previous THROI, THENV, and mconf taken while the user was sitting or standing.
• TH ROI thermal measurements of an ROI
• the affect to TH ROI can be attributed to various physiological and/or metabolic processes that may ensue following the consumption of the confounding substance, which can result (amongst possibly other effects) in a raising or decreasing of the temperature at the ROI in a manner that is unrelated to the physiological response being detected.
• embodiments of this system utilize indications indicative of consumption of a confounding substance (such as medication, an alcoholic beverage, a caffeinated beverage, and/or a cigarette) to improve the system's detection accuracy.
• a confounding substance such as medication, an alcoholic beverage, a caffeinated beverage, and/or a cigarette
• the system includes a CAM and a computer.
• CAM is worn on the user's head and takes thermal measurements of an ROI (TH ROI ) on the user's face.
• the system includes a frame to which CAM and the device are physically coupled.
• CAM is located less than 15 cm from the face and/or weighs below 10 g.
• the ROI may cover different regions on the face and CAM may be located at different locations on a frame worn on the user's head and/or at different distances from the user's face.
• the ROI is on the forehead, and CAM is physically coupled to an eyeglasses frame, located below the ROI, and does not occlude the ROI.
• the physiological response detected in this embodiment is stress, a headache, and/or a stroke.
• the ROI is on the periorbital area, and CAM is located less than 10 cm from the ROI.
• the physiological response detected in this embodiment is stress.
• the ROI is on the nose, and CAM is physically coupled to an eyeglasses frame and is located less than 10 cm from the face.
• the physiological response detected in this embodiment is an allergic reaction.
• the ROI is below the nostrils, and CAM: is physically coupled to an eyeglasses frame, located above the ROI, and does not occlude the ROI.
• the ROI covers one or more areas on the upper lip, the mouth, and/or air volume(s) through which the exhale streams from the nose and/or mouth flow, and the physiological response detected in this embodiment is a respiratory parameter such as the user's breathing rate.
• the computer may receive, from a device, an indication indicative of consuming a confounding substance that is expected to affects TH O I, such as an alcoholic beverage, a medication, caffeine, and/or a cigarette.
• a confounding substance such as an alcoholic beverage, a medication, caffeine, and/or a cigarette.
• Various types of devices may be utilized in different embodiments in order to identify consumption of various confounding substances.
• the device includes a visible-light camera that takes images of the user and/or the user's environment.
• the visible-light camera is a head-mounted visible-light camera having in its field of view a volume that protrudes out of the user's mouth.
• the computer identifies a consumption of the confounding substance based on analyzing the images.
• the visible-light camera may belong to a camera-based system such as OrCam (http://www.orcam.com/), which is utilized to identify various objects, products, faces, and/or recognize text.
• images captured by the visible-light camera may be utilized to determine the nutritional composition of food a user consumes.
• the device includes a microphone that records the user, and the computer identifies a consumption of the confounding substance utilizing a sound recognition algorithm operated on a recording of the user.
• the sound recognition algorithm comprises a speech recognition algorithm configured to identify words that are indicative of consuming the confounding substance.
• the confounding substance is a medication
• the device includes a pill dispenser that provides an indication indicating that the user took a medication.
• the indication indicates the type of medication and/or its dosage.
• the device is a refrigerator, a pantry, and/or a serving robot.
• the device provides an indication indicative of the user taking an alcoholic beverage and/or a food item.
• the device has an internet -of-things (IoT) capability through which the indication is provided to the system.
• IoT internet -of-things
• the device may be part of a "smart device" with network connectivity.
• the device belongs to a user interface that receives an indication from the user or/or a third party about the consuming of the confounding substance.
• THROI confounding substance
• drugs are known to act on the hypothalamus and other brain centers involved in controlling the body's thermoregulatory system.
• stating "the confounding substance affects THROI" means that consuming the confounding substance leads to a measureable change of the temperature at the ROI, which would likely not have occurred had the confounding substance not been consumed.
• a time in which "confounding substance did not affect TH RO i" is a time that occurs after at least a certain duration has elapsed since the confounding substance was last consumed (or was not consumed at all), and the consumption of the confounding substance is no longer expected to have a noticeable effect on the ROI temperature.
• This certain duration may depend on factors such as the type of substance, the amount consumed, and previous consumption patterns. For example, the certain duration may be at least: 30 minutes, two hours, or a day.
• the duration of the effect of a confounding substance may vary between substances, and may depend on various factors such as the amount of substance, sex, weight, genetic characteristics, and the user's state. For example, consumption of alcohol on an empty stomach often has a greater effect on THR O I than consumption of alcohol with a meal. Some confounding substances may have a long- lasting effect, possibly throughout the period they are taken. For example, hormonal contraceptives can significantly alter daily body temperatures. Other confounding factors, such as caffeine and nicotine, may have shorter lasting effects, typically subsiding within less than an hour or two following their consumption.
• the computer detects the physiological response based on THR O I and the indication indicative of consuming the confounding substance. In one embodiment, the computer refrains from detecting the physiological response within a certain window during which the confounding substance affected the user (e.g., an hour, two hours, or four hours). In another embodiment, the computer utilizes a model, in addition to THROI and the indication, to detect whether the user had the physiological response during the time that a consumed confounding substance affected THROI. Optionally, the computer detects the physiological response by generating feature values based on THROI and the indication (and possibly other sources of data), and utilizing the model to calculate, based on the feature values, a value indicative of the extent of the physiological response.
• the feature values include a feature value indicative of one or more of the following: the amount of the consumed confounding substance, the dosage of the consumed confounding substance, the time that has elapsed since the confounding substance had last been consumed, and/or the duration during which the confounding factor has been consumed (e.g., how long the user has been taking a certain medication).
• the model was trained based on data collected from the user and/or other users, which includes TH OI, the indications described above, and values representing the physiological response corresponding to when THROI were taken.
• the data is used to generate samples, with each sample comprising feature values and a label.
• the feature values of each sample are generated based on THROI taken during a certain period and an indication indicating whether a confounding substance affected THROI taken during the certain period.
• the label of the sample is generated based on one or more of the values representing the physiological response, and indicates whether (and optionally to what extent) the measured user had the physiological response during the certain period.
• the data used to train the model reflects both being affected and being unaffected by the confounding substance.
• the data used to train the model may include: a first set of THR O I taken while the confounding substance affected THR O I, and a second set of THR O I taken while the confounding substance did not affect THR O I-
• each of the first and second sets comprises at least some THR O I taken while the measured user had the physiological response and at least some THR O I taken while the measured user did not have the physiological response.
• the computer's detection behavior may be as follows: the computer detects the physiological response based on first THR O I for which there is no indication indicating that the first THR O I were affected by a consumption of the confounding substance, and the first THR O I reach the threshold; the computer does not detect the physiological response based on second THR O I for which there is an indication indicating that the second THR O I were affected by a consumption of the confounding substance, and the second THR O I also reach the threshold; and the computer does not detect the physiological response based on third THR O I for which there is no indication indicating that the third THR O I were affected by a consumption the confounding substance, and the third THR O I do not reach the threshold.
• FIG. 28 illustrates that the effect of consuming alcohol on a certain THROI usually decreases after duration typical to the user (e.g., the duration is based on previous measurements of the user). Thus, when the effect remains high there may be a problem and the system may issue an alert.
• the figure illustrates an outward-facing visible-light camera 525 that generates the indications indicative of when the user consumes alcoholic beverages.
• FIG. 29 illustrates a usual increase in a certain THROI while the user smokes.
• the system identifies when the user smoked (e.g., based on images taken by the outward-facing visible-light camera 525) and doesn't alert because of an increase in THROI caused by the smoking.
• the temperature rises without the user having smoked for a certain time then it may be a sign that there is a problem, and the user might need to be alerted.
• FIG. 30 illustrates the expected decrease in a certain THROI after the user takes medication, based on previous THROI of the user.
• the system identifies when the medication is consumed, and does not generate an alert at those times. However, when THROI falls without medication having been taken, it may indicate a physiological response of which the user should be made aware.
• measuring temperatures and/or temperature changes may help determine the amount of stress a person is feeling, or extent of an allergic reaction the person has.
• measuring temperatures at regions of the face can help determine how a user feels, e.g., whether the user is nervous, calm, or happy.
• visible-light images of the face can be analyzed to determine emotional responses and various physiological signals.
• monitoring and analyzing the face can be useful for many health-related and life-logging related applications.
• collecting such data over time when people are going through their daily activities, can be very difficult.
• collection of such data involves utilizing cameras that may be bulky, unaesthetic, and/or expensive, which need to be continually pointed at a person's face.
• collecting the required measurements often involves performing various complex image analysis procedures, such as procedures involving image registration and face tracking.
• Eyeglasses typically do not include sensors that measure the wearer, such as cameras that take images of regions of the face. In order to enable collection of such images, which may be used for various applications, such as detection of physiological responses, some embodiments described herein involve a clip-on device that may be attached to the eyeglasses.
• a head-mounted camera such as CAM or VCAM
• the clip-on device includes a body that may be attached and detached, multiple times, from a pair of eyeglasses in order to secure and release the clip-on device from the eyeglasses.
• the body is a structure that has one or more components fixed to it.
• the body may have one or more inward-facing camera fixed to it.
• the body may have a wireless communication module fixed to it.
• Some additional components that may each be optionally fixed to the body include a processor, a battery, and one or more outward-facing cameras.
• eyeglasses are limited to prescription eyeglasses, prescription sunglasses, piano sunglasses, and/or augmented reality eyeglasses. This means that "eyeglasses” do not refer to helmets, hats, virtual reality devices, and goggles designed to be worn over eyeglasses. Additionally or alternatively, neither attaching the clip-on device to the eyeglasses nor detaching the clip-on device from the eyeglasses should take more than 10 seconds for an average user. This means that manipulating the clip-on device is not a complicated task.
• the body is configured to be detached from the eyeglasses by the user who wears the eyeglasses, who is not a technician, and without using a tool such as a screwdriver or a knife.
• the clip-on device may be attached and detached as needed, e.g., enabling the user to attach the clip-on when there is a need to take measurements, and otherwise have it detached.
• the clip-on device is a lightweight device, weighing less than 40 g (i.e., the total weight of the body and the components fixed to it is less than 40 g).
• the clip-on device weighs below 20 g and/or below 10 g.
• the body is a structure to which components (e.g., an inward-facing camera) may be fixed such that the various components do not fall off while the clip-on device is attached to the eyeglasses.
• components e.g., an inward-facing camera
• at least some of the various components that are fixed to the body remain in the same location and/or orientation when the body is attached to the eyeglasses.
• stating that a component is "fixed" to the body is intended to indicate that, during normal use (e.g., involving securing/releasing the clip-on device), the components are typically not detached from the body. This is opposed to the body itself, which in normal use is separated from the eyeglasses frame, and as such, is not considered “fixed” to the eyeglasses frame.
• the body is a rigid structure made of a material such as plastic, metal, and/or an alloy (e.g., carbon alloy).
• the rigid structure is shaped such that it fits the contours of at least a portion of the frame of the eyeglasses in order to enable a secure and stable attachment to the eyeglasses.
• the body may be made of a flexible material, such as rubber.
• the flexible body is shaped such that it fits the contours of at least a portion of the frame of the eyeglasses in order to enable a secure and stable attachment to the eyeglasses. Additionally or alternatively, the flexible body may assume the shape of a portion of the frame when it is attached to the eyeglasses.
• the body may utilize various mechanisms in order to stay attached to the eyeglasses.
• the body may include a clip member configured to being clipped on the eyeglasses.
• the body may include a magnet configured to attach to a magnet connected to the eyeglasses and/or to a metallic portion of the eyeglasses.
• the body may include a resting tab configured to secure the clip-on to the eyeglasses.
• the body may include a retention member (e.g., a clasp, buckle, clamp, fastener, hook, or latch) configured to impermanently couple the clip-on to the eyeglasses.
• clasp 147 is utilized to secure the clip-on device illustrated in FIG.
• the body may include a spring configured to apply force that presses the body towards the eyeglasses.
• a spring configured to apply force that presses the body towards the eyeglasses.
• FIG. 17a An example of this type of mechanism is illustrated in FIG. 17a where spring 175 is used to apply force that pushes body 170 and secures it in place to frame 176.
• impermanently couple something means to attach in a way that is easily detached without excessive effort.
• coupling something by clipping it on or closing a latch is considered impermanently coupling it.
• Coupling by screwing a screw with a screwdriver, gluing, or welding is not considered impermanently coupling. The latter would be examples of what may be considered to "fix” a component to the body.
• the inward-facing camera is fixed to the body. It takes images of a region of interest on the face of a user who wears the eyeglasses. Optionally, the inward-facing camera remains pointed at the region of interest even when the user's head makes lateral and/or angular movements.
• the inward- facing camera may be any of the CAMs and/or VCAMs described in this disclosure.
• the inward-facing camera weighs less than 10 g, 5 g or 1 g.
• the inward-facing camera is a thermal camera based on a thermopile sensor, a pyroelectric sensor, or a microbolometer sensor, which may be a FPA sensor.
• the inward-facing camera includes a multi-pixel sensor and a lens, and the sensor plane is tilted by more than 2° relative to the lens plane according to the Scheimpflug principle in order to capture sharper images when the body is attached to the eyeglasses that are worn by a user.
• the clip-one device may include additional components that are fixed to it.
• the clip-on device include a wireless communication module fixed to the body which transmits measurements (e.g., images and/or thermal measurements) taken by one or more of the cameras that are fixed to the body.
• the clip-on device may include a battery fixed to the body, which provides power to one or more components fixed to the body.
• the clip-on device may include a processor that controls the operation of one or more of the components fixed to the body and/or processes measurements taken by the camera fixed to the body.
• a computer receives measurements taken by the inward-facing camera (and possibly other cameras fixed to the body), and utilizes the measurements to detect a physiological response.
• the computer is not fixed to the body.
• the computer may belong to a device of the user (e.g., a smartphone or a smartwatch), or the computer may be a cloud-based server.
• the computer receives, over a wireless channel, the measurements, which are sent by the wireless communication module.
• inward- and outward-facing cameras that are fixed to the body, which may be used to take images of various regions of interest on the face of the user who wears the eyeglasses. It is to be noted that while the discussion below generally refers to a single "inward-facing camera” and/or a single “outward-facing camera", embodiments of the clip-on device may include multiple inward- and/or outward-facing cameras.
• the inward-facing camera is a thermal camera.
• the thermal camera when the body is attached to the eyeglasses, the thermal camera is located less than 5 cm from the user's face.
• measurements taken by the thermal camera are transmitted by the wireless communication module and are received by a computer that uses them to detect a physiological response of the user.
• the optical axis of the thermal camera when the body is attached to the eyeglasses, is above 20° from the Frankfort horizontal plane, and the thermal camera takes thermal measurements of a region on the user's forehead.
• the thermal camera takes thermal measurements of a region on the user's nose.
• the thermal camera takes thermal measurements of a region on a periorbital area of the user.
• the inward-facing camera is a thermal camera.
• the thermal camera is located below eye -level of a user who wears the eyeglasses and at least 2 cm from the vertical symmetry axis that divides the user's face (i.e., the axis the goes down the center of the user's forehead and nose).
• the inward-facing thermal camera takes thermal measurements of a region on at least one of the following parts of the user's face: upper lip, lips, and a cheek.
• measurements taken by the thermal camera are transmitted by the wireless communication module and are received by a computer that uses them to detect a physiological response of the user.
• the inward-facing camera is a visible -light camera.
• the visible -light camera when the body is attached to the eyeglasses, the visible -light camera is located less than 10 cm from the user's face.
• images taken by the visible -light camera are transmitted by the wireless communication module and are received by a computer that uses them to detect a physiological response of the user.
• the computer detects the physiological response based on facial skin color changes (FSCC) that are recognizable in the images.
• FSCC facial skin color changes
• the optical axis of the visible -light camera is above 20° from the Frankfort horizontal plane, and the visible-light camera takes images of a region located above the user's eyes.
• the visible -light camera takes images of a region on the nose of a user who wears the eyeglasses.
• the computer detects the physiological response based on facial expressions, and when the body is attached to the eyeglasses, the visible-light camera takes images of a region above or below the user's eyes.
• the inward-facing camera is a visible -light camera
• the visible-light camera takes images of a region on an eye (IME) of a user who wears the eyeglasses, and is located less than 10 cm from the user's face.
• the images are transmitted by the wireless communication module and are received by a computer that detects a physiological response based in IME.
• the computer detects the physiological response based on color changes to certain parts of the eye, such as the sclera and or the iris. Due to the many blood vessels that are close to the surface of the eye, physiological responses that are manifested through changes to the blood flow (e.g., a cardiac pulse and certain emotional responses), may cause recognizable changes to the color of the certain parts of the eye.
• certain parts of the eye such as the sclera and or the iris. Due to the many blood vessels that are close to the surface of the eye, physiological responses that are manifested through changes to the blood flow (e.g., a cardiac pulse and certain emotional responses), may cause recognizable changes to the color of the certain parts of the eye.
• the various techniques described in this disclosure for detecting a physiological response based on FSCC that is recognizable in images can be applied by one skilled in the art to detect a physiological response based on color changes to the sclera and/or iris; while the sclera and iris are not the same color as a person's skin, they too exhibit blood flow -related color changes that are qualitatively similar to FSCC, and thus may be analyzed using similar techniques to the techniques used to analyze FSCC involving the forehead, nose, and/or cheeks.
• IME may be utilized to determine the size of the pupil, which may be utilized by the computer to detect certain emotional responses (such as based on the assumption that the pupil's response reflects emotional arousal associated with increased sympathetic activity).
• identifying which portions of IME correspond to certain parts of the eye can be done utilizing various image processing techniques known in the art. For example, identifying the iris and pupil size may be done using the techniques described in US patent application US20060147094, or in Hayes, Taylor R., and Alexander A. Petrov. "Mapping and correcting the influence of gaze position on pupil size measurements.” Behavior Research Methods 48.2 (2016): 510-527. Additionally, due to the distinct color differences between the skin, the iris, and the sclera, identification of the iris and/or the white sclera can be easily done by image processing methods known in the art.
• the inward-facing camera is a visible-light camera; when the body is attached to the eyeglasses, the visible-light camera is located below eye-level of a user who wears the eyeglasses, and at least 2 cm from the vertical symmetry axis that divides the user's face.
• the visible - light camera takes images (IM O I) of a region on the upper lip, lips, and/or a cheek.
• IMR O I are transmitted by the wireless communication module and are received by a computer that uses them to detect a physiological response.
• the physiological response is an emotional response, which is detected based on extracting facial expressions from IMROI.
• the physiological response is an emotional response, which is detected based on FSCC recognizable in IMROI.
• the physiological response, which is detected based FSCC recognizable in IMROI is heart rate and/or breathing rate.
• the body may include an outward-facing camera that may be utilized to provide measurements that may be used to account for various environmental interferences that can decrease detections of the physiological response of a user who wears the eyeglasses.
• the outward-facing camera is a head-mounted camera.
• the outward-facing camera is fixed to the body.
• the inward-facing camera is a thermal camera, and when the body is attached to the eyeglasses, the thermal camera is located less than 10 cm from the face of the user who wears the eyeglasses, and takes thermal measurements of a region of interest (THROI) on the face of the user.
• THROI region of interest
• an outward-facing head-mounted thermal camera takes thermal measurements of the environment (THENV).
• the wireless communication module transmits THROI and THENV to a computer that detects an emotional response of the user based on THROI and THENV-
• the computer utilizes THENV to account for thermal interferences from the environment, as discussed elsewhere herein.
• the inward-facing camera is a visible -light camera, and when the body is attached to the eyeglasses, the visible-light camera is located less than 10 cm from the face of the user who wears the eyeglasses and takes images of a region of interest (IMR O I) on the face of the user.
• IMR O I region of interest
• an outward-facing head-mounted visible -light camera takes images of the environment (IMENV).
• the wireless communication module transmits IMROI and IMENV to a computer that detects an emotional response of the user based on IMR O I and IME NV -
• the computer detects the physiological response based on FSCC recognizable in IMROI, and utilizes IMENV to account for variations in ambient light, as discussed elsewhere herein.
• Inward-facing cameras attached to the body may be utilized for additional purposes, beyond detection of physiological responses.
• the inward-facing camera is a visible -light camera
• the clip-on device includes a second visible-light camera that is also fixed to the body.
• the visible -light camera and/or the second visible-light camera are light field cameras.
• the first and second visible -light cameras are located less than 10 cm from the user's face, and take images of a first region above eye -level and a second region on the upper lip (IMROI and IMROE, respectively).
• the wireless communication module transmits IMR O I and IMR O E to a computer that generates an avatar of the user based on IMROI and IMROE-
• FIG. 14a to FIG. 18 illustrate some examples of clip- on devices.
• most of the clip-on device may be located in front of the frame of the eyeglasses, as illustrated in FIG. 14b, FIG. 15b, and FIG. 18, or alternatively, most of the clip-on device may be located behind the frame, as illustrated in FIG. 16b and FIG. 17b.
• Some clip-on devices may include a single unit, such as illustrated in FIG. 15a and FIG. 17a. While other clip-on devices may include multiple units (which each may optionally be considered a separate clip-on device).
• FIG. 14a, FIG. 14b, and FIG. 14c illustrate two right and left clip-on devices comprising bodies 141 and 142, respectively, which are configured to attached/detached from an eyeglasses frame 140.
• the body 142 has multiple inward-facing cameras fixed to it, such as camera 143 that points at a region on the lower part of the face (such as the upper lip, mouth, nose, and/or cheek), and camera 144 that points at the forehead.
• the body 142 may include other electronics 145, such as a processor, a battery, and/or a wireless communication module.
• the bodies 141 and 142 of the left and right clip- on devices may include additional cameras illustrated in the drawings as black circles.
• the eyeglasses include left and right lenses, and when the body is attached to the eyeglasses, most of the volume of the clip -on device is located to the left of the left lens or to the right of the right lens.
• the inward-facing camera takes images of at least one of: a region on the nose of a user wearing the eyeglasses, and a region on the mouth of the user.
• a portion of the clip-on device that is located to the left of the left lens or to the right of the right lens does not obstruct the sight of the user when looking forward.
• FIG. 15a and FIG. 15b illustrate a clip-on device that includes a body 150, to which two head- mounted cameras are fixed: a head-mounted camera 148 that points at a region on the lower part of the face (such as the nose), and a head-mounted camera 149 that points at the forehead.
• the other electronics such as a processor, a battery, and/or a wireless communication module
• the clip-on device is attached and detached from the frame of the eyeglasses with the clasp 147.
• the inward-facing camera takes images of a region on the forehead of a user who wears the eyeglasses.
• a portion of the clip-on device that is located above the lenses of the eyeglasses does not obstruct the sight of the user when looking forward.
• the clip-on device may include various protruding arms.
• these arms may be utilized in order to position one or more cameras in a position suitable for taking images of certain regions of the face.
• FIG. 18 illustrates right and left clip-on devices that include bodies 153 and 154, respectively, which are configured to attached/detached from an eyeglasses frame. These bodies have protruding arms that hold the head-mounted cameras.
• Head-mounted camera 155 measures a region on the lower part of the face, head-mounted camera 156 measures regions on the forehead.
• the left clip-on device also includes other electronics 157 (such as a processor, a battery, and/or a wireless communication module).
• the clip-on devices illustrated in this figure may include additional cameras illustrated in the drawings as black circles.
• FIG. 16b and FIG. 17b illustrate two examples of clip-on devices that are mostly attached behind the frame. The following are some additional examples of embodiments in which a portion of the clip-on device may be located behind the frame.
• FIG. 16a and FIG. 16b illustrate two, right and left, clip-on devices with bodies 160 and 161, respectively, configured to be attached behind an eyeglasses frame 165.
• the body 160 has various components fixed to it which include: an inward-facing head-mounted camera 162 pointed at a region below eye-level (such as the upper lip, mouth, nose, and/or cheek), an inward-facing head-mounted camera 163 pointed at a region above eye -level (such as the forehead), and other electronics 164 (such as a processor, a battery, and/or a wireless communication module).
• the right and left clip-on devices may include additional cameras illustrated in the drawings as black circles.
• FIG. 17a and FIG. 17b illustrate a single-unit clip-on device that includes the body 170, which is configured to be attached behind the eyeglasses frame 176.
• the body 170 has various cameras fixed to it, such as head-mounted cameras 171 and 172 that are pointed at regions on the lower part of the face (such as the upper lip, mouth, nose, and/or cheek), and head-mounted cameras 173 and 174 that are pointed at the forehead.
• the spring 175 is configured to apply force that holds the body 170 to the frame 176.
• Other electronics 177 such as a processor, a battery, and/or a wireless communication module, may also be fixed to the body 170.
• the clip-on device may include additional cameras illustrated in the drawings as black circles.
• the body when the body is attached to the eyeglasses, more than 50% of the outfacing surface of the clip-on device is located behind the eyeglasses frame.
• a portion of the clip-on device that is located behind the eyeglasses frame is occluded from a viewer positioned directly opposite to the eyeglasses, at the same height as the eyeglasses.
• a portion of the clip-on device that is behind the frame might not be visible to other people from many angles, which can make the clip-on device less conspicuous and/or more aesthetically pleasing.
• a larger portion of the clip-on device is behind the frame when the body is attached to the eyeglasses, such as more than 75% or 90% of the out-facing surface.
• FSCC facial skin color changes
• FSCC are typically a result of changes in the concentration levels of hemoglobin and blood oxygenation under a user's facial skin due to a physiological response that involves changes in the user's emotional state and/or changes in the user's physical state, and/or due to normal biological processes. These changes in the concentration levels of hemoglobin and blood oxygenation can cause subtle changes in the hue and saturation components of the user's facial skin color.
• US patent application 20160098592 describes extracting emotions based on hemoglobin concentration changes (HCC) from red, green and blue (RGB) video.
• HCC hemoglobin concentration changes
• US patents number 8768438, 8977347, 8855384, 9020185, 8617081 and US patent application number 20130215244 describe extracting heart rate and related parameters from RGB video, near-IR video, and multi-spectral video streams.
• Some aspects of this disclosure involve detection of a physiological response based on facial skin color changes (FSCC) recognizable in images taken with an inward-facing head-mounted visible -light camera (VCAM m ).
• FSCC facial skin color changes
• VCAM m inward-facing head-mounted visible -light camera
• the transformation model (to relate one image to another) may be restricted to the maximum possible relative movement between VCAM m and the ROI, which is confined as a result of coupling the camera to the frame. This restricted transformation model is much less computational intensive than a full transformation model as in the prior art configurations where the camera is not head-mounted.
• a system configured to detect a physiological response based on FSCC includes at least an inward-facing head-mounted visible-light camera (VCAM m ) and a computer.
• VCAM m inward-facing head-mounted visible-light camera
• the system may optionally include additional elements such as a frame and additional inward-facing camera(s) and/or outward-facing camera(s).
• FIG. 23 illustrates one embodiment of the system configured to detect a physiological response based on FSCC.
• the system includes a frame 735 (e.g., an eyeglasses frame) to which various cameras are physically coupled.
• These cameras include visible -light cameras 740, 741, 742, and 743, which may each take images of regions on the user's cheeks and/or nose. Each of these cameras may possibly be VCAMm, which is discussed in more detail below. Another possibility for VCAM; n is camera 745 that takes images of a region on the user's forehead and is coupled to the upper portion of the frame. Visible-light camera 737, which takes images of the environment (IM ENV ), is an example of VCAM o u t discussed below, which may optionally be included in some embodiments. Additional cameras that may optionally be included in some embodiments are outward-facing thermal camera 738 (which may be used to take TH ENV mentioned below) and inward-facing thermal camera 739 (which may be used to take TH ROO mentioned below).
• VCAM m is worn on the user's head and takes images of a region of interest (IM ROI ) on the user's face.
• the OI may cover various regions on the user's face.
• the ROI is on a cheek of the user, a region on the user's nose, and/or a region on the user's forehead.
• VCAMin does not occlude the ROI, is located less than 10 cm from the user's face, and weighs below 10 g.
• the ROI is illuminated by ambient light.
• the system does not occlude the ROI, and the ROI is not illuminated by a head-mounted light source.
• the ROI may be illuminated by a head-mounted light source that is weaker than the ambient light.
• the computer detects the physiological response based on IM ROI by relying on effects of FSCC that are recognizable in IMROI-
• sentences of the form "FSCC recognizable in IMROI” refer to effects of FSCC that may be identified and/or utilized by the computer, which are usually not recognized by the naked eye.
• the FSCC phenomenon may be utilized to detect various types of physiological responses.
• the physiological response that is detected may involve an expression of emotional response of the user.
• the computer may detect whether the user's emotional response is neutral, positive, or negative.
• the computer may detect an emotional response that falls into a more specific category such as distress, happiness, anxiousness, sadness, frustration, intrigue, joy, disgust, anger, etc.
• the expression of the emotional response may involve the user making a facial expression and/or a microexpression (whose occurrence may optionally be detected based on IMROI).
• detecting the physiological response involves determining one or more physiological signals of the user, such as a heart rate (which may also be referred to as "cardiac pulse”), heart rate variability, and/or a breathing rate.
• IM ROI are images generated based on ambient light illumination that is reflected from the user's face. Variations in the reflected ambient light may cause FSCC that are unrelated to the physiological response being detected, and thus possibly lead to errors in the detection of the physiological response.
• the system includes an outward-facing head-mounted visible -light camera (VCAM 0U t), which is worn on the user's head, and takes images of the environment (IMENV).
• VCAM 0U t is located less than 10 cm from the user's face and weighs below 10 g.
• VCAM 0U t may include optics that provide it with a wide field of view.
• the computer detects the physiological response based on both IMROI and IMENV.
• IMENV is indicative of illumination towards the face
• IMROI is indicative of reflections from the face
• utilizing IMENV in the detection of the physiological response can account, at least in part, for variations in ambient light that, when left unaccounted, may possibly lead to errors in detection of the physiological response.
• the system may include multiple VCAM m configured to take images of various ROIs on the face, IMROI may include images taken from the multiple VCAMm, and multiple VCAM 0U t located at different locations and/or orientation relative to the face may be used to take images of the environment.
• VCAMi Vietnamese and/or VCAM 0U t are physically coupled to a frame, such as an eyeglasses frame or an augmented realty device frame.
• a frame such as an eyeglasses frame or an augmented realty device frame.
• the angle between the optical axes of VCAM m and VCAM 0Ut is known to the computer, and may be utilized in the detection of the physiological response.
• the angle between the optical axes of VCAMiliens and VCAM 0Ut is fixed.
• VCAMi n Due to the proximity of VCAMi n to the face, in some embodiments, there may be an acute angle between the optical axis of VCAM m and the ROI (e.g., when the ROI includes a region on the forehead). In order to improve the sharpness of IMR O I, VCAMi n may be configured to operate in a way that takes advantage of the Scheimpflug principle.
• VCAM m includes a sensor and a lens; the sensor plane is tilted by a fixed angle greater than 2° relative to the lens plane according to the Scheimpflug principle in order to capture a sharper image when VCAM; n is worn by the user (where the lens plane refers to a plane that is perpendicular to the optical axis of the lens, which may include one or more lenses).
• VCAMizie does not occlude the ROI.
• VCAMi lending includes a sensor, a lens, and a motor; the motor tilts the lens relative to the sensor according to the Scheimpflug principle. The tilt improves the sharpness of IMR O I when VCAM m is worn by the user.
• VCAMin and/or VCAM 0U t may include optics and sensors that capture light rays in at least one of the following NIR spectrum intervals: 700-800 nm, 700-900 nm, 700-1 ,000 nm.
• the computer may utilize data obtained in a NIR spectrum interval to detect the physiological response (in addition to or instead of data obtained from the visible spectrum).
• the sensors may be CCD sensors designed to be sensitive in the NIR spectrum and/or CMOS sensors designed to be sensitive in the NIR spectrum.
• the computer may utilize various approaches in order to detect the physiological response based on IMROI- Some examples of how such a detection may be implemented are provided in the prior art references mentioned above, which rely on FSCC to detect the physiological response. It is to be noted that while the prior art approaches involve analysis of video obtained from cameras that are not head-mounted, are typically more distant from the ROI than VCAMin, and are possibly at different orientations relative to the ROI, the computational approaches described in the prior art used to detect physiological responses can be readily adapted by one skilled in the art to handle IMROI. In some cases, embodiments described herein may provide video in which a desired signal is more easily detectable compared to some of the prior art approaches.
• the ROI is expected to cover a larger portion of the images in IMROI compared to images obtained by video cameras in some of the prior art references.
• additional illumination that is required in some prior art approaches, such as illuminating the skin for a pulse oximeter to obtain a photoplethysmographic (PPG) signal, may not be needed.
• PPG photoplethysmographic
• VCAMi n ' s fixed location and orientation relative to the ROI (even when the user makes lateral and/or angular movements)
• many preprocessing steps that need to be implemented by the prior art approaches, such as image registration and/or face tracking, are extremely simplified in embodiments described herein, or may be foregone altogether.
• IMR O I may undergo various preprocessing steps prior to being used by the computer to detect the physiological response and/or as part of the process of the detection of the physiological response.
• the preprocessing include: normalization of pixel intensities (e.g., to obtain a zero-mean unit variance time series signal), and conditioning a time series signal by constructing a square wave, a sine wave, or a user defined shape, such as that obtained from an ECG signal or a PPG signal as described in US patent number 8617081.
• some embodiments may involve generating feature values based on a single image or a sequence of images. In some examples, generation of feature values from one or more images may involve utilization of some of the various approaches described in this disclosure for generation of high-level and/or low- level image-based features.
• IME NV may also be utilized by the computer to detect the physiological response (in addition to IMROI), as explained in more detail below.
• the physiological response may be detected using signal processing and/or analytical approaches.
• these approaches may be used for detecting repetitive physiological signals (e.g., a heart rate, heart rate variability, or a breathing rate) in IMROI taken during a certain period.
• the detected physiological response represents the value of the physiological signal of the user during the certain period.
• US patent number 8768438 titled “Determining cardiac arrhythmia from a video of a subject being monitored for cardiac function" describes how a heart rate may be determined based on FSCC, which are represented in a PPG signal obtained from video of the user.
• a time series signal is generated from video images of a subject's exposed skin, and a reference signal is used to perform a constrained source separation (which is a variant of ICA) on the time series signals to obtain the PPG signal.
• Peak-to-peak pulse points are detected in the PPG signal, which may be analyzed to determine parameters such as heart rate, heart rate variability, and/or to obtain peak-to-peak pulse dynamics that can be indicative of conditions such as cardiac arrhythmia.
• US patent number 8977347 titled “Video-based estimation of heart rate variability” describes how a times-series signal similar to the one described above may be subjected to a different type of analysis to detect the heart rate variability.
• the time series data are de-trended to remove slow non-stationary trends from the signal and filtered (e.g., using bandpass filtering).
• bandpass filtering e.g., using bandpass filtering
• low frequency and high frequency components of the integrated power spectrum within the time series signal are extracted using Fast Fourier Transform (FFT). A ratio of the low and high frequency of the integrated power spectrum within these components is computed. And analysis of the dynamics of this ratio over time is used to estimate heart rate variability.
• FFT Fast Fourier Transform
• US patent number 9020185 titled “Systems and methods for non- contact heart rate sensing” describes how a times-series signals obtained from video of a user can be filtered and processed to separate an underlying pulsing signal by, for example, using an ICA algorithm.
• the separated pulsing signal from the algorithm can be transformed into frequency spacing data using FFT, in which the heart rate can be extracted or estimated.
• the physiological response may be detected using machine learning- based methods.
• these approaches may be used for detecting expressions of emotions and/or values of physiological signals.
• machine learning-based approaches involve training a model on samples, with each sample including: feature values generated based on IMR O I taken during a certain period, and a label indicative of the physiological response during the certain period.
• the model may be personalized for a user by training the model on samples including: feature values generated based on IMROI of the user, and corresponding labels indicative of the user's respective physiological responses.
• Some of the feature values in a sample may be generated based on other sources of data (besides IMROI), such as measurements of the user generated using thermal cameras, movement sensors, and/or other physiological sensors, and/or measurements of the environment.
• IMROI of the user taken during an earlier period may serve as a baseline to which to compare.
• some of the feature values may include indications of confounding factors, which may affect FSCC, but are unrelated to the physiological response being detected.
• confounding factors include touching the face, thermal radiation directed at the face, and consuming certain substances such as a medication, alcohol, caffeine, or nicotine.
• Training the model may involve utilization of various training algorithms known in the art (e.g., algorithms for training neural networks and/or other approaches described herein). After the model is trained, feature values may be generated for IMROI for which the label (physiological response) is unknown, and the computer can utilize the model to detect the physiological response based on these feature values.
• the model is trained based on data that includes measurements of the user, in which case it may be considered a personalized model of the user. In other embodiments, the model is trained based on data that includes measurements of one or more other users, in which case it may be considered a general model.
• the samples used in the training may include samples based on IMR O I taken in different conditions and include samples with various labels (e.g., expressing or not expressing certain emotions, or different values of physiological signals).
• the samples are generated based on IMR O I taken on different days.
• the "measured user” in the four examples below may be "the user” who is mentioned above (e.g., when the model is a personalized model that was trained on data that includes measurements of the user), or a user from among one or more other users (e.g., when the model is a general model that was trained on data that includes measurements of the other users).
• the system does not occlude the ROI, and the model is trained on samples generated from a first set of IMR O I taken while the measured user was indoors and not in direct sunlight, and is also trained on other samples generated from a second set of IMR O I taken while the measured user was outdoors, in direct sunlight.
• the model is trained on samples generated from a first set of IMR O I taken during daytime, and is also trained on other samples generated from a second set of IMR O I taken during nighttime.
• the model is trained on samples generated from a first set of IMR O I taken while the measured user was exercising and moving, and is also trained on other samples generated from a second set of IMR O I taken while the measured user was sitting and not exercising.
• the model is trained on samples generated from a first set of IMR O I taken less than 30 minutes after the measured user had an alcoholic beverage, and is also trained on other samples generated from a second set of IMR O I taken on a day in which the measured user did not have an alcoholic beverage.
• Labels for the samples may be obtained from various sources.
• the labels may be obtained utilizing one or more sensors that are not VCAM; n .
• a heart rate and/or heart rate variability may be measured using an ECG sensor.
• the breathing rate may be determined using a smart shirt with sensors attached to the chest (e.g., a smart shirt by Hexoskin®).
• a type emotional response of the user may be determined based on analysis of a facial expression made by the user, analysis of the user's voice, analysis of thermal measurements of regions of the face of the user, and/or analysis of one or more of the following sensor-measured physiological signals of the user: a heart rate, heart rate variability, breathing rate, and galvanic skin response.
• a label describing an emotional response of the user may be inferred.
• the label may be based on semantic analysis of a communication of the user, which is indicative of the user's emotional state at the time IMROI were taken.
• the label may be generated in a process in which the user is exposed to certain content, and a label is determined based on an expected emotional response corresponding to the certain content (e.g., happiness is an expected response to a nice image while distress is an expected response to a disturbing image).
• the model may include parameters describing multiple hidden layers of a neural network.
• the model may include a convolution neural network (CNN).
• CNN convolution neural network
• the CNN may be utilized to identify certain patterns in the video images, such as the patterns of the reflected FSCC due to the physiological response.
• detecting the physiological response may be done based on multiple, possibly successive, images that display a certain pattern of change over time (i.e., across multiple frames), which characterizes the physiological response being detected.
• detecting the physiological response may involve retaining state information that is based on previous images.
• the model may include parameters that describe an architecture that supports such a capability.
• the model may include parameters of a recurrent neural network (RNN), which is a connectionist model that captures the dynamics of sequences of samples via cycles in the network's nodes. This enables RNNs to retain a state that can represent information from an arbitrarily long context window.
• RNN recurrent neural network
• the RNN may be implemented using a long short-term memory (LSTM) architecture.
• LSTM long short-term memory
• the RNN may be implemented using a bidirectional recurrent neural network architecture (BRNN).
• IMENV may be utilized in the detection of the physiological response to account, at least in part, for illumination interferences that may lead to errors in the detection of the physiological response.
• illumination interferences may lead to errors in the detection of the physiological response.
• IMENV may be utilized for this purpose.
• the computer may refrain from detecting the physiological response.
• a certain threshold e.g., which may correspond to ambient light variations above a certain extent
• IMENV may be utilized to normalize IMROI with respect to the ambient light.
• the intensity of pixels in IMROI may be adjusted based on the intensity of pixels in IM ENV when IM ROI were taken.
• US patent application number 20130215244 describes a method of normalization in which values of pixels from a region that does not contain a signal (e.g., background regions that include a different body part of the user or an object behind the user) are subtracted from regions of the image that contain the signal of the physiological response.
• training data that includes a ground-truth signal may be utilized to optimize the normalization procedure used to correct IM ROI with respect to the ambient light measured in IM ENV - For example, such optimization may be used to determine parameter values of a function that performs the subtraction above, which lead to the most accurate detections of the physiological response.
• IM ENV may be utilized to generate feature values in addition to IM ROI .
• at least some of the same types of feature values generated based on IM ROI may also be generated based on IM ENV -
• at least some of the feature values generated based on IMENV may relate to portions of images, such as average intensity of patches of pixels in IMENV-
• a machine learning-based model may be trained to be robust, and less susceptible, to environmental interferences such as ambient light variations. For example, if the training data used to train the model includes samples in which no physiological response was present (e.g., no measured emotional response or microexpression was made), but some ambient light variations might have introduced some FSCC -related signal, the model will be trained such that feature values based on IMENV are used to account for such cases. This can enable the computer to negate, at least in part, the effects of such environmental interferences, and possibly make more accurate detections of the physiological response.
• the computer receives an indication indicative of the user consuming a confounding substance that is expected to affect FSCC (e.g., alcohol, drugs, certain medications, and/or cigarettes).
• the computer detects the physiological response, while the consumed confounding substance affects FSCC, based on: IMROI, the indication, and a model that was trained on: a first set of IMROI taken while the confounding substance affected FSCC, and a second set of IMROI taken while the confounding substance did not affect FSCC.
• Prior art FSCC systems are sensitive to user movements and do not operate well while the user is running. This is because state-of-the-art FSCC systems use hardware and automatic image trackers that are not accurate enough to crop correctly the ROI from the entire image while running, and the large errors in cropping the ROI are detrimental to the performances of the FSCC algorithms. Contrary to the prior art FSCC systems, the disclosed VCAMi n remains pointed at its ROI also when the user's head makes angular and lateral movements, and thus the complicated challenges related to image registration and ROI tracking are much simplified or even eliminated. Therefore, systems based on VCAMi n (such as the one illustrated in FIG. 23) may detect the physiological response (based on FSCC) also while the user is running.
• VCAMi n may be pointed at different regions on the face.
• the ROI is on the forehead, VCAMi beau is located less than 10 cm from the user's face, and optionally the optical axis of VCAMi n is above 20° from the Frankfort horizontal plane.
• the ROI is on the nose, and VCAMin is located less than 10 cm from the user's face. Because VCAMin is located close to the face, it is possible to calculate the FSCC based on a small ROI, which is irrelevant to the non-head-mounted prior arts that are limited by the accuracy of their automatic image tracker.
• VCAM; n is pointed at an eye of the user.
• the computer selects the sclera as the ROI and detects the physiological response based on color changes recognizable in IM ROI of the sclera.
• VCAMi n is pointed at an eye of the user.
• the computer selects the iris as the ROI and detects the physiological response based on color changes recognizable in IM ROI of the iris.
• the computer further calculates changes to the pupil diameter based on the IM ROI of the iris, and detects an emotional response of the user based on the changes to the pupil diameter.
• the computer may utilize measurements of one or more head-mounted thermal cameras in the detection of the physiological response.
• the system may include an inward- facing head-mounted thermal camera that takes thermal measurements of a second ROI (TH ROO ) on the user's face.
• ROI and ROI2 overlap, and the computer utilizes THROU to detect the physiological response.
• the computer utilizes THROO to account, at least in part, for temperature changes, which may occur due to physical activity and/or consumption of certain medications that affect the blood flow.
• the computer utilizes TH ROE by generating feature values based on THROO, and utilizing a model that was trained on data comprising THROO in order to detect the physiological response.
• the system may include an outward-facing head-mounted thermal camera that takes thermal measurements of the environment (THENV).
• the computer may utilize THENV to detect the physiological response (e.g., by generating feature values based on THENV and utilizing a model trained on data comprising THENV).
• detecting the physiological response based on both FSCC recognizable in IMROI and THENV is more accurate than detecting the physiological response based on the FSCC without THENV.
• the computer utilizes THENV to account, at least in part, for thermal interferences from the environment, such as direct sunlight and/or a nearby heater.
• the computer may utilize IMROI to generate an avatar of the user (e.g., in order to represent the user in a virtual environment).
• the avatar may express emotional responses of the user, which are detected based on IMR O I-
• the computer may modify the avatar of the user to show synthesized facial expressions that are not manifested in the user's actual facial expressions.
• the synthesized facial expressions correspond to emotional responses detected based on FSCC that are recognizable in IMR O I-
• the synthesized facial expressions correspond to emotional responses detected based on thermal measurements taken by CAM.
• HMS may be connected, using wires and/or wirelessly, with a device carried by the user and/or a non-wearable device.
• the HMS may include a battery, a computer, sensors, and a transceiver.
• FIG. 31a and FIG. 31b are schematic illustrations of possible embodiments for computers (400, 410) that are able to realize one or more of the embodiments discussed herein that include a "computer".
• the computer (400, 410) may be implemented in various ways, such as, but not limited to, a server, a client, a personal computer, a network device, a handheld device (e.g., a smartphone), an HMS (such as smart glasses, an augmented reality system, and/or a virtual reality system), a computing device embedded in a wearable device (e.g., a smartwatch or a computer embedded in clothing), a computing device implanted in the human body, and/or any other computer form capable of executing a set of computer instructions.
• a server a client
• a personal computer e.g., a personal computer
• a network device e.g., a smartphone
• HMS such as smart glasses, an augmented reality system, and/or a virtual reality system
• an augmented reality system refers also to a mixed reality system.
• references to a computer or processor include any collection of one or more computers and/or processors (which may be at different locations) that individually or jointly execute one or more sets of computer instructions.
• a first computer may be embedded in the HMS that communicates with a second computer embedded in the user's smartphone that communicates over the Internet with a cloud computer.
• the computer 400 includes one or more of the following components: processor 401 , memory 402, computer readable medium 403, user interface 404, communication interface 405, and bus 406.
• the computer 410 includes one or more of the following components: processor 411 , memory 412, and communication interface 413.
• Thermal measurements that are forwarded to a processor/computer may include "raw" values that are essentially the same as the values measured by thermal cameras, and/or processed values that are the result of applying some form of preprocessing and/or analysis to the raw values. Examples of methods that may be used to process the raw values include analog signal processing, digital signal processing, and various forms of normalization, noise cancellation, and/or feature extraction.
• At least some of the methods described herein are "computer-implemented methods" that are implemented on a computer, such as the computer (400, 410), by executing instructions on the processor (401 , 411).
• the instructions may be stored on a computer -readable medium, which may optionally be a non-transitory computer-readable medium.
• the instructions In response to execution by a system including a processor and memory, the instructions cause the system to perform the method steps.
• a direction of the optical axis of a VCAM or a CAM that has focusing optics is determined by the focusing optics, while the direction of the optical axis of a CAM without focusing optics (such as a single pixel thermopile) is determined by the angle of maximum responsivity of its sensor.
• the term CAM includes the optics (e.g., one or more lenses).
• the optics of a CAM may include one or more lenses made of a material suitable for the required wavelength, such as one or more of the following materials: Calcium Fluoride, Gallium Arsenide, Germanium, Potassium Bromide, Sapphire, Silicon, Sodium Chloride, and Zinc Sulfide.
• the CAM optics may include one or more diffractive optical elements, and/or or a combination of one or more diffractive optical elements and one or more refractive optical elements.
• CAM when CAM includes an optical limiter/ field limiter/ FOV limiter (such as a thermopile sensor inside a standard TO-39 package with a window, or a thermopile sensor with a polished metal field limiter), then the term CAM may also refer to the optical limiter. Depending on the context, the term CAM may also refer to a readout circuit adjacent to CAM, and/or to the housing that holds CAM.
• an optical limiter/ field limiter/ FOV limiter such as a thermopile sensor inside a standard TO-39 package with a window, or a thermopile sensor with a polished metal field limiter
• references to thermal measurements in the context of calculating values based on thermal measurements, generating feature values based on thermal measurements, or comparison of thermal measurements relate to the values of the thermal measurements (which are values of temperature or values of temperature changes).
• a sentence in the form of "calculating based on THROI” may be interpreted as “calculating based on the values of THROI”
• a sentence in the form of "comparing THROII and THROU” may be interpreted as "comparing values of THROII and values of THROU”.
• thermal measurements of an ROI may have various forms, such as time series, measurements taken according to a varying sampling frequency, and/or measurements taken at irregular intervals.
• thermal measurements may include various statistics of the temperature measurements ( ⁇ ) and/or the changes to temperature measurements ( ⁇ ), such as minimum, maximum, and/or average values.
• Thermal measurements may be raw and/or processed values.
• the thermal measurements may include values corresponding to each of the pixels, and/or include values representing processing of the values of the pixels.
• the thermal measurements may be normalized, such as normalized with respect to a baseline (which is based on earlier thermal measurements), time of day, day in the month, type of activity being conducted by the user, and/or various environmental parameters (e.g., the environment's temperature, humidity, radiation level, etc.).
• references to “one embodiment” mean that the feature being referred to may be included in at least one embodiment of the invention. Moreover, separate references to “one embodiment”, “some embodiments”, “another embodiment”, “still another embodiment”, etc., may refer to the same embodiment, may illustrate different aspects of an embodiment, and/or may refer to different embodiments.
• sentences in the form of "X is indicative of Y” mean that X includes information correlated with Y, up to the case where X equals Y.
• sentences in the form of "thermal measurements indicative of a physiological response” mean that the thermal measurements include information from which it is possible to infer the physiological response. Stating that "X indicates Y” or "X indicating Y” may be interpreted as "X being indicative of Y”.
• sentences in the form of "provide/receive an indication indicating whether X happened” may refer herein to any indication method, including but not limited to: sending/receiving a signal when X happened and not sending/receiving a signal when X did not happen, not sending/receiving a signal when X happened and sending/receiving a signal when X did not happen, and/or sending/receiving a first signal when X happened and sending/receiving a second signal X did not happen.
• a "portion” of something and a “region” of something refer herein to a value between a fraction of the something and 100% of the something.
• sentences in the form of a "portion of an area” may cover between 0.1% and 100% of the area.
• sentences in the form of a "region on the user's forehead” may cover between the smallest area captured by a single pixel (such as 0.1% or 5% of the forehead) and 100% of the forehead.
• the word “region” refers to an open-ended claim language, and a camera said to capture a specific region on the face may capture just a small part of the specific region, the entire specific region, and/or a portion of the specific region together with additional region(s).
• Sentences in the form of "angle greater than 20°” refer to absolute values (which may be +20° or -20° in this example), unless specifically indicated, such as in a phrase having the form of "the optical axis of CAM is 20° above/below the Frankfort horizontal plane” where it is clearly indicated that the CAM is pointed upwards/downwards.
• the Frankfort horizontal plane is created by two lines from the superior aspects of the right/left external auditory canal to the most inferior point of the right/left orbital rims.
• a predetermined value is a fixed value and/or a value determined any time before performing a calculation that compares a certain value with the predetermined value.
• a value is also considered to be a predetermined value when the logic, used to determine whether a threshold that utilizes the value is reached, is known before start performing computations to determine whether the threshold is reached.
• the embodiments of the invention may include any variety of combinations and/or integrations of the features of the embodiments described herein. Although some embodiments may depict serial operations, the embodiments may perform certain operations in parallel and/or in different orders from those depicted. Moreover, the use of repeated reference numerals and/or letters in the text and/or drawings is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. The embodiments are not limited in their applications to the order of steps of the methods, or to details of implementation of the devices, set in the description, drawings, or examples. Moreover, individual blocks illustrated in the figures may be functional in nature and therefore may not necessarily correspond to discrete hardware elements.
• a system configured to detect a physiological response, comprising:
• an inward-facing head-mounted thermal camera configured to take thermal measurements of a first region of interest (TH OII) on a user's face;
• VCAM head-mounted visible -light camera
• IMROE second region of interest
• a computer configured to detect the physiological response based on THROII , IMROO, and a model.
• the computer is further configured to generate feature values based on THR O II and IMR O I 2 , and to utilize the model to detect the physiological response based on the feature values; wherein, on average, detections of the physiological response based on THROII and IMROO are more accurate than detections of the physiological response based on THROII without IMROE-
• the model was trained based on previous THROII and IMROE of the user taken during different days, and the physiological response is indicative of an occurrence of at least one of the following emotional states of the user: joy, fear, sadness, and anger.
• the model was trained based on previous THROII and IM OO of the user taken during different days, and the physiological response is indicative of an occurrence of one or more of the following: stress, mental workload, an allergic reaction, a headache, dehydration, intoxication, and a stroke.
• the model was trained based on previous THROII and IMROO taken during different days, and the physiological response is a physiological signal selected from among: a heart rate, a breathing rate, and an extent of frontal lobe brain activity.
• ROI2 covers at least half of ROIi, each of CAM and VCAM weighs below 10 g, is physically coupled to a frame configured to be worn on the user's head, and is located less than 15 cm from the user's face; and further comprising a second thermal camera that is located less than 15 cm from the user's face, is physically coupled to the frame, and is configured to take thermal measurements of a third region of interest (THROD) on the face; the center of ROIi is to the right of the center of the third region of interest (ROI 3 ), and the symmetric overlapping between ROIi and ROI3 is above 50%; whereby to detect the physiological response, the computer is configured to account for facial thermal asymmetry, based on a difference between THROII and THROD
• the account for thermal asymmetry comprises utilizing at least one of the following calculations: (i) utilizing different thresholds to which THROII and THROD are compared; (ii) utilizing different reference time series to which THROU and THROD are compared; (iii) utilizing a machine learning-based model that provides different results for first and second events that involve the same average change in THROII and THROD with different extents of asymmetry in THROII and THROD; and (iv) utilizing the asymmetry for differentiating between (a) temperature changes in THROII and THROD that are related to the physiological response and (b) temperature changes in THROII and THROD that are unrelated to the physiological response.
• ROI 2 covers at least half of ROIi, each of CAM and VCAM weighs below 10 g, is physically coupled to a frame configured to be worn on the user's head, and is located less than 15 cm from the user's face; and further comprising a second visible -light camera (VCAM2) that is physically coupled to the frame and is configured to take images of a third ROI (IMR O D) on the face; wherein VCAM and VCAM2 are located at least 0.5 cm to the right and to the left of the vertical symmetry axis that divides the face, respectively, and the symmetric overlapping between ROI 2 and ROI 3 is above 50% ; and wherein the computer is configured to detect the physiological response also based on IMROD-
• the physiological response is emotional response
• the computer is further configured to calculate, based on IMR OO , facial skin color changes (FSCC), and to detect the emotional response of the user based on THR O II and FSCC; wherein, on average, detections of the emotional responses based on both THR O II and FSCC are more accurate than detections of the emotional responses based on either THROII or FSCC.
• IMR OO facial skin color changes
• FSCC facial skin color changes
• the physiological response is emotional response
• the computer is further configured to identify facial expressions from IMR OO , and to detect the emotional response of the user based on THR O II and the identified facial expressions; wherein, on average, detections of the emotional response based on both THR O II and the identified facial expressions are more accurate than detections of the emotional response based on either THR O II or the identified facial expressions.
• the computer is further configured to generate an avatar of the user based on IMR O D, and to modify the avatar to shows a synthesized facial expression that corresponds to an emotional response detected based on THROII, which is not manifested in the user's facial expression.
• the model was trained on: samples generated based on a first set of THROII and IMROI 2 taken after cosmetics were applied to a portion of the overlapping region between ROIi and ROI2, and other samples generated based on a second set of THROII and IMROD taken while the overlapping region was bare of cosmetics; whereby utilizing the model enables the computer to account for presence of cosmetics on the overlapping region.
• the model was trained on: samples generated from a first set of THROII and IMROD taken while sweat was detectable on a portion of the overlapping region between ROIi and ROI2, and additional samples generated from a second set of TH OII and IMROO taken while sweat was not detectable on the overlapping region; whereby utilizing the model enables the computer to account for sweat on the overlapping region.
• the model was trained on: samples generated from a first set of THROII and EViRon taken while hair density on a portion of the overlapping region between ROIi and ROI2 was at a first level, and additional samples generated from a second set of THROII and IMROO taken while hair density on the portion of the overlapping region between was at a second level higher than the first level; whereby utilizing the model enables the computer to account for hair on the portion of the overlapping region.
• the model was trained on: samples generated from a first set of THROII and IMROO taken while skin inflammation was detectable on a portion of the overlapping region between ROIi and ROI2, and additional samples generated from a second set of THROII and IMROO taken while skin inflammation was not detectable on the overlapping region; whereby utilizing the model enables the computer to account for skin inflammation on the overlapping region.
• the model was trained on: samples generated from a first set of THROII and IMROO taken while detecting that the user touches a portion of the overlapping region between ROIi and ROI2, and additional samples generated from a second set of THROII and IMROO taken while detecting that the user does not touch the overlapping region; whereby utilizing the model enables the computer to account for touching the overlapping region.
• the computer is further configured to: (i) identify, based on IMR OO , occurrences of one or more of the following disruptive activities that alter THR O II : talking, eating, and drinking, (ii) generate feature values based on the identified disruptive activities, and (iii) utilize the model to detect the physiological response based on the feature values and THR O II-
• ROIi and ROI2 are on the mouth, and IMR OO are indicative of a change in a facial expression during a certain period that involves a transition from a facial expression in which the lips are in contact to a facial expression with an open mouth; whereby a detection of the physiological response based on THR O II taken during the certain period involves attributing a change in THR O II to opening the mouth rather than a change in the temperature of the lips.
• ROIi and ROI2 are on the nose and upper lip
• IMROO are indicative of a change in a facial expression during a certain period that involves a transition from a neutral facial expression to a facial expression of disgust; whereby in a detection of the physiological response based on THROII taken during the certain period involves attributing a change in THROU to a raised upper lip and wrinkled nose instead of a change in the temperature of the nose and upper lip.
• ROIi and ROI2 are on the user's forehead located about 1 cm above at least one of the user's eyebrows, and IMROE are indicative of a change in a facial expression during a certain period that involves a transition from a neutral expression to a facial expression involving raised eyebrows; whereby in a detection of the physiological response, based on THROII taken during the certain period, involves attributing a change in THROII to raising the eyebrows instead of a change in the temperature of forehead.
• VCAM comprises a multi-pixel sensor and a lens
• the sensor plane is tilted by more than 2° relative to the lens plane according to the Scheimpflug principle in order to improve the sharpness of images of ROI2
• CAM comprises a focal-plane array (FPA) sensor and an infrared lens
• FPA focal-plane array
• the computer responsive to receiving a first set of THROII and IMROE taken during a first period in which the user expressed a certain facial expression, the computer detects a first emotional response of the user, and responsive to receiving a second set of THROII and IMROE taken during a second period in which the user expressed again the certain facial expression, the computer detects a second emotional response of the user, which is not the same as the first emotional response; and wherein TH ROII of the first set are indicative of a first physiological response that reaches a threshold, while TH ROII of the second set are indicative of a second physiological response that does not reach the threshold.
• the first set comprises IM ROE indicative of a facial expression that is a smile and TH ROII indicative of stress below a certain threshold, and the first emotional response is happiness; and the second set comprises IMROE indicative of a facial expression that is a smile and THROII indicative of stress above the certain threshold, and the second emotional response is discomfort.
• the first set comprises IM ROE indicative of a facial expression that is a neutral expression and TH ROII indicative of stress below a certain threshold, and the first emotional response is comfort; and the second set comprises IM ROE indicative of a facial expression that is neutral and TH ROII indicative of stress above the certain threshold, and the second emotional response is concealment.
• the first set comprises IM ROE indicative of a facial expression that is a an expression of anger and TH ROII indicative of stress above a certain threshold, and the first emotional response is anger; and the second set comprises IM ROE indicative of a facial expression that is an expression of anger and THROII indicative of stress below the certain threshold, and the second emotional response is indicative of pretending to be angry.
• a method for detecting a physiological response comprising:
• the physiological response is emotional response, and further comprising calculating, based on IMROE, a value indicative of facial skin color changes (FSCC), and utilizing the value indicative of FSCC to generate at least one of the feature values used to detect the physiological response.
• FSCC facial skin color changes
• a system configured to detect a physiological response, comprising:
• an inward-facing head-mounted thermal camera configured to take thermal measurements of a region of interest (THR O I) on a user's face; wherein CAMi n does not occlude the region of interest (ROI) and is located less than 15 cm from the user's face;
• an outward-facing head-mounted thermal camera configured to take thermal measurements of the environment (THE NV ), and is located less than 15 cm from the face;
• a computer configured to detect the physiological response based on THROI and THENV-
• the computer is further configured to generate feature values based on sets of THR O I and THE NV , and to utilize a machine learning-based model to detect, based on the feature values, the physiological response; wherein responsive to receiving a first set of measurements in which THR O I reaches a first threshold while THE NV does not reach a second threshold, the computer detects the physiological response; and wherein responsive to receiving a second set of measurements in which THR O I reaches the first threshold while THE NV reaches the second threshold, the computer does not detect the physiological response.
• the field of view (FOV) of CAM in is larger than the FOV of CAM 0Ut , and the noise equivalent differential temperature (NEDT) of CAM; n is lower than NEDT of CAM 0U t-
• CAM in has a field of view (FOV) smaller than 80°
• CAM out has a FOV larger than 80°
• CAMin has more sensing elements than CAM 0U t-
• the angle between the optical axes of CAMin and CAMout is at least one of the following angles: 45°, 90°, 130°, 170°, and 180°.
• CAM; n and CAM 0Ut are based on sensors of the same type and are located less than 5 cm apart.
• the region of interest includes a region on the forehead of the user and the physiological response comprises at least one of the following: stress, a headache, and a stroke.
• the region of interest includes a region on the nose of the user and the physiological response is an allergic reaction.
• CAM 0 ut2 a second outward-facing head-mounted thermal camera
• THENV2 thermal measurements of the environment
• the computer responsive to receiving a first set of measurements in which THROI reach a first threshold while the difference between THENV and THENV2 does not reach a second threshold, the computer detects the physiological response; and responsive to receiving a second set of measurements in which THR O I reach the first threshold while the difference between THENV and THENV2 reaches the second threshold, the computer does not detect the physiological response.
• CAM 0 ut2 a second outward-facing head-mounted thermal camera
• THENV2 thermal measurements of the environment
• the computer is configured to detect the physiological response based on a difference between THROI, THENV, and THENV2, while taking into account the angle between the optical axes of CAM 0Ut and CAM 0U t2 and a graph of responsivity as function of the angle from the optical axes of each of CAM oot and CAM 0U t2-
• CAM; n and CAM 0Ut are located to the right of the vertical symmetry axis that divides the user's face, and the ROI is on the right side of the face; and further comprising a second inward- facing head-mounted thermal camera (CAM ⁇ ) and a second outward-facing head-mounted thermal camera (CAM ou t2) located to the left of the vertical symmetry axis;
• CAMM is configured to take thermal measurements of a second ROI (THR O E) on the left side of the face, and does not occlude ROI2;
• CAM 0 ut2 is configured to take thermal measurements of an environment (THENV2) that is more to the left relative to the environment measured by CAM OLLT ; wherein the computer is configured to detect the physiological response also based on THROE and THENV2-
• the optical axes of CAMikos and CAM out are above the Frankfort horizontal plane; and further comprising a second inward-facing head-mounted thermal camera (CAM ⁇ ) and a second outward-facing head- mounted thermal camera (CAM 0Ut 2), located such that their optical axes are below the Frankfort horizontal plane, and configured to take thermal measurements THROO and THENV2, respectively; wherein the computer is configured to detect the physiological response also based on THROE and THENV2.
• CAM ⁇ inward-facing head-mounted thermal camera
• CAM 0Ut 2 second outward-facing head- mounted thermal camera
• a sensor configured to take measurements (m CO nf) that are indicative of at least one of the following: an extent of the user's activity, an orientation of the user's head, and a change in a position of the user's body; wherein the computer is further configured to detect the physiological response also based on rriconf-
• the computer is further configured to generate feature values based on THROI, THENV, and iriconf, and to utilize a model to detect the physiological response based on the feature values; wherein the model was trained based on: a first set of previous THROI, THENV, and iriconf taken while the user was walking or running, and a second set of previous THROI, THENV, and iriconf taken while the user was sitting or standing.
• the senor comprises one or more of the following sensors: (i) a movement sensor that is physically coupled to a frame worn on the user's head or to a wearable device worn by the user, (ii) a visible-light camera configured to take images of the user, and (iii) an active 3D tracking device configured to emit electromagnetic waves and generate 3D images based on received reflections of the emitted electromagnetic waves.
• the computer is configured to detect the physiological response based on a difference between TH ROI and TH ENV ; whereby detecting the physiological response based on the difference enables the system to operate well in an uncontrolled environment that does not maintain environmental temperature in a range below +3 0 C and does not maintain humidity in a range below ⁇ 10%.
• a method for detecting a physiological response comprising:
• TH ROI region of interest
• CAM m inward- facing head-mounted thermal camera
• a system configured to detect a physiological response while taking into account consumption of a confounding substance, comprising:
• an inward-facing head-mounted thermal camera configured to take thermal measurements of a region of interest (THROI) on a user's face;
• a computer configured to:
• the model was trained on: a first set of THROI taken while the confounding substance affected THROI, and a second set of THROI taken while the confounding substance did not affect THROI.
• CAM is located less than 10 cm from the user's face, and the confounding substance comprises at least one of an alcoholic beverage, a medication, and a cigarette; and wherein the first and second sets are previous THROI of the user.
• the computer detects the physiological response based on first THROI, for which there is no indication indicating that the first THROI were affected by a consumption of the confounding substance, and the first THR O I reach a threshold; wherein the computer does not detect the physiological response based on second THR O I, for which there is an indication indicating that the second THR O I were affected by a consumption the confounding substance, and the second THR O I also reach the threshold.
• the model is a machine learning-based model
• the computer is further configured to generate feature values based on THR O I and the indication, and to utilize the machine learning-based model to detect the physiological response based on the feature values.
• the confounding substance comprises a medication
• the indication is received from a pill dispenser
• the indication indicates that the user took a medication.
• the confounding substance comprises an alcoholic beverage
• the indication is received from at least one of a refrigerator, a pantry, and a serving robot; and wherein the indication indicates that the user took an alcoholic beverage.
• the indication is received from a head-mounted visible-light camera having in its field of view a volume that protrudes out of the user's mouth, and the computer is further configured to identify the consuming of the confounding substance based on analysis of images taken by the head-mounted visible -light camera.
• the indication is received from a device having an internet-of -things (IoT) capability through which the indication is provided to the computer.
• IoT internet-of -things
• the indication is received from a microphone configured to record the user; and the computer is further configured to identify the consuming of the confounding substance utilizing a sound recognition algorithm operated on a recording of the user.
• the sound recognition algorithm comprises a speech recognition algorithm configured to identify words that are indicative of consuming the confounding substance.
• the indication is received from a user interface configured to receive an input from the user or a third party about the consuming of the confounding substance.
• the region of interest is on the forehead
• CAM is physically coupled to an eyeglasses frame, located below the ROI, and does not occlude the ROI.
• the region of interest is on the periorbital area, and CAM is located less than 10 cm from the face.
• the region of interest is on the nose, and CAM: is physically coupled to an eyeglasses frame, and located less than 10 cm from the face.
• the region of interest is below the nostrils
• CAM is physically coupled to an eyeglasses frame, located above the ROI, and does not occlude the ROI.
• a method for detecting a physiological response while taking into account consumption of a confounding substance comprising:
• the model is a machine learning-based model, and further comprising generating feature values based on THR O I and the indication, and utilizing the machine learning-based model to detect the physiological response based on the feature values.
• the confounding substance comprises a medication
• the indication is received from a pill dispenser.
• the confounding substance comprises an alcoholic beverage
• the indication is received from at least one of a refrigerator, a pantry, and a serving robot.
• a clip-on device comprising:
• a body configured to be attached and detached, multiple times, from a pair of eyeglasses in order to secure and release the clip-on device from the eyeglasses;
• the clip-on device weighs less than 40 g.
• the wireless communication module is configured to transmit measurements taken by the inward-facing camera to a computer that is not fixed to the body and is configured to detect a physiological response based on measurements.
• the inward-facing camera is a thermal camera; wherein when the body is attached to the eyeglasses, the thermal camera is: configured to take thermal measurements of a region on the forehead (THF) of a user who wears the eyeglasses, located less than 5 cm from the user's face, and its optical axis is above 20° from the Frankfort horizontal plane; and wherein the wireless communication module is configured to transmit THF to a computer configured to detect a physiological response based on THF.
• THF forehead
• the inward-facing camera is a thermal camera; wherein when the body is attached to the eyeglasses, the thermal camera is: configured to take thermal measurements of a region on the nose (THN) of a user who wears the eyeglasses, and located less than 5 cm from the user's face; and wherein the wireless communication module is configured to transmit THN to a computer configured to detect a physiological response based on THN.
• TBN region on the nose
• the inward-facing camera is a thermal camera; wherein when the body is attached to the eyeglasses, the thermal camera is: configured to take thermal measurements of a region on a periorbital area (THp) of a user who wears the eyeglasses, and located less than 5 cm from the user's face; and wherein the wireless communication module is configured to transmit THp to a computer configured to detect a physiological response based on THp.
• THp periorbital area
• the inward-facing camera is a thermal camera; wherein when the body is attached to the eyeglasses, the thermal camera is: located below eye -level of a user who wears the eyeglasses, located at least 2 cm from the vertical symmetry axis that divides the user's face, and configured to take thermal measurements (THROI) of a region on at least one of the following parts of the user's face: upper lip, lips, and a cheek; and wherein the wireless communication module is configured to transmit THR O I to a computer configured to detect a physiological response based on THR O I-
• the inward-facing camera is a visible -light camera; wherein when the body is attached to the eyeglasses, the visible-light camera is: configured to take images (IMR O I) of a region above eye-level of a user who wears the eyeglasses, located less than 10 cm from the user's face, and its optical axis is above 20° from the Frankfort horizontal plane; and wherein the wireless communication module is configured to transmit IMR O I to a computer configured to detect a physiological response based on IMROI.
• IMR O I images
• the region is on the forehead
• the computer is configured to detect the physiological response based on facial skin color changes (FSCC) recognizable in IMR O I.
• FSCC facial skin color changes
• the inward-facing camera is a visible -light camera; wherein when the body is attached to the eyeglasses, the visible-light camera is: configured to take images (IM N ) of a region on the nose of a user who wears the eyeglasses, and is located less than 10 cm from the user's face; and wherein the wireless communication module is configured to transmit IMN to a computer configured to detect a physiological response based on IMN.
• IM N images
• the computer is configured to detect the physiological response based on facial skin color changes (FSCC) recognizable in IMN-
• the inward-facing camera is a visible -light camera; wherein when the body is attached to the eyeglasses, the visible-light camera is: configured to take images (IME) of a region on an eye of a user who wears the eyeglasses, and located less than 10 cm from the user's face; and wherein the wireless communication module is configured to transmit IME to a computer configured to detect a physiological response of the user based on color changes to the sclera, which are recognizable in IME.
• IME images
• the inward-facing camera is a visible -light camera; wherein when the body is attached to the eyeglasses, the visible-light camera is: configured to take images (IME) of a region on an eye of a user who wears the eyeglasses, and located less than 10 cm from the user's face; and wherein the wireless communication module is configured to transmit IME to a computer configured to detect a physiological response of the user based on color changes to the iris, which are recognizable in IME.
• IME images
• the inward-facing camera is a visible -light camera; wherein when the body is attached to the eyeglasses, the visible-light camera is: configured to take images (IME) of a region on an eye of a user who wears the eyeglasses, and located less than 10 cm from the user's face; and wherein the wireless communication module is configured to transmit IME to a computer configured to detect an emotional response of the user based on the changes to the pupil diameter.
• IME images
• the inward-facing camera is a visible -light camera; wherein when the body is attached to the eyeglasses, the visible-light camera is: located below eye-level of a user who wears the eyeglasses, located at least 2 cm from the vertical symmetry axis that divides the user's face, and configured to take images (IMR O I) of a region on at least one of the following parts of the user's face: upper lip, lips, and a cheek; and wherein the wireless communication module is configured to transmit MR O I to a computer configured to detect an emotional response of the user based on IMR O I-
• the inward-facing camera is a thermal camera; wherein when the body is attached to the eyeglasses, the thermal camera is: configured to take thermal measurements of a region of interest (THR O I) on the face of a user who wears the eyeglasses, and is located less than 10 cm from the user's face; and further comprising an outward-facing head-mounted thermal camera configured to take thermal measurements of the environment (THE NV ); wherein the wireless communication module is configured to transmit THR O I and THE NV to a computer configured to detect an emotional response of the user based on THROI and THENV-
• THR O I region of interest
• THE NV an outward-facing head-mounted thermal camera
• the computer is configured to utilize THE NV to account for thermal interferences from the environment.
• the inward-facing camera is a visible -light camera; wherein when the body is attached to the eyeglasses, the visible -light camera is: configured to take images of a region of interest (IMROI) on the face of a user who wears the eyeglasses, and is located less than 10 cm from the user's face; and further comprising an outward-facing head-mounted visible -light camera configured to take images of the environment (IMENV); wherein the wireless communication module is configured to transmit IMROI and IMENV to a computer configured to detect an emotional response of the user based on IMROI and IMENV-
• IMROI region of interest
• IMENV outward-facing head-mounted visible -light camera
• the inward-facing camera is a visible -light camera; wherein when the body is attached to the eyeglasses, the visible -light camera is: configured to take images of a region of interest (IM OI) on the face of a user who wears the eyeglasses, and is located less than 10 cm from the user's face; and further comprising an outward-facing head-mounted visible -light camera configured to take images of the environment (IMENV); wherein the wireless communication module is configured to transmit IMROI and IMENV to a computer configured to detect a physiological response based on facial skin color changes (FSCC) recognizable in IMROI, and to utilize IMENV to account for variations in ambient light.
• IM OI region of interest
• IMENV outward-facing head-mounted visible -light camera
• the wireless communication module is configured to transmit IMROI and IMENV to a computer configured to detect a physiological response based on facial skin color changes (FSCC) recognizable in IMROI, and to utilize IMENV to account
• the inward-facing camera is a visible-light camera, and further comprising a second inward-facing visible-light camera; wherein, when the body is attached to the eyeglasses, the visible- light camera and the second visible -light camera are: configured to take images of a first region above eye-level and a second region on the upper lip (IMR O I and IMR O E, respectively), and located less than 10 cm from the user's face; wherein the wireless communication module is configured to transmit IMROI and IMROU to a computer configured to generate an avatar of the user based on IMROI and IMROIZ-
• the inward-facing camera comprises a multi-pixel sensor and a lens, and the sensor plane is tilted by more than 2° relative to the lens plane according to the Scheimpflug principle in order to capture sharper images when the body is attached to the eyeglasses which are worn by a user.
• the inward-facing camera is configured to take images of a region on the forehead of a user who wears the eyeglasses.
• the eyeglasses comprise left and right lenses, and when the body is attached to the eyeglasses, most of the volume of the clip-on device is located to the left of the left lens or to the right of the right lens; wherein the inward-facing camera is configured to take images of at least one of: a region on the nose of a user wearing the eyeglasses, and a region on the mouth of the user.
• the body utilizes at least one of the following mechanisms to stay attached to the eyeglasses: a clip member configured to being clipped on the eyeglasses, a magnet configured to attach to a magnet connected to the eyeglasses, a magnet configured to attach to a metallic portion of the eyeglasses, a resting tab configured to secure the clip-on to the eyeglasses, a retention member configured to impermanently couple the clip-on to the eyeglasses, and a spring configured to apply force that presses the body towards the eyeglasses.
• a clip member configured to being clipped on the eyeglasses
• a magnet configured to attach to a magnet connected to the eyeglasses
• a magnet configured to attach to a metallic portion of the eyeglasses
• a resting tab configured to secure the clip-on to the eyeglasses
• a retention member configured to impermanently couple the clip-on to the eyeglasses
• a spring configured to apply force that presses the body towards the eyeglasses
• the body is configured to be detached from the eyeglasses, by a user who uses the eyeglasses, without using a screwdriver or a knife.
• the eyeglasses consist at least one of: prescription eyeglasses, prescription sunglasses, piano sunglasses, and augmented reality eyeglasses; and wherein neither attaching the clip-on device to the eyeglasses nor detaching the clip-on device from the eyeglasses should take more than 10 seconds for an average user.
• the clip-on device weighs less than 20 g.
• a system configured to detect a physiological response based on facial skin color changes (FSCC), comprising:
• VCAMi n an inward-facing head-mounted visible -light camera configured to take images of a region of interest (IMROI) on a user's face; wherein the region of interest (ROI) is illuminated by ambient light; and
• a computer configured to detect the physiological response based on FSCC recognizable in IMROI.
• VCAMom head-mounted visible-light camera
• IMENV is indicative of illumination towards the face
• IMROI is indicative of reflections from the face
• utilizing IM ENV to detect the physiological response accounts, at least in part, for variations in ambient light that cause errors in detections of the physiological response.
• the physiological response comprises an expression of emotional response of the user.
• the physiological response comprises at least one of a heart rate of the user and heart rate variability of the user.
• the physiological response comprises a breathing rate of the user.
• At least one of VCAMi n and VCAM out comprise sensors configured to capture light rays also in at least one of the following near infrared (NIR) spectrum intervals: 700-800 nm, 700-900 nm, 700-1,000 nm; and the computer is further configured to detect the physiological response also based on data obtained in the NIR spectrum interval.
• NIR near infrared
• VCAMi n and VCAM 0Ut are physically coupled to the frame, are located less than 10 cm from the user's face, and each weighs below 10 g; and wherein the angle between the optical axes of VCAMi hinge and VCAMout is known to the computer.
• the model was trained on: a first set of IMROI and IMENV taken while the user was indoors and not in direct sunlight, and a second set of IMROI and IMENV taken while the user was outdoors in direct sunlight.
• the model was trained on: a first set of IM ROI and IM ENV taken during daytime, and a second set of IMROI and IMENV taken during nighttime.
• the model was trained on: a first set of IM ROI and IM ENV taken while the user was exercising and moving, and a second set of IM ROI and IM ENV taken while the user was sitting and not exercising.
• VCAMin does not occlude the ROI
• the computer is further configured to utilize a model to detect the physiological response; and wherein the model was trained on: a first set of IMROI taken while the user was indoors and not in direct sunlight, and a second set of IMROI taken while the user was outdoors in direct sunlight.
• the computer is further configured to utilize a model to detect the physiological response; and wherein the model was trained on: a first set of IM ROI taken while the user was exercising and moving, and a second set of IM ROI taken while the user was sitting and not exercising.
• the computer is further configured to utilize a model to detect the physiological response; and wherein the model was trained on: a first set of IM ROI taken less than 30 minutes after the user drank an alcoholic beverage, and a second set of IM ROI taken on a day in which the user did not drink an alcoholic beverage.
• the computer is further configured to utilize a model to detect the physiological response; wherein the model was trained on samples, each sample comprising: feature values generated based on IM ROI of the user, and a label indicative of an emotional response of the user; and wherein the label is generated based on one or more of the following: semantic analysis of a communication of the user, analysis of a facial expression made by the user, thermal measurements of regions of the face of the user, and one or more of the following physiological signals of the user: heart rate, heart rate variability, breathing rate, and galvanic skin response.
• the computer is further configured to utilize a model to detect the physiological response; wherein the model was trained on samples, each sample comprising: feature values generated based on IM ROI of the user taken while the user was exposed to certain content, and a label indicative of an expected emotional response corresponding to the certain content.
• the computer is further configured to receive an indication indicative of the user consuming a confounding substance that affects FSCC, and to detect the physiological response, while the consumed confounding substance affects FSCC, based on: IM ROI , the indication, and a model; wherein the model was trained on: a first set of IM ROI taken while the confounding substance affected FSCC, and a second set of IM ROI taken while the confounding substance did not affect FSCC.
• the physiological response is still detected by the computer based on FSCC recognizable in IMROI taken while the user is running.
• the ROI is on the forehead
• VCAMin is located less than 10 cm from the user's face
• the optical axis of VCAMin is above 20° from the Frankfort horizontal plane.
• the ROI is on the nose, and VCAM; n is located less than 10 cm from the user's face.
• VCAMi n is pointed at an eye of the user; and the computer is further configured to select the sclera as the ROI, and to detect the physiological response based on color changes recognizable in IM ROI of the sclera.
• VCAM; n is pointed at an eye of the user; and the computer is further configured to select the iris as the ROI, and to detect the physiological response based on color changes recognizable in IMROI of the iris.
• the computer is further configured to calculate changes to the pupil diameter based on the IMROI of the iris, and to detect an emotional response of the user based on the changes to the pupil diameter.
• a second ROI THROD
• the computer is further configured to utilize TH ROO to detect the physiological response; whereby, on average, detecting the physiological response based on both FSCC recognizable in IM ROI and TH ROE is more accurate than detecting the physiological response based on FSCC recognizable in IMROI without TH RO i2.
• TH ENV head-mounted thermal camera
• the computer is further configured to utilize TH ENV to detect the physiological response; whereby, on average, detecting the physiological response based on both FSCC recognizable in IMROI and THENV is more accurate than detecting the physiological response based on FSCC recognizable in IMROI without THENV-
• VCAMi n does not occlude the ROI
• VCAMi n comprises a sensor and a lens; the sensor plane is tilted by a fixed angle greater than 2° relative to the lens plane according to the Scheimpflug principle in order to capture a sharper image when VCAMi n is worn by the user.
• VCAMi n comprises a sensor, a lens, and a motor; the motor is configured to tilt the lens relative to the sensor according to the Scheimpflug principle; whereby the tilt improves the sharpness of IMROI when VCAM; n is worn by the user.
• the computer is further configured to generate an avatar of the user based on IM ROI , and to modify the avatar to show a synthesized facial expression that corresponds to an emotional response detected based on FSCC recognizable in IM ROI , which is not manifested in the user's facial expression.
• a method for detecting a physiological response based on facial skin color changes comprising:
• detecting the physiological response involves generating feature values based on IMROI and utilizing a model to calculate, based on the feature values, a value indicative of the extent of the physiological response; wherein the model was trained based on IMROI of the user taken during different days.

Landscapes

• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Physics & Mathematics (AREA)
• General Health & Medical Sciences (AREA)
• Engineering & Computer Science (AREA)
• Public Health (AREA)
• Animal Behavior & Ethology (AREA)
• Surgery (AREA)
• Molecular Biology (AREA)
• Veterinary Medicine (AREA)
• Biophysics (AREA)
• Pathology (AREA)
• Biomedical Technology (AREA)
• Heart & Thoracic Surgery (AREA)
• Medical Informatics (AREA)
• General Physics & Mathematics (AREA)
• Social Psychology (AREA)
• Otolaryngology (AREA)
• Psychiatry (AREA)
• Hospice & Palliative Care (AREA)
• Educational Technology (AREA)
• Developmental Disabilities (AREA)
• Physiology (AREA)
• Optics & Photonics (AREA)
• Acoustics & Sound (AREA)
• Psychology (AREA)
• Child & Adolescent Psychology (AREA)
• Ophthalmology & Optometry (AREA)
• Pulmonology (AREA)
• Spectroscopy & Molecular Physics (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
• Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
• Radiation Pyrometers (AREA)

Priority Applications (2)

Applications Claiming Priority (6)

Publications (1)

Family

ID=61905195

Family Applications (3)

Family Applications Before (1)

Family Applications After (1)

Country Status (3)

Cited By (2)

Families Citing this family (15)

Citations (5)