US20220133156A1 - Cardiopulmonary health monitoring using thermal camera and audio sensor - Google Patents

Cardiopulmonary health monitoring using thermal camera and audio sensor Download PDF

Info

Publication number
US20220133156A1
US20220133156A1 US17/510,357 US202117510357A US2022133156A1 US 20220133156 A1 US20220133156 A1 US 20220133156A1 US 202117510357 A US202117510357 A US 202117510357A US 2022133156 A1 US2022133156 A1 US 2022133156A1
Authority
US
United States
Prior art keywords
data
patient
thermal imaging
audio
machine learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US17/510,357
Other languages
English (en)
Inventor
Frank Thomas Hartley
Charles Henry ROSS
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ROC8SCI Co
Original Assignee
ROC8SCI Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ROC8SCI Co filed Critical ROC8SCI Co
Priority to US17/510,357 priority Critical patent/US20220133156A1/en
Assigned to Roc8Sci Co. reassignment Roc8Sci Co. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HARTLEY, FRANK THOMAS, ROSS, Charles Henry
Publication of US20220133156A1 publication Critical patent/US20220133156A1/en
Priority to US18/527,127 priority patent/US12023135B2/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/72Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235Details of waveform analysis
    • A61B5/7264Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0077Devices for viewing the surface of the body, e.g. camera, magnifying lens
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/01Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue
    • A61B5/015By temperature mapping of body part
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/02Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
    • A61B5/0205Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/103Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1102Ballistocardiography
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/103Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/113Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb occurring during breathing
    • A61B5/1135Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb occurring during breathing by monitoring thoracic expansion
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/72Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235Details of waveform analysis
    • A61B5/7264Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/74Details of notification to user or communication with user or patient ; user input means
    • A61B5/742Details of notification to user or communication with user or patient ; user input means using visual displays
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B7/00Instruments for auscultation
    • A61B7/003Detecting lung or respiration noise
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/74Details of notification to user or communication with user or patient ; user input means
    • A61B5/746Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms

Definitions

  • This application relates generally to non-contact health monitoring, and more particularly to non-contact cardiopulmonary monitoring in home and healthcare settings.
  • Apparatus for monitoring subjects during sleep may be used to diagnose sleep disorder.
  • Such apparatus are generally bulky, complex and expensive.
  • Sleep tracker devices can be more cost effective, but sleep tracker devices typically require contact with the user and are generally less precise than polysomnography tests.
  • What is needed is systems and methods for acquiring and analyzing cardiopulmonary characteristics of a subject that do not require physical contact between the subject and a monitoring device. What is needed is methods for health monitoring of subjects that do not cause discomfort and do not disturb subjects during sleep. Disclosed embodiments provide a cost-effective device for monitoring health, e.g., device suitable for use in the home. The health monitoring device disclosed herein can achieve improved precision in determining cardiopulmonary characteristics of a subject.
  • Disclosed embodiments employ a health monitoring device to monitor vital signs of a subject, such as a sleeping patient.
  • the health monitoring device may include a thermal camera such as an uncooled microbolometer array to monitor breathing, pulse, temperature, and other vital signs of the patient.
  • An audio sensor e.g., microphone, may be used for monitoring patient respiratory sounds and other patient sounds. Further information such as blood pressure and heart health can be calculated from these signals and their waveforms.
  • the health monitoring device utilizes the acquired signals and higher order data in analyzing patient conditions and behaviors. Higher order data may include visual data based upon signals output by the thermal camera and audio data based upon signals output by the audio sensor.
  • the thermal camera and signal processing of camera outputs track the pulse rate, breathing, and temperature of the subject.
  • a thermal camera may sense the sinusoidal motion of the heart rate by imaging the carotid artery in the neck, and the temple.
  • the thermal camera also may sense the sinusoidal motion of the heart rate by imaging the subject's arms and hands.
  • monitoring breathing the thermal camera may image one or more of the subject's chest, nostrils, and mouth.
  • the health monitoring device incorporates an uncooled microbolometer array in communication with a mobile computing device.
  • the health monitoring device incorporates audio data in monitoring and characterizing vital signs of a subject.
  • the audio data may include spectrograms in the audio spectrum, such as spectrograms derived from audio clips recorded by the audio sensor.
  • the audio sensor may generate audio signals via microphone, handset, or other transducer that converts sound into an analog electrical signal.
  • the microphone or an external device converts the analog signal into digital audio signals, also herein called audio data.
  • audio monitoring may be used for monitoring and characterizing breath rate and abnormal respiratory sounds, and for recognizing the subject's speech.
  • the health monitoring device may activate audio monitoring in the event video monitoring fails to detect presence of the subject at a primary location.
  • the health monitoring device includes a processor configured to output a health determination relating to the one or more health parameters of the patient by inputting one or both visual and audio data into one or more machine learning models.
  • the health determination includes a value of the one or more health parameters, a binary classification of the one or more health parameters, a multiclass classification of the one or more health parameters, an event relating to the one or more health parameters, or a health anomaly relating to the one or more health parameters.
  • the one or more machine learning models include a supervised learning model including a factorization machine.
  • the machine learning models include an unsupervised learning model trained to identify key features of interest.
  • a monitoring device comprises a set of sensors configured to receive signals pertaining to one or more health parameters of a patient through non-physical contact with the patient, wherein the set of sensors comprise a thermal camera and an audio sensor, wherein the monitoring device is configured to monitor the one or more health parameters of the patient; a signal processing unit configured to generate visual data based upon signals output by the thermal camera and to generate audio data based upon signals output by the audio sensor; and a processor configured to output a health determination relating to the one or more health parameters of the patient by inputting one or both of the visual data and the audio data into one or more machine learning models.
  • a method comprises receiving, by a set of sensors, signals pertaining to one or more health parameters of a patient through non-physical contact with the patient, wherein the sensing unit comprises a thermal camera and an audio sensor; generating, by a processor coupled to the set of sensors, visual data based upon signals output by the thermal camera and audio data based upon signals output by the audio sensor; and outputting, by the processor, a health determination relating to the one or more health parameters of the patient by inputting one or both of the visual data and the audio data into one or more machine learning models.
  • Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying figures, which are schematic and are not intended to be drawn to scale. Unless indicated as representing the background art, the figures represent aspects of the disclosure.
  • FIG. 1 illustrates an architecture of a non-contact health monitoring device, according to an embodiment.
  • FIG. 2 shows a schematic diagram of a factoring machine, according to an embodiment.
  • FIG. 3 is a flow chart diagram of recursive processes of a factorization machine, according to an embodiment.
  • FIG. 4 shows a schematic block diagram of a method for training a chest motion model, according to an embodiment.
  • FIG. 5 is a schematic diagram of an SQL architecture of features identified via recognition/identification of facial features, according to an embodiment.
  • FIG. 6 is a representative graph of a blood pressure waveform including systolic and diastolic pressures, according to an embodiment.
  • FIG. 7 shows a training process for determining blood pressure, according to an embodiment.
  • FIG. 8 illustrates field names, field data types, and PK primary key values of an example SQL architecture, according to an embodiment.
  • FIG. 9 illustrates an audio pre-processing method for identifying respiratory ailments, according to an embodiment.
  • FIG. 10 is a flow chart diagram of training methods for analysis of audio events, according to an embodiment.
  • FIG. 11 is a graph of maximum facial temperature vs. camera distance, according to an embodiment.
  • FIG. 12 is a schematic block diagram of a general health monitoring architecture, according to an embodiment.
  • Disclosed embodiments employ a health monitoring device to monitor vital signs of a subject, such as a sleeping patient.
  • Health monitoring signals may be acquired and analyzed via multiple subsystems to ensure greater reliability.
  • the health monitoring device may include a thermal camera such as an uncooled microbolometer array to monitor breathing, pulse, temperature, and other vital signs of the patient. Further information such as blood pressure and heart health can be inferred from these signals and their waveforms.
  • An audio sensor e.g., microphone, may be used for monitoring breath rate.
  • An audio recognition system can be trained to characterize abnormal respiratory sounds.
  • the health monitoring device utilizes acquired signals and higher order data as source data for analyzing patient conditions and behaviors. The higher order data may include visual data based upon signals output by the thermal camera, and audio data based upon signals output by the audio sensor.
  • the thermal camera and signal processing of camera outputs track the heart rate, breathing, and temperature of a sleeping individual.
  • a thermal camera may sense the sinusoidal motion of the heart rate by imaging one or both the carotid artery in the neck and the temple.
  • the thermal camera also may sense the sinusoidal motion of the heart rate by imaging one or both the subject's arms and hands.
  • thermal imaging targets may include one or more of the subject's chest, nostrils, and mouth. Together, these signals can be employed in deriving data extrapolating further information about the health of a monitored subject.
  • the health monitoring device incorporates an uncooled microbolometer array in communication with a mobile computing device.
  • the health monitoring device incorporates audio data in monitoring and characterizing vital signs of a subject.
  • Audio data may include spectrograms in the audio spectrum, e.g., spectrograms derived from audio clips recorded by the audio sensor.
  • the audio sensor may generate audio signals via microphone, handset, or other transducer that converts sound into an electrical signal.
  • the microphone generates AC signals representing air pressure variations of a sound wave, e.g., sounds resulting from speech, breathing, respiratory sounds, or other sounds.
  • the microphone or an external device converts the AC signal into digital audio signals, also herein called audio data.
  • audio monitoring may be used for monitoring breath rate, for identifying abnormal respiratory sounds, and for recognizing subject's speech.
  • the health monitoring device may activate an audio mode in the event video monitoring fails to detect presence of the subject at a primary location.
  • the health monitoring device includes a processor configured to output a health determination relating to the one or more health parameters of the patient by inputting one or both of visual data and audio data into one or more machine learning models.
  • the health determination includes a value of the one or more health parameters, a binary classification of the one or more health parameters, a multiclass classification of the one or more health parameters, an event relating to the one or more health parameters, or a health anomaly relating to the one or more health parameters.
  • An example of a value of one or more health parameters includes systolic and diastolic blood pressure values.
  • An example of a binary classification of one or more health parameters is a binary flag.
  • An example of a multiclass classification of one or more health parameters is a multiclass classification of respiratory sounds.
  • An example of an event relating to one or more health parameters is an apnea event.
  • An example of a health anomaly relating to one or more health parameters is a sleep disorder.
  • one or more machine learning models include a supervised learning model trained to recognize features of interest.
  • a supervised learning model such as regression model includes a factorization machine.
  • machine learning models include an unsupervised learning model.
  • machine learning models may refer to methods such as logistic regression, decision trees, Random forest ensembles, neural networks, linear models, matrix reduction, and/or Bayesian models.
  • multiple machine learning models are employed in monitoring health parameters.
  • multiple machine learning models are combined in multimodal artificial intelligence (AI), in which two or more data types (e.g., thermal imaging spatial data, thermal imaging temperature data, audio data) are combined via multiple AI algorithms.
  • AI multimodal artificial intelligence
  • multiple machine learning models may operate simultaneously or may operate sequentially in monitoring health parameters.
  • a thermal imaging sensor monitors presence of a subject at a primary location. In the event thermal imaging generates a “no patient present” flag, this activates one or more other monitoring modality such as audio monitoring.
  • FIG. 1 illustrates an architecture of a health monitoring device 100 .
  • the health monitoring device includes a set of sensors 140 configured to receive signals pertaining to one or more health parameters without physical contact of the sensors with the subject 120 .
  • the sensors 140 are configured to receive signals through non-physical contact 125 with the subject 120 .
  • the sensors 140 include a thermal camera 144 that measures the thermal heat of the subject 120 and other objects in the field of view.
  • the thermal camera includes an infrared (IR) sensor, such as a microbolometer array.
  • the set of sensors 140 also include an audio sensor 148 .
  • the system 100 monitors health of a sleeping subject 120 .
  • the system 100 and its components can easily be adapted to other health monitoring applications that incorporate thermal imaging and audio sensing for monitoring a subject.
  • Other applications may include, e.g., monitoring pilots; car, bus, truck and train drivers; cyclists; automated preliminary health assessment, e.g., for triage in hospital or in doctor's office; infant care; and other individuals, e.g., persons under care of a healthcare worker or care giver.
  • one or both thermal imaging systems and audio systems track eye motion, heart rate, HRV, breathing rate, and breathing rate variability of a pilot or a professional driver to ensure they are alert and healthy throughout their shifts.
  • preliminary health assessment/triage metrology for use in hospital and doctor's office settings could be non-contact, self-administered by subject, and could take a few minutes to administer.
  • Mid-infrared imaging offers the advantage in various applications that it is not affected by environmental illumination. For example, MIR thermal imaging to observe a professional driver would not be affected by oncoming headlights at night or by occasional glare during the day. Using MIR thermal imaging in a kiosk for preliminary health assessment would not be affected by changing ambient light conditions in a doctor's office.
  • thermal camera 144 incorporates uncooled IR image sensors called microbolometers, built around arrays of tiny thermal detectors.
  • thermal camera is also referred to as infrared (IR) camera and as microbolometer.
  • IR infrared
  • a microbolometer is a specific type of bolometer used as a detector in a thermal camera. Infrared radiation in the mid-IR range, e.g., with wavelengths between 7.5-14 ⁇ m, strikes the detector material, heats it, and changes its electrical resistance. Arrays of many thermally isolated microbolometers can record images in the thermal IR. This resistance change is measured and processed into temperatures that can be used to create an image. Unlike other types of infrared detection equipment, microbolometers do not require cooling, thereby reducing their cost, size, and complexity. Microbolometers are commercially available in handheld devices, including smartphone cameras.
  • thermal imaging allows measurement to be performed in a completely dark room so the subject 120 is not disturbed by an external source light required by an optical camera.
  • IR cameras have the advantages that they measure photons radiated from a regarded object, they do not need any external light that may distract or disturb a subject, and they are insensitive to viewing angle.
  • Health monitoring device 100 may utilize uncooled microbolometer arrays for non-contact measurement of a subject's stance, rate of breathing, temperature, pulse rate and pulse profile.
  • IR cameras measure radiated photons from a regarded object, they do not need any external light source that can be distracting to the subject 120 . Additionally, IR cameras are insensitive to viewing angle.
  • thermal imaging subjects could be pilots; car, bus, truck and train drivers; cyclists; hospital and doctor triage users; and individuals. For drivers, stance monitoring could establish drowsiness, fatigue, inattention or stress.
  • signal processing 154 of output of thermal camera 144 may employ functions from OpenCV (Open Source Computer Vision Library) a library of programming functions employed in at real-time computer vision.
  • OpenCV Open Source Computer Vision Library
  • functionalities of facial feature recognition and object recognition are expanded to work with a thermal camera.
  • Features of the face ducts, eyes, temples, carotid artery, etc.
  • an existing feature map is compared with one or more acquired thermal images. This comparison may include a series of convolutions for feature extraction, with bounding boxes for each relevant feature.
  • signal processing 154 extracts patterns of thermal intensity of sensed features as digital data streams for conversion to frequency domain.
  • This conversion may employ the Fast Fourier Transform used by NumPy.
  • NumPy is a library for the Python programming language including a large collection of high-level mathematical functions to operate on multi-dimensional arrays and matrices.
  • the NumPy library includes the Fourier transfer function Discrete Fourier Transform (numpy.fft). In discrete Fourier transform (DFT), both the original signal and its Fourier transform are replaced with discretized counterparts. The DFT separates its input into components that contribute at discrete frequencies.
  • NumPy Fast Fourier Transform is represented by the following formula:
  • the signal in the Fourier domain typically has a dominant feature that will correspond to the breathing rate or pulse rate.
  • Signal processing 154 may filter the acquired signal via band pass filter around this frequency to improve the signal.
  • calculated rates are further refined using a factorization machine in machine learning models 160 .
  • the factorization machine fits the calculated rates to appropriate variables, given by the following formula:
  • the signal also may be converted to the frequency domain using a wavelet transform. This allows for the signals of interest to change in the frequency domain as a function of time, unlike the more common Fourier transform. Typically, Fast Fourier Transform using a rolling window average is sufficient, however. This is because health monitoring signals most often change slowly with occasional periods of extreme change correlated with a health event.
  • model development seeks to find a fitted model and to gain insights into how an arbitrary health feature relates to pulse, breathing, and other physical characteristics.
  • factorization machine is used as a measurement device that can return an accurate value for training purposes. Data exploration of the calculated factors may identify a relationship between the raw data returned by the camera and the desired value.
  • processing unit 150 analyzes each frame of dynamic thermal images captured by thermal camera 144 .
  • machine learning models 160 incorporate a factorization machine, as shown in the scheme 200 of FIG. 2 .
  • Factorization machines are supervised learning models that can perform both regression and classification. They are non-linear models that model interactions of variables 210 (x 1 , x 2 , x 3 . . . ) by generating feature vector representations 220 (including vectors v 1 , v 2 , v 3 . . . ) and by mapping 230 , 240 feature interactions to a model output 250 in a low dimensional space.
  • a factorization machine has several advantages over traditional regression techniques commonly employed in machine learning. First, by decomposing a variable into multiple factors, higher order effects, e.g., biological effects, may be observed. Additionally, while a single variable may be over-fitted and thus fail to respond appropriately to a change in the system, a factorization machine is less susceptible to this.
  • FIG. 3 shows a flow chart of recursive processes 300 of a factorization machine.
  • the factorization machine algorithm employs reinforcement learning that will continually adjust to be in agreement with whatever secondary data stream is being used.
  • a variable “figure of merit” is defined as
  • Factorization machines can provide supervised machine learning models with excellent interpretability of modeled health parameters. For example, a matrix reduction and averaging procedure can derive matrix-reduced data to reconstruct waveforms with intuitive relevance to health parameters.
  • machine learning models apply a factorization machine to determine the subject's tidal volume.
  • tidal volume refers to the amount of air that moves in or out of the subject's lungs with each respiratory cycle. Coupled with pulse and breath rate, knowledge of the tidal volume would be an important indication of the subject's health.
  • a sensor may detect a mass of hot, moist air being exhaled, but this technique is sensitive to humidity, ambient temperature, and viewing angle. Instead, by using the motion of the chest, mouth and nostrils, and frequencies as factors, machine learning models may be trained to predict the tidal volume of the breathing. By examining spatial displacement as well as temperature gradients, machine learning models may incorporate contributions of these factors to the overall factorization machine.
  • the canny edge detection algorithm with an applied Gaussian filter may be used to determine the motion of the chest, distinguishing the relatively hot body against the cool background. Temperature differences make it straightforward for the algorithm to distinguish between the different temperatures present on the body, which includes covered portions, hands, and hair, relative to the much cooler sheets and blankets.
  • the magnitude and profile of chest motion can be used in training a machine learning model 160 , provided with actual measured breath volume, to predict tidal volume.
  • chest motion can be predictive of tidal volume, while the frequency is used to determine breathing rate. This motion can be detected from a variety of angles and under covers.
  • FIG. 4 is a schematic block diagram of a method 400 for training a chest motion model.
  • This method takes as inputs measurements of breath volume, and thermal camera output data.
  • Thermal camera output data may comprise a float array.
  • Thermal camera data and breath volume data are passed through a band pass Gaussian filter to derive chest motion.
  • the band pass is centered at a calculated value for average chest motion over a preceding set of thermal images, e.g., the last 1000 frames.
  • the chest motion data is analyzed to derive breathing rate.
  • the canny edge algorithm may be applied to thermal camera data to derive magnitude of displacement of the chest d/dt.
  • Breathing rate, d/dt, and numeric integration of chest motion are inputs to a model to determine plume volume.
  • a computer vision (CV) algorithm may extract the pixels related to each important feature from a digitized thermal image in performing feature recognition of a subject's face and neck.
  • this algorithm seeks to identify the subject's neck and temples. For each point, all pixels within a certain square radius are used. A weighting function is applied so that the maximum thermal point has the greatest weighting, since presumably this corresponds to the point closest to the artery. Each of these points may be represented by a 2D array that varies in time.
  • FIG. 5 is a schematic diagram of an SQL architecture 500 of features identified via facial feature recognition. Facial feature/element recognition techniques based on thermal imaging methods of the disclosure can derive spatial and temperature visual data that may be used in modeling health parameters.
  • health monitoring device 110 analyzes these data points over time to extract information about the heart rate, heart rate waveform, and variability. For a moment in time, each 2D array corresponding to a feature of interest is added to a single array. For two features (neck, temple) with each feature represented by an M ⁇ N array, this step results in a 2 ⁇ M ⁇ N array for a given moment in time.
  • CV analysis may perform principal component analysis to extract the most important information from this array.
  • Principal component analysis can be calculated several ways, such as truncated singular value decomposition. This PCA technique does not require calculating a matrix inverse. Principal components can be considered a linear projection along the axis such that higher dimensional data is efficiently represented as the function of a single variable.
  • the first principal component corresponds to the axis along which covariance is maximized, and variance is minimized.
  • this first principal component represents a stable pulse rate measurement.
  • the second principal component corresponds to maximized variance.
  • L2-norms (vector distance) of principal components in the data set indicate stability of the measurements of pulse rate variability. If these values change rapidly, further investigation is required.
  • the pulse waveform becomes readily apparent when matrix reduction is performed on successive arrays.
  • This analysis provides an approximate waveform that can be used to calculate its mean, standard deviation, Gaussian spread, and other parameters.
  • These derived data may be employed to construct a blood pressure model.
  • a supervised learning model may take as input a vector of values corresponding to the waveform.
  • a factorization machine then may train the model to fit an experimentally obtained invasive blood pressure for the waveform.
  • Two factorization machines may be used to fit the systolic and diastolic pressures, e.g., as shown in the representative graph 600 of FIG. 6 .
  • the factorization machines may be applied to qualify and quantify atrial fibrillation and other cardiac ailments.
  • FIG. 7 shows a training process 700 for determining blood pressure.
  • Blood pressure has the systolic and diastolic numbers associated with the waveform of the heart rate. Using the ratio of the rise and fall times, the method 700 can predict these numbers.
  • the method 700 uses blood pressure profile and a secondary source in a supervised learning model to predict blood pressure.
  • the method derives the blood pressure profile by inputting a floating-point array of thermal imaging data and a feature coordinate list to determine pulse data in the time domain.
  • the method applies numeric integration to the time domain pulse data to derive a blood pressure profile, including wave power and first derivative of the pulse data.
  • the blood pressure profile is input to a regression/factorization machine, along with data obtained from a secondary source.
  • the secondary source includes ground truth data obtained from an intra-arterial measurement method.
  • Disclosed embodiments apply CV image processing to respiratory systems, which are lower frequency. Respiratory systems analysis has the advantage that it may incorporate audio data as additional acquired data. For two features, chest and throat movement, with each feature represented by an M ⁇ N array, this process derives a 2 ⁇ M ⁇ N array for a given moment in time. In an embodiment, health monitoring device 110 analyzes these data points over time to extract information about breathing rate, respiration volume (waveform), and breathing rate variability.
  • PCA may be performed to address noise and many similar data points.
  • the principal component the most important feature, the breathing waveform, becomes readily apparent when feature movements are represented in successive arrays.
  • the first principal component corresponds to the axis along which covariance is maximized, and variance is minimized. This represents a stable breathing rate measurement.
  • the second principal component corresponds to maximized variance representing breathing rate variability.
  • the L2-norms of the principal components give a sense of stability of the system, e.g., a metric of breathing rate variability such as apnea. If they change rapidly, further investigation is required.
  • Blood oxygen saturation is another useful medical characteristic that can be determined from thermal imaging.
  • Disclosed embodiments apply similar CV image processing techniques to pulse oximetry of blood oxygen saturation.
  • Thermal imaging outputs an approximate waveform that may be analyzed via CV techniques to calculate its mean, standard deviation, Gaussian spread, and other characteristics.
  • these visual data are employed in constructing a blood oxygen saturation model.
  • a supervised machine learning model may take as input a vector of values corresponding to the waveforms.
  • a regression/factorization machine trains the model to fit experimentally obtained via oximeter measurements for the waveform.
  • Oximetry is a traditional technique in which blood oxygen saturation is measured by passing light through a fingertip and comparing the absorption of the light through oxygenated versus unoxygenated blood.
  • oximetry is used as ground truth in training the blood oxygen saturation model.
  • Respiratory system analysis may incorporate audio data, such as spectrograms in the audio spectrum, in addition to the vision data. Although these audio data have different units, supervised learning models may be trained to convert between them. Adding audio spectrum values to the thermal imaging values in a factorization machine may enable the factorization machine to automatically learn relationships between audio variables and thermal/spatial variables. In an example, this procedure was applied to three features with each feature represented by an M ⁇ N array and a 3 ⁇ M ⁇ N array for a given moment in time.
  • health monitoring device 110 communicates with a mobile computing device, e.g., smart phone 130 .
  • the mobile computing device may act as power source and device to compute signals.
  • raw data is processed on the phone, and derived signals and flags 158 are saved remotely when the run is stopped.
  • a mobile computing device 130 also may perform calculation for real-time monitoring of vital signs.
  • Device 130 includes sufficient RAM and computing power to undertake all real-time simultaneous modeling and statistical analysis.
  • a code base for smartphone 130 was developed in python, C, and C++, and converted using public cloud computing tools provided by Microsoft Azure. The viewing angle may also be easily chosen with simple wall mounts in a wide variety of settings.
  • Using smart phone 130 as processor provides a cost-effective design for health monitoring applications, such as monitoring subject 120 during sleep.
  • Other health monitoring applications may replace smart phone 130 with another computing device suitable for the application.
  • the health monitoring processor may be included in a vehicular computer system.
  • the processor may be included in a device such as a kiosk.
  • a smart phone or other mobile device 130 can extend sensing capabilities for health monitoring of a subject.
  • a back facing camera of a smart phone effects video imaging in visible light spectrum of neck and temple of a subject to track and create blood pulse profile.
  • These optical sensor readings can provide measurements of pulse rate and pulse rate variability as inputs for predictive modeling of systolic and diastolic blood pressures.
  • Processing unit 150 can be implemented using a single-processor system including one processor, or a multi-processor system including any number of suitable processors that may be employed to provide for parallel and/or sequential execution of one or more portions of the techniques described herein. Processing unit 150 performs these operations as a result of central processing unit executing software instructions contained within a computer-readable medium, such as within memory. As used herein, a module may represent functionality (or at least a part of the functionality) performed by a processor.
  • Device 100 includes a power supply 174 for powering components of health monitoring device 110 , including the mobile device 130 and IR camera 144 .
  • power supply 174 is a battery that can be recharged by power source 172 , e.g., via continuing mains feed charging.
  • Power supply 172 may be configured to provide non-interruptible operation during power outs.
  • Communications module 178 may support various wired and wireless various communication protocols such as Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), and IEEE communication protocols, and standard wireless protocols such as Bluetooth®, BLE, Wi-Fi, NFC, ZigBee®, and the like.
  • BLUETOOTH is a registered trademark of Bluetooth Sig, Inc., Kirkland Wash.
  • communication protocols may include wireless communications according to BLUETOOTH specification sets or another standard or proprietary wireless communication protocol.
  • communication protocols may also include communications over a cellular network, including, e.g., a GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), or EDGE (Enhanced Data for Global Evolution) network.
  • GSM Global System for Mobile Communications
  • CDMA Code Division Multiple Access
  • EDGE Enhanced Data for Global Evolution
  • Cellular communications may accommodate operational and system uploads, real-time alarm monitoring outputs to healthcare professionals and care givers, session reports archiving, and reporting to healthcare professionals and care givers.
  • wireless fidelity (Wi-Fi) communications may accommodate functions such as operational and system uploads, real-time alarm monitoring outputs to healthcare professionals and care givers, session reports archiving, and GP/care giver reporting to healthcare professionals and care givers.
  • communication protocols may include wired duplex communications with IR camera 144 .
  • Audio sensor 148 may generate audio signals via microphone, handset, or other transducer that converts sound into an electrical signal.
  • a microphone converts the air pressure variations of a sound wave, e.g., resulting from speech, breathing, or other respiratory sounds from subject 120 .
  • Microphones convert sound waves into AC electrical audio signals and are therefore analog devices.
  • analog signals from audio sensor 148 signals are converted to digital audio signals processed by filtering/signal processing module 154 .
  • a microphone or other transducer may output digital audio signals via built-in analog-to-digital converters, or may be coupled to an external analog-to-digital converter device that outputs digital audio signals.
  • Audio monitoring may be used for monitoring breath rate, and may be trained to identify abnormal respiratory sounds. Audio monitoring can measure various physical attributes: frequency, e.g., the number of the sound waves or vibrations per second; intensity or amplitude, e.g., the height of sound waves from their mean; and quality or timbre. Quality or timbre differentiates two sounds with the same frequency and intensity. The timbre of a sound depends on its wave form, including the number of overtones or harmonics, their frequencies, and their relative intensities (amplitudes).
  • audio sensor 148 is used to obtain audio intensities and other audio parameters of breathing, which may provide an additional signal to confirm the accuracy of vision analysis of chest motion and breathing.
  • filtering/signal processing module 154 passes the audio signal to the frequency domain, in which a large magnitude signal is taken as the breathing rate. Module 154 may filter out background noise, as most sounds are broad-bandwidth, low intensity in nature. If several prominent peaks occur in the frequency spectrum, the peak closest to the frequency of breathing calculated by the chest and throat motion and exhalations is chosen as the target feature in performing band pass filtering.
  • a digital band pass filter may be employed. This procedure identifies the two frequencies with the greatest magnitudes, corresponding to breathing of the two individuals.
  • the original signal is duplicated then passed through band pass filters centered at each of these two frequencies in order to find the breath rate for each person.
  • the band pass filters may filter out extraneous noises that might briefly conceal breathing sounds, such as thunder or vehicle noise.
  • the central frequency of the band pass filter is calculated via a rolling average, in which the frequency of a preceding time interval (e.g., previous 5 minutes) is used. This ensures that the signal is not lost if, over the course of a night, the breathing slows, as might be expected.
  • filtering/signal processing module 154 does not perform band pass filtering since training models use higher-order overtones are used as inputs.
  • Flags 158 may be classified into several categories. If the patient leaves the frame, the temperature sensed by thermal camera 144 will drop to ambient bed temperature. This flag will note the patient has left the bed. If the maximum temperature in frame remains surface temperature, but the position of sleep impedes one or more signals, the flag will note which signals are impeded and attempt to make a guess as to why, with the position of the sleeping patient and pillows and blankets covering them being two possibilities. These regions are distinguished by their relative temperatures. Further, in the event a signal is lost for no obvious reason, processing unit 150 can check motion data during the preceding time period (e.g., five minutes). Motion may be measured, e.g., by the displacement vector of the canny edge. If this metric falls below a percentage threshold, it may be assumed some system error is responsible. In this case, the temporarily saved data may be stored in a data log for the night.
  • the preceding time period e.g., five minutes. Motion may be measured, e.g., by the displacement vector of the canny edge.
  • audio signal 148 is analyzed to recognize the subject's speech.
  • speech recognition employs Carnegie Mellon's Hephaestus Speech-related software systems to throw flags 158 when key phrases, such as “help” or “I've fallen,” are spoken.
  • the spectrograms of these audio clips are passed through a triangular Mel filter, which weights frequencies dominant in human vocalizations.
  • the filtered spectrograms are subsequently trained in a supervised learning model.
  • Key phrase flags 158 may trigger alarms 176 , such as audible alarms or visual alarms displayed to healthcare professionals and care givers.
  • system 100 may acknowledge that ‘cries’ were received over phone speaker 130 .
  • keyword recognition can be used to begin a monitoring run and/or to end a monitoring run.
  • the visual system 144 , 154 includes a patient presence model that identifies when no subject 120 is in the field of view, e.g., generating a “no patient present” flag.
  • an additional audio monitoring measure 148 , 154 is activated, e.g., to identify falling sounds and sounds likely to occur when a patient rises, e.g., doors opening and closing, and toilet flushing.
  • a fall has a broad spectrum, making it an impracticable measure to track continuously.
  • monitoring for fall stops once the patient is in the field of the view of the camera. If a fall occurs, a flag 158 is thrown and care givers are notified 176 .
  • Other modalities may be activated by “no patient present” flag, such as activating a front facing camera and directing local WiFi to turn on lights in a room. These additional modalities may be deactivated if the video system later detects patient presence.
  • a flag 158 is thrown to indicate a signal has been lost. If the maximum temperature sensed by thermal camera 144 is below 25° C., it can reasonably be inferred the patient has moved out of the field of view, and the flag thrown notes this. All flags thrown over the course of a night may be saved to a text file as well as to a data log for the time of each flag. In an embodiment, frame data may be saved for a limited time (e.g., 5 minutes) before being replaced, while attributes derived from the frame data may be stored in a data log for the night.
  • a limited time e.g., 5 minutes
  • standard deviation, first and second derivative, and splining fit are all stored by default. These data have various applications in determining patients' health. Using the derivative and standard deviation, health anomalies such as apnea and atrial fibrillation can be discovered and flagged 158 , which may result in an alert 176 to a healthcare worker or care giver. In a use case, this system serves as a smart monitor to aid nurses and night-staff in rest home and hospice settings. In an embodiment, a flag 158 is a binary classifier for which minimizing false negatives are prioritized over minimizing false positives. Model training may select weighting errors to provide a priori Bayesian distributions for probabilistic alerting of healthcare workers. Model training may employ joint probabilities to allow incorporating other signals relevant for the patient's health.
  • the system 100 may store data in local databases of health monitoring device 110 and mobile computing device 130 .
  • the system also may store data, e.g., archived data, in cloud databases 180 .
  • Cloud 180 may be a third-party cloud.
  • Databases are organized collections of data, stored in non-transitory machine-readable storage.
  • the databases may execute or may be managed by database management systems (DBMS), which may be computer software applications that interact with users, other applications, and the database itself, to capture (e.g., store data, update data) and analyze data (e.g., query data, execute data analysis algorithms).
  • DBMS database management systems
  • the DBMS may execute or facilitate the definition, creation, querying, updating, and/or administration of databases.
  • the databases may conform to a well-known structural representational model, such as relational databases, object-oriented databases, or network databases.
  • Example database management systems include MySQL, PostgreSQL, SQLite, Microsoft SQL Server, Microsoft Access, Oracle, SAP, dBASE, FoxPro, IBM DB2, LibreOffice Base, and FileMaker Pro.
  • Example database management systems also include NoSQL databases, i.e., non-relational or distributed databases that encompass various categories: key-value stores, document databases, wide-column databases, and graph databases.
  • FIG. 8 illustrates field names, field data types, and PK primary key values of an example SQL architecture.
  • sampling rate per Nyquist's Theorem.
  • the discrete nature of sampling may be fitted to assumed continuous functions as a splining curve whose higher-order terms could be useful for determining additional data.
  • One goal of disclosed systems and methods is to extrapolate general diagnoses from data collected.
  • a corpus of data collected over many patients concerning their breathing rate, pulse, lung capacity, and sleep cycle may offer deeper insights into the well-being of monitored patients.
  • the system applies data mining methods to create correlation matrices.
  • Correlation matrices may be used to describe the health of a patient and to identify higher risk conditions of the patient based on covariance.
  • higher-order terms may correspond to physical phenomena such as the correlation of blood pressure with pulse waveform.
  • Machine learning modeling 160 may calculates the posterior distribution of each pair to determine which signal is most likely flawed. This information can then be used to adjust the neural network and filters that determine that signal. Bayesian machine learning then can compare the signals again to find better agreement.
  • Disclosed embodiments may apply audio monitoring to recognize coughs.
  • Coughs are typically characterized by three phases: an initial loud, peak in intensity, followed by a quick attenuation and finally a voiced phase.
  • a recognition algorithm may consider the number of cough sounds, the portion of breaths that include or are disrupted by a cough, and cough epochs, defined herein as the number of coughs with no more than two second interval between them. Chest movement is a further classifying parameter. Also pertinent are the audio breathing phase analysis and chest dynamics for characterizing asthma, pneumonia, and other abnormal respiratory conditions.
  • FIG. 9 illustrates an audio pre-processing method 900 for identifying respiratory ailments.
  • adventitious lung sounds may be classified in frequency bands, e.g., low frequency (100 to 300 Hz), middle frequency (300 to 600 Hz), and high (600 to 1,200 Hz) frequency bands.
  • Frequency is measured objectively, while pitch is the subjective perception of sound's frequency.
  • Stridor are high pitched sounds generated by turbulent airflow in the larynx or bronchial tree, and are related to an upper airway obstruction as with epiglottitis, croup, and laryngeal edema.
  • Inspiratory gasp sounds or whoops e.g., after a bout of coughing, are caused by fast moving air through the respiratory tract and a pathognomonic symptom of whooping cough (pertussis).
  • Squawks, or short wheezes are generated by oscillation at the peripheral airways and associated hypersensitivity to pneumonia.
  • Fine crackle sounds are caused by explosive openings of the small airway and usually associated with pneumonia, congestive heart failure, and lung fibrosis.
  • Coarse crackle sounds are generated by air bubbles in large bronchi and can be heard on patients with chronic bronchitis, bronchiectasis, as well as COPD.
  • Pleural rub sounds are non-musical rhythmic sounds due to the rubbing of pleural membranes when breathing and are usually caused by inflammation of the pleural membrane. These qualitative differences can be rigorously characterized by their respective spectrums. Chest and throat video dynamics typically correlate, particularly for respiratory events associated with lower frequency sounds.
  • System 100 includes tools for training machine learning models for the diagnosis of lower respiratory tract disease, upper respiratory tract infection, pneumonia, bronchiolitis, croup, asthma exacerbation/reactive airway disease, chronic obstructive pulmonary disease, chronic obstructive pulmonary disease exacerbation and obstructive sleep apnea.
  • Potential use cases include healthcare providers in telehealth, emergency department, urgent care and primary care settings as well as humanitarian facilities in the developing world.
  • FIG. 10 is a flow chart diagram 1000 of training methods for analysis of audio events.
  • Process 1010 applies Fourier transform and Hann smoothing to audio input signals.
  • Process 1010 then performs segmentation to produce pre-processed audio spectra, e.g., for neural network modeling.
  • Process 1020 trains a model to identify respiratory conditions and patient movements.
  • Process 1020 inputs pre-processed, segmented spectra from process 910 along with annotated data into a convolutional neural network (CNN).
  • the CNN may be trained as a recurrent neural network, a network in which connections between nodes form a directed graph along a temporal sequence.
  • the CNN performs feature matching to respiratory conditions and to sounds associated with patient movements.
  • Processes 1010 , 1020 illustrate an approach to acquisition and analysis of adventitious lung sounds, which may signify pathological processes in pulmonary tissue or airways.
  • This approach employs a convolutional neural network to classify spectrograms.
  • the respective spectrograms of relevant sounds are preprocessed via Hann smoothing.
  • Hann smoothing generally has the advantage of removing broad spectrum, low intensity noise that may be present from traffic, television, or weather.
  • the recordings are split into samples, e.g., 1 second samples, and are identified using a deep convolutional neural network.
  • Processes 1010 , 1020 also encompass harmonics of the fundamental frequency present from breathing. Using a regression classifier model, higher order harmonics correspond to additional parameters fitting the model to the sounds, with the constant term in the model being the natural breathing rate.
  • Process 1030 trains a model for word/phrase recognition. Processes input a corpus of keywords and key phrases. Process 1030 applies the Hephaestus model for audio recognition of keywords and key phrases.
  • health monitoring methods disclosed herein utilize two additional independent metrics to assess and classify respiratory effort (depth of inspiration) and tidal volume estimations from chest movement modelling. These metrics are N/M expiration and chest and throat movement.
  • Real-time processing algorithms can offer fundamental information in detection and classification of adventitious sounds and can enable timely identification of diseases, as well as changes in their severity.
  • Physical characteristics that may be derived from raw frame data of thermal sensor 144 include breath-rate waveforms.
  • the area feature (integral under period), distance feature (period), and their derivatives can operate at between 0-6 Hz and have medical applications.
  • calculating the waveform for both features follows a similar procedure (“shift-sum procedure”).
  • a filter is not suitable because the waveform is a complex, multi-peaked function. Therefore, to reduce noise and ensure the waveform is properly populated, the average period is calculated over a time interval (e.g., 5 minutes). If the standard deviation is too high, the waveform is not calculated because the period is changing which indicates the patient's vital signs are shifting.
  • each period is transformed according to sin(w(t ⁇ nT)), where n is the period number and T is the average period. Using this procedure, all waveforms nearly coincide and a waveform with a large sample set and low uncertainty may be obtained.
  • heart rate refers to the number of times a subject's heart beats per minute.
  • Heart rate variability measures the time between each heartbeat, also called an R-R interval in ECG signals. Beat-to-beat interval variation may be measured in milliseconds and can vary depending on a number of factors. For instance, the interval between heartbeats is generally longer while exhaling and shorter while inhaling. Various factors can influence or change a subject's HRV metrics, such as exercise volume and intensity, chronic health conditions, quality of sleep, and diet. Another reason for HRV is operation of the autonomic nervous system, which controls the involuntary aspects of physiology, via two branches, parasympathetic (deactivating) and sympathetic (activating).
  • HRV is an extremely sensitive metric that fluctuates greatly throughout the day, from one day to the next, and from one person to another. Younger people tend to have higher HRV than older people, and males often have slightly higher HRV than females. Elite athletes usually have greater HRV.
  • non-contact health monitoring systems and methods of the present disclosure can measure HRV directly from blood pressure pulse profiles representing heart hydraulic performance, which can be more informative than EKG pulse commands.
  • HI Sudden hemodynamic instability due to cardiovascular and/or cardiorespiratory distress is a common occurrence.
  • Causes can include hemorrhage, sepsis, pneumonia, heart failure, and others.
  • Due to the body's compensatory mechanisms, heart and respiratory rate, and blood pressure can be indicators of HI. When detected late or left unrecognized, HI can lead to complications and even death.
  • Signs of hemodynamic instability include having arterial systolic blood pressure ⁇ 80 or >220 mmHg, a diastolic blood pressure of >110 mmHg, a pulse rate of ⁇ 40 or >160, a respiratory rate of ⁇ 8 or >36, a SpO 2 ⁇ 90%, and temperature.
  • Pulse rate variability (PRV) has been demonstrated to reflect status of the autonomic nervous system. Pulse rate has been identified as a minor indicator, while PRV has been shown to indicate significant perturbations.
  • Non-contact monitoring methods disclosed herein may be employed in accurately predicting problems before they occur. This capability can aid in diagnosis and treatment of a deteriorating patient. In various embodiments, these predictions may be based on one or more metrics including core temperature from thermal imaging; respiratory rate from audio; thermal/spatial analysis of mouth, nose and chest analysis; and respiratory volume from spatial chest analysis. These real-time measurements can provide a rich and comprehensive data set for training a machine-learning HI predictor to improve model performance. Additionally, ECG is a contact non-invasive device while embodiments disclosed herein provide a non-contact device. In use cases, a non-contact HI prediction model could be deployed to monitor patients at home, in the field and even while driving.
  • disclosed embodiments may be applied to understand disordered sleep of a subject in order to identify a solution.
  • Disordered sleep varies considerably from person to person and can be of a physiological and psychological nature.
  • a disrupted biorhythm, poor sleep posture, lowered resilience to stress, and psychological distress such as worry about being able to fall asleep all can lead to sleep disruption.
  • disclosed embodiments may apply measurements of breathing rate and breath volume and could implement a feedback function to reduce breathing rate. This process may output an audio signal via mobile device 130 or via Bluetooth ear buds.
  • Remedial measures could include, e.g., reducing pulse rate, and implementing an audio meditation session to induce sleep.
  • a photoplethysmogram is an optically obtained plethysmogram that can be used to detect blood volume changes in a microvascular bed of tissue, effectively a spatial measurement.
  • Systems and methods of the present disclosure provide thermal plethysmogrammy (thermal PG), offering various advantages over optically obtained plethysmogram.
  • Optical PPG is much noisier and indistinct than thermal PG, and only thermal PG can identify the end and start of diastolic pressure pulses.
  • PPGs acquired at finger or wrist have much more damped and indistinct pressure pulse profiles than are available from thermal PG acquired at arteries at temple, neck or upper arms of a subject.
  • a machine learning model 160 may input thermal PG data to estimate diastolic pressures.
  • Blood pressure has systolic and diastolic numbers associated with the waveform of the heart rate ( FIG. 6 ).
  • a supervised learning model may be used to predict blood pressure from the thermal PG system. Given a reasonable accurate characterization of the waveform, model exploration can investigate its properties to infer the atrial fibrillation of the patient.
  • Pulse shape and energy of the heart may be derived from the raw frame thermal PG data.
  • the area feature (integral under period), distance feature (period), and their derivatives can operate at between 0-6 Hz and have medical applications.
  • calculating the waveform for both features follows a similar procedure (herein called shift-sum procedure).
  • a filter is not suitable because the waveform is a complex, multi-peaked function.
  • the average period is calculated over a time interval (e.g., 5 minutes). If the standard deviation is too high, the waveform is not calculated because the period is changing which indicates the patient's vital signs are shifting.
  • each period is transformed according to sin (w(t ⁇ nT)), where n is the period number and T is the average period. Using this procedure, all waveforms nearly coincide and a waveform with a large sample set and low uncertainty may be obtained.
  • the pulse and its waveform may be extracted from two signals, the spatial variation of the pulse and the intensity (temperature) variation of the pulse.
  • this procedure can calculate the periodicity using the systolic peak.
  • the upstroke and decline then are used as inputs to the learning model for the blood pressure.
  • the shape and intensity of the pulse are used to approximate the variations of pressure throughout the waveform.
  • An individual waveform which may be under 1 second and may be sampled, e.g., 15 times, generally does not neatly define the shape of the pulse from which blood pressure may be derived.
  • the above-described shift-sum procedure addresses this problem.
  • temperature measurements can provide results superior to spatial measurements. Temperature measurements typically show reduced sensor noise.
  • Disclosed systems and methods can monitor physical characteristics of a sleeping subject 120 associated with REM sleep.
  • REM sleep the stage of sleep associated with dreaming, is very different physiologically from the other stages of sleep. Five stages of sleep include:
  • REM atonia an almost complete paralysis of the body, is effected through the inhibition of motor neurons. Some localized twitching and reflexes still can occur. Lack of REM atonia causes REM behavior disorder.
  • RBD REM sleep behavior disorder
  • the paralysis that normally occurs during REM sleep is incomplete or absent, allowing the individual to “act out” his or her dreams.
  • RBD is often characterized by the acting out of dreams that are vivid, intense, and violent.
  • Dream-enacting behaviors can include talking, yelling, punching, kicking, sitting, jumping from bed, arm flailing, and grabbing.
  • An acute form may occur during withdrawal from alcohol or sedative-hypnotic drugs
  • Systems and methods of the disclosure may analyze these and other characteristics in monitoring sleep, e.g., to measure REM sleep and monitor movements of a resting body.
  • a larger set of metrics that may be employed in sleep metrology include core temperature, heart rate, HRV, breathing rate, respiration rate, thermal sensing of REM, audio correlations with breathing rate and spectral analysis of breathing ailments, and visual monitoring of RBD movements. In a use case, this data is employed in monitoring and characterizing any RBD.
  • This ensemble of metrics can be used to more precise monitor and identify stages of sleep of a subject 120 . Facial feature recognition ( FIG. 5 ) may be applied to detect edges for identifying eye motion in monitoring REM sleep.
  • the system ranks quality of sleep based on several factors. These factors may include standard deviation of the breath rate, standard deviation of the heart rate, motion of the eyes (which could indicate REM sleep), and amount of motion of the patient, e.g., as defined by velocity of tracked features.
  • a machine learning model applying classifier/factorization machine prediction acts as a multivariate sleep quality classifier.
  • a machine learning model applying random forest classification acts as a multivariate sleep quality classifier.
  • Disclosed embodiments monitor several sleep apneas.
  • Obstructive sleep apnea is caused by a blockage of the airway.
  • CSA central sleep apnea
  • Complex sleep apnea is a combination of the two conditions.
  • the brain rouses the sleeper, usually only partially, to signal breathing to resume.
  • REM rapid-eye-movement
  • the sleeping CSA patient displays not a periodic failure to breathe but a periodic shallow breathing or under-breathing that alternates with deep over-breathing, a condition known as Cheyne-Stokes breathing.
  • the disorder reduces oxygenation of the blood, further stressing the sleeper's health.
  • Disclosed embodiments recognize several sleep apneas and may generate an alert/alarm if an apnea event occurs.
  • thermal imaging correlates facial feature recognition of tear ducts temperature with core temperature, e.g., to determine if an individual is healthy. Skin temperature is different than core temperature but useful to monitor to check if person is in field of view and for possible health issues. Tear duct temperature is representative of core temperature. Over the course of a night, core temperature decreases slightly.
  • feature detection observes the tear ducts. Locations of the tear ducts may be identified using the same machine learning model that has been trained to identify other facial features ( FIG. 5 ).
  • This thermal imaging data may be employed to calculate core temperature.
  • FIG. 11 is a graph of maximum facial temperature as a function of camera distance, measured for 44 year old female and 7 year old male. The imaging system estimates proximity of the thermal camera to the subject to correct core temperature readings. Based on field of view and number of pixels occupied by face, the system can estimate distance from the camera. Using this estimate, the correction reduces temperature recorded by camera for range.
  • This calculation and predetermined relationship shown in FIG. 11 enable accurate determination of core temperature from tear duct temperature observations. In an example, distance is measured using FLIR® Research Studio software. FLIR is a registered trademark of Teledyne FLIR, LLC, Wilsonville Oreg.
  • FIG. 12 is a schematic block diagram of a general health monitoring architecture 1100 . Key functions include:
  • process flow diagrams are provided merely as illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order presented.
  • the steps in the foregoing embodiments may be performed in any order. Words such as “then,” “next,” etc., are not intended to limit the order of the steps; these words are simply used to guide the reader through the description of the methods.
  • process flow diagrams may describe the operations as addition, the order of the operations may be rearranged.
  • a process may correspond to a method, a function, a procedure, a subroutine, a subprogram, and the like.
  • the process termination may correspond to a return of the function to a calling function or a main function.
  • Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof.
  • a code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements.
  • a code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents.
  • Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.
  • the functions When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium.
  • the steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a computer-readable or processor-readable storage medium.
  • a non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another.
  • a non-transitory processor-readable storage media may be any available media that may be accessed by a computer.
  • non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor.
  • Disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
  • the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable medium and/or computer-readable medium, which may be incorporated into a computer program product.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Biomedical Technology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Pathology (AREA)
  • Biophysics (AREA)
  • Physiology (AREA)
  • Artificial Intelligence (AREA)
  • Cardiology (AREA)
  • Pulmonology (AREA)
  • Mathematical Physics (AREA)
  • Psychiatry (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Fuzzy Systems (AREA)
  • Evolutionary Computation (AREA)
  • Dentistry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)
  • Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
US17/510,357 2020-10-29 2021-10-25 Cardiopulmonary health monitoring using thermal camera and audio sensor Abandoned US20220133156A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/510,357 US20220133156A1 (en) 2020-10-29 2021-10-25 Cardiopulmonary health monitoring using thermal camera and audio sensor
US18/527,127 US12023135B2 (en) 2020-10-29 2023-12-01 Cardiopulmonary health monitoring using thermal camera and audio sensor

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202063107036P 2020-10-29 2020-10-29
US17/510,357 US20220133156A1 (en) 2020-10-29 2021-10-25 Cardiopulmonary health monitoring using thermal camera and audio sensor

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/527,127 Continuation US12023135B2 (en) 2020-10-29 2023-12-01 Cardiopulmonary health monitoring using thermal camera and audio sensor

Publications (1)

Publication Number Publication Date
US20220133156A1 true US20220133156A1 (en) 2022-05-05

Family

ID=81381221

Family Applications (2)

Application Number Title Priority Date Filing Date
US17/510,357 Abandoned US20220133156A1 (en) 2020-10-29 2021-10-25 Cardiopulmonary health monitoring using thermal camera and audio sensor
US18/527,127 Active US12023135B2 (en) 2020-10-29 2023-12-01 Cardiopulmonary health monitoring using thermal camera and audio sensor

Family Applications After (1)

Application Number Title Priority Date Filing Date
US18/527,127 Active US12023135B2 (en) 2020-10-29 2023-12-01 Cardiopulmonary health monitoring using thermal camera and audio sensor

Country Status (3)

Country Link
US (2) US20220133156A1 (de)
EP (1) EP4236775A1 (de)
WO (1) WO2022093707A1 (de)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115410138A (zh) * 2022-11-02 2022-11-29 山东省人工智能研究院 基于特征编码器分类和定位串行的视频动作检测方法
CN115496170A (zh) * 2022-11-17 2022-12-20 中南民族大学 一种人体姿态识别方法、系统、电子设备及存储介质
US20230238113A1 (en) * 2022-01-25 2023-07-27 Unitedhealth Group Incorporated Machine learning techniques for parasomnia episode management
CN117558449A (zh) * 2023-11-24 2024-02-13 启康保(北京)健康科技有限公司 基于大数据的医疗健康数据管理系统
WO2024129685A1 (en) * 2022-12-13 2024-06-20 Lifeq B.V. Pre-diagnostic disease screening using wearable derived data

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110112442A1 (en) * 2007-05-02 2011-05-12 Earlysense Ltd. Monitoring, Predicting and Treating Clinical Episodes
US20120289850A1 (en) * 2011-05-09 2012-11-15 Xerox Corporation Monitoring respiration with a thermal imaging system

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070118054A1 (en) 2005-11-01 2007-05-24 Earlysense Ltd. Methods and systems for monitoring patients for clinical episodes
US8360986B2 (en) * 2006-06-30 2013-01-29 University Of Louisville Research Foundation, Inc. Non-contact and passive measurement of arterial pulse through thermal IR imaging, and analysis of thermal IR imagery
US8109875B2 (en) * 2007-01-03 2012-02-07 Gizewski Theodore M Derma diagnostic and automated data analysis system
US8638364B2 (en) * 2010-09-23 2014-01-28 Sony Computer Entertainment Inc. User interface system and method using thermal imaging
US9000371B2 (en) 2011-08-26 2015-04-07 Custom Scene Technology, Inc. Camera, computer program and method for measuring thermal radiation and thermal rates of change
US20140152424A1 (en) * 2012-12-05 2014-06-05 Unisys Corporation Method and system for security screening using biometric variables
US9486146B2 (en) * 2015-03-25 2016-11-08 Xerox Corporation Detecting tumorous breast tissue in a thermal image
WO2017178292A1 (en) * 2016-04-15 2017-10-19 Koninklijke Philips N.V. Annotating data points associated with clinical decision support application
US11241167B2 (en) 2016-08-17 2022-02-08 The Regents Of The University Of Colorado, A Body Corporate Apparatus and methods for continuous and fine-grained breathing volume monitoring
US10861605B2 (en) 2016-08-22 2020-12-08 Aic Innovations Group, Inc. Method and apparatus for determining health status
JP2020503102A (ja) 2016-08-26 2020-01-30 ライオット ソリューションズ インコーポレイテッドRiot Solutions,Inc. 非侵襲的に非接触で健康状態をモニタリングするためのシステム及び方法
WO2018112613A1 (en) * 2016-12-19 2018-06-28 Nuralogix Corporation System and method for contactless blood pressure determination
WO2018212710A1 (en) * 2017-05-19 2018-11-22 National University Of Singapore Predictive analysis methods and systems
US10918331B2 (en) 2017-09-19 2021-02-16 Aic Innovations Group, Inc. Method and apparatus for determining health status
US20210212576A1 (en) * 2018-02-23 2021-07-15 Alunos Ag Monitoring of Physiological Parameters
US20190385711A1 (en) 2018-06-19 2019-12-19 Ellipsis Health, Inc. Systems and methods for mental health assessment
TWI687206B (zh) * 2018-12-06 2020-03-11 財團法人工業技術研究院 異常肺音標記的方法及裝置
WO2021067860A1 (en) 2019-10-03 2021-04-08 DawnLight Technologies Inc. Systems and methods for contactless sleep monitoring
KR102199020B1 (ko) 2020-05-08 2021-01-06 성균관대학교산학협력단 천장형 인공지능 건강 모니터링 장치 및 이를 이용한 원격 의료 진단 방법

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110112442A1 (en) * 2007-05-02 2011-05-12 Earlysense Ltd. Monitoring, Predicting and Treating Clinical Episodes
US20120289850A1 (en) * 2011-05-09 2012-11-15 Xerox Corporation Monitoring respiration with a thermal imaging system

Non-Patent Citations (8)

* Cited by examiner, † Cited by third party
Title
Ali I. Siam et al., "Efficient video-based breathing pattern and respiration rate monitoring for remote health monitoring," J. Opt. Soc. Am. A 37, C118-C124 (2020), Published: August 31, 2020, https://doi.org/10.1364/JOSAA.399284, viewed on 1/27/23 *
Carina Pereira et al. Remote monitoring of breathing dynamics using infrared thermography. Biomed Opt Express. 2015 Oct 16;6(11):4378-94. doi: 10.1364/BOE.6.004378. PMID: 26601003; PMCID: PMC4646547. viewed on 2/3/22; *
Carina Pereira et al., "Noncontact Monitoring of Respiratory Rate in Newborn Infants Using Thermal Imaging," in IEEE Transactions on Biomedical Engineering, vol. 66, no. 4, pp. 1105-1114, April 2019, doi: 10.1109/TBME.2018.2866878, viewed on 1/27/23; *
Ching Wang et al., "Unconstrained Video Monitoring of Breathing Behavior and Application to Diagnosis of Sleep Apnea," in IEEE Transactions on Biomedical Engineering, vol. 61, no. 2, pp. 396-404, Feb. 2014, doi: 10.1109/TBME.2013.2280132. viewed on 2/3/23 *
Christian Hessler et al., "A Non-contact Method for Extracting Heart and Respiration Rates," 2020 17th Conference on Computer and Robot Vision (CRV), Ottawa, ON, Canada, 2020, pp. 1-8, doi: 10.1109/CRV50864.2020.00009. viewed on 2/3/22; *
FLIR, How Do Thermal Cameras Work?, https://www.flir.com/discover/rd-science/how-do-thermal-cameras-work/, 6/16/20, viewed on 8/1/22 *
Martin Manullang et al. Implementation of Thermal Camera for Non-Contact Physiological Measurement: A Systematic Review. Sensors (Basel). 2021 Nov 23;21(23):7777. doi: 10.3390/s21237777. PMID: 34883780; PMCID: PMC8659982. Viewed on 2/2/23 *
Prasara Jakkaew et al. Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging. Sensors (Basel). 2020 Nov 5;20(21):6307. doi: 10.3390/s20216307. PMID: 33167556; PMCID: PMC7663997. viewed on 8/1/22 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230238113A1 (en) * 2022-01-25 2023-07-27 Unitedhealth Group Incorporated Machine learning techniques for parasomnia episode management
CN115410138A (zh) * 2022-11-02 2022-11-29 山东省人工智能研究院 基于特征编码器分类和定位串行的视频动作检测方法
CN115496170A (zh) * 2022-11-17 2022-12-20 中南民族大学 一种人体姿态识别方法、系统、电子设备及存储介质
WO2024129685A1 (en) * 2022-12-13 2024-06-20 Lifeq B.V. Pre-diagnostic disease screening using wearable derived data
CN117558449A (zh) * 2023-11-24 2024-02-13 启康保(北京)健康科技有限公司 基于大数据的医疗健康数据管理系统

Also Published As

Publication number Publication date
US20240090778A1 (en) 2024-03-21
EP4236775A1 (de) 2023-09-06
WO2022093707A1 (en) 2022-05-05
US12023135B2 (en) 2024-07-02

Similar Documents

Publication Publication Date Title
US12023135B2 (en) Cardiopulmonary health monitoring using thermal camera and audio sensor
Mendonca et al. A review of obstructive sleep apnea detection approaches
US11678838B2 (en) Automated detection of breathing disturbances
US11712198B2 (en) Estimation of sleep quality parameters from whole night audio analysis
CA2872785C (en) Sound-based spirometric devices, systems, and methods
US9655559B2 (en) Automated sleep staging using wearable sensors
US20230190140A1 (en) Methods and apparatus for detection and monitoring of health parameters
JeyaJothi et al. [Retracted] A Comprehensive Review: Computational Models for Obstructive Sleep Apnea Detection in Biomedical Applications
Casal et al. Classifying sleep–wake stages through recurrent neural networks using pulse oximetry signals
US20180233234A1 (en) System and Method for Sympathetic and Parasympathetic Activity Monitoring by Heartbeat
WO2021208656A1 (zh) 睡眠风险预测方法、装置和终端设备
US20220218273A1 (en) System and Method for Noninvasive Sleep Monitoring and Reporting
Tabatabaei et al. Methods for adventitious respiratory sound analyzing applications based on smartphones: A survey
Paraschiv et al. Machine learning approaches based on wearable devices for respiratory diseases diagnosis
WO2021053645A1 (en) System and method for determining sleep stages based on non-cardiac body signals
US20220378377A1 (en) Augmented artificial intelligence system and methods for physiological data processing
US20220165393A1 (en) System for the detection and management of mental, emotional, and behavioral disorders
Pandey et al. Nocturnal sleep sounds classification with artificial neural network for sleep monitoring
US20240350021A1 (en) Cardiopulmonary health monitoring using thermal camera and audio sensor
US20210282736A1 (en) Respiration rate detection metholody for nebulizers
Adithya et al. A Deep Learning Framework for Comprehensive Sleep Analysis and Quality Tracking
Almazaydeh An interactive, real-time, high precision and portable monitoring system of obstructive sleep apnea
Bello et al. A wearable, cloud-based system to enable Alzheimer's disease analysis, diagnosis, and progression monitoring
Shewinvanakitkul Automated detection and prediction of sleep apnea events
Dhole et al. A Research Study On The Diagnostic Capabilities Of Deep Learning Regarding Sleep Apnea

Legal Events

Date Code Title Description
AS Assignment

Owner name: ROC8SCI CO., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HARTLEY, FRANK THOMAS;ROSS, CHARLES HENRY;SIGNING DATES FROM 20210804 TO 20210807;REEL/FRAME:057906/0187

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: ADVISORY ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STCV Information on status: appeal procedure

Free format text: NOTICE OF APPEAL FILED

STCV Information on status: appeal procedure

Free format text: APPEAL BRIEF (OR SUPPLEMENTAL BRIEF) ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: TC RETURN OF APPEAL

STCV Information on status: appeal procedure

Free format text: ON APPEAL -- AWAITING DECISION BY THE BOARD OF APPEALS

STCV Information on status: appeal procedure

Free format text: BOARD OF APPEALS DECISION RENDERED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION