US20220133156A1 - Cardiopulmonary health monitoring using thermal camera and audio sensor - Google Patents
Cardiopulmonary health monitoring using thermal camera and audio sensor Download PDFInfo
- Publication number
- US20220133156A1 US20220133156A1 US17/510,357 US202117510357A US2022133156A1 US 20220133156 A1 US20220133156 A1 US 20220133156A1 US 202117510357 A US202117510357 A US 202117510357A US 2022133156 A1 US2022133156 A1 US 2022133156A1
- Authority
- US
- United States
- Prior art keywords
- data
- patient
- thermal imaging
- audio
- machine learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 230000036541 health Effects 0.000 title claims abstract description 124
- 238000012544 monitoring process Methods 0.000 title abstract description 67
- 230000002612 cardiopulmonary effect Effects 0.000 title description 6
- 238000000034 method Methods 0.000 claims abstract description 90
- 230000029058 respiratory gaseous exchange Effects 0.000 claims abstract description 55
- 238000010801 machine learning Methods 0.000 claims abstract description 52
- 238000012806 monitoring device Methods 0.000 claims abstract description 34
- 208000037656 Respiratory Sounds Diseases 0.000 claims abstract description 30
- 230000036772 blood pressure Effects 0.000 claims abstract description 29
- 238000001931 thermography Methods 0.000 claims description 55
- 230000033001 locomotion Effects 0.000 claims description 40
- 238000012545 processing Methods 0.000 claims description 19
- 238000004422 calculation algorithm Methods 0.000 claims description 14
- 230000000241 respiratory effect Effects 0.000 claims description 13
- 208000008784 apnea Diseases 0.000 claims description 9
- 230000004461 rapid eye movement Effects 0.000 claims description 9
- 230000008667 sleep stage Effects 0.000 claims description 9
- 210000001715 carotid artery Anatomy 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 abstract description 16
- 230000006399 behavior Effects 0.000 abstract description 5
- 238000006213 oxygenation reaction Methods 0.000 abstract description 2
- 230000007958 sleep Effects 0.000 description 30
- 230000006870 function Effects 0.000 description 21
- 230000008569 process Effects 0.000 description 21
- 238000004458 analytical method Methods 0.000 description 15
- 238000012549 training Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 238000005259 measurement Methods 0.000 description 12
- 230000005236 sound signal Effects 0.000 description 12
- 238000003860 storage Methods 0.000 description 11
- 239000008280 blood Substances 0.000 description 9
- 210000004369 blood Anatomy 0.000 description 9
- 230000001815 facial effect Effects 0.000 description 9
- 230000036385 rapid eye movement (rem) sleep Effects 0.000 description 9
- 238000001228 spectrum Methods 0.000 description 9
- 206010011224 Cough Diseases 0.000 description 8
- 230000008859 change Effects 0.000 description 8
- 238000003491 array Methods 0.000 description 7
- 230000008901 benefit Effects 0.000 description 7
- 230000035487 diastolic blood pressure Effects 0.000 description 7
- 208000011316 hemodynamic instability Diseases 0.000 description 7
- 208000025535 REM sleep behavior disease Diseases 0.000 description 6
- 230000002159 abnormal effect Effects 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 6
- 238000003384 imaging method Methods 0.000 description 6
- 239000013598 vector Substances 0.000 description 6
- 206010035664 Pneumonia Diseases 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 5
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 description 5
- 238000001914 filtration Methods 0.000 description 5
- 239000011159 matrix material Substances 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 229910052760 oxygen Inorganic materials 0.000 description 5
- 239000001301 oxygen Substances 0.000 description 5
- 210000003800 pharynx Anatomy 0.000 description 5
- 238000000513 principal component analysis Methods 0.000 description 5
- 230000035488 systolic blood pressure Effects 0.000 description 5
- 208000006545 Chronic Obstructive Pulmonary Disease Diseases 0.000 description 4
- 230000000875 corresponding effect Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 230000004438 eyesight Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000003330 mid-infrared imaging Methods 0.000 description 4
- 230000036387 respiratory rate Effects 0.000 description 4
- 210000002345 respiratory system Anatomy 0.000 description 4
- 206010003658 Atrial Fibrillation Diseases 0.000 description 3
- 208000003417 Central Sleep Apnea Diseases 0.000 description 3
- 206010047924 Wheezing Diseases 0.000 description 3
- 208000006673 asthma Diseases 0.000 description 3
- 210000004556 brain Anatomy 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000009499 grossing Methods 0.000 description 3
- 210000004247 hand Anatomy 0.000 description 3
- 210000003205 muscle Anatomy 0.000 description 3
- 208000001797 obstructive sleep apnea Diseases 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 201000002859 sleep apnea Diseases 0.000 description 3
- 230000035882 stress Effects 0.000 description 3
- 206010011416 Croup infectious Diseases 0.000 description 2
- 241001669679 Eleotris Species 0.000 description 2
- 206010019280 Heart failures Diseases 0.000 description 2
- 241000423732 Hephaestus Species 0.000 description 2
- 206010021118 Hypotonia Diseases 0.000 description 2
- 206010033799 Paralysis Diseases 0.000 description 2
- 208000037273 Pathologic Processes Diseases 0.000 description 2
- 201000005702 Pertussis Diseases 0.000 description 2
- 206010062519 Poor quality sleep Diseases 0.000 description 2
- 206010042241 Stridor Diseases 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 210000001367 artery Anatomy 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 210000003403 autonomic nervous system Anatomy 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000009529 body temperature measurement Methods 0.000 description 2
- 206010006451 bronchitis Diseases 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000005189 cardiac health Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 201000010549 croup Diseases 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 238000009795 derivation Methods 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 230000003205 diastolic effect Effects 0.000 description 2
- 230000037213 diet Effects 0.000 description 2
- 235000005911 diet Nutrition 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000011835 investigation Methods 0.000 description 2
- 210000004072 lung Anatomy 0.000 description 2
- 239000012528 membrane Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000002496 oximetry Methods 0.000 description 2
- 230000009054 pathological process Effects 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 210000004879 pulmonary tissue Anatomy 0.000 description 2
- 238000007637 random forest analysis Methods 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 238000005096 rolling process Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 208000019116 sleep disease Diseases 0.000 description 2
- 230000003860 sleep quality Effects 0.000 description 2
- 208000020685 sleep-wake disease Diseases 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 208000037874 Asthma exacerbation Diseases 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- 206010006448 Bronchiolitis Diseases 0.000 description 1
- 206010006458 Bronchitis chronic Diseases 0.000 description 1
- 201000004813 Bronchopneumonia Diseases 0.000 description 1
- 206010007559 Cardiac failure congestive Diseases 0.000 description 1
- 206010049874 Cardio-respiratory distress Diseases 0.000 description 1
- LFQSCWFLJHTTHZ-UHFFFAOYSA-N Ethanol Chemical compound CCO LFQSCWFLJHTTHZ-UHFFFAOYSA-N 0.000 description 1
- 208000032843 Hemorrhage Diseases 0.000 description 1
- 206010020751 Hypersensitivity Diseases 0.000 description 1
- 206010021079 Hypopnoea Diseases 0.000 description 1
- 206010061218 Inflammation Diseases 0.000 description 1
- 208000009481 Laryngeal Edema Diseases 0.000 description 1
- 206010023845 Laryngeal oedema Diseases 0.000 description 1
- 206010028347 Muscle twitching Diseases 0.000 description 1
- 206010035615 Pleural rub Diseases 0.000 description 1
- 206010057190 Respiratory tract infections Diseases 0.000 description 1
- 206010039109 Rhonchi Diseases 0.000 description 1
- 206010040047 Sepsis Diseases 0.000 description 1
- 206010041349 Somnolence Diseases 0.000 description 1
- 206010067775 Upper airway obstruction Diseases 0.000 description 1
- 206010046306 Upper respiratory tract infection Diseases 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 230000001154 acute effect Effects 0.000 description 1
- 208000026935 allergic disease Diseases 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000004071 biological effect Effects 0.000 description 1
- 238000009530 blood pressure measurement Methods 0.000 description 1
- 230000036760 body temperature Effects 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 210000000621 bronchi Anatomy 0.000 description 1
- 201000009267 bronchiectasis Diseases 0.000 description 1
- 230000000747 cardiac effect Effects 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 208000007451 chronic bronchitis Diseases 0.000 description 1
- 208000013116 chronic cough Diseases 0.000 description 1
- 230000001684 chronic effect Effects 0.000 description 1
- 230000001447 compensatory effect Effects 0.000 description 1
- 208000020020 complex sleep apnea Diseases 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000034994 death Effects 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000002542 deteriorative effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 208000035475 disorder Diseases 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 229940079593 drug Drugs 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000002526 effect on cardiovascular system Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 208000001606 epiglottitis Diseases 0.000 description 1
- 230000005713 exacerbation Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000011010 flushing procedure Methods 0.000 description 1
- 230000004313 glare Effects 0.000 description 1
- 210000004704 glottis Anatomy 0.000 description 1
- 238000009532 heart rate measurement Methods 0.000 description 1
- 230000009610 hypersensitivity Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000004054 inflammatory process Effects 0.000 description 1
- 208000030603 inherited susceptibility to asthma Diseases 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 230000003434 inspiratory effect Effects 0.000 description 1
- 238000001361 intraarterial administration Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 210000000867 larynx Anatomy 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 208000030500 lower respiratory tract disease Diseases 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000007620 mathematical function Methods 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000004476 mid-IR spectroscopy Methods 0.000 description 1
- 201000006646 mixed sleep apnea Diseases 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 210000002161 motor neuron Anatomy 0.000 description 1
- 210000003097 mucus Anatomy 0.000 description 1
- 230000010355 oscillation Effects 0.000 description 1
- 230000000803 paradoxical effect Effects 0.000 description 1
- 230000001734 parasympathetic effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000035479 physiological effects, processes and functions Effects 0.000 description 1
- 229920001451 polypropylene glycol Polymers 0.000 description 1
- 230000009430 psychological distress Effects 0.000 description 1
- 208000005069 pulmonary fibrosis Diseases 0.000 description 1
- 238000002106 pulse oximetry Methods 0.000 description 1
- 238000004080 punching Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 208000036273 reactive airway disease Diseases 0.000 description 1
- 230000011514 reflex Effects 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 230000000246 remedial effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 208000023504 respiratory system disease Diseases 0.000 description 1
- 230000000284 resting effect Effects 0.000 description 1
- 230000001020 rhythmical effect Effects 0.000 description 1
- 230000028327 secretion Effects 0.000 description 1
- 230000004799 sedative–hypnotic effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000013707 sensory perception of sound Effects 0.000 description 1
- 210000002027 skeletal muscle Anatomy 0.000 description 1
- 230000037322 slow-wave sleep Effects 0.000 description 1
- 238000012732 spatial analysis Methods 0.000 description 1
- 238000010183 spectrum analysis Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000002889 sympathetic effect Effects 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000002076 thermal analysis method Methods 0.000 description 1
- 230000008719 thickening Effects 0.000 description 1
- 210000001519 tissue Anatomy 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000002618 waking effect Effects 0.000 description 1
- 230000036642 wellbeing Effects 0.000 description 1
- 210000000707 wrist Anatomy 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/01—Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue
- A61B5/015—By temperature mapping of body part
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/0205—Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1102—Ballistocardiography
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/113—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb occurring during breathing
- A61B5/1135—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb occurring during breathing by monitoring thoracic expansion
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/74—Details of notification to user or communication with user or patient ; user input means
- A61B5/742—Details of notification to user or communication with user or patient ; user input means using visual displays
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B7/00—Instruments for auscultation
- A61B7/003—Detecting lung or respiration noise
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/74—Details of notification to user or communication with user or patient ; user input means
- A61B5/746—Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms
Definitions
- This application relates generally to non-contact health monitoring, and more particularly to non-contact cardiopulmonary monitoring in home and healthcare settings.
- Apparatus for monitoring subjects during sleep may be used to diagnose sleep disorder.
- Such apparatus are generally bulky, complex and expensive.
- Sleep tracker devices can be more cost effective, but sleep tracker devices typically require contact with the user and are generally less precise than polysomnography tests.
- What is needed is systems and methods for acquiring and analyzing cardiopulmonary characteristics of a subject that do not require physical contact between the subject and a monitoring device. What is needed is methods for health monitoring of subjects that do not cause discomfort and do not disturb subjects during sleep. Disclosed embodiments provide a cost-effective device for monitoring health, e.g., device suitable for use in the home. The health monitoring device disclosed herein can achieve improved precision in determining cardiopulmonary characteristics of a subject.
- Disclosed embodiments employ a health monitoring device to monitor vital signs of a subject, such as a sleeping patient.
- the health monitoring device may include a thermal camera such as an uncooled microbolometer array to monitor breathing, pulse, temperature, and other vital signs of the patient.
- An audio sensor e.g., microphone, may be used for monitoring patient respiratory sounds and other patient sounds. Further information such as blood pressure and heart health can be calculated from these signals and their waveforms.
- the health monitoring device utilizes the acquired signals and higher order data in analyzing patient conditions and behaviors. Higher order data may include visual data based upon signals output by the thermal camera and audio data based upon signals output by the audio sensor.
- the thermal camera and signal processing of camera outputs track the pulse rate, breathing, and temperature of the subject.
- a thermal camera may sense the sinusoidal motion of the heart rate by imaging the carotid artery in the neck, and the temple.
- the thermal camera also may sense the sinusoidal motion of the heart rate by imaging the subject's arms and hands.
- monitoring breathing the thermal camera may image one or more of the subject's chest, nostrils, and mouth.
- the health monitoring device incorporates an uncooled microbolometer array in communication with a mobile computing device.
- the health monitoring device incorporates audio data in monitoring and characterizing vital signs of a subject.
- the audio data may include spectrograms in the audio spectrum, such as spectrograms derived from audio clips recorded by the audio sensor.
- the audio sensor may generate audio signals via microphone, handset, or other transducer that converts sound into an analog electrical signal.
- the microphone or an external device converts the analog signal into digital audio signals, also herein called audio data.
- audio monitoring may be used for monitoring and characterizing breath rate and abnormal respiratory sounds, and for recognizing the subject's speech.
- the health monitoring device may activate audio monitoring in the event video monitoring fails to detect presence of the subject at a primary location.
- the health monitoring device includes a processor configured to output a health determination relating to the one or more health parameters of the patient by inputting one or both visual and audio data into one or more machine learning models.
- the health determination includes a value of the one or more health parameters, a binary classification of the one or more health parameters, a multiclass classification of the one or more health parameters, an event relating to the one or more health parameters, or a health anomaly relating to the one or more health parameters.
- the one or more machine learning models include a supervised learning model including a factorization machine.
- the machine learning models include an unsupervised learning model trained to identify key features of interest.
- a monitoring device comprises a set of sensors configured to receive signals pertaining to one or more health parameters of a patient through non-physical contact with the patient, wherein the set of sensors comprise a thermal camera and an audio sensor, wherein the monitoring device is configured to monitor the one or more health parameters of the patient; a signal processing unit configured to generate visual data based upon signals output by the thermal camera and to generate audio data based upon signals output by the audio sensor; and a processor configured to output a health determination relating to the one or more health parameters of the patient by inputting one or both of the visual data and the audio data into one or more machine learning models.
- a method comprises receiving, by a set of sensors, signals pertaining to one or more health parameters of a patient through non-physical contact with the patient, wherein the sensing unit comprises a thermal camera and an audio sensor; generating, by a processor coupled to the set of sensors, visual data based upon signals output by the thermal camera and audio data based upon signals output by the audio sensor; and outputting, by the processor, a health determination relating to the one or more health parameters of the patient by inputting one or both of the visual data and the audio data into one or more machine learning models.
- Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying figures, which are schematic and are not intended to be drawn to scale. Unless indicated as representing the background art, the figures represent aspects of the disclosure.
- FIG. 1 illustrates an architecture of a non-contact health monitoring device, according to an embodiment.
- FIG. 2 shows a schematic diagram of a factoring machine, according to an embodiment.
- FIG. 3 is a flow chart diagram of recursive processes of a factorization machine, according to an embodiment.
- FIG. 4 shows a schematic block diagram of a method for training a chest motion model, according to an embodiment.
- FIG. 5 is a schematic diagram of an SQL architecture of features identified via recognition/identification of facial features, according to an embodiment.
- FIG. 6 is a representative graph of a blood pressure waveform including systolic and diastolic pressures, according to an embodiment.
- FIG. 7 shows a training process for determining blood pressure, according to an embodiment.
- FIG. 8 illustrates field names, field data types, and PK primary key values of an example SQL architecture, according to an embodiment.
- FIG. 9 illustrates an audio pre-processing method for identifying respiratory ailments, according to an embodiment.
- FIG. 10 is a flow chart diagram of training methods for analysis of audio events, according to an embodiment.
- FIG. 11 is a graph of maximum facial temperature vs. camera distance, according to an embodiment.
- FIG. 12 is a schematic block diagram of a general health monitoring architecture, according to an embodiment.
- Disclosed embodiments employ a health monitoring device to monitor vital signs of a subject, such as a sleeping patient.
- Health monitoring signals may be acquired and analyzed via multiple subsystems to ensure greater reliability.
- the health monitoring device may include a thermal camera such as an uncooled microbolometer array to monitor breathing, pulse, temperature, and other vital signs of the patient. Further information such as blood pressure and heart health can be inferred from these signals and their waveforms.
- An audio sensor e.g., microphone, may be used for monitoring breath rate.
- An audio recognition system can be trained to characterize abnormal respiratory sounds.
- the health monitoring device utilizes acquired signals and higher order data as source data for analyzing patient conditions and behaviors. The higher order data may include visual data based upon signals output by the thermal camera, and audio data based upon signals output by the audio sensor.
- the thermal camera and signal processing of camera outputs track the heart rate, breathing, and temperature of a sleeping individual.
- a thermal camera may sense the sinusoidal motion of the heart rate by imaging one or both the carotid artery in the neck and the temple.
- the thermal camera also may sense the sinusoidal motion of the heart rate by imaging one or both the subject's arms and hands.
- thermal imaging targets may include one or more of the subject's chest, nostrils, and mouth. Together, these signals can be employed in deriving data extrapolating further information about the health of a monitored subject.
- the health monitoring device incorporates an uncooled microbolometer array in communication with a mobile computing device.
- the health monitoring device incorporates audio data in monitoring and characterizing vital signs of a subject.
- Audio data may include spectrograms in the audio spectrum, e.g., spectrograms derived from audio clips recorded by the audio sensor.
- the audio sensor may generate audio signals via microphone, handset, or other transducer that converts sound into an electrical signal.
- the microphone generates AC signals representing air pressure variations of a sound wave, e.g., sounds resulting from speech, breathing, respiratory sounds, or other sounds.
- the microphone or an external device converts the AC signal into digital audio signals, also herein called audio data.
- audio monitoring may be used for monitoring breath rate, for identifying abnormal respiratory sounds, and for recognizing subject's speech.
- the health monitoring device may activate an audio mode in the event video monitoring fails to detect presence of the subject at a primary location.
- the health monitoring device includes a processor configured to output a health determination relating to the one or more health parameters of the patient by inputting one or both of visual data and audio data into one or more machine learning models.
- the health determination includes a value of the one or more health parameters, a binary classification of the one or more health parameters, a multiclass classification of the one or more health parameters, an event relating to the one or more health parameters, or a health anomaly relating to the one or more health parameters.
- An example of a value of one or more health parameters includes systolic and diastolic blood pressure values.
- An example of a binary classification of one or more health parameters is a binary flag.
- An example of a multiclass classification of one or more health parameters is a multiclass classification of respiratory sounds.
- An example of an event relating to one or more health parameters is an apnea event.
- An example of a health anomaly relating to one or more health parameters is a sleep disorder.
- one or more machine learning models include a supervised learning model trained to recognize features of interest.
- a supervised learning model such as regression model includes a factorization machine.
- machine learning models include an unsupervised learning model.
- machine learning models may refer to methods such as logistic regression, decision trees, Random forest ensembles, neural networks, linear models, matrix reduction, and/or Bayesian models.
- multiple machine learning models are employed in monitoring health parameters.
- multiple machine learning models are combined in multimodal artificial intelligence (AI), in which two or more data types (e.g., thermal imaging spatial data, thermal imaging temperature data, audio data) are combined via multiple AI algorithms.
- AI multimodal artificial intelligence
- multiple machine learning models may operate simultaneously or may operate sequentially in monitoring health parameters.
- a thermal imaging sensor monitors presence of a subject at a primary location. In the event thermal imaging generates a “no patient present” flag, this activates one or more other monitoring modality such as audio monitoring.
- FIG. 1 illustrates an architecture of a health monitoring device 100 .
- the health monitoring device includes a set of sensors 140 configured to receive signals pertaining to one or more health parameters without physical contact of the sensors with the subject 120 .
- the sensors 140 are configured to receive signals through non-physical contact 125 with the subject 120 .
- the sensors 140 include a thermal camera 144 that measures the thermal heat of the subject 120 and other objects in the field of view.
- the thermal camera includes an infrared (IR) sensor, such as a microbolometer array.
- the set of sensors 140 also include an audio sensor 148 .
- the system 100 monitors health of a sleeping subject 120 .
- the system 100 and its components can easily be adapted to other health monitoring applications that incorporate thermal imaging and audio sensing for monitoring a subject.
- Other applications may include, e.g., monitoring pilots; car, bus, truck and train drivers; cyclists; automated preliminary health assessment, e.g., for triage in hospital or in doctor's office; infant care; and other individuals, e.g., persons under care of a healthcare worker or care giver.
- one or both thermal imaging systems and audio systems track eye motion, heart rate, HRV, breathing rate, and breathing rate variability of a pilot or a professional driver to ensure they are alert and healthy throughout their shifts.
- preliminary health assessment/triage metrology for use in hospital and doctor's office settings could be non-contact, self-administered by subject, and could take a few minutes to administer.
- Mid-infrared imaging offers the advantage in various applications that it is not affected by environmental illumination. For example, MIR thermal imaging to observe a professional driver would not be affected by oncoming headlights at night or by occasional glare during the day. Using MIR thermal imaging in a kiosk for preliminary health assessment would not be affected by changing ambient light conditions in a doctor's office.
- thermal camera 144 incorporates uncooled IR image sensors called microbolometers, built around arrays of tiny thermal detectors.
- thermal camera is also referred to as infrared (IR) camera and as microbolometer.
- IR infrared
- a microbolometer is a specific type of bolometer used as a detector in a thermal camera. Infrared radiation in the mid-IR range, e.g., with wavelengths between 7.5-14 ⁇ m, strikes the detector material, heats it, and changes its electrical resistance. Arrays of many thermally isolated microbolometers can record images in the thermal IR. This resistance change is measured and processed into temperatures that can be used to create an image. Unlike other types of infrared detection equipment, microbolometers do not require cooling, thereby reducing their cost, size, and complexity. Microbolometers are commercially available in handheld devices, including smartphone cameras.
- thermal imaging allows measurement to be performed in a completely dark room so the subject 120 is not disturbed by an external source light required by an optical camera.
- IR cameras have the advantages that they measure photons radiated from a regarded object, they do not need any external light that may distract or disturb a subject, and they are insensitive to viewing angle.
- Health monitoring device 100 may utilize uncooled microbolometer arrays for non-contact measurement of a subject's stance, rate of breathing, temperature, pulse rate and pulse profile.
- IR cameras measure radiated photons from a regarded object, they do not need any external light source that can be distracting to the subject 120 . Additionally, IR cameras are insensitive to viewing angle.
- thermal imaging subjects could be pilots; car, bus, truck and train drivers; cyclists; hospital and doctor triage users; and individuals. For drivers, stance monitoring could establish drowsiness, fatigue, inattention or stress.
- signal processing 154 of output of thermal camera 144 may employ functions from OpenCV (Open Source Computer Vision Library) a library of programming functions employed in at real-time computer vision.
- OpenCV Open Source Computer Vision Library
- functionalities of facial feature recognition and object recognition are expanded to work with a thermal camera.
- Features of the face ducts, eyes, temples, carotid artery, etc.
- an existing feature map is compared with one or more acquired thermal images. This comparison may include a series of convolutions for feature extraction, with bounding boxes for each relevant feature.
- signal processing 154 extracts patterns of thermal intensity of sensed features as digital data streams for conversion to frequency domain.
- This conversion may employ the Fast Fourier Transform used by NumPy.
- NumPy is a library for the Python programming language including a large collection of high-level mathematical functions to operate on multi-dimensional arrays and matrices.
- the NumPy library includes the Fourier transfer function Discrete Fourier Transform (numpy.fft). In discrete Fourier transform (DFT), both the original signal and its Fourier transform are replaced with discretized counterparts. The DFT separates its input into components that contribute at discrete frequencies.
- NumPy Fast Fourier Transform is represented by the following formula:
- the signal in the Fourier domain typically has a dominant feature that will correspond to the breathing rate or pulse rate.
- Signal processing 154 may filter the acquired signal via band pass filter around this frequency to improve the signal.
- calculated rates are further refined using a factorization machine in machine learning models 160 .
- the factorization machine fits the calculated rates to appropriate variables, given by the following formula:
- the signal also may be converted to the frequency domain using a wavelet transform. This allows for the signals of interest to change in the frequency domain as a function of time, unlike the more common Fourier transform. Typically, Fast Fourier Transform using a rolling window average is sufficient, however. This is because health monitoring signals most often change slowly with occasional periods of extreme change correlated with a health event.
- model development seeks to find a fitted model and to gain insights into how an arbitrary health feature relates to pulse, breathing, and other physical characteristics.
- factorization machine is used as a measurement device that can return an accurate value for training purposes. Data exploration of the calculated factors may identify a relationship between the raw data returned by the camera and the desired value.
- processing unit 150 analyzes each frame of dynamic thermal images captured by thermal camera 144 .
- machine learning models 160 incorporate a factorization machine, as shown in the scheme 200 of FIG. 2 .
- Factorization machines are supervised learning models that can perform both regression and classification. They are non-linear models that model interactions of variables 210 (x 1 , x 2 , x 3 . . . ) by generating feature vector representations 220 (including vectors v 1 , v 2 , v 3 . . . ) and by mapping 230 , 240 feature interactions to a model output 250 in a low dimensional space.
- a factorization machine has several advantages over traditional regression techniques commonly employed in machine learning. First, by decomposing a variable into multiple factors, higher order effects, e.g., biological effects, may be observed. Additionally, while a single variable may be over-fitted and thus fail to respond appropriately to a change in the system, a factorization machine is less susceptible to this.
- FIG. 3 shows a flow chart of recursive processes 300 of a factorization machine.
- the factorization machine algorithm employs reinforcement learning that will continually adjust to be in agreement with whatever secondary data stream is being used.
- a variable “figure of merit” is defined as
- Factorization machines can provide supervised machine learning models with excellent interpretability of modeled health parameters. For example, a matrix reduction and averaging procedure can derive matrix-reduced data to reconstruct waveforms with intuitive relevance to health parameters.
- machine learning models apply a factorization machine to determine the subject's tidal volume.
- tidal volume refers to the amount of air that moves in or out of the subject's lungs with each respiratory cycle. Coupled with pulse and breath rate, knowledge of the tidal volume would be an important indication of the subject's health.
- a sensor may detect a mass of hot, moist air being exhaled, but this technique is sensitive to humidity, ambient temperature, and viewing angle. Instead, by using the motion of the chest, mouth and nostrils, and frequencies as factors, machine learning models may be trained to predict the tidal volume of the breathing. By examining spatial displacement as well as temperature gradients, machine learning models may incorporate contributions of these factors to the overall factorization machine.
- the canny edge detection algorithm with an applied Gaussian filter may be used to determine the motion of the chest, distinguishing the relatively hot body against the cool background. Temperature differences make it straightforward for the algorithm to distinguish between the different temperatures present on the body, which includes covered portions, hands, and hair, relative to the much cooler sheets and blankets.
- the magnitude and profile of chest motion can be used in training a machine learning model 160 , provided with actual measured breath volume, to predict tidal volume.
- chest motion can be predictive of tidal volume, while the frequency is used to determine breathing rate. This motion can be detected from a variety of angles and under covers.
- FIG. 4 is a schematic block diagram of a method 400 for training a chest motion model.
- This method takes as inputs measurements of breath volume, and thermal camera output data.
- Thermal camera output data may comprise a float array.
- Thermal camera data and breath volume data are passed through a band pass Gaussian filter to derive chest motion.
- the band pass is centered at a calculated value for average chest motion over a preceding set of thermal images, e.g., the last 1000 frames.
- the chest motion data is analyzed to derive breathing rate.
- the canny edge algorithm may be applied to thermal camera data to derive magnitude of displacement of the chest d/dt.
- Breathing rate, d/dt, and numeric integration of chest motion are inputs to a model to determine plume volume.
- a computer vision (CV) algorithm may extract the pixels related to each important feature from a digitized thermal image in performing feature recognition of a subject's face and neck.
- this algorithm seeks to identify the subject's neck and temples. For each point, all pixels within a certain square radius are used. A weighting function is applied so that the maximum thermal point has the greatest weighting, since presumably this corresponds to the point closest to the artery. Each of these points may be represented by a 2D array that varies in time.
- FIG. 5 is a schematic diagram of an SQL architecture 500 of features identified via facial feature recognition. Facial feature/element recognition techniques based on thermal imaging methods of the disclosure can derive spatial and temperature visual data that may be used in modeling health parameters.
- health monitoring device 110 analyzes these data points over time to extract information about the heart rate, heart rate waveform, and variability. For a moment in time, each 2D array corresponding to a feature of interest is added to a single array. For two features (neck, temple) with each feature represented by an M ⁇ N array, this step results in a 2 ⁇ M ⁇ N array for a given moment in time.
- CV analysis may perform principal component analysis to extract the most important information from this array.
- Principal component analysis can be calculated several ways, such as truncated singular value decomposition. This PCA technique does not require calculating a matrix inverse. Principal components can be considered a linear projection along the axis such that higher dimensional data is efficiently represented as the function of a single variable.
- the first principal component corresponds to the axis along which covariance is maximized, and variance is minimized.
- this first principal component represents a stable pulse rate measurement.
- the second principal component corresponds to maximized variance.
- L2-norms (vector distance) of principal components in the data set indicate stability of the measurements of pulse rate variability. If these values change rapidly, further investigation is required.
- the pulse waveform becomes readily apparent when matrix reduction is performed on successive arrays.
- This analysis provides an approximate waveform that can be used to calculate its mean, standard deviation, Gaussian spread, and other parameters.
- These derived data may be employed to construct a blood pressure model.
- a supervised learning model may take as input a vector of values corresponding to the waveform.
- a factorization machine then may train the model to fit an experimentally obtained invasive blood pressure for the waveform.
- Two factorization machines may be used to fit the systolic and diastolic pressures, e.g., as shown in the representative graph 600 of FIG. 6 .
- the factorization machines may be applied to qualify and quantify atrial fibrillation and other cardiac ailments.
- FIG. 7 shows a training process 700 for determining blood pressure.
- Blood pressure has the systolic and diastolic numbers associated with the waveform of the heart rate. Using the ratio of the rise and fall times, the method 700 can predict these numbers.
- the method 700 uses blood pressure profile and a secondary source in a supervised learning model to predict blood pressure.
- the method derives the blood pressure profile by inputting a floating-point array of thermal imaging data and a feature coordinate list to determine pulse data in the time domain.
- the method applies numeric integration to the time domain pulse data to derive a blood pressure profile, including wave power and first derivative of the pulse data.
- the blood pressure profile is input to a regression/factorization machine, along with data obtained from a secondary source.
- the secondary source includes ground truth data obtained from an intra-arterial measurement method.
- Disclosed embodiments apply CV image processing to respiratory systems, which are lower frequency. Respiratory systems analysis has the advantage that it may incorporate audio data as additional acquired data. For two features, chest and throat movement, with each feature represented by an M ⁇ N array, this process derives a 2 ⁇ M ⁇ N array for a given moment in time. In an embodiment, health monitoring device 110 analyzes these data points over time to extract information about breathing rate, respiration volume (waveform), and breathing rate variability.
- PCA may be performed to address noise and many similar data points.
- the principal component the most important feature, the breathing waveform, becomes readily apparent when feature movements are represented in successive arrays.
- the first principal component corresponds to the axis along which covariance is maximized, and variance is minimized. This represents a stable breathing rate measurement.
- the second principal component corresponds to maximized variance representing breathing rate variability.
- the L2-norms of the principal components give a sense of stability of the system, e.g., a metric of breathing rate variability such as apnea. If they change rapidly, further investigation is required.
- Blood oxygen saturation is another useful medical characteristic that can be determined from thermal imaging.
- Disclosed embodiments apply similar CV image processing techniques to pulse oximetry of blood oxygen saturation.
- Thermal imaging outputs an approximate waveform that may be analyzed via CV techniques to calculate its mean, standard deviation, Gaussian spread, and other characteristics.
- these visual data are employed in constructing a blood oxygen saturation model.
- a supervised machine learning model may take as input a vector of values corresponding to the waveforms.
- a regression/factorization machine trains the model to fit experimentally obtained via oximeter measurements for the waveform.
- Oximetry is a traditional technique in which blood oxygen saturation is measured by passing light through a fingertip and comparing the absorption of the light through oxygenated versus unoxygenated blood.
- oximetry is used as ground truth in training the blood oxygen saturation model.
- Respiratory system analysis may incorporate audio data, such as spectrograms in the audio spectrum, in addition to the vision data. Although these audio data have different units, supervised learning models may be trained to convert between them. Adding audio spectrum values to the thermal imaging values in a factorization machine may enable the factorization machine to automatically learn relationships between audio variables and thermal/spatial variables. In an example, this procedure was applied to three features with each feature represented by an M ⁇ N array and a 3 ⁇ M ⁇ N array for a given moment in time.
- health monitoring device 110 communicates with a mobile computing device, e.g., smart phone 130 .
- the mobile computing device may act as power source and device to compute signals.
- raw data is processed on the phone, and derived signals and flags 158 are saved remotely when the run is stopped.
- a mobile computing device 130 also may perform calculation for real-time monitoring of vital signs.
- Device 130 includes sufficient RAM and computing power to undertake all real-time simultaneous modeling and statistical analysis.
- a code base for smartphone 130 was developed in python, C, and C++, and converted using public cloud computing tools provided by Microsoft Azure. The viewing angle may also be easily chosen with simple wall mounts in a wide variety of settings.
- Using smart phone 130 as processor provides a cost-effective design for health monitoring applications, such as monitoring subject 120 during sleep.
- Other health monitoring applications may replace smart phone 130 with another computing device suitable for the application.
- the health monitoring processor may be included in a vehicular computer system.
- the processor may be included in a device such as a kiosk.
- a smart phone or other mobile device 130 can extend sensing capabilities for health monitoring of a subject.
- a back facing camera of a smart phone effects video imaging in visible light spectrum of neck and temple of a subject to track and create blood pulse profile.
- These optical sensor readings can provide measurements of pulse rate and pulse rate variability as inputs for predictive modeling of systolic and diastolic blood pressures.
- Processing unit 150 can be implemented using a single-processor system including one processor, or a multi-processor system including any number of suitable processors that may be employed to provide for parallel and/or sequential execution of one or more portions of the techniques described herein. Processing unit 150 performs these operations as a result of central processing unit executing software instructions contained within a computer-readable medium, such as within memory. As used herein, a module may represent functionality (or at least a part of the functionality) performed by a processor.
- Device 100 includes a power supply 174 for powering components of health monitoring device 110 , including the mobile device 130 and IR camera 144 .
- power supply 174 is a battery that can be recharged by power source 172 , e.g., via continuing mains feed charging.
- Power supply 172 may be configured to provide non-interruptible operation during power outs.
- Communications module 178 may support various wired and wireless various communication protocols such as Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), and IEEE communication protocols, and standard wireless protocols such as Bluetooth®, BLE, Wi-Fi, NFC, ZigBee®, and the like.
- BLUETOOTH is a registered trademark of Bluetooth Sig, Inc., Kirkland Wash.
- communication protocols may include wireless communications according to BLUETOOTH specification sets or another standard or proprietary wireless communication protocol.
- communication protocols may also include communications over a cellular network, including, e.g., a GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), or EDGE (Enhanced Data for Global Evolution) network.
- GSM Global System for Mobile Communications
- CDMA Code Division Multiple Access
- EDGE Enhanced Data for Global Evolution
- Cellular communications may accommodate operational and system uploads, real-time alarm monitoring outputs to healthcare professionals and care givers, session reports archiving, and reporting to healthcare professionals and care givers.
- wireless fidelity (Wi-Fi) communications may accommodate functions such as operational and system uploads, real-time alarm monitoring outputs to healthcare professionals and care givers, session reports archiving, and GP/care giver reporting to healthcare professionals and care givers.
- communication protocols may include wired duplex communications with IR camera 144 .
- Audio sensor 148 may generate audio signals via microphone, handset, or other transducer that converts sound into an electrical signal.
- a microphone converts the air pressure variations of a sound wave, e.g., resulting from speech, breathing, or other respiratory sounds from subject 120 .
- Microphones convert sound waves into AC electrical audio signals and are therefore analog devices.
- analog signals from audio sensor 148 signals are converted to digital audio signals processed by filtering/signal processing module 154 .
- a microphone or other transducer may output digital audio signals via built-in analog-to-digital converters, or may be coupled to an external analog-to-digital converter device that outputs digital audio signals.
- Audio monitoring may be used for monitoring breath rate, and may be trained to identify abnormal respiratory sounds. Audio monitoring can measure various physical attributes: frequency, e.g., the number of the sound waves or vibrations per second; intensity or amplitude, e.g., the height of sound waves from their mean; and quality or timbre. Quality or timbre differentiates two sounds with the same frequency and intensity. The timbre of a sound depends on its wave form, including the number of overtones or harmonics, their frequencies, and their relative intensities (amplitudes).
- audio sensor 148 is used to obtain audio intensities and other audio parameters of breathing, which may provide an additional signal to confirm the accuracy of vision analysis of chest motion and breathing.
- filtering/signal processing module 154 passes the audio signal to the frequency domain, in which a large magnitude signal is taken as the breathing rate. Module 154 may filter out background noise, as most sounds are broad-bandwidth, low intensity in nature. If several prominent peaks occur in the frequency spectrum, the peak closest to the frequency of breathing calculated by the chest and throat motion and exhalations is chosen as the target feature in performing band pass filtering.
- a digital band pass filter may be employed. This procedure identifies the two frequencies with the greatest magnitudes, corresponding to breathing of the two individuals.
- the original signal is duplicated then passed through band pass filters centered at each of these two frequencies in order to find the breath rate for each person.
- the band pass filters may filter out extraneous noises that might briefly conceal breathing sounds, such as thunder or vehicle noise.
- the central frequency of the band pass filter is calculated via a rolling average, in which the frequency of a preceding time interval (e.g., previous 5 minutes) is used. This ensures that the signal is not lost if, over the course of a night, the breathing slows, as might be expected.
- filtering/signal processing module 154 does not perform band pass filtering since training models use higher-order overtones are used as inputs.
- Flags 158 may be classified into several categories. If the patient leaves the frame, the temperature sensed by thermal camera 144 will drop to ambient bed temperature. This flag will note the patient has left the bed. If the maximum temperature in frame remains surface temperature, but the position of sleep impedes one or more signals, the flag will note which signals are impeded and attempt to make a guess as to why, with the position of the sleeping patient and pillows and blankets covering them being two possibilities. These regions are distinguished by their relative temperatures. Further, in the event a signal is lost for no obvious reason, processing unit 150 can check motion data during the preceding time period (e.g., five minutes). Motion may be measured, e.g., by the displacement vector of the canny edge. If this metric falls below a percentage threshold, it may be assumed some system error is responsible. In this case, the temporarily saved data may be stored in a data log for the night.
- the preceding time period e.g., five minutes. Motion may be measured, e.g., by the displacement vector of the canny edge.
- audio signal 148 is analyzed to recognize the subject's speech.
- speech recognition employs Carnegie Mellon's Hephaestus Speech-related software systems to throw flags 158 when key phrases, such as “help” or “I've fallen,” are spoken.
- the spectrograms of these audio clips are passed through a triangular Mel filter, which weights frequencies dominant in human vocalizations.
- the filtered spectrograms are subsequently trained in a supervised learning model.
- Key phrase flags 158 may trigger alarms 176 , such as audible alarms or visual alarms displayed to healthcare professionals and care givers.
- system 100 may acknowledge that ‘cries’ were received over phone speaker 130 .
- keyword recognition can be used to begin a monitoring run and/or to end a monitoring run.
- the visual system 144 , 154 includes a patient presence model that identifies when no subject 120 is in the field of view, e.g., generating a “no patient present” flag.
- an additional audio monitoring measure 148 , 154 is activated, e.g., to identify falling sounds and sounds likely to occur when a patient rises, e.g., doors opening and closing, and toilet flushing.
- a fall has a broad spectrum, making it an impracticable measure to track continuously.
- monitoring for fall stops once the patient is in the field of the view of the camera. If a fall occurs, a flag 158 is thrown and care givers are notified 176 .
- Other modalities may be activated by “no patient present” flag, such as activating a front facing camera and directing local WiFi to turn on lights in a room. These additional modalities may be deactivated if the video system later detects patient presence.
- a flag 158 is thrown to indicate a signal has been lost. If the maximum temperature sensed by thermal camera 144 is below 25° C., it can reasonably be inferred the patient has moved out of the field of view, and the flag thrown notes this. All flags thrown over the course of a night may be saved to a text file as well as to a data log for the time of each flag. In an embodiment, frame data may be saved for a limited time (e.g., 5 minutes) before being replaced, while attributes derived from the frame data may be stored in a data log for the night.
- a limited time e.g., 5 minutes
- standard deviation, first and second derivative, and splining fit are all stored by default. These data have various applications in determining patients' health. Using the derivative and standard deviation, health anomalies such as apnea and atrial fibrillation can be discovered and flagged 158 , which may result in an alert 176 to a healthcare worker or care giver. In a use case, this system serves as a smart monitor to aid nurses and night-staff in rest home and hospice settings. In an embodiment, a flag 158 is a binary classifier for which minimizing false negatives are prioritized over minimizing false positives. Model training may select weighting errors to provide a priori Bayesian distributions for probabilistic alerting of healthcare workers. Model training may employ joint probabilities to allow incorporating other signals relevant for the patient's health.
- the system 100 may store data in local databases of health monitoring device 110 and mobile computing device 130 .
- the system also may store data, e.g., archived data, in cloud databases 180 .
- Cloud 180 may be a third-party cloud.
- Databases are organized collections of data, stored in non-transitory machine-readable storage.
- the databases may execute or may be managed by database management systems (DBMS), which may be computer software applications that interact with users, other applications, and the database itself, to capture (e.g., store data, update data) and analyze data (e.g., query data, execute data analysis algorithms).
- DBMS database management systems
- the DBMS may execute or facilitate the definition, creation, querying, updating, and/or administration of databases.
- the databases may conform to a well-known structural representational model, such as relational databases, object-oriented databases, or network databases.
- Example database management systems include MySQL, PostgreSQL, SQLite, Microsoft SQL Server, Microsoft Access, Oracle, SAP, dBASE, FoxPro, IBM DB2, LibreOffice Base, and FileMaker Pro.
- Example database management systems also include NoSQL databases, i.e., non-relational or distributed databases that encompass various categories: key-value stores, document databases, wide-column databases, and graph databases.
- FIG. 8 illustrates field names, field data types, and PK primary key values of an example SQL architecture.
- sampling rate per Nyquist's Theorem.
- the discrete nature of sampling may be fitted to assumed continuous functions as a splining curve whose higher-order terms could be useful for determining additional data.
- One goal of disclosed systems and methods is to extrapolate general diagnoses from data collected.
- a corpus of data collected over many patients concerning their breathing rate, pulse, lung capacity, and sleep cycle may offer deeper insights into the well-being of monitored patients.
- the system applies data mining methods to create correlation matrices.
- Correlation matrices may be used to describe the health of a patient and to identify higher risk conditions of the patient based on covariance.
- higher-order terms may correspond to physical phenomena such as the correlation of blood pressure with pulse waveform.
- Machine learning modeling 160 may calculates the posterior distribution of each pair to determine which signal is most likely flawed. This information can then be used to adjust the neural network and filters that determine that signal. Bayesian machine learning then can compare the signals again to find better agreement.
- Disclosed embodiments may apply audio monitoring to recognize coughs.
- Coughs are typically characterized by three phases: an initial loud, peak in intensity, followed by a quick attenuation and finally a voiced phase.
- a recognition algorithm may consider the number of cough sounds, the portion of breaths that include or are disrupted by a cough, and cough epochs, defined herein as the number of coughs with no more than two second interval between them. Chest movement is a further classifying parameter. Also pertinent are the audio breathing phase analysis and chest dynamics for characterizing asthma, pneumonia, and other abnormal respiratory conditions.
- FIG. 9 illustrates an audio pre-processing method 900 for identifying respiratory ailments.
- adventitious lung sounds may be classified in frequency bands, e.g., low frequency (100 to 300 Hz), middle frequency (300 to 600 Hz), and high (600 to 1,200 Hz) frequency bands.
- Frequency is measured objectively, while pitch is the subjective perception of sound's frequency.
- Stridor are high pitched sounds generated by turbulent airflow in the larynx or bronchial tree, and are related to an upper airway obstruction as with epiglottitis, croup, and laryngeal edema.
- Inspiratory gasp sounds or whoops e.g., after a bout of coughing, are caused by fast moving air through the respiratory tract and a pathognomonic symptom of whooping cough (pertussis).
- Squawks, or short wheezes are generated by oscillation at the peripheral airways and associated hypersensitivity to pneumonia.
- Fine crackle sounds are caused by explosive openings of the small airway and usually associated with pneumonia, congestive heart failure, and lung fibrosis.
- Coarse crackle sounds are generated by air bubbles in large bronchi and can be heard on patients with chronic bronchitis, bronchiectasis, as well as COPD.
- Pleural rub sounds are non-musical rhythmic sounds due to the rubbing of pleural membranes when breathing and are usually caused by inflammation of the pleural membrane. These qualitative differences can be rigorously characterized by their respective spectrums. Chest and throat video dynamics typically correlate, particularly for respiratory events associated with lower frequency sounds.
- System 100 includes tools for training machine learning models for the diagnosis of lower respiratory tract disease, upper respiratory tract infection, pneumonia, bronchiolitis, croup, asthma exacerbation/reactive airway disease, chronic obstructive pulmonary disease, chronic obstructive pulmonary disease exacerbation and obstructive sleep apnea.
- Potential use cases include healthcare providers in telehealth, emergency department, urgent care and primary care settings as well as humanitarian facilities in the developing world.
- FIG. 10 is a flow chart diagram 1000 of training methods for analysis of audio events.
- Process 1010 applies Fourier transform and Hann smoothing to audio input signals.
- Process 1010 then performs segmentation to produce pre-processed audio spectra, e.g., for neural network modeling.
- Process 1020 trains a model to identify respiratory conditions and patient movements.
- Process 1020 inputs pre-processed, segmented spectra from process 910 along with annotated data into a convolutional neural network (CNN).
- the CNN may be trained as a recurrent neural network, a network in which connections between nodes form a directed graph along a temporal sequence.
- the CNN performs feature matching to respiratory conditions and to sounds associated with patient movements.
- Processes 1010 , 1020 illustrate an approach to acquisition and analysis of adventitious lung sounds, which may signify pathological processes in pulmonary tissue or airways.
- This approach employs a convolutional neural network to classify spectrograms.
- the respective spectrograms of relevant sounds are preprocessed via Hann smoothing.
- Hann smoothing generally has the advantage of removing broad spectrum, low intensity noise that may be present from traffic, television, or weather.
- the recordings are split into samples, e.g., 1 second samples, and are identified using a deep convolutional neural network.
- Processes 1010 , 1020 also encompass harmonics of the fundamental frequency present from breathing. Using a regression classifier model, higher order harmonics correspond to additional parameters fitting the model to the sounds, with the constant term in the model being the natural breathing rate.
- Process 1030 trains a model for word/phrase recognition. Processes input a corpus of keywords and key phrases. Process 1030 applies the Hephaestus model for audio recognition of keywords and key phrases.
- health monitoring methods disclosed herein utilize two additional independent metrics to assess and classify respiratory effort (depth of inspiration) and tidal volume estimations from chest movement modelling. These metrics are N/M expiration and chest and throat movement.
- Real-time processing algorithms can offer fundamental information in detection and classification of adventitious sounds and can enable timely identification of diseases, as well as changes in their severity.
- Physical characteristics that may be derived from raw frame data of thermal sensor 144 include breath-rate waveforms.
- the area feature (integral under period), distance feature (period), and their derivatives can operate at between 0-6 Hz and have medical applications.
- calculating the waveform for both features follows a similar procedure (“shift-sum procedure”).
- a filter is not suitable because the waveform is a complex, multi-peaked function. Therefore, to reduce noise and ensure the waveform is properly populated, the average period is calculated over a time interval (e.g., 5 minutes). If the standard deviation is too high, the waveform is not calculated because the period is changing which indicates the patient's vital signs are shifting.
- each period is transformed according to sin(w(t ⁇ nT)), where n is the period number and T is the average period. Using this procedure, all waveforms nearly coincide and a waveform with a large sample set and low uncertainty may be obtained.
- heart rate refers to the number of times a subject's heart beats per minute.
- Heart rate variability measures the time between each heartbeat, also called an R-R interval in ECG signals. Beat-to-beat interval variation may be measured in milliseconds and can vary depending on a number of factors. For instance, the interval between heartbeats is generally longer while exhaling and shorter while inhaling. Various factors can influence or change a subject's HRV metrics, such as exercise volume and intensity, chronic health conditions, quality of sleep, and diet. Another reason for HRV is operation of the autonomic nervous system, which controls the involuntary aspects of physiology, via two branches, parasympathetic (deactivating) and sympathetic (activating).
- HRV is an extremely sensitive metric that fluctuates greatly throughout the day, from one day to the next, and from one person to another. Younger people tend to have higher HRV than older people, and males often have slightly higher HRV than females. Elite athletes usually have greater HRV.
- non-contact health monitoring systems and methods of the present disclosure can measure HRV directly from blood pressure pulse profiles representing heart hydraulic performance, which can be more informative than EKG pulse commands.
- HI Sudden hemodynamic instability due to cardiovascular and/or cardiorespiratory distress is a common occurrence.
- Causes can include hemorrhage, sepsis, pneumonia, heart failure, and others.
- Due to the body's compensatory mechanisms, heart and respiratory rate, and blood pressure can be indicators of HI. When detected late or left unrecognized, HI can lead to complications and even death.
- Signs of hemodynamic instability include having arterial systolic blood pressure ⁇ 80 or >220 mmHg, a diastolic blood pressure of >110 mmHg, a pulse rate of ⁇ 40 or >160, a respiratory rate of ⁇ 8 or >36, a SpO 2 ⁇ 90%, and temperature.
- Pulse rate variability (PRV) has been demonstrated to reflect status of the autonomic nervous system. Pulse rate has been identified as a minor indicator, while PRV has been shown to indicate significant perturbations.
- Non-contact monitoring methods disclosed herein may be employed in accurately predicting problems before they occur. This capability can aid in diagnosis and treatment of a deteriorating patient. In various embodiments, these predictions may be based on one or more metrics including core temperature from thermal imaging; respiratory rate from audio; thermal/spatial analysis of mouth, nose and chest analysis; and respiratory volume from spatial chest analysis. These real-time measurements can provide a rich and comprehensive data set for training a machine-learning HI predictor to improve model performance. Additionally, ECG is a contact non-invasive device while embodiments disclosed herein provide a non-contact device. In use cases, a non-contact HI prediction model could be deployed to monitor patients at home, in the field and even while driving.
- disclosed embodiments may be applied to understand disordered sleep of a subject in order to identify a solution.
- Disordered sleep varies considerably from person to person and can be of a physiological and psychological nature.
- a disrupted biorhythm, poor sleep posture, lowered resilience to stress, and psychological distress such as worry about being able to fall asleep all can lead to sleep disruption.
- disclosed embodiments may apply measurements of breathing rate and breath volume and could implement a feedback function to reduce breathing rate. This process may output an audio signal via mobile device 130 or via Bluetooth ear buds.
- Remedial measures could include, e.g., reducing pulse rate, and implementing an audio meditation session to induce sleep.
- a photoplethysmogram is an optically obtained plethysmogram that can be used to detect blood volume changes in a microvascular bed of tissue, effectively a spatial measurement.
- Systems and methods of the present disclosure provide thermal plethysmogrammy (thermal PG), offering various advantages over optically obtained plethysmogram.
- Optical PPG is much noisier and indistinct than thermal PG, and only thermal PG can identify the end and start of diastolic pressure pulses.
- PPGs acquired at finger or wrist have much more damped and indistinct pressure pulse profiles than are available from thermal PG acquired at arteries at temple, neck or upper arms of a subject.
- a machine learning model 160 may input thermal PG data to estimate diastolic pressures.
- Blood pressure has systolic and diastolic numbers associated with the waveform of the heart rate ( FIG. 6 ).
- a supervised learning model may be used to predict blood pressure from the thermal PG system. Given a reasonable accurate characterization of the waveform, model exploration can investigate its properties to infer the atrial fibrillation of the patient.
- Pulse shape and energy of the heart may be derived from the raw frame thermal PG data.
- the area feature (integral under period), distance feature (period), and their derivatives can operate at between 0-6 Hz and have medical applications.
- calculating the waveform for both features follows a similar procedure (herein called shift-sum procedure).
- a filter is not suitable because the waveform is a complex, multi-peaked function.
- the average period is calculated over a time interval (e.g., 5 minutes). If the standard deviation is too high, the waveform is not calculated because the period is changing which indicates the patient's vital signs are shifting.
- each period is transformed according to sin (w(t ⁇ nT)), where n is the period number and T is the average period. Using this procedure, all waveforms nearly coincide and a waveform with a large sample set and low uncertainty may be obtained.
- the pulse and its waveform may be extracted from two signals, the spatial variation of the pulse and the intensity (temperature) variation of the pulse.
- this procedure can calculate the periodicity using the systolic peak.
- the upstroke and decline then are used as inputs to the learning model for the blood pressure.
- the shape and intensity of the pulse are used to approximate the variations of pressure throughout the waveform.
- An individual waveform which may be under 1 second and may be sampled, e.g., 15 times, generally does not neatly define the shape of the pulse from which blood pressure may be derived.
- the above-described shift-sum procedure addresses this problem.
- temperature measurements can provide results superior to spatial measurements. Temperature measurements typically show reduced sensor noise.
- Disclosed systems and methods can monitor physical characteristics of a sleeping subject 120 associated with REM sleep.
- REM sleep the stage of sleep associated with dreaming, is very different physiologically from the other stages of sleep. Five stages of sleep include:
- REM atonia an almost complete paralysis of the body, is effected through the inhibition of motor neurons. Some localized twitching and reflexes still can occur. Lack of REM atonia causes REM behavior disorder.
- RBD REM sleep behavior disorder
- the paralysis that normally occurs during REM sleep is incomplete or absent, allowing the individual to “act out” his or her dreams.
- RBD is often characterized by the acting out of dreams that are vivid, intense, and violent.
- Dream-enacting behaviors can include talking, yelling, punching, kicking, sitting, jumping from bed, arm flailing, and grabbing.
- An acute form may occur during withdrawal from alcohol or sedative-hypnotic drugs
- Systems and methods of the disclosure may analyze these and other characteristics in monitoring sleep, e.g., to measure REM sleep and monitor movements of a resting body.
- a larger set of metrics that may be employed in sleep metrology include core temperature, heart rate, HRV, breathing rate, respiration rate, thermal sensing of REM, audio correlations with breathing rate and spectral analysis of breathing ailments, and visual monitoring of RBD movements. In a use case, this data is employed in monitoring and characterizing any RBD.
- This ensemble of metrics can be used to more precise monitor and identify stages of sleep of a subject 120 . Facial feature recognition ( FIG. 5 ) may be applied to detect edges for identifying eye motion in monitoring REM sleep.
- the system ranks quality of sleep based on several factors. These factors may include standard deviation of the breath rate, standard deviation of the heart rate, motion of the eyes (which could indicate REM sleep), and amount of motion of the patient, e.g., as defined by velocity of tracked features.
- a machine learning model applying classifier/factorization machine prediction acts as a multivariate sleep quality classifier.
- a machine learning model applying random forest classification acts as a multivariate sleep quality classifier.
- Disclosed embodiments monitor several sleep apneas.
- Obstructive sleep apnea is caused by a blockage of the airway.
- CSA central sleep apnea
- Complex sleep apnea is a combination of the two conditions.
- the brain rouses the sleeper, usually only partially, to signal breathing to resume.
- REM rapid-eye-movement
- the sleeping CSA patient displays not a periodic failure to breathe but a periodic shallow breathing or under-breathing that alternates with deep over-breathing, a condition known as Cheyne-Stokes breathing.
- the disorder reduces oxygenation of the blood, further stressing the sleeper's health.
- Disclosed embodiments recognize several sleep apneas and may generate an alert/alarm if an apnea event occurs.
- thermal imaging correlates facial feature recognition of tear ducts temperature with core temperature, e.g., to determine if an individual is healthy. Skin temperature is different than core temperature but useful to monitor to check if person is in field of view and for possible health issues. Tear duct temperature is representative of core temperature. Over the course of a night, core temperature decreases slightly.
- feature detection observes the tear ducts. Locations of the tear ducts may be identified using the same machine learning model that has been trained to identify other facial features ( FIG. 5 ).
- This thermal imaging data may be employed to calculate core temperature.
- FIG. 11 is a graph of maximum facial temperature as a function of camera distance, measured for 44 year old female and 7 year old male. The imaging system estimates proximity of the thermal camera to the subject to correct core temperature readings. Based on field of view and number of pixels occupied by face, the system can estimate distance from the camera. Using this estimate, the correction reduces temperature recorded by camera for range.
- This calculation and predetermined relationship shown in FIG. 11 enable accurate determination of core temperature from tear duct temperature observations. In an example, distance is measured using FLIR® Research Studio software. FLIR is a registered trademark of Teledyne FLIR, LLC, Wilsonville Oreg.
- FIG. 12 is a schematic block diagram of a general health monitoring architecture 1100 . Key functions include:
- process flow diagrams are provided merely as illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order presented.
- the steps in the foregoing embodiments may be performed in any order. Words such as “then,” “next,” etc., are not intended to limit the order of the steps; these words are simply used to guide the reader through the description of the methods.
- process flow diagrams may describe the operations as addition, the order of the operations may be rearranged.
- a process may correspond to a method, a function, a procedure, a subroutine, a subprogram, and the like.
- the process termination may correspond to a return of the function to a calling function or a main function.
- Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof.
- a code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements.
- a code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents.
- Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.
- the functions When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium.
- the steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a computer-readable or processor-readable storage medium.
- a non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another.
- a non-transitory processor-readable storage media may be any available media that may be accessed by a computer.
- non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor.
- Disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
- the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable medium and/or computer-readable medium, which may be incorporated into a computer program product.
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Molecular Biology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Biomedical Technology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Pathology (AREA)
- Biophysics (AREA)
- Physiology (AREA)
- Artificial Intelligence (AREA)
- Cardiology (AREA)
- Pulmonology (AREA)
- Mathematical Physics (AREA)
- Psychiatry (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Fuzzy Systems (AREA)
- Evolutionary Computation (AREA)
- Dentistry (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
- Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/510,357 US20220133156A1 (en) | 2020-10-29 | 2021-10-25 | Cardiopulmonary health monitoring using thermal camera and audio sensor |
US18/527,127 US12023135B2 (en) | 2020-10-29 | 2023-12-01 | Cardiopulmonary health monitoring using thermal camera and audio sensor |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202063107036P | 2020-10-29 | 2020-10-29 | |
US17/510,357 US20220133156A1 (en) | 2020-10-29 | 2021-10-25 | Cardiopulmonary health monitoring using thermal camera and audio sensor |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/527,127 Continuation US12023135B2 (en) | 2020-10-29 | 2023-12-01 | Cardiopulmonary health monitoring using thermal camera and audio sensor |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220133156A1 true US20220133156A1 (en) | 2022-05-05 |
Family
ID=81381221
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/510,357 Abandoned US20220133156A1 (en) | 2020-10-29 | 2021-10-25 | Cardiopulmonary health monitoring using thermal camera and audio sensor |
US18/527,127 Active US12023135B2 (en) | 2020-10-29 | 2023-12-01 | Cardiopulmonary health monitoring using thermal camera and audio sensor |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/527,127 Active US12023135B2 (en) | 2020-10-29 | 2023-12-01 | Cardiopulmonary health monitoring using thermal camera and audio sensor |
Country Status (3)
Country | Link |
---|---|
US (2) | US20220133156A1 (de) |
EP (1) | EP4236775A1 (de) |
WO (1) | WO2022093707A1 (de) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115410138A (zh) * | 2022-11-02 | 2022-11-29 | 山东省人工智能研究院 | 基于特征编码器分类和定位串行的视频动作检测方法 |
CN115496170A (zh) * | 2022-11-17 | 2022-12-20 | 中南民族大学 | 一种人体姿态识别方法、系统、电子设备及存储介质 |
US20230238113A1 (en) * | 2022-01-25 | 2023-07-27 | Unitedhealth Group Incorporated | Machine learning techniques for parasomnia episode management |
CN117558449A (zh) * | 2023-11-24 | 2024-02-13 | 启康保(北京)健康科技有限公司 | 基于大数据的医疗健康数据管理系统 |
WO2024129685A1 (en) * | 2022-12-13 | 2024-06-20 | Lifeq B.V. | Pre-diagnostic disease screening using wearable derived data |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110112442A1 (en) * | 2007-05-02 | 2011-05-12 | Earlysense Ltd. | Monitoring, Predicting and Treating Clinical Episodes |
US20120289850A1 (en) * | 2011-05-09 | 2012-11-15 | Xerox Corporation | Monitoring respiration with a thermal imaging system |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070118054A1 (en) | 2005-11-01 | 2007-05-24 | Earlysense Ltd. | Methods and systems for monitoring patients for clinical episodes |
US8360986B2 (en) * | 2006-06-30 | 2013-01-29 | University Of Louisville Research Foundation, Inc. | Non-contact and passive measurement of arterial pulse through thermal IR imaging, and analysis of thermal IR imagery |
US8109875B2 (en) * | 2007-01-03 | 2012-02-07 | Gizewski Theodore M | Derma diagnostic and automated data analysis system |
US8638364B2 (en) * | 2010-09-23 | 2014-01-28 | Sony Computer Entertainment Inc. | User interface system and method using thermal imaging |
US9000371B2 (en) | 2011-08-26 | 2015-04-07 | Custom Scene Technology, Inc. | Camera, computer program and method for measuring thermal radiation and thermal rates of change |
US20140152424A1 (en) * | 2012-12-05 | 2014-06-05 | Unisys Corporation | Method and system for security screening using biometric variables |
US9486146B2 (en) * | 2015-03-25 | 2016-11-08 | Xerox Corporation | Detecting tumorous breast tissue in a thermal image |
WO2017178292A1 (en) * | 2016-04-15 | 2017-10-19 | Koninklijke Philips N.V. | Annotating data points associated with clinical decision support application |
US11241167B2 (en) | 2016-08-17 | 2022-02-08 | The Regents Of The University Of Colorado, A Body Corporate | Apparatus and methods for continuous and fine-grained breathing volume monitoring |
US10861605B2 (en) | 2016-08-22 | 2020-12-08 | Aic Innovations Group, Inc. | Method and apparatus for determining health status |
JP2020503102A (ja) | 2016-08-26 | 2020-01-30 | ライオット ソリューションズ インコーポレイテッドRiot Solutions,Inc. | 非侵襲的に非接触で健康状態をモニタリングするためのシステム及び方法 |
WO2018112613A1 (en) * | 2016-12-19 | 2018-06-28 | Nuralogix Corporation | System and method for contactless blood pressure determination |
WO2018212710A1 (en) * | 2017-05-19 | 2018-11-22 | National University Of Singapore | Predictive analysis methods and systems |
US10918331B2 (en) | 2017-09-19 | 2021-02-16 | Aic Innovations Group, Inc. | Method and apparatus for determining health status |
US20210212576A1 (en) * | 2018-02-23 | 2021-07-15 | Alunos Ag | Monitoring of Physiological Parameters |
US20190385711A1 (en) | 2018-06-19 | 2019-12-19 | Ellipsis Health, Inc. | Systems and methods for mental health assessment |
TWI687206B (zh) * | 2018-12-06 | 2020-03-11 | 財團法人工業技術研究院 | 異常肺音標記的方法及裝置 |
WO2021067860A1 (en) | 2019-10-03 | 2021-04-08 | DawnLight Technologies Inc. | Systems and methods for contactless sleep monitoring |
KR102199020B1 (ko) | 2020-05-08 | 2021-01-06 | 성균관대학교산학협력단 | 천장형 인공지능 건강 모니터링 장치 및 이를 이용한 원격 의료 진단 방법 |
-
2021
- 2021-10-25 EP EP21887268.7A patent/EP4236775A1/de not_active Withdrawn
- 2021-10-25 US US17/510,357 patent/US20220133156A1/en not_active Abandoned
- 2021-10-25 WO PCT/US2021/056482 patent/WO2022093707A1/en active Application Filing
-
2023
- 2023-12-01 US US18/527,127 patent/US12023135B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110112442A1 (en) * | 2007-05-02 | 2011-05-12 | Earlysense Ltd. | Monitoring, Predicting and Treating Clinical Episodes |
US20120289850A1 (en) * | 2011-05-09 | 2012-11-15 | Xerox Corporation | Monitoring respiration with a thermal imaging system |
Non-Patent Citations (8)
Title |
---|
Ali I. Siam et al., "Efficient video-based breathing pattern and respiration rate monitoring for remote health monitoring," J. Opt. Soc. Am. A 37, C118-C124 (2020), Published: August 31, 2020, https://doi.org/10.1364/JOSAA.399284, viewed on 1/27/23 * |
Carina Pereira et al. Remote monitoring of breathing dynamics using infrared thermography. Biomed Opt Express. 2015 Oct 16;6(11):4378-94. doi: 10.1364/BOE.6.004378. PMID: 26601003; PMCID: PMC4646547. viewed on 2/3/22; * |
Carina Pereira et al., "Noncontact Monitoring of Respiratory Rate in Newborn Infants Using Thermal Imaging," in IEEE Transactions on Biomedical Engineering, vol. 66, no. 4, pp. 1105-1114, April 2019, doi: 10.1109/TBME.2018.2866878, viewed on 1/27/23; * |
Ching Wang et al., "Unconstrained Video Monitoring of Breathing Behavior and Application to Diagnosis of Sleep Apnea," in IEEE Transactions on Biomedical Engineering, vol. 61, no. 2, pp. 396-404, Feb. 2014, doi: 10.1109/TBME.2013.2280132. viewed on 2/3/23 * |
Christian Hessler et al., "A Non-contact Method for Extracting Heart and Respiration Rates," 2020 17th Conference on Computer and Robot Vision (CRV), Ottawa, ON, Canada, 2020, pp. 1-8, doi: 10.1109/CRV50864.2020.00009. viewed on 2/3/22; * |
FLIR, How Do Thermal Cameras Work?, https://www.flir.com/discover/rd-science/how-do-thermal-cameras-work/, 6/16/20, viewed on 8/1/22 * |
Martin Manullang et al. Implementation of Thermal Camera for Non-Contact Physiological Measurement: A Systematic Review. Sensors (Basel). 2021 Nov 23;21(23):7777. doi: 10.3390/s21237777. PMID: 34883780; PMCID: PMC8659982. Viewed on 2/2/23 * |
Prasara Jakkaew et al. Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging. Sensors (Basel). 2020 Nov 5;20(21):6307. doi: 10.3390/s20216307. PMID: 33167556; PMCID: PMC7663997. viewed on 8/1/22 * |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230238113A1 (en) * | 2022-01-25 | 2023-07-27 | Unitedhealth Group Incorporated | Machine learning techniques for parasomnia episode management |
CN115410138A (zh) * | 2022-11-02 | 2022-11-29 | 山东省人工智能研究院 | 基于特征编码器分类和定位串行的视频动作检测方法 |
CN115496170A (zh) * | 2022-11-17 | 2022-12-20 | 中南民族大学 | 一种人体姿态识别方法、系统、电子设备及存储介质 |
WO2024129685A1 (en) * | 2022-12-13 | 2024-06-20 | Lifeq B.V. | Pre-diagnostic disease screening using wearable derived data |
CN117558449A (zh) * | 2023-11-24 | 2024-02-13 | 启康保(北京)健康科技有限公司 | 基于大数据的医疗健康数据管理系统 |
Also Published As
Publication number | Publication date |
---|---|
US20240090778A1 (en) | 2024-03-21 |
EP4236775A1 (de) | 2023-09-06 |
WO2022093707A1 (en) | 2022-05-05 |
US12023135B2 (en) | 2024-07-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US12023135B2 (en) | Cardiopulmonary health monitoring using thermal camera and audio sensor | |
Mendonca et al. | A review of obstructive sleep apnea detection approaches | |
US11678838B2 (en) | Automated detection of breathing disturbances | |
US11712198B2 (en) | Estimation of sleep quality parameters from whole night audio analysis | |
CA2872785C (en) | Sound-based spirometric devices, systems, and methods | |
US9655559B2 (en) | Automated sleep staging using wearable sensors | |
US20230190140A1 (en) | Methods and apparatus for detection and monitoring of health parameters | |
JeyaJothi et al. | [Retracted] A Comprehensive Review: Computational Models for Obstructive Sleep Apnea Detection in Biomedical Applications | |
Casal et al. | Classifying sleep–wake stages through recurrent neural networks using pulse oximetry signals | |
US20180233234A1 (en) | System and Method for Sympathetic and Parasympathetic Activity Monitoring by Heartbeat | |
WO2021208656A1 (zh) | 睡眠风险预测方法、装置和终端设备 | |
US20220218273A1 (en) | System and Method for Noninvasive Sleep Monitoring and Reporting | |
Tabatabaei et al. | Methods for adventitious respiratory sound analyzing applications based on smartphones: A survey | |
Paraschiv et al. | Machine learning approaches based on wearable devices for respiratory diseases diagnosis | |
WO2021053645A1 (en) | System and method for determining sleep stages based on non-cardiac body signals | |
US20220378377A1 (en) | Augmented artificial intelligence system and methods for physiological data processing | |
US20220165393A1 (en) | System for the detection and management of mental, emotional, and behavioral disorders | |
Pandey et al. | Nocturnal sleep sounds classification with artificial neural network for sleep monitoring | |
US20240350021A1 (en) | Cardiopulmonary health monitoring using thermal camera and audio sensor | |
US20210282736A1 (en) | Respiration rate detection metholody for nebulizers | |
Adithya et al. | A Deep Learning Framework for Comprehensive Sleep Analysis and Quality Tracking | |
Almazaydeh | An interactive, real-time, high precision and portable monitoring system of obstructive sleep apnea | |
Bello et al. | A wearable, cloud-based system to enable Alzheimer's disease analysis, diagnosis, and progression monitoring | |
Shewinvanakitkul | Automated detection and prediction of sleep apnea events | |
Dhole et al. | A Research Study On The Diagnostic Capabilities Of Deep Learning Regarding Sleep Apnea |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ROC8SCI CO., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HARTLEY, FRANK THOMAS;ROSS, CHARLES HENRY;SIGNING DATES FROM 20210804 TO 20210807;REEL/FRAME:057906/0187 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STCV | Information on status: appeal procedure |
Free format text: NOTICE OF APPEAL FILED |
|
STCV | Information on status: appeal procedure |
Free format text: APPEAL BRIEF (OR SUPPLEMENTAL BRIEF) ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: TC RETURN OF APPEAL |
|
STCV | Information on status: appeal procedure |
Free format text: ON APPEAL -- AWAITING DECISION BY THE BOARD OF APPEALS |
|
STCV | Information on status: appeal procedure |
Free format text: BOARD OF APPEALS DECISION RENDERED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION |