US20230222805A1 - Machine learning based monitoring system - Google Patents
Machine learning based monitoring system Download PDFInfo
- Publication number
- US20230222805A1 US20230222805A1 US18/153,173 US202318153173A US2023222805A1 US 20230222805 A1 US20230222805 A1 US 20230222805A1 US 202318153173 A US202318153173 A US 202318153173A US 2023222805 A1 US2023222805 A1 US 2023222805A1
- Authority
- US
- United States
- Prior art keywords
- camera
- model
- infant
- person
- potential
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010801 machine learning Methods 0.000 title claims abstract description 100
- 238000012544 monitoring process Methods 0.000 title abstract description 216
- 238000000034 method Methods 0.000 claims abstract description 104
- 238000012216 screening Methods 0.000 claims abstract description 79
- 230000004044 response Effects 0.000 claims abstract description 54
- 230000000694 effects Effects 0.000 claims abstract description 36
- 230000008569 process Effects 0.000 claims abstract description 35
- 238000001514 detection method Methods 0.000 claims description 131
- 230000001815 facial effect Effects 0.000 claims description 59
- 238000012549 training Methods 0.000 claims description 51
- 208000004929 Facial Paralysis Diseases 0.000 claims description 32
- 208000036826 VIIth nerve paralysis Diseases 0.000 claims description 32
- 208000006550 Mydriasis Diseases 0.000 claims description 21
- 238000012360 testing method Methods 0.000 claims description 17
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 claims description 14
- 229910052760 oxygen Inorganic materials 0.000 claims description 14
- 239000001301 oxygen Substances 0.000 claims description 14
- 239000008280 blood Substances 0.000 claims description 13
- 210000004369 blood Anatomy 0.000 claims description 13
- 230000029058 respiratory gaseous exchange Effects 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 11
- 230000010412 perfusion Effects 0.000 claims description 7
- 206010012289 Dementia Diseases 0.000 claims description 5
- 238000002106 pulse oximetry Methods 0.000 claims description 5
- 210000002784 stomach Anatomy 0.000 claims description 5
- 206010003497 Asphyxia Diseases 0.000 claims description 3
- 230000009471 action Effects 0.000 description 18
- 208000003443 Unconsciousness Diseases 0.000 description 17
- 238000013528 artificial neural network Methods 0.000 description 14
- 238000013527 convolutional neural network Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 11
- 239000003814 drug Substances 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 229940079593 drug Drugs 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 210000003128 head Anatomy 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 210000000887 face Anatomy 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000005406 washing Methods 0.000 description 3
- 210000000707 wrist Anatomy 0.000 description 3
- 206010011469 Crying Diseases 0.000 description 2
- 206010039740 Screaming Diseases 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000001680 brushing effect Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000010419 pet care Methods 0.000 description 2
- 230000004461 rapid eye movement Effects 0.000 description 2
- 238000011012 sanitization Methods 0.000 description 2
- 230000007958 sleep Effects 0.000 description 2
- CVOFKRWYWCSDMA-UHFFFAOYSA-N 2-chloro-n-(2,6-diethylphenyl)-n-(methoxymethyl)acetamide;2,6-dinitro-n,n-dipropyl-4-(trifluoromethyl)aniline Chemical compound CCC1=CC=CC(CC)=C1N(COC)C(=O)CCl.CCCN(CCC)C1=C([N+]([O-])=O)C=C(C(F)(F)F)C=C1[N+]([O-])=O CVOFKRWYWCSDMA-UHFFFAOYSA-N 0.000 description 1
- 208000025721 COVID-19 Diseases 0.000 description 1
- 206010011224 Cough Diseases 0.000 description 1
- 206010013887 Dysarthria Diseases 0.000 description 1
- 206010020751 Hypersensitivity Diseases 0.000 description 1
- 208000000224 Night Terrors Diseases 0.000 description 1
- 206010041010 Sleep terror Diseases 0.000 description 1
- 206010041235 Snoring Diseases 0.000 description 1
- 206010041347 Somnambulism Diseases 0.000 description 1
- 206010042496 Sunburn Diseases 0.000 description 1
- 206010044565 Tremor Diseases 0.000 description 1
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000007815 allergy Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 210000000038 chest Anatomy 0.000 description 1
- 230000001055 chewing effect Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000037406 food intake Effects 0.000 description 1
- 235000012631 food intake Nutrition 0.000 description 1
- 210000001061 forehead Anatomy 0.000 description 1
- 238000012074 hearing test Methods 0.000 description 1
- 208000000122 hyperventilation Diseases 0.000 description 1
- 230000000870 hyperventilation Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010348 incorporation Methods 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000000474 nursing effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000004962 physiological condition Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000037380 skin damage Effects 0.000 description 1
- 201000002859 sleep apnea Diseases 0.000 description 1
- 230000003860 sleep quality Effects 0.000 description 1
- 208000026473 slurred speech Diseases 0.000 description 1
- 206010041232 sneezing Diseases 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/04—Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons
- G08B21/0407—Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons based on behaviour analysis
- G08B21/043—Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons based on behaviour analysis detecting an emergency event, e.g. a fall
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/0202—Child monitoring systems using a transmitter-receiver system carried by the parent and the child
- G08B21/0205—Specific application combined with child monitoring using a transmitter-receiver system
- G08B21/0208—Combination with audio or video communication, e.g. combination with "baby phone" function
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/0202—Child monitoring systems using a transmitter-receiver system carried by the parent and the child
- G08B21/0294—Display details on parent unit
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/04—Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons
- G08B21/0438—Sensor means for detecting
- G08B21/0492—Sensor dual technology, i.e. two or more technologies collaborate to extract unsafe condition, e.g. video tracking and RFID tracking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30041—Eye; Retina; Ophthalmic
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30232—Surveillance
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
Definitions
- a smart camera system can be a machine vision system which, in addition to image capture capabilities, is capable of extracting information from captured images. Some smart camera systems are capable of generating event descriptions and/or making decisions that are used in an automated system. Some camera systems can be a self-contained, standalone vision system with a built-in image sensor. The vision system and the image sensor can be integrated into a single hardware device. Some camera systems can include communication interfaces, such as, but not limited to Ethernet and/or wireless interfaces.
- Safety can be important in clinical, hospice, assisted living, and/or home settings. Potentially dangerous events can happen in these environments. Automation can also be beneficial in these environments.
- a system comprising: a storage device configured to store first instructions and second instructions; a camera; a hardware accelerator configured to execute the first instructions; and a hardware processor configured to execute the second instructions to: receive, from the camera, first image data; invoke, on the hardware accelerator, a person detection model based on the first image data, wherein the person detection model outputs first classification result; detect a person based on the first classification result; receive, from the camera, second image data; and in response to detecting the person, invoke, on the hardware accelerator, a fall detection model based on the second image data, wherein the fall detection model outputs a second classification result, detect a potential fall based on the second classification result, and in response to detecting the potential fall, provide an alert.
- the system may further comprise a microphone
- the hardware processor may be configured to execute further instructions to: receive, from the microphone, audio data; and in response to detecting the person, invoke, on the hardware accelerator, a loud noise detection model based on the audio data, wherein the loud noise detection model outputs a third classification result, and detect a potential scream based on the third classification result.
- the hardware processor may be configured to execute additional instructions to: in response to detecting the potential scream, provide a second alert.
- the hardware processor may be configured to execute additional instructions to: in response to detecting the potential fall and the potential scream, provide an escalated alert.
- invoking the loud noise detection model based on the audio data may further comprise: generating spectrogram data from the audio data; and providing the spectrogram data as input to the loud noise detection model.
- the second image data may comprise a plurality of images.
- a method comprising: receiving, from a camera, first image data; invoking, on a hardware accelerator, a person detection model based on the first image data, wherein the person detection model outputs first classification result; detecting a person based on the first classification result; receiving, from the camera, second image data; and in response to detecting the person, invoking, on the hardware accelerator, a plurality of person safety models based on the second image data, for each person safety model from the plurality of person safety models, receiving, from the hardware accelerator, a second classification result, detecting a potential safety issue based on a particular second classification result, and in response to detecting the potential safety issue, providing an alert.
- the method may further comprise: in response to detecting the person, invoking, on the hardware accelerator, a facial feature extraction model based on the second image data, wherein the facial feature extraction model outputs a facial feature vector, executing a query of a facial features database based on the facial feature vector, wherein executing the query indicates that the facial feature vector is not present in the facial features database, and in response to determining that the facial feature vector is not present in the facial features database, providing an unrecognized person alert.
- the plurality of person safety models may comprise a fall detection model
- the method may further comprise: collecting a first set of videos of person falls; collecting a second set of videos of persons without falling; creating a training data set comprising the first set of videos and the second set of videos; and training the fall detection model using the training data set.
- the plurality of person safety models may comprise a handwashing detection model
- the method may further comprise: collecting a first set of videos of with handwashing; collecting a second set of videos without handwashing; creating a training data set comprising the first set of videos and the second set of videos; and training the handwashing detection model using the training data set.
- the method may further comprise: receiving, from a microphone, audio data; and in response to detecting the person, invoking, on the hardware accelerator, a loud noise detection model based on the audio data, wherein the loud noise detection model outputs a third classification result, and detecting a potential scream based on the third classification result.
- the method may further comprise: in response to detecting the potential safety issue and the potential scream, providing an escalated alert.
- the method may further comprise: collecting a first set of videos of with screaming; collecting a second set of videos without screaming; creating a training data set comprising the first set of videos and the second set of videos; and training the loud noise detection model using the training data set.
- a system comprising: a storage device configured to store first instructions and second instructions; a camera; a hardware accelerator configured to execute the first instructions; and a hardware processor configured to execute the second instructions to: receive, from the camera, first image data; invoke, on the hardware accelerator, a person detection model based on the first image data, wherein the person detection model outputs first classification result; detect a person based on the first classification result; receive, from the camera, second image data; and in response to detecting the person, invoke, on the hardware accelerator, a plurality of person safety models based on the second image data, for each person safety model from the plurality of person safety models, receive, from the hardware accelerator, a model result, detect a potential safety issue based on a particular model result, and in response to detecting the potential safety issue, provide an alert.
- the plurality of person safety models may comprise a fall detection model
- invoking the plurality of person safety models may comprise: invoking, on the hardware accelerator, the fall detection model based on the second image data, wherein the fall detection model outputs the particular model result.
- the plurality of person safety models may comprise a handwashing detection model
- invoking the plurality of person safety models may comprise: invoking, on the hardware accelerator, the handwashing detection model based on the second image data, wherein the handwashing detection model outputs the particular model result.
- the system may further comprise a microphone
- the hardware processor may be configured to execute further instructions to: receive, from the microphone, audio data; and in response to detecting the person, invoke, on the hardware accelerator, a loud noise detection model based on the audio data, wherein the loud noise detection model outputs a third classification result, detect a potential loud noise based on the third classification result, and in response to detecting the potential loud noise, provide a second alert.
- the system may further comprise a display, wherein the hardware processor may be configured to execute further instructions to: cause presentation, on the display, of a prompt to cause a person to perform an activity; receive, from the camera, third image data of a recording of the activity; invoke, on the hardware accelerator, a screening machine learning model based on the third image data, wherein the screening machine learning model outputs a third classification result, detect a potential screening issue based on the third classification result, and in response to detecting the potential screening issue, provide a second alert.
- the hardware processor may be configured to execute further instructions to: cause presentation, on the display, of a prompt to cause a person to perform an activity; receive, from the camera, third image data of a recording of the activity; invoke, on the hardware accelerator, a screening machine learning model based on the third image data, wherein the screening machine learning model outputs a third classification result, detect a potential screening issue based on the third classification result, and in response to detecting the potential screening issue, provide a second alert.
- the screening machine learning model may be a pupillometry screening model, and wherein the potential screening issue indicates potential dilated pupils.
- the screening machine learning model may be a facial paralysis screening model, and wherein the potential screening issue indicates potential facial paralysis.
- a system comprising: a storage device configured to store first instructions and second instructions; a wearable device configured to process sensor signals to determine a physiological value for a person; a microphone; a camera; a hardware accelerator configured to execute the first instructions; and a hardware processor configured to execute the second instructions to: receive, from the wearable device, the first physiological value; determine to begin a monitoring process based on the first physiological value; and in response to determining to begin the monitoring process, receive, from the camera, image data; receive, from the microphone, audio data; invoke, on the hardware accelerator, a first unconscious detection model based on the image data, wherein the first unconscious detection model outputs a first classification result, invoke, on the hardware accelerator, a second unconscious detection model based on the audio data, wherein the second unconscious detection model outputs a second classification result, detect a potential state of unconsciousness based on the first classification result and the second classification result, and in response to detecting the potential state of unconsciousness, provide an alert.
- the wearable device may comprise a pulse oximetry sensor and the first physiological value is for blood oxygen saturation, and wherein determining to begin the monitoring process based on the first physiological value further comprises: determining that the first physiological value is below a threshold level.
- the wearable device may comprise a respiration rate sensor and the first physiological value is for respiration rate, and wherein determining to begin the monitoring process based on the first physiological value further comprises: determining that the first physiological value satisfies a threshold alarm level.
- the wearable device comprises a heart rate sensor and the first physiological value is for heart rate
- determining to begin the monitoring process based on the physiological value further comprises: receiving, from the wearable device, a plurality of physiological values measuring heart rate over time; and determining that the plurality of physiological values and the first physiological value satisfies a threshold alarm level.
- a system comprising: a storage device configured to store instructions; a display; a camera; and a hardware processor configured to execute the instructions to: receive a current time; determine to begin a check-up process from the current time; and in response to determining to begin the check-up process, cause presentation, on the display, of a prompt to cause a person to perform a check-up activity, receive, from the camera, image data of a recording of the check-up activity, invoke a screening machine learning model based on the image data, wherein the screening machine learning model outputs a classification result, detect a potential screening issue based on the classification result, and in response to detecting the potential screening issue, provide an alert.
- the screening machine learning model may be a pupillometry screening model, and wherein the potential screening issue indicates potential dilated pupils.
- the screening machine learning model may be a facial paralysis screening model, and wherein the potential screening issue indicates potential facial paralysis.
- the system may further comprise a wearable device configured to process sensor signals to determine a physiological value for the person, wherein the hardware processor may be configured to execute further instructions to: receive, from the wearable device, the physiological value; and generate the alert comprising the physiological value.
- the wearable device may comprise a pulse oximetry sensor and the physiological value is for blood oxygen saturation.
- the wearable device may be further configured to process the sensor signals to measure at least one of blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, or pleth variability index.
- the hardware processor may be configured to execute further instructions to: receive, from a second computing device, first video data; cause presentation, on the display, of the first video data; receive, from the camera, second video data; and transmit, to the second computing device, the second video data.
- a method comprising: receiving a current time; determining to begin a check-up process from the current time; and in response to determining to begin the check-up process, causing presentation, on a display, of a prompt to cause a person to perform a check-up activity, receiving, from a camera, image data of a recording of the check-up activity, invoking a screening machine learning model based on the image data, wherein the screening machine learning model outputs a model result, detecting a potential screening issue based on the model result, and in response to detecting the potential screening issue, providing an alert.
- the screening machine learning model may be a pupillometry screening model, and wherein the potential screening issue indicates potential dilated pupils, the method further comprise: collecting a first set of images of dilated pupils; collecting a second set of images without dilated pupils; creating a training data set comprising the first set of images and the second set of images; and training the pupillometry screening model using the training data set.
- the screening machine learning model may be a facial paralysis screening model, and wherein the potential screening issue indicates potential facial paralysis, the method may further comprise: collecting a first set of images of facial paralysis; collecting a second set of images without facial paralysis; creating a training data set comprising the first set of images and the second set of images; and training the facial paralysis screening model using the training data set.
- the check-up activity may comprise a dementia test
- the screening machine learning model may comprise a gesture detection model
- the gesture detection model may be configured to detect a gesture directed towards a portion of the display.
- the method may further comprise: receiving, from the camera, second image data; invoking a person detection model based on the second image data, wherein the person detection model outputs first classification result; detect a person based on the first classification result; receive, from the camera, third image data; and in response to detecting the person, invoking a handwashing detection model based on the third image data, wherein the handwashing detection model outputs a second classification result, detecting a potential lack of handwashing based on the second classification result, and in response to detecting a lack of handwashing, provide a second alert.
- a system comprising: a storage device configured to store instructions; a camera; and a hardware processor configured to execute the instructions to: receive, from the camera, first image data; invoke an infant detection model based on the first image data, wherein the infant detection model outputs a classification result; detect an infant based on the classification result; receive captured data; and in response to detecting the infant, invoke an infant safety model based on the captured data, wherein the infant safety model outputs a model result, detect a potential safety issue based on the model result, and in response to detecting the potential safety issue, provide an alert.
- the infant safety model may be an infant position model, and wherein the potential safety issue indicates the infant potentially laying on their stomach.
- the hardware processor may be configured to execute further instructions to: receive, from the camera, second image data; and in response to detecting the infant, invoke a facial feature extraction model based on the second image data, wherein the facial feature extraction model outputs a facial feature vector, execute a query of a facial features database based on the facial feature vector, wherein executing the query indicates that the facial feature vector is not present in the facial features database, and in response to determining that the facial feature vector is not present in the facial features database, provide an unrecognized person alert.
- the infant safety model may be an infant color detection model, and wherein the potential safety issue indicates potential asphyxiation.
- the model result may comprise coordinates of a boundary region identifying an infant object in the captured data, and wherein detecting the potential safety issue may comprise: determining that the coordinates of the boundary region exceed a threshold distance from an infant zone.
- the system may further comprise a wearable device configured to process sensor signals to determine a physiological value for the infant, wherein the hardware processor may be configured to execute further instructions to: receive, from the wearable device, the physiological value; and generate the alert comprising the physiological value.
- the system may further comprise a microphone, wherein the captured data is received from the microphone, wherein the infant safety model is a loud noise detection model, and wherein the potential safety issue indicates a potential scream.
- systems and/or computer systems comprise a computer readable storage medium having program instructions embodied therewith, and one or more processors configured to execute the program instructions to cause the one or more processors to perform operations comprising one or more of the above- and/or below-aspects (including one or more aspects of the appended claims).
- FIG. 1 A is a drawing of a camera system in a clinical setting.
- FIG. 1 B is a schematic diagram illustrating a monitoring system.
- FIG. 2 is a schematic drawing of a monitoring system in a clinical setting.
- FIG. 3 is another schematic drawing of a monitoring system in a clinical setting.
- FIG. 4 is a drawing of patient sensor devices that can be used in a monitoring system.
- FIG. 5 illustrates a camera image with object tracking.
- FIG. 6 is a drawing of a monitoring system in a home setting.
- FIG. 7 is a drawing of a monitoring system configured for baby monitoring.
- FIG. 8 is a flowchart of a method for efficiently applying machine learning models.
- FIG. 9 is a flowchart of another method for efficiently applying machine learning models.
- FIG. 10 is a flowchart of a method for efficiently applying machine learning models for infant care.
- FIG. 11 illustrates a block diagram of a computing device that may implement one or more aspects of the present disclosure.
- some camera systems are capable of extracting information from captured images.
- extracting information from images and/or monitoring by existing camera systems can be limited.
- Technical improvements regarding monitoring people and/or objects and automated actions based on the monitoring can advantageously be helpful, improve safety, and possibly save lives.
- a camera system can include a camera and a hardware accelerator.
- the camera system can include multiple machine learning models. Each model of the machine learning models can be configured to detect an object and/or an activity.
- the hardware accelerator can be special hardware that is configured to accelerate machine learning applications.
- the camera system can be configured to execute the machine learning models on the hardware accelerator.
- the camera system can advantageously be configured to execute conditional logic to determine which machine learning models should be applied and when. For example, until a person is detected in an area, the camera system may not apply any machine learning models related to persons, such as, but not limited to, fall detection, person identification, stroke detection, medication tracking, activity tracking, etc.
- Some existing monitoring systems can have limited artificial intelligence capabilities. For example, some existing monitoring systems may only have basic person, object, or vehicle detection. Moreover, some existing monitoring systems may require a network connection from local cameras to backend servers that perform the artificial intelligence processing. Some existing cameras may have limited or no artificial intelligence capabilities. Performing artificial intelligence processing locally on cameras can be technically challenging. For example, the hardware processors and/or memory devices in existing cameras may be so limited as being unable to execute machine learning models locally. Moreover, existing cameras may have limited software to be able to execute machine learning models locally in an efficient manner. The systems and methods described herein may efficiently process camera data either locally and/or in a distributed manner with machine learning models. Accordingly, the systems and methods described herein may improve over existing artificial intelligence monitoring technology.
- camera and “camera system” can be used interchangeably. Moreover, as used herein, “camera” and “camera system” can be used interchangeably with “monitoring system” since a camera system can encompass a monitoring system in some aspects.
- FIG. 1 A depicts a camera system 114 in a clinical setting 101 .
- the clinical setting 101 can be, but is not limited to, a hospital, nursing home, or hospice.
- the clinical setting 101 can include the camera system 114 , a display 104 , and a user computing device 108 .
- the camera system 114 can be housed in a soundbar enclosure or a tabletop speaker enclosure (not illustrated).
- the camera system 114 can include multiple cameras (such as 1080 p or 4 k camera and/or an infrared image camera), an output speaker, an input microphone (such as a microphone array), an infrared blaster, and/or multiple hardware processors (including one or more hardware accelerators).
- the camera system 114 can have optical zoom.
- the camera system 114 can include a privacy switch that allows the monitoring system's 100 A, 100 B cameras to be closed.
- the camera system 114 may receive voice commands.
- the camera system 114 can include one or more hardware components for Bluetooth®, Bluetooth Low Energy (BLE), Ethernet, Wi-Fi, cellular (such as 4G/5G/LTE), near-field communication (NFC), radio-frequency identification (RFID), High-Definition Multimedia Interface (HDMI), and/or HDMI Consumer Electronics Control (CEC).
- the camera system 114 can be connected to the display 104 (such as a television) and the camera system 114 can control the display 104 .
- the camera system 114 can be wirelessly connected to the user computing device 108 (such as a tablet).
- the camera system 114 can be wireles sly connected to a hub device and the hub device can be wirelessly connected to the user computing device 108 .
- the camera system 114 may include machine learning capabilities.
- the camera system 114 can include machine learning models.
- the machine learning models can include, but are not limited to, convolutional neural network (CNN) models and other models.
- a CNN model can be trained to extract features from images for object identification (such as person identification).
- a CNN can feed the extracted features to a recurrent neural network (RNN) for further processing.
- RNN recurrent neural network
- the camera system 114 may track movements of individuals inside the room without using any facial recognition or identification tag tracking.
- Identification tags can include, but are not limited to, badges and/or RFID tags. This feature allows the camera system 114 to track an individual's movements even when the identification of the individual is unknown. A person in the room may not be identifiable for various reasons.
- the person may be wearing a mask so that facial recognition modules may not be able to extract any features.
- the person may be a visitor who is not issued an identification tag, unlike the clinicians, who typically wear identification tags.
- the camera system 114 may combine the motion tracking with the identification of the individual to further improve accuracy in tracking the activity of the individual in the room. Having the identity of at least one person in the room may also improve accuracy in tracking the activity of other individuals in the room whose identity is unknown by reducing the number of anonymous individuals in the room. Additional details regarding machine learning capabilities and models that the camera system 114 can use are provided herein.
- the camera system 114 can be included in a monitoring system, as described herein.
- the monitoring system can include remote interaction capabilities.
- a patient in the clinical setting 101 can be in isolation due to an illness, such as COVID- 19 .
- the patient can ask for assistance via a button (such as by selecting an element in the graphical user interface on the user computing device 108 ) and/or by issuing a voice command.
- the camera system 114 can be configured to respond to voice commands, such as, but not limited to, activating or deactivating cameras or other functions.
- a remote clinician 106 can interact with the patient via the display 104 and the camera system 114 , which can include an input microphone and an output speaker.
- the monitoring system can also allow the patient to remotely maintain contact with friends and family via the display 104 and camera system 114 .
- the camera system 114 can be connected to internet of things (IOT) devices.
- IOT internet of things
- closing of the privacy switch can cause the camera system 114 and/or a monitoring system to disable monitoring.
- the monitoring system can still issue alerts if the privacy switch has been closed.
- the camera system 114 can record activity via cameras based on a trigger, such as, but not limited to, detection of motion via a motion sensor.
- FIG. 1 B is a diagram depicting a monitoring system 100 A, 100 B.
- the clinical side monitoring system 100 B can track and monitor a patient via a first camera system 114 in a clinical setting.
- the patient can be monitored via wearable sensor devices.
- a clinician 110 can interact with the patient via the first display 104 and the first camera system 114 .
- Friends and family can also use a user computing device 102 to interact with the patient via the first display 104 and the first camera system 114 .
- the home/assisted living side monitoring system 100 A can track and monitor a person (which can be an infant) via a second camera system 134 in a home/assisted living setting.
- a person can be recovering at home or live in an assisted living home.
- the person can be monitored via wearable sensor devices.
- a clinician 110 can interact with the person via the second display 124 and the second camera system 134 .
- the clinical side to the monitoring system 100 B can securely communicate with the home/assisted living side to the monitoring system 100 A, which can allow communications between the clinician 110 and persons in the home or assisted living home. Friends and family can use the user computing device 102 to interact with the patient via the second display 124 and the second camera system 134 .
- the monitoring system 100 A, 100 B can include server(s) 130 A, 130 B.
- the server(s) 130 A, 130 B can facilitate communication between the clinician 110 and a person via the second display 124 and the second camera system 134 .
- the server(s) 130 A, 130 B can facilitate communication between the user computing device 102 and the patient via the first display 104 and the first camera system 114 .
- the server(s) 130 A, 130 B can communicate with the camera system(s) 114 , 134 .
- the server(s) 130 A, 130 B can transmit machine learning model(s) to the camera system(s) 114 , 134 .
- the server(s) 130 A, 130 B can train machine learning models based on training data sets.
- the monitoring system 100 A, 100 B can present modified images (which can be in a video format) to clinician(s) or other monitoring users. For example, instead of showing actual persons, the monitoring system 100 A, 100 B can present images where a person has been replaced with a virtual representation (such as a stick figure) and/or a redacted area such as a rectangle.
- a virtual representation such as a stick figure
- FIG. 2 is a diagram depicting a monitoring system 200 in another clinical setting with an accompanying legend.
- the monitoring system 200 can include, but is not limited to, cameras 272 A, 272 B, 280 A, 280 B, 286 , 290 , 294 , displays 292 A, 292 B, 292 C, and a server 276 .
- Some of the cameras 272 A, 272 B, 280 A, 280 B, 286 , 290 , 294 can be the same as or similar to the camera system 114 of FIG. 1 A .
- the cameras 272 A, 272 B, 280 A, 280 B, 286 , 290 , 294 can send data and/or images to the server 276 .
- the server 276 can be located in the hospital room, or elsewhere in the hospital, or at a remote location outside the hospital (not illustrated). As shown, in a clinical setting, such as a hospital, hospitalized patients can be lying on hospital beds, such as the hospital bed 274 .
- the bed cameras 272 A, 272 B can be near a head side of the bed 274 facing toward a foot side of the bed 274 .
- the clinical setting may have a handwashing area 278 .
- the handwashing cameras 280 A, 280 B can face the handwashing area 278 .
- the handwashing cameras 280 A, 280 B can have a combined field of view 282 C so as to maximize the ability to detect a person's face and/or identification tag when the person is standing next to the handwashing area 278 facing the sink.
- the monitoring system 200 can detect whether the clinician (or a visitor) is within a patient zone 275 , which can be located within a field of view 282 A, 282 B of the bed camera(s) 272 A, 272 B.
- Patient zones can be customized.
- the patient zone 275 can be defined as a proximity threshold around the hospital bed 274 and/or a patient.
- the clinician 281 is within the patient zone 275 if the clinician is at least partially within a proximity threshold distance to the hospital bed and/or the patient.
- the bed cameras 272 A, 272 B can be located above a head side of the bed 274 , where the patient's head would be at when the patient lies on the bed 274 .
- the bed cameras 272 A, 272 B can be separated by a distance, which can be wider than a width of the bed 274 , and can both be pointing toward the bed 274 .
- the fields of view 282 A, 282 B of the bed cameras 272 A, 272 B can overlap at least partially over the bed 274 .
- the combined field of view 282 A, 282 B can cover an area surrounding the bed 274 so that a person standing by any of the four sides of the bed 274 can be in the combined field of view 282 A, 282 B.
- the bed cameras 272 A, 272 B can each be installed at a predetermined height and pointing downward at a predetermined angle.
- the bed cameras 272 A, 272 B can be configured so as to maximize the ability to detect the face of a person standing next to or near the bed 274 , independent of the orientation of the person's face, and/or the ability to detect an identification tag that is worn on the person's body, for example, hanging by the neck, the belt, etc.
- the bed cameras 272 A, 272 B need not be able to identify the patient lying on the bed 274 , as the identity of the patient is typically known in clinical and other settings.
- the cameras 272 A, 272 B, 280 A, 280 B, 286 , 290 , 294 can be configured, including but not limited to being installed at a height and/or angle, to allow the monitoring system 200 to detect a person's face and/or identification tag, if any.
- at least some of the cameras 272 A, 272 B, 280 A, 280 B, 286 , 290 , 294 can be installed at a ceiling of the room or at a predetermined height above the floor of the room.
- the cameras 272 A, 272 B, 280 A, 280 B, 286 , 290 , 294 can be configured to detect an identification tag.
- the cameras 272 A, 272 B, 280 A, 280 B, 286 , 290 , 294 can detect faces, which can include extracting facial recognition features of the detected face, and/or to detect a face and the identification tag substantially simultaneously.
- the monitoring system 200 can monitor one or more aspects about the patient, the clinician 281 , and/or zones.
- the monitoring system 200 can determine whether the patient is in the bed 274 .
- the monitoring system 200 can detect whether the patient is within a bed zone, which can be within the patient zone 275 .
- the monitoring system 200 can determine an angle of the patient in the bed 274 .
- the monitoring system 200 can include a wearable, wireless sensor device (not illustrated) that can track a patient's posture, orientation, and activity.
- a wearable, wireless sensor device can include, but is not limited to, a Centroid® device by Masimo Corporation, Irvine, Calif..
- the monitoring system 200 can determine how often the patient has turned in the bed 274 and/or gotten up from the bed 274 .
- the monitoring system 200 can detect turning and/or getting up based on the bed zone and/or facial recognition of the patient.
- the monitoring system 200 can detect whether the clinician 281 is within the patient zone 275 or another zone. As described herein, the monitoring system 200 can detect whether the clinician 281 is present or not present via one or more methods, such as, but not limited to, facial recognition, identification via an image of an identification tag, and/or RFID based tracking.
- the monitoring system 200 can detect intruders that are unauthorized in one or more zones via one or more methods, such as, but not limited to, facial recognition, identification via an image of an identification tag, and/or RFID based tracking.
- the monitoring system 200 can issue an alert based on one or more of the following factors: facial detection of an unrecognized face; no positive visual identification of authorized persons via identification tags; and/or no positive identification of authorized persons via RFID tags.
- the monitoring system 200 can detect falls via one or more methods, such as, but not limited to, machine-vision based fall detection and/or fall detection via wearable device, such as using accelerometer data. Any of the alerts described herein can be presented on the displays 292 A, 292 B, 292 C.
- the system 200 can assign a “contaminated” status to the clinician 281 .
- the monitoring system 200 can detect a touch action by detecting the actual act of touching by the clinician 281 and/or by detecting the clinician 281 being in close proximity, for example, within less than 1 foot, 6 inches, or otherwise, of the patient. If the clinician 281 moves outside the patient zone 275 , then the monitoring system 200 can assign a “contaminated-prime” status to the clinician 281 . If the clinician 281 with the “contaminated-prime” status re-enters the same patient zone 275 or enters a new patient zone, monitoring system 200 can output an alarm or warning. If the monitoring system 200 detects a handwashing activity by the clinician 281 with a “contaminated-prime” status, then the monitoring system 200 can assign a “not contaminated” status to the clinician 281 .
- a person may also be contaminated by entering contaminated areas other than a patient zone.
- the contaminated areas can include a patient consultation area 284 .
- the patient consultation area 284 can be considered a contaminated area with or without the presence of a patient.
- the monitoring system 200 can include a consultation area camera 286 , which has a field of view 282 D that overlaps with and covers the patient consultation area 284 .
- the contaminated areas can further include a check-in area 288 that is next to a door of the hospital room. Alternatively and/or additionally, the check-in area 288 can extend to include the door.
- the check-in area 288 can be considered a contaminated area with or without the presence of a patient.
- the monitoring system 200 can include an entrance camera 290 , which has a field of view 282 E that overlaps with and covers the check-in area 288 .
- the monitoring system 200 can include an additional camera 294 .
- Additional cameras may not be directed to any specific contaminated and/or handwashing areas.
- the additional camera 294 can have a field of view 282 F that covers substantially an area that a person likely has to pass when moving from one area to another area of the hospital room, such as from the patient zone 275 to the consultation area 284 .
- Additional camera can provide data to the server 276 to facilitate tracking of movements of the people in the room.
- FIG. 3 depicts a monitoring system 300 in another clinical setting.
- the monitoring system 300 may monitor the activities of anyone present in the room such as medical personnel, visitors, patients, custodians, etc.
- the monitoring system 300 may be located in a clinical setting such as a hospital room.
- the hospital room may include one or more patient beds 308 .
- the hospital room may include an entrance/exit 329 to the room.
- the entrance/exit 329 may be the only entrance/exit to the room.
- the monitoring system 300 can include a server 322 , a display 316 , one or more camera systems 314 , 318 , 320 , and an additional device 310 .
- the camera systems 314 , 318 , 320 may be connected to the server 322 .
- the server 322 may be a remote server.
- the one or more camera systems may include a first camera system 318 , a second camera system 320 , and/or additional camera systems 314 .
- the camera systems 314 , 318 , 320 may include one or more processors, which can include one or more hardware accelerators.
- the processors can be enclosed in an enclosure 313 , 324 , 326 of the camera systems 314 , 318 , 320 .
- the processors can include, but are not limited to, an embedded processing unit, such as an Nvidia® Jetson XavierTM NX/AGX, that is embedded in an enclosure of the camera systems 314 , 318 , 320 .
- the one or more processors may be physically located outside of the room.
- the processors may include microcontrollers such as, but not limited to, ASICs, FPGAs, etc.
- the camera systems 314 , 318 , 320 may each include a camera.
- the camera(s) may be communication with the one or more processors and may transmit image data to the processor(s).
- the camera systems 314 , 318 , 320 can exchange data and state information with other camera systems.
- the monitoring system 300 may include a database.
- the database can include information relating to the location of items in the room such as camera systems, patient beds, handwashing stations, and/or entrance/exits.
- the database can include locations of the camera systems 314 , 318 , 320 and the items in the field of view of each camera system 314 , 318 , 320 .
- the database can further include settings for each of the camera systems.
- Each camera system 314 , 318 , 320 can be associated with an identifier, which can be stored in the database.
- the server 322 may use the identifiers to configure each of the camera systems 314 , 318 , 320 .
- the first camera system 318 can include a first enclosure 324 and a first camera 302 .
- the first enclosure 324 can enclose one or more hardware processors.
- the first camera 302 may be a camera capable of sensing depth and color, such as, but not limited to, an RGB-D stereo depth camera.
- the first camera 302 may be positioned in a location of the room to monitor the entire room or substantially all of the room.
- the first camera 302 may be tilted downward at a higher location in the room.
- the first camera 302 may be set up to minimize blind spots in the field of view of the first camera 302 .
- the first camera 302 may be located in a corner of the room.
- the first camera 302 may be facing the entrance/exit 329 and may have a view of the entrance/exit 329 of the room.
- the second camera system 320 can include a second enclosure 326 (which can include one or more processors) and a second camera 304 .
- the second camera 304 may be a RGB color camera.
- the second camera 304 may be an RGB-D stereo depth camera.
- the second camera 304 may be installed over a hand hygiene compliance area 306 .
- the hand hygiene compliance area 306 may include a sink and/or a hand sanitizer dispenser.
- the second camera 304 may be located above the hand hygiene compliance area 306 and may be point downwards toward the hand hygiene compliance area 306 .
- the second camera 304 may be located on or close to the ceiling and may have a view the hand hygiene compliance area 306 from above.
- the first and second camera systems 318 , 320 may be sufficient for monitoring the room.
- the system 300 may include any number of additional camera systems, such as a third camera system 314 .
- the third camera system 314 may include a third enclosure 313 (which can include one or more processors) and a third camera 312 .
- the third camera 312 of the third camera system 314 may be located near the patient's bed 308 or in a corner of the room, for example, a corner of the room that is different than (for example, opposite or diagonal to) the corner of the room where the first camera 302 of the first camera system 318 is located.
- the third camera 312 may be located at any other suitable location of the room to aid in reducing blind spots in the combined fields of view of the first camera 302 and the second camera 304 .
- the third camera 312 of the third camera system 314 may have a field of view covering the entire room.
- the third camera system 314 may operate similarly to the first camera system 318 , as described herein.
- the monitoring system 300 may include one or more additional devices 310 .
- the additional device 310 can be, but is not limited to, a patient monitoring and connectivity hub, bedside monitor, or other patient monitoring device.
- the additional device 310 can be a Root® monitor by Masimo Corporation, Irvine, Calif..
- the additional device 310 can be, but is not limited to, a display device of a data aggregation and/or alarm visualization platform.
- the additional device 310 can be a display device (not illustrated) for the Uniview® platform by Masimo Corporation, Irvine, Calif.
- the additional device(s) 310 can include smartphones or tablets (not illustrated).
- the additional device(s) may be in communication with the server 322 and/or the camera systems 318 , 320 , 314 .
- the monitoring system 300 can output alerts on the additional device(s) 310 and/or the display 316 .
- the outputted alert may be any auditory and/or visual signal.
- Outputted alerts can include, but are not limited to, a fall alert, an unauthorized person alert, an alert that a patient should be turned, or an alert that a person has not complied the hand hygiene protocol. For example, someone outside of the room can be notified on an additional device 310 and/or the display 316 that an emergency has occurred in the room.
- the monitoring system 300 can provide a graphical user interface, which can be presented on the display 316 . A configuration user can configure the monitoring system 300 via the graphical user interface presented on the display 316 .
- FIG. 4 depicts patient sensor devices 404 , 406 , 408 (such as a wearable device) and a user computing device 402 (which may not be drawn to scale) that can be used in a monitoring system.
- patient sensor devices 404 , 406 , 408 can be optionally used in a monitoring system.
- patient sensor devices can be used with the monitoring system that are different than the devices 404 , 406 , 408 depicted in FIG. 4 .
- a patient sensor device can non-invasively measure physiological parameters from a fingertip, wrist, chest, forehead, or other portion of the body.
- the first, second, and third patient sensor devices 404 , 406 , 408 can be wirelessly connected to the user computing device 402 and/or a server in the monitoring system.
- the first patient sensor device 404 can include a display and a touchpad and/or touchscreen.
- the first patient sensor device 404 can be a pulse oximeter that is designed to non-invasively monitor patient physiological parameters from a fingertip.
- the first patient sensor device 404 can measure physiological parameters such as, but not limited to, blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, and/or pleth variability index.
- the first patient sensor device 404 can be a MightyS at® fingertip pulse oximeter by Masimo Corporation, Irvine, Calif.
- the second patient sensor device 406 can be configured to be worn on a patient's wrist to non-invasively monitor patient physiological parameters from a wrist.
- the second patient sensor device 406 can be a smartwatch.
- the second patient sensor device 406 can include a display and/or touchscreen.
- the second patient sensor device 406 can measure physiological parameters including, but not limited to, blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, and/or pleth variability index.
- the third patient sensor device 408 can be a temperature sensor that is designed to non-invasively monitor physiological parameters of a patient.
- the third patient sensor device 408 can measure a temperature of the patient.
- the third patient sensor device 408 can be a Radius T°TM sensor by Masimo Corporation, Irvine, Calif. A patient, clinician, or other authorized user can use the user computing device 408 to view physiological information and other information from the monitoring system.
- a graphical user interface can be presented on the user computing device 402 .
- the graphical user interface can present physiological parameters that have been measured by the patient sensor devices 404 , 406 , 408 .
- the graphical user interface can also present alerts and information from the monitoring system.
- the graphical user interface can present alerts such as, but not limited to, a fall alert, an unauthorized person alert, an alert that a patient should be turned, or an alert that a person has not complied the hand hygiene protocol.
- FIG. 5 illustrates a camera image 500 with object tracking.
- the monitoring system can track the persons 502 A, 502 B, 502 C in the camera image 500 with the boundary regions 504 , 506 , 508 .
- each camera system in a monitoring system can be configured to perform object detection.
- some monitoring systems can have a single camera system while other monitoring systems can have multiple camera systems.
- Each camera system can be configured with multiple machine learning models for object detection.
- a camera system can receive image data from a camera. The camera can capture a sequence of images (which can be referred to as frames).
- the camera system can process the frame with a YOLO (You Only Look Once) deep learning network, which can be trained to detect objects (such as persons 502 A, 502 B, 502 C) and return coordinates of the boundary regions 504 , 506 , 508 .
- the camera system can process the frame with an inception CNN, which can be trained to detect activities, such as hand sanitizing or hand washing (not illustrated).
- the machine learning models, such as the inception CNN can be trained using a dataset of a particular activity type, such as handwashing or hand sanitizing demonstration videos, for example.
- the camera system can determine processed data that consists of the boundary regions 504 , 506 , 508 surrounding a detected person 502 A, 502 B, 502 C in the room, such as coordinates of the boundary regions.
- the camera system can provide the boundary regions to a server in the monitoring system.
- processed data may not include the images captured by the camera.
- the images from the camera can be processed locally at the camera system and may not be transmitted outside of the camera system.
- the monitoring system can ensure anonymity and protect privacy of imaged persons by not transmitting the images outside of each camera system.
- the camera system can track objects using the boundary regions.
- the camera system can compare the intersection of boundary regions in consecutive frames.
- a sequence of boundary regions associated with an object through consecutive frames can be referred to as a “track.”
- the camera system may associate boundary regions if the boundary regions of consecutive frames overlap by a threshold distance or are within of a threshold distance of another.
- the camera system may determine that boundary regions from consecutive frames that are adjacent (or the closest with each other) are associated with the same object. Thus, whenever object detection occurs in the field of view of one camera, that object may be associated with the nearest track.
- the camera system can use one or more computer vision algorithms.
- a computer vision algorithm can identify a boundary region around a person's face or around a person's body.
- the camera system can detect faces using a machine learning model, such as, but not limited to, Google's FaceNet.
- the machine learning model can receive an image of the person's face as input and output a vector of numbers, which can represent features of a face.
- the camera system can send the extracted facial features to the server.
- the monitoring system can map the extracted facial features to a person.
- the vector numbers can represent facial features corresponding to points on ones' face.
- Facial features of known people can be stored in a facial features database, which can be part of the database described herein.
- the monitoring system can initially mark the unknown person as unknown and subsequently identify the same person in multiple camera images.
- the monitoring system can populate a database with the facial features of the new person.
- FIG. 6 depicts a monitoring system 600 in a home setting.
- the monitoring system 600 can include, but is not limited to, one or more cameras 602 , 604 , 606 . Some of the cameras, such as a first camera 602 of the monitoring system 600 , can be the same as or similar to the camera system 114 of FIG. 1 A . In some aspects, the cameras 602 , 604 , 606 can send data and/or images to a server (not illustrated).
- the monitoring system 600 can be configured to detect a pet 610 using the object identification techniques described herein. The monitoring system 600 can be further configured to determine if a pet 610 was fed or if the pet 610 is chewing or otherwise damaging the furniture 612 .
- the monitoring system 600 can be configured to communicate with a home automation system. For example, if the monitoring system 600 detects that the pet 610 is near a door, the monitoring system 600 can instruct the home automation system to open the door. In some aspects, the monitoring system 600 can provide alerts and/or commands in the home setting to deter a pet from some activity (such as biting a couch, for example).
- FIG. 7 depicts a monitoring system 700 in an infant care setting.
- the monitoring system 700 can include one or more cameras 702 .
- a camera in the monitoring system 700 can send data and/or images to a server (not illustrated).
- the monitoring system 700 can be configured to detect an infant 704 using the object identification techniques described herein. Via a camera, the monitoring system 700 can detect whether a person is within an infant zone, which can be located within a field of view of the camera 702 .
- Infant zones can be similar to patient zones, as described herein. For example, an infant zone can be defined as a proximity threshold around a crib 706 and/or the infant 704 .
- a person is within the infant zone if the person is at least partially within a proximity threshold distance to the crib 706 and/or the infant 704 .
- the monitoring system 700 can use object tracking, as described herein, to determine if the infant 704 is moved. For example, the monitoring system 700 can issue an alert if the infant 704 leaves the crib 706 .
- the monitoring system 700 can include one or more machine learning models.
- the monitoring system 700 can detect whether an unauthorized person is within the infant zone.
- the monitoring system 700 can detect whether an unauthorized person is present using one or more methods, such as, but not limited to, facial recognition, identification via an image of an identification tag, and/or RFID based tracking.
- Identification tag tracking (whether an identification badge, RFID tracking, or some other tracking) can be appliable to hospital-infant settings.
- the monitoring system 700 can issue an alert based on one or more of the following factors: facial detection of an unrecognized face; no positive visual identification of authorized persons via identification tags; and/or no positive identification of authorized persons via RFID tags.
- a machine learning model of the monitoring system 700 can receive an image of a person's face as input and output a vector of numbers, which can represent features of a face.
- the monitoring system 700 can map the extracted facial features to a known person.
- a database of the monitoring system 700 can store a mapping from facial features (but not actual pictures of faces) to person profiles. If the monitoring system 700 cannot match the features to features from a known person, the monitoring system 700 can mark person as unknown and issue an alert. Moreover, the monitoring system 700 can issue another alert if the unknown person moves the infant 704 outside of a zone.
- the monitoring system 700 can monitor movements of the infant 704 .
- the monitoring system 700 can monitor a color of the infant for physiological concerns. For example, the monitoring system can detect a change in color of skin (such as a bluish color) since that might indicate potential asphyxiation.
- the monitoring system 700 can use trained machine learning models to identify skin color changes.
- the monitoring system 700 can detect a position of the infant 704 . For example, if the infant 704 rolls onto their stomach, the monitoring system 700 can issue a warning since it may be safer for the infant 704 to lay on their back.
- the monitoring system 700 can use trained machine learning models to identify potentially dangerous positions.
- a non-invasive sensor device can be attached to the infant 704 (such as a wristband or a band that wraps around the infant's foot) to monitor physiological parameters of the infant.
- the monitoring system 700 can receive the physiological parameters, such as, but not limited to, blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, and/or pleth variability index.
- the monitoring system 700 can include a microphone that can capture audio data.
- the monitoring system 700 can detect sounds from the infant 704 , such as crying.
- the monitoring system 700 can issue an alert if the detected sounds are above a threshold decibel level. Additionally or alternatively, the monitoring system 700 can process the sounds with a machine learning model.
- the monitoring system 700 can convert sound data into spectrograms, input them into a CNN and a linear classifier model, and output a prediction whether the sounds (such as excessive crying) should cause a warning to be issued.
- the monitoring system 700 can include a thermal camera.
- the monitoring system 700 can use trained machine learning models to identify a potentially wet diaper from an input thermal image.
- FIG. 8 is a flowchart of a method 800 for efficiently applying machine learning models, according to some aspects of the present disclosure.
- a monitoring system which can include a camera system, may implement aspects of the method 800 as described herein.
- the method 800 may include fewer or additional blocks and/or the blocks may be performed in order different than is illustrated.
- image data can be received.
- a camera system (such as the camera systems 114 , 318 of FIGS. 1 A, 3 described herein) can receive image data from a camera. Depending on the type of camera and configuration of the camera, the camera system can receive different types of images, such as 4K, 1080p, 8MP images. Image data can also include, but is not limited to, a sequence of images.
- a camera in a camera system can continuously capture images. Therefore, the camera in a camera system can capture images of objects (such as a patient, a clinician, an intruder, the elderly, an infant, a youth, or a pet) in a room either at a clinical facility, a home, or an assisted living home.
- a person detection model can be applied.
- the camera system can apply the person detection model based on the image data.
- the camera system can invoke the person detection model on a hardware accelerator.
- the hardware accelerator can be configured to accelerate the application of machine learning models, including a person detection model.
- the person detection model can be configured to receive image data as input.
- the person detection model can be configured to output a classification result.
- the classification result can indicate a likelihood (such as a percentage chance) that the image data includes a person.
- the classification result can be a binary result: either the object is predicted as present in the image or not.
- the person detection model can be, but is not limited to, a CNN.
- the person detection model can be trained to detect persons. For example, the person detection model can be trained with a training data set with labeled examples indicating whether the input data includes a person or not.
- the camera system can determine whether a person is present.
- the camera system can determine whether a person object is located in the image data.
- the camera system can receive from the person detection model (which can execute on the hardware accelerator) the output of a classification result.
- the output can be a binary result, such as, “yes” there is a person object present or “no” there is not a person object present.
- the output can be a percentage result and the camera system can determine the presence of a person if the percentage result is above a threshold. If a person is detected, the method 800 proceeds to the block 810 to receive second image data. If a person is not detected, the method 800 proceeds to repeat the previous blocks 802 , 806 , 808 to continue checking for persons.
- second image data can be received.
- the block 810 for receiving the second image data can be similar to the previous block for receiving image data.
- the camera in the camera system can continuously capture images, which can lead to the second image data.
- the image data can include multiple images, such as a sequence of images.
- one or more person safety models can be applied.
- the camera system can apply one or more person safety models.
- the camera system can invoke (which can be invoked on a hardware accelerator) a fall detection model based on the second image data.
- the fall detection model can output a classification result.
- the fall detection model can be or include a CNN.
- the camera system can pre-process the image data.
- the camera system can covert an image into an RGB image, which can be a m-by-n-by- 3 data array that defines red, green, and blue color components for each individual pixel in the image.
- the camera system can compute an optical flow from the image data (such as the RGB images), which can be a two-dimensional vector field between two images.
- the two-dimensional vector field can show how the pixels of an object in the first image move to form the same object in the second image.
- the fall detection model can be pre-trained to perform feature extraction and classification of the image data (which can be pre-processed image data) to output a classification result.
- the fall detection model can be made of various layers, such as, but not limited to, a convolution layer, a max pooling layer, and a regularization layer, and a classifier, such as, but not limited to, a softmax classifier.
- an advantage of performing the previous blocks 802 , 806 , 808 for checking whether a person is present is that more computationally expensive operations, such as applying one or more person safety models, can be delayed until a person is detected.
- the camera system can invoke (which can be invoked on a hardware accelerator) multiple person safety models based on the second image data. For each person safety model that is invoked, the camera system can receive a model result, such as but not limited to, a classification result.
- the person safety models can include a fall detection model, a handwashing detection model, and/or an intruder detection model.
- the camera system can determine whether there is a person safety issue.
- the camera system can receive a model result as output.
- the output can be a binary result, such as, “yes” a fall has been detected or “no” a fall has not been detected.
- the output can be a percentage result and the camera system can determine a person safety issue exists if the percentage result is above a threshold.
- evaluation of the one or more person safety models can result in an issue detection if at least one model returns a result that indicates issue detection. If a person safety issue is detected, the method 800 proceeds to block 816 to provide an alert and/or take an action. If a person safety issue is not detected, the method 800 proceeds to repeat the previous blocks 802 , 806 , 808 to continue checking for persons.
- an alert can be provided and/or an action can be taken.
- the camera system can initiate an alert.
- the camera system can notify a monitoring system to provide an alert.
- a user computing device 102 can receive an alert about a safety issue.
- a clinician 110 can receive an alert about the safety issue.
- the camera system can initiate an action.
- the camera system can cause the monitoring system to take an action. For example, the monitoring system can automatically notify emergency services (such as an emergency hotline and/or an ambulance service) to send someone to help.
- emergency services such as an emergency hotline and/or an ambulance service
- FIG. 9 is a flowchart of another method 900 for efficiently applying machine learning models, according to some aspects of the present disclosure.
- a monitoring system which can include a camera system, may implement aspects of the method 900 as described herein.
- the method 900 may include fewer or additional blocks and/or the blocks may be performed in order different than is illustrated.
- the block(s) of the method 900 of FIG. 9 can be similar to the block(s) of the method 800 of FIG. 8 .
- the block(s) of the method 900 of FIG. 9 can be used in conjunction with the block(s) of the method 800 of FIG. 8 .
- a training data set can be received.
- the monitoring system can receive a training data set.
- a first set of videos of person falls can be collected and a second set of videos of persons without falling can be collected.
- a training data set can be created with the first set of videos and the second set of videos.
- Other training data sets can be created. For example, for machine learning of handwashing, a first set of videos of with handwashing and a second set of videos without handwashing can be collected; and a training data set can be created from the first set of videos and the second set of videos.
- a first set of images of with dilated pupils and a second set of images without dilated pupils can be collected; and a training data set can be created from the first set of images and the second set of images.
- a training data set can be created from the first set of images and the second set of images.
- a first set of images of with facial paralysis and a second set of images without facial paralysis can be collected; and a training data set can be created from the first set of images and the second set of images.
- a first set of images of with an infant and a second set of images without an infant can be collected; and a training data set can be created from the first set of images and the second set of images.
- a training data set can be created from the first set of images and the second set of images.
- a first set of videos of persons in an unconscious state and a second set of videos of a person in a state of consciousness; and a training data set can be created from the first set of videos and the second set of videos.
- a first set of audio recordings of persons in an unconscious state and a second set of audio recordings of a person in a state of consciousness; and a training data set can be created from the first set of audio recordings and the second set of audio recordings.
- the monitoring system can receive training data sets for any of the machine learning models described herein that can be trained with supervised machine learning.
- a machine learning model can be trained.
- the monitoring system can train one or more machine learning models.
- the monitoring system can train a fall detection model using the training data set from the previous block 902 .
- the monitoring system can train a handwashing detection model using the training data set from the previous block 902 .
- the monitoring system can train any of the machine learning models described herein that use supervised machine learning.
- the monitoring system can train a neural network, such as, but not limited to, a CNN.
- the monitoring system can initiate the neural network with random weights.
- the monitoring system feeds labelled data from the training data set to the neural network.
- Class labels can include, but are not limited to, fall, no fall, hand washing, no hand washing, loud noise, no loud noise, normal pupils, dilated pupils, no facial paralysis, facial paralysis, infant, no infant, supine position, prone position, side position, unconscious, conscious, etc.
- the neural network can process each input vector with its values being assigned randomly and then make comparisons with the class label of the input vector.
- an adjustment to the weights of the neural network neurons are made so that output correctly matches the class label.
- the corrections to the value of weights can be made through a technique, such as, but not limited to backpropagation.
- Each run of training of the neural network can be called an “epoch.”
- the neural network can go through several series of epochs during the process of training, which results in further adjusting of the neural network weights. After each epoch step, the neural network can become more accurate at classifying and correctly predicting the class of the training data.
- the monitoring system can use a test dataset to verify the neural network's accuracy.
- the test dataset can be a set of labelled test data that were not included in the training process.
- Each test vector can be fed to the neural network, and the monitoring system can compare the output to the actual class label of the test input vector.
- input data can be received.
- the camera system can receive input data.
- the block 906 for receiving input data can be similar to the block 802 of FIG. 8 for receiving image data.
- the camera system can receive image data from a camera.
- other input data can be received.
- the camera system can receive a current time.
- the camera system can receive an RFID signal (which can be used for identification purposes, as described herein).
- the camera system can receive physiological values (such as blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, and/or pleth variability index) from a patient sensor device, such as a wearable device.
- the camera system can determine whether a trigger has been satisfied to apply one or more machine learning models. In some aspects, the camera system can determine whether a trigger has been satisfied by checking whether a person has been detected. In some aspects, the camera system can determine whether a trigger has been satisfied by checking whether the current time satisfies a trigger time window, such as, but not limited to, a daily time check-up window. If a trigger is satisfied, the method 900 proceeds to the block 910 to receive captured data. If a trigger is not detected, the method 900 proceeds to repeat the previous blocks 906 , 908 to continue checking for triggers.
- a trigger time window such as, but not limited to, a daily time check-up window
- a trigger can be determined based on a received physiological value.
- the camera system can determine to begin a monitoring process based on a physiological value.
- the wearable device can include a pulse oximetry sensor and the physiological value is for blood oxygen saturation.
- the camera system can determine that the physiological value is below a threshold level (such as blood oxygen below 88%, 80%, or 70%, etc.).
- the wearable device can include a respiration rate sensor and the physiological value is for respiration rate.
- the camera system can determine that the physiological value satisfies a threshold alarm level (such as respiration rate under 12 or over 25 breaths per minute).
- the wearable device can include a heart rate sensor, the physiological value is for heart rate, and the multiple physiological values measuring heart rate over time can be received from the wearable device.
- the camera system can determine that the physiological values satisfies a threshold alarm level, such as, but not limited to, heart rate being above 100 beats per minute for a threshold period of time or under a threshold level for threshold period of time.
- captured data can be received.
- the block 910 for receiving captured data can be similar to the previous block 906 for receiving input data.
- the camera in the camera system can continuously capture images, which can lead to the captured data.
- the camera system can receive audio data from a microphone.
- the camera system can be configured to cause presentation, on a display, of a prompt to cause a person to perform an activity.
- the camera system can receive, from a camera, image data of a recording of the activity.
- one or more machine learning models can be applied.
- the camera system can apply one or more machine learning models based on the captured data.
- the camera system can invoke (which can be invoked on a hardware accelerator) one or more machine learning models, which can output a model result.
- the camera system can invoke a fall detection model based on image data where the fall detection model can output a classification result.
- the camera system can invoke a loud noise detection model based on the audio data where the loud noise detection model can output a classification result.
- the camera system can generate a spectrogram data from the audio data and provide the spectrogram data as input to the loud noise detection model.
- the camera system can invoke a facial feature extraction model based on the image data where the facial feature extraction model can output a facial feature vector.
- the camera system can invoke a handwashing detection model based on the image data where the handwashing detection model can output a classification result.
- the camera system can invoke a screening machine learning model based on image data where the screening machine learning model can output a model result.
- the screening machine learning model can include, but is not limited to, a pupillometry screening model or a facial paralysis screening model.
- the camera system in response to determining to begin the monitoring process, can invoke one or more machine learning models.
- the camera system can invoke (which can be on a hardware accelerator) a first unconscious detection model based on the image data where the first unconscious detection model outputs a first classification result.
- the camera system can invoke (which can be on the hardware accelerator) a second unconscious detection model based on the audio data where the second unconscious detection model outputs a second classification result.
- the camera system can determine whether there is a safety issue.
- the camera system can receive a classification result as output.
- the output can be a binary result, such as, “yes” a fall has been detected or “no” a fall has not been detected.
- the output can be a percentage result and the camera system can determine a safety issue exists if the percentage result is above a threshold.
- evaluation of the one or more machine learning models can result in an issue detection if at least one model returns a result that indicates issue detection.
- the camera system can detect a potential fall based on the classification result.
- the camera system can detect a potential scream or loud noise based on the classification result from a loud noise detection model.
- the camera system can execute a query of a facial features database based on the facial feature vector where executing the query can indicate that the facial feature vector is not present in a facial features database, which can indicate a safety issue.
- the camera system can detect a potential screening issue based on the classification result.
- the potential screening issue can indicate, but is not limited to, potential dilated pupils or potential facial paralysis.
- the camera system can detect a potential state of unconsciousness. If a safety issue is detected, the method 900 proceeds to block 916 to provide an alert and/or take an action. If a safety issue is not detected, the method 900 proceeds to repeat the previous blocks 906 , 908 to continue checking for triggers.
- an alert can be provided and/or an action can be taken.
- the camera system can initiate an alert.
- the camera system can notify a monitoring system to provide an alert.
- the camera system can initiate an action.
- the block 916 for providing an alert and/or taking an action can be similar to the block 816 of FIG. 8 for providing an alert and/or taking an action.
- the monitoring system can provide an alert.
- the monitoring system can escalate alerts.
- the monitoring system in response to detecting a potential fall and a potential scream or loud noise, can provide an escalated alert.
- the camera system can cause the monitoring system to take an action.
- the monitoring system can automatically notify emergency services (such as an emergency hotline and/or an ambulance service) to send someone to help.
- emergency services such as an emergency hotline and/or an ambulance service
- the monitoring system can allow privacy options. For example, some user profiles can specify that the user computing devices associated with those profiles should not receive alerts (which can be specified for a period of time). However, the monitoring system can include an alert escalation policy such that alerts can be presented via user computing devices based on one or more escalation conditions. For example, if an alert isn't responded to a for a period of time, the monitoring system can escalate the alert. As another example, if a quantity of alerts exceed a threshold, then the monitoring system can present an alert via user computing devices despite user preferences otherwise.
- a communications system can be provided.
- the monitoring system can provide a communications system.
- the camera system can receive, from a computing device, first video data (such as, but not limited to, video data of a clinician, friends, or family of a patient).
- the camera system can cause presentation, on the display, of the first video data.
- the camera system can receive, from the camera, second video data and transmit, to the computing device, the second video data.
- the monitoring systems described herein can be applied to assisted living and/or home settings for the elderly.
- the monitoring systems described herein which can include camera systems, can generally monitor activities of the elderly.
- the monitoring systems described herein can initiate check-up processes, including, but not limited to, dementia checks.
- a check-up process can detect a color of skin to detect possible physiological changes.
- the monitoring system can perform stroke detection by determining changes in facial movements and/or speech patterns.
- the monitoring system can track medication administration and provide reminders if medication is not taken. For example, the monitoring can monitor a cupboard or medicine drawer and determine whether medication is taken based on activity in those areas.
- some of the camera systems can be outdoor camera systems.
- the monitoring system can track when a person goes for a walk, log when the person leaves and returns, and potentially issues an alert if a walk exceeds a threshold period of time.
- the monitoring system can track usage of good hygiene practices, such as but not limited to, handwashing, brushing teeth, or showering (e.g., tracking that a person enters a bathroom at a showering time).
- the monitoring system can keep track of whether a person misses a check-up.
- a camera system can include a thermal camera, which can be used to identify a potentially wet adult diaper from an input thermal image.
- a training data set can be received.
- the monitoring system can receive a training data set, which can be used to train machine learning models to be used in check-up processes for the elderly, such as checking for dilated pupils or facial paralysis.
- a training data set can be created from the first set of images and the second set of images.
- a training data set can be created from the first set of images and the second set of images.
- a machine learning model can be trained.
- a server in the monitoring system can train a pupillometry screening model using the training data set.
- the server in the monitoring system can train a facial paralysis screening model using the training data set.
- input data can be received.
- the camera system can receive input data, which can be used to determine if a trigger has been satisfied for application of one or more machine learning models.
- the camera system can receive image data from a camera.
- the camera system can receive a current time.
- the camera system can receive an RFID signal, which can be used for person identification and/or detection.
- the monitoring system can include patient sensor devices, such as, but not limited to, wearable devices.
- the wearable device can be configured to process sensor signals to determine a physiological value for the person.
- the monitoring system can receive a physiological value from the wearable device.
- the wearable device can include a pulse oximetry sensor and the physiological value can be for blood oxygen saturation.
- the wearable device can be configured to process the sensor signals to measure at least one of blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, or pleth variability index. Some of the wearable devices can be used for an infant.
- the camera system can determine whether a trigger has been satisfied to apply one or more machine learning models.
- the camera system can determine whether a check-up process should begin from a current time. For example, the monitoring system can conduct check-up processes at regular intervals, such as once or two a day, which can be at particular times, such as a morning check-up time or an afternoon check-up time.
- another trigger type can be detection of a person.
- the camera system can invoke a person detection model based on image data where the person detection model outputs classification result; and detect a person based on the classification result. If a trigger is satisfied, the method 900 proceeds to the block 910 to receive captured data. If a trigger is not detected, the method 900 proceeds to repeat the previous blocks 906 , 908 to continue checking for triggers.
- captured data can be received.
- the monitoring system can cause presentation, on a display, of a prompt to cause a person to perform a check-up activity.
- the check-up activity can check for signs of dementia.
- a check-up activity can include having a person standing a particular distance from the camera system.
- a check-up activity can include simple exercises.
- the prompts can cause a user to say something or perform tasks.
- the person can be prompted to perform math tasks, pattern recognition, solve puzzles, and/or identify photos of family members. For example, the person can be prompted to point to sections of the display, which can correspond to answers to check-up tests.
- the check-up tests can check for loss of motor skills.
- the check-up activity can include a virtual physical or appointment conducted by a clinician.
- the camera system can receive, from the camera, image data of a recording of the check-up activity.
- the camera system can receive other input, such as, but not limited to, audio data from a microphone.
- one or more machine learning models can be applied.
- the camera system can apply one or more machine learning models based on the captured data.
- the camera system in response to determining to begin the check-up process, can invoke a screening machine learning model based on image data where the screening machine learning model can output a model result (such as a classification result).
- the screening machine learning model can include, but is not limited to, a pupillometry screening model, a facial paralysis screening model, or a gesture detection model.
- the gesture detection model can be configured to detect a gesture directed towards a portion of the display.
- the person can be prompted to point to a portion of the display and the gesture detection model can identify a point gesture, such as but not limited to, pointing to a quadrant on the display.
- the camera system in response to detecting a person, can invoke a handwashing detection model based on image data wherein the handwashing detection model outputs a classification result.
- the camera system can determine whether there is a safety issue, such as a potential screening issue.
- the camera system can detect a potential screening issue based on the model result.
- the potential screening issue can indicate, but is not limited to, potential dilated pupils or potential facial paralysis.
- the monitoring system can determine whether there is a potential screening issue based on output from a gesture detection model. For example, the monitoring system can use detected gesture to determine an answer and an incorrect answer can indicate a potential screening issue. If a safety issue is detected, the method 900 proceeds to block 916 to provide an alert and/or take an action. If a safety issue is not detected, the method 900 proceeds to repeat the previous blocks 906 , 908 to continue checking for triggers.
- an alert can be provided.
- the camera system can initiate an alert.
- the camera system can notify a monitoring system to provide one or more alerts.
- the monitoring system can provide an alert.
- the monitoring system can escalate alerts. For example, in response to detecting a potential fall and a potential scream or loud noise, the monitoring system can provide an escalated alert.
- the monitoring system can provide alerts via different networks (such as Wi-Fi or cellular) and/or technologies (such as Bluetooth).
- a communications system can be provided.
- the monitoring system can provide a communications system in an elderly care setting.
- the camera system can receive, from a computing device, first video data (such as, but not limited to, video data of a clinician, friends, or family of a patient).
- the camera system can cause presentation, on the display, of the first video data.
- the camera system can receive, from the camera, second video data and transmit, to the computing device, the second video data.
- FIG. 10 is a flowchart of a method 1000 for efficiently applying machine learning models for infant care, according to some aspects of the present disclosure.
- a monitoring system which can include a camera system, may implement aspects of the method 1000 as described herein.
- the block(s) of the method 1000 of FIG. 10 can be similar to the block(s) of the methods 800 , 900 of FIGS. 8 and/or 9 .
- the method 1000 may include fewer or additional blocks and/or the blocks may be performed in order different than is illustrated.
- image data can be received.
- a camera system can receive image data from a camera, which can be positioned in an infant area, such as a nursery.
- Image data can also include, but is not limited to, a sequence of images.
- a camera in a camera system can continuously capture images of the infant area. Therefore, the camera in a camera system can capture images of objects, such as an infant, in a room either at a home or a clinical facility.
- an infant detection model can be applied.
- the camera system can apply the infant detection model based on the image data.
- the camera system can invoke the infant detection model on a hardware accelerator.
- the infant detection model can be configured to receive image data as input.
- the infant detection model can be configured to output a classification result.
- the classification result can indicate a likelihood (such as a percentage chance) that the image data includes an infant.
- the classification result can be a binary result: either the infant object is predicted as present in the image or not.
- the infant detection model can be, but is not limited to, a CNN.
- the infant detection model can be trained to detect persons. For example, the infant detection model can be trained with a training data set with labeled examples indicating whether the input data includes an infant or not.
- the camera system can determine whether an infant is present.
- the camera system can determine whether an infant object is located in the image data.
- the camera system can receive from the infant detection model the output of a classification result.
- the output can be a binary result, such as, “yes” there is an infant object present or “no” there is not an infant object present.
- the output can be a percentage result and the camera system can determine the presence of an infant if the percentage result is above a threshold. If an infant is detected, the method 1000 proceeds to the block 1010 to receive captured data. If an infant is not detected, the method 1000 proceeds to repeat the previous blocks 1002 , 1006 , 1008 to continue checking for infants.
- captured data can be received.
- the camera in the camera system can continuously capture images, which can lead to the captured data.
- the camera system can receive audio data from a microphone.
- one or more infant safety models can be applied.
- the camera system can apply one or more infant safety models that outputs a model result.
- the camera system can invoke (which can be invoked on a hardware accelerator) an infant position model based on the captured data.
- the infant position model can output a classification result.
- the infant position model can be or include a CNN.
- the camera system can invoke a facial feature extraction model based on second image data where the facial feature extraction model outputs a facial feature vector.
- the camera system can execute a query of a facial features database based on the facial feature vector where executing the query indicates that the facial feature vector is not present in the facial features database.
- An infant safety model can be an infant color detection model.
- the model result can include coordinates of a boundary region identifying an infant object in the image data.
- the camera system can invoke a loud noise detection model based on the audio data where the loud noise detection model can output a classification result.
- the camera system can determine whether there is an infant safety issue.
- the camera system can receive a model result as output.
- the output can be a binary result, such as, “yes” an infant is in a supine position or “no” a supine position has not been detected (such as the infant potentially laying on their stomach).
- the output can be a percentage result and the camera system can determine an infant safety issue exists if the percentage result is above a threshold. The camera system can determine that an unrecognized person has been detected.
- the camera system determine that the coordinates of the boundary region exceed a threshold distance from an infant zone (which can indicate that an infant is being removed from the infant zone).
- the camera system can determine a potential scream from the model result.
- evaluation of the one or more infant safety models can result in an issue detection if at least one model returns a result that indicates issue detection. If an infant safety issue is detected, the method 1000 proceeds to block 1016 to provide an alert and/or take an action. If an infant safety issue is not detected, the method 1000 proceeds to repeat the previous blocks 1002 , 1006 , 1008 to continue checking for infants.
- an alert can be provided and/or an action can be taken.
- the camera system can initiate an alert associated with the infant.
- the camera system can notify a monitoring system to provide an alert.
- a user computing device 102 can receive an alert about an infant safety issue.
- a clinician 110 can receive an alert about the infant safety issue.
- the camera system can initiate an action.
- the camera system can cause the monitoring system to take an action. For example, the monitoring system can automatically notify emergency services (such as an emergency hotline and/or an ambulance service) to send someone to help.
- emergency services such as an emergency hotline and/or an ambulance service
- the monitoring systems described herein can be applied to monitoring in a home.
- the monitoring system can accomplish one or more of the following features using the machine learning techniques described herein.
- the monitoring system can monitor the time spent on various tasks by members of a household (such as youth at home), such as time spent watching television or time spent studying.
- the monitoring system can be configured to confirm that certain tasks (such as chores) are completed.
- the monitoring system can allow parents to monitor an amount of time spent using electronics.
- the camera system can be configured to detect night terrors and amount and types of sleep.
- the monitoring system can track usage of good hygiene practices at home, such as but not limited to, handwashing, brushing teeth, or showering (e.g., tracking that a person enters a bathroom at a showering time).
- zones can be used to provide alerts, such as monitoring a pool zone or other spaces youth should not be allowed, such as, but not limited to, certain rooms at certain times and/or unaccompanied by an adult.
- the camera system can monitor a gun storage location to alert adults to unauthorized access of weapons.
- a camera system can have local storage for an image and/or video feed. In some aspects, remote access of the local storage may be restricted and/or limited. In some aspects, the camera system can use a calibration factor which can be useful for correcting color drift in the image data from a camera. In some aspects, the camera system can add or remove filters on camera to provide certain effects. The camera system may include infrared filters. In some aspects, the monitoring system can monitor food intake of subject and/or estimate calories. In some aspects, the monitoring system can detect mask wearing (such as wearing or not wearing an oxygen mask).
- the monitoring system can perform one or more check-up tests.
- the monitoring system using a machine learning model, can detect slurred speech, drunkenness, drug use, and/or adverse behavior. Based on other check-up tests the monitoring system can detect shaking, microtremors, tremors, which can indicate a potential disease state such as Parkinson's.
- the monitoring system can track exercise movements to determine a potential physiological condition.
- a check-up test can be used by the monitoring system for a cognitive assessment, such as, detecting vocabulary decline.
- the monitoring system can check a user's smile where the monitoring system prompts the user to stand a specified distance away from the camera system.
- a check-up test can request a subject to do one or more exercise, read something outload (to test muscles of a face), reach for an object.
- the camera system can perform an automated physical, perform a hearing test, and/or perform an eye test.
- a check-up test can be for Alzheimer's detection.
- the monitoring system can provide memory exercises, monitor for good/bad days, and/or monitor basic behavior to prevent injury.
- the camera system can monitor skin color changes to detect skin damage and/or sunburn detection.
- the camera system can take a trend of skin color, advise or remind to take corrective action, and/or detect a tan line.
- the monitoring system can monitor sleep cycles and/or heart rate variability.
- the monitoring system can monitor snoring, rapid eye movement (REM), and/or sleep quality, which can be indicative of sleep apnea or another disease.
- the camera system can be tried to detect sleep walking.
- the camera system can be configured to detect coughing or sneezing to determine potential allergies or illness.
- the camera system can also provide an alert if a possible hyperventilation is detected. Any of the monitoring features described herein can be implemented with the machine learning techniques described herein.
- FIG. 11 is a block diagram that illustrates example components of a computing device 1100 , which can be a camera system.
- the computing device 1100 can implement aspects of the present disclosure, and, in particular, aspects of the monitoring system 100 A, 100 B, such as the camera system 114 .
- the computing device 1100 can communicate with other computing devices.
- the computing device 1100 can include a hardware processor 1102 , a hardware accelerator, a data storage device 1104 , a memory device 1106 , a bus 1108 , a display 1112 , one or more input/output devices 1114 , and a camera 1118 .
- a processor 1102 can also be implemented as a combination of computing devices, e.g., a combination of a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor, or any other such configuration.
- the processor 1102 can be configured, among other things, to process data, execute instructions to perform one or more functions, such as apply one or more machine learning models, as described herein.
- the hardware accelerator 1116 can be special hardware that is configured to accelerate machine learning applications.
- the data storage device 1104 can include a magnetic disk, optical disk, or flash drive, etc., and is provided and coupled to the bus 1108 for storing information and instructions.
- the memory 1106 can include one or more memory devices that store data, including without limitation, random access memory (RAM) and read-only memory (ROM).
- RAM random access memory
- ROM read-only memory
- the computing device 1100 may be coupled via the bus 1108 to a display 1112 , such as an LCD display or touch screen, for displaying information to a user, such as a patient.
- the computing device 1100 may be coupled via the bus 1108 to one or more input/output devices 1114 .
- the input device 1114 can include, but is not limited to, a keyboard, mouse, digital pen, microphone, touch screen, gesture recognition system, voice recognition system, imaging device (which may capture eye, hand, head, or body tracking data and/or placement), gamepad, accelerometer, or gyroscope.
- the camera 1118 can include, but is not limited to, a 1080p or 4k camera and/or an infrared image camera.
- the term “patient” can refer to any person that is monitored using the systems, methods, devices, and/or techniques described herein. As used herein, a “patient” is not required—to be admitted to a hospital, rather, the term “patient” can refer to a person that is being monitored. As used herein, in some cases the terms “patient” and “user” can be used interchangeably.
- While some features described herein may be discussed in a specific context, such as adult, youth, infant, elderly, or pet care, those features can be applied to other contexts, such as, but not limited to, a different one of adult, youth, infant, elderly, or pet care contexts.
- the apparatuses and methods described herein may be implemented by one or more computer programs executed by one or more processors.
- the computer programs include processor-executable instructions that are stored on a non-transitory tangible computer readable medium.
- the computer programs may also include stored data.
- Non-limiting examples of the non-transitory tangible computer readable medium are nonvolatile memory, magnetic storage, and optical storage.
- Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is otherwise understood with the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z). Such disjunctive language is not generally intended to, and should not, imply that certain aspects require at least one of X, at least one of Y, or at least one of Z to each be present.
- the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list.
- the term “each,” as used herein, in addition to having its ordinary meaning, can mean any subset of a set of elements to which the term “each” is applied.
Abstract
Systems and methods are provided for machine learning based monitoring. A current time is received. The system determines to begin a check-up process from the current time. In response to determining to begin the check-up process, a prompt to cause a person to perform a check-up activity is presented on a display. Image data of a recording of the check-up activity is received from the camera. The system invokes a screening machine learning model based on the image data. The screening machine learning model outputs a classification result. The system detects a potential screening issue based on the classification result. In response to detecting the potential screening issue, the system provides an alert.
Description
- The present application claims benefit of U.S. Provisional Application No. 63/298,569 entitled “Intelligent Camera System” filed Jan. 11, 2022 and U.S. Provisional Application No. 63/299,168 entitled “Intelligent Camera System” filed Jan. 13, 2022, the entirety of each of which is hereby incorporated by reference. Any and all applications for which a foreign or domestic priority claim is identified in the Application Data Sheet as filed with the present application are hereby incorporated by reference under 37 CFR 1.57.
- A smart camera system can be a machine vision system which, in addition to image capture capabilities, is capable of extracting information from captured images. Some smart camera systems are capable of generating event descriptions and/or making decisions that are used in an automated system. Some camera systems can be a self-contained, standalone vision system with a built-in image sensor. The vision system and the image sensor can be integrated into a single hardware device. Some camera systems can include communication interfaces, such as, but not limited to Ethernet and/or wireless interfaces.
- Safety can be important in clinical, hospice, assisted living, and/or home settings. Potentially dangerous events can happen in these environments. Automation can also be beneficial in these environments.
- The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of this disclosure, several non-limiting features will now be discussed briefly.
- According to an aspect, a system is disclosed comprising: a storage device configured to store first instructions and second instructions; a camera; a hardware accelerator configured to execute the first instructions; and a hardware processor configured to execute the second instructions to: receive, from the camera, first image data; invoke, on the hardware accelerator, a person detection model based on the first image data, wherein the person detection model outputs first classification result; detect a person based on the first classification result; receive, from the camera, second image data; and in response to detecting the person, invoke, on the hardware accelerator, a fall detection model based on the second image data, wherein the fall detection model outputs a second classification result, detect a potential fall based on the second classification result, and in response to detecting the potential fall, provide an alert.
- According to an aspect, the system may further comprise a microphone, wherein the hardware processor may be configured to execute further instructions to: receive, from the microphone, audio data; and in response to detecting the person, invoke, on the hardware accelerator, a loud noise detection model based on the audio data, wherein the loud noise detection model outputs a third classification result, and detect a potential scream based on the third classification result.
- According to an aspect, the hardware processor may be configured to execute additional instructions to: in response to detecting the potential scream, provide a second alert.
- According to an aspect, the hardware processor may be configured to execute additional instructions to: in response to detecting the potential fall and the potential scream, provide an escalated alert.
- According to an aspect, invoking the loud noise detection model based on the audio data may further comprise: generating spectrogram data from the audio data; and providing the spectrogram data as input to the loud noise detection model.
- According to an aspect, the second image data may comprise a plurality of images.
- According to an aspect, a method is disclosed comprising: receiving, from a camera, first image data; invoking, on a hardware accelerator, a person detection model based on the first image data, wherein the person detection model outputs first classification result; detecting a person based on the first classification result; receiving, from the camera, second image data; and in response to detecting the person, invoking, on the hardware accelerator, a plurality of person safety models based on the second image data, for each person safety model from the plurality of person safety models, receiving, from the hardware accelerator, a second classification result, detecting a potential safety issue based on a particular second classification result, and in response to detecting the potential safety issue, providing an alert.
- According to an aspect, the method may further comprise: in response to detecting the person, invoking, on the hardware accelerator, a facial feature extraction model based on the second image data, wherein the facial feature extraction model outputs a facial feature vector, executing a query of a facial features database based on the facial feature vector, wherein executing the query indicates that the facial feature vector is not present in the facial features database, and in response to determining that the facial feature vector is not present in the facial features database, providing an unrecognized person alert.
- According to an aspect, the plurality of person safety models may comprise a fall detection model, the method may further comprise: collecting a first set of videos of person falls; collecting a second set of videos of persons without falling; creating a training data set comprising the first set of videos and the second set of videos; and training the fall detection model using the training data set.
- According to an aspect, the plurality of person safety models may comprise a handwashing detection model, the method may further comprise: collecting a first set of videos of with handwashing; collecting a second set of videos without handwashing; creating a training data set comprising the first set of videos and the second set of videos; and training the handwashing detection model using the training data set.
- According to an aspect, the method may further comprise: receiving, from a microphone, audio data; and in response to detecting the person, invoking, on the hardware accelerator, a loud noise detection model based on the audio data, wherein the loud noise detection model outputs a third classification result, and detecting a potential scream based on the third classification result.
- According to an aspect, the method may further comprise: in response to detecting the potential safety issue and the potential scream, providing an escalated alert.
- According to an aspect, the method may further comprise: collecting a first set of videos of with screaming; collecting a second set of videos without screaming; creating a training data set comprising the first set of videos and the second set of videos; and training the loud noise detection model using the training data set.
- According to an aspect, a system is disclosed comprising: a storage device configured to store first instructions and second instructions; a camera; a hardware accelerator configured to execute the first instructions; and a hardware processor configured to execute the second instructions to: receive, from the camera, first image data; invoke, on the hardware accelerator, a person detection model based on the first image data, wherein the person detection model outputs first classification result; detect a person based on the first classification result; receive, from the camera, second image data; and in response to detecting the person, invoke, on the hardware accelerator, a plurality of person safety models based on the second image data, for each person safety model from the plurality of person safety models, receive, from the hardware accelerator, a model result, detect a potential safety issue based on a particular model result, and in response to detecting the potential safety issue, provide an alert.
- According to an aspect, the plurality of person safety models may comprise a fall detection model, and wherein invoking the plurality of person safety models may comprise: invoking, on the hardware accelerator, the fall detection model based on the second image data, wherein the fall detection model outputs the particular model result.
- According to an aspect, the plurality of person safety models may comprise a handwashing detection model, and wherein invoking the plurality of person safety models may comprise: invoking, on the hardware accelerator, the handwashing detection model based on the second image data, wherein the handwashing detection model outputs the particular model result.
- According to an aspect, the system may further comprise a microphone, wherein the hardware processor may be configured to execute further instructions to: receive, from the microphone, audio data; and in response to detecting the person, invoke, on the hardware accelerator, a loud noise detection model based on the audio data, wherein the loud noise detection model outputs a third classification result, detect a potential loud noise based on the third classification result, and in response to detecting the potential loud noise, provide a second alert.
- According to an aspect, the system may further comprise a display, wherein the hardware processor may be configured to execute further instructions to: cause presentation, on the display, of a prompt to cause a person to perform an activity; receive, from the camera, third image data of a recording of the activity; invoke, on the hardware accelerator, a screening machine learning model based on the third image data, wherein the screening machine learning model outputs a third classification result, detect a potential screening issue based on the third classification result, and in response to detecting the potential screening issue, provide a second alert.
- According to an aspect, the screening machine learning model may be a pupillometry screening model, and wherein the potential screening issue indicates potential dilated pupils.
- According to an aspect, the screening machine learning model may be a facial paralysis screening model, and wherein the potential screening issue indicates potential facial paralysis.
- According to an aspect, a system is disclosed comprising: a storage device configured to store first instructions and second instructions; a wearable device configured to process sensor signals to determine a physiological value for a person; a microphone; a camera; a hardware accelerator configured to execute the first instructions; and a hardware processor configured to execute the second instructions to: receive, from the wearable device, the first physiological value; determine to begin a monitoring process based on the first physiological value; and in response to determining to begin the monitoring process, receive, from the camera, image data; receive, from the microphone, audio data; invoke, on the hardware accelerator, a first unconscious detection model based on the image data, wherein the first unconscious detection model outputs a first classification result, invoke, on the hardware accelerator, a second unconscious detection model based on the audio data, wherein the second unconscious detection model outputs a second classification result, detect a potential state of unconsciousness based on the first classification result and the second classification result, and in response to detecting the potential state of unconsciousness, provide an alert.
- According to an aspect, the wearable device may comprise a pulse oximetry sensor and the first physiological value is for blood oxygen saturation, and wherein determining to begin the monitoring process based on the first physiological value further comprises: determining that the first physiological value is below a threshold level.
- According to an aspect, the wearable device may comprise a respiration rate sensor and the first physiological value is for respiration rate, and wherein determining to begin the monitoring process based on the first physiological value further comprises: determining that the first physiological value satisfies a threshold alarm level.
- According to an aspect, the wearable device comprises a heart rate sensor and the first physiological value is for heart rate, and wherein determining to begin the monitoring process based on the physiological value further comprises: receiving, from the wearable device, a plurality of physiological values measuring heart rate over time; and determining that the plurality of physiological values and the first physiological value satisfies a threshold alarm level.
- According to an aspect, a system is disclosed comprising: a storage device configured to store instructions; a display; a camera; and a hardware processor configured to execute the instructions to: receive a current time; determine to begin a check-up process from the current time; and in response to determining to begin the check-up process, cause presentation, on the display, of a prompt to cause a person to perform a check-up activity, receive, from the camera, image data of a recording of the check-up activity, invoke a screening machine learning model based on the image data, wherein the screening machine learning model outputs a classification result, detect a potential screening issue based on the classification result, and in response to detecting the potential screening issue, provide an alert.
- According to an aspect, the screening machine learning model may be a pupillometry screening model, and wherein the potential screening issue indicates potential dilated pupils.
- According to an aspect, the screening machine learning model may be a facial paralysis screening model, and wherein the potential screening issue indicates potential facial paralysis.
- According to an aspect, the system may further comprise a wearable device configured to process sensor signals to determine a physiological value for the person, wherein the hardware processor may be configured to execute further instructions to: receive, from the wearable device, the physiological value; and generate the alert comprising the physiological value.
- According to an aspect, the wearable device may comprise a pulse oximetry sensor and the physiological value is for blood oxygen saturation.
- According to an aspect, the wearable device may be further configured to process the sensor signals to measure at least one of blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, or pleth variability index.
- According to an aspect, the hardware processor may be configured to execute further instructions to: receive, from a second computing device, first video data; cause presentation, on the display, of the first video data; receive, from the camera, second video data; and transmit, to the second computing device, the second video data.
- According to an aspect, a method is disclosed comprising: receiving a current time; determining to begin a check-up process from the current time; and in response to determining to begin the check-up process, causing presentation, on a display, of a prompt to cause a person to perform a check-up activity, receiving, from a camera, image data of a recording of the check-up activity, invoking a screening machine learning model based on the image data, wherein the screening machine learning model outputs a model result, detecting a potential screening issue based on the model result, and in response to detecting the potential screening issue, providing an alert.
- According to an aspect, the screening machine learning model may be a pupillometry screening model, and wherein the potential screening issue indicates potential dilated pupils, the method further comprise: collecting a first set of images of dilated pupils; collecting a second set of images without dilated pupils; creating a training data set comprising the first set of images and the second set of images; and training the pupillometry screening model using the training data set.
- According to an aspect, the screening machine learning model may be a facial paralysis screening model, and wherein the potential screening issue indicates potential facial paralysis, the method may further comprise: collecting a first set of images of facial paralysis; collecting a second set of images without facial paralysis; creating a training data set comprising the first set of images and the second set of images; and training the facial paralysis screening model using the training data set.
- According to an aspect, the check-up activity may comprise a dementia test, and wherein the screening machine learning model may comprise a gesture detection model.
- According to an aspect, the gesture detection model may be configured to detect a gesture directed towards a portion of the display.
- According to an aspect, the method may further comprise: receiving, from the camera, second image data; invoking a person detection model based on the second image data, wherein the person detection model outputs first classification result; detect a person based on the first classification result; receive, from the camera, third image data; and in response to detecting the person, invoking a handwashing detection model based on the third image data, wherein the handwashing detection model outputs a second classification result, detecting a potential lack of handwashing based on the second classification result, and in response to detecting a lack of handwashing, provide a second alert.
- According to an aspect, a system is disclosed comprising: a storage device configured to store instructions; a camera; and a hardware processor configured to execute the instructions to: receive, from the camera, first image data; invoke an infant detection model based on the first image data, wherein the infant detection model outputs a classification result; detect an infant based on the classification result; receive captured data; and in response to detecting the infant, invoke an infant safety model based on the captured data, wherein the infant safety model outputs a model result, detect a potential safety issue based on the model result, and in response to detecting the potential safety issue, provide an alert.
- According to an aspect, the infant safety model may be an infant position model, and wherein the potential safety issue indicates the infant potentially laying on their stomach.
- According to an aspect, the hardware processor may be configured to execute further instructions to: receive, from the camera, second image data; and in response to detecting the infant, invoke a facial feature extraction model based on the second image data, wherein the facial feature extraction model outputs a facial feature vector, execute a query of a facial features database based on the facial feature vector, wherein executing the query indicates that the facial feature vector is not present in the facial features database, and in response to determining that the facial feature vector is not present in the facial features database, provide an unrecognized person alert.
- According to an aspect, the infant safety model may be an infant color detection model, and wherein the potential safety issue indicates potential asphyxiation.
- According to an aspect, the model result may comprise coordinates of a boundary region identifying an infant object in the captured data, and wherein detecting the potential safety issue may comprise: determining that the coordinates of the boundary region exceed a threshold distance from an infant zone.
- According to an aspect, the system may further comprise a wearable device configured to process sensor signals to determine a physiological value for the infant, wherein the hardware processor may be configured to execute further instructions to: receive, from the wearable device, the physiological value; and generate the alert comprising the physiological value.
- According to an aspect, the system may further comprise a microphone, wherein the captured data is received from the microphone, wherein the infant safety model is a loud noise detection model, and wherein the potential safety issue indicates a potential scream.
- In various aspects, systems and/or computer systems are disclosed that comprise a computer readable storage medium having program instructions embodied therewith, and one or more processors configured to execute the program instructions to cause the one or more processors to perform operations comprising one or more of the above- and/or below-aspects (including one or more aspects of the appended claims).
- In various aspects, computer-implemented methods are disclosed in which, by one or more processors executing program instructions, one or more of the above- and/or below-described aspects (including one or more aspects of the appended claims) are implemented and/or performed.
- These and other features, aspects, and advantages are described below with reference to the drawings, which are intended for illustrative purposes and should in no way be interpreted as limiting. Furthermore, the various features described herein can be combined to form new combinations, which are part of this disclosure. In the drawings, like reference characters can denote corresponding features. The following is a brief description of each of the drawings.
-
FIG. 1A is a drawing of a camera system in a clinical setting. -
FIG. 1B is a schematic diagram illustrating a monitoring system. -
FIG. 2 is a schematic drawing of a monitoring system in a clinical setting. -
FIG. 3 is another schematic drawing of a monitoring system in a clinical setting. -
FIG. 4 is a drawing of patient sensor devices that can be used in a monitoring system. -
FIG. 5 illustrates a camera image with object tracking. -
FIG. 6 is a drawing of a monitoring system in a home setting. -
FIG. 7 is a drawing of a monitoring system configured for baby monitoring. -
FIG. 8 is a flowchart of a method for efficiently applying machine learning models. -
FIG. 9 is a flowchart of another method for efficiently applying machine learning models. -
FIG. 10 is a flowchart of a method for efficiently applying machine learning models for infant care. -
FIG. 11 illustrates a block diagram of a computing device that may implement one or more aspects of the present disclosure. - As described above, some camera systems are capable of extracting information from captured images. However, extracting information from images and/or monitoring by existing camera systems can be limited. Technical improvements regarding monitoring people and/or objects and automated actions based on the monitoring can advantageously be helpful, improve safety, and possibly save lives.
- Generally described, aspects of the present disclosure are directed to improved monitoring systems. In some aspects, a camera system can include a camera and a hardware accelerator. The camera system can include multiple machine learning models. Each model of the machine learning models can be configured to detect an object and/or an activity. The hardware accelerator can be special hardware that is configured to accelerate machine learning applications. The camera system can be configured to execute the machine learning models on the hardware accelerator. The camera system can advantageously be configured to execute conditional logic to determine which machine learning models should be applied and when. For example, until a person is detected in an area, the camera system may not apply any machine learning models related to persons, such as, but not limited to, fall detection, person identification, stroke detection, medication tracking, activity tracking, etc.
- Some existing monitoring systems can have limited artificial intelligence capabilities. For example, some existing monitoring systems may only have basic person, object, or vehicle detection. Moreover, some existing monitoring systems may require a network connection from local cameras to backend servers that perform the artificial intelligence processing. Some existing cameras may have limited or no artificial intelligence capabilities. Performing artificial intelligence processing locally on cameras can be technically challenging. For example, the hardware processors and/or memory devices in existing cameras may be so limited as being unable to execute machine learning models locally. Moreover, existing cameras may have limited software to be able to execute machine learning models locally in an efficient manner. The systems and methods described herein may efficiently process camera data either locally and/or in a distributed manner with machine learning models. Accordingly, the systems and methods described herein may improve over existing artificial intelligence monitoring technology.
- As used herein, “camera” and “camera system” can be used interchangeably. Moreover, as used herein, “camera” and “camera system” can be used interchangeably with “monitoring system” since a camera system can encompass a monitoring system in some aspects.
-
FIG. 1A depicts acamera system 114 in aclinical setting 101. Theclinical setting 101 can be, but is not limited to, a hospital, nursing home, or hospice. Theclinical setting 101 can include thecamera system 114, adisplay 104, and auser computing device 108. In some aspects, thecamera system 114 can be housed in a soundbar enclosure or a tabletop speaker enclosure (not illustrated). Thecamera system 114 can include multiple cameras (such as 1080p or 4k camera and/or an infrared image camera), an output speaker, an input microphone (such as a microphone array), an infrared blaster, and/or multiple hardware processors (including one or more hardware accelerators). In some aspects, thecamera system 114 can have optical zoom. In some aspects, thecamera system 114 can include a privacy switch that allows the monitoring system's 100A, 100B cameras to be closed. Thecamera system 114 may receive voice commands. Thecamera system 114 can include one or more hardware components for Bluetooth®, Bluetooth Low Energy (BLE), Ethernet, Wi-Fi, cellular (such as 4G/5G/LTE), near-field communication (NFC), radio-frequency identification (RFID), High-Definition Multimedia Interface (HDMI), and/or HDMI Consumer Electronics Control (CEC). Thecamera system 114 can be connected to the display 104 (such as a television) and thecamera system 114 can control thedisplay 104. In some aspects, thecamera system 114 can be wirelessly connected to the user computing device 108 (such as a tablet). In particular, thecamera system 114 can be wireles sly connected to a hub device and the hub device can be wirelessly connected to theuser computing device 108. - The
camera system 114 may include machine learning capabilities. Thecamera system 114 can include machine learning models. The machine learning models can include, but are not limited to, convolutional neural network (CNN) models and other models. A CNN model can be trained to extract features from images for object identification (such as person identification). In some aspects, a CNN can feed the extracted features to a recurrent neural network (RNN) for further processing. Thecamera system 114 may track movements of individuals inside the room without using any facial recognition or identification tag tracking. Identification tags can include, but are not limited to, badges and/or RFID tags. This feature allows thecamera system 114 to track an individual's movements even when the identification of the individual is unknown. A person in the room may not be identifiable for various reasons. For example, the person may be wearing a mask so that facial recognition modules may not be able to extract any features. As another example, the person may be a visitor who is not issued an identification tag, unlike the clinicians, who typically wear identification tags. Alternatively, when the person is not wearing a mask and/or is wearing an identification tag, thecamera system 114 may combine the motion tracking with the identification of the individual to further improve accuracy in tracking the activity of the individual in the room. Having the identity of at least one person in the room may also improve accuracy in tracking the activity of other individuals in the room whose identity is unknown by reducing the number of anonymous individuals in the room. Additional details regarding machine learning capabilities and models that thecamera system 114 can use are provided herein. - The
camera system 114 can be included in a monitoring system, as described herein. The monitoring system can include remote interaction capabilities. A patient in the clinical setting 101 can be in isolation due to an illness, such as COVID-19. The patient can ask for assistance via a button (such as by selecting an element in the graphical user interface on the user computing device 108) and/or by issuing a voice command. In some aspects, thecamera system 114 can be configured to respond to voice commands, such as, but not limited to, activating or deactivating cameras or other functions. In response to the request, aremote clinician 106 can interact with the patient via thedisplay 104 and thecamera system 114, which can include an input microphone and an output speaker. The monitoring system can also allow the patient to remotely maintain contact with friends and family via thedisplay 104 andcamera system 114. In some aspects, thecamera system 114 can be connected to internet of things (IOT) devices. In some aspects, closing of the privacy switch can cause thecamera system 114 and/or a monitoring system to disable monitoring. In other aspects, the monitoring system can still issue alerts if the privacy switch has been closed. In some aspects, thecamera system 114 can record activity via cameras based on a trigger, such as, but not limited to, detection of motion via a motion sensor. -
FIG. 1B is a diagram depicting amonitoring system monitoring system 100A and a clinical side to themonitoring system 100B. As described herein, the clinicalside monitoring system 100B can track and monitor a patient via afirst camera system 114 in a clinical setting. As described herein, the patient can be monitored via wearable sensor devices. Aclinician 110 can interact with the patient via thefirst display 104 and thefirst camera system 114. Friends and family can also use auser computing device 102 to interact with the patient via thefirst display 104 and thefirst camera system 114. - The home/assisted living
side monitoring system 100A can track and monitor a person (which can be an infant) via asecond camera system 134 in a home/assisted living setting. For example, a person can be recovering at home or live in an assisted living home. As described herein, the person can be monitored via wearable sensor devices. Aclinician 110 can interact with the person via thesecond display 124 and thesecond camera system 134. As shown, the clinical side to themonitoring system 100B can securely communicate with the home/assisted living side to themonitoring system 100A, which can allow communications between theclinician 110 and persons in the home or assisted living home. Friends and family can use theuser computing device 102 to interact with the patient via thesecond display 124 and thesecond camera system 134. - In some aspects, the
monitoring system clinician 110 and a person via thesecond display 124 and thesecond camera system 134. The server(s) 130A, 130B can facilitate communication between theuser computing device 102 and the patient via thefirst display 104 and thefirst camera system 114. As described herein, the server(s) 130A, 130B can communicate with the camera system(s) 114, 134. In some aspects, the server(s) 130A, 130B can transmit machine learning model(s) to the camera system(s) 114, 134. In some aspects, the server(s) 130A, 130B can train machine learning models based on training data sets. - In some aspects, the
monitoring system monitoring system -
FIG. 2 is a diagram depicting amonitoring system 200 in another clinical setting with an accompanying legend. Themonitoring system 200 can include, but is not limited to,cameras server 276. Some of thecameras camera system 114 ofFIG. 1A . Thecameras server 276. Theserver 276 can be located in the hospital room, or elsewhere in the hospital, or at a remote location outside the hospital (not illustrated). As shown, in a clinical setting, such as a hospital, hospitalized patients can be lying on hospital beds, such as thehospital bed 274. Thebed cameras bed 274 facing toward a foot side of thebed 274. The clinical setting may have ahandwashing area 278. Thehandwashing cameras handwashing area 278. Thehandwashing cameras view 282C so as to maximize the ability to detect a person's face and/or identification tag when the person is standing next to thehandwashing area 278 facing the sink. Via the bed camera(s) 272A, 272B, themonitoring system 200 can detect whether the clinician (or a visitor) is within apatient zone 275, which can be located within a field ofview patient zone 275 can be defined as a proximity threshold around thehospital bed 274 and/or a patient. In some aspects, theclinician 281 is within thepatient zone 275 if the clinician is at least partially within a proximity threshold distance to the hospital bed and/or the patient. - The
bed cameras bed 274, where the patient's head would be at when the patient lies on thebed 274. Thebed cameras bed 274, and can both be pointing toward thebed 274. The fields ofview bed cameras bed 274. The combined field ofview bed 274 so that a person standing by any of the four sides of thebed 274 can be in the combined field ofview bed cameras bed cameras bed 274, independent of the orientation of the person's face, and/or the ability to detect an identification tag that is worn on the person's body, for example, hanging by the neck, the belt, etc. Optionally, thebed cameras bed 274, as the identity of the patient is typically known in clinical and other settings. - In some aspects, the
cameras monitoring system 200 to detect a person's face and/or identification tag, if any. For example, at least some of thecameras cameras cameras - In some aspects, the
monitoring system 200 can monitor one or more aspects about the patient, theclinician 281, and/or zones. Themonitoring system 200 can determine whether the patient is in thebed 274. Themonitoring system 200 can detect whether the patient is within a bed zone, which can be within thepatient zone 275. Themonitoring system 200 can determine an angle of the patient in thebed 274. In some aspects, themonitoring system 200 can include a wearable, wireless sensor device (not illustrated) that can track a patient's posture, orientation, and activity. In some aspects, a wearable, wireless sensor device can include, but is not limited to, a Centroid® device by Masimo Corporation, Irvine, Calif.. Themonitoring system 200 can determine how often the patient has turned in thebed 274 and/or gotten up from thebed 274. Themonitoring system 200 can detect turning and/or getting up based on the bed zone and/or facial recognition of the patient. Themonitoring system 200 can detect whether theclinician 281 is within thepatient zone 275 or another zone. As described herein, themonitoring system 200 can detect whether theclinician 281 is present or not present via one or more methods, such as, but not limited to, facial recognition, identification via an image of an identification tag, and/or RFID based tracking. Similarly, themonitoring system 200 can detect intruders that are unauthorized in one or more zones via one or more methods, such as, but not limited to, facial recognition, identification via an image of an identification tag, and/or RFID based tracking. In some aspects, themonitoring system 200 can issue an alert based on one or more of the following factors: facial detection of an unrecognized face; no positive visual identification of authorized persons via identification tags; and/or no positive identification of authorized persons via RFID tags. In some aspects, themonitoring system 200 can detect falls via one or more methods, such as, but not limited to, machine-vision based fall detection and/or fall detection via wearable device, such as using accelerometer data. Any of the alerts described herein can be presented on thedisplays - In some aspects, if the
monitoring system 200 detects that theclinician 281 is within thepatient zone 275 and/or has touched the patient, then thesystem 200 can assign a “contaminated” status to theclinician 281. Themonitoring system 200 can detect a touch action by detecting the actual act of touching by theclinician 281 and/or by detecting theclinician 281 being in close proximity, for example, within less than 1 foot, 6 inches, or otherwise, of the patient. If theclinician 281 moves outside thepatient zone 275, then themonitoring system 200 can assign a “contaminated-prime” status to theclinician 281. If theclinician 281 with the “contaminated-prime” status re-enters thesame patient zone 275 or enters a new patient zone,monitoring system 200 can output an alarm or warning. If themonitoring system 200 detects a handwashing activity by theclinician 281 with a “contaminated-prime” status, then themonitoring system 200 can assign a “not contaminated” status to theclinician 281. - A person may also be contaminated by entering contaminated areas other than a patient zone. For example, as shown in
FIG. 2 , the contaminated areas can include apatient consultation area 284. Thepatient consultation area 284 can be considered a contaminated area with or without the presence of a patient. Themonitoring system 200 can include aconsultation area camera 286, which has a field ofview 282D that overlaps with and covers thepatient consultation area 284. The contaminated areas can further include a check-inarea 288 that is next to a door of the hospital room. Alternatively and/or additionally, the check-inarea 288 can extend to include the door. The check-inarea 288 can be considered a contaminated area with or without the presence of a patient. Themonitoring system 200 can include anentrance camera 290, which has a field ofview 282E that overlaps with and covers the check-inarea 288. - As shown in
FIG. 2 , themonitoring system 200 can include anadditional camera 294. Additional cameras may not be directed to any specific contaminated and/or handwashing areas. For example, theadditional camera 294 can have a field ofview 282F that covers substantially an area that a person likely has to pass when moving from one area to another area of the hospital room, such as from thepatient zone 275 to theconsultation area 284. Additional camera can provide data to theserver 276 to facilitate tracking of movements of the people in the room. -
FIG. 3 depicts amonitoring system 300 in another clinical setting. Themonitoring system 300 may monitor the activities of anyone present in the room such as medical personnel, visitors, patients, custodians, etc. As described herein, themonitoring system 300 may be located in a clinical setting such as a hospital room. The hospital room may include one or morepatient beds 308. The hospital room may include an entrance/exit 329 to the room. The entrance/exit 329 may be the only entrance/exit to the room. - The
monitoring system 300 can include aserver 322, adisplay 316, one ormore camera systems additional device 310. Thecamera systems server 322. Theserver 322 may be a remote server. The one or more camera systems may include afirst camera system 318, asecond camera system 320, and/oradditional camera systems 314. Thecamera systems enclosure camera systems camera systems camera systems camera systems - The
monitoring system 300 may include a database. The database can include information relating to the location of items in the room such as camera systems, patient beds, handwashing stations, and/or entrance/exits. The database can include locations of thecamera systems camera system camera system server 322 may use the identifiers to configure each of thecamera systems - As shown in
FIG. 3 , thefirst camera system 318 can include afirst enclosure 324 and afirst camera 302. Thefirst enclosure 324 can enclose one or more hardware processors. Thefirst camera 302 may be a camera capable of sensing depth and color, such as, but not limited to, an RGB-D stereo depth camera. Thefirst camera 302 may be positioned in a location of the room to monitor the entire room or substantially all of the room. Thefirst camera 302 may be tilted downward at a higher location in the room. Thefirst camera 302 may be set up to minimize blind spots in the field of view of thefirst camera 302. For example, thefirst camera 302 may be located in a corner of the room. Thefirst camera 302 may be facing the entrance/exit 329 and may have a view of the entrance/exit 329 of the room. - As shown in
FIG. 3 , thesecond camera system 320 can include a second enclosure 326 (which can include one or more processors) and asecond camera 304. Thesecond camera 304 may be a RGB color camera. Alternatively, thesecond camera 304 may be an RGB-D stereo depth camera. Thesecond camera 304 may be installed over a handhygiene compliance area 306. The handhygiene compliance area 306 may include a sink and/or a hand sanitizer dispenser. Thesecond camera 304 may be located above the handhygiene compliance area 306 and may be point downwards toward the handhygiene compliance area 306. For example, thesecond camera 304 may be located on or close to the ceiling and may have a view the handhygiene compliance area 306 from above. - In a room of a relatively small size, the first and
second camera systems system 300 may include any number of additional camera systems, such as athird camera system 314. Thethird camera system 314 may include a third enclosure 313 (which can include one or more processors) and athird camera 312. Thethird camera 312 of thethird camera system 314 may be located near the patient'sbed 308 or in a corner of the room, for example, a corner of the room that is different than (for example, opposite or diagonal to) the corner of the room where thefirst camera 302 of thefirst camera system 318 is located. Thethird camera 312 may be located at any other suitable location of the room to aid in reducing blind spots in the combined fields of view of thefirst camera 302 and thesecond camera 304. Thethird camera 312 of thethird camera system 314 may have a field of view covering the entire room. Thethird camera system 314 may operate similarly to thefirst camera system 318, as described herein. - The
monitoring system 300 may include one or moreadditional devices 310. Theadditional device 310 can be, but is not limited to, a patient monitoring and connectivity hub, bedside monitor, or other patient monitoring device. For example, theadditional device 310 can be a Root® monitor by Masimo Corporation, Irvine, Calif.. Additionally or alternatively, theadditional device 310 can be, but is not limited to, a display device of a data aggregation and/or alarm visualization platform. For example, theadditional device 310 can be a display device (not illustrated) for the Uniview® platform by Masimo Corporation, Irvine, Calif. The additional device(s) 310 can include smartphones or tablets (not illustrated). The additional device(s) may be in communication with theserver 322 and/or thecamera systems - The
monitoring system 300 can output alerts on the additional device(s) 310 and/or thedisplay 316. The outputted alert may be any auditory and/or visual signal. Outputted alerts can include, but are not limited to, a fall alert, an unauthorized person alert, an alert that a patient should be turned, or an alert that a person has not complied the hand hygiene protocol. For example, someone outside of the room can be notified on anadditional device 310 and/or thedisplay 316 that an emergency has occurred in the room. In some aspects, themonitoring system 300 can provide a graphical user interface, which can be presented on thedisplay 316. A configuration user can configure themonitoring system 300 via the graphical user interface presented on thedisplay 316. -
FIG. 4 depictspatient sensor devices patient sensor devices devices FIG. 4 . A patient sensor device can non-invasively measure physiological parameters from a fingertip, wrist, chest, forehead, or other portion of the body. The first, second, and thirdpatient sensor devices user computing device 402 and/or a server in the monitoring system. The firstpatient sensor device 404 can include a display and a touchpad and/or touchscreen. The firstpatient sensor device 404 can be a pulse oximeter that is designed to non-invasively monitor patient physiological parameters from a fingertip. The firstpatient sensor device 404 can measure physiological parameters such as, but not limited to, blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, and/or pleth variability index. The firstpatient sensor device 404 can be a MightyS at® fingertip pulse oximeter by Masimo Corporation, Irvine, Calif. The secondpatient sensor device 406 can be configured to be worn on a patient's wrist to non-invasively monitor patient physiological parameters from a wrist. The secondpatient sensor device 406 can be a smartwatch. The secondpatient sensor device 406 can include a display and/or touchscreen. The secondpatient sensor device 406 can measure physiological parameters including, but not limited to, blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, and/or pleth variability index. The thirdpatient sensor device 408 can be a temperature sensor that is designed to non-invasively monitor physiological parameters of a patient. In particular, the thirdpatient sensor device 408 can measure a temperature of the patient. The thirdpatient sensor device 408 can be a Radius T°™ sensor by Masimo Corporation, Irvine, Calif. A patient, clinician, or other authorized user can use theuser computing device 408 to view physiological information and other information from the monitoring system. - As shown, a graphical user interface can be presented on the
user computing device 402. The graphical user interface can present physiological parameters that have been measured by thepatient sensor devices -
FIG. 5 illustrates acamera image 500 with object tracking. The monitoring system can track thepersons camera image 500 with theboundary regions persons boundary regions - The camera system can determine processed data that consists of the
boundary regions person - The camera system can track objects using the boundary regions. The camera system can compare the intersection of boundary regions in consecutive frames. A sequence of boundary regions associated with an object through consecutive frames can be referred to as a “track.” The camera system may associate boundary regions if the boundary regions of consecutive frames overlap by a threshold distance or are within of a threshold distance of another. The camera system may determine that boundary regions from consecutive frames that are adjacent (or the closest with each other) are associated with the same object. Thus, whenever object detection occurs in the field of view of one camera, that object may be associated with the nearest track.
- As described herein, the camera system can use one or more computer vision algorithms. For example, a computer vision algorithm can identify a boundary region around a person's face or around a person's body. In some aspects, the camera system can detect faces using a machine learning model, such as, but not limited to, Google's FaceNet. The machine learning model can receive an image of the person's face as input and output a vector of numbers, which can represent features of a face. In some aspects, the camera system can send the extracted facial features to the server. The monitoring system can map the extracted facial features to a person. The vector numbers can represent facial features corresponding to points on ones' face. Facial features of known people (such as clinicians or staff) can be stored in a facial features database, which can be part of the database described herein. To identify an unknown individual, such as a new patient or a visitor, the monitoring system can initially mark the unknown person as unknown and subsequently identify the same person in multiple camera images. The monitoring system can populate a database with the facial features of the new person.
-
FIG. 6 depicts amonitoring system 600 in a home setting. Themonitoring system 600 can include, but is not limited to, one ormore cameras first camera 602 of themonitoring system 600, can be the same as or similar to thecamera system 114 ofFIG. 1A . In some aspects, thecameras monitoring system 600 can be configured to detect apet 610 using the object identification techniques described herein. Themonitoring system 600 can be further configured to determine if apet 610 was fed or if thepet 610 is chewing or otherwise damaging thefurniture 612. In some aspects, themonitoring system 600 can be configured to communicate with a home automation system. For example, if themonitoring system 600 detects that thepet 610 is near a door, themonitoring system 600 can instruct the home automation system to open the door. In some aspects, themonitoring system 600 can provide alerts and/or commands in the home setting to deter a pet from some activity (such as biting a couch, for example). -
FIG. 7 depicts amonitoring system 700 in an infant care setting. Themonitoring system 700 can include one ormore cameras 702. In some aspects, a camera in themonitoring system 700 can send data and/or images to a server (not illustrated). Themonitoring system 700 can be configured to detect aninfant 704 using the object identification techniques described herein. Via a camera, themonitoring system 700 can detect whether a person is within an infant zone, which can be located within a field of view of thecamera 702. Infant zones can be similar to patient zones, as described herein. For example, an infant zone can be defined as a proximity threshold around acrib 706 and/or theinfant 704. In some aspects, a person is within the infant zone if the person is at least partially within a proximity threshold distance to thecrib 706 and/or theinfant 704. Themonitoring system 700 can use object tracking, as described herein, to determine if theinfant 704 is moved. For example, themonitoring system 700 can issue an alert if theinfant 704 leaves thecrib 706. Themonitoring system 700 can include one or more machine learning models. - The
monitoring system 700 can detect whether an unauthorized person is within the infant zone. Themonitoring system 700 can detect whether an unauthorized person is present using one or more methods, such as, but not limited to, facial recognition, identification via an image of an identification tag, and/or RFID based tracking. Identification tag tracking (whether an identification badge, RFID tracking, or some other tracking) can be appliable to hospital-infant settings. In some aspects, themonitoring system 700 can issue an alert based on one or more of the following factors: facial detection of an unrecognized face; no positive visual identification of authorized persons via identification tags; and/or no positive identification of authorized persons via RFID tags. - As described herein, a machine learning model of the
monitoring system 700 can receive an image of a person's face as input and output a vector of numbers, which can represent features of a face. Themonitoring system 700 can map the extracted facial features to a known person. For example, a database of themonitoring system 700 can store a mapping from facial features (but not actual pictures of faces) to person profiles. If themonitoring system 700 cannot match the features to features from a known person, themonitoring system 700 can mark person as unknown and issue an alert. Moreover, themonitoring system 700 can issue another alert if the unknown person moves theinfant 704 outside of a zone. - In some aspects, the
monitoring system 700 can monitor movements of theinfant 704. Themonitoring system 700 can monitor a color of the infant for physiological concerns. For example, the monitoring system can detect a change in color of skin (such as a bluish color) since that might indicate potential asphyxiation. Themonitoring system 700 can use trained machine learning models to identify skin color changes. Themonitoring system 700 can detect a position of theinfant 704. For example, if theinfant 704 rolls onto their stomach, themonitoring system 700 can issue a warning since it may be safer for theinfant 704 to lay on their back. Themonitoring system 700 can use trained machine learning models to identify potentially dangerous positions. In some aspects, a non-invasive sensor device (not illustrated) can be attached to the infant 704 (such as a wristband or a band that wraps around the infant's foot) to monitor physiological parameters of the infant. Themonitoring system 700 can receive the physiological parameters, such as, but not limited to, blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, and/or pleth variability index. In some aspects, themonitoring system 700 can include a microphone that can capture audio data. Themonitoring system 700 can detect sounds from theinfant 704, such as crying. Themonitoring system 700 can issue an alert if the detected sounds are above a threshold decibel level. Additionally or alternatively, themonitoring system 700 can process the sounds with a machine learning model. For example, themonitoring system 700 can convert sound data into spectrograms, input them into a CNN and a linear classifier model, and output a prediction whether the sounds (such as excessive crying) should cause a warning to be issued. In some aspects, themonitoring system 700 can include a thermal camera. Themonitoring system 700 can use trained machine learning models to identify a potentially wet diaper from an input thermal image. -
FIG. 8 is a flowchart of amethod 800 for efficiently applying machine learning models, according to some aspects of the present disclosure. As described herein, a monitoring system, which can include a camera system, may implement aspects of themethod 800 as described herein. Themethod 800 may include fewer or additional blocks and/or the blocks may be performed in order different than is illustrated. - Beginning at
block 802, image data can be received. A camera system (such as thecamera systems FIGS. 1A, 3 described herein) can receive image data from a camera. Depending on the type of camera and configuration of the camera, the camera system can receive different types of images, such as 4K, 1080p, 8MP images. Image data can also include, but is not limited to, a sequence of images. A camera in a camera system can continuously capture images. Therefore, the camera in a camera system can capture images of objects (such as a patient, a clinician, an intruder, the elderly, an infant, a youth, or a pet) in a room either at a clinical facility, a home, or an assisted living home. - At
block 806, a person detection model can be applied. The camera system can apply the person detection model based on the image data. In some aspects, the camera system can invoke the person detection model on a hardware accelerator. The hardware accelerator can be configured to accelerate the application of machine learning models, including a person detection model. The person detection model can be configured to receive image data as input. The person detection model can be configured to output a classification result. In some aspects, the classification result can indicate a likelihood (such as a percentage chance) that the image data includes a person. In other aspects, the classification result can be a binary result: either the object is predicted as present in the image or not. The person detection model can be, but is not limited to, a CNN. The person detection model can be trained to detect persons. For example, the person detection model can be trained with a training data set with labeled examples indicating whether the input data includes a person or not. - At
block 808, it can be determined whether a person is present. The camera system can determine whether a person is present. The camera system can determine whether a person object is located in the image data. The camera system can receive from the person detection model (which can execute on the hardware accelerator) the output of a classification result. In some aspects, the output can be a binary result, such as, “yes” there is a person object present or “no” there is not a person object present. In other aspects, the output can be a percentage result and the camera system can determine the presence of a person if the percentage result is above a threshold. If a person is detected, themethod 800 proceeds to theblock 810 to receive second image data. If a person is not detected, themethod 800 proceeds to repeat theprevious blocks - At
block 810, second image data can be received. Theblock 810 for receiving the second image data can be similar to the previous block for receiving image data. Moreover, the camera in the camera system can continuously capture images, which can lead to the second image data. As described herein, the image data can include multiple images, such as a sequence of images. - At
block 812, one or more person safety models can be applied. In response to detecting a person, the camera system can apply one or more person safety models. The camera system can invoke (which can be invoked on a hardware accelerator) a fall detection model based on the second image data. The fall detection model can output a classification result. In some aspects, the fall detection model can be or include a CNN. The camera system can pre-process the image data. In some aspects, the camera system can covert an image into an RGB image, which can be a m-by-n-by-3 data array that defines red, green, and blue color components for each individual pixel in the image. In some aspects, the camera system can compute an optical flow from the image data (such as the RGB images), which can be a two-dimensional vector field between two images. The two-dimensional vector field can show how the pixels of an object in the first image move to form the same object in the second image. The fall detection model can be pre-trained to perform feature extraction and classification of the image data (which can be pre-processed image data) to output a classification result. In some aspects, the fall detection model can be made of various layers, such as, but not limited to, a convolution layer, a max pooling layer, and a regularization layer, and a classifier, such as, but not limited to, a softmax classifier. - As described herein, in some aspects, an advantage of performing the
previous blocks - At
block 814, it can be determined whether there is a person safety issue. The camera system can determine whether there is a person safety issue. As described above, for each person safety model that is invoked, the camera system can receive a model result as output. For some models, the output can be a binary result, such as, “yes” a fall has been detected or “no” a fall has not been detected. For other models, the output can be a percentage result and the camera system can determine a person safety issue exists if the percentage result is above a threshold. In some aspects, evaluation of the one or more person safety models can result in an issue detection if at least one model returns a result that indicates issue detection. If a person safety issue is detected, themethod 800 proceeds to block 816 to provide an alert and/or take an action. If a person safety issue is not detected, themethod 800 proceeds to repeat theprevious blocks - At
block 816, an alert can be provided and/or an action can be taken. In some aspects, the camera system can initiate an alert. The camera system can notify a monitoring system to provide an alert. In some aspects, auser computing device 102 can receive an alert about a safety issue. In some aspects, aclinician 110 can receive an alert about the safety issue. In some aspects, the camera system can initiate an action. The camera system can cause the monitoring system to take an action. For example, the monitoring system can automatically notify emergency services (such as an emergency hotline and/or an ambulance service) to send someone to help. -
FIG. 9 is a flowchart of anothermethod 900 for efficiently applying machine learning models, according to some aspects of the present disclosure. As described herein, a monitoring system, which can include a camera system, may implement aspects of themethod 900 as described herein. Themethod 900 may include fewer or additional blocks and/or the blocks may be performed in order different than is illustrated. The block(s) of themethod 900 ofFIG. 9 can be similar to the block(s) of themethod 800 ofFIG. 8 . In some aspects, the block(s) of themethod 900 ofFIG. 9 can be used in conjunction with the block(s) of themethod 800 ofFIG. 8 . - Beginning at
block 902, a training data set can be received. The monitoring system can receive a training data set. In some aspects, a first set of videos of person falls can be collected and a second set of videos of persons without falling can be collected. A training data set can be created with the first set of videos and the second set of videos. Other training data sets can be created. For example, for machine learning of handwashing, a first set of videos of with handwashing and a second set of videos without handwashing can be collected; and a training data set can be created from the first set of videos and the second set of videos. For machine learning detection of dilated pupils, a first set of images of with dilated pupils and a second set of images without dilated pupils can be collected; and a training data set can be created from the first set of images and the second set of images. For machine learning detection of facial paralysis, a first set of images of with facial paralysis and a second set of images without facial paralysis can be collected; and a training data set can be created from the first set of images and the second set of images. For machine learning detection of an infant, a first set of images of with an infant and a second set of images without an infant can be collected; and a training data set can be created from the first set of images and the second set of images. For machine learning detection of an infant's position, a first set of images of an infant on their back and a second set of images of an infant on their stomach or their side; and a training data set can be created from the first set of images and the second set of images. For machine learning detection of an unconsciousness state, a first set of videos of persons in an unconscious state and a second set of videos of a person in a state of consciousness; and a training data set can be created from the first set of videos and the second set of videos. For other machine learning detection of an unconsciousness state, a first set of audio recordings of persons in an unconscious state and a second set of audio recordings of a person in a state of consciousness; and a training data set can be created from the first set of audio recordings and the second set of audio recordings. The monitoring system can receive training data sets for any of the machine learning models described herein that can be trained with supervised machine learning. - At
block 904, a machine learning model can be trained. The monitoring system can train one or more machine learning models. The monitoring system can train a fall detection model using the training data set from theprevious block 902. The monitoring system can train a handwashing detection model using the training data set from theprevious block 902. The monitoring system can train any of the machine learning models described herein that use supervised machine learning. - In some aspects, the monitoring system can train a neural network, such as, but not limited to, a CNN. The monitoring system can initiate the neural network with random weights. During the training of the neural network, the monitoring system feeds labelled data from the training data set to the neural network. Class labels can include, but are not limited to, fall, no fall, hand washing, no hand washing, loud noise, no loud noise, normal pupils, dilated pupils, no facial paralysis, facial paralysis, infant, no infant, supine position, prone position, side position, unconscious, conscious, etc. The neural network can process each input vector with its values being assigned randomly and then make comparisons with the class label of the input vector. If the output prediction does not match the class label, an adjustment to the weights of the neural network neurons are made so that output correctly matches the class label. The corrections to the value of weights can be made through a technique, such as, but not limited to backpropagation. Each run of training of the neural network can be called an “epoch.” The neural network can go through several series of epochs during the process of training, which results in further adjusting of the neural network weights. After each epoch step, the neural network can become more accurate at classifying and correctly predicting the class of the training data. After training the neural network, the monitoring system can use a test dataset to verify the neural network's accuracy. The test dataset can be a set of labelled test data that were not included in the training process. Each test vector can be fed to the neural network, and the monitoring system can compare the output to the actual class label of the test input vector.
- At
block 906, input data can be received. The camera system can receive input data. In some aspects, theblock 906 for receiving input data can be similar to theblock 802 ofFIG. 8 for receiving image data. The camera system can receive image data from a camera. In some aspects, other input data can be received. For example, the camera system can receive a current time. The camera system can receive an RFID signal (which can be used for identification purposes, as described herein). The camera system can receive physiological values (such as blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, and/or pleth variability index) from a patient sensor device, such as a wearable device. - At
block 908, it can be determined whether a trigger has been satisfied. The camera system can determine whether a trigger has been satisfied to apply one or more machine learning models. In some aspects, the camera system can determine whether a trigger has been satisfied by checking whether a person has been detected. In some aspects, the camera system can determine whether a trigger has been satisfied by checking whether the current time satisfies a trigger time window, such as, but not limited to, a daily time check-up window. If a trigger is satisfied, themethod 900 proceeds to theblock 910 to receive captured data. If a trigger is not detected, themethod 900 proceeds to repeat theprevious blocks - In some aspects, a trigger can be determined based on a received physiological value. The camera system can determine to begin a monitoring process based on a physiological value. In some aspects, the wearable device can include a pulse oximetry sensor and the physiological value is for blood oxygen saturation. The camera system can determine that the physiological value is below a threshold level (such as blood oxygen below 88%, 80%, or 70%, etc.). In some aspects, the wearable device can include a respiration rate sensor and the physiological value is for respiration rate. The camera system can determine that the physiological value satisfies a threshold alarm level (such as respiration rate under 12 or over 25 breaths per minute). In some aspects, the wearable device can include a heart rate sensor, the physiological value is for heart rate, and the multiple physiological values measuring heart rate over time can be received from the wearable device. The camera system can determine that the physiological values satisfies a threshold alarm level, such as, but not limited to, heart rate being above 100 beats per minute for a threshold period of time or under a threshold level for threshold period of time.
- At
block 910, captured data can be received. Theblock 910 for receiving captured data can be similar to theprevious block 906 for receiving input data. Moreover, the camera in the camera system can continuously capture images, which can lead to the captured data. In some aspects, the camera system can receive audio data from a microphone. In some aspects, the camera system can be configured to cause presentation, on a display, of a prompt to cause a person to perform an activity. The camera system can receive, from a camera, image data of a recording of the activity. - At
block 912, one or more machine learning models can be applied. In response to determining that a trigger has been satisfied , the camera system can apply one or more machine learning models based on the captured data. The camera system can invoke (which can be invoked on a hardware accelerator) one or more machine learning models, which can output a model result. The camera system can invoke a fall detection model based on image data where the fall detection model can output a classification result. The camera system can invoke a loud noise detection model based on the audio data where the loud noise detection model can output a classification result. In some aspects, the camera system can generate a spectrogram data from the audio data and provide the spectrogram data as input to the loud noise detection model. The camera system can invoke a facial feature extraction model based on the image data where the facial feature extraction model can output a facial feature vector. The camera system can invoke a handwashing detection model based on the image data where the handwashing detection model can output a classification result. The camera system can invoke a screening machine learning model based on image data where the screening machine learning model can output a model result. The screening machine learning model can include, but is not limited to, a pupillometry screening model or a facial paralysis screening model. - In some aspects, in response to determining to begin the monitoring process, the camera system can invoke one or more machine learning models. The camera system can invoke (which can be on a hardware accelerator) a first unconscious detection model based on the image data where the first unconscious detection model outputs a first classification result. The camera system can invoke (which can be on the hardware accelerator) a second unconscious detection model based on the audio data where the second unconscious detection model outputs a second classification result.
- At
block 914, it can be determined whether there is a safety issue. The camera system can determine whether there is a safety issue. For each machine learning model that is invoked, the camera system can receive a classification result as output. For some models, the output can be a binary result, such as, “yes” a fall has been detected or “no” a fall has not been detected. For other models, the output can be a percentage result and the camera system can determine a safety issue exists if the percentage result is above a threshold. In some aspects, evaluation of the one or more machine learning models can result in an issue detection if at least one model returns a result that indicates issue detection. The camera system can detect a potential fall based on the classification result. The camera system can detect a potential scream or loud noise based on the classification result from a loud noise detection model. The camera system can execute a query of a facial features database based on the facial feature vector where executing the query can indicate that the facial feature vector is not present in a facial features database, which can indicate a safety issue. The camera system can detect a potential screening issue based on the classification result. The potential screening issue can indicate, but is not limited to, potential dilated pupils or potential facial paralysis. In some aspects, based on the output from one or more machine learning models, the camera system can detect a potential state of unconsciousness. If a safety issue is detected, themethod 900 proceeds to block 916 to provide an alert and/or take an action. If a safety issue is not detected, themethod 900 proceeds to repeat theprevious blocks - At
block 916, an alert can be provided and/or an action can be taken. In some aspects, the camera system can initiate an alert. The camera system can notify a monitoring system to provide an alert. In some aspects, the camera system can initiate an action. In some aspects, theblock 916 for providing an alert and/or taking an action can be similar to theblock 816 ofFIG. 8 for providing an alert and/or taking an action. In response to detecting an issue, such as, but not limited to, detecting a potential fall, loud noise, scream, lack of handwashing, dilated pupils, facial paralysis, intruder, state of unconsciousness, etc., the monitoring system can provide an alert. The monitoring system can escalate alerts. For example, in response to detecting a potential fall and a potential scream or loud noise, the monitoring system can provide an escalated alert. The camera system can cause the monitoring system to take an action. For example, the monitoring system can automatically notify emergency services (such as an emergency hotline and/or an ambulance service) to send someone to help. - In some aspects, the monitoring system can allow privacy options. For example, some user profiles can specify that the user computing devices associated with those profiles should not receive alerts (which can be specified for a period of time). However, the monitoring system can include an alert escalation policy such that alerts can be presented via user computing devices based on one or more escalation conditions. For example, if an alert isn't responded to a for a period of time, the monitoring system can escalate the alert. As another example, if a quantity of alerts exceed a threshold, then the monitoring system can present an alert via user computing devices despite user preferences otherwise.
- At
block 918, a communications system can be provided. The monitoring system can provide a communications system. The camera system can receive, from a computing device, first video data (such as, but not limited to, video data of a clinician, friends, or family of a patient). The camera system can cause presentation, on the display, of the first video data. The camera system can receive, from the camera, second video data and transmit, to the computing device, the second video data. - Some of the aspects described herein can be directed towards elderly care features. The monitoring systems described herein can be applied to assisted living and/or home settings for the elderly. The monitoring systems described herein, which can include camera systems, can generally monitor activities of the elderly. The monitoring systems described herein can initiate check-up processes, including, but not limited to, dementia checks. In some aspects, a check-up process can detect a color of skin to detect possible physiological changes. The monitoring system can perform stroke detection by determining changes in facial movements and/or speech patterns. The monitoring system can track medication administration and provide reminders if medication is not taken. For example, the monitoring can monitor a cupboard or medicine drawer and determine whether medication is taken based on activity in those areas. In some aspects, some of the camera systems can be outdoor camera systems. The monitoring system can track when a person goes for a walk, log when the person leaves and returns, and potentially issues an alert if a walk exceeds a threshold period of time. In some aspects, the monitoring system can track usage of good hygiene practices, such as but not limited to, handwashing, brushing teeth, or showering (e.g., tracking that a person enters a bathroom at a showering time). The monitoring system can keep track of whether a person misses a check-up. In some aspects, a camera system can include a thermal camera, which can be used to identify a potentially wet adult diaper from an input thermal image.
- With respect to
FIG. 9 , themethod 900 for efficiently applying machine learning models can be applied to elderly care settings. Atblock 902, a training data set can be received. The monitoring system can receive a training data set, which can be used to train machine learning models to be used in check-up processes for the elderly, such as checking for dilated pupils or facial paralysis. For machine learning of dilated pupils, a first set of images of with dilated pupils and a second set of images without dilated pupils can be collected; and a training data set can be created from the first set of images and the second set of images. For machine learning of facial paralysis, a first set of images of with facial paralysis and a second set of images without facial paralysis can be collected; and a training data set can be created from the first set of images and the second set of images. - At
block 904, a machine learning model can be trained. A server in the monitoring system can train a pupillometry screening model using the training data set. The server in the monitoring system can train a facial paralysis screening model using the training data set. - At
block 906, input data can be received. The camera system can receive input data, which can be used to determine if a trigger has been satisfied for application of one or more machine learning models. The camera system can receive image data from a camera. The camera system can receive a current time. The camera system can receive an RFID signal, which can be used for person identification and/or detection. - In some aspects, the monitoring system can include patient sensor devices, such as, but not limited to, wearable devices. The wearable device can be configured to process sensor signals to determine a physiological value for the person. The monitoring system can receive a physiological value from the wearable device. In some aspects, the wearable device can include a pulse oximetry sensor and the physiological value can be for blood oxygen saturation. In some aspects, the wearable device can be configured to process the sensor signals to measure at least one of blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, or pleth variability index. Some of the wearable devices can be used for an infant.
- At
block 908, it can be determined whether a trigger has been satisfied. The camera system can determine whether a trigger has been satisfied to apply one or more machine learning models. The camera system can determine whether a check-up process should begin from a current time. For example, the monitoring system can conduct check-up processes at regular intervals, such as once or two a day, which can be at particular times, such as a morning check-up time or an afternoon check-up time. As described herein, another trigger type can be detection of a person. The camera system can invoke a person detection model based on image data where the person detection model outputs classification result; and detect a person based on the classification result. If a trigger is satisfied, themethod 900 proceeds to theblock 910 to receive captured data. If a trigger is not detected, themethod 900 proceeds to repeat theprevious blocks - At
block 910, captured data can be received. In response to determining to begin the check-up process, the monitoring system can cause presentation, on a display, of a prompt to cause a person to perform a check-up activity. In some aspects, the check-up activity can check for signs of dementia. A check-up activity can include having a person standing a particular distance from the camera system. A check-up activity can include simple exercises. The prompts can cause a user to say something or perform tasks. The person can be prompted to perform math tasks, pattern recognition, solve puzzles, and/or identify photos of family members. For example, the person can be prompted to point to sections of the display, which can correspond to answers to check-up tests. The check-up tests can check for loss of motor skills. In some aspects, the check-up activity can include a virtual physical or appointment conducted by a clinician. The camera system can receive, from the camera, image data of a recording of the check-up activity. In some aspects, the camera system can receive other input, such as, but not limited to, audio data from a microphone. - At
block 912, one or more machine learning models can be applied. In response to determining that a trigger has been satisfied , the camera system can apply one or more machine learning models based on the captured data. In some aspects, in response to determining to begin the check-up process, the camera system can invoke a screening machine learning model based on image data where the screening machine learning model can output a model result (such as a classification result). The screening machine learning model can include, but is not limited to, a pupillometry screening model, a facial paralysis screening model, or a gesture detection model. The gesture detection model can be configured to detect a gesture directed towards a portion of the display. For example, during a dementia test, the person can be prompted to point to a portion of the display and the gesture detection model can identify a point gesture, such as but not limited to, pointing to a quadrant on the display. In some aspects, in response to detecting a person, the camera system can invoke a handwashing detection model based on image data wherein the handwashing detection model outputs a classification result. - At
block 914, it can be determined whether there is a safety issue. The camera system can determine whether there is a safety issue, such as a potential screening issue. The camera system can detect a potential screening issue based on the model result. The potential screening issue can indicate, but is not limited to, potential dilated pupils or potential facial paralysis. The monitoring system can determine whether there is a potential screening issue based on output from a gesture detection model. For example, the monitoring system can use detected gesture to determine an answer and an incorrect answer can indicate a potential screening issue. If a safety issue is detected, themethod 900 proceeds to block 916 to provide an alert and/or take an action. If a safety issue is not detected, themethod 900 proceeds to repeat theprevious blocks - At
block 916, an alert can be provided. In some aspects, the camera system can initiate an alert. The camera system can notify a monitoring system to provide one or more alerts. In response to detecting an issue in an elderly care setting, such as, but not limited to, detecting a potential fall, loud noise, scream, lack of handwashing, dilated pupils, facial paralysis, intruder, etc., the monitoring system can provide an alert. The monitoring system can escalate alerts. For example, in response to detecting a potential fall and a potential scream or loud noise, the monitoring system can provide an escalated alert. In some aspects, the monitoring system can provide alerts via different networks (such as Wi-Fi or cellular) and/or technologies (such as Bluetooth). - At
block 918, a communications system can be provided. The monitoring system can provide a communications system in an elderly care setting. The camera system can receive, from a computing device, first video data (such as, but not limited to, video data of a clinician, friends, or family of a patient). The camera system can cause presentation, on the display, of the first video data. The camera system can receive, from the camera, second video data and transmit, to the computing device, the second video data. - Some of the aspects described herein can be directed towards infant care features. The monitoring systems described herein can be applied to monitoring an infant.
FIG. 10 is a flowchart of amethod 1000 for efficiently applying machine learning models for infant care, according to some aspects of the present disclosure. As described herein, a monitoring system, which can include a camera system, may implement aspects of themethod 1000 as described herein. The block(s) of themethod 1000 ofFIG. 10 can be similar to the block(s) of themethods FIGS. 8 and/or 9 . Themethod 1000 may include fewer or additional blocks and/or the blocks may be performed in order different than is illustrated. - Beginning at
block 1002, image data can be received. A camera system can receive image data from a camera, which can be positioned in an infant area, such as a nursery. Image data can also include, but is not limited to, a sequence of images. A camera in a camera system can continuously capture images of the infant area. Therefore, the camera in a camera system can capture images of objects, such as an infant, in a room either at a home or a clinical facility. - At
block 1006, an infant detection model can be applied. The camera system can apply the infant detection model based on the image data. In some aspects, the camera system can invoke the infant detection model on a hardware accelerator. The infant detection model can be configured to receive image data as input. The infant detection model can be configured to output a classification result. In some aspects, the classification result can indicate a likelihood (such as a percentage chance) that the image data includes an infant. In other aspects, the classification result can be a binary result: either the infant object is predicted as present in the image or not. The infant detection model can be, but is not limited to, a CNN. The infant detection model can be trained to detect persons. For example, the infant detection model can be trained with a training data set with labeled examples indicating whether the input data includes an infant or not. - At
block 1008, it can be determined whether an infant is present. The camera system can determine whether an infant is present. The camera system can determine whether an infant object is located in the image data. The camera system can receive from the infant detection model the output of a classification result. In some aspects, the output can be a binary result, such as, “yes” there is an infant object present or “no” there is not an infant object present. In other aspects, the output can be a percentage result and the camera system can determine the presence of an infant if the percentage result is above a threshold. If an infant is detected, themethod 1000 proceeds to theblock 1010 to receive captured data. If an infant is not detected, themethod 1000 proceeds to repeat theprevious blocks - At
block 1010, captured data can be received. The camera in the camera system can continuously capture images, which can lead to the captured data. In some aspects, the camera system can receive audio data from a microphone. - At
block 1012, one or more infant safety models can be applied. In response to detecting an infant, the camera system can apply one or more infant safety models that outputs a model result. The camera system can invoke (which can be invoked on a hardware accelerator) an infant position model based on the captured data. The infant position model can output a classification result. In some aspects, the infant position model can be or include a CNN. In response to detecting an infant, the camera system can invoke a facial feature extraction model based on second image data where the facial feature extraction model outputs a facial feature vector. The camera system can execute a query of a facial features database based on the facial feature vector where executing the query indicates that the facial feature vector is not present in the facial features database. An infant safety model can be an infant color detection model. In some aspects, the model result can include coordinates of a boundary region identifying an infant object in the image data. As described herein, the camera system can invoke a loud noise detection model based on the audio data where the loud noise detection model can output a classification result. - At
block 1014, it can be determined whether there is an infant safety issue. The camera system can determine whether there is an infant safety issue. As described above, for each person safety model that is invoked, the camera system can receive a model result as output. For some models, the output can be a binary result, such as, “yes” an infant is in a supine position or “no” a supine position has not been detected (such as the infant potentially laying on their stomach). For other models, the output can be a percentage result and the camera system can determine an infant safety issue exists if the percentage result is above a threshold. The camera system can determine that an unrecognized person has been detected. In some aspects, the camera system determine that the coordinates of the boundary region exceed a threshold distance from an infant zone (which can indicate that an infant is being removed from the infant zone). The camera system can determine a potential scream from the model result. In some aspects, evaluation of the one or more infant safety models can result in an issue detection if at least one model returns a result that indicates issue detection. If an infant safety issue is detected, themethod 1000 proceeds to block 1016 to provide an alert and/or take an action. If an infant safety issue is not detected, themethod 1000 proceeds to repeat theprevious blocks - At
block 1016, an alert can be provided and/or an action can be taken. In some aspects, the camera system can initiate an alert associated with the infant. The camera system can notify a monitoring system to provide an alert. In some aspects, auser computing device 102 can receive an alert about an infant safety issue. In some aspects, aclinician 110 can receive an alert about the infant safety issue. In some aspects, the camera system can initiate an action. The camera system can cause the monitoring system to take an action. For example, the monitoring system can automatically notify emergency services (such as an emergency hotline and/or an ambulance service) to send someone to help. - Some of the aspects described herein can be directed towards at-home monitoring features. The monitoring systems described herein can be applied to monitoring in a home. The monitoring system can accomplish one or more of the following features using the machine learning techniques described herein. The monitoring system can monitor the time spent on various tasks by members of a household (such as youth at home), such as time spent watching television or time spent studying. The monitoring system can be configured to confirm that certain tasks (such as chores) are completed. In some aspects, the monitoring system can allow parents to monitor an amount of time spent using electronics. In some aspects, the camera system can be configured to detect night terrors and amount and types of sleep. As described herein, in some aspects, the monitoring system can track usage of good hygiene practices at home, such as but not limited to, handwashing, brushing teeth, or showering (e.g., tracking that a person enters a bathroom at a showering time). As described herein, zones can be used to provide alerts, such as monitoring a pool zone or other spaces youth should not be allowed, such as, but not limited to, certain rooms at certain times and/or unaccompanied by an adult. For example, the camera system can monitor a gun storage location to alert adults to unauthorized access of weapons.
- Some of the aspects described herein can include any of the following features, which can be applied in different settings. In some aspects, a camera system can have local storage for an image and/or video feed. In some aspects, remote access of the local storage may be restricted and/or limited. In some aspects, the camera system can use a calibration factor which can be useful for correcting color drift in the image data from a camera. In some aspects, the camera system can add or remove filters on camera to provide certain effects. The camera system may include infrared filters. In some aspects, the monitoring system can monitor food intake of subject and/or estimate calories. In some aspects, the monitoring system can detect mask wearing (such as wearing or not wearing an oxygen mask).
- The monitoring system can perform one or more check-up tests. The monitoring system, using a machine learning model, can detect slurred speech, drunkenness, drug use, and/or adverse behavior. Based on other check-up tests the monitoring system can detect shaking, microtremors, tremors, which can indicate a potential disease state such as Parkinson's. The monitoring system can track exercise movements to determine a potential physiological condition. A check-up test can be used by the monitoring system for a cognitive assessment, such as, detecting vocabulary decline. In some aspects, the monitoring system can check a user's smile where the monitoring system prompts the user to stand a specified distance away from the camera system. A check-up test can request a subject to do one or more exercise, read something outload (to test muscles of a face), reach for an object. In some aspects, the camera system can perform an automated physical, perform a hearing test, and/or perform an eye test. In some aspects, a check-up test can be for Alzheimer's detection. The monitoring system can provide memory exercises, monitor for good/bad days, and/or monitor basic behavior to prevent injury. In some aspects, the camera system can monitor skin color changes to detect skin damage and/or sunburn detection. The camera system can take a trend of skin color, advise or remind to take corrective action, and/or detect a tan line. The monitoring system can monitor sleep cycles and/or heart rate variability. In some aspects, the monitoring system can monitor snoring, rapid eye movement (REM), and/or sleep quality, which can be indicative of sleep apnea or another disease. As described herein, the camera system can be tried to detect sleep walking. The camera system can be configured to detect coughing or sneezing to determine potential allergies or illness. The camera system can also provide an alert if a possible hyperventilation is detected. Any of the monitoring features described herein can be implemented with the machine learning techniques described herein.
-
FIG. 11 is a block diagram that illustrates example components of acomputing device 1100, which can be a camera system. Thecomputing device 1100 can implement aspects of the present disclosure, and, in particular, aspects of themonitoring system camera system 114. Thecomputing device 1100 can communicate with other computing devices. - The
computing device 1100 can include ahardware processor 1102, a hardware accelerator, adata storage device 1104, amemory device 1106, abus 1108, adisplay 1112, one or more input/output devices 1114, and acamera 1118. Aprocessor 1102 can also be implemented as a combination of computing devices, e.g., a combination of a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor, or any other such configuration. Theprocessor 1102 can be configured, among other things, to process data, execute instructions to perform one or more functions, such as apply one or more machine learning models, as described herein. Thehardware accelerator 1116 can be special hardware that is configured to accelerate machine learning applications. Thedata storage device 1104 can include a magnetic disk, optical disk, or flash drive, etc., and is provided and coupled to thebus 1108 for storing information and instructions. Thememory 1106 can include one or more memory devices that store data, including without limitation, random access memory (RAM) and read-only memory (ROM). Thecomputing device 1100 may be coupled via thebus 1108 to adisplay 1112, such as an LCD display or touch screen, for displaying information to a user, such as a patient. Thecomputing device 1100 may be coupled via thebus 1108 to one or more input/output devices 1114. Theinput device 1114 can include, but is not limited to, a keyboard, mouse, digital pen, microphone, touch screen, gesture recognition system, voice recognition system, imaging device (which may capture eye, hand, head, or body tracking data and/or placement), gamepad, accelerometer, or gyroscope. Thecamera 1118 can include, but is not limited to, a 1080p or 4k camera and/or an infrared image camera. - As used herein, the term “patient” can refer to any person that is monitored using the systems, methods, devices, and/or techniques described herein. As used herein, a “patient” is not required—to be admitted to a hospital, rather, the term “patient” can refer to a person that is being monitored. As used herein, in some cases the terms “patient” and “user” can be used interchangeably.
- While some features described herein may be discussed in a specific context, such as adult, youth, infant, elderly, or pet care, those features can be applied to other contexts, such as, but not limited to, a different one of adult, youth, infant, elderly, or pet care contexts.
- The apparatuses and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions that are stored on a non-transitory tangible computer readable medium. The computer programs may also include stored data. Non-limiting examples of the non-transitory tangible computer readable medium are nonvolatile memory, magnetic storage, and optical storage.
- The term “substantially” when used in conjunction with the term “real-time” forms a phrase that will be readily understood by a person of ordinary skill in the art. For example, it is readily understood that such language will include speeds in which no or little delay or waiting is discernible, or where such delay is sufficiently short so as not to be disruptive, irritating, or otherwise vexing to a user.
- Conditional language used herein, such as, among others, “can,” “might,” “may,” “e.g.,” “for example,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain aspects described herein include, while other aspects described herein do not include, certain features, elements, or states. Thus, such conditional language is not generally intended to imply that features, elements, or states are in any way required for one or more aspects described herein.
- Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is otherwise understood with the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z). Such disjunctive language is not generally intended to, and should not, imply that certain aspects require at least one of X, at least one of Y, or at least one of Z to each be present. Thus, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Further, the term “each,” as used herein, in addition to having its ordinary meaning, can mean any subset of a set of elements to which the term “each” is applied.
- The term “a” as used herein should be given an inclusive rather than exclusive interpretation. For example, unless specifically noted, the term “a” should not be understood to mean “exactly one” or “one and only one”; instead, the term “a” means “one or more” or “at least one,” whether used in the claims or elsewhere in the specification and regardless of uses of quantifiers such as “at least one,” “one or more,” or “a plurality” elsewhere in the claims or specification.
- The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth.
- While the above detailed description has shown, described, and pointed out novel features as applied to various aspects described herein, it will be understood that various omissions, substitutions, and changes in the form and details of the devices or algorithms illustrated can be made without departing from the spirit of the disclosure. As will be recognized, certain aspects described herein can be embodied within a form that does not provide all of the features and benefits set forth herein, as some features can be used or practiced separately from others.
Claims (20)
1. A system comprising:
a storage device configured to store instructions;
a display;
a camera; and
a hardware processor configured to execute the instructions to:
receive a current time;
determine to begin a check-up process from the current time; and
in response to determining to begin the check-up process,
cause presentation, on the display, of a prompt to cause a person to perform a check-up activity,
receive, from the camera, image data of a recording of the check-up activity,
invoke a screening machine learning model based on the image data, wherein the screening machine learning model outputs a classification result,
detect a potential screening issue based on the classification result, and
in response to detecting the potential screening issue, provide an alert.
2. The system of claim 1 , wherein the screening machine learning model is a pupillometry screening model, and wherein the potential screening issue indicates potential dilated pupils.
3. The system of claim 1 , wherein the screening machine learning model is a facial paralysis screening model, and wherein the potential screening issue indicates potential facial paralysis.
4. The system of claim 1 , further comprising a wearable device configured to process sensor signals to determine a physiological value for the person, wherein the hardware processor is configured to execute further instructions to:
receive, from the wearable device, the physiological value; and
generate the alert comprising the physiological value.
5. The system of claim 4 , wherein the wearable device comprises a pulse oximetry sensor and the physiological value is for blood oxygen saturation.
6. The system of claim 4 , wherein the wearable device is further configured to process the sensor signals to measure at least one of blood oxygen saturation, pulse rate, perfusion index, respiration rate, heart rate, or pleth variability index.
7. The system of claim 1 , wherein the hardware processor is configured to execute further instructions to:
receive, from a second computing device, first video data;
cause presentation, on the display, of the first video data;
receive, from the camera, second video data; and
transmit, to the second computing device, the second video data.
8. A method comprising:
receiving a current time;
determining to begin a check-up process from the current time; and
in response to determining to begin the check-up process,
causing presentation, on a display, of a prompt to cause a person to perform a check-up activity,
receiving, from a camera, image data of a recording of the check-up activity,
invoking a screening machine learning model based on the image data, wherein the screening machine learning model outputs a model result,
detecting a potential screening issue based on the model result, and
in response to detecting the potential screening issue, providing an alert.
9. The method of claim 8 , wherein the screening machine learning model is a pupillometry screening model, and wherein the potential screening issue indicates potential dilated pupils, further comprising:
collecting a first set of images of dilated pupils;
collecting a second set of images without dilated pupils;
creating a training data set comprising the first set of images and the second set of images; and
training the pupillometry screening model using the training data set.
10. The method of claim 8 , wherein the screening machine learning model is a facial paralysis screening model, and wherein the potential screening issue indicates potential facial paralysis, further comprising:
collecting a first set of images of facial paralysis;
collecting a second set of images without facial paralysis;
creating a training data set comprising the first set of images and the second set of images; and
training the facial paralysis screening model using the training data set.
11. The method of claim 8 , wherein the check-up activity comprises a dementia test, and wherein the screening machine learning model comprises a gesture detection model.
12. The method of claim 11 , wherein the gesture detection model is configured to detect a gesture directed towards a portion of the display.
13. The method of claim 8 , further comprising:
receiving, from the camera, second image data;
invoking a person detection model based on the second image data, wherein the person detection model outputs first classification result;
detect a person based on the first classification result;
receive, from the camera, third image data; and
in response to detecting the person,
invoking a handwashing detection model based on the third image data, wherein the handwashing detection model outputs a second classification result,
detecting a potential lack of handwashing based on the second classification result, and
in response to detecting a lack of handwashing, provide a second alert.
14. A system comprising:
a storage device configured to store instructions;
a camera; and
a hardware processor configured to execute the instructions to:
receive, from the camera, first image data;
invoke an infant detection model based on the first image data, wherein
the infant detection model outputs a classification result;
detect an infant based on the classification result;
receive captured data; and
in response to detecting the infant, invoke an infant safety model based on the captured data,
wherein the infant safety model outputs a model result,
detect a potential safety issue based on the model result, and
in response to detecting the potential safety issue, provide an alert.
15. The system of claim 14 , wherein the infant safety model is an infant position model, and wherein the potential safety issue indicates the infant potentially laying on their stomach.
16. The system of claim 15 , wherein the hardware processor is configured to execute further instructions to:
receive, from the camera, second image data; and
in response to detecting the infant, invoke a facial feature extraction model based on the second image data,
wherein the facial feature extraction model outputs a facial feature vector,
execute a query of a facial features database based on the facial feature vector, wherein executing the query indicates that the facial feature vector is not present in the facial features database, and
in response to determining that the facial feature vector is not present in the facial features database, provide an unrecognized person alert.
17. The system of claim 14 , wherein the infant safety model is an infant color detection model, and wherein the potential safety issue indicates potential asphyxiation.
18. The system of claim 14 , wherein the model result comprises coordinates of a boundary region identifying an infant object in the captured data, and wherein detecting the potential safety issue comprises:
determining that the coordinates of the boundary region exceed a threshold distance from an infant zone.
19. The system of claim 14 , further comprising a wearable device configured to process sensor signals to determine a physiological value for the infant, wherein the hardware processor is configured to execute further instructions to:
receive, from the wearable device, the physiological value; and
generate the alert comprising the physiological value.
20. The system of claim 14 , further comprising a microphone, wherein the captured data is received from the microphone, wherein the infant safety model is a loud noise detection model, and wherein the potential safety issue indicates a potential scream.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/153,173 US20230222805A1 (en) | 2022-01-11 | 2023-01-11 | Machine learning based monitoring system |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202263298569P | 2022-01-11 | 2022-01-11 | |
US202263299168P | 2022-01-13 | 2022-01-13 | |
US18/153,173 US20230222805A1 (en) | 2022-01-11 | 2023-01-11 | Machine learning based monitoring system |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230222805A1 true US20230222805A1 (en) | 2023-07-13 |
Family
ID=87069870
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/153,108 Pending US20230222887A1 (en) | 2022-01-11 | 2023-01-11 | Machine learning based monitoring system |
US18/153,173 Pending US20230222805A1 (en) | 2022-01-11 | 2023-01-11 | Machine learning based monitoring system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/153,108 Pending US20230222887A1 (en) | 2022-01-11 | 2023-01-11 | Machine learning based monitoring system |
Country Status (1)
Country | Link |
---|---|
US (2) | US20230222887A1 (en) |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11857315B2 (en) | 2006-10-12 | 2024-01-02 | Masimo Corporation | Patient monitor capable of monitoring the quality of attached probes and accessories |
US11864922B2 (en) | 2015-09-04 | 2024-01-09 | Cercacor Laboratories, Inc. | Low-noise sensor system |
US11872156B2 (en) | 2018-08-22 | 2024-01-16 | Masimo Corporation | Core body temperature measurement |
US11879960B2 (en) | 2020-02-13 | 2024-01-23 | Masimo Corporation | System and method for monitoring clinical activities |
US11886858B2 (en) | 2017-02-24 | 2024-01-30 | Masimo Corporation | Medical monitoring hub |
US11894640B2 (en) | 2015-02-06 | 2024-02-06 | Masimo Corporation | Pogo pin connector |
US11901070B2 (en) | 2017-02-24 | 2024-02-13 | Masimo Corporation | System for displaying medical monitoring data |
US11903140B2 (en) | 2015-02-06 | 2024-02-13 | Masimo Corporation | Fold flex circuit for LNOP |
US11925445B2 (en) | 2011-06-21 | 2024-03-12 | Masimo Corporation | Patient monitoring system |
US11931176B2 (en) | 2016-03-04 | 2024-03-19 | Masimo Corporation | Nose sensor |
US11944431B2 (en) | 2006-03-17 | 2024-04-02 | Masimo Corportation | Apparatus and method for creating a stable optical interface |
US11944415B2 (en) | 2013-08-05 | 2024-04-02 | Masimo Corporation | Systems and methods for measuring blood pressure |
US11951186B2 (en) | 2019-10-25 | 2024-04-09 | Willow Laboratories, Inc. | Indicator compounds, devices comprising indicator compounds, and methods of making and using the same |
US11961616B2 (en) | 2023-01-20 | 2024-04-16 | Vccb Holdings, Inc. | Real-time monitoring systems and methods in a healthcare environment |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117372437B (en) * | 2023-12-08 | 2024-02-23 | 安徽农业大学 | Intelligent detection and quantification method and system for facial paralysis |
-
2023
- 2023-01-11 US US18/153,108 patent/US20230222887A1/en active Pending
- 2023-01-11 US US18/153,173 patent/US20230222805A1/en active Pending
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11944431B2 (en) | 2006-03-17 | 2024-04-02 | Masimo Corportation | Apparatus and method for creating a stable optical interface |
US11857315B2 (en) | 2006-10-12 | 2024-01-02 | Masimo Corporation | Patient monitor capable of monitoring the quality of attached probes and accessories |
US11925445B2 (en) | 2011-06-21 | 2024-03-12 | Masimo Corporation | Patient monitoring system |
US11944415B2 (en) | 2013-08-05 | 2024-04-02 | Masimo Corporation | Systems and methods for measuring blood pressure |
US11894640B2 (en) | 2015-02-06 | 2024-02-06 | Masimo Corporation | Pogo pin connector |
US11903140B2 (en) | 2015-02-06 | 2024-02-13 | Masimo Corporation | Fold flex circuit for LNOP |
US11864922B2 (en) | 2015-09-04 | 2024-01-09 | Cercacor Laboratories, Inc. | Low-noise sensor system |
US11931176B2 (en) | 2016-03-04 | 2024-03-19 | Masimo Corporation | Nose sensor |
US11886858B2 (en) | 2017-02-24 | 2024-01-30 | Masimo Corporation | Medical monitoring hub |
US11901070B2 (en) | 2017-02-24 | 2024-02-13 | Masimo Corporation | System for displaying medical monitoring data |
US11872156B2 (en) | 2018-08-22 | 2024-01-16 | Masimo Corporation | Core body temperature measurement |
US11963749B2 (en) | 2019-08-30 | 2024-04-23 | Masimo Corporation | Acoustic physiological monitoring system |
US11951186B2 (en) | 2019-10-25 | 2024-04-09 | Willow Laboratories, Inc. | Indicator compounds, devices comprising indicator compounds, and methods of making and using the same |
US11879960B2 (en) | 2020-02-13 | 2024-01-23 | Masimo Corporation | System and method for monitoring clinical activities |
US11963736B2 (en) | 2020-12-30 | 2024-04-23 | Masimo Corporation | Wireless patient monitoring system |
USD1022729S1 (en) | 2022-12-20 | 2024-04-16 | Masimo Corporation | Wearable temperature measurement device |
US11961616B2 (en) | 2023-01-20 | 2024-04-16 | Vccb Holdings, Inc. | Real-time monitoring systems and methods in a healthcare environment |
Also Published As
Publication number | Publication date |
---|---|
US20230222887A1 (en) | 2023-07-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20230222805A1 (en) | Machine learning based monitoring system | |
Pham et al. | Delivering home healthcare through a cloud-based smart home environment (CoSHE) | |
US10643061B2 (en) | Detecting unauthorized visitors | |
US11369321B2 (en) | Monitoring and tracking system, method, article and device | |
US9795324B2 (en) | System for monitoring individuals as they age in place | |
JP6975230B2 (en) | Patient monitoring system and method | |
US7420472B2 (en) | Patient monitoring apparatus | |
JP7197475B2 (en) | Patient monitoring system and method | |
EP3432772B1 (en) | Using visual context to timely trigger measuring physiological parameters | |
Bathrinarayanan et al. | Evaluation of a monitoring system for event recognition of older people | |
WO2018037026A1 (en) | Device, system and method for patient monitoring to predict and prevent bed falls | |
WO2019013257A1 (en) | Monitoring assistance system and method for controlling same, and program | |
KR20190006670A (en) | Seizure monitoring method and apparatus using video | |
EP3504649B1 (en) | Device, system and method for patient monitoring to predict and prevent bed falls | |
Yi et al. | Home interactive elderly care two-way video healthcare system design | |
Ianculescu et al. | Improving the Elderly’s Fall Management through Innovative Personalized Remote Monitoring Solution | |
Inoue et al. | Bed exit action detection based on patient posture with long short-term memory | |
WO2021122136A1 (en) | Device, system and method for monitoring of a subject | |
CN116945156A (en) | Intelligent elderly accompanying system based on computer vision technology | |
CN112132112A (en) | Behavior prejudging system | |
Vijay et al. | Deep Learning-Based Smart Healthcare System for Patient's Discomfort Detection |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: MASIMO CORPORATION, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MUHSIN, BILAL;PRIDDELL, RICHARD;TELFORT, VALERY G.;AND OTHERS;SIGNING DATES FROM 20230424 TO 20240129;REEL/FRAME:066311/0291 |