US20210398416A1 - Systems and methods for a hand hygiene compliance checking system with explainable feedback - Google Patents
Systems and methods for a hand hygiene compliance checking system with explainable feedback Download PDFInfo
- Publication number
- US20210398416A1 US20210398416A1 US17/353,241 US202117353241A US2021398416A1 US 20210398416 A1 US20210398416 A1 US 20210398416A1 US 202117353241 A US202117353241 A US 202117353241A US 2021398416 A1 US2021398416 A1 US 2021398416A1
- Authority
- US
- United States
- Prior art keywords
- routine
- component
- compliant
- test data
- segments
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 25
- 238000012360 testing method Methods 0.000 claims abstract description 41
- 230000002123 temporal effect Effects 0.000 claims abstract description 16
- 238000005406 washing Methods 0.000 claims description 34
- 238000012706 support-vector machine Methods 0.000 claims description 29
- 238000001514 detection method Methods 0.000 claims description 27
- 230000008859 change Effects 0.000 claims description 17
- 238000004891 communication Methods 0.000 claims description 15
- 239000013598 vector Substances 0.000 claims description 14
- 238000012549 training Methods 0.000 claims description 9
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000012544 monitoring process Methods 0.000 claims description 7
- 238000005259 measurement Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 2
- 230000008569 process Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 8
- 238000011156 evaluation Methods 0.000 description 8
- 230000009471 action Effects 0.000 description 7
- 230000007246 mechanism Effects 0.000 description 7
- 238000002567 electromyography Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000006855 networking Effects 0.000 description 5
- 210000004247 hand Anatomy 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 241000700605 Viruses Species 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 210000003811 finger Anatomy 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 239000000344 soap Substances 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 210000001142 back Anatomy 0.000 description 2
- 238000002790 cross-validation Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 208000025721 COVID-19 Diseases 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000034994 death Effects 0.000 description 1
- 231100000517 death Toxicity 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000009474 immediate action Effects 0.000 description 1
- 208000015181 infectious disease Diseases 0.000 description 1
- 230000002045 lasting effect Effects 0.000 description 1
- 210000004936 left thumb Anatomy 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 210000004935 right thumb Anatomy 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/18—Status alarms
- G08B21/24—Reminder alarms, e.g. anti-loss alarms
- G08B21/245—Reminder of hygiene compliance policies, e.g. of washing hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/16—Constructional details or arrangements
- G06F1/1613—Constructional details or arrangements for portable computers
- G06F1/163—Wearable computers, e.g. on a belt
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
- H04W4/025—Services making use of location information using location based information parameters
- H04W4/027—Services making use of location information using location based information parameters using movement velocity, acceleration information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/80—Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
Definitions
- the present disclosure generally relates to hygiene compliance system, and in particular to a system and associated method for monitoring handwashing hygiene practices and providing explainable feedback.
- the Covid19 outbreak has affected the lives of billions of people in this world with over 2.79 million infections and 196,000 deaths as of Apr. 24, 2020.
- One of the primary modes of human to human transmission is through droplets from an infected person landing on the hands of a healthy individual, who in turn touches their face.
- An effective mode of defense against the SARS-Cov2 virus is to thoroughly wash hands regularly before touching one's face.
- the World Health Organization (WHO) and Center for Disease Control (CDC) has published guidelines for proper handwashing, which includes six components ( FIG. 1 ): i) take soap and rub in both the palms, ii) right palm over left dorsum with interlaced fingers, iii) left palm over right dorsum with interlaced fingers, iv) rotational rubbing of right thumb clasped in left palm, v) rotational rubbing of left thumb clasped in right palm, vi) rotational rubbing backwards and forwards while rinsing.
- the entire duration of hand washing exercise should last for at least 20 seconds. Adherence to this protocol is necessary to eliminate the virus from ones' hands. However, adhering to these guidelines can be a challenge for many, especially when done frequently throughout the day.
- FIG. 1 is a series of photographs showing WHO/CDC recommended components of a compliant handwashing routine
- FIG. 2 is a simplified diagram showing an embodiment of a hand hygiene monitoring system
- FIG. 3 is a simplified block diagram showing an additional view of the system of FIG. 2 ;
- FIG. 4 is a diagram showing generation of models for each handwashing routine component of the system of FIG. 2 ;
- FIG. 5 is a diagram showing identification of handwashing routine components and generation of explanations for missed or improperly completed handwashing routine components of the system of FIG. 2 ;
- FIG. 6 is a process flow diagram showing a process for hand hygiene monitoring by the system of FIG. 2 ;
- FIG. 7 is a diagram showing criteria for compliant hand washing data
- FIG. 8 is a diagram showing criteria for non-compliant hand washing data
- FIGS. 9A and 9B are graphical representations showing examples of RuLSIF results for compliant hand washing routines.
- FIG. 10 is a simplified diagram showing an exemplary computing system for implementation of the system of FIG. 2 .
- a proper handwashing routine one of the categories of hand hygiene, includes several components that must be performed in a temporally aligned sequence as recommended by WHO/CDC.
- the present system is an armband-based hand wash compliance checking mechanism that can identify if a component of the WHO/CDC recommendations are missing during a hand washing routine. An appropriate alert mechanism can then inform the user to redo the missing component(s) of the handwashing routine based on a compliance factor provided by the present system. Tests on data collected from 20 subjects, show a ten-fold user-independent accuracy of detecting compliant hand washing at 88%, precision at 0.89 and recall at 0.85. The present system has been found to identify any missing components in about 350 out of 500 non-compliant hand wash routines. Referring to the drawings, embodiments of a system and associated process for hand hygiene compliance checking are illustrated and generally indicated as 100 and 200 in FIGS. 1-10 .
- an armband-based hand hygiene compliance checking system 100 (hereinafter “system”) is disclosed that evaluates user compliance with WHO/CDC handwashing guidelines.
- the system 100 further includes an explanation framework to alert a user of a potential reason for non-compliance, so that the user can immediately correct themselves.
- the present WHO/CDC compliant hand washing routine includes six well-defined components, there can still be significant variations in correct handwashing routines performed by individuals.
- a user may perform the same component many times during the hand washing routine, resulting in potentially infinite ways of following the WHO/CDC guidelines.
- a classification approach that considers compliant and non-compliant hand washing examples and trains a machine to verify compliance can be cumbersome and also infeasible.
- the system is intended to scale to many users within a facility. As such it is not feasible to collect training data from each user. Hence, the compliance checking system 100 should be user independent.
- the present system 100 is an armband-based hand hygiene monitoring system.
- the user wears an armband 104 which connects to a smartphone or other computing device 102 via Bluetooth or another suitable wireless or wired connection.
- the armband 104 collects motion data from onboard sensors 112 when a user is executing the handwashing routine. The user does not need to continuously wear the armband 104 , it is required to be worn when performing a hand washing routine.
- the WHO/CDC recommendations for a compliant hand washing routine can be interpreted as a language, which can generate an infinite number of compliant hand washing sequences of routine components.
- Each component of the compliant handwashing routine may be considered a letter in the alphabet of the language described by the WHO/CDC recommendations.
- the alphabet is the set of all steps in the recommendation.
- a given hand washing routine then needs to match a language expressed using a context free grammar.
- This component-wise recognition and language-based expression allows the present system 100 to identify a potential reason for noncompliance and provide feedback to the user.
- the present system 100 considers differential features of a given component of a handwashing routine.
- the differential feature captures the relative differences in the alignment of the arms with respect to the first and the sixth step of the routine.
- Compliant and non-compliant hand washing examples were collected from 20 users for a total of 1300 samples.
- the system 100 reached a 10-fold cross-validation accuracy of 88% with a precision of 0.89 and recall of 0.85. This is with 60 (12 users)-40 (8 users) train test split among users.
- the system 100 was able to identify the missing component in 75% of the instances. Once a non-compliant hand washing routine is detected by the system 100 , the system 100 then asks the user to redo only the component(s) that were identified as missing.
- Each component of the WHO/CDC recommended handwashing routine can be modeled as a respective letter S i , i ⁇ 1 . . . 6 in an alphabet ⁇ .
- the WHO/CDC recommendation can be encoded in the following context-free grammar:
- S i ⁇ i ⁇ 1, 2, 3, 4, 5, 6 ⁇ are symbols representing each component of the recommended routine.
- An asterisk (*) as a superscript on a symbol denotes a Kleene start operation, which means any number of occurrences of the symbol can occur within the handwashing routine including no occurrence.
- HandWash.dur denotes the duration of the hand washing routine. According to the current WHO/CDC recommendations at the time of this filing, HandWash.dur ⁇ 20 s.
- the system 100 represents each handwashing routine in the form of a context free grammar, such as the context-free grammar described in Equation 1.
- a context free grammar such as the context-free grammar described in Equation 1.
- the context-free grammar allows concepts or components of the routine to be represented in a modular fashion; that is, the definition allows flexibility in repetition or order of some components of the handwashing routine. Further, the context-free grammar enables compliance checking of handwashing routines even if the components of the routine are repeated, out of order, or have an extended duration.
- Test data from a handwashing routine including accelerometer data and inertial measurement unit (IMU) data.
- IMU inertial measurement unit
- System Outputs 1. Identify whether the test data matches grammar of Equation 1, 2 that denote an acceptable handwashing routine. If the test data does not match the acceptable handwashing routine then determine which component of the grammar is missing. and 3. Duration of handwash.
- the armband 104 is a Myo device, developed by Thalmic Labs.
- the armband 104 is a band-shaped gesture control device that is worn on an arm of a user.
- the armband 104 determines the motion of the arm and transmits a live feed via Bluetooth or another wireless or wired connection protocol to a connected device 102 such as a smartphone in order to access raw EMG and/or IMU data streamed at 200 Hz and 50 Hz respectively.
- the armband 104 includes a plurality of sensors 112 that include eight surface ElectroMyoGraphy (EMG) sensors and a highly sensitive 9-axis Inertial Measurement Unit (IMU) sensor 114 that includes a 3-axis gyroscope, a 3-axis accelerometer, and a 3-axis magnetometer.
- EMG ElectroMyoGraphy
- IMU Inertial Measurement Unit
- the EMG sensor is not necessary for operation of the system 100 . This is because EMG sensors are currently not available in commercial wristbands or smart watches. Hence, if one wants to deploy the system 100 using a commercial wristband or smartwatch, the system 100 cannot rely exclusively on EMG.
- the armband 104 can include network interface 116 that communicates data to the computing device 102 across the network. Referring to FIG.
- the computing device 102 includes a processor 150 in communication with a memory 120 and a network interface 160 .
- the network interface 160 receives sensor data from the armband 104 for processing by the computing device 102 .
- the memory 120 includes instructions for an operating system 140 and instructions for execution of a compliance checking processes/services application 130 corresponding with process 200 .
- data is recorded from the armband 104 through an associated data recordation application.
- the data recordation application is Myo Recode.
- the app collects the data from the sensors 112 and stores them in the app folder as five individual text (.txt) files i.e., EMG.txt, IMU.txt, location.txt, process.txt and service.txt as shown in FIG. 2 .
- the computing system 102 of the system 100 includes three modules: a) a compliance definition module 170 including models for each component of the WHO/CDC recommended hand washing routine, b) a component identification module 180 operable to identify components in a handwashing routine performed by a user based on the models in the compliance definition module 170 , and c) a compliance checking and feedback module 190 .
- training data examples 171 of compliant handwashing routines were used to develop the compliance definition module 170 .
- the start and end time stamps of each component of the compliant handwashing routing examples are labeled.
- Training data including orientation X, Y and accelerometer X, Y data from the armband 104 were considered for the examples.
- Differential features are extracted from each compliant handwashing example as shown in FIG. 4 .
- For a given component S i a relative alignment of S i is considered with respect to all other components. The alignment was computed using a Dynamic Time Warping (DTW) distance measure by a DTW module 183 .
- DTW Dynamic Time Warping
- each gesture component S i there are five DTW-based alignments from each mode of sensing. Hence in total there are 20 such alignments, which forms a feature vector for a given handwash component.
- Such feature vectors are extracted for all components S 2 , S 3 , S 4 , and S 5 .
- a Support Vector Machine (SVM) 184 is trained with alignment feature vectors for S i as a positive class and alignment feature vectors from S i , j ⁇ i as a negative class.
- SVM Support Vector Machine
- Each routine component S i is associated with an individual SVM 184
- the Radial Basis Function was used as the kernel. To maintain balanced data, if one had x i number of positive samples for S i , the system 100 considers x i number of negative class samples using a uniform distribution.
- the trained SVM 184 encodes a model for a handwash routine component S i .
- the system 100 does not require SVM models of S 1 and S 6 as they are already encoded in the other models, given that they are used as references, however other embodiments may have other constraints on handwashing routine component arrangements.
- the component identification module 180 is shown in FIG. 5 and a corresponding process flow 200 is illustrated in FIG. 6 .
- the system 100 first identifies segments including a number of segments within the test data 181 using a change point detection module 182 .
- the change point detection module 182 can include a change point detection algorithm such as RuL-SIF, which is a divergence-based change point detection algorithm.
- the change point detection module 182 first divides the entire sample into windows. A window size is selected; one example window size is 4 seconds, i.e. 200 data points given a 50 Hz sampling ratio of the armband.
- the window size can be selected empirically; it was found that on average each component will take around 4 seconds to complete for a 20 second handwashing routine.
- the change point detection module 182 then computes a Pearson's divergence coefficient between two consecutive windows. The windows are then slid across the entire sample, hence obtaining a variation of the Pearson's Divergence coefficient over time. Peaks in a divergence coefficient indicates that there is a change in the underlying process generating the data.
- the change point detection module 182 extracts peaks from the Pearson's Divergence coefficient and the peak locations are considered to be start and end points of segments of test data 181 . Based on these change points, the component identification module 180 segments the handwash test sample.
- the system 100 assumes that a first segment of the test data 181 corresponds to S 1 , i.e. take soap, and the last segment corresponds to S 6 , i.e. rinse.
- the system 100 then computes relative DTW alignments of consecutive segments at a DTW module 183 .
- the component identification module 180 considers all possible sequences of each segment of the test data 181 by temporally swapping one segment with another segment. For each temporal combination of segments, the component identification module 180 computes DTW alignments of consecutive segments.
- each handwash sample generates a feature vector of length at-least 120, (4C2 ⁇ 5) assuming that each segment is performed once.
- the compliance checking and feedback module 190 evaluates the segments of the test data 181 to ensure that the hand washing routine is in compliance with the recommended hand washing routine.
- the feature vector for each segment is passed to each of the four SVM component models 184 A-D to classify segments of the test data 181 into components of a compliant handwashing routine.
- each temporal combination of segments is compared against each SVM component model 184 of a compliant routine.
- Each temporal combination of segments is passed through the SVM models 184 .
- Each temporal combination results in a 4-tuple of SVM outputs.
- the system 100 obtains at least a 24 ⁇ 4 matrix.
- the compliance checking and feedback module 190 obtains aggregate results from the SVM component models 184 to evaluate the SVM outputs This matrix is then summed up by the rows to get aggregate results of all combinations to yield a 1 ⁇ 4 matrix. These are the aggregate outputs of the SVM component models 184 A-D.
- the i th element of the matrix signifies the detection of the i th component.
- the aggregate outputs of the SVM component models 184 A- 1840 are evaluated to identify a percentage of positive detections across the plurality of temporal combinations of segments.
- the percentage of positive detections for each SVM component model 184 is compared with a threshold percentage to determine whether the corresponding component of the compliant routine is sufficiently present within the test data.
- a threshold of 3% was set, which means that the component S i should be recognized by the SVM model 184 for S i in at least 30% of the temporal combinations.
- the compliance checking and feedback module 190 considers that the hand washing routine is non-compliant.
- the system 100 compares an aggregation recognition result for test data 181 with a threshold to identify missing routine components that do not meet the threshold. If the aggregate recognition results are lower than this threshold for a component S i , then the compliance checking and feedback module 190 considers that that component S i was missing and at block 270 asks the user to redo that component.
- Evaluation of the system 100 has two aims: a) determine the accuracy of identification of compliant hand washing routines, and b) determine the correctness of the explanation generation mechanism.
- the evaluation considers among how many instances the reason for non-compliance was correctly identified over the total number of non-compliant instances.
- Compliant handwashing is defined as the hand washes that have all the 6 required step actions performed for approximately 20 seconds either (a) sequentially or (b) in different combination orders with fixed step 1 and step 6 being the start and finish steps. Additionally, the data included correct handwashings lasting for approximately 40 seconds, twice the recommended amount of time.
- each step action is performed for 4 times.
- each step action is performed for 4 times having different sequences of steps keeping the first step 1 and last step 6 at the start and finish, hence there are 24 different combinations of handwashing styles i.e., S i , different step orders of S 2 , S 3 , S 4 , S 5 , S 6 .
- criteria (c) every step is performed 7 times.
- Non-compliant hand washing is defined as the hand washes that have performed for less than 15 seconds having (i) all the required steps sequentially or (ii) either one component/step is missing with different combination order of steps and (iii) handwashing that have one missing step action with different combination order of steps performed for 20 seconds. Missing component is between the steps 2 to 5.
- every step action is performed 2 times.
- every step action is performed 2 times with one missing component of step and different order sequence of steps keeping the first step 1 and last step 6 at the start and finish i.e., S 1 , missing of S 2 or S 3 or S 4 or S 5 and different step orders non-missing steps, S 6 .
- criteria (iii) it is similar fashion as criteria (ii) but every action is performed 5 or 6 times to get the 20 seconds duration.
- FIGS. 9A and 9B show an example output of RuISIF based change point detection algorithm on the Orientation X axis of a compliant handwash routine.
- Intra-user recognition with a 60-40 split was performed. This means that out of the 65 samples collected from each user, the evaluation included 60% data of each user in training and 40% from each user for testing. The accuracy increased to 92% and the precision was 96% while the recall was 90%. This is expected since performed user dependent analysis was also performed. However, the difference in accuracy is only 7% which indicates that the system 100 was successful in extracting user-independent characteristics of handwash routines.
- the evaluation only focused on the non-compliant handwash routines. Duration of hand wash routine is easy to identify by just checking the number of data points in each routine. Hence, if a non-compliant handwash violates the duration rule, the system 100 can identify it 100% of the time. For the handwash routines that are more than 20 seconds, the system 100 attempts to identify the missing components.
- the dataset includes nearly 500 samples of non-compliant handwashing routines with one or more missing components. It is assumed that the explanation for non-compliance is successful if the system 100 can identify at least one component that is missing. Out of the 500 non-compliant samples the system 100 could correctly identify at least one missing component in 75% of the samples. The system 100 could correctly identify two missing components in 41% of the samples.
- FIG. 10 illustrates an example of a suitable computing and networking environment (computer system 300 ) which may be used to implement various aspects of the present disclosure.
- Example embodiments described herein may be implemented at least in part in electronic circuitry; in computer hardware executing firmware and/or software instructions; and/or in combinations thereof.
- Example embodiments also may be implemented using a computer program product (e.g., a computer program tangibly or non-transitorily embodied in a machine-readable medium and including instructions for execution by, or to control the operation of, a data processing apparatus, such as, for example, one or more programmable processors or computers).
- a computer program product e.g., a computer program tangibly or non-transitorily embodied in a machine-readable medium and including instructions for execution by, or to control the operation of, a data processing apparatus, such as, for example, one or more programmable processors or computers.
- a computer program may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a subroutine or other unit suitable for use in a computing environment. Also, a computer program can be deployed to be executed on one computer, or to be executed on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
- modules are hardware-implemented, and thus include at least one tangible unit capable of performing certain operations and may be configured or arranged in a certain manner.
- a hardware-implemented module may comprise dedicated circuitry that is permanently configured (e.g., as a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations.
- a hardware-implemented module may also comprise programmable circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software or firmware to perform certain operations.
- one or more computer systems e.g., a standalone system, a client and/or server computer system, or a peer-to-peer computer system
- one or more processors may be configured by software (e.g., an application or application portion) as a hardware-implemented module that operates to perform certain operations as described herein.
- the term “hardware-implemented module” encompasses a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein.
- hardware-implemented modules are temporarily configured (e.g., programmed)
- each of the hardware-implemented modules need not be configured or instantiated at any one instance in time.
- the hardware-implemented modules comprise a general-purpose processor configured using software
- the general-purpose processor may be configured as respective different hardware-implemented modules at different times.
- Software in the form of the system 100 or process 200 or otherwise, may include a hardware-implemented module and may accordingly configure a processor 302 , for example, to constitute a particular hardware-implemented module at one instance of time and to constitute a different hardware-implemented module at a different instance of time.
- Hardware-implemented modules may provide information to, and/or receive information from, other hardware-implemented modules. Accordingly, the described hardware-implemented modules may be regarded as being communicatively coupled. Where multiple of such hardware-implemented modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware-implemented modules. In embodiments in which multiple hardware-implemented modules are configured or instantiated at different times, communications between such hardware-implemented modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware-implemented modules have access.
- one hardware-implemented module may perform an operation, and may store the output of that operation in a memory device to which it is communicatively coupled. A further hardware-implemented module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware-implemented modules may also initiate communications with input or output devices.
- the computing and networking environment 300 may be a general purpose computing device 300 , although it is contemplated that the networking environment 300 may include other computing systems, such as personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, digital signal processors, state machines, logic circuitries, distributed computing environments that include any of the above computing systems or devices, and the like.
- the networking environment 300 may include other computing systems, such as personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, digital signal processors, state machines, logic circuitries, distributed computing environments that include any of the above computing systems or devices, and the like.
- Components of the general purpose computing device 300 may include various hardware components, such as a processing unit 302 , a main memory 304 (e.g., a memory or a system memory), and a system bus 301 that couples various system components of the general purpose computing device 300 to the processing unit 302 .
- the system bus 301 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
- bus architectures may include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
- ISA Industry Standard Architecture
- MCA Micro Channel Architecture
- EISA Enhanced ISA
- VESA Video Electronics Standards Association
- PCI Peripheral Component Interconnect
- the general-purpose computing device 300 may further include a variety of computer-readable media 307 that includes removable/non-removable media and volatile/nonvolatile media but excludes transitory propagated signals.
- Computer-readable media 307 may also include computer storage media and communication media.
- Computer storage media includes removable/non-removable media and volatile/nonvolatile media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules or other data, such as RAM, ROM, EPSOM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information/data and which may be accessed by the general purpose computing device 300 .
- Communication media includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
- modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
- communication media may include wired media such as a wired network or direct-wired connection and wireless media such as acoustic, RF, infrared, and/or other wireless media, or some combination thereof.
- Computer-readable media may be embodied as a computer program product, such as software stored on computer storage media.
- the main memory 304 includes computer storage media in the form of volatile/nonvolatile memory such as read only memory (ROM) and random access memory (RAM).
- ROM read only memory
- RAM random access memory
- BIOS basic input/output system
- RAM typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 302 .
- data storage 306 holds an operating system, application programs, and other program modules and program data.
- Data storage 306 may also include other removable/non-removable, volatile/nonvolatile computer storage media.
- data storage 306 may be: a hard disk drive that reads from or writes to non-removable, nonvolatile magnetic media; a magnetic disk drive that reads from or writes to a removable, nonvolatile magnetic disk; and/or an optical disk drive that reads from or writes to a removable, nonvolatile optical disk such as a CD-ROM or other optical media.
- Other removable/non-removable, volatile/nonvolatile computer storage media may include magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like.
- the drives and their associated computer storage media provide storage of computer-readable instructions, data structures, program modules and other data for the general purpose computing device 300 .
- a user may enter commands and information through a user interface 340 or other input devices 345 such as a tablet, electronic digitizer, a microphone, keyboard, and/or pointing device, commonly referred to as mouse, trackball, or touch pad.
- Other input devices 345 may include a joystick, game pad, satellite dish, scanner, or the like.
- voice inputs, gesture inputs (e.g., via hands or fingers), or other natural user interfaces may also be used with the appropriate input devices, such as a microphone, camera, tablet, touch pad, glove, or other sensor.
- a monitor 360 or other type of display device is also connected to the system bus 301 via user interface 340 , such as a video interface.
- the monitor 360 may also be integrated with a touch-screen panel or the like.
- the general purpose computing device 300 may operate in a networked or cloud-computing environment using logical connections of a network Interface 303 to one or more remote devices, such as a remote computer.
- the remote computer may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the general purpose computing device 300 .
- the logical connection may include one or more local area networks (LAN) and one or more wide area networks (WAN), but may also include other networks.
- LAN local area network
- WAN wide area network
- Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.
- the general purpose computing device 300 When used in a networked or cloud-computing environment, the general purpose computing device 300 may be connected to a public and/or private network through the network interface 303 . In such embodiments, a modem or other means for establishing communications over the network is connected to the system bus 301 via the network interface 303 or other appropriate mechanism.
- a wireless networking component including an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a network.
- program modules depicted relative to the general purpose computing device 300 may be stored in the remote memory storage device.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Hardware Design (AREA)
- Signal Processing (AREA)
- Medical Informatics (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Databases & Information Systems (AREA)
- Image Analysis (AREA)
- User Interface Of Digital Computer (AREA)
- Epidemiology (AREA)
- Public Health (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Description
- This is a non-provisional application that claims benefit to U.S. Provisional Patent Application Ser. No. 63/041,746 filed 19 Jun. 2020, which is herein incorporated by reference in its entirety.
- The present disclosure generally relates to hygiene compliance system, and in particular to a system and associated method for monitoring handwashing hygiene practices and providing explainable feedback.
- The Covid19 outbreak has affected the lives of billions of people in this world with over 2.79 million infections and 196,000 deaths as of Apr. 24, 2020. One of the primary modes of human to human transmission is through droplets from an infected person landing on the hands of a healthy individual, who in turn touches their face. An effective mode of defense against the SARS-Cov2 virus is to thoroughly wash hands regularly before touching one's face.
- The World Health Organization (WHO) and Center for Disease Control (CDC) has published guidelines for proper handwashing, which includes six components (
FIG. 1 ): i) take soap and rub in both the palms, ii) right palm over left dorsum with interlaced fingers, iii) left palm over right dorsum with interlaced fingers, iv) rotational rubbing of right thumb clasped in left palm, v) rotational rubbing of left thumb clasped in right palm, vi) rotational rubbing backwards and forwards while rinsing. The entire duration of hand washing exercise should last for at least 20 seconds. Adherence to this protocol is necessary to eliminate the virus from ones' hands. However, adhering to these guidelines can be a challenge for many, especially when done frequently throughout the day. - It is with these observations in mind, among others, that various aspects of the present disclosure were conceived and developed.
- The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
-
FIG. 1 is a series of photographs showing WHO/CDC recommended components of a compliant handwashing routine; -
FIG. 2 is a simplified diagram showing an embodiment of a hand hygiene monitoring system; -
FIG. 3 is a simplified block diagram showing an additional view of the system ofFIG. 2 ; -
FIG. 4 is a diagram showing generation of models for each handwashing routine component of the system ofFIG. 2 ; -
FIG. 5 is a diagram showing identification of handwashing routine components and generation of explanations for missed or improperly completed handwashing routine components of the system ofFIG. 2 ; -
FIG. 6 is a process flow diagram showing a process for hand hygiene monitoring by the system ofFIG. 2 ; -
FIG. 7 is a diagram showing criteria for compliant hand washing data; -
FIG. 8 is a diagram showing criteria for non-compliant hand washing data; -
FIGS. 9A and 9B are graphical representations showing examples of RuLSIF results for compliant hand washing routines; and -
FIG. 10 is a simplified diagram showing an exemplary computing system for implementation of the system ofFIG. 2 . - Corresponding reference characters indicate corresponding elements among the view of the drawings. The headings used in the figures do not limit the scope of the claims.
- Monitoring hand hygiene practices, feedback on hygiene evaluation, and immediate action are important steps toward protection against a virus that transmits through droplets such as SARS-Cov2. A proper handwashing routine, one of the categories of hand hygiene, includes several components that must be performed in a temporally aligned sequence as recommended by WHO/CDC. The present system is an armband-based hand wash compliance checking mechanism that can identify if a component of the WHO/CDC recommendations are missing during a hand washing routine. An appropriate alert mechanism can then inform the user to redo the missing component(s) of the handwashing routine based on a compliance factor provided by the present system. Tests on data collected from 20 subjects, show a ten-fold user-independent accuracy of detecting compliant hand washing at 88%, precision at 0.89 and recall at 0.85. The present system has been found to identify any missing components in about 350 out of 500 non-compliant hand wash routines. Referring to the drawings, embodiments of a system and associated process for hand hygiene compliance checking are illustrated and generally indicated as 100 and 200 in
FIGS. 1-10 . - In this disclosure, an armband-based hand hygiene compliance checking system 100 (hereinafter “system”) is disclosed that evaluates user compliance with WHO/CDC handwashing guidelines. The
system 100 further includes an explanation framework to alert a user of a potential reason for non-compliance, so that the user can immediately correct themselves. - Given that the present WHO/CDC compliant hand washing routine includes six well-defined components, there can still be significant variations in correct handwashing routines performed by individuals. First, there is no specific order in which the components must be executed. The first step, taking soap, and the last step, rinsing, should be fixed, however, the rest of the components can occur in any order. Moreover, there is no restriction on the number of times each component is performed. Hence, a user may perform the same component many times during the hand washing routine, resulting in potentially infinite ways of following the WHO/CDC guidelines. As such a classification approach that considers compliant and non-compliant hand washing examples and trains a machine to verify compliance can be cumbersome and also infeasible.
- Secondly, in case a user misses a given component of the hand washing routine, the user should be immediately notified so that the user can correct themselves. This necessitates an explainable compliance checking system that can identify the reason a particular handwashing routine is non-compliant.
- Thirdly, the system is intended to scale to many users within a facility. As such it is not feasible to collect training data from each user. Hence, the
compliance checking system 100 should be user independent. - The
present system 100 is an armband-based hand hygiene monitoring system. In one aspect, the user wears anarmband 104 which connects to a smartphone orother computing device 102 via Bluetooth or another suitable wireless or wired connection. Thearmband 104 collects motion data fromonboard sensors 112 when a user is executing the handwashing routine. The user does not need to continuously wear thearmband 104, it is required to be worn when performing a hand washing routine. - To address the challenges mentioned above, the WHO/CDC recommendations for a compliant hand washing routine can be interpreted as a language, which can generate an infinite number of compliant hand washing sequences of routine components. Each component of the compliant handwashing routine may be considered a letter in the alphabet of the language described by the WHO/CDC recommendations. The alphabet is the set of all steps in the recommendation. A given hand washing routine then needs to match a language expressed using a context free grammar.
- For the present system, individual component models must first be developed for each component of the compliant hand washing routine.
Data 171 from a compliant hand washingroutine using armband 104 is recorded and then segmented into a plurality of segments of the routine using a change point detection module. Each segment of the plurality of segments is then checked against models of all the components of the compliant hand washing routine to determine if all components are present in an order enforced by a grammar (Eq. 1, below). This ensures that the user can perform all the components any number of times in any sequence. In the present embodiment, thesystem 100 assumes that the first and the sixth components of the routine are fixed, however, in other embodiments, alternative temporal arrangement of components may be required. - This component-wise recognition and language-based expression allows the
present system 100 to identify a potential reason for noncompliance and provide feedback to the user. - To address the user-independence requirements, the
present system 100 considers differential features of a given component of a handwashing routine. The differential feature captures the relative differences in the alignment of the arms with respect to the first and the sixth step of the routine. - Compliant and non-compliant hand washing examples were collected from 20 users for a total of 1300 samples. The
system 100 reached a 10-fold cross-validation accuracy of 88% with a precision of 0.89 and recall of 0.85. This is with 60 (12 users)-40 (8 users) train test split among users. For non-compliant handwashing routines, thesystem 100 was able to identify the missing component in 75% of the instances. Once a non-compliant hand washing routine is detected by thesystem 100, thesystem 100 then asks the user to redo only the component(s) that were identified as missing. - Each component of the WHO/CDC recommended handwashing routine can be modeled as a respective letter Si, i∈1 . . . 6 in an alphabet Σ. The WHO/CDC recommendation can be encoded in the following context-free grammar:
-
S 2R →S 3 *S 4 *S 5 *S 2 S 2 *S 3 *S 4 *S 5* -
S 3R →S 2 *S 4 *S 5 *S 3 S 3 *S 2 *S 4 *S 5* -
S 4R →S 2 *S 3 *S 5 *S 4 S 4 *S 2 *S 3 *S 5* -
S 5R →S 2 *S 3 *S 4 *S 4 S 4 *S 2 *S 3 *S 4* -
H and Wash→S 1 S 2R S 3R S 4R S 5R S 6 (1) - Here Si∀i∈{1, 2, 3, 4, 5, 6} are symbols representing each component of the recommended routine. An asterisk (*) as a superscript on a symbol denotes a Kleene start operation, which means any number of occurrences of the symbol can occur within the handwashing routine including no occurrence. HandWash.dur denotes the duration of the hand washing routine. According to the current WHO/CDC recommendations at the time of this filing, HandWash.dur≥20 s.
- The
system 100 represents each handwashing routine in the form of a context free grammar, such as the context-free grammar described inEquation 1. The context-free grammar allows concepts or components of the routine to be represented in a modular fashion; that is, the definition allows flexibility in repetition or order of some components of the handwashing routine. Further, the context-free grammar enables compliance checking of handwashing routines even if the components of the routine are repeated, out of order, or have an extended duration. - Problem Statement: Given sets of training samples TSii∈{1 . . . 6}, for each handwashing component Si,
- System Input: Test data from a handwashing routine including accelerometer data and inertial measurement unit (IMU) data.
- System Outputs: 1. Identify whether the test data matches grammar of
Equation - An embodiment of the
system 100 is shown inFIGS. 2 and 3 . In one embodiment, thearmband 104 is a Myo device, developed by Thalmic Labs. Thearmband 104 is a band-shaped gesture control device that is worn on an arm of a user. Thearmband 104 determines the motion of the arm and transmits a live feed via Bluetooth or another wireless or wired connection protocol to aconnected device 102 such as a smartphone in order to access raw EMG and/or IMU data streamed at 200 Hz and 50 Hz respectively. In one embodiment, thearmband 104 includes a plurality ofsensors 112 that include eight surface ElectroMyoGraphy (EMG) sensors and a highly sensitive 9-axis Inertial Measurement Unit (IMU)sensor 114 that includes a 3-axis gyroscope, a 3-axis accelerometer, and a 3-axis magnetometer. In some embodiments, the EMG sensor is not necessary for operation of thesystem 100. This is because EMG sensors are currently not available in commercial wristbands or smart watches. Hence, if one wants to deploy thesystem 100 using a commercial wristband or smartwatch, thesystem 100 cannot rely exclusively on EMG. Thearmband 104 can includenetwork interface 116 that communicates data to thecomputing device 102 across the network. Referring toFIG. 3 , thecomputing device 102 includes aprocessor 150 in communication with amemory 120 and anetwork interface 160. Thenetwork interface 160 receives sensor data from thearmband 104 for processing by thecomputing device 102. Thememory 120 includes instructions for anoperating system 140 and instructions for execution of a compliance checking processes/services application 130 corresponding withprocess 200. - In some embodiments, data is recorded from the
armband 104 through an associated data recordation application. In some embodiments, the data recordation application is Myo Recode. The app collects the data from thesensors 112 and stores them in the app folder as five individual text (.txt) files i.e., EMG.txt, IMU.txt, location.txt, process.txt and service.txt as shown inFIG. 2 . - Referring to
FIGS. 3-6 , thecomputing system 102 of thesystem 100 includes three modules: a) acompliance definition module 170 including models for each component of the WHO/CDC recommended hand washing routine, b) acomponent identification module 180 operable to identify components in a handwashing routine performed by a user based on the models in thecompliance definition module 170, and c) a compliance checking andfeedback module 190. - Referring to
FIG. 4 , training data examples 171 of compliant handwashing routines were used to develop thecompliance definition module 170. The start and end time stamps of each component of the compliant handwashing routing examples are labeled. Training data including orientation X, Y and accelerometer X, Y data from thearmband 104 were considered for the examples. Differential features are extracted from each compliant handwashing example as shown inFIG. 4 . For a given component Si, a relative alignment of Si is considered with respect to all other components. The alignment was computed using a Dynamic Time Warping (DTW) distance measure by aDTW module 183. Hence, for each gesture component Si there are five DTW-based alignments from each mode of sensing. Hence in total there are 20 such alignments, which forms a feature vector for a given handwash component. Such feature vectors are extracted for all components S2, S3, S4, and S5. - In one implementation, to build a model of one component Si of the compliant routine, a Support Vector Machine (SVM) 184 is trained with alignment feature vectors for Si as a positive class and alignment feature vectors from Si, j≠i as a negative class. Each routine component Si is associated with an
individual SVM 184 The Radial Basis Function was used as the kernel. To maintain balanced data, if one had xi number of positive samples for Si, thesystem 100 considers xi number of negative class samples using a uniform distribution. The trainedSVM 184 encodes a model for a handwash routine component Si. In some embodiments, there are a plurality ofdifferential SVM models 184A-D for components S2, S3, S4, S5. In some embodiments, thesystem 100 does not require SVM models of S1 and S6 as they are already encoded in the other models, given that they are used as references, however other embodiments may have other constraints on handwashing routine component arrangements. - Such differential models enable the
system 100 to perform user-independent recognition. The primary reason is that the alignment operation using DTW ensures that artifacts related to initialization and origin shift are negated by the DTW operation. - For deployment of the
system 100, thecomponent identification module 180 is shown inFIG. 5 and acorresponding process flow 200 is illustrated inFIG. 6 . Referring to block 210 ofprocess 200, giventest data 181 of a handwashing routine captured bysensors 112, thesystem 100 first identifies segments including a number of segments within thetest data 181 using a changepoint detection module 182. In particular, the changepoint detection module 182 can include a change point detection algorithm such as RuL-SIF, which is a divergence-based change point detection algorithm. The changepoint detection module 182 first divides the entire sample into windows. A window size is selected; one example window size is 4 seconds, i.e. 200 data points given a 50 Hz sampling ratio of the armband. The window size can be selected empirically; it was found that on average each component will take around 4 seconds to complete for a 20 second handwashing routine. The changepoint detection module 182 then computes a Pearson's divergence coefficient between two consecutive windows. The windows are then slid across the entire sample, hence obtaining a variation of the Pearson's Divergence coefficient over time. Peaks in a divergence coefficient indicates that there is a change in the underlying process generating the data. The changepoint detection module 182 extracts peaks from the Pearson's Divergence coefficient and the peak locations are considered to be start and end points of segments oftest data 181. Based on these change points, thecomponent identification module 180 segments the handwash test sample. - For this particular application, the
system 100 assumes that a first segment of thetest data 181 corresponds to S1, i.e. take soap, and the last segment corresponds to S6, i.e. rinse. Referring to block 220, thesystem 100 then computes relative DTW alignments of consecutive segments at aDTW module 183. Thecomponent identification module 180 considers all possible sequences of each segment of thetest data 181 by temporally swapping one segment with another segment. For each temporal combination of segments, thecomponent identification module 180 computes DTW alignments of consecutive segments. Referring to block 230, each handwash sample generates a feature vector of length at-least 120, (4C2×5) assuming that each segment is performed once. - The compliance checking and
feedback module 190 evaluates the segments of thetest data 181 to ensure that the hand washing routine is in compliance with the recommended hand washing routine. Atblock 240, the feature vector for each segment is passed to each of the fourSVM component models 184A-D to classify segments of thetest data 181 into components of a compliant handwashing routine. In particular, each temporal combination of segments is compared against eachSVM component model 184 of a compliant routine. Each temporal combination of segments is passed through theSVM models 184. Each temporal combination results in a 4-tuple of SVM outputs. Hence, for each test handwash routine there will be at least 24 instances of SVM model outputs. At the end of this step, thesystem 100 obtains at least a 24×4 matrix. Atblock 250, the compliance checking andfeedback module 190 obtains aggregate results from theSVM component models 184 to evaluate the SVM outputs This matrix is then summed up by the rows to get aggregate results of all combinations to yield a 1×4 matrix. These are the aggregate outputs of theSVM component models 184A-D. The ith element of the matrix signifies the detection of the ith component. - At
block 250, the aggregate outputs of theSVM component models 184A-1840 are evaluated to identify a percentage of positive detections across the plurality of temporal combinations of segments. The percentage of positive detections for eachSVM component model 184 is compared with a threshold percentage to determine whether the corresponding component of the compliant routine is sufficiently present within the test data. A threshold of 3% was set, which means that the component Si should be recognized by theSVM model 184 for Si in at least 30% of the temporal combinations. When allSVMs 184 return a positive class, then the compliance checking andfeedback module 190 considers that the handwashing sample routine is compliant. If at least oneSVM 184 returns a negative result, the compliance checking andfeedback module 190 considers that the hand washing routine is non-compliant. Atblock 260, thesystem 100 compares an aggregation recognition result fortest data 181 with a threshold to identify missing routine components that do not meet the threshold. If the aggregate recognition results are lower than this threshold for a component Si, then the compliance checking andfeedback module 190 considers that that component Si was missing and atblock 270 asks the user to redo that component. - Evaluation of the
system 100 has two aims: a) determine the accuracy of identification of compliant hand washing routines, and b) determine the correctness of the explanation generation mechanism. - To address evaluation goals there are two sets of metrics. For the first aim, accuracy, precision, and recall are considered. In the specific application of the
system 100 discussed herein, it is believed that precision is a more important parameter than recall for public safety. It is necessary to reduce false positives i.e. non-compliant hand washing routines classified as compliant. If compliant handwashing routines are classified as noncompliant, then there is no harm in spending an additional few seconds re-doing the routine. - For the second aim, the evaluation considers among how many instances the reason for non-compliance was correctly identified over the total number of non-compliant instances.
- Data was collected from 20 subjects including correct and incorrect hand washings. Compliant handwashing is defined as the hand washes that have all the 6 required step actions performed for approximately 20 seconds either (a) sequentially or (b) in different combination orders with fixed
step 1 andstep 6 being the start and finish steps. Additionally, the data included correct handwashings lasting for approximately 40 seconds, twice the recommended amount of time. - As shown in
FIG. 7 , in criteria (a), each step action is performed for 4 times. In criteria (b), each step action is performed for 4 times having different sequences of steps keeping thefirst step 1 andlast step 6 at the start and finish, hence there are 24 different combinations of handwashing styles i.e., Si, different step orders of S2, S3, S4, S5, S6. In criteria (c), every step is performed 7 times. - Non-compliant hand washing is defined as the hand washes that have performed for less than 15 seconds having (i) all the required steps sequentially or (ii) either one component/step is missing with different combination order of steps and (iii) handwashing that have one missing step action with different combination order of steps performed for 20 seconds. Missing component is between the
steps 2 to 5. - As shown in
FIG. 8 , in criteria (i), every step action is performed 2 times. In criteria (ii), every step action is performed 2 times with one missing component of step and different order sequence of steps keeping thefirst step 1 andlast step 6 at the start and finish i.e., S1, missing of S2 or S3 or S4 or S5 and different step orders non-missing steps, S6. In criteria (iii), it is similar fashion as criteria (ii) but every action is performed 5 or 6 times to get the 20 seconds duration. - Segmentation is an important component of the entire mechanism. RuLSIF change point detection algorithm was used for segmentation and its accuracy was verified on the collected data. RuLSIF was 93% accurate in determining the change point, which means that out of nearly 8000 change points, it identified nearly 7500 change point time stamps. Out of the 7,500 time stamps, on an average RuLSIF based segmentation was off by +/−13 samples. This error may have contributed to lower the accuracy of our mechanism.
FIGS. 9A and 9B show an example output of RuISIF based change point detection algorithm on the Orientation X axis of a compliant handwash routine. - Ten-fold cross-validation inter-user recognition was performed while considering a 60-40 split of train vs. test data. This implies that 12 users were used to train the system and the rest 8 users were used to test. The average accuracy for recognition of compliant hand washing among the total number of instances was 88%. The mechanism achieved an average precision of 0.89, while the recall was 0.85. A higher precision is an encouraging result since it indicates that false positives are lesser than false negatives.
- Intra-user recognition with a 60-40 split was performed. This means that out of the 65 samples collected from each user, the evaluation included 60% data of each user in training and 40% from each user for testing. The accuracy increased to 92% and the precision was 96% while the recall was 90%. This is expected since performed user dependent analysis was also performed. However, the difference in accuracy is only 7% which indicates that the
system 100 was successful in extracting user-independent characteristics of handwash routines. - For detection of missing components, the evaluation only focused on the non-compliant handwash routines. Duration of hand wash routine is easy to identify by just checking the number of data points in each routine. Hence, if a non-compliant handwash violates the duration rule, the
system 100 can identify it 100% of the time. For the handwash routines that are more than 20 seconds, thesystem 100 attempts to identify the missing components. The dataset includes nearly 500 samples of non-compliant handwashing routines with one or more missing components. It is assumed that the explanation for non-compliance is successful if thesystem 100 can identify at least one component that is missing. Out of the 500 non-compliant samples thesystem 100 could correctly identify at least one missing component in 75% of the samples. Thesystem 100 could correctly identify two missing components in 41% of the samples. -
FIG. 10 illustrates an example of a suitable computing and networking environment (computer system 300) which may be used to implement various aspects of the present disclosure. Example embodiments described herein may be implemented at least in part in electronic circuitry; in computer hardware executing firmware and/or software instructions; and/or in combinations thereof. Example embodiments also may be implemented using a computer program product (e.g., a computer program tangibly or non-transitorily embodied in a machine-readable medium and including instructions for execution by, or to control the operation of, a data processing apparatus, such as, for example, one or more programmable processors or computers). A computer program may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a subroutine or other unit suitable for use in a computing environment. Also, a computer program can be deployed to be executed on one computer, or to be executed on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. - Certain embodiments are described herein as including one or more modules. Such modules are hardware-implemented, and thus include at least one tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. For example, a hardware-implemented module may comprise dedicated circuitry that is permanently configured (e.g., as a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware-implemented module may also comprise programmable circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software or firmware to perform certain operations. In some example embodiments, one or more computer systems (e.g., a standalone system, a client and/or server computer system, or a peer-to-peer computer system) or one or more processors may be configured by software (e.g., an application or application portion) as a hardware-implemented module that operates to perform certain operations as described herein.
- Accordingly, the term “hardware-implemented module” encompasses a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein. Considering embodiments in which hardware-implemented modules are temporarily configured (e.g., programmed), each of the hardware-implemented modules need not be configured or instantiated at any one instance in time. For example, where the hardware-implemented modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware-implemented modules at different times. Software, in the form of the
system 100 orprocess 200 or otherwise, may include a hardware-implemented module and may accordingly configure aprocessor 302, for example, to constitute a particular hardware-implemented module at one instance of time and to constitute a different hardware-implemented module at a different instance of time. - Hardware-implemented modules may provide information to, and/or receive information from, other hardware-implemented modules. Accordingly, the described hardware-implemented modules may be regarded as being communicatively coupled. Where multiple of such hardware-implemented modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware-implemented modules. In embodiments in which multiple hardware-implemented modules are configured or instantiated at different times, communications between such hardware-implemented modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware-implemented modules have access. For example, one hardware-implemented module may perform an operation, and may store the output of that operation in a memory device to which it is communicatively coupled. A further hardware-implemented module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware-implemented modules may also initiate communications with input or output devices.
- As illustrated, the computing and
networking environment 300 may be a generalpurpose computing device 300, although it is contemplated that thenetworking environment 300 may include other computing systems, such as personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, digital signal processors, state machines, logic circuitries, distributed computing environments that include any of the above computing systems or devices, and the like. - Components of the general
purpose computing device 300 may include various hardware components, such as aprocessing unit 302, a main memory 304 (e.g., a memory or a system memory), and asystem bus 301 that couples various system components of the generalpurpose computing device 300 to theprocessing unit 302. Thesystem bus 301 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus. - The general-
purpose computing device 300 may further include a variety of computer-readable media 307 that includes removable/non-removable media and volatile/nonvolatile media but excludes transitory propagated signals. Computer-readable media 307 may also include computer storage media and communication media. Computer storage media includes removable/non-removable media and volatile/nonvolatile media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules or other data, such as RAM, ROM, EPSOM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information/data and which may be accessed by the generalpurpose computing device 300. Communication media includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media may include wired media such as a wired network or direct-wired connection and wireless media such as acoustic, RF, infrared, and/or other wireless media, or some combination thereof. Computer-readable media may be embodied as a computer program product, such as software stored on computer storage media. - The
main memory 304 includes computer storage media in the form of volatile/nonvolatile memory such as read only memory (ROM) and random access memory (RAM). A basic input/output system (BIOS), containing the basic routines that help to transfer information between elements within the general-purpose computing device 300 (e.g., during start-up) is typically stored in ROM. RAM typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processingunit 302. For example, in one embodiment,data storage 306 holds an operating system, application programs, and other program modules and program data. -
Data storage 306 may also include other removable/non-removable, volatile/nonvolatile computer storage media. For example,data storage 306 may be: a hard disk drive that reads from or writes to non-removable, nonvolatile magnetic media; a magnetic disk drive that reads from or writes to a removable, nonvolatile magnetic disk; and/or an optical disk drive that reads from or writes to a removable, nonvolatile optical disk such as a CD-ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media may include magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The drives and their associated computer storage media provide storage of computer-readable instructions, data structures, program modules and other data for the generalpurpose computing device 300. - A user may enter commands and information through a user interface 340 or
other input devices 345 such as a tablet, electronic digitizer, a microphone, keyboard, and/or pointing device, commonly referred to as mouse, trackball, or touch pad.Other input devices 345 may include a joystick, game pad, satellite dish, scanner, or the like. Additionally, voice inputs, gesture inputs (e.g., via hands or fingers), or other natural user interfaces may also be used with the appropriate input devices, such as a microphone, camera, tablet, touch pad, glove, or other sensor. These andother input devices 345 are often connected to theprocessing unit 302 through a user interface 340 that is coupled to thesystem bus 301, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). Amonitor 360 or other type of display device is also connected to thesystem bus 301 via user interface 340, such as a video interface. Themonitor 360 may also be integrated with a touch-screen panel or the like. - The general
purpose computing device 300 may operate in a networked or cloud-computing environment using logical connections of anetwork Interface 303 to one or more remote devices, such as a remote computer. The remote computer may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the generalpurpose computing device 300. The logical connection may include one or more local area networks (LAN) and one or more wide area networks (WAN), but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet. - When used in a networked or cloud-computing environment, the general
purpose computing device 300 may be connected to a public and/or private network through thenetwork interface 303. In such embodiments, a modem or other means for establishing communications over the network is connected to thesystem bus 301 via thenetwork interface 303 or other appropriate mechanism. A wireless networking component including an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a network. In a networked environment, program modules depicted relative to the generalpurpose computing device 300, or portions thereof, may be stored in the remote memory storage device. - It should be understood from the foregoing that, while particular embodiments have been illustrated and described, various modifications can be made thereto without departing from the spirit and scope of the invention as will be apparent to those skilled in the art. Such changes and modifications are within the scope and teachings of this invention as defined in the claims appended hereto.
Claims (22)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/353,241 US20210398416A1 (en) | 2020-06-19 | 2021-06-21 | Systems and methods for a hand hygiene compliance checking system with explainable feedback |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202063041746P | 2020-06-19 | 2020-06-19 | |
US17/353,241 US20210398416A1 (en) | 2020-06-19 | 2021-06-21 | Systems and methods for a hand hygiene compliance checking system with explainable feedback |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210398416A1 true US20210398416A1 (en) | 2021-12-23 |
Family
ID=79023484
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/353,312 Active 2041-08-27 US11783615B2 (en) | 2020-06-19 | 2021-06-21 | Systems and methods for language driven gesture understanding |
US17/353,241 Pending US20210398416A1 (en) | 2020-06-19 | 2021-06-21 | Systems and methods for a hand hygiene compliance checking system with explainable feedback |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/353,312 Active 2041-08-27 US11783615B2 (en) | 2020-06-19 | 2021-06-21 | Systems and methods for language driven gesture understanding |
Country Status (1)
Country | Link |
---|---|
US (2) | US11783615B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220301413A1 (en) * | 2021-03-19 | 2022-09-22 | Kohler Co. | Systems and method for closed loop tracking of hand hygiene compliance |
WO2024064275A1 (en) * | 2022-09-21 | 2024-03-28 | Omnimed I, Llc | Surgical handwashing system |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11275453B1 (en) | 2019-09-30 | 2022-03-15 | Snap Inc. | Smart ring for manipulating virtual objects displayed by a wearable device |
US11798429B1 (en) | 2020-05-04 | 2023-10-24 | Snap Inc. | Virtual tutorials for musical instruments with finger tracking in augmented reality |
US11520399B2 (en) | 2020-05-26 | 2022-12-06 | Snap Inc. | Interactive augmented reality experiences using positional tracking |
US11925863B2 (en) | 2020-09-18 | 2024-03-12 | Snap Inc. | Tracking hand gestures for interactive game control in augmented reality |
US11587362B2 (en) * | 2020-12-16 | 2023-02-21 | Lenovo (Singapore) Pte. Ltd. | Techniques for determining sign language gesture partially shown in image(s) |
US11740313B2 (en) | 2020-12-30 | 2023-08-29 | Snap Inc. | Augmented reality precision tracking and display |
US12013985B1 (en) * | 2021-02-25 | 2024-06-18 | Snap Inc. | Single-handed gestures for reviewing virtual content |
CN117178247A (en) * | 2021-04-19 | 2023-12-05 | 斯纳普公司 | Gestures for animating and controlling virtual and graphical elements |
WO2024072410A1 (en) * | 2022-09-30 | 2024-04-04 | Innopeak Technology, Inc. | Real-time hand gesture tracking and recognition |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140369596A1 (en) * | 2013-06-15 | 2014-12-18 | Purdue Research Foundation | Correlating videos and sentences |
JP2018045512A (en) * | 2016-09-15 | 2018-03-22 | オムロン株式会社 | Workability evaluation device |
US20180182092A1 (en) * | 2014-05-09 | 2018-06-28 | Given Imaging Ltd. | System and method for sequential image analysis of an in vivo image stream |
US20200075167A1 (en) * | 2018-08-30 | 2020-03-05 | Regents Of The University Of Minnesota | Dynamic activity recommendation system |
US20200074836A1 (en) * | 2018-09-05 | 2020-03-05 | Honeywell International Inc. | Methods and systems for improving infection control in a facility |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10488939B2 (en) * | 2017-04-20 | 2019-11-26 | Microsoft Technology Licensing, Llc | Gesture recognition |
US11625557B2 (en) * | 2018-10-29 | 2023-04-11 | Hrl Laboratories, Llc | Process to learn new image classes without labels |
-
2021
- 2021-06-21 US US17/353,312 patent/US11783615B2/en active Active
- 2021-06-21 US US17/353,241 patent/US20210398416A1/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140369596A1 (en) * | 2013-06-15 | 2014-12-18 | Purdue Research Foundation | Correlating videos and sentences |
US20180182092A1 (en) * | 2014-05-09 | 2018-06-28 | Given Imaging Ltd. | System and method for sequential image analysis of an in vivo image stream |
JP2018045512A (en) * | 2016-09-15 | 2018-03-22 | オムロン株式会社 | Workability evaluation device |
US20200075167A1 (en) * | 2018-08-30 | 2020-03-05 | Regents Of The University Of Minnesota | Dynamic activity recommendation system |
US20200074836A1 (en) * | 2018-09-05 | 2020-03-05 | Honeywell International Inc. | Methods and systems for improving infection control in a facility |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220301413A1 (en) * | 2021-03-19 | 2022-09-22 | Kohler Co. | Systems and method for closed loop tracking of hand hygiene compliance |
US11741818B2 (en) * | 2021-03-19 | 2023-08-29 | Kohler Co. | Systems and method for closed loop tracking of hand hygiene compliance |
WO2024064275A1 (en) * | 2022-09-21 | 2024-03-28 | Omnimed I, Llc | Surgical handwashing system |
Also Published As
Publication number | Publication date |
---|---|
US11783615B2 (en) | 2023-10-10 |
US20210397266A1 (en) | 2021-12-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210398416A1 (en) | Systems and methods for a hand hygiene compliance checking system with explainable feedback | |
Lu et al. | Towards unsupervised physical activity recognition using smartphone accelerometers | |
JP6409507B2 (en) | Method, computer system, and program for associating user activity label with gaze | |
Liu et al. | DeepFaceLIFT: interpretable personalized models for automatic estimation of self-reported pain | |
Kukula et al. | The human–biometric-sensor interaction evaluation method: Biometric performance and usability measurements | |
Pang et al. | Automatic detection and quantification of hand movements toward development of an objective assessment of tremor and bradykinesia in Parkinson's disease | |
Pollreisz et al. | A simple algorithm for emotion recognition, using physiological signals of a smart watch | |
Yanay et al. | Air-writing recognition using smart-bands | |
WO2019085331A1 (en) | Fraud possibility analysis method, device, and storage medium | |
Lu et al. | Out-of-distribution representation learning for time series classification | |
Kadir et al. | Can a simple approach identify complex nurse care activity? | |
Kelly et al. | Automatic prediction of health status using smartphone-derived behavior profiles | |
Ali et al. | Spatio-temporal attention and magnification for classification of Parkinson’s disease from videos collected via the internet | |
Cabra et al. | Wide machine learning algorithms evaluation applied to ECG authentication and gender recognition | |
US20210042971A1 (en) | Information processing device, display method, and program storage medium for monitoring object movement | |
Creagh et al. | Interpretable deep learning for the remote characterisation of ambulation in multiple sclerosis using smartphones | |
Piugie et al. | How artificial intelligence can be used for behavioral identification? | |
Lagodzinski et al. | Codebook-based electrooculography data analysis towards cognitive activity recognition | |
Shadman et al. | Keystroke dynamics: Concepts, techniques, and applications | |
Kawulok et al. | Dynamics of facial actions for assessing smile genuineness | |
Wang et al. | Human identification in metaverse using egocentric iris recognition | |
Cohen et al. | Recognition of continuous sign language alphabet using leap motion controller | |
Barra et al. | Continuous authentication on smartphone by means of periocular and virtual keystroke | |
Venkataraman et al. | Distinguishing mislabeled data from correctly labeled data in classifier design | |
Subramanian et al. | Using photoplethysmography for simple hand gesture recognition |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ARIZONA BOARD OF REGENTS ON BEHALF OF ARIZONA STATE UNIVERSITY, ARIZONA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GUPTA, SANDEEP;BANERJEE, AYAN;AMPERAYANI, VENKATA NAGA SAI APURUPA;SIGNING DATES FROM 20210625 TO 20210702;REEL/FRAME:056760/0074 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |