WO2007129289A1 - A hand washing monitoring system - Google Patents
A hand washing monitoring system Download PDFInfo
- Publication number
- WO2007129289A1 WO2007129289A1 PCT/IE2007/000051 IE2007000051W WO2007129289A1 WO 2007129289 A1 WO2007129289 A1 WO 2007129289A1 IE 2007000051 W IE2007000051 W IE 2007000051W WO 2007129289 A1 WO2007129289 A1 WO 2007129289A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- processor
- monitoring system
- hand
- motion
- hands
- Prior art date
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 51
- 238000005406 washing Methods 0.000 title claims abstract description 39
- 239000013598 vector Substances 0.000 claims abstract description 61
- 230000033001 locomotion Effects 0.000 claims abstract description 58
- 238000004458 analytical method Methods 0.000 claims abstract description 27
- 230000011218 segmentation Effects 0.000 claims abstract description 14
- 238000005259 measurement Methods 0.000 claims abstract description 12
- 230000000694 effects Effects 0.000 claims abstract description 4
- 239000000284 extract Substances 0.000 claims abstract description 4
- 238000000034 method Methods 0.000 claims description 33
- 238000012549 training Methods 0.000 claims description 27
- 238000012706 support-vector machine Methods 0.000 claims description 18
- 238000001514 detection method Methods 0.000 claims description 15
- 238000000605 extraction Methods 0.000 claims description 11
- 238000010606 normalization Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 10
- 230000003287 optical effect Effects 0.000 claims description 7
- 239000000344 soap Substances 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000002547 anomalous effect Effects 0.000 claims description 2
- 239000012141 concentrate Substances 0.000 claims description 2
- 238000000513 principal component analysis Methods 0.000 claims description 2
- 210000004247 hand Anatomy 0.000 description 52
- 238000004422 calculation algorithm Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 6
- 230000007704 transition Effects 0.000 description 6
- 238000013459 approach Methods 0.000 description 5
- 210000003811 finger Anatomy 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 238000009826 distribution Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 3
- 230000003749 cleanliness Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 235000013305 food Nutrition 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 239000000700 radioactive tracer Substances 0.000 description 2
- 229910001220 stainless steel Inorganic materials 0.000 description 2
- 239000010935 stainless steel Substances 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 210000003813 thumb Anatomy 0.000 description 2
- 241000894006 Bacteria Species 0.000 description 1
- LFQSCWFLJHTTHZ-UHFFFAOYSA-N Ethanol Chemical compound CCO LFQSCWFLJHTTHZ-UHFFFAOYSA-N 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011109 contamination Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 239000000645 desinfectant Substances 0.000 description 1
- 230000001627 detrimental effect Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 208000015181 infectious disease Diseases 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000001717 pathogenic effect Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 235000020995 raw meat Nutrition 0.000 description 1
- 230000002459 sustained effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 238000009281 ultraviolet germicidal irradiation Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 210000000707 wrist Anatomy 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/18—Status alarms
- G08B21/24—Reminder alarms, e.g. anti-loss alarms
- G08B21/245—Reminder of hygiene compliance policies, e.g. of washing hands
Definitions
- the invention relates to monitoring of hand washing.
- a recommended technique for hand washing consists of the following phases:
- WO03/079278f describes a system which executes a pattern recognition algorithm with digitized images to generate a pass/fail report for hand washing or disinfectant application.
- WO2005/093681 describes a cleanliness monitoring system and method involving a fluorescent tracer, a UV irradiation source, and determining the proportion of the hands which are clear of tracer.
- GB2337327 describes pixel-by-pixel analysis of soap on hands
- WO98/36258 also describes an approach involving pixel analysis to detect soap on hands.
- US5952924 describes an RFID badge to trigger measurement of hand washing.
- a motion sensor is used to trigger the device and an alcohol sensor is used to detect if hand washing occurs.
- US6236317 describes an RFID-based tracking system to ensure that people wash their hands when entering and leaving defined zones such as toilets and raw meat counters. IR reflection sensors are used to localise the user in front of the sink.
- US6426701 describes use of audio visual feedback to guide the user through hand washing. Proximity sensors are used to ensure that the user's hands have remained in the sink for 20 seconds.
- the invention is directed towards achieving improved monitoring of hand washing.
- a hand washing monitoring system comprising a camera, a processor, the processor being adapted to receive from the camera images of hand washing activity, characterized in that, the processor is adapted to:
- the processor analyses the images within a region of interest encompassing joined hands.
- the processor generates the indication independently of the order of motion poses.
- the processor extracts information features from the images and generates feature vectors based on the features, including bimanual hand and arm shape vectors, and executes a classifier with the vectors to determine the poses.
- the processor uses edge segmentation to form at least some of the feature vectors.
- the processor uses pixel spatio-temporal measurements to form at least some of the feature vectors.
- the processor decomposes an image into cells and uses each cell to calculate a histogram, and combines the histograms to form a feature vector.
- the processor generates the feature vectors using rules derived from a training phase performed with reference images. In another embodiment, the processor reduces size of the feature vectors.
- the processor reduces feature vector size using Principal Component Analysis.
- the processor uses linear discriminate analysis to reduce feature vector size.
- a self-organising map is generated to represent clustered datasets of a reduced feature vector.
- the processor classifies the feature vectors by executing a multi-class classifier which is trained with exemplar feature vectors to generate an estimate of a hand pose.
- the processor performs the classification using a support vector machine.
- the processor executes an ensemble of support vector machines.
- the processor determines a pose according to votes from different support vector machines.
- the processor filters bimanual hand pose classifications.
- the processor performs said filtering by using probabilistic bimanual hand motion models to remove anomalous results.
- the processor determines a time duration for a pose by registering a count of the number of image frames for a pose. In one embodiment, the processor assigns a minimum threshold for each pose.
- the processor is calibrated for light level and/or colour changes in an image normalisation process.
- the processor performs segmentation according to colour, texture and/or motion to avoid noise arising from reflections.
- the processor eliminates pixels representing watches or jewellery according to region size and shape analysis.
- the system is battery powered.
- the classifier comprises an ensemble of weak classifiers.
- the processor before feature extraction, detects skin by normalizing pixel colour values to skin and non-skin class bins, and processes bin counts to determine skin probability values.
- the processor executes a lighting compensation filter operating on the basis that the spatial average of surface reflectance in a scene is achromatic.
- the processor performs optical flow calculations based on pixel distance moved and direction of movement, and applies motion data to a filter to eliminate skin false-positive pixels.
- the processor determines increase and decrease factors representing pixel motion increase and decrease, and applies a greater weighting to motion than absence of motion.
- the processor performs hand and arm detection by monitoring geometric characteristics of pixel blobs, including major and minor axes.
- the processor performs feature extraction by generating histograms of gradient orientation in a local part of an image by accumulating votes into each of a plurality of bins, each for an orientation.
- the processor generates a gradient orientation histogram for each of a plurality of pixel cells comprising pre-defined numbers of pixels.
- the processor concentrates all histograms into a single vector.
- the processor normalizes cell histograms.
- the processor performs a training phase in which vectors are generated from training images.
- a soap dispensing unit comprising a monitoring system as defined above.
- a computer readable medium comprising software code for performing operations of a processor of a monitoring system as defined above.
- Fig. 1 is a high-level representation of a monitoring system of the invention
- Fig. 2 is a flow diagram showing operation of the system
- Fig. 3 shows, left, an original image and, right, a compensated image with a standard mean of 97;
- Fig. 5 shows skin and motion detection for a hands/arms segmentation scheme
- Fig. 6 shows hand ROI selection
- Fig. 7 shows HOG description, (a) original image, (b) gray image, (c) gradient image, (d) HOG cells distribution and blocks normalization;
- Fig. 8 is a flow diagram showing training to generate feature selection rules, and performance of feature selection during operation of the system
- Fig. 9 shows single frame classification results in a sequence
- Fig. 10 is a perspective view of a stand-alone battery-operated sensor head of an alternative monitoring system.
- a monitoring system 1 comprises a camera 2, an illuminator 3, a processor 4, a storage device 5, and a display 6.
- the camera 2 captures images of handwashing activity and these images are processed by the processor 4 to determine if the desired hand motions have taken place by using motion tracking techniques.
- the system 1 tracks the hand motions and classifies the type of motion pattern used by performing the following steps: - Detection of the hands. - Detection if the hands are joined.
- a multi-class classifier such as an ensemble of Support Vector Machine (SVM) or a cascade of weak classifiers to determine if the hands are in one of the different hand washing motions.
- SVM Support Vector Machine
- the camera 2 captures images, and the processor 4 analyses the images to analyse bimanual hand washing motions to check if all of the required steps of hand washing have been adequately completed.
- the image processing information is used to determine if the hands are moved in predetermined patterns, that the patterns are each executed for a minimum length of time, and that all required patterns in the hand washing procedure are executed.
- the brightness, colour balance, and contrast of the image are modified on the basis of parameters measured during a set-up phase.
- a typical example of this is to determine the white balance of the camera by the camera capturing an image of a pure white card and associating the values it reads with pure white.
- Lighting compensation or color constancy algorithms are used.
- a greyworld algorithm is used in one embodiment. It is based on the assumption that the spatial average of surface reflectance in a scene is achromatic. Since the light reflected from an achromatic surface is changed equally at all wavelengths, it follows that the spatial average of the light leaving the scene will be the color of the incident illumination.
- the grey-world algorithm is defined as,
- BGR avg is mean values of the specific channel in a specific frame.
- the limitation of the grey-world algorithm is that the constant standard mean value (128) will not fit well for images with a dark background, which will be over-compensated.
- the processor computes the standard mean value as:
- the over-compensation problem is solved.
- an adaptive mean gray value of the whole image is obtained.
- the BGR avg is the mean of the non-black pixels in each channel.
- the scale factor Sc — C s t d /BGR aV g is applied for all pixels in the image.
- a skin detector such as a PoesiaTM filter identify the regions of the image that can be classified as skin. Sometimes reflections from specular surfaces such as stainless steel sinks will also be classified as skin using this method.
- Non-parametric skin modeling with Bayesian models based on histograms is applied. The skin and non-skin colors are modelled through histograms.
- the color space RGB is normalized to a number of bins rgb e RGB and the processor counts the number of color pixels in each bin A ⁇ n (rgb) for skin class as well as N nons un (rgb) for non-skin class.
- each bin is normalized to get the discrete conditional sldn/non-sMn color distribution p(rgb ⁇ skin)/(rgb ⁇ nonski ⁇ ).
- TS and IW denote the total counts contained in the skin and non-skin histograms, i.e., the number of skin and non-skin pixels in the training set, we have:
- Skin and non-skin reference histograms were obtained from the open source filtering according to the Poesia project.
- the project has compiled 323 3D RGB histograms from the Compaq database and has left them available in two files, one for skin pixels and other for non-skin pixels. Prior probabilities of skin and non-skin are assumed to be 0.4 and 0.6 respectively.
- the skin detection results in a skin probability map Xp(i, j).
- a skin binary map Xb(i, j) can be made upon a properly selected threshold Th, 0 ⁇ Th ⁇ 1.
- the pixel is said to be a skin pixel if p(skin/rgb) > Th, or a non-skin pixel if p(skin/rgb) " Th.
- TMs determines the way that pixels move from one image to the next. The movement at each pixel is represented as distance moved and a direction of movement. As the hands are the primary moving object in the scene they will produce the primary optical flow vectors. Reflections and water can also produce optical flow measurements. Motion information is calculated using dense optical flow measurements using the Lucas Kanade method. Coarser methods such as block matching do not provide sufficient per-pixel information to produce a reliable segmentation.
- the technique consists in computing an averaged image integrating both skin and motion features. For each skin pixel, which could be a false positive, the processor measures the motion, after applying an average filter over the motion magnitude image. If the averaged motion magnitude is greater than a fixed threshold Tm the probability is increased. If it is less than Tm, then the probability is decreased.
- Tm a fixed threshold
- I F and D F are the increase and decrease factors with which the processor tunes how fast or slow the variations are when there were motion or not
- M t is the motion magnitude of pixel i
- Tm is the threshold value which decides if there is going to be an increase or a decrease in the output image.
- T J D 50
- the system tries to determine if the hands have gone near the taps. This would be a high risk gesture but only if it occurs at the end of the hand wash cycle.
- the processor applies a simple connected components analysis over the binary image, removing small area components. Then, a region of interest covering the hands is selected. The vertical symmetry axis is used for computing upper and lower boundaries, which are then used for computing lateral ones. This process is depicted in Fig. 6.
- the region of interest should be square sized in order to keep a constant size and not distort the hands pose.
- the processor enlarges the width or the height to obtain a square region of interest. Then it resizes the ROI to a fixed size of 128 x 128 where features will be subsequently extracted.
- the range of hand sizes, large hands, small hands, and the position of the hands, i.e. close to the camera, or far from the camera change the size of the hands in the image.
- the processor passes multiple ROIs to the feature extraction system e.g. 128x128, 64x64, 32x32. Extracting features across a range of scales makes the classification more tolerant to different sizes of hands in the image.
- the system extracts information and measurements from the images.
- the processor uses local Histograms of Oriented Gradients (HOG) as a single frame feature extraction method
- HOG Oriented Gradients
- the aim of this method is to describe an image by a set of local histograms. These histograms count occurrences of gradient orientation in a local part of the image.
- the gradient image is computed.
- the image is split in cells which can be defined as a spatial region like a square with a predefined size in pixels.
- the processor computes the histogram of gradients by accumulating votes into bins for each orientation. Votes are weighted by the magnitude of a gradient, so that the histogram takes into account the importance of gradient at a given point.
- the processor When all histograms have been computed for each cell, the processor builds the descriptor vector of an image, concatenating all histograms in a single vector.
- Cell histograms are locally normalized according to values of the neighboured cell histograms. The normalization is done among a group of cells, which is called a block. A normalization factor is then computed over the block and all histograms within this block are normalized according to this normalization factor. Once this normalization step has been performed, all the histograms can be concatenated in a single feature vector.
- e is a small regularization constant needed because sometimes empty gradients are going to be evaluated.
- a histogram from a given cell can be involved in several block normalization. Thus, there is some redundant information which improves the performance. This method is illustrated in Fig. 7.
- ROI dimension In order to compute the vector dimension several parameters have to be taken into account: ROI dimension, cell size, block size, number of bins, and number of overlapped blocks.
- the ROI size is 128 x 128, each window is divided into cells of size 16 x 16 and each group of 2 x 2 cells is integrated into a block in a sliding fashion, so blocks overlap with each other.
- Each cell consists of a 16-bin Histogram of Oriented Gradient (HOG) and each block contains a concatenated vector of all its cells.
- HOG Oriented Gradient
- Each block is thus represented by a 64 feature vector which is normalized to an L2 unit length.
- Each 128 x 128 region of interest is represented by:
- CellsRow ROIwidth/cellswidth (H)
- CellScolumn ROlHeight/cells H eight ( 12)
- OverlapBlockscoiumn Cellscoiumn - CellsBlockcoiumn + 1 (16)
- the feature vectors are of 3,136 dimension.
- Fig. 8 in a training phase the system uses stored videos of gestures previously identified by human observers. This information is used to select the relevant features, train the classifier and tune the filters.
- "normal operations" the system executes the image processing, feature extraction, feature selection, classification and filtering operations.
- the training phase includes:
- the full set of features are generated using histogram of Orientation Gradients, or histogram of flows (or "spatio-temporal motion"). This generates a large number of features not all of which are contributing positively to the classification.
- the dimensionality of the feature set is reduced to identify and use only that sub-set of the features which contribute most to the classification of the hand motions.
- Mathematical techniques such as Principal Components Analysis or Linear Discriminant Analysis are used to identify this subset of features. This process is called feature selection.
- the subset of features to be used typically about 10% of the original set, is passed to the classifier. The classifier learns to associate the features with the motion patterns and thus identifies the hand motions used in hand washing.
- the final stage of training is to tune the filters that detect and correct errors of classification by examining the sequence of hand motions along with the underlying prior probabilities of the hand poses.
- Mathematical structures such as Kalman filters, Bayesian networks or Markov decision processes are used to implement these filters.
- ROI regions of interest
- feature extraction algorithms are run.
- a subset of the features are selected according to the prior training and concatenated to form a feature vector.
- This feature vector is passed to the classifier that determines which of the hand motion is currently in use. This result is filtered to remove errors resulting in the final hand motion classification.
- An array of the measurements suitable for classification are made and only those measurements which contribute most to the classification are used.
- the measurements from the feature extraction algorithms are combined to create an array called a feature vector, and this could contain over 3000 different measurements.
- Two approaches to feature vector building are processing histograms of edge orientation, and processing histograms of spatio-temporal motion.
- histograms of flows are used to create a feature vector using the spatio- temporal information a region of interest around the hands and arms is created and dense motion vectors are extracted. The image is divided into cells and the histograms of the motion vector directions are calculated. The histograms from several subsequent frames may also be added to calculate the motion direction histogram over a sequence of frames. The histograms are normalised either globally or in overlapping local neighbourhoods. The resulting histograms are combined to form a feature vector that is passed to the multi-class classifier. This is described in more detail above.
- the processor analyses the variance in the feature vector using Principal Components Analysis (PCA). This helps the processor 5 to identify which feature measurements are contributing most to a correct classification. The processor keeps only those features which contribute most to the correct classification; this could typically be only 10% of our original features.
- PCA Principal Components Analysis
- a data reduction technique other than PCA is used, such as linear discriminate analysis (LDA), particularly Fischer LDA.
- LDA linear discriminate analysis
- Using smaller and better clustered data sets can allow the use of a simple K-nearest neighbour classifier that can be implemented easily in hardware.
- the feature vector is passed to a multi-class classifier to produce an estimate of the current hand pose.
- a simple classifier such as KNN (K nearest neighbour, or "supervised K means" can be used. If a more complex and general classification problem exists an ensemble of Support Vector Machine or a Cascade of Weak Classifiers can be used.
- a multi-class classification system is trained using exemplar feature vectors to produce reliable classifications, a computationally efficient structure and one which generalises well to the wide variety of hand shapes, sizes and bi-manual configurations.
- One such approach is to use an ensemble of support vector machines.
- Several strategies may be used to perform this type of ensemble classification.
- One such strategy is to train an individual support vector machine to classify one pose against a different pose.
- Multiple support vector machines are trained to cover all possible combinations.
- the results of the different support vector machines can be combined using a voting scheme with the pose which has the highest number of votes being selected as the correct pose.
- the Support Vector Machine classifier is a binary classifier algorithm that looks for an optimal hyperplane as a decision function in a high dimensional space.
- the class label of x is then obtained by considering the sign of/(x).
- ⁇ , r and d are kernel parameters.
- the LIBSVM library is used for training and test.
- the RBF kernel non-linear maps samples into a higher dimensional space, so it, unlike the linear kernel can handle the case when the relation between class labels and attributes is nonlinear.
- the second reason is the number of hyper-parameters which influences the complexity of the model since polynomial kernel has more hyper-parameters than the RBF kernel.
- RBF kernel has less numerical difficulties.
- the bimanual hand poses are represented as a table of transitions from prior poses with probabilities assigned. For example, if the current pose is B and the previous was E, then the likelihood of this transition is 80%.
- AU prior and current poses are assigned poses based on training and analysis. Low probability poses are deleted as they are likely to be the result of misclassification.
- a post classification filter allows for a small and efficient classifier to be used. Occasional classification errors are caught and rectified based on a probabilistic model of hand position reachability. For example, it is unlikely that hands may move immediately from pose A to pose F without some transition pose. It is therefore more likely that pose F is in fact a misclassification and the filter assigns the most likely transition pose. Filters operate on multiple time intervals; frame-to-frame transitions and weighted averages over several frame intervals to produce a reliable sequence of hand pose classifications to the hand motion analysis system.
- Poses which are sustained for a threshold number of frames are passed to an accumulator which counts the number of frames that each pose has been maintained for, which effectively counts the amount of time spent in each pose.
- Each pose has a configurable threshold minimum time. When all thresholds are exceeded the hands are deemed to have been washed to a sufficiently high standard.
- the hand pose information is used to complete a hand hygiene assessment. Each of the hand poses must be recorded for a configurable time period. Bi-manual hand washing poses can be executed in any order. This information is integrated over time and provided to the user via a graphical display such as a speaker or an audio display device. Site-specific options can be implemented regarding overall hand washing metrics such as total time and critical versus non critical hand poses.
- the system 1 displays on the simple low-power display 6 permanent graphics of the individual hand poses.
- the display is a low power LCD display. This indicates to the user that each pose has been achieved.
- the graphics can be easily customised for each installation.
- the system must deal with the challenges posed by changing lighting.
- a typical scenario is where a hand washing facility is positioned where there is a mix of natural and artificial lighting.
- Each of these light sources cause colour casts in the image and depending on the time of day the proportions of artificial and natural light can vary significantly.
- the detection of the hands is a significant issue and uses colour as a significant feature algorithms have been developed to calibrate for colour changes. These are contained within an image normalisation process.
- a supervised labeling process is used for generating training and test data sets. Firstly a database with ROI images and their labels is generated. Thus the reuse of the database is assured. Secondly an off-line application computes the feature vectors for both training and test procedures. Once we have one classifier correctly trained we can use it for on-line pose recognition. The number of training and test samples is depicted in Table 1. We have one class for each one of the different poses of the recommended hand washing procedure.
- Fig. 9 depicts the single frame classification results in a sequence.
- the detected class is shown with a picture at the top left of the image.
- the system correctly detects when the hands are separated or joined. Even with a single frame approach the classifier correctly detects the transitions between poses, which are classified as other poses class.
- Second, fifth and sixth poses are correctly classified in both cases: when the left hand is rubbing the right one and vice versa.
- the system 160 comprises a housing 161 with a camera window 162 facing towards a sink unit.
- a processor is contained within the housing, and the system is battery powered.
- a background plate 163 extends outwardly.
- the system can provide the hand washing data to an on-system storage device or can be broadcast via a wireless link to central data storage.
- the records can for part of a HACCP (Hazard Analysis and Critical Control Points) records for use in food preparation or other hygiene standards.
- HACCP Hazard Analysis and Critical Control Points
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Psychiatry (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Public Health (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Epidemiology (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Emergency Management (AREA)
- Image Analysis (AREA)
- Emergency Alarm Devices (AREA)
- Domestic Plumbing Installations (AREA)
- Iron Core Of Rotating Electric Machines (AREA)
- Closed-Circuit Television Systems (AREA)
- Control Of Washing Machine And Dryer (AREA)
- Detergent Compositions (AREA)
Abstract
A hand washing monitoring system (1) comprising a camera (2), a processor (4), the processor being adapted to receive from the camera images of hand washing activity. The processor analyses mutual motion of hands to determine if the hands mutually move in desired poses, and if so, the durations of the patterns; and generates a hand washing quality indication according to the analysis. The processor extracts information features from the images and generates feature vectors based on the features, including bimanual hand and arm shape vectors, and executes a classifier with the vectors to determine the poses. The processor uses edge segmentation and pixel spatio-temporal measurements to form at least some of the feature vectors.
Description
"A Hand Washing Monitoring System"
INTRODUCTION
Field of the Invention
The invention relates to monitoring of hand washing.
Prior Art Discussion
It is well known and documented that transmission of infection in environments such as hospitals and kitchens due to incorrect hand washing is widespread, has a major detrimental effect on health, and a huge financial cost.
In both healthcare and food preparation environments training is given in hand washing techniques. While there is a recognized problem to encourage people to wash their hands in the first place, it is also critical that they do so in an effective manner — by following the guidelines. It has been shown that particular areas of the hand are frequently missed with hand washing - notably the tips of fingers and the thumbs. Fingers are thought to be the most important part of the hand in terms of the transfer of pathogenic mirco organisms.
While the recommended time for hand washing is a minimum of 15 seconds of hand rubbing, studies have shown that both long (3 mins) and short (10 seconds) can produce a ten fold reduction in the median number of transient bacteria and suggests that hand washing technique is more important than duration.
A recommended technique for hand washing consists of the following phases:
1. Apply soap, wet hands and rub palms together.
2. Rub right palm over the back of the left hand up to wrist level, and do the same with the other hands.
3. With the right hand over the back of the left hand rub the fingers, do the same with the other hand.
4. Rub palm to palm with the fingers interlaced.
5. Wash the thumbs of each hand separately.
6. Rub the tips of the fingers against the opposite palm.
7. Rinse hands thoroughly removing all traces of soap.
8. Dry hands (without touching the taps).
WO03/079278f describes a system which executes a pattern recognition algorithm with digitized images to generate a pass/fail report for hand washing or disinfectant application.
WO2005/093681 describes a cleanliness monitoring system and method involving a fluorescent tracer, a UV irradiation source, and determining the proportion of the hands which are clear of tracer.
GB2337327 describes pixel-by-pixel analysis of soap on hands, and WO98/36258 also describes an approach involving pixel analysis to detect soap on hands.
US5952924 describes an RFID badge to trigger measurement of hand washing. A motion sensor is used to trigger the device and an alcohol sensor is used to detect if hand washing occurs.
US6236317 describes an RFID-based tracking system to ensure that people wash their hands when entering and leaving defined zones such as toilets and raw meat counters. IR reflection sensors are used to localise the user in front of the sink.
US6426701 describes use of audio visual feedback to guide the user through hand washing. Proximity sensors are used to ensure that the user's hands have remained in the sink for 20 seconds.
While various techniques have been described in the art it appears that they suffer from not analysing the hand washing in a very comprehensive manner.
The invention is directed towards achieving improved monitoring of hand washing.
SUMMARY OF THE INVENTION
According to the invention, there is provided a hand washing monitoring system comprising a camera, a processor, the processor being adapted to receive from the camera images of hand washing activity, characterized in that, the processor is adapted to:
analyse mutual motion of hands to determine if the hands mutually move in desired poses, and if so, the durations of the patterns; and
generate a hand washing quality indication according to the analysis.
In one embodiment, the processor analyses the images within a region of interest encompassing joined hands.
In another embodiment, the processor generates the indication independently of the order of motion poses.
In a further embodiment, the processor extracts information features from the images and generates feature vectors based on the features, including bimanual hand and arm shape vectors, and executes a classifier with the vectors to determine the poses.
In one embodiment, the processor uses edge segmentation to form at least some of the feature vectors.
In another embodiment, the processor uses pixel spatio-temporal measurements to form at least some of the feature vectors.
In a further embodiment, the processor decomposes an image into cells and uses each cell to calculate a histogram, and combines the histograms to form a feature vector.
In one embodiment, the processor generates the feature vectors using rules derived from a training phase performed with reference images.
In another embodiment, the processor reduces size of the feature vectors.
Li a further embodiment, the processor reduces feature vector size using Principal Component Analysis.
In one embodiment, the processor uses linear discriminate analysis to reduce feature vector size.
In another embodiment, a self-organising map is generated to represent clustered datasets of a reduced feature vector.
In a further embodiment, the processor classifies the feature vectors by executing a multi-class classifier which is trained with exemplar feature vectors to generate an estimate of a hand pose.
In one embodiment, the processor performs the classification using a support vector machine.
In another embodiment, the processor executes an ensemble of support vector machines.
In a further embodiment, the processor determines a pose according to votes from different support vector machines.
hi one embodiment, the processor filters bimanual hand pose classifications.
In another embodiment, the processor performs said filtering by using probabilistic bimanual hand motion models to remove anomalous results.
In a further embodiment, the processor determines a time duration for a pose by registering a count of the number of image frames for a pose.
In one embodiment, the processor assigns a minimum threshold for each pose.
In another embodiment, the processor is calibrated for light level and/or colour changes in an image normalisation process.
In a further embodiment, the processor performs segmentation according to colour, texture and/or motion to avoid noise arising from reflections.
In one embodiment, the processor eliminates pixels representing watches or jewellery according to region size and shape analysis.
In another embodiment, the system is battery powered.
In a further embodiment, the classifier comprises an ensemble of weak classifiers.
In one embodiment, the processor, before feature extraction, detects skin by normalizing pixel colour values to skin and non-skin class bins, and processes bin counts to determine skin probability values.
In another embodiment, the processor executes a lighting compensation filter operating on the basis that the spatial average of surface reflectance in a scene is achromatic.
In a further embodiment, the processor performs optical flow calculations based on pixel distance moved and direction of movement, and applies motion data to a filter to eliminate skin false-positive pixels.
hi one embodiment, the processor determines increase and decrease factors representing pixel motion increase and decrease, and applies a greater weighting to motion than absence of motion.
In another embodiment, the processor performs hand and arm detection by monitoring geometric characteristics of pixel blobs, including major and minor axes.
In a further embodiment, the processor performs feature extraction by generating histograms of gradient orientation in a local part of an image by accumulating votes into each of a plurality of bins, each for an orientation.
In one embodiment, the processor generates a gradient orientation histogram for each of a plurality of pixel cells comprising pre-defined numbers of pixels.
In another embodiment, the processor concentrates all histograms into a single vector.
In a further embodiment, the processor normalizes cell histograms.
In one embodiment, the processor performs a training phase in which vectors are generated from training images.
In another aspect, there is provided a soap dispensing unit comprising a monitoring system as defined above.
In a further aspect, there is provided a computer readable medium comprising software code for performing operations of a processor of a monitoring system as defined above.
DETAILED DESCRIPTION QF THE INVENTION
Brief Description of the Drawings
The invention will be more clearly understood from the following description of some embodiments thereof, given by way of example only with reference to the accompanying drawings in which:-
Fig. 1 is a high-level representation of a monitoring system of the invention;
Fig. 2 is a flow diagram showing operation of the system;
Fig. 3 shows, left, an original image and, right, a compensated image with a standard mean of 97;
Fig. 4 shows motion (left) increase and (right) decrease functions with a threshold Tro=50;
Fig. 5 shows skin and motion detection for a hands/arms segmentation scheme;
Fig. 6 shows hand ROI selection;
Fig. 7 shows HOG description, (a) original image, (b) gray image, (c) gradient image, (d) HOG cells distribution and blocks normalization;
Fig. 8 is a flow diagram showing training to generate feature selection rules, and performance of feature selection during operation of the system;
Fig. 9 shows single frame classification results in a sequence; and
Fig. 10 is a perspective view of a stand-alone battery-operated sensor head of an alternative monitoring system.
Description of the Embodiments
Referring to Fig. 1 a monitoring system 1 comprises a camera 2, an illuminator 3, a processor 4, a storage device 5, and a display 6. The camera 2 captures images of handwashing activity and these images are processed by the processor 4 to determine if the desired hand motions have taken place by using motion tracking techniques.
The system 1 tracks the hand motions and classifies the type of motion pattern used by performing the following steps: - Detection of the hands.
- Detection if the hands are joined.
- Creating a region of interest around the joined hands.
- Extracting edge and region information for the hands within the region of interest to provide feature vectors.
- Classifying the bimanual hand motion information (in the form of the feature vectors) using a multi-class classifier such as an ensemble of Support Vector Machine (SVM) or a cascade of weak classifiers to determine if the hands are in one of the different hand washing motions.
- Ensuring that each of the steps of the hand washing have been achieved for the desired amount of time. The sequence of the washing steps is not important. What is important is the set of bimanual hand motions performed.
The camera 2 captures images, and the processor 4 analyses the images to analyse bimanual hand washing motions to check if all of the required steps of hand washing have been adequately completed.
The image processing information is used to determine if the hands are moved in predetermined patterns, that the patterns are each executed for a minimum length of time, and that all required patterns in the hand washing procedure are executed.
The following describes the method in more detail, with reference to the steps of Fig. 2 and also to Figs. 3 - 12.
Image Normalisation, 70
In this step the brightness, colour balance, and contrast of the image are modified on the basis of parameters measured during a set-up phase. A typical example of this is to determine the white balance of the camera by the camera capturing an image of a pure white card and associating the values it reads with pure white.
Lighting compensation or color constancy algorithms are used. A greyworld algorithm is used in one embodiment. It is based on the assumption that the spatial average of surface reflectance in a scene is achromatic. Since the light reflected from an achromatic surface is changed equally at all wavelengths, it follows that the spatial
average of the light leaving the scene will be the color of the incident illumination. The grey-world algorithm is defined as,
B G R (1) avg avg
where 5Gi?c is the scale factor for every channel. BGRavg is mean values of the specific channel in a specific frame. BGRstd is the 50% ideal grey under the canonical, i.e. the BGRavg given by BGRavg = x/ι-BGRcanonicaι = Vilδβ = 128. The limitation of the grey-world algorithm is that the constant standard mean value (128) will not fit well for images with a dark background, which will be over-compensated. The processor computes the standard mean value as:
_ ∑ :iM^.,G,.?J?;.)+minfe,Gi.;i?j] Csid _ _ (2)
where m stands for the number of pixels in the image and n stands for the number of non-black pixels in the image. Thus the over-compensation problem is solved. By calculating the average of the maximum and minimum channel percentage, an adaptive mean gray value of the whole image is obtained. The BGRavg is the mean of the non-black pixels in each channel. The scale factor Sc — Cstd/BGRaVg is applied for all pixels in the image.
Skin Detection, 71
Using a skin detector such as a Poesia™ filter identify the regions of the image that can be classified as skin. Sometimes reflections from specular surfaces such as stainless steel sinks will also be classified as skin using this method.
Non-parametric skin modeling with Bayesian models based on histograms is applied. The skin and non-skin colors are modelled through histograms. The color space RGB is normalized to a number of bins rgb e RGB and the processor counts the number of color pixels in each bin A^n (rgb) for skin class as well as Nnonsun (rgb) for non-skin class. Finally, each bin is normalized to get the discrete conditional sldn/non-sMn color distribution p(rgb\skin)/(rgb\nonskiή). Letting TS and IW denote the total counts contained in the skin and non-skin histograms, i.e., the number of skin and non-skin pixels in the training set, we have:
P(rgb IsMn) = N^rgb) ; P(rgb I nonsldn) = N»°»^rgb) (4)
1 N
As well as the priors:
P(sMn) = — ^ — ; P(nonskiή) = TN = 1 - P(skin) (5)
Ts + FN TS + TN
Then the Bayesian formula is employed to estimate the skin/non-skin probability according to the color of a given pixel:
Skin and non-skin reference histograms were obtained from the open source filtering according to the Poesia project. The project has compiled 323 3D RGB histograms from the Compaq database and has left them available in two files, one for skin pixels and other for non-skin pixels. Prior probabilities of skin and non-skin are assumed to be 0.4 and 0.6 respectively. After applying equation (6) the skin detection results in a skin probability map Xp(i, j). A skin binary map Xb(i, j) can be made upon a properly selected threshold Th, 0 < Th < 1. The pixel is said to be a skin pixel if p(skin/rgb) > Th, or a non-skin pixel if p(skin/rgb) " Th.
Optical Flow Calculation. 72
TMs determines the way that pixels move from one image to the next. The movement at each pixel is represented as distance moved and a direction of movement. As the hands are the primary moving object in the scene they will produce the primary optical flow vectors. Reflections and water can also produce optical flow measurements. Motion information is calculated using dense optical flow measurements using the Lucas Kanade method. Coarser methods such as block matching do not provide sufficient per-pixel information to produce a reliable segmentation.
Background Subtraction, 73
In order to have a robust segmentation process motion information is also used. The technique consists in computing an averaged image integrating both skin and motion features. For each skin pixel, which could be a false positive, the processor measures the motion, after applying an average filter over the motion magnitude image. If the averaged motion magnitude is greater than a fixed threshold Tm the probability is increased. If it is less than Tm, then the probability is decreased. The increase and decrease values are computed according to the following equations:
where IF and DF are the increase and decrease factors with which the processor tunes how fast or slow the variations are when there were motion or not, Mt is the motion magnitude of pixel i, and Tm is the threshold value which decides if there is going to be an increase or a decrease in the output image. In Fig. 4 the increase and decrease functions are depicted for a threshold TJD = 50. The increase function reaches the maximum value quickly, whereas the decrease function is less steep. This is because
to stress quickly the motion when it appears and penalize the absence of motion in a slower way.
Thus skin pixels due to metal sinks and specular reflections, are removed while the hands/arms are correctly segmented only if they are moving.
Accordingly, the problem of hands/arms miss detection when these ones are stopped remains unresolved. In Fig. 5 we can see the global architecture of the skin-motion segmentation.
Hand and Arm Detection, 75
Even with the filtering in steps 71, 72 and 73 some erroneous regions can still exist. The largest blobs in the scene are analysed to find regions that may be arms and hands. The shape parameters of the blobs: size, major versus minor axis, and angle of major and minor axes are used to identify the hands and arms.
Hands Joined decision, 76
By analysing the shape parameters of the hand and arm blobs a decision can be made about whether the hands are joined in a rubbing motion.
High Risk Gesture Recognition, 77
If the hands are not joined the system tries to determine if the hands have gone near the taps. This would be a high risk gesture but only if it occurs at the end of the hand wash cycle.
Monitoring hand motions during washing is important to ensure the cleanliness of hands. However it is also important to prevent contamination of the hands at the end of hand washing because the hands rather than an elbow were used to turn off the water supply. These actions must also be interpreted in context as it is hygienic to use the hands at the start of hand washing to turn on the water but not hygienic to do so at the end of hand washing. Thus the detection of the actions is separate to the classification of the risk behaviour.
Region of Interest Calculation, 78
Based on the location of the hands within the image a region of interest around the hands is created.
Once the hands and the arms are segmented the processor applies a simple connected components analysis over the binary image, removing small area components. Then, a region of interest covering the hands is selected. The vertical symmetry axis is used for computing upper and lower boundaries, which are then used for computing lateral ones. This process is depicted in Fig. 6.
The region of interest should be square sized in order to keep a constant size and not distort the hands pose. Sometimes, the processor enlarges the width or the height to obtain a square region of interest. Then it resizes the ROI to a fixed size of 128 x 128 where features will be subsequently extracted. Sometimes the range of hand sizes, large hands, small hands, and the position of the hands, i.e. close to the camera, or far from the camera change the size of the hands in the image. To compensate for these scale changes the processor passes multiple ROIs to the feature extraction system e.g. 128x128, 64x64, 32x32. Extracting features across a range of scales makes the classification more tolerant to different sizes of hands in the image.
Feature Extraction. 79
Using algorithms such as the histograms of optical flows, histograms of orientation gradients, or reduced scale images of the hands, the system extracts information and measurements from the images.
Choosing discriminating and independent features is important to pattern recognition being successful in classification. Where the processor uses local Histograms of Oriented Gradients (HOG) as a single frame feature extraction method, the aim of this method is to describe an image by a set of local histograms. These histograms count occurrences of gradient orientation in a local part of the image. First the gradient image is computed. Then the image is split in cells which can be defined as a spatial region like a square with a predefined size in pixels. For each cell, the processor computes the histogram of gradients by accumulating votes into bins for each
orientation. Votes are weighted by the magnitude of a gradient, so that the histogram takes into account the importance of gradient at a given point.
When all histograms have been computed for each cell, the processor builds the descriptor vector of an image, concatenating all histograms in a single vector. However, due to the variability in the images, it is necessary to normalize cell histograms. Cell histograms are locally normalized according to values of the neighboured cell histograms. The normalization is done among a group of cells, which is called a block. A normalization factor is then computed over the block and all histograms within this block are normalized according to this normalization factor. Once this normalization step has been performed, all the histograms can be concatenated in a single feature vector. We use the L2-norm scheme:
v → * (10)
Where e is a small regularization constant needed because sometimes empty gradients are going to be evaluated. According to how each block has been built, a histogram from a given cell can be involved in several block normalization. Thus, there is some redundant information which improves the performance. This method is illustrated in Fig. 7.
In order to compute the vector dimension several parameters have to be taken into account: ROI dimension, cell size, block size, number of bins, and number of overlapped blocks. The ROI size is 128 x 128, each window is divided into cells of size 16 x 16 and each group of 2 x 2 cells is integrated into a block in a sliding fashion, so blocks overlap with each other. Each cell consists of a 16-bin Histogram of Oriented Gradient (HOG) and each block contains a concatenated vector of all its cells. Each block is thus represented by a 64 feature vector which is normalized to an L2 unit length. Each 128 x 128 region of interest is represented by:
CellsRow = ROIwidth/cellswidth (H)
CellScolumn = ROlHeight/cellsHeight ( 12)
CellsBlockRow = blockwidth/cellswidth (13)
CellsBlcokcoiumn = blockπeight/cellsHeight (14)
OverlapBlocksROW = CellsRow - CellsBlockRow + 1 (15)
OverlapBlockscoiumn = Cellscoiumn - CellsBlockcoiumn + 1 (16)
VectorDIM = OverlapBlocksRow * OverlapBlockscoiumn * (17)
CellsBlockRow * CellsBlockcoiumn * binsH0G (18)
Thus, the feature vectors are of 3,136 dimension.
Referring to Fig. 8 in a training phase the system uses stored videos of gestures previously identified by human observers. This information is used to select the relevant features, train the classifier and tune the filters. In a second phase, "normal operations", the system executes the image processing, feature extraction, feature selection, classification and filtering operations.
Referring to Fig. 3, the training phase includes:
101, human labelling of images,
102, image processing of the stored video,
103, feature extraction,
104, dimensionality reduction,
105, creating and applying feature selection rules,
106, constructing a feature vector,
107, training a classifier, and
108, tuning the filter rules
In the training phase the full set of features are generated using histogram of Orientation Gradients, or histogram of flows (or "spatio-temporal motion"). This generates a large number of features not all of which are contributing positively to the classification. The dimensionality of the feature set is reduced to identify and use only
that sub-set of the features which contribute most to the classification of the hand motions. Mathematical techniques such as Principal Components Analysis or Linear Discriminant Analysis are used to identify this subset of features. This process is called feature selection. The subset of features to be used, typically about 10% of the original set, is passed to the classifier. The classifier learns to associate the features with the motion patterns and thus identifies the hand motions used in hand washing. The final stage of training is to tune the filters that detect and correct errors of classification by examining the sequence of hand motions along with the underlying prior probabilities of the hand poses. Mathematical structures such as Kalman filters, Bayesian networks or Markov decision processes are used to implement these filters.
In the normal operations phase the image processing filters are applied, regions of interest (ROI) are identified and the feature extraction algorithms are run. A subset of the features are selected according to the prior training and concatenated to form a feature vector. This feature vector is passed to the classifier that determines which of the hand motion is currently in use. This result is filtered to remove errors resulting in the final hand motion classification.
Build Feature Vectors, 80
An array of the measurements suitable for classification are made and only those measurements which contribute most to the classification are used. The measurements from the feature extraction algorithms are combined to create an array called a feature vector, and this could contain over 3000 different measurements.
Two approaches to feature vector building are processing histograms of edge orientation, and processing histograms of spatio-temporal motion.
Where histograms of flows are used to create a feature vector using the spatio- temporal information a region of interest around the hands and arms is created and dense motion vectors are extracted. The image is divided into cells and the histograms of the motion vector directions are calculated. The histograms from several subsequent frames may also be added to calculate the motion direction histogram over
a sequence of frames. The histograms are normalised either globally or in overlapping local neighbourhoods. The resulting histograms are combined to form a feature vector that is passed to the multi-class classifier. This is described in more detail above.
In order to reduce the dimensionality of the feature vectors the processor analyses the variance in the feature vector using Principal Components Analysis (PCA). This helps the processor 5 to identify which feature measurements are contributing most to a correct classification. The processor keeps only those features which contribute most to the correct classification; this could typically be only 10% of our original features.
Also, in another embodiment a data reduction technique other than PCA is used, such as linear discriminate analysis (LDA), particularly Fischer LDA. Using smaller and better clustered data sets can allow the use of a simple K-nearest neighbour classifier that can be implemented easily in hardware.
Classification, 81
The feature vector is passed to a multi-class classifier to produce an estimate of the current hand pose. In situations where there is little noise (good lighting, non-specular sinks) a simple classifier such a KNN (K nearest neighbour, or "supervised K means") can be used. If a more complex and general classification problem exists an ensemble of Support Vector Machine or a Cascade of Weak Classifiers can be used.
A multi-class classification system is trained using exemplar feature vectors to produce reliable classifications, a computationally efficient structure and one which generalises well to the wide variety of hand shapes, sizes and bi-manual configurations. One such approach is to use an ensemble of support vector machines. Several strategies may be used to perform this type of ensemble classification. One such strategy is to train an individual support vector machine to classify one pose against a different pose. Multiple support vector machines are trained to cover all possible combinations. The results of the different support vector machines can be combined using a voting scheme with the pose which has the highest number of votes being selected as the correct pose.
In more detail, the Support Vector Machine classifier is a binary classifier algorithm that looks for an optimal hyperplane as a decision function in a high dimensional space. It is a kind of example based machine learning method for both classification and regression problems. This technique has several features that make it particularly attractive. Traditional training techniques for classifiers, such as multi-layer perceptrons (MLP) use empirical risk minimisation and only guarantee minimum error over the training set. In contrast, SVM based on the structural risk minimisation principle minimise a bound on the generalization error and therefore perform better on novel data.
Thus, consider one has a training data set {xk, y^} e χκ{-l, I}, where Xk are the training examples (feature vectors) and y^ the class label. At first, the method consists in mapping xk in a high dimensional space owing to a function φ . Then, it looks for a decision function of the form: j{x) = w. φ (x)+b andyfø) is optimal in the sense that maximizes the distance between the nearest point φ (xι) and the hyperplane. The class label of x is then obtained by considering the sign of/(x).
This optimization problem can be tuned, in the case of soft-margin SVM classifier (misclassified examples are linearly penalized), in the following way:
™|M2 +CΣ 9)
J > (1 t=I
under the constraint Vk, ykflxk) >_ 1 - ξk- The solution of this problem is obtained using the Lagrangian theory and it is possible to show that the vector w is of the form:
m
W=∑«;ΛΦ*)
/t=l (2°)
where ak * is the solution of the following quadratic optimization problem:
m 1 in msxW(μ) = ∑ak --^a^y^K^x,) (21)
A=I 2 - w
subject to^™=1 jfccε,, = 0 and Wς 0<ak<C, where Z (xto *,) = (φ (xk), φ (xι))- K(xh xi) is called the kernel function. Even though new kernels are being proposed by researchers, beginners may find in SVM books the following four basic kernels:
linear : K(x» xj) = xfxj . polynomial : K{xt, Xj) = (yxfxj + ff, γ > 0. radial basis function(RBF) : K{xu xj) = exp
- x Jl 2 ), γ > 0. sigmoid : K(xu xj) = tanh( pζ Xj + r)
where γ , r and d are kernel parameters.
The LIBSVM library is used for training and test. The RBF kernel non-linear maps samples into a higher dimensional space, so it, unlike the linear kernel can handle the case when the relation between class labels and attributes is nonlinear. The second reason is the number of hyper-parameters which influences the complexity of the model since polynomial kernel has more hyper-parameters than the RBF kernel. In addition RBF kernel has less numerical difficulties.
In order to perform the multi-class classification the "one against one" method is used, in which k(h — l)/2 (with k = 7 we have 21 classifiers) different binary classifiers are constructed and each one trains data from two different classes. In practice "one against one" method is one of the more suitable strategies for multi-classification problems [14]. After training data from the z'th and the y th classes, the following voting strategy is used: if sign((wiJ)T φ (x) + bij) says x is in the zth class, then the vote for the zth is added by one. Otherwise they th is increased by one. Then x is predicted to be in the class with the largest vote. This approach is usually referred as the "max wins" strategy. In case that two classes have identical votes, the decision values
(distance to the decision surface) of each one of the classifications is taken into account as in.
Pose Filtering, 82
The bimanual hand poses are represented as a table of transitions from prior poses with probabilities assigned. For example, if the current pose is B and the previous was E, then the likelihood of this transition is 80%. AU prior and current poses are assigned poses based on training and analysis. Low probability poses are deleted as they are likely to be the result of misclassification.
The use of a post classification filter allows for a small and efficient classifier to be used. Occasional classification errors are caught and rectified based on a probabilistic model of hand position reachability. For example, it is unlikely that hands may move immediately from pose A to pose F without some transition pose. It is therefore more likely that pose F is in fact a misclassification and the filter assigns the most likely transition pose. Filters operate on multiple time intervals; frame-to-frame transitions and weighted averages over several frame intervals to produce a reliable sequence of hand pose classifications to the hand motion analysis system.
Time and Motion analysis, 83
Poses which are sustained for a threshold number of frames (e.g. 3) are passed to an accumulator which counts the number of frames that each pose has been maintained for, which effectively counts the amount of time spent in each pose.
Each pose has a configurable threshold minimum time. When all thresholds are exceeded the hands are deemed to have been washed to a sufficiently high standard.
The hand pose information is used to complete a hand hygiene assessment. Each of the hand poses must be recorded for a configurable time period. Bi-manual hand washing poses can be executed in any order. This information is integrated over time and provided to the user via a graphical display such as a speaker or an audio display device. Site-specific options can be implemented regarding overall hand washing metrics such as total time and critical versus non critical hand poses.
The system 1 displays on the simple low-power display 6 permanent graphics of the individual hand poses. The display is a low power LCD display. This indicates to the user that each pose has been achieved. The graphics can be easily customised for each installation.
The system must deal with the challenges posed by changing lighting. A typical scenario is where a hand washing facility is positioned where there is a mix of natural and artificial lighting. Each of these light sources cause colour casts in the image and depending on the time of day the proportions of artificial and natural light can vary significantly. Because the detection of the hands is a significant issue and uses colour as a significant feature algorithms have been developed to calibrate for colour changes. These are contained within an image normalisation process.
Many hand wash stations are high reflective, particularly stainless steel wash hand basins. Basing the segmentation of the hands purely on colour information can lead to significant false positives as the colour of the hands are reflected in the specular surface. By segmenting based on a combination of colour, texture and motion the specular reflections in the typical hand wash station it is possible to achieve reliable segmentation.
When colour and texture are used to segment hands and arms from the background, personal jewellery such as watches and rings often result in small disconnected regions being created. Typically, small regions are also created due to background segmentation errors. Small regions would normally be deleted based on a size filter. However shape analysis of the region boundary is also used to assess if the small region is due to a background segmentation error or a personal jewellery item. The signature shape that is used is if the two shapes on either side of the split share a narrow parallel boundary and in addition the colour and texture on either side of the divide are identical. If this is the case the regions are merged.
Results
Supervised training and test
A supervised labeling process is used for generating training and test data sets. Firstly a database with ROI images and their labels is generated. Thus the reuse of the database is assured. Secondly an off-line application computes the feature vectors for both training and test procedures. Once we have one classifier correctly trained we can use it for on-line pose recognition. The number of training and test samples is depicted in Table 1. We have one class for each one of the different poses of the recommended hand washing procedure.
Table 1. Class distribution of the training and test data sets
We can see the results after applying multi-classification with HOG features and SVM classifier in Table 2. These are single frame results which means that they are not filtered in a post-processing stage. Although detection rate is low for the "other poses" case, which is reasonable taking into account that it is the class with higher variability, all the recommended poses are classified with detection rates greater than 85%. Three of them are classified with accuracy greater than 91%, and the best classified class has a detection rate of 96.09%. A multi-frame validation process would improve these results by far.
Table 2. Detection Rate for the different classes
Fig. 9 depicts the single frame classification results in a sequence. The detected class is shown with a picture at the top left of the image. The system correctly detects when the hands are separated or joined. Even with a single frame approach the classifier
correctly detects the transitions between poses, which are classified as other poses class. Second, fifth and sixth poses are correctly classified in both cases: when the left hand is rubbing the right one and vice versa.
Referring to Fig. 10 another system, 160, is illustrated. The system 160 comprises a housing 161 with a camera window 162 facing towards a sink unit. A processor is contained within the housing, and the system is battery powered. A background plate 163 extends outwardly.
By implementing a system based on a cascade of efficient but error-prone segmentation and classification methods an overall highly reliable performance is achieved. This allows the system to be implemented on low cost and low power computer hardware. The resulting system can operate on a battery and could be integrated into a replaceable soap dispensing unit.
The system can provide the hand washing data to an on-system storage device or can be broadcast via a wireless link to central data storage. In either case the records can for part of a HACCP (Hazard Analysis and Critical Control Points) records for use in food preparation or other hygiene standards.
The invention is not limited to the embodiments described but may be varied in construction and detail.
Claims
1. A hand washing monitoring system comprising a camera (2), a processor (4), the processor being adapted to receive from the camera images of hand washing activity, characterized in that, the processor is adapted to:
analyse mutual motion of hands to determine if the hands mutually move in desired poses, and if so, the durations of the patterns; and
generate a hand washing quality indication according to the analysis.
2. A monitoring system as claimed in claims 1, wherein the processor analyses the images within a region of interest encompassing joined hands.
3. A monitoring system as claimed in claims 1 or 2, wherein the processor generates the indication independently of the order of motion poses.
4. A monitoring system as claimed in any preceding claim, wherein the processor extracts information features from the images and generates feature vectors based on the features, including bimanual hand and arm shape vectors, and executes a classifier with the vectors to determine the poses.
5. A monitoring system as claimed in claim 4, wherein the processor uses edge segmentation to form at least some of the feature vectors.
6. A monitoring system as claimed in claim 4 or 5, wherein the processor uses pixel spatio-temporal measurements to form at least some of the feature vectors.
7. A monitoring system as claimed in any of claims 4 to 6, wherein the processor decomposes an image into cells and uses each cell to calculate a histogram, and combines the histograms to form a feature vector.
8. A monitoring system as claimed in any of claims 4 to 7, wherein the processor generates the feature vectors using rules derived from a training phase performed with reference images.
9. A monitoring system as claimed in any of claims 4 to 8, wherein the processor reduces size of the feature vectors.
10. A monitoring system as claimed in claim 9, wherein the processor reduces feature vector size using Principal Component Analysis.
11. A monitoring system as claimed in claim 9, wherein the processor uses linear discriminate analysis to reduce feature vector size.
12. A monitoring system as claimed in any of claims 9 to 11, wherein a self- organising map is generated to represent clustered datasets of a reduced feature vector.
13. A monitoring system as claimed in any of claims 4 to 12, wherein the processor classifies the feature vectors by executing a multi-class classifier which is trained with exemplar feature vectors to generate an estimate of a hand pose.
14. A monitoring system as claimed in any of claims 4 to 13, wherein the processor performs the classification using a support vector machine.
15. A monitoring system as claimed in claim 14, wherein the processor executes an ensemble of support vector machines.
16. A monitoring system as claimed in claim 15, wherein the processor determines a pose according to votes from different support vector machines.
17. A monitoring system as claimed in any of claims 5 to 15, wherein the processor filters bimanual hand pose classifications.
18. A monitoring system as claimed in claim 17, wherein the processor performs said filtering by using probabilistic bimanual hand motion models to remove anomalous results.
19. A monitoring system as claimed in any preceding claim, wherein the processor determines a time duration for a pose by registering a count of the number of image frames for a pose.
20. A monitoring system as claimed in claim 19, wherein the processor assigns a minimum threshold for each pose.
21. A monitoring system as claimed in any preceding claim, wherein the processor is calibrated for light level and/or colour changes in an image normalisation process.
22. A monitoring system as claimed in any preceding claim, wherein the processor performs segmentation according to colour, texture and/or motion to avoid noise arising from reflections.
23. A monitoring system as claimed in claim 22, wherein the processor eliminates pixels representing watches or jewellery according to region size and shape analysis.
24. A monitoring system as claimed in any preceding claim, wherein the system is battery powered.
25. A monitoring system as claimed in any of claims 4 to 24, wherein the classifier comprises an ensemble of weak classifiers.
26. A monitoring system as claimed in any of claims 4 to 25 wherein the processor, before feature extraction, detects skin by normalizing pixel colour values to skin and non-skin class bins, and processes bin counts to determine skin probability values.
27. A monitoring system as claimed in any preceding claim, wherein the processor executes a lighting compensation filter operating on the basis that the spatial average of surface reflectance in a scene is achromatic.
28. A monitoring system as claimed in claims 26 or 27, wherein the processor performs optical flow calculations based on pixel distance moved and direction of movement, and applies motion data to a filter to eliminate skin false- positive pixels.
29. A monitoring system as claimed in claim 28, wherein the processor determines increase and decrease factors representing pixel motion increase and decrease, and applies a greater weighting to motion than absence of motion.
30. A monitoring system as claimed in any of claims 26 to 29, wherein the processor performs hand and arm detection by monitoring geometric characteristics of pixel blobs, including major and minor axes.
31. A monitoring system as claimed in any of claims 4 to 30, wherein the processor performs feature extraction by generating histograms of gradient orientation in a local part of an image by accumulating votes into each of a plurality of bins, each for an orientation.
32. A monitoring system as claimed in claim 31 , wherein the processor generates a gradient orientation histogram for each of a plurality of pixel cells comprising pre-defined numbers of pixels.
33. A monitoring system as claimed in claims 32, wherein the processor concentrates all histograms into a single vector.
34. A monitoring system as claimed in claim 33 wherein the processor normalizes cell histograms.
35. A monitoring system as claimed in claim 34, wherein the processor performs a training phase in which vectors are generated from training images.
36. A soap dispensing unit comprising a monitoring system as claimed in any preceding claim.
37. A computer readable medium comprising software code for performing operations of a processor of a monitoring system as claimed in any preceding claim.
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE602007002068T DE602007002068D1 (en) | 2006-05-04 | 2007-05-04 | HAND WASHING MONITORING SYSTEM |
PL07736107T PL2015665T3 (en) | 2006-05-04 | 2007-05-04 | A hand washing monitoring system |
US12/226,983 US8090155B2 (en) | 2006-05-04 | 2007-05-04 | Hand washing monitoring system |
EP07736107A EP2015665B1 (en) | 2006-05-04 | 2007-05-04 | A hand washing monitoring system |
DK07736107T DK2015665T3 (en) | 2006-05-04 | 2007-05-04 | Monitoring system |
AT07736107T ATE439787T1 (en) | 2006-05-04 | 2007-05-04 | HAND WASH MONITORING SYSTEM |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
IE20060350 | 2006-05-04 | ||
IE2006/0350 | 2006-05-04 | ||
IE20060905 | 2006-12-12 | ||
IE2006/0905 | 2006-12-12 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2007129289A1 true WO2007129289A1 (en) | 2007-11-15 |
Family
ID=38180608
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IE2007/000051 WO2007129289A1 (en) | 2006-05-04 | 2007-05-04 | A hand washing monitoring system |
Country Status (9)
Country | Link |
---|---|
US (1) | US8090155B2 (en) |
EP (1) | EP2015665B1 (en) |
AT (1) | ATE439787T1 (en) |
DE (1) | DE602007002068D1 (en) |
DK (1) | DK2015665T3 (en) |
ES (1) | ES2330489T3 (en) |
IE (1) | IE20070336A1 (en) |
PL (1) | PL2015665T3 (en) |
WO (1) | WO2007129289A1 (en) |
Cited By (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
ES2359082A1 (en) * | 2009-04-01 | 2011-05-18 | Fundacion Fibao, Fundacion Para La Investigacion Biosanitaria De Andalucia Oriental-Alejandro Otero | Device for handwashing and handwash control procedure. (Machine-translation by Google Translate, not legally binding) |
WO2011123743A1 (en) * | 2010-04-01 | 2011-10-06 | Sealed Air Corporation (Us) | Automated monitoring and control of contamination in a production area |
WO2011123757A1 (en) * | 2010-04-01 | 2011-10-06 | Sealed Air Corporation (Us) | Automated monitoring and control of contamination activity in a production area |
US8040245B2 (en) | 2007-08-23 | 2011-10-18 | Gt Angel, Llc | Hand washing monitor for detecting the entry and identification of a person |
WO2012037192A1 (en) * | 2010-09-14 | 2012-03-22 | General Electric Company | System and method for protocol adherence |
US8395515B2 (en) | 2009-06-12 | 2013-03-12 | Ecolab Usa Inc. | Hand hygiene compliance monitoring |
US8639527B2 (en) | 2008-04-30 | 2014-01-28 | Ecolab Usa Inc. | Validated healthcare cleaning and sanitizing practices |
DE102013104556B3 (en) * | 2013-05-03 | 2014-03-27 | Itec-Ingenieurbüro für Hygiene Und Lebensmitteltechnik GmbH | Automatic hand cleaning method used in medical and food industry, involves supplying fixed amount of cleaning or disinfecting agent to two hands, by recognizing two spread hands positioned in cleaning space |
US8990098B2 (en) | 2008-04-30 | 2015-03-24 | Ecolab Inc. | Validated healthcare cleaning and sanitizing practices |
US9011607B2 (en) | 2010-10-07 | 2015-04-21 | Sealed Air Corporation (Us) | Automated monitoring and control of cleaning in a production area |
US9143843B2 (en) | 2010-12-09 | 2015-09-22 | Sealed Air Corporation | Automated monitoring and control of safety in a production area |
US9189949B2 (en) | 2010-12-09 | 2015-11-17 | Sealed Air Corporation (Us) | Automated monitoring and control of contamination in a production area |
US9824569B2 (en) | 2011-01-28 | 2017-11-21 | Ecolab Usa Inc. | Wireless communication for dispenser beacons |
US10529219B2 (en) | 2017-11-10 | 2020-01-07 | Ecolab Usa Inc. | Hand hygiene compliance monitoring |
WO2020117150A3 (en) * | 2018-12-03 | 2021-03-04 | Erg Kontrol Si̇stemleri̇ San. Ve Ti̇c. A.Ş. | A hand wash monitoring system |
JP2021174415A (en) * | 2020-04-30 | 2021-11-01 | 株式会社コンテック | Hygiene management system |
USRE48951E1 (en) | 2015-08-05 | 2022-03-01 | Ecolab Usa Inc. | Hand hygiene compliance monitoring |
US11272815B2 (en) | 2017-03-07 | 2022-03-15 | Ecolab Usa Inc. | Monitoring modules for hand hygiene dispensers |
WO2022056356A1 (en) * | 2020-09-11 | 2022-03-17 | AI Data Innovation Corporation | Disinfection monitoring system and method |
US11284333B2 (en) | 2018-12-20 | 2022-03-22 | Ecolab Usa Inc. | Adaptive route, bi-directional network communication |
Families Citing this family (44)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9165199B2 (en) * | 2007-12-21 | 2015-10-20 | Honda Motor Co., Ltd. | Controlled human pose estimation from depth image streams |
US9098766B2 (en) * | 2007-12-21 | 2015-08-04 | Honda Motor Co., Ltd. | Controlled human pose estimation from depth image streams |
KR101592999B1 (en) * | 2009-02-09 | 2016-02-11 | 삼성전자주식회사 | Apparatus and method for recognzing hand shafe in portable terminal |
US9904845B2 (en) * | 2009-02-25 | 2018-02-27 | Honda Motor Co., Ltd. | Body feature detection and human pose estimation using inner distance shape contexts |
US8600166B2 (en) | 2009-11-06 | 2013-12-03 | Sony Corporation | Real time hand tracking, pose classification and interface control |
WO2011131800A1 (en) * | 2010-04-22 | 2011-10-27 | Fundación Fibao, Fundación Para La Investigación Biosanitaria De Andalucía Oriental-Alejandro Otero | Hand-washing device and method for monitoring hand washing |
GB2480654B (en) * | 2010-05-27 | 2012-08-15 | Infrared Integrated Syst Ltd | Monitoring handwashing |
US8792722B2 (en) * | 2010-08-02 | 2014-07-29 | Sony Corporation | Hand gesture detection |
US8750573B2 (en) * | 2010-08-02 | 2014-06-10 | Sony Corporation | Hand gesture detection |
US8730157B2 (en) * | 2010-11-15 | 2014-05-20 | Hewlett-Packard Development Company, L.P. | Hand pose recognition |
TW201223491A (en) * | 2010-12-03 | 2012-06-16 | Nat Taipei University Oftechnology | Intelligent hand-washing monitoring system |
WO2012081012A1 (en) * | 2010-12-16 | 2012-06-21 | Pointgrab Ltd. | Computer vision based hand identification |
CN103299342B (en) * | 2010-12-31 | 2017-02-15 | 诺基亚技术有限公司 | Method and apparatus for providing a mechanism for gesture recognition |
US20150302769A1 (en) * | 2011-12-05 | 2015-10-22 | Raymond C. Johnson | Virtual Hand-Washing Coach |
WO2014074585A1 (en) | 2012-11-06 | 2014-05-15 | Johnson Raymond C | Virtual hand-washing coach |
JP5780142B2 (en) * | 2011-12-07 | 2015-09-16 | 富士通株式会社 | Image processing apparatus and image processing method |
GB2497992B (en) * | 2011-12-30 | 2015-06-03 | Hyintel Ltd | A drip tray |
US20130243077A1 (en) * | 2012-03-13 | 2013-09-19 | Canon Kabushiki Kaisha | Method and apparatus for processing moving image information, and method and apparatus for identifying moving image pattern |
TWI476702B (en) * | 2012-03-16 | 2015-03-11 | Pixart Imaging Inc | User identification system and method for identifying user |
CH706677A1 (en) * | 2012-06-21 | 2013-12-31 | Hopitaux Universitaires Geneve | Process and handwashing verification device. |
WO2014062755A1 (en) * | 2012-10-16 | 2014-04-24 | Johnson Raymond C | Hand washing monitoring system for public restrooms |
US9076044B2 (en) * | 2012-12-15 | 2015-07-07 | Joseph Ernest Dryer | Apparatus and method for monitoring hand washing |
CN104239844A (en) * | 2013-06-18 | 2014-12-24 | 华硕电脑股份有限公司 | Image recognition system and image recognition method |
GB2521844A (en) | 2014-01-03 | 2015-07-08 | Fluke Corp | A method and system for monitoring hand washing |
US9729833B1 (en) | 2014-01-17 | 2017-08-08 | Cerner Innovation, Inc. | Method and system for determining whether an individual takes appropriate measures to prevent the spread of healthcare-associated infections along with centralized monitoring |
JP6436385B2 (en) * | 2014-12-17 | 2018-12-12 | パナソニックIpマネジメント株式会社 | Imaging apparatus, image processing apparatus, and image processing method |
US9881485B2 (en) | 2015-03-17 | 2018-01-30 | Julio Hajdenberg | Device based hygiene reminder, alarm, and reporting system |
US9724443B2 (en) * | 2015-04-10 | 2017-08-08 | Rememdia LC | System, method, and device for decontamination |
US9418535B1 (en) * | 2015-04-24 | 2016-08-16 | WashSense Inc. | Hand-wash monitoring and compliance system |
US9721452B2 (en) * | 2015-04-24 | 2017-08-01 | WashSense, Inc. | Hand-wash management and compliance system |
US9418536B1 (en) * | 2015-04-24 | 2016-08-16 | WashSense Inc. | Hand-washing compliance system |
WO2016172390A1 (en) * | 2015-04-24 | 2016-10-27 | WashSense Inc. | Hand-wash monitoring and compliance system |
US9542828B1 (en) * | 2015-06-22 | 2017-01-10 | Peter D. Haaland | System, device, and method for measurement of hand hygiene technique |
WO2019160919A1 (en) * | 2018-02-13 | 2019-08-22 | Gojo Industries, Inc. | Modular people counters |
US20200388193A1 (en) * | 2018-08-14 | 2020-12-10 | Food Service Monitoring, Inc. | Method and system for motivating and monitoring hand washing in a food service or related environment |
US11017655B2 (en) * | 2019-10-09 | 2021-05-25 | Visualq | Hand sanitation compliance enforcement systems and methods |
JP7476599B2 (en) * | 2020-03-24 | 2024-05-01 | 大日本印刷株式会社 | Information processing system and information processing method |
EP4128153A4 (en) * | 2020-03-27 | 2024-08-07 | Hs Vk Idea Factory Llc | Apparatus and method for protecting against environmental hazards |
WO2021222245A1 (en) | 2020-04-27 | 2021-11-04 | Municipal Parking Services, Inc. | Health and sanitation monitoring methods and systems for controlled access to a premises |
JP6875028B1 (en) * | 2020-04-30 | 2021-05-19 | 株式会社Acculus | Hand wash evaluation device and hand wash evaluation program |
JP7356954B2 (en) * | 2020-07-17 | 2023-10-05 | 株式会社Nsd先端技術研究所 | Handwashing evaluation system and method |
CN111860448A (en) * | 2020-07-30 | 2020-10-30 | 北京华捷艾米科技有限公司 | Hand washing action recognition method and system |
WO2022023615A2 (en) * | 2020-07-30 | 2022-02-03 | URBANO RODRÍGUEZ, Manuel | System for controlling access by verifying hand hygiene and application of the system to a controlled-access dressing trolley |
EP3964653A1 (en) | 2020-09-08 | 2022-03-09 | Ideal Standard International NV | Hygiene system and method for operating a hygiene system |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1998036258A2 (en) * | 1997-02-18 | 1998-08-20 | Johnson, Raymond, C. | Apparatus and method for monitoring hand washing |
WO2003079278A1 (en) * | 2002-03-12 | 2003-09-25 | Johnson Raymond C | Pattern recognition system and method for monitoring hand washing or application of a disinfectant |
JP2006132277A (en) * | 2004-11-09 | 2006-05-25 | Seiken Ika Kogyo Kk | Automatic washhand unit |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6236317B1 (en) | 1998-04-29 | 2001-05-22 | Food Safety Solution Corp. | Method and apparatus for monitoring actions taken by a user for enhancing hygiene |
US5952927A (en) | 1998-06-02 | 1999-09-14 | Eshman; Richard | Portable child safety alarm system |
US6426701B1 (en) | 2000-09-20 | 2002-07-30 | Ultraclenz Engineering Group | Handwash monitoring system |
US6970574B1 (en) * | 2001-03-13 | 2005-11-29 | Johnson Raymond C | Pattern recognition system and method for monitoring hand washing or application of a disinfectant |
US7542586B2 (en) * | 2001-03-13 | 2009-06-02 | Johnson Raymond C | Touchless identification system for monitoring hand washing or application of a disinfectant |
GB0406704D0 (en) | 2004-03-25 | 2004-04-28 | Bourne Leisure Ltd | Cleanliness monitoring system and related method |
-
2007
- 2007-05-04 DE DE602007002068T patent/DE602007002068D1/en active Active
- 2007-05-04 EP EP07736107A patent/EP2015665B1/en active Active
- 2007-05-04 WO PCT/IE2007/000051 patent/WO2007129289A1/en active Application Filing
- 2007-05-04 DK DK07736107T patent/DK2015665T3/en active
- 2007-05-04 AT AT07736107T patent/ATE439787T1/en not_active IP Right Cessation
- 2007-05-04 US US12/226,983 patent/US8090155B2/en active Active
- 2007-05-04 PL PL07736107T patent/PL2015665T3/en unknown
- 2007-05-04 IE IE20070336A patent/IE20070336A1/en not_active Application Discontinuation
- 2007-05-04 ES ES07736107T patent/ES2330489T3/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1998036258A2 (en) * | 1997-02-18 | 1998-08-20 | Johnson, Raymond, C. | Apparatus and method for monitoring hand washing |
WO2003079278A1 (en) * | 2002-03-12 | 2003-09-25 | Johnson Raymond C | Pattern recognition system and method for monitoring hand washing or application of a disinfectant |
JP2006132277A (en) * | 2004-11-09 | 2006-05-25 | Seiken Ika Kogyo Kk | Automatic washhand unit |
Cited By (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8040245B2 (en) | 2007-08-23 | 2011-10-18 | Gt Angel, Llc | Hand washing monitor for detecting the entry and identification of a person |
US8990098B2 (en) | 2008-04-30 | 2015-03-24 | Ecolab Inc. | Validated healthcare cleaning and sanitizing practices |
US8639527B2 (en) | 2008-04-30 | 2014-01-28 | Ecolab Usa Inc. | Validated healthcare cleaning and sanitizing practices |
ES2359082A1 (en) * | 2009-04-01 | 2011-05-18 | Fundacion Fibao, Fundacion Para La Investigacion Biosanitaria De Andalucia Oriental-Alejandro Otero | Device for handwashing and handwash control procedure. (Machine-translation by Google Translate, not legally binding) |
US8395515B2 (en) | 2009-06-12 | 2013-03-12 | Ecolab Usa Inc. | Hand hygiene compliance monitoring |
US8502680B2 (en) | 2009-06-12 | 2013-08-06 | Ecolab Usa Inc. | Hand hygiene compliance monitoring |
US9406212B2 (en) | 2010-04-01 | 2016-08-02 | Sealed Air Corporation (Us) | Automated monitoring and control of contamination activity in a production area |
WO2011123757A1 (en) * | 2010-04-01 | 2011-10-06 | Sealed Air Corporation (Us) | Automated monitoring and control of contamination activity in a production area |
EP2553627A1 (en) * | 2010-04-01 | 2013-02-06 | Sealed Air Corporation (US) | Automated monitoring and control of contamination activity in a production area |
WO2011123743A1 (en) * | 2010-04-01 | 2011-10-06 | Sealed Air Corporation (Us) | Automated monitoring and control of contamination in a production area |
US9785744B2 (en) | 2010-09-14 | 2017-10-10 | General Electric Company | System and method for protocol adherence |
WO2012037192A1 (en) * | 2010-09-14 | 2012-03-22 | General Electric Company | System and method for protocol adherence |
US10361000B2 (en) | 2010-09-14 | 2019-07-23 | General Electric Company | System and method for protocol adherence |
US9934358B2 (en) | 2010-09-14 | 2018-04-03 | General Electric Company | System and method for protocol adherence |
US9011607B2 (en) | 2010-10-07 | 2015-04-21 | Sealed Air Corporation (Us) | Automated monitoring and control of cleaning in a production area |
US9143843B2 (en) | 2010-12-09 | 2015-09-22 | Sealed Air Corporation | Automated monitoring and control of safety in a production area |
US9189949B2 (en) | 2010-12-09 | 2015-11-17 | Sealed Air Corporation (Us) | Automated monitoring and control of contamination in a production area |
US9824569B2 (en) | 2011-01-28 | 2017-11-21 | Ecolab Usa Inc. | Wireless communication for dispenser beacons |
DE102013104556B3 (en) * | 2013-05-03 | 2014-03-27 | Itec-Ingenieurbüro für Hygiene Und Lebensmitteltechnik GmbH | Automatic hand cleaning method used in medical and food industry, involves supplying fixed amount of cleaning or disinfecting agent to two hands, by recognizing two spread hands positioned in cleaning space |
USRE48951E1 (en) | 2015-08-05 | 2022-03-01 | Ecolab Usa Inc. | Hand hygiene compliance monitoring |
US11903537B2 (en) | 2017-03-07 | 2024-02-20 | Ecolab Usa Inc. | Monitoring modules for hand hygiene dispensers |
US11272815B2 (en) | 2017-03-07 | 2022-03-15 | Ecolab Usa Inc. | Monitoring modules for hand hygiene dispensers |
US10529219B2 (en) | 2017-11-10 | 2020-01-07 | Ecolab Usa Inc. | Hand hygiene compliance monitoring |
WO2020117150A3 (en) * | 2018-12-03 | 2021-03-04 | Erg Kontrol Si̇stemleri̇ San. Ve Ti̇c. A.Ş. | A hand wash monitoring system |
US11284333B2 (en) | 2018-12-20 | 2022-03-22 | Ecolab Usa Inc. | Adaptive route, bi-directional network communication |
US11711745B2 (en) | 2018-12-20 | 2023-07-25 | Ecolab Usa Inc. | Adaptive route, bi-directional network communication |
JP2021174415A (en) * | 2020-04-30 | 2021-11-01 | 株式会社コンテック | Hygiene management system |
JP7492850B2 (en) | 2020-04-30 | 2024-05-30 | 株式会社コンテック | Sanitation Management System |
WO2022056356A1 (en) * | 2020-09-11 | 2022-03-17 | AI Data Innovation Corporation | Disinfection monitoring system and method |
Also Published As
Publication number | Publication date |
---|---|
ATE439787T1 (en) | 2009-09-15 |
ES2330489T3 (en) | 2009-12-10 |
IE20070336A1 (en) | 2008-06-11 |
EP2015665A1 (en) | 2009-01-21 |
DK2015665T3 (en) | 2009-11-16 |
DE602007002068D1 (en) | 2009-10-01 |
EP2015665B1 (en) | 2009-08-19 |
US20090087028A1 (en) | 2009-04-02 |
US8090155B2 (en) | 2012-01-03 |
PL2015665T3 (en) | 2010-01-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2015665B1 (en) | A hand washing monitoring system | |
Salti et al. | Adaptive appearance modeling for video tracking: Survey and evaluation | |
Yang et al. | Detecting faces in images: A survey | |
Gowsikhaa et al. | Suspicious Human Activity Detection from Surveillance Videos. | |
Llorca et al. | A vision-based system for automatic hand washing quality assessment | |
Iosifidis et al. | Multi-view action recognition based on action volumes, fuzzy distances and cluster discriminant analysis | |
US9001199B2 (en) | System and method for human detection and counting using background modeling, HOG and Haar features | |
Dias et al. | Gaze estimation for assisted living environments | |
Li et al. | Measuring the intensity of spontaneous facial action units with dynamic Bayesian network | |
Shoyaib et al. | A skin detection approach based on the Dempster–Shafer theory of evidence | |
Iosifidis et al. | Dynamic action recognition based on dynemes and extreme learning machine | |
US20110304541A1 (en) | Method and system for detecting gestures | |
Zafar et al. | Pain intensity evaluation through facial action units | |
WO2012004281A1 (en) | Optimization of human activity determination from video | |
Kobayashi et al. | Three-way auto-correlation approach to motion recognition | |
Fuentes-Jimenez et al. | DPDnet: A robust people detector using deep learning with an overhead depth camera | |
Elder et al. | Pre-attentive and attentive detection of humans in wide-field scenes | |
Devanne et al. | Recognition of activities of daily living via hierarchical long-short term memory networks | |
Neili et al. | Human posture recognition approach based on ConvNets and SVM classifier | |
Sun et al. | A hybrid system for on-line blink detection | |
Haghpanah et al. | Real-time hand rubbing quality estimation using deep learning enhanced by separation index and feature-based confidence metric | |
Venkat et al. | Recognizing occluded faces by exploiting psychophysically inspired similarity maps | |
Ryan | Crowd monitoring using computer vision | |
Llorca et al. | A multi-class SVM classifier ensemble for automatic hand washing quality assessment | |
Bostanci et al. | A neuro fuzzy embedded agent approach towards the development of an intelligent refrigerator |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 07736107 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 12226983 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2007736107 Country of ref document: EP |