JP6875028B1 - Hand wash evaluation device and hand wash evaluation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect which part of both hands is not washed in hand washing by a simple and unprepared method, and present it to the user.
An imaging unit captures an image of both hands of a person, and a hand position shape estimation unit estimates a three-dimensional position and a surface shape of both hands of the person. The contact area estimation unit estimates contact with other small areas in units of small areas on the surface of both hands estimated by the hand position shape estimation unit. The hand-washing start determination unit detects that the person has started hand-washing, and the score calculation unit calculates the contact score after the hand-washing is started for each small area of both hands. The information output unit notifies the user of the score distribution of both hands.
[Selection diagram] FIG. 15

Description

The present invention relates to a technique for determining the quality of a person's hand-washing operation from image information and presenting the technique to the user.

Many of the harmful substances such as viruses and pathogens are said to infect humans through fingers, and it is important to remove them by hand washing for prevention. However, the procedure and degree of hand washing varies depending on the person and the case, and sometimes both hands are washed cleanly, and sometimes there are unwashed parts. If someone who knows a good hand-washing procedure can see and teach hand-washing, the hand-washing method can be improved.

Alternatively, there is a technique disclosed in Patent Documents 1 to 4 below, which measures hand washing with a camera or a sensor and automatically evaluates it.

Patent Document 1 is a hygiene evaluation device that calculates a hand-washing score by attaching a substance to a hand in advance, taking an image of the hand after washing the hand, and determining whether or not the substance is attached to the hand.

Patent Document 2 attaches a motion detection sensor such as an acceleration sensor to a person in advance, recognizes the action of the person based on the detection result of the motion detection means, and evaluates the number of hand hygiene actions and the amount of movement.

Patent Document 3 is a hand-washing assist system that captures a finger image and displays a model hand-washing image showing a model operation of a hand-washing procedure performed by a person.

Patent Document 4 extracts the shape of the hand region from the imaging, identifies the type of washing performed, adds the number of seconds when the identified washing methods are the washing methods in a predetermined order, and rubs the state. This is a hand-washing monitoring method that determines the quality.

Japanese Patent Application No. 2017-202611 Japanese Patent No. 5097965 Japanese Patent Application No. 2018-117779 Patent No. 6305448

In Patent Document 1, it is necessary for a person to prepare in advance to attach a substance to his or her hand. It is good if the person does it for the purpose of hand-washing evaluation, but it is not suitable for use in daily life.

Patent Document 2 also requires that a person be equipped with a motion detecting means in advance, and is not suitable for use by an unspecified number of people in daily life. In addition, the motion detecting means cannot recognize the detailed motion of the fingers.

Patent Document 3 only recognizes the hand-washing procedure performed by a person and displays a model motion, and does not evaluate the detailed state of the fingers.

In Patent Document 4, the quality of the rubbing state of the washing method is determined by identifying the type of washing method performed by extracting the two-dimensional shape of the hand area. It does not detect the state and evaluate the state for each area. In addition, the recognition accuracy largely depends on the state of the number of fingers of the hand, and it is difficult to obtain practical recognition accuracy under various actual environments and the state of the fingers.

In order for an unspecified number of people to use it easily on a daily basis, it is a challenge that a person can use it without preparations such as attaching a sensor, and that the hand-washing state of each of the three-dimensional areas of both hands can be accurately determined. Met.

According to one embodiment of the present invention, there is provided a hand-washing evaluation device that calculates a score for each three-dimensional small area of both hands and notifies the user. The imaging unit captures images of both hands of the person, and the hand position shape estimation unit estimates the three-dimensional position and surface shape of both hands of the person. The contact area estimation unit estimates contact with other small areas in units of small areas on the surface of both hands estimated by the hand position shape estimation unit. The hand-washing start determination unit detects that the person has started hand-washing, and the score calculation unit calculates the contact score after the hand-washing is started for each small area of both hands. The information output unit notifies the user of the score distribution of both hands.

According to the present invention, the user can know the evaluation score of hand washing for each small area of both hands simply by washing the hands without prior preparation. By visualizing areas where hand washing is inadequate, users can intuitively understand areas where improvement in washing is required. In addition, since the effect of hand washing is visibly reflected, there is an effect of motivating the user to improve hand washing by himself / herself.

It is an installation situation in the 1st Embodiment of this invention. It is a configuration in the first embodiment of the present invention. It is a flowchart for demonstrating the processing procedure in 1st Embodiment. This is an example of an image captured by the imaging unit. It is a flowchart for demonstrating the processing procedure of the hand position shape estimation part. It is a figure which shows that both hands overlap each other and hide. It is a figure of a set of three-dimensional vertices representing the surface shape of a hand. It is a figure of the joint position of a hand. It is a schematic diagram of the vertex estimation process. It is a schematic diagram of the fitting process of a hand model data. It is a figure which shows the distance between vertices. It is a figure when the vertex group is approximated to the sphere information. It is a score table. It is a figure at the time of the operation that both hands rub against each other. It is a figure that an information output part visualizes the score distribution of both hands as a display device and notifies a user. It is a figure of the operation of washing the fingertip of the right hand.

(First Embodiment)
FIG. 1 shows an installation situation in the first embodiment of the present invention. A tap 102 is installed at the hand-washing place 101, and the user 103 uses the water from the faucet to wash his hands. The user may also use soap or a hand-washing cleaning solution. The imaging unit 104, which is a part of the present invention, is installed in a place where the user's hand washing can be imaged. Further, the information output unit 105, which is a part of the present invention, is installed as a display device and presents information to the user 103.

FIG. 2 is a configuration according to the first embodiment of the present invention. The image pickup unit 201 is installed at 104, and the information output unit 206 is installed at 105. The hand position shape estimation unit 202, the contact area estimation unit 203, the hand wash start determination unit 204, and the score calculation unit 205 are numerical calculation processes, and the CPU The calculation is performed by an arithmetic unit such as (Central-Processing-Unit) or GPU (Graphics-Processing-Unit).

The hand position shape estimation unit 202 owns the weight coefficient 207 and the hand model data 208. The weighting coefficient data 207 and the hand model data 208 are stored in a ROM (Read-Only-Memory).

The score calculation unit 205 owns the score data 209. The score data 209 is stored in a RAM (Random-Access-Memory).

Arithmetic logic units such as CPUs, ROMs, RAMs, and other hardware required for arithmetic operations may be installed in the imaging unit 104, in the information output unit 105, or in other places.

FIG. 3 is a flowchart for explaining a processing procedure according to the first embodiment of the present invention.

The imaging unit 201 captures a moving image of the user 103 washing their hands at the hand-washing place 101 (S301). The image pickup unit 201 is a camera that captures visible light, and acquires RGB data as an image. Alternatively, the imaging unit 201 is any one of a stereo camera, a Time-Of-Flight system camera, and a Structured-Light system camera, and acquires distance data as an image. A visible light camera and a distance camera may be used together.

FIG. 4 is an example of an image captured by the imaging unit 201. The user's left hand 402 and right hand 403 are imaged as image 401. When the field of view of the imaging unit 201 is wide, the place where the hand-washing is performed at the hand-washing place 101 is limited, so that the partial area of the image may be used as the input image in the subsequent steps.

The hand position shape estimation unit 202 uses an image as an input to estimate the three-dimensional position and surface shape of both hands (S302). FIG. 5 is a flowchart for explaining the processing procedure in the hand position shape estimation unit 202.

In order to estimate the contact condition of both hands, the position of the entire surface shape of both hands is required. However, the image pickup unit 201 is photographed from one direction, and the back surface thereof cannot be imaged. Further, as shown in FIG. 6, when washing hands, both hands often hide each other, and it is difficult to capture the entire surface shape.

Therefore, hand model data that models the surface shape of the hand is required. The surface shape of the hand can be represented by a set of three-dimensional vertices 701 as shown in FIG. Edge 702 in FIG. 7 connects each vertex with a line segment. Each vertex is given a unique label depending on its position in the hand. The surface shape of one hand can be expressed by a set of N of three-dimensional vertices 701, but it can also be expressed by a linear combination of M less than N by principal component analysis. When the state of the hand in each input image is set as the parameter θ, N vertices can be restored by using the hand model data which is the principal component information calculated in advance.

The hand state parameter θ may include a three-axis rotation vector or rotation matrix of the hand, and a three-axis translation vector and scale value.

The hand state parameter θ may include a parameter for restoring the hand joint position 801 shown in FIG.

The hand position shape estimation unit 202 receives an image as an input (S501) and performs vertex estimation processing (S502). Figure 9 is an overview of the vertex estimation process. In the vertex estimation process, the image 901 captured by the left hand 905 and the right hand 906 is processed by the convolutional neural network 902, and the image 903 is output. The image 903 is the vertex label information for each pixel, and the vertex label of the left hand 907 and the vertex label of the right hand 908 are output as an image. The convolutional neural network 902 includes pre-supervised machine learning weighted coefficient data 904.

The hand position shape estimation unit 202 calculates the three-dimensional data of the measured vertices based on the image output by the vertex estimation process (S503). When the imaging unit 201 is a distance camera that captures a distance image, the three-dimensional position of each vertex label is calculated from the horizontal and vertical positions in the image 903 and the information of the distance image.

When the imaging unit 201 is a visible light camera that captures an RGB image, the convolutional neural network 902 also estimates the distance information and outputs it in addition to the image 903 to calculate the three-dimensional position of each vertex label. Since it is difficult to estimate absolute distance information in an RGB image, it is a relative three-dimensional position, but there is no problem in the present invention because the purpose is to determine the contact between both hands.

If the hand position shape estimation unit 202 cannot obtain valid three-dimensional vertex data in S502 and S503, it determines that the hand has not been imaged and outputs information that there is no hand (S504).

When valid 3D vertex data is obtained, the hand position shape estimation unit 202 calculates the hand state parameter θ that best fits the 3D vertex data of each of the left and right hands with respect to the hand model data (S505). ).

FIG. 10 shows the fitting process of the hand model data. The fitting process 1003 is performed on the measured three-dimensional vertex data 1001 of the left hand, and the state parameter θL of the left hand and the restored three-dimensional shape 1004 of the left hand are calculated. Similarly, for the three-dimensional vertex data 1002 of the right hand, the state parameter θR of the right hand and the three-dimensional shape 1005 of the right hand restored by it are calculated. The fitting process 1003 includes the hand model data 1006.

The fitting process 1003 is an optimization process for finding the parameter θ that minimizes the defined energy function. For the energy function, the squared error between the measured 3D vertex data and the vertex data restored by the parameters is used.

A regularization term may be added to the energy function to prevent overfitting.

In the above, the fitting process of the hand model data is performed by each of the left and right hands, but the fitting process may be performed at the same time in consideration of the positional relationship between both hands.

With S302, the restored 3D vertex data which is the presence / absence of a hand and the 3D position and surface shape when there is a hand can be obtained. If there is no hand, the steps S304 to S307 are not performed (S303).

The hand-washing start determination unit 204 determines whether or not the user has newly started hand-washing (S304). The presence or absence of hands is determined by S302, but if it is determined that there are hands after the state of no hands for a certain period of time or more, it is determined that new hand washing has been started. Alternatively, if it is determined that a new person has entered by measuring the entry and exit of the person to the hand-washing place 101, it may be determined that the hand-washing has been newly started.

As shown in the table of FIG. 13, the score data holds the evaluation score for each small area. The small area may be any of the vertices of both hands shown in FIG. 7, a small set of vertices, and each part area of both hands.

If it is determined that hand washing has started in S304, all the values in the score data are initialized to zero (S305).

The contact area estimation unit 203 processes the contact area estimation (S306). In FIG. 11, when the distance 1103 between the apex 1101 of the left hand and the apex 1102 of the right hand in the three-dimensional space is equal to or less than the threshold value, it is determined that 1101 and 1102 are in contact with each other. This process is performed between all the vertices of the left hand and the vertices of the right hand. Contact determination may be performed within the same group of vertices of one hand. The threshold may be changed according to the scale value of the hand.

As shown in FIG. 12, the contact area estimation unit 203 may estimate the contact area by approximating the vertex group to the sphere information in order to speed up the processing. The vertex group is approximated by a set of spheres 1201 having center coordinates and radii, and it is determined whether or not the left-hand sphere and the right-hand sphere are in contact with each other. If the distance in the three-dimensional space between the center 1202 of the left-hand sphere and the center 1203 of the right-hand sphere is less than or equal to the sum of the radius 1204 of the left-hand sphere and the radius 1205 of the right-hand sphere, it is determined that they have touched each other. In this case, the smallest unit of the small area is the unit of the sphere. In addition to the sphere, an ellipsoid or a combination thereof may be used for approximation.

The score calculation unit 205 adds the score based on the presence or absence of contact in the small area (S307). When it is determined that the vertices 1101 and 1102 are in contact with each other in FIG. 11, +1 is added to the scores of 1101 and 1102. Specifically, it is added to the score of the corresponding small area number in the score table of FIG.

In each small area, the higher the score, the greater the number of contacts, which means that hand washing is performed well. The smaller the score, the fewer contacts there are, which means that hand washing is inadequate.

In FIG. 14, when it is determined that the apex 1401 of the left hand and the apex 1402 of the right hand are in contact with each other, and the apex 1402 is moving in the direction 1405 orthogonal to the normal vector of the apex 1402, the apex 1401 and the apex 1402 Add higher scores. This means that both hands are not only in contact but also rubbing against each other. For the orthogonal orientation 1405, the normal may be obtained from the surface including the vertices 1403 and 1404 in the vicinity of the vertices 1402, and may be obtained from the normal.

The information output unit 206 presents the score table to the user (S308). The information output unit 206 is a display device, which displays an image imitating both hands as shown in FIG. 15, and visualizes and displays the score for each small area. By displaying the area around the small area with a high score in white and the area around the small area with a low score in black, the user can visually know where in both hands is not washed.

Alternatively, the information output unit 206 may notify the user of the score distribution by voice. For example, the location of the fingers and the degree of hand washing can be notified by voice, such as "The tip of the index finger of the right hand cannot be washed".

Alternatively, the information output unit 206 may store the score table as statistical data in another storage medium, or communicate with the information output unit 206 to notify another device.

(Second embodiment)
The second embodiment of the present invention includes a hand-washing operation type estimation unit in addition to FIG. 2 which is the first embodiment.

The hand-washing motion type estimation unit outputs the hand-washing motion type using the surface shape data (FIG. 7) or the joint position (FIG. 8) output by the hand position shape estimation unit 202.

The types of hand-washing operations include, for example, washing between fingers, washing wrists, washing between fingertips and nails, and the like.

As shown in FIG. 16, when the left hand 1601 and the right hand 1602 wash the fingertips of the right hand, the fingertip region of the right hand and the palm region of the left hand indicated by 1603 are in contact with each other. In this case, points are added to the scores (FIG. 13) of the fingertip area of the right hand and the small area corresponding to the palm area of the left hand.

In this way, in addition to the first embodiment, when it is estimated that a predetermined hand-washing motion type has been performed, a score of a predetermined small area corresponding to the hand-washing motion is added to give points. Even if there is imaging noise or fitting error to the hand model data, the contact status of a small area is estimated more robustly.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and it is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

101 Hand wash location, 102 Faucet, 103 User, 104 Imaging unit, 105 Information output unit, 201 Imaging unit, 202 Hand position shape estimation unit, 203 Contact area estimation unit, 204 Hand wash start determination unit, 205 Score calculation unit, 206 information Output, 207 weight coefficient data, 208 hand model data, 209 score data, 401 image, 402 left hand, 403 right hand, 601 left hand, 602 right hand, 701 apex, 702 edge, 801 joint site, 802 link, 901 input image, 902 Neural network, 903 output image, 904 weighting coefficient data, 905 left hand, 906 right hand, 907 left hand, 908 right hand, 1001 left hand, 1002 right hand, 1003 optimizer, 1004 left hand, 1005 right hand, 1006 hand model data, 1101 left hand apex , 1102 Right hand apex, 1103 Distance between apex, 1201 Sphere representation, 1202 Left hand sphere center, 1203 Right hand sphere center, 1204 Left hand sphere radius, 1205 Right hand sphere radius, 1401 Left hand apex, 1402 Right hand apex 1, 1403 Right hand apex 2, 1404 Right hand apex 4, 1405 Right hand apex 1, 2, 3 Vector orthogonal to normal, 1601 Left hand, 1602 Right hand, 1603 Area where left and right hands meet

Claims (16)

It is a hand-washing evaluation device that evaluates the hand-washing movement of a person.
An imaging unit that captures an image of the area to be hand-washed,
A hand position shape estimation unit that estimates the three-dimensional position and surface shape of both hands of a person from an image captured by the imaging unit, and a hand position shape estimation unit.
A contact area estimation unit that estimates contact with other small areas in units of small areas on the surface of both hands estimated by the hand position shape estimation unit, and a contact area estimation unit.
A hand-washing start determination unit that determines that a person has started hand-washing,
For each of the small areas, a score calculation unit that calculates the contact score after the start of hand washing,
The score distribution the score calculation unit has calculated, notify the user, an information output unit for notifying stored in another storage medium, or by communicating to another device,
A hand-washing evaluation device equipped with. The imaging unit is a stereo camera, a Time-Of-Flight system camera, capable of acquiring an image including depth information.
The hand-washing evaluation device according to claim 1, which comprises any of the Structured-Light type cameras. The hand-washing evaluation device according to claim 1, wherein the imaging unit is an RGB camera that acquires color information, and the hand position shape estimation unit estimates the relative three-dimensional position and surface shape of both hands. The hand position shape estimation unit receives all or partial images of the images acquired by the imaging unit as inputs.
The hand-washing evaluation device according to claim 1, wherein the three-dimensional position and surface shape of both hands are estimated using a Convolutional-Neural-Network. The hand position shape estimation unit uses a three-dimensional shape model of the hand prepared in advance.
The hand-washing evaluation device according to claim 1, wherein the three-dimensional position and surface shape of both hands are estimated by taking correspondence with the information measured in the image. The small regions are the vertices of a data structure in which the surface shape of both hands is represented by a group of three-dimensional vertices, a small set of vertices, and each region of both hands.
The hand-washing evaluation device according to claim 1, which is any of the above. The contact area estimation unit determines the presence or absence of contact based on whether or not the distance between the small areas in the three-dimensional space is equal to or less than the threshold value.
The hand-washing evaluation device according to claim 1. The hand washing start determination unit, by interval of the presence or absence of hands within the image which the image pickup unit takes an image, determines whether a new hand washing is started, hand washing evaluation apparatus according to claim 1. The hand-washing start determination unit measures the entry and exit of a person in the space in front of the hand-washing evaluation device.
Newly determining whether hand washing is started, hand washing evaluation apparatus according to claim 1. The hand-washing evaluation device according to claim 1, wherein the score calculation unit initializes a score when it is determined by the hand-washing start determination unit that hand washing has been started. The imaging unit continuously acquires images in the time direction, and the contact area estimation unit estimates the presence or absence of contact in each image.
The score calculation unit adds points to each of the small area units and calculates the final score by accumulating the scores in a plurality of images.
The hand-washing evaluation device according to claim 1. When the small area is in contact, the score calculation unit adds a score when the small area or the small area in contact is moving in a direction orthogonal to the three-dimensional normal on the surface of the small area. , The hand-washing evaluation device according to claim 1. Further, it is provided with a hand-washing motion type estimation unit that determines the hand-washing motion type from the motions of both hands detected by the hand position shape estimation unit.
A score for a specific small area unit is added according to the hand-washing motion type estimated by the hand-washing motion type estimation unit.
The hand-washing evaluation device according to claim 1. The information output unit is provided with a display device, and the score of contact between both hands calculated by the score calculation unit is visualized together with an image expressing both hands and displayed on the display device.
The hand-washing evaluation device according to claim 1, which presents the score distribution of both hands to the user. The information output unit is provided with a sounding device, converts the score of contact between both hands calculated by the score calculation unit into a voice for explaining, and reproduces the converted voice with the sounding device.
The hand-washing evaluation device according to claim 1, which notifies the user of the score distribution of both hands. A hand-washing evaluation program that evaluates a person's hand-washing behavior.
An imaging step that captures an image of the area to be hand-washed,
A hand position shape estimation step that estimates the three-dimensional position and surface shape of both hands of a person from the captured image,
A contact area estimation step that estimates contact with other small areas in units of small areas on the surface of both hands,
A hand-washing start determination step that determines that a person has started hand-washing,
A score calculation step to calculate the contact score after the start of hand washing for each small area of both hands,
An information output step of notifying the user of the calculated score distribution , storing it in another storage medium, or communicating to notify another device.
A hand-washing evaluation program for letting a computer run.

Images