JP2021152759A - Information processing system, information processing method and learning model generation method - Google Patents

Abstract

To provide an information processing system or the like capable of accurately identifying the type of hand-washing movement.SOLUTION: An information processing system includes a photographing unit that photographs hand fingers of a worker who washes the hands or a distance measuring unit that measures a distance to the hand fingers of the worker who washes the hands. The information processing system identifies a type of hand-washing movement by the worker based on a photographed image photographed by the photographing unit or distance data measured by the distance measuring unit. Further, the information processing system determines a progress degree of the hand-washing movement for each type of the identified hand-washing movement, and reports the determined progress degree.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing system, an information processing method, and a learning model generation method.

Those engaged in factories or stores that manufacture or sell pharmaceuticals, groceries, cosmetics, etc., medical institutions, various research institutes, etc. are required to keep their bodies clean, and in particular, work to keep their hands and fingers clean. At that time, and after the work is completed, the hands are washed. Patent Document 1 discloses a system for determining whether or not hand washing has been performed by a predetermined washing method by image processing. In the system disclosed in Patent Document 1, the operator can determine whether or not the hand washing is properly performed only by washing the hands by a predetermined washing method.

Japanese Patent No. 6305448

In the system disclosed in Patent Document 1, since conditions such as the number of fingers are used when recognizing the washing method, only the washing method under preset conditions can be recognized. In addition, since the hand area is extracted based on the skin color area and the background area, it may not be possible to accurately extract the hand area from the photographed image depending on the shooting environment, the foaming state of soap, the color of the gloves worn, and the like. In that case, there is a risk that the washing method cannot be properly recognized.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide an information processing system or the like capable of accurately specifying the type of hand-washing operation.

The information processing system according to one aspect of the present invention includes an imaging unit that photographs the fingers of a worker who washes hands, a ranging unit that measures the distance to the fingers of a worker who is washing hands, and the imaging unit. Based on the captured image or the distance data measured by the distance measuring unit, the progress of the hand-washing operation is determined for each specific unit that specifies the type of hand-washing operation by the operator and the specified type of hand-washing operation. It is provided with a determination unit for determining the above and a notification unit for notifying the progress of the hand-washing operation.

In one aspect of the present invention, the type of hand-washing operation performed by the operator can be accurately specified regardless of the photographing environment, the foaming state of soap, the color of the gloves worn, and the like.

It is a block diagram which shows the configuration example of a monitoring device. It is a schematic diagram which shows the example of the photographing state by a monitoring device. It is a schematic diagram which shows the structural example of the hand-washing operation discrimination model. It is a flowchart which shows an example of the determination processing procedure. It is a schematic diagram for demonstrating the determination process. It is a schematic diagram which shows the screen example. It is a block diagram which shows the structural example of the learning apparatus. It is a flowchart which shows an example of the generation processing procedure of the hand-washing operation discrimination model. It is a schematic diagram which shows the structural example of the operation order DB. It is a flowchart which shows an example of the determination processing procedure of Embodiment 2. It is a block diagram which shows the structural example of the monitoring apparatus of Embodiment 4.

Hereinafter, the information processing system, the information processing method, and the method of generating the learning model of the present disclosure will be described in detail with reference to the drawings showing the embodiments thereof.

(Embodiment 1)
An embodiment applied to a monitoring device that monitors whether or not a hand-washing operation for keeping fingers clean is properly performed based on a photographed image of an operator will be described. FIG. 1 is a block diagram showing a configuration example of a monitoring device, and FIG. 2 is a schematic diagram showing an example of an imaging state by the monitoring device. FIG. 2A shows a state in which a hand-washing sink (hand-washing place) provided in each facility such as a food factory, a drug factory, a medical institution, various laboratories, a cooking facility, and a store is viewed from the ceiling side, and FIG. 2B shows a state. , The state where the hand-washing sink is viewed from the right side of FIG. 2A is shown. The operator performs the hand-washing operation while standing on the lower side in FIG. 2A and the left side in FIG. 2B, for example. The upper side in FIG. 2A and the right side in FIG. 2B are the front side of the operator.

The monitoring device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a notification unit 16, a camera 17, a reading unit 18, and the like, and these units are connected to each other via a bus. Has been done. The control unit 11 includes one or a plurality of processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). The control unit 11 appropriately executes the control program 12P stored in the storage unit 12 to execute various information processing and control processing to be performed by the monitoring device 10.

The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores in advance various data and the like necessary for executing the control program 12P and the control program 12P executed by the control unit 11. Further, the storage unit 12 temporarily stores data or the like generated when the control unit 11 executes the control program 12P. Further, the storage unit 12 stores the hand-washing motion discriminating model M, which is a learned model trained by machine learning. The hand-washing motion discrimination model M is expected to be used as a program module that functions as a part of artificial intelligence software.

The communication unit 13 has an interface for connecting to a network such as the Internet by wired communication or wireless communication, and transmits / receives information to / from another device via the network. The input unit 14 receives an operation input by a user who operates the monitoring device 10 and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 11. The input unit 14 and the display unit 15 may be a touch panel configured as an integral body. The display unit 15 may be provided at a position that can be visually recognized by the workers who use each hand-washing sink, for example, as shown in FIG. 2A, and is 1 for each of a plurality of workers (plurality of hand-washing sinks). One display unit 15 may be provided.

The notification unit 16 includes a lamp, a buzzer, a speaker, a vibrator, and the like, and a worker who is performing a hand-washing operation by lighting or blinking the lamp, ringing the buzzer, outputting a message by the speaker, vibrating by the vibrator, and the like. Notify the predetermined situation (situation such as whether or not the hand-washing operation is properly performed). Further, the notification unit 16 may notify the operator of a predetermined situation by displaying a message on the display unit 15.

The camera 17 is an image pickup apparatus having a lens, an image pickup device, and the like, and acquires image data of a subject image through the lens. The camera 17 takes a picture according to an instruction from the control unit 11, and sequentially acquires one image (one frame) of image data (taken image). The camera 17 is configured to acquire, for example, 60 frames, 30 frames, or 15 frames of video data per second, and the acquired video data is sequentially stored in the storage unit 12. In addition to the configuration built in the monitoring device 10, the camera 17 may be externally attached to the monitoring device 10 and is connected to the monitoring device 10 via a network such as a LAN (Local Area Network) or the Internet. May be configured. In this case, the monitoring device 10 includes a connection unit to which an external camera can be connected or a camera communication unit capable of wireless communication with the external camera, and video data taken by the external camera is transmitted via the connection unit or the camera communication unit. get. In the present embodiment, the camera 17 is provided at a position where the fingers (state of the hand-washing operation) of the operator performing the hand-washing operation in the hand-washing sink can be photographed. It is provided so that the operator's fingers can be photographed from above the front side. In addition, in FIG. 2A and FIG. 2B, the image of the shooting range of the camera 17 is shown by a straight line. In the present embodiment, one camera 17 is used to photograph two workers, but one camera 17 may be used to photograph one worker. Further, the camera 17 only needs to be able to photograph at least the fingers of the operator, and may be configured to photograph the entire body of the operator. Further, the camera 17 may be configured to shoot not only from above the front side of the operator but also from above in the lateral direction, and may be configured to shoot only from above in the lateral direction. May be good.

The reading unit 18 reads information stored in a portable storage medium 1a including a CD (Compact Disc) -ROM, a DVD (Digital Versatile Disc) -ROM, a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, and the like. read. The control program 12P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 1a via the reading unit 18 and stored in the storage unit 12. Further, the control program 12P and various data stored in the storage unit 12 may be downloaded from another device by the control unit 11 via the communication unit 13 and stored in the storage unit 12.

FIG. 3 is a schematic diagram showing a configuration example of the hand-washing operation discrimination model M. The hand-washing motion discrimination model M can be configured by, for example, a CNN (Convolution Neural Network). The hand-washing motion discrimination model M is input to a photographed image (video data) obtained by photographing the fingers of an operator who is performing a hygienic hand-washing operation, and is executed by the photographed operator based on the input photographed image. This is a trained model that has been trained to perform an operation to determine the type of hand-washing motion (the part of the finger that is the target of hand-washing) and output the calculated result. Specifically, the hand-washing operation discrimination model M performs filter processing and compression processing on the input captured image, further calculates an output value using various functions and threshold values, and outputs the calculated output value. do. The hand-washing motion discrimination model M of the present embodiment has six output nodes, and the output node 0 outputs the probability that the operator in the captured image should be discriminated as performing the hand-washing motion on the palm and the abdomen of the hand. Then, the output node 1 outputs the probability that it should be determined that the hand washing operation is performed on the back of the hand and the back of the finger. Further, the output node 2 outputs the probability that it should be determined that the hand washing operation is performed between the fingers and the crotch, and the output node 3 outputs the probability that it should be determined that the thumb and the base of the thumb are performing the hand washing operation. .. Further, the output node 4 outputs the probability that it should be determined that the hand washing operation is performed on the fingertip, and the output node 5 outputs the probability that it should be determined that the hand washing operation is performed on the wrist. The output value of each output node is, for example, a value of 0 to 1.0, and the total of the discrimination probabilities output from all the output nodes is 1.0. The hand-washing motion discriminating model M is not limited to the above-described configuration, and for example, the types of hand-washing motion discriminated by the hand-washing motion discriminating model M are not limited to six. The type of hand-washing motion discriminated by the hand-washing motion discriminating model M is the type of hand-washing motion defined by the hygiene management standards used in each facility, and the number of types to be discriminated may be different for each facility. .. Further, in addition to the type of hand-washing operation, the hand-washing operation determination model M is used to determine whether or not the operation of applying the cleaning agent to the fingers before the hand-washing operation, the operation of flushing the cleaning agent with running water after the hand-washing operation, and the like are performed. It may be configured in. Specifically, it should be determined that the hand-washing operation determination model M is performing an output node that outputs the probability that the operation of applying the cleaning agent to the fingers is being performed, or an operation of flowing the cleaning agent with running water. An output node or the like that outputs the probability may be provided.

The hand-washing motion discriminating model M learns using training data including a type of hygienic hand-washing motion (correct label) and a photographed image of the fingers of a worker performing each hand-washing motion. In the hand washing motion discrimination model M, when the captured image included in the training data is input, an output value of 1.0 is output from the output node corresponding to the type indicated by the correct answer label included in the training data, and other outputs are output. Learn so that the output value of 0.0 is output from the node. In the learning process, the hand-washing motion discrimination model M learns to optimize various coefficients in the filtering process and the compression process to be executed, the coefficient of the function, the weighting coefficient, and the like. As a result, when the captured image is input, the trained hand-washing motion discriminating model M learned so that the operator in the captured image identifies (discriminates) the type of the hand-washing motion being executed is obtained. Be done. The learning of the hand-washing motion determination model M is performed by another learning device, but may be performed by the monitoring device 10. Further, the hand-washing motion discrimination model M can be constructed by using various machine learning algorithms.

The hand-washing operation determination model M may be configured to process a plurality of frames (time-series image data) over a predetermined unit time. In this case, the hand-washing operation determination model M can be configured by combining CNN and RNN (Recurrent Neural Network), or may be configured by using a three-dimensional CNN (3D-CNN). In the hand-washing motion discriminating model M having such a configuration, a time-series photographed image (moving image data) of the fingers of the operator who is executing the hand-washing operation is input, and is photographed based on the input photographed image. It can be configured so that the type of hand-washing operation (part of the finger to be hand-washed) being executed by the worker is discriminated and the discrimination result is output.

Hereinafter, a process of determining whether or not the operator is performing a hygienic hand-washing operation will be described based on a photographed image of the operator's fingers photographed by the monitoring device 10 of the present embodiment using the camera 17. .. FIG. 4 is a flowchart showing an example of the determination processing procedure, FIG. 5 is a schematic diagram for explaining the determination processing, and FIG. 6 is a schematic diagram showing a screen example. The following processing is executed by the control unit 11 according to the control program 12P stored in the storage unit 12 of the monitoring device 10. A part of the following processing may be realized by a dedicated hardware circuit.

In the monitoring device 10 of the present embodiment, the control unit 11 photographs the hand-washing sink using the camera 17 (imaging unit) (S11). The control unit 11 may be constantly taking pictures with the camera 17. Further, when a motion sensor for detecting the presence or absence of a person in the hand-washing sink or the vicinity of the hand-washing sink is connected to the monitoring device 10, the control unit 11 uses the camera 17 after the presence of the person is detected by the motion sensor. You may start shooting. The control unit 11 photographs the hand-washing sink with the camera 17, acquires the photographed images of the fingers of the operator who is performing the hand-washing operation with the hand-washing sink, and sequentially stores the acquired captured images in the storage unit 12. FIG. 5A shows an example of a captured image.

Since the monitoring device 10 of the present embodiment captures two hand-washing sinks (workers performing the hand-washing operation) with one camera 17, the control unit 11 extracts the imaging area of each hand-washing sink from the acquired captured images. The trimming process to be performed is executed (S12). For example, the control unit 11 extracts two imaging regions as shown in FIG. 5B from the captured image shown in FIG. 5A. When shooting one hand-washing sink with one camera 17, the control unit 11 does not need to perform the process of step S12. Further, the control unit 11 determines whether or not the hand-washing operation is performed in each hand-washing sink based on, for example, a captured image, and the hand-washing sink in which the hand-washing operation is not performed, specifically, the operator's finger is captured. The hand-washing sink that has not been used may be configured so that the trimming process is not performed.

The control unit 11 repeats the shooting process and the trimming process at a predetermined timing (for example, a timing corresponding to the shooting timing), and performs the processes after step S13. First, the control unit 11 (detection unit) determines (detects) whether or not the hand-washing operation has been started in any of the hand-washing sinks (S13). For example, a motion sensor may be installed in each hand-washing sink, and when a worker's finger is detected by the motion sensor, it may be determined that the hand-washing operation is started in the hand-washing sink in which the motion sensor is installed. .. In this case, the operator can input the start of the hand-washing operation by holding his / her finger over the motion sensor when starting the hand-washing operation. As such a motion sensor, a sensor that detects a user's finger when automatically discharging water from a faucet and stopping water can be used. In addition, each hand-washing sink is provided with an operation button for inputting the start of the hand-washing operation, and when the operation button is operated, it is determined that the hand-washing operation is started in the hand-washing sink in which the operation button is installed. May be good. In addition, when the worker's fingers are reflected in the shooting area of each hand-washing sink extracted by the trimming process, or when the photographed worker's fingers are moving in a predetermined manner, the hand-washing sink is used for hand-washing. It may be determined that the operation is started. In this case, the operator can input the start of the hand-washing operation only by putting his or her finger in the shooting range of the camera 17 or by starting the hand-washing operation within the shooting range of the camera 17.

When it is determined that the hand washing operation has not been started (S13: NO), the control unit 11 returns to the process of step S11 and repeats the processes of steps S11 to S13. When it is determined that the hand-washing operation has started (S13: YES), the control unit 11 performs the processes after step S14 for each of the hand-washing sinks from which the hand-washing operation has started. First, the control unit 11 (specific unit) specifies the type of the started hand-washing operation for each hand-washing sink in which the hand-washing operation is started (S14). Here, the control unit 11 determines which of a plurality of predetermined types of hand-washing operations is the started hand-washing operation. The pre-specified hand-washing motion is a hand-washing motion defined by hygiene management standards in each facility. For example, a hand-washing motion for washing the palm and the abdomen of the finger, a hand-washing motion for washing the back of the hand and the back of the finger, and a finger There are six types of hand-washing operations: hand-washing to wash the space (side) and crotch (base), hand-washing to wash the thumb and the bulging part of the base of the thumb, hand-washing to wash the fingertips, and hand-washing to wash the wrist. The type of such hand-washing operation is an example, and the type of hand-washing operation specified for each facility can be used.

In the present embodiment, the control unit 11 specifies the type of hand-washing operation started in the hand-washing sink based on the photographing area extracted by the trimming process in step S12 with respect to the hand-washing sink in which the hand-washing operation is started. .. Specifically, the control unit 11 inputs the photographing area of the hand-washing sink into the hand-washing motion discriminating model M, and based on the output information from the hand-washing motion discriminating model M, the hand-washing performed by the operator in the photographing area is performed. Identify (recognize) the type of operation. For example, the control unit 11 sets the type corresponding to the output node that outputs the maximum output value when the maximum output value (discrimination probability) is a predetermined value (for example, 0.7) or more in the hand-washing operation discrimination model M. , Specify the type of hand-washing operation performed by the operator in the shooting area. If the maximum output value is less than a predetermined value, the control unit 11 determines that the type of hand-washing operation performed by the operator in the photographing area cannot be specified.

The control unit 11 determines whether or not the type of hand-washing operation performed by the operator in the photographing area can be specified based on the output information from the hand-washing operation determination model M (S15). When the control unit 11 determines that the type of the hand-washing operation cannot be specified (S15: NO), the control unit 11 returns to the process of step S14 and repeats the process of step S14 based on the captured images to be sequentially acquired. When the control unit 11 determines that the type of hand-washing operation can be specified (S15: YES), the control unit 11 measures the operation time (hand-washing time) of the hand-washing operation for the specified type (S16). For example, the control unit 11 can calculate the hand-washing time for each hand-washing operation from the number of frames of the captured image to be processed. Specifically, in the case of a captured image in which 30 frames are captured per second, for example, in the case of a configuration in which one frame captured image is input to the hand-washing motion determination model M every three frames to determine the type of hand-washing motion. The control unit 11 adds 0.1 seconds as the operation time of the specified hand-washing operation each time the type of the hand-washing operation is specified once. This makes it possible to time the hand-washing time for each hand-washing operation. Further, the control unit 11 stores in the storage unit 12 the hand-washing time measured for each hand-washing operation.

The control unit 11 (notification unit) calculates the progress of each type of hand-washing operation based on the timed hand-washing time, and displays a screen showing the progress of each hand-washing operation on the display unit 15 (S17). For example, the control unit 11 (determination unit) reads the operation completion time for the specified hand-washing operation from the storage unit 12, calculates the ratio of the timed hand-washing time to the operation completion time, and thereby determines the progress of the hand-washing operation. calculate. In the present embodiment, the progress of each hand-washing operation is calculated based on the hand-washing time. For example, the progress of each hand-washing operation is calculated based on the number of frames of the captured image input to the hand-washing operation determination model M. May be configured to calculate. In this case, the number of image frames corresponding to the hand-washing time to be executed for each hand-washing operation (specifically, the number of frames corresponding to the number of frames taken per unit time) is stored in the storage unit 12. Remember. Then, the control unit 11 reads out the number of operation completion frames for the specified hand-washing operation from the storage unit 12, and calculates the ratio of the counted number of frames (the number of frames corresponding to the hand-washing time) to the number of operation completion frames. , The progress of the hand-washing operation may be determined.

FIG. 6A shows an example of a screen showing the progress status, and displays the progress of each of the six types of hand-washing operations (progress for each type of hand-washing operation). On the screen shown in FIG. 6A, a photographed image of a worker performing a hand-washing operation in one hand-washing sink, a progress bar (indicator) showing the progress of six types of hand-washing operations, and a model image showing a model of the hand-washing operation. Is displayed. Each time the operator is photographed by the camera 17 and the photographed area is extracted by the trimming process, the control unit 11 displays the extracted photographed area in the display area of the photographed image on the screen shown in FIG. 6A. Further, each time the control unit 11 calculates the progress of each hand-washing operation, in the progress status display area on the screen shown in FIG. 6A, the progress bar calculated by the progress bar corresponding to the hand-washing operation for which the progress is calculated is used to calculate the progress. indicate. The progress bar in the screen shown in FIG. 6A shows the operation completion time set for each hand-washing operation by a broken line and the executed hand-washing time by hatching. With such a configuration, the progress of each hand-washing operation is shown. Can be easily grasped.

When determining the progress of each hand-washing operation, the control unit 11 may consider not only the executed hand-washing time but also the output value (discrimination probability) from the hand-washing operation determination model M. For example, if the output value from the hand-washing operation determination model M is a predetermined value (for example, 0.95) or more, the operation completion time set for this hand-washing operation is shortened (for example, operation completion time × 0.9). The progress of the hand-washing time with respect to the shortened operation completion time may be calculated. Conversely, if the output value from the hand-washing operation discrimination model M is less than a predetermined value (for example, 0.9) (that is, if the output value is 0.7 or more and less than 0.9), this hand-washing operation is set. The operation completion time may be lengthened (for example, the operation completion time × 1.1), and the progress of the hand-washing time with respect to the lengthened operation completion time may be calculated.

The model image on the screen shown in FIG. 6A is stored in the storage unit 12 in advance. In the screen shown in FIG. 6A, the numbers 1 to 6 of the model image correspond to the numbers 1 to 6 assigned to each progress bar. When the control unit 11 identifies the hand-washing operation during execution, the progress bar number corresponding to the hand-washing operation during execution and the model image number are different from the other numbers on the screen shown in FIG. 6A. Display with. This makes it possible to notify which hand-washing operation is currently being performed. Specifically, the screen shown in FIG. 6A shows a display example when the operator is washing the back of the hand and the back of the finger, and No. 2 is highlighted in the progress bar and the model image, and No. 2 is displayed. Indicates that the hand-washing operation of is in progress.

The control unit 11 determines whether or not the hand-washing operation is completed in the hand-washing sink that is timing the hand-washing time (S18). For example, the control unit 11 determines that the hand-washing operation has been completed when the progress of all the hand-washing operations reaches 100%. In addition, for example, when the presence (absence) of the worker or the worker's fingers is detected by the motion sensor installed in each hand-washing sink, or the worker or the worker's fingers disappear from the captured image (shooting area). In this case, it may be determined that the hand washing operation has been completed. Further, for example, when the number of times the operator's finger is detected by the motion sensor is the second time, it may be determined that the hand washing operation is completed. Further, each hand-washing sink may be provided with an operation button for inputting the end of the hand-washing operation, and when the operation button is operated, it may be determined that the hand-washing operation is completed. When it is determined that the hand washing operation is not completed (S18: NO), the control unit 11 returns to the process of step S14 and repeats the processes of steps S14 to S17. As a result, the control unit 11 repeats the shooting process, the trimming process, the process of timing the time of each hand-washing operation, and the like, and on the screen shown in FIG. The progress indicated by the progress bar of each hand-washing operation is updated. Therefore, the operator who is performing the hand-washing operation can easily grasp the progress status of his / her own hand-washing operation by the progress bar of each hand-washing operation.

When it is determined that the hand-washing operation is completed (S18: YES), the control unit 11 determines whether or not all the hand-washing operations are normally completed (S19). Here, the control unit 11 determines that the process has ended normally when the progress of each hand-washing operation stored in the storage unit 12 and displayed on the screen reaches 100%. When it is determined that the process has ended normally (S19: YES), the control unit 11 notifies that the hand-washing operation has ended normally (S20), and ends the process. For example, the control unit 11 displays a message such as "appropriate hand-washing operation is completed" on the display unit 15 to notify the display unit 11. When the notification unit 16 has a speaker, the control unit 11 may output the above-mentioned message by voice from the speaker.

When it is determined that the hand-washing operation has not been normally completed (interrupted) (S19: NO), the control unit 11 (abnormality notification unit) indicates that the hand-washing operation has been interrupted (operation interrupted, abnormal termination). Notify (S21). For example, the control unit 11 displays a message such as "hand washing operation is not completed" or "hand washing is insufficient" on the display unit 15 to notify the display unit 11. When the notification unit 16 has a speaker, the control unit 11 may output a message as described above by voice from the speaker, and when the notification unit 16 has a lamp, a buzzer, or a vibrator, the control unit 11 may output a message as described above. , The lamp may be lit or blinking, the buzzer may sound, or the vibrator may vibrate to notify the operator of the abnormality.

The control unit 11 determines whether or not the interrupted (not normally completed) hand-washing operation has been resumed (S22). For example, the control unit 11 performs a process of determining whether or not the hand-washing operation has been started by the same process as in step S13, and the hand-washing operation is started within a predetermined time (for example, several seconds) after the notification of step S21 is given. If it is determined that the hand-washing operation has been interrupted, it is determined that the interrupted hand-washing operation has been resumed as the hand-washing operation by the same operator. When it is determined that the hand washing operation has been restarted (S22: YES), the control unit 11 returns to the process of step S14. As a result, the control unit 11 performs imaging processing and trimming processing of the operator performing the restarted hand-washing operation, identification processing of the type of hand-washing operation, timekeeping processing of the hand-washing time, and the like. In step S16 here, the hand-washing time that was timed before the interruption is taken over, and the time-taking process is restarted from the time of each hand-washing operation stored in the storage unit 12. When the hand-washing operation is interrupted, the hand-washing time measured before the interruption may be reset. In this case, the operator who interrupted the hand-washing operation needs to restart the hand-washing operation from the beginning. When it is determined that the hand-washing operation has not been restarted (S22: NO), the control unit 11 ends the process. At this time, the control unit 11 may reset the time of each hand-washing operation stored in the storage unit 12 for this worker.

By the above-described processing, the monitoring device 10 monitors the hand-washing operation performed by the worker based on the captured image of the state of the hand-washing operation of the worker, and manages the progress of the hand-washing operation. Then, when the operator appropriately performs the predetermined hand-washing operation, the monitoring device 10 notifies the operator to that effect (normal end). Further, when the operator interrupts the hand-washing operation in the middle, the monitoring device 10 notifies the operator to that effect (interruption). Therefore, the operator can judge whether or not his / her hand-washing operation is appropriate not only by his / her own judgment but also by the judgment process by the monitoring device 10. In the present embodiment, the progress bar on the screen shown in FIG. 6A notifies the progress of each hand-washing operation, so that the operator performing the hand-washing operation can easily grasp the progress status of each hand-washing operation. can. Therefore, when the operator performs the hand-washing operation while checking the progress, the degree of cleanliness varies depending on the washing method (type of hand-washing operation) such as the hand-washing operation of excessively washing the same part or insufficient washing. It is possible to carry out an efficient hand-washing operation.

In the present embodiment, by displaying the screen as shown in FIG. 6A on the display unit 15, the operator is notified of the progress of the hand-washing operation being executed, but the present invention is not limited to such a configuration. FIG. 6B shows another display example of the progress status, and shows an example of displaying the progress of each hand-washing operation using, for example, a lamp. When the lamp panel shown in FIG. 6B is used, the lamp panel is provided at the installation position of the display unit 15 in FIG. 2A instead of the display unit 15. The lamp panel shown in FIG. 6B has a state lamp L1 indicating the start and end of the hand-washing operation, and progress lamps L21 to L26 indicating the progress of the six types of hand-washing operations. When the control unit 11 of the monitoring device 10 determines that the hand-washing operation has started in step S13 in FIG. 4 (S13: YES), the state lamp L1 is turned on to notify the start of the hand-washing operation. Further, when the control unit 11 determines in step S19 in FIG. 4 that the hand-washing operation is normally completed (S19: YES), the control unit 11 turns off the status lamp L1 and notifies the end of the hand-washing operation. Further, in step S17, the control unit 11 turns on or blinks the progress lamps L21 to L26 corresponding to each hand-washing operation in a manner corresponding to the progress based on the time-measured hand-washing time for each hand-washing operation. Notify the progress of each hand-washing operation. For example, the control unit 11 may light the progress lamps L21 to L26 with a color or brightness according to the progress, or may blink the progress lamps L21 to L26 at a blinking speed according to the progress. .. Further, the control unit 11 may notify the operator of the hand-washing operation during execution by blinking the progress lamps L21 to L26 corresponding to the hand-washing operation during execution. When a lamp panel as shown in FIG. 6B is used, a poster printed with a model image as shown in FIG. 6A is attached at a position where the operator who performs the hand-washing operation can visually recognize the model. The operation can be notified to the worker.

In the present embodiment, the hand-washing motion discriminating model M is used to discriminate the type of hand-washing motion performed by the operator in the captured image. Therefore, it is not easily affected by the color of the hand-washing sink and the background, the color of the operator's fingers, and the shooting environment, and it is possible to accurately discriminate the hand-washing operation performed by the worker in the captured image. Further, even when the type of hand-washing motion to be discriminated is changed, the set washing method (hand-washing motion) can be flexibly changed by appropriately learning the hand-washing motion discriminating model M. Further, in the present embodiment, the hand-washing motion determination model M has a configuration for discriminating the type of hand-washing motion performed by the operator in the captured image, for example, a hand-washing operation defined by the operator in the captured image. It may be configured to determine whether or not it is done. For example, the hand-washing motion determination model M may be configured to input a captured image and output a probability that an operator in the input captured image should be determined to perform a predetermined hand-washing motion. When such a hand-washing motion determination model M is used, it is possible to monitor whether or not the photographed worker is performing the prescribed hand-washing motion.

In the present embodiment, the monitoring device 10 may further include a configuration for identifying a worker from among pre-registered employees. For example, the monitoring device 10 is provided with a voice input function, and the worker inputs his / her name by voice before starting the hand-washing operation, so that the monitoring device 10 makes the worker an employee with the voice-input name. It may be specified. Further, when a name tag on which the worker's name is written is attached to the work clothes and the name tag can be photographed by the camera 17, the monitoring device 10 extracts the area of the name tag from the photographed image and performs OCR (OCR) from the extracted area. The name of the worker may be acquired by generating text data by (Optical Character Recognition). Further, when the face image of each employee is registered in advance and the camera 17 can take a picture of the worker's face, the monitoring device 10 takes a picture based on the worker's face image in the picture. The worker inside may be identified from among the employees. When the worker is specified in this way, it is possible to collect the execution results of the hand-washing operation of each employee, and it can be used for improving hygiene guidance and hygiene management.

(Embodiment 2)
The generation process of the hand-washing operation determination model M used in the monitoring device 10 of the first embodiment will be described. The learning of the hand-washing motion discrimination model M is performed by, for example, a predetermined learning device, and the learned hand-washing motion discrimination model M generated by learning by the learning device is via the network or via the portable storage medium 1a. It is downloaded from the learning device to the monitoring device 10 and stored in the storage unit 12. FIG. 7 is a block diagram showing a configuration example of the learning device. The learning device 50 is composed of a personal computer, a server computer, and the like, and includes a control unit 51, a storage unit 52, a communication unit 53, an input unit 54, a display unit 55, a reading unit 56, and the like, and each of these units has a bus. They are connected to each other via. Since each part 51 to 56 of the learning device 50 is the same as each part 11 to 15 and 18 of the monitoring device 10 of the first embodiment, detailed description thereof will be omitted. The storage unit 52 of the learning device 50 stores the training data DB (database) 52a in which the training data used for learning the hand washing motion discriminating model M is stored in addition to the control program 52P and the hand washing motion discriminating model M. .. The training data DB 52a stores training data in which information indicating the type of hand-washing operation (correct label) is associated with a photographed image of a worker's finger performing each type of hand-washing operation. ..

FIG. 8 is a flowchart showing an example of the generation processing procedure of the hand-washing operation determination model M. The control unit 51 of the learning device 50 acquires one training data from the training data DB 52a (S31). The control unit 51 performs a learning process of the hand-washing motion discrimination model M using the read training data (S32). Here, when the control unit 51 inputs the captured image included in the training data into the hand-washing motion determination model M, the output value from the output node corresponding to the correct answer label included in the training data approaches 1.0. The hand-washing operation discrimination model M is trained so that the output values from other output nodes approach 0. In the learning process, the hand-washing motion discrimination model M learns to optimize various coefficients in the filter process and the compression process to be executed, the coefficients of the function, and the like.

The control unit 51 determines whether or not there is unprocessed training data in the training data stored in the training data DB 52a (S33), and determines that there is unprocessed training data (S33: YES). , Return to the process of step S31. Then, the control unit 51 reads the unprocessed training data from the training data DB 52a, and repeats the learning process using the read training data. When it is determined that there is no unprocessed training data (S33: NO), the control unit 51 ends the learning process. As a result, when the captured image is input, the hand-washing motion discrimination model M learned to output the discrimination probability for each type of hand-washing motion as the discrimination result of the hand-washing motion performed by the operator in the captured image. Is generated. The hand-washing motion discriminating model M that has already been learned can also be re-learned by performing the above-described processing, and in this case, a hand-washing motion discriminating model M with higher discrimination accuracy can be generated. Further, the hand-washing motion discrimination model M can be optimized by repeatedly performing the learning process shown in FIG. 8 using the training data stored in the training data DB 52a.

When the type of hand-washing motion of the worker is discriminated by using the hand-washing motion discriminating model M as described above, the hand-washing motion performed by the worker in the captured image can be accurately discriminated without depending on the shooting environment or the like. Further, when the hand-washing motion discriminating model M is trained using a huge amount of training data, it is possible to realize a hand-washing motion discriminating model M capable of more accurately discriminating the hand-washing motion performed by the operator in the captured image.

The hand-washing motion discrimination model M generated by the learning process described above can be subjected to transfer learning (additional learning) using the training data collected at the facility where the monitoring device 10 is installed. Specifically, the information indicating the type of hand-washing operation (correct answer label) and the fingers of the worker performing each type of hand-washing operation were photographed at the hand-washing sink (predetermined hand-washing place) of the facility where the monitoring device 10 is installed. The training data associated with the captured image is used as training data for transfer learning. In the transfer learning as well, the hand-washing motion discrimination model M performs a learning process for optimizing the coefficients and weighting coefficients of various functions used in the calculation by the same process as in FIG. By using the training data for transfer learning to transfer-learn the hand-washing motion discrimination model M by the same process as in FIG. 8, the hand-washing motion discrimination model M capable of performing discrimination processing suitable for the facility where the monitoring device 10 is installed. Can be realized. Therefore, after learning the hand-washing motion discrimination model M using the photographed image of the hand-washing motion in a general hand-washing sink, transfer learning is performed using the photographed image of the hand-washing motion in the hand-washing sink of each facility. It is possible to efficiently generate a hand-washing operation discriminating model M suitable for the above.

In the learning process of the hand-washing motion discrimination model M, for example, a hand-washing motion for washing the back of the hand and the back of the finger, a hand-washing motion for washing the thumb and the base of the thumb, a hand-washing motion for washing the fingertips, a hand-washing motion for washing the wrist, etc. For the hand-washing operation in which there is a washing method, the left and right fingers may be learned separately. In this case, the hand-washing operation determination model M determines whether or not the above-mentioned hand-washing operation is performed for each of the left and right fingers. Further, the hand-washing motion determination model M is, for example, a model in which a photographed image of a worker's finger taken from above the front side of the worker is input, and a photograph taken of the worker's finger from above the left or right side of the worker. A model for inputting an image, a model for inputting a photographed image of the operator's fingers taken from diagonally above the operator, and the like may be generated separately. In this case, the hand-washing operation discrimination model M according to the shooting situation by the camera 17 can be used, and the discrimination processing with high accuracy becomes possible.

Further, the hand-washing motion discrimination model M may be configured to input one captured image and an optical flow showing the movement of the worker in the captured image as a vector instead of the time-series captured image (video data). good. Also in this case, the hand-washing motion determination model M can be configured to discriminate the type of hand-washing motion performed by the worker being photographed based on the input captured image and optical flow. Further, when the monitoring device 10 has a configuration for measuring the distance to the operator's finger using the depth sensor, the hand-washing motion determination model M includes the photographed image and the optical flow (motion vector) extracted from the photographed image. , The depth information (distance information) by the depth sensor may be input. In this case, the hand-washing motion discrimination model M is a learning model (neural network) that outputs the features of the input captured image, a learning model that outputs the features of the input optical flow, and the features of the input depth information. It may be configured by a multimodal model including three learning models of a learning model for outputting the above and discriminating the type of hand washing operation based on each feature amount output from the three learning models.

(Embodiment 3)
A monitoring device for monitoring whether or not each hand-washing operation is properly performed based on a photographed image of an operator when the order to be performed for a plurality of types of hand-washing operations is determined will be described. .. Since the monitoring device of the present embodiment has the same configuration as the monitoring device 10 of the first embodiment, detailed description of the configuration will be omitted. In addition to the configuration of the first embodiment shown in FIG. 1, the monitoring device 10 of the present embodiment stores the operation sequence DB 12a in the storage unit 12.

FIG. 9 is a schematic diagram showing a configuration example of the operation sequence DB 12a. The operation order DB 12a stores the order in which the hand washing operation is performed on each part of the fingers. The operation sequence DB 12a shown in FIG. 9 includes an order sequence, a hand-washing operation sequence, an operation completion time sequence, and the like, and stores the type of hand-washing operation (the part of the finger to be hand-washed) and the operation completion time in association with the order. Each order of 1 to 6 is stored in advance in the sequence column, the hand-washing operation column stores the parts of the fingers (type of hand-washing operation) to be hand-washed in each order, and the operation completion time column is used for each hand-washing operation. The set time for hand washing (operation completion time) is memorized. The hand-washing operation (portion of fingers) and the operation completion time stored in the operation sequence DB 12a are obtained when the control unit 11 acquires the hand-washing operation and the operation completion time for each order via the communication unit 13 or the input unit 14. It is stored by the control unit 11. The contents stored in the operation sequence DB 12a are registered according to the hygiene management standard in the facility where the monitoring device 10 is installed, and can be changed as appropriate. The stored contents of the operation sequence DB 12a are not limited to the example shown in FIG. 9, and the number of sequences, that is, the number of types of hand washing operations to be discriminated is not limited to six.

Hereinafter, the process of determining whether or not the operator is properly performing the hand-washing operation based on the captured image of the operator's fingers by the monitoring device 10 of the present embodiment will be described. FIG. 10 is a flowchart showing an example of the determination processing procedure of the second embodiment. In the process shown in FIG. 10, steps S41 to S42 are added between YES and step S16 in step S15, and steps S43 to S46 are added in place of steps S18 to S19. The same steps as in FIG. 4 are omitted, and in FIG. 10, steps S11 to S13 in FIG. 4 are not shown.

In the monitoring device 10 of the present embodiment, the control unit 11 performs the processes of steps S11 to S15 in FIG. 4 described in the first embodiment. As a result, the control unit 11 identifies the hand-washing operation being executed by the operator based on the shooting area extracted by the trimming process from the shot image shot by the camera 17. When it is determined in step S15 that the operator in the photographing area can specify the type of hand-washing operation being executed (S15: YES), the control unit 11 determines that the specified hand-washing operation is in the order registered in the operation order DB 12a. It is determined whether or not the operation is appropriate according to (S41). For example, when the hand-washing operation is specified for the first time, the control unit 11 reads "palm and finger pad" as the first hand-washing operation from the operation sequence DB12a, and the specified hand-washing action is the operation of washing the palm and finger pad ( Judge whether or not it matches the read operation). Then, the control unit 11 determines that the specified hand-washing operation is an appropriate operation if they match, and determines that the specified hand-washing operation is not an appropriate operation if they do not match.

When it is determined that the operation is appropriate (S41: YES), the control unit 11 shifts to the process of step S16 and performs the processes of steps S16 to S17 in FIG. 4 described in the first embodiment. As a result, the control unit 11 displays a screen as shown in FIG. 6A on the display unit 15 and notifies the operator of the progress of each hand-washing operation. When it is determined that the operation is not appropriate (S41: NO), the control unit 11 notifies that the operation is not appropriate (inappropriate) (S42). Here, the control unit 11 displays a message such as "The order of hand washing is incorrect. Please wash your hands according to the manual" on the display unit 15 or outputs a voice from the speaker (notification unit 16) to the operator. Notice. In addition, the control unit 11 displays or voices a message on the display unit 15 that specifies a part (part) to be washed by hand, such as "Please wash from the palm / finger pad" and "Next, wash your fingertips". It may be output and notified. After the process of step S42, the control unit 11 returns to the process of step S14, and based on the captured image, the process of identifying the hand-washing operation being executed by the operator and the specified hand-washing operation follow the set order. The process of determining whether or not the operation is appropriate is repeated.

After the process of step S17, the control unit 11 determines whether or not the appropriate hand-washing operation is normally completed in the hand-washing sink that is timing the hand-washing time (S43). Here, the control unit 11 determines that the hand-washing operation here is normally completed when the progress of the hand-washing operation during the timekeeping of the hand-washing time reaches 100%. When it is determined that the hand-washing operation during the timekeeping is not normally completed (interrupted) (S43: NO), the control unit 11 shifts to the process of step S21 and the hand-washing operation is interrupted. Is notified (S21). Then, when the control unit 11 determines that the interrupted hand-washing operation has been restarted (S22: YES), the control unit 11 returns to the process of step S14, and the restarted hand-washing operation follows the order registered in the operation order DB 12a. If the operation is appropriate, the hand washing time before the interruption is taken over, and the time counting process of the hand washing time of the restarted hand washing operation is restarted.

When it is determined that the hand-washing operation during the timekeeping of the hand-washing time is completed normally (S43: YES), the control unit 11 has elapsed the operation completion time for the hand-washing operation being executed (the hand-washing operation has been completed normally). ) Is notified (S44). For example, the control unit 11 displays a message such as "The part being washed by hand has been sufficiently washed" on the display unit 15 or outputs a voice from the speaker (notification unit 16) to notify the operator. Next, the control unit 11 determines whether or not there is an unfinished hand-washing operation among the hand-washing operations stored in the operation sequence DB 12a (S45). Here, the control unit 11 determines whether or not the hand-washing operation determined to have been normally completed in step S43 is the sixth hand-washing operation (hand-washing operation for washing the wrist), and the sixth hand-washing operation. If so, it is determined that there is no incomplete hand-washing operation, and if it is not the sixth hand-washing operation, it is determined that there is an incomplete hand-washing operation.

When it is determined that there is an incomplete hand-washing operation (S45: YES), the control unit 11 gives a notification prompting the change of the type of the hand-washing operation (S46). For example, the control unit 11 displays a message such as "Please wash your hands in the next washing method" or "Next, wash your fingertips" on the display unit 15 or outputs a voice to notify the operator. After that, the control unit 11 returns to the process of step S14, and based on the captured images taken sequentially, the process of specifying the hand-washing operation being executed by the operator and the specified hand-washing operation are appropriate according to the set order. The process of determining whether or not the operation is normal, and the process of timing the hand-washing time when the hand-washing operation is appropriate are repeated. Then, when it is determined in step S45 that there is no unfinished hand-washing operation (S45: NO), the control unit 11 proceeds to the process of step S20 and notifies that all the hand-washing operations have been normally completed (S20). ), End the process.

By the above-described processing, in the monitoring device 10 of the present embodiment, not only whether or not the worker properly performs each preset hand-washing operation, but also the worker performs each hand-washing operation according to a preset order. You can monitor whether you are doing it properly. Further, also in the present embodiment, the monitoring device 10 can notify the operator of the progress of the hand-washing operation by the operator. Therefore, the operator can confirm whether or not his / her hand-washing operation is appropriate not only by his / her own judgment but also by the judgment process by the monitoring device 10, and can grasp the remaining hand-washing time and the like for the ongoing hand-washing operation. Because it can be done, efficient hand washing operation becomes possible.

In this embodiment, the same effects as those in each of the above-described embodiments can be obtained. Further, in the present embodiment, it is possible to monitor whether or not the operator appropriately performs each hand-washing operation in a predetermined order in consideration of the operation order preset for the washing method (each hand-washing operation). Therefore, in the facility where the monitoring device 10 is installed, even if the hand-washing order for the hand-washing operation (parts of fingers) is specified by the hygiene management standard, the monitoring process in accordance with the hygiene management standard becomes possible. .. Also in this embodiment, it is possible to apply the modifications described as appropriate in each of the above-described embodiments. Further, the hand-washing operation discrimination model M in the present embodiment can also be generated by the generation process described in the second embodiment.

(Embodiment 4)
In the monitoring devices 10 of the first to third embodiments described above, the monitoring device 10 that uses the depth sensor instead of the camera 17 will be described. FIG. 11 is a block diagram showing a configuration example of the monitoring device 10 of the fourth embodiment. The monitoring device 10 of the present embodiment has a depth sensor 19 instead of the camera 17 in the monitoring device 10 of the first embodiment shown in FIG. Since the configurations other than the depth sensor 19 are the same as those of the first embodiment, detailed description thereof will be omitted.

The depth sensor 19 (distance measuring unit) includes, for example, a light emitting unit that emits light such as laser light, near-infrared light, or infrared light, and a light receiving unit that receives reflected light from an object of the light emitted by the light emitting unit. And measures the distance to the object based on the received reflected light. The depth sensor 19 measures the distance to the object according to the instruction from the control unit 11, and acquires the distance data indicating the measured distance. The depth sensor 19 is configured to acquire distance data of, for example, 30 frames or 15 frames per second, and the acquired distance data is sequentially stored in the storage unit 12. In addition to the configuration built in the monitoring device 10, the depth sensor 19 may be configured to be externally attached to the monitoring device 10 or may be connected to the monitoring device 10 via a network such as LAN or the Internet. .. In this case, the monitoring device 10 includes a connection unit capable of connecting an external sensor or a sensor communication unit capable of wireless communication with the external sensor, and transmits the distance data acquired by the external sensor via the connection unit or the sensor communication unit. get.

In the present embodiment, the depth sensor 19 is provided at a position where the distance to the fingers of the operator who performs the hand-washing operation in the hand-washing sink can be measured, and is provided at the same position as the camera 17 in FIGS. 2A and 2B, for example. Has been done. The depth sensor 19 may be configured to measure the distance to two workers as shown in FIG. 2A, and one depth sensor 19 may be used to measure the distance to one worker. It may be configured to measure. Further, the depth sensor 19 may be provided with at least the operator's fingers as the distance measurement target, and may be provided with the entire body of the worker as the distance measurement target. Further, the depth sensor 19 may be configured to measure the distance not only from above the front side of the operator but also from above in the lateral direction, and also measures the distance only from above in the lateral direction. It may be configured in.

The hand-washing motion discriminating model M of the present embodiment is realized by the same configuration as the hand-washing motion discriminating model M of the first embodiment, but in the hand-washing motion discriminating model M of the present embodiment, the operator who is executing the hand-washing motion The distance data measured by the depth sensor 19 for the distance to the finger is used as input. That is, in the hand-washing operation determination model M of the present embodiment, when the distance data acquired by the depth sensor 19 is input, the hand-washing operation being executed by the operator to be distance-measured is performed based on the input distance data. This is a trained model that has been trained to discriminate the type (the part of the finger that is the target of hand washing) and output the discrimination result. Even when the hand-washing motion determination model M having such a configuration is used, the type of hand-washing motion being executed by the operator can be specified.

The monitoring device 10 of the present embodiment including the depth sensor 19 can execute the same processing as the processing shown in FIG. In the monitoring device 10 of the present embodiment, in step S11 in FIG. 4, the control unit 11 uses the depth sensor 19 to perform distance measurement processing up to the fingers of the operator who performs the hand-washing operation in the hand-washing sink. .. That is, the control unit 11 measures the distance to the operator's finger, acquires the distance data, and extracts the operator's area from the acquired distance data by the trimming process. Further, in step S14 in FIG. 4, the control unit 11 inputs the distance data of the worker's area into the hand-washing motion discriminating model M, and based on the output information from the hand-washing motion discriminating model M, the operator to be distance-measured. Identify the type of hand-washing operation performed by.

By the above-described processing, the same effect as that of the first embodiment can be obtained in the monitoring device 10 of the present embodiment. Specifically, in the present embodiment, it is possible to monitor the hand-washing operation performed by the worker and manage the progress of the hand-washing operation based on the distance data obtained by measuring the distance to the fingers of the worker performing the hand-washing operation. Further, since the monitoring device 10 can notify the worker of the progress of the hand-washing operation, the worker can grasp the progress of the hand-washing operation performed by himself / herself, and not only his / her own judgment but also the judgment processing by the monitoring device 10 This makes it possible to carry out an efficient hand-washing operation.

The hand-washing operation discrimination model M in the present embodiment can be generated by the generation process described in the second embodiment. In the generation process of the hand-washing motion discrimination model M of the present embodiment, the type of hand-washing motion (correct label) and the distance to the fingers of the operator performing each hand-washing motion are measured by the depth sensor 19. Training data, including data, is used. Further, the configuration of this embodiment can be applied to the monitoring device 10 of the above-described third embodiment. When applied to the monitoring device 10 of the third embodiment, whether the operator appropriately performs each hand-washing operation according to a preset order for each hand-washing operation based on the distance data acquired by the depth sensor 19. You can monitor whether or not.

In each of the above-described embodiments, the monitoring device 10 is configured to perform a process of determining the type of hand-washing operation being executed by the operator on a predetermined server connected to the network based on the captured image or the distance data. You may. In this case, the control unit 11 of the monitoring device 10 transmits the captured images or distance data to be sequentially acquired to a predetermined server via the network, and the determination result determined by the predetermined server (worker in the captured image or distance data). The type of hand-washing operation being executed) may be acquired, and the operator in the captured image or distance data may specify the hand-washing operation being executed according to the acquired determination result. At this time, the server stores the hand-washing motion discriminating model M in the storage unit, and the operator is executing the process based on the captured image or distance data received from the monitoring device 10 using the hand-washing motion discriminating model M. The type of hand-washing operation may be determined. The server here may be realized by using a server computer or a personal computer, may be realized by using a plurality of virtual machines provided in one server, and may be realized by using a cloud server. May be done.

The embodiments disclosed this time should be considered as exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

10 Monitoring device 11 Control unit 12 Storage unit 15 Display unit 16 Notification unit 17 Camera 19 Depth sensor M Hand-washing operation discrimination model

Claims (12)

An imaging unit that captures the fingers of the worker who washes the hands, or a distance measuring unit that measures the distance to the fingers of the operator who washes the hands.
A specific unit that specifies the type of hand-washing operation by the operator based on the captured image captured by the imaging unit or the distance data measured by the distance measuring unit.
For each type of the specified hand-washing operation, a determination unit for determining the progress of the hand-washing operation, and
An information processing system including a notification unit that notifies the progress of the hand-washing operation. The notification unit notifies the progress of each type of hand-washing operation.
The information processing system according to claim 1, further comprising a display unit that displays a model image for each type of hand-washing operation. The information processing system according to claim 2, wherein the notification unit includes an indicator indicating the progress of each type of hand-washing operation. The information processing system according to any one of claims 1 to 3, wherein the determination unit determines the degree of progress based on the operation time of the hand-washing operation. The specific unit measures the captured image captured by the imaging unit or the distance measuring unit in a learning model learned to output a value related to the type of hand washing operation when a captured image or distance data is input. The information processing system according to any one of claims 1 to 4, wherein the distance data is input and the type of the hand washing operation is specified based on the output value from the learning model. The information processing system according to claim 5, wherein the determination unit determines the progress of the hand-washing operation based on the output value from the learning model. The information processing system according to any one of claims 1 to 6, wherein the determination unit determines the progress of a plurality of types of hand washing operations in a predetermined order. The information processing system according to any one of claims 1 to 7, further comprising a detection unit that detects the start of a hand-washing operation by the operator. The information processing system according to any one of claims 1 to 8, further comprising an abnormality notification unit for notifying an abnormal end when the hand washing operation is interrupted before the determination unit determines the completion of the hand washing operation. .. Acquire a photographed image of the hand-washing worker's fingers or distance data that measures the distance to the hand-washing worker's fingers.
The acquired photographed image or distance data is input to a learning model trained to output a value related to the type of hand washing operation when the photographed image or distance data is input, and the type of hand washing operation by the worker is specified. An information processing method in which a computer executes processing. Obtained a photographed image of the hand-washing worker's fingers, distance data obtained by measuring the distance to the hand-washing worker's fingers, and training data including the type of hand-washing operation by the worker.
A method of generating a learning model in which a computer executes a process of generating a learning model that outputs a value related to the type of hand-washing operation when a photographed image or distance data is input using the acquired training data. Photographed images of the fingers of a worker washing hands at a predetermined hand-washing place, distance data measuring the distance to the fingers of a worker washing hands at a predetermined hand-washing place, and types of hand-washing operations by the worker. Obtain training data for transfer learning, including
The method for generating a learning model according to claim 11, wherein the computer executes a process of performing transfer learning of the learning model using the acquired training data for transfer learning and generating a learning model according to the hand-washing place. ..

Images