JP2022186188A - Monitoring device and monitoring method - Google Patents

Abstract

To provide a monitoring device and a monitoring method that improve the accuracy of determining whether an operator's behavior is appropriate or not.SOLUTION: A monitoring device 14 includes an image acquisition unit 21 that acquires an image including a part or all of an operator's hand and an operation target operated by the operator using an imaging device 11 attached to the arm of the operator who performs the operation, an operation determination unit 23 that determines whether the operation is a predetermined operation using the image acquired by the image acquisition unit 21, and a determination result output unit 25 that outputs a determination result from the operation determination unit 23.SELECTED DRAWING: Figure 1

Description

The present invention relates to an apparatus and method for monitoring an operator who performs a predetermined operation.

Detecting the operation of an operation article placed on the operation panel by associating the coordinates of the operation article on the operation panel with the time when the operation article was operated by an operator (hereinafter also referred to as "operator"), and detecting the operation of the operation article. The coordinates of the operator's line of sight are detected in association with the time when the line of sight was detected, an image showing the coordinates of the operator's operation position on the control panel is acquired in association with the time when the image was acquired, and the operation item is operated by the operator By determining whether or not the coordinates of the line of sight of the operator and the operation position match the coordinates of the operation article, it is possible to determine whether or not the operator has acted appropriately without looking away. (Patent Document 1).

JP 2018-13958 A

However, in the conventional technology described above, if the operator's body or the operation article becomes an obstacle and the operator's operation position cannot be detected by the camera, it cannot be determined whether the operator's action was appropriate. , there is a problem that the determination accuracy decreases.

A problem to be solved by the present invention is to provide a monitoring apparatus and a monitoring method that improve the accuracy of determining whether an operator's behavior is appropriate.

The present invention is an imaging device attached to the arm of an operator who performs an operation, acquires an image including part or all of the operator's hand and an operation target operated by the operator, and obtains the image. The above problem is solved by determining whether or not the operation is a predetermined operation using the image acquired by the acquisition unit.

According to the present invention, it is possible to obtain an image suitable for determining the behavior of the operator and improve the accuracy of determining the behavior of the operator.

1 is a block diagram showing one embodiment of a monitoring system including a monitoring device according to the present invention; FIG. 1. It is a side view of an operator's arm which shows an example of the positional relationship of the imaging device 11, the detection apparatus 12, and the display apparatus 13 which are shown in FIG. FIG. 3 is an explanatory diagram showing an example of a learned model used in the operation specifying unit in FIG. 1; It is explanatory drawing which shows an example of a predetermined procedure. FIG. 2 is an explanatory diagram showing an example of a learned model used in the operation procedure determination unit in FIG. 1; 2 is a flow chart showing an example of an information processing procedure in the monitoring system of FIG. 1;

Embodiments of a monitoring device and a monitoring method according to the present invention will be described below with reference to the drawings.

[Monitoring system]
FIG. 1 is a block diagram showing a monitoring system 1 according to the invention. The monitoring system 1 is a device that monitors the operation of an operator who performs a certain operation on a certain operation target. is operated, the vehicle dealer (hereinafter also referred to as "dealer") has performed maintenance according to the procedures specified in the manual by the mechanic engaged in vehicle maintenance. It can be used, for example, to monitor whether Operators whose operations are monitored by the monitoring system 1 are not particularly limited, and examples thereof include factory workers and dealer mechanics.

As shown in FIG. 1 , the monitoring system 1 includes an imaging device 11 , a detection device 12 , a display device 13 and a monitoring device 14 . Devices that make up the monitoring system 1 are connected in a state in which data can be exchanged with each other by known means such as a wired or wireless LAN. The numbers of the imaging device 11, the detection device 12, and the display device 13 are not particularly limited as long as they are at least one or more. Also, the imaging device 11 , the detection device 12 , and the display device 13 do not have to be provided together with the monitoring device 14 , and may be installed at a location away from the monitoring device 14 . For example, the imager 11, detector 12, and display 13 may be installed near the assembly line of an assembly plant, and the monitoring device 14 may be installed in a central control room away from the assembly line, or in a remote location away from the assembly plant. Can be set on the server.

The imaging device 11 is a device for acquiring image data of surrounding objects, and is, for example, a camera including an imaging element such as a CCD, an ultrasonic camera, an infrared camera, or the like. Objects include objects existing around the operator in addition to the operator. The objects include, for example, switches and touch panels placed around the worker, parts assembled by the worker, tools used by the worker, and vehicles maintained by dealer mechanics. Further, the imaging device 11 is attached to the arm of the operator, and thereby, the image of the operator operating the operation target can be reliably acquired. The imaging device 11 has a mounting portion for mounting on the operator's arm, and can be mounted on the palm side of the operator's upper arm using, for example, a hook-and-loop fastener or an adhesive pad.

The detection device 12 is a device for detecting the operator's condition and the surrounding conditions of the operator, and includes all sensors. Sensors include, for example, an electromyograph that measures the operator's myoelectric potential, a microphone that acquires the surrounding sounds of the operator as audio data, and an accelerometer attached to the operator to detect the operator's body movements. and a thermometer that measures the temperature around the operator. The electrodes provided in the electromyograph may be needle electrodes, surface electrodes, or wire electrodes. An electromyograph is attached to an operator's body, particularly an arm including an upper arm and/or a forearm, for example, by an adhesive pad or hook-and-loop fastener, and measures the myoelectric potential of a portion of the attached operator's body that is in contact with the body. do.

Microphones include stand mics, close-talking mics, gun mics, etc., and may be omnidirectional or directional. Also, the communication method may be either wired or wireless. The microphone acquires sounds caused by the operator's actions in addition to the operator's voice and the voices of people around the operator. Sounds caused by user actions include sounds generated from switches operated by workers, sounds generated from speakers when workers touch the touch panel, and sounds generated when workers assemble multiple parts. Examples include the sound of gears meshing together and the sound of tools being used by dealer mechanics. Microphones are placed in locations where sounds around the operator can be detected, such as on the assembly line of the assembly plant, on the workbenches, and on the tools used by the workers.

Although the accelerometer is not particularly limited, it is preferably of a size that can be attached to the operator's body, especially the arm including the upper arm and / or the forearm. can be used. The thermometer is also not particularly limited, and may be a contact thermometer or a non-contact thermometer using infrared rays. Monitoring device 14 acquires detection results from these sensors at predetermined time intervals.

The display device 13 is a device for notifying an operator and/or a supervisor of operations monitored by the monitoring device 14 . The observer need not be near the operator and may be in the factory's central control room or in a remote location away from the factory. The display device 13 is, for example, a liquid crystal display, a projector, or the like, and may include a speaker. The display device 13 is located near the operator, such as a work place of a worker in an assembly plant, at which necessary information can be notified to the operator, and a monitor at a location away from the operator. It is installed at a position where it can notify necessary information to Moreover, the display device 13 may be attached to the operator as a wearable terminal. Furthermore, the imaging device 11, the detection device 12, the display device 13, and the monitoring device 14 may be integrated into one wearable terminal and attached to the operator.

An example of the positional relationship among the imaging device 11, the detection device 12, and the display device 13 is shown in FIG. In FIG. 2, the imaging device 11 is attached to the forearm A1 of the operator using the attachment portion 11a, and the detection device 12 is attached to the upper arm A2 of the operator using an adhesive pad. . Further, an operation target O is held in the hand H of the operator. Here, assuming that the viewing angle of the imaging device 11 is within the range indicated by VA, the imaging device 11 can capture an image including a portion of the operator's hand H and the operation target O. The image data captured by the imaging device 11 and the detection result of the detection device 12 are sent to the monitoring device 14 by wireless communication. In addition, in FIG. 2, the display device 13 is provided in front of the operator, and the operator can confirm the display of the display device 13 .

Returning to FIG. 1, the monitoring device 14 is a device for monitoring whether or not the operator has appropriately performed a certain operation. The image data and sensor detection results used for the monitoring are obtained from the imaging device 11 and the detection device 12, respectively, at predetermined time intervals. The monitoring device 14 uses the processor 15 to implement functions such as processing acquired data, determining whether or not an operation has been properly executed, and outputting the result of the determination. The processor 15 includes a ROM (Read Only Memory) 152 storing a program, and a CPU (Central Processing Unit) 151 which is an operating circuit for functioning as the monitoring device 14 by executing the program stored in the ROM 152. , and a RAM (Random Access Memory) 153 functioning as an accessible storage device.

[Operation monitor]
The program used by the monitoring device 14 of this embodiment is a program for realizing functions such as processing of acquired data, determination of whether or not an operation has been properly executed, and output of the result of the determination. It includes an operation monitoring section 2 which is a functional block. The operation monitoring unit 2 acquires image data from the imaging device 11, acquires detection results from the detection device 12, specifies operations performed by the user based on the acquired data, and checks that the specified operations are appropriate according to a predetermined procedure. It has a function of determining whether or not it has been executed and outputting the determination result. The operation monitoring unit 2 includes an image acquisition unit 21, an operation specifying unit 22, an operation determination unit 23, an operation procedure determination unit 24, and a determination result output unit 25, as shown in FIG. In FIG. 1, each part is extracted and shown for convenience.

Although the monitoring device 14 shown in FIG. 1 includes all of the above functional blocks, it is not necessary for a single monitoring device 14 to include all of the functional blocks. It may be provided in other equipment included or another information processing device (not shown). For example, in the monitoring system 1 of FIG. 1 , the determination result output section 25 may be provided in the display device 13 . In this case, the functions of the determination result output unit 25 are executed using the CPU, ROM, and RAM of the display device 13 .

Moreover, it is not necessary to execute all the processing of each functional block in a single device, and the function of each functional block may be realized across a plurality of devices connected in a state in which data can be exchanged. For example, in the monitoring system 1 of FIG. 1, part of the processing performed by the operation specifying unit 22 is performed by the imaging device 11, and the remaining processing is performed by the monitoring device 14. good too. In this case, the CPU, ROM, and RAM of the imaging device 11 are used to perform part of the processing for realizing the function of the operation specifying unit 22 . As another example, among the processes executed by the operation determination unit 23 and/or the operation procedure determination unit 24, part of the processing is performed by the imaging device 11, and the remaining processing is performed by the monitoring device 14. may be executed.

Functions performed by each functional block of the operation monitoring unit 2 will be described below.

The image acquisition unit 21 has a function of acquiring an image including an operator performing an operation and an operation target operated by the operator. In particular, the image acquisition unit 21 of the present embodiment acquires image data from the imaging device 11 attached to the operator's arm, and the acquired image data includes part or all of the operator's hand H and the and an operation target O operated by a person. The operator's hand H may be either the right hand or the left hand, and at least a part of the hand H may be included. is included in the image, the operation can be determined more accurately.

The operation performed by the operator is not particularly limited, and includes all operations for the operator to make some kind of input to the operation target. Specifically, the assembly factory worker touches the touch panel to change the display of the equipment on the assembly line, and the assembly factory worker connects the coupler connected to the sensor and the electronic control unit (ECU). An assembly factory worker uses a tool to tighten the bolts to attach the exhaust manifold to the engine block. A dealer's mechanic installs a spark plug in the engine. For example, a mechanic uses a torque wrench to tighten a bolt.

Also, the operation target is not particularly limited, and includes any object that is the target of some input operation performed by the operator. Examples include ON/OFF switches, touch panel user interfaces, parts such as couplers and exhaust manifolds, and tools such as torque wrenches and electric screwdrivers.

Furthermore, the image acquisition unit 21 can change the quality of the image acquired from the imaging device 11, and the operation determination unit 23, which will be described later, does not process image data while the operator is not performing an operation. During this period, the frame rate of time-series images (moving images) to be acquired may be reduced. For example, when the operation determination unit 23, which will be described later, determines that the motion of the operator's hand does not correspond to the predetermined motion, the frame rate of the time-series images to be obtained is set to a predetermined frame rate (for example, 60 fps). ). An appropriate value can be set for the predetermined frame rate within a range in which the operation can be properly monitored by the monitoring device 14 . On the other hand, while the operator is actually performing the operation, the frame rate of the time-series images to be obtained may be increased from the predetermined frame rate.

The operation identifying unit 22 has a function of identifying an operation performed by the operator from the image data acquired by the image acquiring unit 21 . The operation specifying unit 22 performs analysis such as pattern matching on image data acquired from the imaging device 11, and classifies objects included in the image data. Next, an operator is selected from the classified objects, and data relating to the part of the operator's hand extracted from the image data is obtained. Similarly, an operation target is selected from the classified objects, and data relating to the operation target extracted from the image data is acquired. Then, from these data, the positional relationship between the operator's hand and the operation target is recognized, and the user's operation is specified from the recognized positional relationship.

A correspondence relationship between the positional relationship between the operator's hand H and the operation target O and the operation that the operator is about to perform is obtained in advance for each operation and stored in a database such as the database 16, for example. . The operation specifying unit 22 can acquire the correspondence relationship between the positional relationship and the operation from the database 16 as necessary. The operation identification unit 22 identifies an operation that the operator is about to perform from the image data acquired by the image acquisition unit 21 based on the correspondence acquired from the database 16 .

As an example, the image data acquired by the image acquisition unit 21 includes a state in which a worker working on an assembly line in a factory holds a coupler in his left hand and extends his right hand toward another coupler. A case will be described. In this case, the operation specifying unit 22 searches for an operation corresponding to the positional relationship between the right hand extending to the coupler and the coupler from the correspondence relationship between the positional relationship and the operation acquired from the database 16 . The operations corresponding to this action include the operation of removing one coupler from a pair of couplers already engaged, and the operation of engaging the coupler connected to the sensor with the coupler connected to the electronic control unit. Then, since the operator holds the coupler in his left hand, the operation specifying unit 22 determines that the positional relationship is the operation of fitting the coupler connected to the sensor and the coupler connected to the electronic control unit. to be identified as corresponding to

As another example, a case where the image data acquired by the image acquisition unit 21 includes a state in which a dealer's mechanic holds engine oil in his left hand and extends his right hand into the engine room will be described. . In this case, the operation specifying unit 22 searches for an operation corresponding to the positional relationship between the right hand extending into the engine room and the right hand inside the engine room, based on the correspondence between the positional relationship and the operation acquired from the database 16. . Assuming that an operation for replenishing the coolant of the engine and an operation for replenishing the engine oil correspond to the operation corresponding to the operation, the operation specifying unit 22 determines that the operator holds the engine oil in the left hand. , the positional relationship is specified as corresponding to the operation of replenishing the engine oil.

Further, the operation identifying unit 22 may identify the operation using time-series images such as moving images as the images acquired by the image acquiring unit 21 . The operation specifying unit 22 acquires image data from the image acquiring unit 21 at predetermined time intervals and analyzes the image data. In this way, by using time-series images having predetermined time intervals, the operation can be specified more accurately. The time interval can be appropriately set according to the computing power of the CPU 151 . Alternatively or additionally, the operation identifying unit 22 may identify the operation using the detection result of the detection device 12 . For example, the operator's myoelectric potential value detected by an electromyograph attached to the operator's upper arm and the operator's acceleration value detected by an accelerometer attached to the operator's forearm. A motion may be detected and the operation may be identified using the detected motion.

Furthermore, the operation identifying unit 22 may identify the operation using a first trained model that has been trained to identify the operation from the image data acquired by the image acquiring unit 21 . A trained model is a model that has been learned in advance by machine learning so that appropriate output data can be obtained for certain input data. At least, a program that performs operations from input data to output data, and a weighting coefficient (parameter) used for the calculation. When the image data acquired by the image acquisition unit 21 is input as input data, the first trained model of the present embodiment is based on the input data, and the operation that the operator is about to perform on the image is performed. The computer (in particular, the CPU 151 of the processor 15) is operated so that the specified output data is output. By using such a first learned model, it is possible to specify an operation even for image data indicating a positional relationship for which a correspondence relationship has not been set in advance.

Although the first trained model of this embodiment is not particularly limited, it is, for example, the first neural network 3 as shown in FIG. The first neural network 3 comprises a first input layer 31, a first intermediate layer 32 and a first output layer 33, each layer containing at least one neuron. First input data 34 including at least one of the image data acquired by the image acquisition unit 21 and the detection result of the detection device 12 is input to the first input layer 31 , and the input data is input to the first input layer 31 . Output to intermediate layer 32 . The first intermediate layer 32 extracts the positional relationship between the operator's hand and the operation target from the data input from the first input layer 31 . Then, an operation corresponding to the extracted positional relationship is specified. The first output layer 33 outputs the data input from the first intermediate layer 32 as first output data 35 including the specified operation.

In the first intermediate layer 32, the correspondence relationship between the positional relationship between the operator's hand and the operation target for specifying the operation that the operator is about to perform, and the operation that the operator is about to perform is machine-learned. established by As a result, the positional relationship between the worker's hand and the switch, the positional relationship between the worker's hand and the touch panel user interface, the positional relationship between the worker's hand and the part, the positional relationship between the mechanic's hand and the tool, etc. The operation that the operator is about to perform can be identified from the positional relationship between the operator's hand and the operation target. In addition to the positional relationship between the operator's hand and the operation target, the corresponding relationship is associated with a plurality of factors such as the operator's surrounding conditions detected by the detection device 12, and the input data The parameters are set so that the appropriate operation is output for For example, a worker holding a coupler in each hand, a mechanic holding engine oil in his left hand, etc. are taken into account as parameters in identifying operations.

Further, the correspondence relationship of the first intermediate layer 32 described above may be learned in advance by machine learning, but the first input data 34 input to the first neural network 3 in the past and the first neural network 3 in the past New learning may be performed using the first teacher data 36 including the first output data 35 output from the model, or a learned model that has been previously learned by machine learning may be further learned. The learning is performed by the first machine learning unit 22a included in the operation specifying unit 22. FIG. The first training data 36 is stored in a database such as the database 16 shown in FIG. 1, for example, and can be obtained as needed. By learning using a combination of the first input data 34 and the first output data 35 in the past, the operation identification unit 22 can more accurately identify the operation based on the positional relationship between the operator's hand and the operation target. become.

Returning to FIG. 1 , the operation determination unit 23 has a function of determining whether or not the operation specified by the operation determination unit 23 is a predetermined operation using the image data acquired by the image acquisition unit 21 . A predetermined operation is an operation that is predetermined as an operation performed by an operator. For example, in the case of a factory worker, a coupler connected to a sensor and a coupler connected to an electronic control unit (ECU) and the operation of tightening the bolts using a tool to attach the exhaust manifold to the engine block are set in advance as predetermined operations. The operation determination unit 23 uses, for example, a series of images acquired by the image acquisition unit 21 to determine whether or not the operation specified by the operation determination unit 23 is a predetermined operation. It outputs to the procedure determination unit 24 and the determination result output unit 25 .

Further, the operation determination unit 23 can acquire the motion of the operator's hand H from the image acquired by the image acquisition unit 21 and determine whether or not the acquired motion corresponds to a predetermined motion. . Then, when it is determined that the movement of the hand H corresponds to the predetermined movement, the movement of the hand is determined to be the predetermined movement from a predetermined time before the time when the movement of the hand H is determined to be the predetermined movement. It is determined whether or not the identified operation is the predetermined operation only during the period from when it is determined to be the movement to when the predetermined time elapses. In this way, it is possible to perform the determination only at the time before and after the motion of the hand H is determined to be the predetermined motion, which requires the determination of the operation. The movement of the operator's hand H is determined by analysis of image data acquired by the image acquisition unit 21, myoelectric potential values detected by an electromyograph attached to the operator's upper arm A2, and the operator's forearm. It can be obtained from the value of the acceleration detected by the accelerometer attached to the portion A1. Further, the predetermined movement is set in advance as an initial movement of the hand H for the predetermined operation and stored in the database 16 or the like. The operation determination unit 23 acquires data of a predetermined movement from the database 16 as necessary.

The operation procedure determination unit 24 has a function of determining whether or not the predetermined operation has been executed according to a predetermined procedure when the operation determination unit 23 determines that the operation identified by the operation identification unit 22 corresponds to the predetermined operation. have In particular, the operation procedure determination unit 24 uses the time-series image data acquired by the image acquisition unit 21 to determine whether a predetermined operation has been performed according to a predetermined procedure. A predetermined procedure is preset for each predetermined operation and stored in a database such as the database 16 .

The operating procedure determination unit 24 acquires the specified operation from the operation specifying unit 22 at predetermined time intervals, and grasps the positional relationship between the operator's hand and the operation target included in the image data. Then, the operator's operations are grasped in chronological order, and it is determined each time whether or not the predetermined procedure described above matches the specified operation. If there is no difference from the predetermined procedure by the time the identified operation is completed, the operating procedure determination unit 24 determines that the predetermined operation has been performed according to the predetermined procedure. On the other hand, if the identified operation is different from the predetermined procedure, or if the identified operation deviates from the predetermined procedure, it is determined that the predetermined operation has not been performed according to the predetermined procedure. These determination results are output to the determination result output unit 25 .

As an example, a case where the predetermined operation is an operation of fitting two couplers will be described with reference to FIG. In FIG. 4, the coupler C1 held by the left hand of the worker and the coupler C2 held by the left hand of the worker are fitted together, and when the fitting portion Z of the coupler C1 and the coupler C2 is properly fitted, the operation is performed. Suppose that a sound S is generated. The predetermined procedure in this case is, for example, as shown in FIG. The couplers C1 and C2 are pushed into each other until the fitting portion Z is engaged and a clicking sound S is generated, and step 4) when the operating sound S is generated, the couplers C1 and C2 are released. be. The operation procedure determining unit 24 acquires the specified operation from the operation specifying unit 22 at predetermined time intervals, and grasps the positional relationship between the coupler and the hand of the operator who engages the coupler. Then, how the operator who engages the coupler operates the coupler is grasped in chronological order, and each time, whether or not the four steps described above match the specified operation is checked. judge.

If the coupler has been mated according to the steps described above, it is determined that the coupler has been properly mated according to the predetermined procedure. On the other hand, if the coupler was not properly fitted due to insufficient pushing of the coupler in step 3), it is determined that the coupler was not properly fitted. Here, the operation procedure determination unit 24 may use the detection result of the detection device 12 when determining whether or not the predetermined operation has been performed according to the predetermined procedure. In this case, for example, from the value of myoelectric potential obtained by an electromyograph attached to the upper arm of the operator and the value of acceleration obtained by an accelerometer attached to the forearm accelerometer of the operator Alternatively, the motion of the operator who engages the coupler may be detected and used for determination by the operation procedure determination unit 24 . In addition, a microphone installed near the operator who engages the coupler detects a click sound S that is generated when the engagement portion Z of the coupler is engaged, and the operation procedure determination unit 24 uses it for determination. may

As another example, a case where the predetermined operation is an operation to replenish engine oil will be described. In this case, the predetermined procedure consists of 1) removing the oil filler cap, 2) pouring engine oil from the oil inlet, 3) closing the oil filler cap, and 4) checking the amount of engine oil with the oil level gauge. It is a procedure consisting of steps. The operation procedure determining unit 24 acquires the specified operation from the operation specifying unit 22 at predetermined time intervals, and grasps the positional relationship between the hand of the operator who replenishes the oil, the engine oil, and the oil level gauge. Then, the operations performed by the operator who refills the oil are grasped in chronological order, and each time, it is determined whether or not the four steps described above match the specified operations. do.

If the oil has been replenished according to the steps described above, it is determined that the engine oil has been properly replenished according to the predetermined procedure. On the other hand, if the amount of engine oil is not confirmed in step 4), it is determined that the engine oil has not been properly replenished. Further, similarly to the above example, the operation procedure determination unit 24 may use the detection result of the detection device 12 when determining whether or not the predetermined operation has been performed according to the predetermined procedure. In this case, for example, from the value of myoelectric potential obtained by an electromyograph attached to the upper arm of the operator and the value of acceleration obtained by an accelerometer attached to the forearm accelerometer of the operator Alternatively, a motion of gripping the oil level gauge may be detected and used for determination by the operation procedure determination unit 24 .

Furthermore, the operation procedure determination unit 24 uses the time-series image data acquired by the image acquisition unit 21 and/or the detection result of the detection device 12 that detects the surrounding state of the operator to determine whether the predetermined operation is a predetermined operation. It may be determined whether or not the predetermined operation has been performed according to a predetermined procedure using a second trained model that has been trained to determine whether or not the operation has been performed according to a predetermined sequence. In the second trained model of the present embodiment, time-series image data acquired by the image acquisition unit 21 and/or detection results of the detection device 12 that detects the surrounding conditions of the operator are input as input data. Then, based on the input data, an operation that the operator is about to perform in the image is specified, and output data including a judgment result as to whether or not the operator has performed the specified operation in accordance with a predetermined procedure is generated. The computer (especially the CPU 151 of the processor 15) is operated to output. By using such a second trained model, it is possible to determine whether an operation has been performed according to a predetermined procedure even for image data indicating a positional relationship for which a correspondence relationship has not been set in advance.

Although the second trained model is not particularly limited, it is, for example, the second neural network 4 as shown in FIG. The second neural network 4 comprises a second input layer 41, a second intermediate layer 42 and a second output layer 43, each layer containing at least one neuron. Second input data 44 including at least one of the time-series image data acquired by the image acquisition unit 21 and the detection result of the detection device 12 is input to the second input layer 41, and the input data is output to the second intermediate layer 42 . The second intermediate layer 42 extracts the positional relationship between the operator's hand and the operation target from the data input from the second input layer 41 . Next, an operation corresponding to the extracted positional relationship is specified. Then, it is determined whether or not the specified operation corresponds to the predetermined operation, and if the specified operation corresponds to the predetermined operation, it is determined whether or not the predetermined operation has been performed according to the predetermined procedure. The second output layer 43 outputs the data input from the second intermediate layer 42 as second output data 45 including details of determination as to whether or not a predetermined operation has been performed according to a predetermined procedure.

In the second intermediate layer 42, similarly to the first intermediate layer 32 described above, the positional relationship between the operator's hand and the operation target and the positional relationship between the operator's hand and the operation target for specifying the operation that the operator is about to perform and the Machine learning has established a correspondence relationship with the operation of . In addition to the positional relationship between the operator's hand and the operation target, the corresponding relationship is associated with a plurality of factors such as the operator's surrounding conditions detected by the detection device 12, and the input data The parameters are set so that the appropriate operation is output for In addition to this, in the second intermediate layer 42, when the identified operation corresponds to the predetermined operation, it is determined whether the predetermined operation was performed according to the predetermined procedure. relationship is established. Specifically, machine learning learns the correspondence between the data in which the identified operations are arranged in chronological order and each step of the predetermined procedure, and the identified operations and the predetermined procedure can be appropriately compared. Required parameters are set.

Further, the correspondence relationship of the second intermediate layer 42 described above may be learned in advance by machine learning, but the second input data 44 input to the second neural network 4 in the past and the second neural network 4 in the past A new learning may be performed using the second teacher data 46 including the second output data 45 output from the model, or a learned model previously learned by machine learning may be further learned. The learning is performed by the second machine learning section 24a included in the operating procedure determination section 24. FIG. The second teacher data 46 is stored in a database such as the database 16 shown in FIG. 1, for example, and can be obtained as needed. By learning using the combination of the past second input data 44 and the second output data 45, the operation procedure determination unit 24 can more accurately determine whether the predetermined operation has been performed according to the predetermined procedure.

Returning to FIG. 1, the determination result output unit 25 has a function of acquiring determination results from the operation determination unit 23 and the operation procedure determination unit 24 and outputting them to an external device. When it is determined that the operation performed by the user is not the predetermined operation, it outputs to the display device 13 that the operation is inappropriate. Further, even when the operation procedure determination unit 24 determines that the predetermined operation has not been performed according to the predetermined procedure, it outputs to the display device 13 that the operation is not appropriate. By outputting these determination results to the display device 13 and displaying them on the display device 13, it is possible to notify the operator and/or the supervisor that the predetermined operation has not been properly executed.

Note that the power of the monitoring device 14 may be turned on after confirming that the operator has started using the monitoring system 1 based on the detection results of the detection device 12 . Also, the determination results of the operation determination unit 23 and the operation procedure determination unit 24 may be output to a server (not shown).

[Processing in operation detection system]
With reference to FIG. 6, the procedure for processing information by the monitoring device 14 will be described. FIG. 6 is an example of a flowchart showing information processing in the monitoring system 1 of this embodiment. The processing described below is executed by the processor 15 of the monitoring device 14 at predetermined time intervals.

First, in step S<b>1 , the function of the image acquisition unit 21 acquires from the imaging device 11 an image including part or all of an operator's hand performing an operation and an operation target operated by the operator. For example, when a worker engaged in assembly work in a factory uses a tool to tighten a bolt, an image including a portion of the worker's hand, the bolt, and the tool is acquired.

In subsequent step S2, the detection device 12 detects the state of the surroundings of the operator by the function of the operation specifying unit 22. FIG. For example, when a worker engaged in assembly work in a factory uses a tool to tighten a bolt, an electromyograph attached to the worker's arm acquires the value of the worker's myoelectric potential.

In subsequent step S<b>3 , the operation that the operator is about to perform is specified from the image data acquired by the image acquiring unit 21 and the detection result of the detecting device 12 by the function of the operation specifying unit 22 . For example, when a worker engaged in assembly work in a factory tightens a bolt using a tool, it is specified that the worker is about to tighten the bolt and which bolt the worker is about to tighten.

In subsequent step S4, the function of the operation determination unit 23 determines whether the specified operation corresponds to the predetermined operation. If the specified operation corresponds to the predetermined operation, the process proceeds to step S5. On the other hand, if the specified operation does not correspond to the predetermined operation, the process proceeds to step S6. For example, when a worker engaged in assembly work in a factory tightens a bolt using a tool, if the worker tightening the bolt is the predetermined operation, the process proceeds to step S5. On the other hand, if the worker's work does not include bolt fastening, the process proceeds to step S6.

In step S5, the function of the operation procedure determination unit 24 determines whether or not the predetermined operation has been performed according to the predetermined procedure. When it is determined that the predetermined operation has been performed in accordance with the predetermined procedure, execution of the routine is stopped and the monitoring process is terminated. On the other hand, when it is determined that the predetermined operation has not been performed according to the predetermined procedure, the process proceeds to step S6. For example, when a worker engaged in assembly work in a factory tightens a bolt using a tool, if the worker tightens the bolt according to a predetermined procedure, execution of the routine is stopped and monitoring processing ends. On the other hand, if the worker left the bolt loose, the process proceeds to step S6.

In step S6, the function of the determination result output unit 25 outputs to the display device 13 that the identified operation was not the predetermined operation and that the predetermined operation was not properly executed according to the predetermined procedure. Then, the display device 13 receives the output of the determination result output unit 25 and notifies the operator and/or the supervisor that the operation was not properly executed. For example, when a worker engaged in assembly work in a factory uses a tool to tighten a bolt, the observer is notified that the worker did not properly tighten the bolt. After the display, the execution of the routine is stopped and the monitoring process ends.

As another example, when a worker engaging in assembly work in a factory connects two couplers, for example, in step S1, image data is captured by the imaging device 11 attached to the worker as a wearable terminal. is obtained, and in step S2, the function of the operation specifying unit 22 detects sounds around the worker. In the subsequent step S3, based on the image data acquired by the image acquisition unit 21 and the detection result of the detection device 12, it is specified that the operator's operation is coupling of the coupler. In subsequent step S4, the function of the operation determination unit 23 determines whether the specified operation corresponds to the predetermined operation. If the coupler fitting operation, which is the specified operation, is the predetermined operation, the process proceeds to step S5. On the other hand, if fitting of the coupler is not included in the worker's work, the process proceeds to step S6. In step S5, the function of the operation procedure determination unit 24 determines whether or not the predetermined operation has been performed according to the predetermined procedure. If it is determined that the coupling operation of the coupler has been performed according to the predetermined procedure, execution of the routine is stopped and the monitoring process is terminated. On the other hand, if it is determined that the coupling operation of the coupler has not been performed according to the predetermined procedure, the process proceeds to step S6. Then, in step S6, by the function of the determination result output unit 25, it is determined that the specified coupling operation of the coupler was not the predetermined operation, or that the coupling operation of the coupler was not properly executed according to the predetermined procedure. It is output to the display device 13 .

[Embodiment of the present invention]
As described above, according to the monitoring device 14 of the present embodiment, using the imaging device 11 attached to the arm of the operator who performs the operation, a part or all of the operator's hand H and the operator An image acquisition unit 21 that acquires an image including an operation target O to be operated, and an operation determination unit that uses the image acquired by the image acquisition unit 21 to determine whether the operation is a predetermined operation. 23 and a determination result output unit 25 that outputs the determination result of the operation determination unit 23. A monitoring device is provided. Accordingly, an image in which the operator operates the operation target O can be reliably acquired. In addition, it is possible to obtain an image suitable for determining the behavior of the operator and improve the accuracy of determining the behavior of the operator.

Further, according to the monitoring device 14 of the present embodiment, the image acquisition unit 21 acquires an image including part or all of the operator's palm and the operation target. This makes it possible to more reliably acquire an image in which the operator manipulates the manipulation target.

Further, according to the monitoring device 14 of the present embodiment, the imaging device 11 has the mounting portion for mounting on the palm side of the operator's upper arm A2. This makes it possible to more reliably acquire an image in which the operator manipulates the manipulation target.

Further, according to the monitoring device 14 of the present embodiment, the operation identification unit 22 is provided for identifying the operation from the image acquired by the image acquisition unit 21, and the operation determination unit 23 is the image acquisition unit From the series of images acquired in 21, it is determined whether or not the operation specified by the operation specifying unit 22 is the predetermined operation. This makes it possible to specify the operation more accurately.

Further, according to the monitoring device 14 of the present embodiment, the operation specifying unit 22 detects the time-series images acquired by the image acquiring unit 21 and the surrounding conditions of the operator. The operation is specified using the detection result of . This makes it possible to specify the operation more accurately.

Further, according to the monitoring device 14 of the present embodiment, the determination result output unit 25 displays the operation on the display device 13 when the operation determination unit 23 determines that the operation is not the predetermined operation. is not appropriate. Thereby, the operator and/or the supervisor can be notified that the operation is not proper.

Further, according to the monitoring device 14 of the present embodiment, the operation specifying unit 22 uses the first trained model learned to specify the operation from the image acquired by the image acquiring unit 21. Identify the operation. This makes it possible to specify an operation for a positional relationship that has not been set in advance.

Further, according to the monitoring device 14 of the present embodiment, the first trained model is obtained by detecting the image acquired by the image acquisition unit 21 and the detection device 12 that detects the surrounding conditions of the operator. A first neural network 3 in which first input data 34 containing at least one of the results is input to the first input layer 31 and first output data 35 containing the specified operation is output from the first output layer 33. The operation specifying unit 22 converts the first input data 34 input to the first neural network 3 in the past and the first output data 35 output from the first neural network 3 in the past into the first teacher data A first machine learning unit 22a for learning the first neural network 3 is provided as 36, and the first neural network 3 is learned by the first machine learning unit 22a. Accordingly, it is possible to more accurately specify an operation for a positional relationship that has not been set in advance.

Further, according to the monitoring device 14 of the present embodiment, when the operation identifying unit 22 determines that the operation is the predetermined operation, it is determined whether or not the predetermined operation has been performed in a predetermined procedure. An operation procedure determination unit 24 is provided, and the determination result output unit 25 outputs the determination result of the operation procedure determination unit 24 . This makes it possible to determine the appropriateness of the procedure for performing the operation.

Further, according to the monitoring device 14 of the present embodiment, the operation procedure determination unit 24 uses the time-series images acquired by the image acquisition unit 21 to determine whether the predetermined operation is performed in the predetermined procedure. determine whether or not Thereby, it is possible to more accurately determine whether or not the predetermined operation has been performed according to the predetermined procedure.

Further, according to the monitoring device 14 of the present embodiment, the determination result output unit 25 outputs the display device 13 when the operation procedure determination unit 24 determines that the predetermined operation has not been performed according to the predetermined procedure. , output that the operation is not appropriate. Thereby, the operator and/or the observer can be notified that the operation is not proper.

Further, according to the monitoring device 14 of the present embodiment, the operation procedure determination unit 24 is a detection device that detects the time-series images acquired by the image acquisition unit 21 and the surrounding conditions of the operator. 12, using a second trained model trained to determine whether the predetermined operation has been performed in the predetermined procedure, using the detection result of 12, whether the predetermined operation has been performed in the predetermined procedure. determine whether As a result, it is possible to determine the appropriateness of the operation and the procedure for the positional relationship that has not been set in advance.

Further, according to the monitoring device 14 of the present embodiment, the second trained model includes the image acquired by the image acquisition unit 21 and the detection of the detection device 12 that detects the surrounding conditions of the operator. When the second input data 44 including the result is input to the second input layer 41, the second output data 45 including the judgment result as to whether or not the predetermined operation has been performed according to the predetermined procedure is output from the second output layer 43. The operation procedure determination unit 24 includes second input data 44 that has been input to the second neural network 4 in the past, and second output that has been output from the second neural network 4 in the past A second machine learning section 24a for learning the second neural network 4 using the data 45 as second teacher data 46 is provided, and the second neural network 4 is learned by the second machine learning section 24a. This makes it possible to more accurately determine the appropriateness of operations and procedures for positional relationships that have not been set in advance.

Further, according to the monitoring device 14 of the present embodiment, the operation determination unit 23 acquires the motion of the hand H of the operator from the image acquired by the image acquisition unit 21, and the acquired motion determines whether the movement corresponds to the predetermined movement, and if it is determined that the movement corresponds to the predetermined movement, a predetermined time before the time when the movement of the hand H is determined to be the predetermined movement. , it is determined whether or not the operation is the predetermined operation only during a predetermined period of time after the movement of the hand H is determined to be the predetermined movement. Thereby, the power consumption of the monitoring device 14 can be suppressed.

Further, according to the monitoring device 14 of the present embodiment, when it is determined that the motion does not correspond to the predetermined motion, the image acquisition unit 21 reduces the frame rate of the time-series images to be acquired. As a result, the capacity for storing image data can be reduced.

Further, according to the monitoring method of the present embodiment, in the monitoring method of monitoring an operator who performs an operation using a processor, the processor monitors the image of the operator from the imaging device 11 attached to the operator's arm. acquiring an image including part or all of the hand H and an operation target O to be operated by the operator, and using the acquired image, determining whether the operation is a predetermined operation; A monitoring method for outputting a determination result as to whether or not the operation is the predetermined operation is provided. Accordingly, an image in which the operator operates the operation target O can be reliably acquired. In addition, it is possible to obtain an image suitable for determining the behavior of the operator and improve the accuracy of determining the behavior of the operator.

DESCRIPTION OF SYMBOLS 1... Surveillance system 11... Imaging device 11a... Mounting part 12... Detecting device 13... Display device 14... Monitoring device 15... Processor 151... CPU
152 ROM
153 RAM
DESCRIPTION OF SYMBOLS 16... Database 2... Operation monitoring part 21... Image acquisition part 22... Operation specification part 22a... First machine learning part 23... Operation determination part 24... Operation procedure determination part 24a... Second machine learning part 25... Judgment result output part 3 First Neural Network 31 First Input Layer 32 First Intermediate Layer 33 First Output Layer 34 First Input Data 35 First Output Data 36 First Teacher Data 4 Second Neural Network 41 Second 2 input layer 42 second intermediate layer 43 second output layer 44 second input data 45 second output data 46 second teacher data A1 forearm A2 upper arm C1, C2 coupler H hand O … Operation object S … Operation sound VA … Viewing angle Z … Fitting part

Claims (16)

an image acquisition unit that acquires an image including part or all of the operator's hand and an operation target operated by the operator using an imaging device attached to the arm of the operator who performs the operation;
an operation determination unit that determines whether the operation is a predetermined operation using the image acquired by the image acquisition unit;
and a determination result output unit that outputs a determination result of the operation determination unit. 2. The monitoring apparatus according to claim 1, wherein said image acquisition unit acquires an image including part or all of said operator's palm and said operation target. 3. The monitoring device according to claim 1, wherein said imaging device has a mounting portion for mounting on the palm side of said operator's upper arm. an operation identification unit that identifies the operation from the image acquired by the image acquisition unit;
4. The operation determining unit determines whether the operation specified by the operation specifying unit is the predetermined operation from the series of images acquired by the image acquiring unit. A monitoring device according to any one of Claims 1 to 3. 5. The operation identifying unit identifies the operation using the time-series images obtained by the image obtaining unit and detection results of a detection device that detects a surrounding state of the operator. A monitoring device as described. 6. The determination result output unit according to claim 4, wherein when the operation determination unit determines that the operation is not the predetermined operation, the determination result output unit outputs to a display device that the operation is inappropriate. monitoring equipment. 7. The operation identifying unit identifies the operation using a first trained model trained to identify the operation from the image acquired by the image acquiring unit. A monitoring device as described in Clause. The first trained model receives first input data including at least one of the image acquired by the image acquisition unit and a detection result of a detection device that detects a surrounding state of the operator. A first neural network that, when input to an input layer, outputs first output data including the identified operation from a first output layer;
The operation specifying unit learns the first neural network using first input data input to the first neural network in the past and first output data output from the first neural network in the past as first teacher data. 8. The monitoring device according to claim 7, further comprising a first machine learning unit that allows the first machine learning unit to learn the first neural network. an operation procedure determining unit that determines whether or not the predetermined operation has been performed in accordance with a predetermined procedure when the operation identifying unit determines that the operation is the predetermined operation;
The monitoring device according to any one of claims 4 to 8, wherein said determination result output unit outputs a determination result of said operation procedure determination unit. 10. The monitoring apparatus according to claim 9, wherein said operation procedure determination unit determines whether said predetermined operation has been performed according to said predetermined procedure, using said time-series images acquired by said image acquisition unit. . 10. When the operation procedure determination unit determines that the predetermined operation has not been performed according to the predetermined procedure, the determination result output unit outputs to a display device that the operation is inappropriate. Or the monitoring device according to 10. The operation procedure determination unit uses the time-series images acquired by the image acquisition unit and the detection result of a detection device that detects the surrounding state of the operator to determine whether the predetermined operation is performed in the predetermined procedure. Determining whether the predetermined operation has been performed in the predetermined procedure using a second learned model that has been learned to determine whether it has been performed, according to any one of claims 9 to 11 A monitoring device as described. When the second input data including the image acquired by the image acquisition unit and the detection result of a detection device that detects the surrounding state of the operator is input to the second input layer, the second trained model inputs: a second neural network for outputting from a second output layer second output data including a determination result as to whether the predetermined operation has been performed according to the predetermined procedure;
The operation procedure determination unit determines the second neural network using second input data that has been input to the second neural network in the past and second output data that has been output from the second neural network in the past as second teacher data. 13. The monitoring device according to claim 12, further comprising a second machine learning unit for learning, wherein the second machine learning unit learns the second neural network. The operation determination unit is
Acquiring the movement of the hand of the operator from the image acquired by the image acquisition unit;
Determining whether the acquired movement corresponds to a predetermined movement,
When it is determined that the motion corresponds to the predetermined motion, the motion of the hand is determined to be the predetermined motion from a predetermined time before the time when the motion of the hand is determined to be the predetermined motion. 14. The monitoring device according to any one of claims 1 to 13, wherein it is determined whether or not the operation is the predetermined operation only during a period from the time when the determination is made until the time after the predetermined time. 15. The monitoring apparatus according to claim 14, wherein when it is determined that the motion does not correspond to the predetermined motion, the image acquisition unit reduces the frame rate of the time-series images to be acquired. In a monitoring method for monitoring an operator performing an operation using a processor,
The processor
acquiring an image including part or all of the operator's hand and an operation target operated by the operator from an imaging device attached to the operator's arm;
Determining whether the operation is a predetermined operation using the acquired image,
A monitoring method of outputting a determination result as to whether or not the operation is the predetermined operation.

Images