JP7483179B1 - Estimation device, learning device, estimation method, and estimation program - Google Patents

Abstract

The time division unit (11) divides the work engagement time into a work performance time zone during which the worker is performing any one of the multiple elemental tasks and a non-work time zone during which the worker is not performing any one of the multiple elemental tasks, using at least one of the results of a displacement amount determination that determines the amount of displacement of the worker's hand during work engagement time, which is the time during which the worker is engaged in work including multiple elemental tasks, each of which has non-uniform attributes, a tool gripping state determination that determines the gripping state of the tool in the worker's hand during the work engagement time, and an appearance state determination that determines the appearance state of the worker's hand in an image captured during the work engagement time.The estimation unit (12) estimates the elemental task being performed by the worker during the work performance time zone, based on a partial image that is a portion of the image captured during the work performance time zone.

Description

The present disclosure relates to a technique for estimating an elemental work being performed by a worker.
An elemental task is an action that can be recognized as a single unit, and is a component of a task. In other words, a task is composed of a combination of multiple elemental tasks. For example, the elemental task "Placing the cover in the installation position" and the elemental task "Screwing the cover in place" are combined to form one task, "Attach the cover."

Japanese Patent Laid-Open No. 2003-233693 discloses a technique for capturing an image of a person's behavior and estimating the person's behavior from the captured image.
In the technology of Patent Document 1, time-series position data of human body parts obtained from a video is classified into multiple position data. Furthermore, in the technology of Patent Document 1, each position data is analyzed, and a motion sequence (movement, change, rest, etc.) is generated from the analysis result. Then, in the technology of Patent Document 1, a neural network that handles time-series data, such as an RNN (Recurrent Neural Network), analyzes the motion sequence. Furthermore, in the technology of Patent Document 1, a memory neural network processes the analysis result.

JP 2021-22323 A

An operation includes element tasks with various attributes, such as element tasks that take a short time to perform, element tasks that take a long time to perform, element tasks that occur infrequently, and element tasks that occur frequently.
When the technology of Patent Document 1 is applied to a task that includes element tasks with such various attributes, the following problems arise.

Issue (1)
In the technology of Patent Document 1, position data, which is time-series data divided into short intervals, is analyzed by a neural network that handles time-series data.
When a neural network is trained on time-series data separated by short intervals for an operation that includes multiple elemental operations with various execution times and/or occurrence frequencies, if the amount of time-series data for each category is large, it becomes difficult to train the network to generate a model for estimating the elemental operations.
A category is a label that indicates the type of element work that a worker is performing and a label that indicates a time period during which the worker is not performing any element work.
Issue (2)
Furthermore, even if a new category of elemental work is created in order to improve the balance of the amount of time-series data between categories, additional data collection is required for the new category.
Issue (3)
In addition, even if the time series data of an element task that takes a long time to perform is divided into multiple time series data for short periods of time, each time series data only represents a part of the element task. This increases the variability of the time series data even within the same category, making it difficult to learn to generate a model that estimates the element task.
Issue (4)
Due to the above problems (1) to (3), when a task includes multiple elemental tasks whose attributes are not uniform, such as having different execution times and/or occurrence frequencies, it is not possible to accurately estimate the elemental tasks performed by a worker.

One of the main objectives of the present disclosure is to solve the above-mentioned problems. More specifically, the main objective of the present disclosure is to enable accurate estimation of elemental tasks performed by a worker even when the task includes multiple elemental tasks, each of which has non-uniform attributes.

The estimation device according to the present disclosure comprises:
a time division unit that divides a work engagement time into a work performance time zone during which the worker is performing any one of the plurality of elemental tasks and a non-work time zone during which the worker is not performing any one of the plurality of elemental tasks, using at least one of a result of a displacement amount determination that determines an amount of displacement of the hand of a worker during a work engagement time that is a time during which the worker is engaged in a task including a plurality of elemental tasks, each of which has non-uniform attributes, while the worker is engaged in the task, a result of a tool gripping state determination that determines a gripping state of a tool in the hand of the worker during the work engagement time, and a result of an appearance state determination that determines an appearance state of the hand of the worker in an image captured during the work engagement time;
The system further includes an estimation unit that estimates an elemental work being performed by the worker during the work performance time period based on a partial image that is a portion of the image captured during the work performance time period.

According to the present disclosure, it is possible to accurately estimate the component tasks performed by a worker even when the task includes multiple component tasks, each of which has non-uniform attributes.

FIG. 2 is a diagram showing an overview of an example of a functional configuration of an estimation device according to the first embodiment. 6 is a diagram showing an example of a time division process using only the determination result of the displacement amount determination according to the first embodiment; 11A to 11C are diagrams showing an example of a time division process using only the determination result of the tool gripping state determination in the first embodiment. 11 is a diagram showing an example of a time division process using only the determination result of the appearance situation determination in the first embodiment. FIG. 11A and 11B are diagrams showing an example of time division processing using three determination results according to the first embodiment; 11A to 11C are diagrams showing examples of tool movements during non-work time periods and work execution time periods in the first embodiment. FIG. 2 is a diagram showing an example of a detailed functional configuration of the estimation device according to the first embodiment. 4 is a diagram showing an example of the internal configuration of a task execution time zone detection unit and an element task estimation unit according to the first embodiment; FIG. 4 is a flowchart showing an example of the operation of the estimation device according to the first embodiment. 10 is a flowchart showing an example of the operation of an operation execution time period detection unit according to the first embodiment. 10 is a flowchart showing an example of the operation of an element work estimation unit according to the first embodiment. FIG. 13 is a diagram showing an example of the functional configuration of a learning device according to a second embodiment. FIG. 13 is a diagram showing an example of the internal configuration of an element work estimation model generation unit according to the second embodiment. 13 is a flowchart showing an example of the operation of the learning device according to the second embodiment. FIG. 13 is a diagram showing an example of the functional configuration of a learning device according to a third embodiment. FIG. 13 is a diagram showing an example of the internal configuration of an operation execution time period detection model generation unit according to the third embodiment. 13 is a flowchart showing an example of the operation of the learning device according to the third embodiment. FIG. 13 is a diagram showing an example of a functional configuration of an estimation device according to a fourth embodiment. FIG. 13 is a diagram showing an example of the functional configuration of a learning device according to a fifth embodiment. 1 is a diagram showing an overview of imaging processing according to the first embodiment; 4 is a diagram showing an overview of an operation execution time period detection process, an element operation estimation process, and an estimation result process according to the first embodiment. FIG. FIG. 1 is a diagram showing an example of a hardware configuration of an estimation device according to first and fourth embodiments. FIG. 13 is a diagram showing an example of the hardware configuration of a learning device according to the second, third and fifth embodiments.

The following describes the embodiments with reference to the drawings. In the following description of the embodiments and in the drawings, the same reference numerals indicate the same or corresponding parts.

Embodiment 1.
This disclosure describes a method for accurately estimating the task elements being performed by workers.
An "operator" is a person (worker) or robot (working robot) engaged in work. Additionally, an "operator's hand" includes a human (worker's) hand and a component part of a robot (working robot) that has the same function and/or role as a human hand. Additionally, an "operator's hand" includes at least one of the palm, back of the hand, wrist, and fingers of a human (worker) and a component part of a robot (working robot) that corresponds to any one of the palm, back of the hand, wrist, and fingers.
In the following, an example will be described in which a person (worker) is engaged in machine adjustment work.

***Configuration Description***
1 shows an overview of an example of a functional configuration of an estimation device 100 according to the present embodiment. The estimation device 100 operates in an estimation phase.
The estimation device 100 is connected to an imaging device 110 .
The image capturing device 110 captures an image of the worker's hands during work engagement time. Work engagement time is the time during which the worker is engaged in work.
The imaging device 110 outputs an image of the worker's hands obtained by imaging to the estimation device 100.
The imaging device 110 is, for example, worn on the head of a worker and captures a first-person perspective image. In this embodiment, the imaging device 110 is assumed to be worn on the head of the worker. However, the imaging device 110 does not have to be worn on the head of the worker as long as it can capture an image of the worker's hands.
Furthermore, the image capturing device 110 may not only be a device that captures normal color images, but may also include a sensor having another modality, such as a depth sensor.
A depth sensor is capable of capturing images at a wider angle. When the imaging device 110 includes a depth sensor, even if the hand is not captured in the color image, the estimation device 100 may determine that the hand is present as long as the hand is present within the imaging range of the depth sensor.

**Description of a Simplified Configuration Example of the Estimation Device 100**
The estimation device 100 includes a time division unit 11 and an estimation unit 12 .
As described below, the estimation device 100 has a detailed functional configuration shown in FIGS. 7 and 8. However, in FIG. 1, the functional configuration of the estimation device 100 is shown in a simplified manner for ease of understanding.
Before describing the detailed configuration of the estimation device 100 shown in FIGS. 7 and 8, the functional configuration of the simplified estimation device 100 shown in FIG. 1 will be described.

**Explanation of time division unit 11**
The time division unit 11 divides the work time into a work execution time period and a non-work time period using at least one of the results of the displacement amount determination, the results of the tool holding status determination, and the results of the appearance status determination.
The displacement amount determination is a determination process for determining the amount of displacement of the worker's hand.
The tool gripping state determination is a process for determining the state in which a tool is gripped by the worker's hand. The tool is a tool used in machine adjustment work.
The appearance status determination is a determination process for determining the appearance status of the worker's hands in the video captured during the work engagement time.
Further, a work performance time period is a time period during which a worker performs one of a plurality of elemental works.
Moreover, a non-working time period is a time period during which the worker is not performing any of the elemental works among a plurality of elemental works.
The process performed by the time division unit 11 corresponds to a time division process.

**Description of Estimation Unit 12**
The estimation unit 12 estimates the elemental work being performed by the worker during the work performance time period based on a partial image that is a portion of the image captured during the work performance time period that is captured during the work performance time period.
Specifically, the estimation unit 12 estimates the element work performed by the worker during the work performance time period using a trained model generated by training.

**Explanation of time division process**
2 to 5 show examples of the time division process by the time division unit 11. FIG.
FIG. 2 shows an example in which the time dividing unit 11 divides the work engagement time into a work execution time period and a non-work time period using only the result of the displacement amount determination.
FIG. 3 shows an example in which the time dividing unit 11 divides the work engagement time into a work execution time period and a non-work time period using only the result of the tool gripping state determination.
FIG. 4 shows an example in which the time dividing unit 11 divides the work engagement time into a work performance time period and a non-work time period using only the determination result of the appearance status determination.
FIG. 5 shows an example in which the time division unit 11 divides work time into a work performance time period and a non-work time period using the results of the displacement amount determination, the results of the tool holding status determination, and the results of the appearance status determination.

The result of the displacement amount determination in Fig. 2 indicates the displacement amount of the worker's hand during the work engagement time. In Fig. 2, the result of the displacement amount determination is shown in a graph.
The time division unit 11 acquires time series data of the joints of the hand derived from an image of the worker's hand captured by the imaging device 110. Then, the amount of displacement of the hand per unit time is determined from the time series data of the joints of the hand.
Although not shown in FIG. 1, the estimation device 100 is assumed to include a mechanism for converting an image of the worker's hand into time-series data of the hand joints.
The time division unit 11 designates a time period during which the amount of hand displacement per unit time is less than the threshold as a task execution time period, while the time division unit 11 designates a time period during which the amount of hand displacement per unit time is equal to or greater than the threshold as a non-task time period.
As shown in (a) of Fig. 6, in machine adjustment work, when no work is being performed, the hand holding the tool is thought to move quickly toward (or away from) the work target (this is also thought to be the case for a hand not holding a tool). For this reason, the time period when the amount of hand displacement is large is thought to be a non-work time period. On the other hand, as shown in (b) of Fig. 6, when work is being performed, the tool is thought to be moved slowly or the range of movement of the tool is small (this is also thought to be the case for a hand not holding a tool). For this reason, the time period when the amount of hand displacement is small is thought to be a work time period.

The result of the tool holding state determination in FIG. 3 indicates whether or not a tool is being held.
Although not shown in FIG. 1, the estimation device 100 is assumed to include a mechanism for identifying the tool held by the worker's hand.
In the example of Figure 3, first the worker was holding a wrench. Next, the worker was holding a screwdriver. After that, the worker was not holding any tools. Finally, the worker was holding a steel ruler.
The time division unit 11 designates a time period during which the worker's hands are holding a tool as a work performance time period, while the time division unit 11 designates a time period during which the worker's hands are not holding any tool as a non-work time period.
In addition, when the type of tool held by the worker's hand changes over time, the time division unit 11 designates each time period during which the worker's hand holds a different type of tool as a different work performance time period. In the example of Fig. 3, the time division unit 11 designates a time period during which the worker's hand holds a wrench and a time period during which the worker's hand holds a screwdriver as different work performance time periods.

The determination result of the appearance status determination in FIG. 4 indicates whether or not the worker's hands are appearing in the image captured by the imaging device 110.
The time division unit 11 can determine whether or not the worker's hand is present by analyzing the image captured by the imaging device 110 or the time-series data of the hand joints.
In the example of Figure 4, first the worker's hand appears in the image. Then the worker's hand disappears from the image. After that, the worker's hand appears in the image again.
The time division unit 11 designates a time period during which the worker's hands appear in the video as a work performance time period, while the time division unit 11 designates a time period during which the worker's hands do not appear in the video as a non-work time period.

FIG. 5 shows an example in which the time dividing unit 11 divides the work engagement time into a work execution time period and a non-work time period using three determination results.
In the example of Figure 5, if the displacement of the worker's hand is equal to or greater than a threshold value, the time division unit 11 designates that time period as a non-work time period, regardless of whether the worker's hand is holding a tool or not and regardless of whether the worker's hand appears in the image or not.
Furthermore, when the amount of displacement of the worker's hand is less than the threshold and the worker's hand appears in the video, the time division unit 11 designates the time period as a work performance time period regardless of whether the worker's hand is holding a tool or not. On the other hand, even when the amount of displacement of the worker's hand is less than the threshold, if the worker's hand does not appear in the video, the time division unit 11 designates the time period as a non-work time period regardless of whether the worker's hand is holding a tool or not.
In this way, when two or more of the judgment results of the displacement amount judgment, the judgment result of the tool holding status judgment, and the judgment result of the appearance status judgment are used, the user of the estimation device 100 defines in advance the application criteria for determining the priority order in which the two or more judgment results are to be applied.

The estimation unit 12 applies the time series data of the hand joints derived from the partial video captured during the work performance time period to the learned model, and estimates the component work performed by the worker during the work performance time period.
The trained model used by the estimation unit 12 is generated by a training device 200 described later.

In this way, the estimation device 100 according to the present embodiment performs a two-stage process of detecting a task execution time period by the time dividing unit 11 and estimating an element task using a partial video by the estimation unit 12.
Therefore, the estimating device 100 according to the present embodiment can accurately estimate the work elements performed by the workers.

**Explanation of an Example of the Hardware Configuration of the Estimation Apparatus 100**
The estimation device 100 according to this embodiment is a computer having a hardware configuration exemplified in FIG.
As shown in FIG. 22, the estimation device 100 includes, as hardware, a processor 801, a main memory device 802, an auxiliary memory device 803, and a communication device 804.
The functions of the time division unit 11 and the estimation unit 12 shown in FIG. 1, and further the components shown in FIGS. 7 and 8, are realized by, for example, a program.
The auxiliary storage device 803 stores programs that realize the functions of these components.
These programs are loaded from the auxiliary storage device 803 to the main storage device 802. Then, the processor 801 executes these programs to perform the operations of these components.

The operating procedure of the estimation device 100 corresponds to an estimation method. Furthermore, the program that realizes the operation of the estimation device 100 corresponds to an estimation program.

**Description of a detailed configuration example of the estimation device 100**
FIG. 7 shows an example of a detailed functional configuration of the estimation device 100 shown in FIG.
In FIG. 7 , the estimation device 100 includes a joint position time series data acquisition unit 120, a joint velocity calculation unit 121, a joint time series data imaging unit 122, an task performance time zone detection unit 123, an element work estimation unit 124, an estimation result processing unit 125, an element work estimation model storage unit 130, a pre-processing statistics storage unit 131, and an estimation result storage unit 132.
The task execution time zone detection unit 123 corresponds to the time division unit 11 shown in Fig. 1. The element task estimation unit 124 corresponds to the estimation unit 12 shown in Fig. 1.

**Description of joint position time-series data acquisition unit 120**
The joint position time-series data acquisition unit 120 acquires an image V obtained by capturing an image of the worker's hand from the imaging device 110. The image V is an image captured during the work engagement time in the estimation phase. The image V includes frames in which the worker's hand is captured and frames in which the worker's hand is not captured.
The joint position time series data acquisition unit 120 generates joint position time series data HPT from the video V. The joint position time series data HPT is data indicating the spatial position coordinates of each joint of the hand.
The joint position time-series data acquisition unit 120 generates the joint position time-series data HPT, for example, by using an existing hand tracking model that estimates the positions of the hand joints.
The joint position time series data acquisition unit 120 generates the right hand joint position time series data HPT and the left hand joint position time series data HPT. Note that, hereinafter, when simply referring to the joint position time series data HPT, it refers to both the right hand joint position time series data HPT and the left hand joint position time series data HPT.
When there is a period of time in which no hands appear in the video V, the joint position time series data acquisition unit 120 inserts a value that enables determination that no hands appear into the joint position time series data HPT.
Then, the joint position time series data acquiring unit 120 outputs the joint position time series data HPT to the joint velocity calculating unit 121 and the joint time series data imaging unit 122. In addition, the joint position time series data acquiring unit 120 may output the joint position time series data HPT to the task execution time zone detecting unit 123.

**Explanation of the joint velocity calculation unit 121**
The joint velocity calculation unit 121 acquires the joint position time series data HPT from the joint position time series data acquisition unit 120 .
Then, the joint velocity calculation unit 121 calculates the difference (velocity) in the time direction for each joint of the joint position time series data HPT. Then, the joint velocity calculation unit 121 outputs the joint velocity time series data HVT indicating the calculation result to the joint time series data imaging unit 122. The joint velocity calculation unit 121 may output the joint velocity time series data HVT to the task execution time zone detection unit 123.
The joint velocity calculation unit 121 generates the joint velocity time series data HVT of the right hand and the joint velocity time series data HVT of the left hand. Note that, hereinafter, when simply referring to the joint velocity time series data HVT, it refers to both the joint velocity time series data HVT of the right hand and the joint velocity time series data HVT of the left hand.

**Explanation of the joint time-series data imaging unit 122**
The joint time series data imaging unit 122 acquires the joint position time series data HPT from the joint position time series data acquisition unit 120. The joint time series data imaging unit 122 also acquires the joint velocity time series data HVT from the joint velocity calculation unit 121.
The joint time-series data imaging unit 122 images the joint position time-series data HPT to generate a left wrist joint position image LJPI and a right wrist joint position image RJPI. The joint time-series data imaging unit 122 also images the joint velocity time-series data HVT to generate a left wrist joint velocity image LJVI and a right wrist joint velocity image RJVI.
Then, the joint time-series data visualization unit 122 outputs the left hand joint position image LJPI, the right hand joint position image RJPI, the left hand joint velocity image LJVI, and the right hand joint velocity image RJVI to the task execution time zone detection unit 123.

Here, imaging of the joint position time series data HPT and imaging of the joint velocity time series data HVT by the joint time series data imaging unit 122 will be described.
20 shows an overview of the imaging process of the joint time-series data imaging unit 122. Here, the imaging process of the joint position time-series data HPT will be described. Imaging refers to generating time-series data of the three-dimensional coordinates of a joint by regarding the x, y, and z values of the three-dimensional coordinates as R, G, and B of an image.
20, the joint position time-series data HPT for each time t includes spatial position coordinates of J (J≧2) joints. The coordinate axes C are the x-axis, the y-axis, and the z-axis.

The joint time-series data imaging unit 122 generates a tensor having elements of each joint j and each axis coordinate value c of the J joints in the left hand joint position time-series data HPT, for example. The joint time-series data imaging unit 122 then combines the tensors in the time direction to generate a left hand joint position image LJPI.
The joint time-series data imaging unit 122 also generates the right wrist joint position image RJPI in a similar procedure.
Further, the joint time-series data imaging unit 122 generates a left hand joint velocity image LJVI from the left hand joint velocity time-series data HVT in a similar procedure.
Further, the joint time-series data imaging unit 122 generates a right hand joint velocity image RJVI from the right hand joint velocity time-series data HVT in a similar procedure.

**Description of the gripping tool time-series data generating unit 126**
The gripping tool time-series data generator 126 acquires the video V or the sensor information RRS.
The video V is an image captured by the imaging device 110. In other words, the gripping tool time-series data generation unit 126 may acquire the video V from the joint position time-series data acquisition unit 120, or may acquire the video V from the imaging device 110 independently of the joint position time-series data acquisition unit 120.
The sensor information RRS is information that indicates whether the worker is holding a tool or not, and if the worker is holding a tool, the sensor information RRS indicates the type of tool that the worker is holding.
For example, when a tag-type wireless communication sensor such as an RFID is attached to the hand of a worker, the tag-type wireless communication sensor receives a signal transmitted from the tool held by the worker. The tag-type wireless communication sensor then determines the type of tool held by the worker based on the received signal. Furthermore, when the tag-type wireless communication sensor does not receive a signal, it determines that the worker is not holding a tool. The tag-type wireless communication sensor outputs the determination result as sensor information RRS to the held tool time-series data generator 126.

The gripped tool time-series data generator 126 uses the video V or the sensor information RRS to identify whether the worker's hand is gripping a tool. If the worker's hand is gripping a tool, the gripped tool time-series data generator 126 identifies the type of tool the worker's hand is gripping.
When acquiring the video V, the held tool time-series data generation unit 126 identifies from each frame of the video V whether the worker is holding a tool and the type of tool, for example, by using a known object detection model.
When acquiring the sensor information RRS, the held tool time-series data generator 126 analyzes the sensor information RRS to identify whether or not the worker is holding a tool and the type of tool.

The gripped tool time-series data generating unit 126 outputs the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 indicating the identification results to the task execution time zone detecting unit 123.
The left hand held tool data LTO1 is data that indicates in chronological order whether or not the worker's left hand is holding a tool and the type of tool being held. In other words, by analyzing the left hand held tool data LTO1, it is possible to identify the time period when the worker's left hand is holding a tool and the time period when the worker's left hand is not holding a tool. Furthermore, by analyzing the left hand held tool data LTO1, it is possible to identify in chronological order the type of tool being held by the worker's left hand during the time period when the worker's left hand is holding a tool.
Similarly, the right hand held tool data RTO1 is data that indicates in a time series whether the worker's right hand is holding a tool and the type of tool being held. In other words, by analyzing the right hand held tool data RTO1, it is possible to identify in a time series the time period when the worker's right hand is holding a tool and the time period when the worker's right hand is not holding a tool. In addition, by analyzing the right hand held tool data RTO1, it is possible to identify the type of tool being held by the worker's right hand during the time period when the worker's right hand is holding a tool.

**Explanation of the work execution time zone detection unit 123**
The task execution time zone detection unit 123 acquires a left wrist joint position image LJPI, a left wrist joint velocity image LJVI, a right wrist joint position image RJPI, and a right wrist joint velocity image RJVI from the joint time-series data imaging unit 122.
Furthermore, the task execution time zone detection unit 123 acquires the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 from the gripped tool time series data generation unit 126.

The task performance time zone detection unit 123 analyzes the left hand joint position image LJPI and/or the left hand joint velocity image LJVI. Then, the task performance time zone detection unit 123 generates a left hand appearance time zone set LS, which is a set of time zones in which the left hand appears in the video.
Similarly, the task performance time zone detection unit 123 analyzes the right hand joint position image RJPI and/or the right hand joint velocity image RJVI. Then, the task performance time zone detection unit 123 generates a right hand appearance time zone set RS which is a set of time zones in which the right hand appears in the image.
In this manner, the operation execution time zone detection unit 123 performs the occurrence status determination shown in FIG.
Furthermore, when the task performance time zone detection unit 123 acquires the joint position time series data HPT from the joint position time series data acquisition unit 120, the task performance time zone detection unit 123 may determine the appearance status using the joint position time series data HPT. Furthermore, when the task performance time zone detection unit 123 acquires the joint velocity time series data HVT from the joint velocity calculation unit 121, the task performance time zone detection unit 123 may determine the appearance status using the joint velocity time series data HVT.

Furthermore, the work performance time zone detection unit 123 extracts, in chronological order, time zones during which a tool is being held and time zones during which a tool is not being held for each of the left and right hands based on the left hand held tool data LTO1 and the right hand held tool data RTO1. Furthermore, when the type of tool held by the worker is changing, the work performance time zone detection unit 123 extracts different time zones for each type of tool.
Hereinafter, the extraction result of the work execution time zone detection unit 123 is referred to as the tool gripping state determination result TS.
The tool holding status determination result TS indicates, in chronological order, the time periods when the worker is holding the tool and the time periods when the worker is not holding the tool. The tool holding status determination result TS also associates the time periods when the worker is holding the tool with the type of tool held by the worker in that time period.
In this manner, the task execution time zone detection unit 123 performs the tool holding state determination shown in FIG.

Furthermore, the task execution time zone detection unit 123 analyzes the left hand joint position image LJPI and/or the left hand joint velocity image LJVI to calculate the amount of displacement of the left hand per unit time.
Similarly, the task execution time zone detection unit 123 analyzes the right hand joint position image RJPI and/or the right hand joint velocity image RJVI to calculate the amount of displacement of the right hand per unit time.
In this manner, the work execution time zone detection unit 123 performs the displacement amount determination shown in FIG.
Furthermore, when the task execution time zone detection unit 123 acquires the joint position time series data HPT from the joint position time series data acquisition unit 120, the task execution time zone detection unit 123 may determine the amount of displacement using the joint position time series data HPT. When the task execution time zone detection unit 123 acquires the joint velocity time series data HVT from the joint velocity calculation unit 121, the task execution time zone detection unit 123 may determine the amount of displacement using the joint velocity time series data HVT.

The work performance time zone detection unit 123 divides the work performance time zone into a work performance time zone and a non-work time zone based on at least one of the determination results.
Specifically, the work performance time zone detection unit 123 identifies the start time and end time of the work performance time zone, and identifies the start time and end time of the non-work time zone, and divides the work performance time zone into a work performance time zone and a non-work time zone.
In addition, when the work performance time zone detection unit 123 divides the work engagement time into a work performance time zone and a non-work time zone using only the results of the appearance status determination, it does not need to perform the tool holding status determination and the displacement amount determination.
Similarly, when the work performance time zone detection unit 123 divides the work engagement time into a work performance time zone and a non-work time zone using only the results of the tool holding status determination, it does not need to perform the appearance status determination and the displacement amount determination.
Similarly, when the work performance time zone detection unit 123 divides the work engagement time into work performance time zones and non-work time zones using only the results of the displacement amount determination, it does not need to perform appearance status determination and tool grasping status determination.

The work implementation time zone detection unit 123 outputs a work implementation time zone set FS, which is a set of work implementation time zones, to the element work estimation unit 124. The work implementation time zone set FS indicates the start time and end time of the work implementation time zone for each work implementation time zone.
Furthermore, the work implementation time zone detection unit 123 outputs a non-work time zone set NFS, which is a set of non-work time zones, to the estimation result processing unit 125. The non-work time zone set NFS indicates the start time and end time of each non-work time zone.

**Explanation of the element work estimation unit 124**
The element work estimation unit 124 acquires the work performance time zone set FS from the work performance time zone detection unit 123 .
Furthermore, the element work estimation unit 124 acquires a left wrist joint position image LJPI, a left wrist joint velocity image LJVI, a right wrist joint position image RJPI, and a right wrist joint velocity image RJVI from the joint time-series data imaging unit 122 .
Furthermore, the element work estimation unit 124 acquires the element work estimation model M from the element work estimation model storage unit 130. The element work estimation model M corresponds to the trained model shown in FIG.
Furthermore, the element work estimation unit 124 acquires preprocessing statistics from the estimation result storage unit 132. Specifically, the element work estimation unit 124 acquires, as the preprocessing statistics, a joint position luminance average value PM, a joint position luminance standard deviation PS, a joint velocity luminance average value VM, and a joint velocity luminance standard deviation VS.

The joint position luminance average value PM is the average luminance value of the joint position image for learning obtained by the learning device 200 described later.
The joint position luminance standard deviation PS is the standard deviation of the luminance of the joint position images for training obtained by the training device 200.
The brightness is the coordinate value for each coordinate axis C included in the joint position image.
The learning joint position images are joint position images used in the learning phase, and correspond to the left hand joint position image LJPI and right hand joint position image RJPI described above.

Strictly speaking, the average joint position luminance value PM includes the average left wrist joint position luminance value LPM and the average right wrist joint position luminance value RPM. The average left wrist joint position luminance value LPM is the average value obtained from the left wrist joint position label image sLJPI, which is a learning left wrist joint position image. The average right wrist joint position luminance value RPM is the average value obtained from the right wrist joint position label image sRJPI, which is a learning right wrist joint position image.
Moreover, the left hand joint position luminance average value LPM is obtained for each coordinate axis C. Therefore, strictly speaking, the left hand joint position luminance average value LPM is expressed as the left hand joint position luminance average value LPM(C).
Similarly, the right hand joint position luminance average value RPM is also found for each coordinate axis C. Therefore, strictly speaking, the right hand joint position luminance average value RPM is expressed as the right hand joint position luminance average value RPM(C).
Unless otherwise specified strictly, the left wrist joint position luminance average value LPM and the right wrist joint position luminance average value RPM will be collectively referred to as the joint position luminance average value PM.
Note that, instead of using the average luminance value for the left hand and the average luminance value for the right hand as described above, the average luminance value for both the left hand and the right hand may be used. In the following, an example in which the average luminance value for the left hand and the average luminance value for the right hand are used will be described, but the following description also applies to the case in which the average luminance value for both the left hand and the right hand is used.

Strictly speaking, the joint position luminance standard deviation PS also includes a left hand joint position luminance standard deviation LPS and a right hand joint position luminance standard deviation RPS. The left hand joint position luminance standard deviation LPS is a standard deviation obtained from a left hand joint position label image sLJPI, which is a left hand joint position image for learning. The right hand joint position luminance standard deviation RPS is a standard deviation obtained from a right hand joint position label image sRJPI, which is a right hand joint position image for learning.
In addition, the left hand joint position luminance standard deviation LPS is also found for each coordinate axis C. Therefore, in a strict sense, the left hand joint position luminance standard deviation LPS is expressed as the left hand joint position luminance standard deviation LPS(C).
Similarly, the right hand joint position luminance standard deviation RPS is also found for each coordinate axis C. Therefore, in a strict sense, the right hand joint position luminance standard deviation RPS is expressed as the right hand joint position luminance standard deviation RPS(C).
Unless a strict notation is required, the left hand joint position luminance standard deviation LPS and the right hand joint position luminance standard deviation RPS will be collectively referred to as the joint position luminance standard deviation PS.
Note that, instead of using the luminance standard deviation for the left hand and the luminance standard deviation for the right hand as described above, the luminance standard deviation over both the left hand and the right hand may be used. In the following, an example in which the luminance standard deviation for the left hand and the luminance standard deviation for the right hand are used will be described, but the following description also applies to the case in which the luminance standard deviation over both the left hand and the right hand is used.

The joint velocity luminance average value VM is the average luminance value of the training joint velocity image obtained by the training device 200.
The joint velocity luminance standard deviation VS is the standard deviation of the luminance of the training joint velocity image obtained by the training device 200.
The learning joint velocity images are joint velocity images used in the learning phase, and correspond to the left wrist joint velocity image LJVI and right wrist joint velocity image RJVI described above.

Strictly speaking, the joint velocity brightness average value VM also includes a left wrist joint velocity brightness average value LVM and a right wrist joint velocity brightness average value RVM. The left wrist joint velocity brightness average value LVM is the average value obtained from the left wrist joint velocity label image sLJVI, which is a left wrist joint velocity image for learning. The right wrist joint velocity brightness average value RVM is the average value obtained from the right wrist joint velocity label image sRJVI, which is a right wrist joint velocity image for learning.
The left wrist joint velocity brightness average value LVM is also calculated for each coordinate axis C. Therefore, the left wrist joint velocity brightness average value LVM can be expressed strictly as the left wrist joint velocity brightness average value LVM(C).
Similarly, the right wrist joint velocity luminance average value RVM is also calculated for each coordinate axis C. Therefore, in a strict sense, the right wrist joint velocity luminance average value RVM is expressed as the right wrist joint velocity luminance average value RVM(C).
Unless a strict notation is required, the left wrist joint velocity brightness average value LVM and the right wrist joint velocity brightness average value RVM will be collectively referred to as the joint velocity brightness average value VM.
Note that, instead of using the average luminance value for the left hand and the average luminance value for the right hand as described above, the average luminance value for both the left hand and the right hand may be used. In the following, an example in which the average luminance value for the left hand and the average luminance value for the right hand are used will be described, but the following description also applies to the case in which the average luminance value for both the left hand and the right hand is used.

Strictly speaking, the joint velocity brightness standard deviation VS also includes a left hand joint velocity brightness standard deviation LVS and a right hand joint velocity brightness standard deviation RVS. The left hand joint velocity brightness standard deviation LVS is a standard deviation obtained from a left hand joint velocity label image sLJVI, which is a left hand joint velocity image for learning. The right hand joint velocity brightness standard deviation RVS is a standard deviation obtained from a right hand joint velocity label image sRJVI, which is a right hand joint velocity image for learning.
In addition, the left wrist joint velocity luminance standard deviation LVS is also obtained for each coordinate axis C. Therefore, in a strict expression, the left wrist joint velocity luminance standard deviation LVS is expressed as the left wrist joint velocity luminance standard deviation LVS(C).
Similarly, the right hand joint velocity luminance standard deviation RVS is also calculated for each coordinate axis C. Therefore, in a strict sense, the right hand joint velocity luminance standard deviation RVS is expressed as the right hand joint velocity luminance standard deviation RVS(C).
Unless a strict notation is required, the left wrist joint velocity luminance standard deviation LVS and the right wrist joint velocity luminance standard deviation RVS will be collectively referred to as the joint velocity luminance standard deviation VS.
Note that, instead of using the luminance standard deviation for the left hand and the luminance standard deviation for the right hand as described above, the luminance standard deviation over both the left hand and the right hand may be used. In the following, an example in which the luminance standard deviation for the left hand and the luminance standard deviation for the right hand are used will be described, but the following description also applies to the case in which the luminance standard deviation over both the left hand and the right hand is used.

The element work estimation unit 124 extracts partial images constituting a partial video captured during a work performance time period from each of the left hand joint position image LJPI, the left hand joint velocity image LJVI, the right hand joint position image RJPI, and the right hand joint velocity image RJVI for each work performance time period included in the work performance time period set FS. That is, the element work estimation unit 124 extracts images constituting a partial video captured during a work performance time period from the left hand joint position image LJPI as partial images. The element work estimation unit 124 also extracts similar partial images from the left hand joint velocity image LJVI, the right hand joint position image RJPI, and the right hand joint velocity image RJV.
In the following, a set of partial images extracted from each of the left wrist joint position image LJPI, left wrist joint velocity image LJVI, right wrist joint position image RJPI, and right wrist joint velocity image RJVI will be referred to as a partial image set IS.

Then, the element work estimation unit 124 resizes each partial image included in the partial image set IS for each work time period into an image having a certain width and height.
Furthermore, the element work estimation unit 124 uses the pre-processing statistics to standardize the pixel values included in each partial image after resizing for each coordinate axis C. Note that the order of resizing and standardization may be reversed.
Specifically, the element work estimation unit 124 standardizes the pixel values of the resized partial image extracted from the left hand joint position image LJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS.
In addition, the element work estimation unit 124 standardizes the pixel values of the resized partial image extracted from the right hand joint position image RJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS.
More specifically, the element work estimation unit 124 standardizes the pixel values of the resized partial image extracted from the left hand joint position image LJPI using the left hand joint position luminance average value LPM(C) and the left hand joint position luminance standard deviation LPS(C).
In addition, the element work estimation unit 124 standardizes the pixel values of the resized partial image extracted from the right hand joint position image RJPI using the right hand joint position luminance average value RPM(C) and the right hand joint position luminance standard deviation RPS(C).

Furthermore, the element work estimation unit 124 standardizes the pixel values of the resized partial image extracted from the left hand joint velocity image LJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS.
In addition, the element work estimation unit 124 standardizes the pixel values of the resized partial image extracted from the right hand joint velocity image RJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS.
More specifically, the element work estimation unit 124 standardizes the pixel values of the resized partial image extracted from the left hand joint velocity image LJVI using the left hand joint velocity brightness average value LVM(C) and the left hand joint velocity brightness standard deviation LVS(C).
In addition, the element work estimation unit 124 standardizes the pixels of the resized partial image extracted from the right hand joint velocity image RJVI using the right hand joint velocity brightness average value RVM(C) and the right hand joint velocity brightness standard deviation RVS(C).

Then, the element work estimation unit 124 inputs each partial image after resizing and standardization to the element work estimation model M.
Then, the element work estimation unit 124 acquires the estimation result qs from the element work estimation model M. The estimation result qs indicates, in association with a work time period, an element work that is estimated to be performed by a worker during that work time period.
The element work estimation unit 124 performs the above procedure for all work execution time periods included in the work execution time period set FS, and obtains estimation results qs for each work time period.
Then, the element work estimation unit 124 outputs an estimation result set QS, which is a set of the estimation results qs for each work time period, to the estimation result processing unit 125.

**Explanation of the estimation result processing unit 125**
The estimation result processing unit 125 acquires the estimation result set QS from the element work estimation unit 124 .
In addition, the estimation result processing unit 125 acquires the non-work time slot set NFS from the work implementation time slot detection unit 123 .
The estimation result processing unit 125 sorts the multiple estimation results qs included in the estimation result set QS in ascending order of the work implementation time zones to which each estimation result qs corresponds.
The estimation result processing unit 125 also assigns a label representing a non-working time period to each of the non-working time periods included in the non-working time period set NFS. Furthermore, the estimation result processing unit 125 inserts each non-working time period after the label assignment into the sorted estimation result set QS at the corresponding time position.
Each non-work time slot after labeling and the sorted estimation result set QS are output to the estimation result storage unit 132 as the final estimation result AS.

Figure 21 shows an overview of the operation of the work performance time zone detection unit 123, element work estimation unit 124 and estimation result processing unit 125.

**Explanation of the element work estimation model storage unit 130**
The element work estimation model storage unit 130 stores the element work estimation model M.

**Explanation of Pre-Processing Statistics Storage Unit 131**
The preprocessing statistics storage unit 131 stores the preprocessing statistics (the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS).

**Explanation of Estimation Result Storage Unit 132**
The estimation result storage unit 132 stores the final estimation result AS.

**Explanation of an example of the internal configuration of the task execution time zone detection unit 123 and an example of the internal configuration of the element task estimation unit 124**
FIG. 8 shows an example of the internal configuration of the work execution time zone detection unit 123 and the internal configuration of the element work estimation unit 124.
An example of the internal configuration of the work execution time zone detection unit 123 and an example of the internal configuration of the element work estimation unit 124 will be described with reference to FIG.
In addition, in FIG. 8, of the functional components of the estimation device 100, only the functional components necessary for explaining an example of the internal configuration of the work execution time zone detection unit 123 and the example of the internal configuration of the element work estimation unit 124 are shown.

**Explanation of an example of the internal configuration of the work execution time zone detection unit 123**
The task execution time zone detection unit 123 has, as its internal components, an appearance status determination unit 1231, a tool gripping status determination unit 1232, a displacement amount determination unit 1233, and a task execution time zone determination unit 1234.

**Explanation of the appearance status determination unit 1231**
The appearance status determination unit 1231 performs appearance status determination.
The appearance status determination unit 1231 acquires a left wrist joint position image LJPI, a left wrist joint velocity image LJVI, a right wrist joint position image RJPI, and a right wrist joint velocity image RJVI from the joint time-series data imaging unit 122.
Then, the appearance status determination unit 1231 analyzes the left hand joint position image LJPI and/or the left hand joint velocity image LJVI to generate a left hand appearance time period set LS. Similarly, the appearance status determination unit 1231 analyzes the right hand joint position image RJPI and/or the right hand joint velocity image RJVI to generate a right hand appearance time period set RS. As described above, the left hand appearance time period set LS is a set of time periods in which the left hand appears in the image. Also, the right hand appearance time period set RS is a set of time periods in which the right hand appears in the image. The left hand appearance time period set LS and the right hand appearance time period set RS are time series data shown in FIG. 4 as the determination result of the appearance status determination.
The appearance status determination unit 1231 outputs the left hand appearance time period set LS and the right hand appearance time period set RS to the work implementation time period determination unit 1234 .
As described above, the appearance status determination unit 1231 may perform appearance status determination using the joint position time series data HPT and/or the joint velocity time series data HVT.
In FIG. 8, input of the joint position time series data HPT and/or the joint velocity time series data HVT to the appearance status determination unit 1231 is omitted.

**Description of the tool gripping state determination unit 1232**
The tool holding status determination unit 1232 determines the tool holding status.
The tool gripping state determination unit 1232 acquires the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 from the gripped tool time-series data generation unit 126.
The tool gripping status determination unit 1232 extracts, in chronological order, time periods during which a tool is gripped and time periods during which a tool is not gripped for each of the left and right hands based on the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1. In addition, when the type of tool held by the worker is changing, the tool gripping status determination unit 1232 treats the time periods during which a tool is gripped as different time periods for each type of tool.
The tool gripping situation determination unit 1232 outputs a left-hand tool gripping situation determination result LTS indicating the extraction result for the left hand to the work implementation time zone determination unit 1234. In addition, the tool gripping situation determination unit 1232 outputs a right-hand tool gripping situation determination result RTS indicating the extraction result for the right hand to the work implementation time zone determination unit 1234. The left-hand tool gripping situation determination result LTS and the right-hand tool gripping situation determination result RTS are time-series data shown in Fig. 4 as the determination results of the tool gripping situation determination.

**Explanation of the Displacement Amount Determination Unit 1233**
The displacement amount determination unit 1233 performs displacement amount determination.
The displacement amount determining unit 1233 acquires a left wrist joint position image LJPI, a left wrist joint velocity image LJVI, a right wrist joint position image RJPI, and a right wrist joint velocity image RJVI from the joint time-series data imaging unit 122.
Then, the displacement amount determination unit 1233 analyzes the left hand joint position image LJPI and/or the left hand joint velocity image LJVI to calculate the displacement amount of the left hand per unit time. For example, the displacement amount determination unit 1233 calculates the displacement amount of the wrist joint of the left hand per unit time. Similarly, the displacement amount determination unit 1233 analyzes the right hand joint position image RJPI and/or the right hand joint velocity image RJVI to calculate the displacement amount of the right hand per unit time. For example, the displacement amount determination unit 1233 calculates the displacement amount of the wrist joint of the right hand per unit time.
Then, the displacement amount determination unit 1233 outputs left hand displacement amount data LZS indicating the displacement amount of the left hand per unit time in a time series to the work execution time zone determination unit 1234. The displacement amount determination unit 1233 also outputs right hand displacement amount data RZS indicating the displacement amount of the right hand per unit time in a time series to the work execution time zone determination unit 1234. The left hand displacement amount data LZS and right hand displacement amount data RZS are time series data shown in FIG. 2 as the determination result of the displacement amount determination.
As described above, the displacement amount determining unit 1233 may determine the displacement amount using the joint position time series data HPT and/or the joint velocity time series data HVT.
In FIG. 8, the input of the joint position time series data HPT and/or the joint velocity time series data HVT to the displacement amount determination unit 1233 is omitted.

**Explanation of the work execution time zone determination unit 1234**
The work implementation time zone determination unit 1234 acquires a left hand appearance time zone set LS and a right hand appearance time zone set RS from the appearance status determination unit 1231.
In addition, the work execution time zone determination unit 1234 acquires the left hand tool gripping situation determination result LTS and the right hand tool gripping situation determination result RTS from the tool gripping situation determination unit 1232.
In addition, the work implementation time zone determination unit 1234 acquires the left hand displacement amount data LZS and the right hand displacement amount data RZS from the displacement amount determination unit 1233.
Then, the work execution time zone determining unit 1234 divides the work engagement time into a work execution time zone and a non-work time zone based on these.

For example, if the amount of displacement of at least one of the left hand and the right hand is equal to or greater than a threshold value, the work performance time zone determination unit 1234 designates the time zone as a non-work time zone, regardless of whether at least one of the left hand and the right hand is holding a tool or whether at least one of the left hand and the right hand appears in the image or not.

Furthermore, for example, when the displacement amounts of the left and right hands are both less than a threshold, if at least one of the left and right hands appears in the video, the work performance time zone determination unit 1234 designates the time zone as a work performance time zone, regardless of whether at least one of the left and right hands is holding a tool. On the other hand, even if the displacement amounts of the left and right hands are both less than the threshold, if neither the left nor right hand appears in the video, the time division unit 11 designates the time zone as a non-work time zone, regardless of whether at least one of the left and right hands is holding a tool.
In this case, the work execution time zone determination unit 1234 may treat the left hand and the right hand separately. Specifically, the work execution time zone determination unit 1234 treats the following (A1) to (A3) as separate work execution time zones. Also, when the work engagement time is divided into a work execution time zone and a non-working time zone only by the appearance status determination, the work execution time zone determination unit 1234 treats (A1) to (A3) as separate work execution time zones.
(A1) A time period when the left hand appears but the right hand does not appear. (A2) A time period when the left hand does not appear but the right hand does appear. (A3) A time period when both the left hand and the right hand appear.

Furthermore, the task execution time zone determination unit 1234 designates a time zone in which at least one of the left hand and the right hand is holding a tool as the task execution time zone, regardless of the amount of displacement of the left hand and the right hand, for example.
In this case, the work execution time zone determination unit 1234 may treat the left hand and the right hand separately. Specifically, the work execution time zone determination unit 1234 treats the following (B1) to (B3) as separate work execution time zones. Also, when dividing the work engagement time into a work execution time zone and a non-working time zone only based on the tool gripping state determination, the work execution time zone determination unit 1234 treats (B1) to (B3) as separate work execution time zones.
(B1) A time period when the left hand is holding a tool, but the right hand is not holding a tool. (B2) A time period when the left hand is not holding a tool, but the right hand is holding a tool. (B3) A time period when both the left and right hands are holding a tool. In addition, the work performance time period determination unit 1234 may subdivide the above (B3) and treat the following (B3-1) and (B3-2) as separate work performance time periods.
(B3-1) A time period when the left and right hands are holding the same tool. (B3-2) A time period when the left and right hands are holding different tools.

As described above, the work execution time zone determination unit 1234 designates a time zone in which the amount of displacement is less than the threshold as the work execution time zone.
The work implementation time zone determination unit 1234 may compare the amount of displacement with a threshold value, for example, as follows.
The task execution time zone determination unit 1234 regards the left hand appearance time zone set LS as a set of task execution time zone candidates. The task execution time zone determination unit 1234 also extracts time series data of partial displacement amounts of time zones of each candidate in the left hand displacement amount data LZS. This time series data of partial displacement amounts is called a left hand partial displacement amount time series data set sLZS. The i-th data of the left hand partial displacement amount time series data set sLZS is also called sLZS(i).
The task execution time zone determination unit 1234 divides sLZS(i) into a certain time interval, and the kth partial displacement amount data of the divided sLZS(i) is called short-time partial displacement amount data sLZS(i, k).
The task execution time zone determination unit 1234 calculates statistics of the amount of variation contained in the short-time partial variation data sLZS(i, k). The statistics may be, for example, an average value.
If the statistic is equal to or greater than the threshold, the work execution time zone determination unit 1234 determines that the time zone corresponding to the short-time partial variation data sLZS(i,k) is a non-work execution time zone. On the other hand, if the statistic is less than the threshold, the work execution time zone determination unit 1234 determines that the time zone corresponding to the short-time partial variation data sLZS(i,k) is a work execution time zone.
The task implementation time zone determination unit 1234 performs similar processing for the right hand appearance time zone RS and the right hand tool gripping state determination result RTS.
Furthermore, when a left hand and a right hand appear at the same time, one of which is holding a tool and the other is not, the task execution time zone determination unit 1234 may determine whether or not it is a task execution time zone based on the amount of displacement of the hand holding the tool. Alternatively, the task execution time zone determination unit 1234 may determine whether or not it is a task execution time zone based on a comparison between a threshold value and a statistical amount of displacement of the hand holding the tool obtained by the process described in this paragraph.
When both hands are holding a tool, the task execution time zone determination unit 1234 compares the displacement amount of the left hand with the displacement amount of the right hand, or compares the statistics of the displacement amount of the left hand obtained by the processing described in this paragraph with the statistics of the displacement amount of the right hand. Then, the task execution time zone determination unit 1234 may compare the smaller of the displacement amount or the statistics of the displacement amount with a threshold value to determine whether or not it is a task execution time zone.

In addition, the work execution time zone determination unit 1234 may divide the work engagement time into work execution time zones and non-work time zones based on other criteria.

The work implementation time slot determination unit 1234 outputs a work implementation time slot set FS, which is a set of work implementation time slots, to the element work estimation unit 124. The work implementation time slot set FS indicates the start time and end time of the work implementation time slot for each work implementation time slot.
Furthermore, the work implementation time slot determination unit 1234 outputs a non-work time slot set NFS, which is a set of non-work time slots, to the estimation result processing unit 125. The non-work time slot set NFS indicates the start time and end time of each non-work time slot.

**Explanation of an example of the internal configuration of the element work estimation unit 124**
The element work estimation unit 124 has, as its internal components, a joint position image acquisition unit 1241 , a joint velocity image acquisition unit 1242 , a joint position feature extraction unit 1243 , a joint velocity feature extraction unit 1244 , and a joint image feature classification unit 1245 .

**Explanation of the joint position image acquisition unit 1241**
The joint position image acquisition unit 1241 acquires the work execution time period set FS from the work execution time period determination unit 1234 .
In addition, the joint position image acquisition section 1241 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging section 122.
Furthermore, the joint position image acquisition unit 1241 acquires preprocessing statistics from the preprocessing statistics storage unit 131. Specifically, the joint position image acquisition unit 1241 acquires a joint position luminance average value PM, a joint position luminance standard deviation PS, a joint velocity luminance average value VM, and a joint velocity luminance standard deviation VS as the preprocessing statistics.

Then, the joint position image acquisition unit 1241 extracts partial images constituting a partial video captured during a work performance time period from each of the left hand joint position image LJPI and the right hand joint position image RJPI for each work performance time period included in the work performance time period set FS.
Then, the joint position image acquisition unit 1241 resizes each partial image to an image having a certain width and height for each working time period. Furthermore, the joint position image acquisition unit 1241 standardizes the pixel values of the resized partial image extracted from the left hand joint position image LJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. Furthermore, the joint position image acquisition unit 1241 standardizes the pixel values of the resized partial image extracted from the right hand joint position image RJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. As described above, the order of resizing and standardization may be reversed.
The joint position image acquisition section 1241 outputs the resized and standardized partial images of each of the left hand joint position image LJPI and the right hand joint position image RJPI to the joint position feature extraction section 1243.

In addition, the joint position image acquisition section 1241 outputs the task execution time period set FS, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the joint velocity brightness average value VM, and the joint velocity brightness standard deviation VS to the joint velocity image acquisition section 1242.
Here, an example is described in which the joint position image acquisition unit 1241 acquires a work performance time period set FS, a left hand joint velocity image LJVI, a right hand joint velocity image RJVI, a joint velocity brightness average VM and a joint velocity brightness standard deviation VS, and outputs these values to the joint velocity image acquisition unit 1242.
Alternatively, the joint velocity image acquisition unit 1242 may acquire a work performance time zone set FS from the work performance time zone determination unit 1234, acquire a left hand joint velocity image LJVI and a right hand joint velocity image RJVI from the joint time series data imaging unit 122, and acquire a joint velocity brightness average value VM and a joint velocity brightness standard deviation VS from the pre-processing statistics storage unit 131.
In this case, the joint position image acquisition unit 1241 acquires only the left hand joint position image LJPI and the right hand joint position image RJPI from the joint time series data imaging unit 122, and acquires only the joint position luminance average value PM and the joint position luminance standard deviation PS from the preprocessing statistics storage unit 131.

**Explanation of joint velocity image acquisition unit 1242**
The joint velocity image acquisition section 1242 acquires from the joint position image acquisition section 1241 a task execution time period set FS, a left hand joint velocity image LJVI, a right hand joint velocity image RJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS.
Then, the joint velocity image acquisition unit 1242 extracts partial images that constitute partial videos captured during the work performance time period from each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI for each work performance time period included in the work performance time period set FS.
Then, the joint velocity image acquisition unit 1242 resizes each partial image to an image with a constant width and height for each work time period. Furthermore, the joint velocity image acquisition unit 1242 standardizes the pixel values of the resized partial image extracted from the left hand joint velocity image LJVI using the joint velocity luminance average VM and the joint velocity luminance standard deviation VS. Furthermore, the joint velocity image acquisition unit 1242 standardizes the pixel values of the resized partial image extracted from the right hand joint velocity image RJVI using the joint velocity luminance average VM and the joint velocity luminance standard deviation VS.
The joint velocity image acquisition unit 1242 outputs the resized and standardized partial images of each of the left wrist joint velocity image LJVI and the right wrist joint velocity image RJVI to the joint velocity feature extraction unit 1244.

**Explanation of the joint position feature extraction unit 1243**
The joint position feature extraction section 1243 acquires from the joint position image acquisition section 1241 partial images after resizing and standardization for each of the left hand joint position image LJPI and the right hand joint position image RJPI.
Then, the joint position feature extraction unit 1243 inputs the acquired partial image to the element work estimation model M. The element work estimation model M used here is, for example, a pre-trained convolutional neural network.
The joint position feature extraction unit 1243 extracts a joint position feature vector, which is a feature vector, from each partial image. Here, the joint position feature vector obtained from the partial image of the left hand joint position image LJPI is called a left hand position feature vector fLP. Also, the joint position feature vector obtained from the partial image of the right hand joint position image RJPI is called a right hand position feature vector fRP.
The joint position feature extraction unit 1243 outputs the left hand position feature vector fLP and the right hand position feature vector fRP to the joint image feature classification unit 1245.

**Description of Joint Velocity Feature Extraction Unit 1244**
The joint velocity feature extraction unit 1244 acquires, from the joint velocity image acquisition unit 1242, partial images after resizing and standardization for each of the left wrist joint velocity image LJVI and the right wrist joint velocity image RJVI.
Then, the joint velocity feature extraction unit 1244 inputs the acquired partial image to the element work estimation model M. The element work estimation model M used here is also, for example, a pre-trained convolutional neural network. Note that the convolutional neural network used here may be the same as or different from the convolutional neural network used by the joint position feature extraction unit 1243.
The joint velocity feature extraction unit 1244 extracts a joint velocity feature vector, which is a feature vector, from each partial image. Here, the joint velocity feature vector obtained from the partial image of the left hand joint velocity image LJVI is referred to as a left hand velocity feature vector fLV. Also, the joint velocity feature vector obtained from the partial image of the right hand joint velocity image RJVI is referred to as a right hand velocity feature vector fRV.
The joint velocity feature extraction unit 1244 outputs the left hand velocity feature vector fLV and the right hand velocity feature vector fRV to the joint image feature classification unit 1245.

**Explanation of the joint image feature classification unit 1245**
The joint image feature classification unit 1245 acquires the left hand position feature vector fLP and the right hand position feature vector fRP from the joint position feature extraction unit 1243. In addition, the joint image feature classification unit 1245 acquires the left hand velocity feature vector fLV and the right hand velocity feature vector fRV from the joint velocity feature extraction unit 1244.

Then, the joint image feature classifying unit 1245 combines the left hand position feature vector fLP, the right hand position feature vector fRP, the left hand velocity feature vector fLV, and the right hand velocity feature vector fRV.
Furthermore, the joint image feature classification unit 1245 inputs the combined feature vector to the element work estimation model M. More specifically, the joint image feature classification unit 1245 inputs the combined feature vector to a layer of a neural network that performs classification processing of the element work estimation model M. The layer configuration of the element work estimation model M can be arbitrarily specified by the user of the estimation device 100 depending on the amount of data and the task. The element work estimation model M is generally configured using multiple fully connected layers and multiple activation layers.
The joint image feature classification unit 1245 estimates, for each work performance time period, the component work estimated by the component work estimation model M, which has the highest probability, as the component work being performed by the worker during that work time period.
Then, the joint image feature classification unit 1245 outputs an estimation result set QS, which is a set of the estimation results qs for each work time period, to the estimation result processing unit 125.

*** Operation Description ***
Next, an example of the operation of the estimation device 100 according to the present embodiment will be described.
FIG. 9 is a flowchart showing an example of the operation of the estimation device 100.

First, in step S11, the joint position time series data acquisition unit 120 acquires the joint position time series data HPT. That is, the joint position time series data acquisition unit 120 generates the joint position time series data HPT from the video V from the imaging device 110.

Next, in step S12, the joint velocity calculation unit 121 generates joint velocity time series data HVT from the joint position time series data HPT.

Next, in step S13, the joint time series data imaging unit 122 images the joint position time series data HPT to generate a left wrist joint position image LJPI and a right wrist joint position image RJPI. The joint time series data imaging unit 122 also images the joint velocity time series data HVT to generate a left wrist joint velocity image LJVI and a right wrist joint velocity image RJVI.

Next, in step S14, the gripped tool time series data generation unit 126 generates left hand gripped tool data LTO1 and right hand gripped tool data RTO1 using the video V or sensor information RRS.

Next, in step S15, the work performance time zone detection unit 123 detects the work performance time zone using the left hand joint position image LJPI, the right hand joint position image RJPI, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the left hand held tool data LTO1, and the right hand held tool data RTO1.

Next, in step S16, the element work estimation unit 124 uses the element work estimation model M to estimate the element work performed by the worker for each work performance time period.

Finally, in step S17, the estimation result processing unit 125 outputs the final estimation result AS.

Next, an example of the operation of the work execution time zone detection unit 123 will be described in detail.
FIG. 10 is a flowchart showing details of an example of the operation of the work execution time zone detection unit 123.

First, in step S111, the appearance status determination unit 1231 determines the appearance status.
The appearance status determination unit 1231 acquires a left wrist joint position image LJPI, a left wrist joint velocity image LJVI, a right wrist joint position image RJPI, and a right wrist joint velocity image RJVI from the joint time-series data imaging unit 122.
The appearance status determination unit 1231 then analyzes the left hand joint position image LJPI and/or the left hand joint velocity image LJVI to generate a left hand appearance time period set LS. Similarly, the appearance status determination unit 1231 analyzes the right hand joint position image RJPI and/or the right hand joint velocity image RJVI to generate a right hand appearance time period set RS.
The appearance status determination unit 1231 outputs the left hand appearance time period set LS and the right hand appearance time period set RS to the work implementation time period determination unit 1234.

Next, in step S112, the tool holding status determination unit 1232 determines the tool holding status.
The tool gripping state determination unit 1232 acquires the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 from the gripped tool time-series data generation unit 126.
The tool holding status determination unit 1232 extracts, in chronological order, time periods during which a tool is being held and time periods during which a tool is not being held for each of the left and right hands based on the left hand held tool data LTO1 and the right hand held tool data RTO1. If the type of tool held by the worker is changing, the tool holding status determination unit 1232 treats the time periods during which a tool is being held as different time periods for each type of tool.
The tool holding situation determination unit 1232 outputs a left hand tool holding situation determination result LTS indicating the extraction result for the left hand and a right hand tool holding situation determination result RTS indicating the extraction result for the right hand to the work execution time zone determination unit 1234.

Next, in step S113, the displacement amount determination unit 1233 determines the displacement amount.
The displacement amount determining unit 1233 acquires a left wrist joint position image LJPI, a left wrist joint velocity image LJVI, a right wrist joint position image RJPI, and a right wrist joint velocity image RJVI from the joint time-series data imaging unit 122.
The displacement amount determination unit 1233 then analyzes the left wrist joint position image LJPI and/or the left wrist joint velocity image LJVI to calculate the displacement amount of the left hand per unit time. Similarly, the displacement amount determination unit 1233 analyzes the right wrist joint position image RJPI and/or the right wrist joint velocity image RJVI to calculate the displacement amount of the right hand per unit time.
Then, the displacement amount determination unit 1233 outputs left hand displacement amount data LZS which indicates the displacement amount of the left hand per unit time in a time series, and right hand displacement amount data RZS which indicates the displacement amount of the right hand per unit time in a time series, to the work performance time zone determination unit 1234.

Finally, in step S114, the work implementation time zone determination unit 1234 detects the work implementation time zone.
The work implementation time zone determination unit 1234 acquires a left hand appearance time zone set LS and a right hand appearance time zone set RS from the appearance status determination unit 1231.
In addition, the work execution time zone determination unit 1234 acquires the left hand tool gripping situation determination result LTS and the right hand tool gripping situation determination result RTS from the tool gripping situation determination unit 1232.
In addition, the work implementation time zone determination unit 1234 acquires the left hand displacement amount data LZS and the right hand displacement amount data RZS from the displacement amount determination unit 1233.
Then, the work execution time zone determining unit 1234 divides the work engagement time into a work execution time zone and a non-work time zone based on these.
Finally, the task performance time zone determination unit 1234 outputs a task performance time zone set FS, which is a set of task performance time zones, to the joint image feature classification unit 1245.
In addition, the work implementation time zone determination unit 1234 outputs a non-work time zone set NFS, which is a set of non-work time zones, to the estimation result processing unit 125.

Next, an example of the operation of the element work estimation unit 124 will be described in detail.
FIG. 11 is a flowchart showing details of an example of the operation of the element work estimation unit 124.

First, in step S121, the joint position image acquisition unit 1241 generates a partial image of the joint position image, and resizes and standardizes the partial image.
The joint position image acquisition unit 1241 acquires the work execution time period set FS from the work execution time period determination unit 1234 .
In addition, the joint position image acquisition section 1241 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging section 122.
In addition, the joint position image acquisition section 1241 acquires the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS as preprocessing statistics from the preprocessing statistics storage section 131.

Then, the joint position image acquisition unit 1241 extracts partial images constituting a partial video captured during a work performance time period from each of the left hand joint position image LJPI and the right hand joint position image RJPI for each work performance time period included in the work performance time period set FS.
Then, the joint position image acquisition unit 1241 resizes each partial image to an image having a certain width and height for each working time period. Furthermore, the joint position image acquisition unit 1241 standardizes the pixel values of the resized partial image extracted from the left hand joint position image LJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. Furthermore, the joint position image acquisition unit 1241 standardizes the pixel values of the resized partial image extracted from the right hand joint position image RJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. As described above, the order of resizing and standardization may be reversed.
The joint position image acquisition section 1241 outputs the resized and standardized partial images of each of the left hand joint position image LJPI and the right hand joint position image RJPI to the joint position feature extraction section 1243.
In addition, the joint position image acquisition section 1241 outputs the task execution time period set FS, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the joint velocity brightness average VM, and the joint velocity brightness standard deviation VS to the joint velocity image acquisition section 1242.

Next, in step S122, the joint velocity image acquisition unit 1242 generates a partial image of the joint velocity image, and resizes and standardizes the partial image.
The joint velocity image acquisition section 1242 acquires from the joint position image acquisition section 1241 a task execution time period set FS, a left hand joint velocity image LJVI, a right hand joint velocity image RJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS.
Then, the joint velocity image acquisition unit 1242 extracts partial images that constitute partial videos captured during the work performance time period from each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI for each work performance time period included in the work performance time period set FS.
Then, the joint velocity image acquisition unit 1242 resizes each partial image to an image having a constant width and height for each work time period. Furthermore, the joint velocity image acquisition unit 1242 standardizes the pixel values of the resized partial image extracted from the left hand joint velocity image LJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. Furthermore, the joint velocity image acquisition unit 1242 standardizes the pixel values of the resized partial image extracted from the right hand joint velocity image RJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. As described above, the order of resizing and standardization may be reversed.
The joint velocity image acquisition unit 1242 outputs the resized and standardized partial images of each of the left wrist joint velocity image LJVI and the right wrist joint velocity image RJVI to the joint velocity feature extraction unit 1244.

Next, in step S123, the joint position feature extraction unit 1243 extracts a joint position feature vector.
The joint position feature extraction section 1243 acquires, from the joint position image acquisition section 1241, partial images after resizing and standardization for each of the left hand joint position image LJPI and the right hand joint position image RJPI.
Then, the joint position feature extraction unit 1243 inputs the acquired partial image to the element work estimation model M.
Then, the joint position feature extraction unit 1243 extracts a left hand position feature vector fLP and a right hand position feature vector fRP.
The joint position feature extraction unit 1243 outputs the left hand position feature vector fLP and the right hand position feature vector fRP to the joint image feature classification unit 1245.

Next, in step S124, the joint velocity feature extractor 1244 extracts a joint velocity feature vector.
The joint velocity feature extraction unit 1244 acquires, from the joint velocity image acquisition unit 1242, partial images after resizing and standardization for each of the left wrist joint velocity image LJVI and the right wrist joint velocity image RJVI.
Then, the joint velocity feature extraction unit 1244 inputs the acquired partial images to the element task estimation model M.
Then, the joint velocity feature extraction unit 1244 extracts a left hand velocity feature vector fLV and a right hand velocity feature vector fRV.
The joint velocity feature extraction unit 1244 outputs the left hand velocity feature vector fLV and the right hand velocity feature vector fRV to the joint image feature classification unit 1245.

Finally, in step S125, the joint image feature classifying unit 1245 combines the joint position feature vector and the joint velocity feature vector to estimate the element work.
The joint image feature classification unit 1245 acquires the left hand position feature vector fLP and the right hand position feature vector fRP from the joint position feature extraction unit 1243. In addition, the joint image feature classification unit 1245 acquires the left hand velocity feature vector fLV and the right hand velocity feature vector fRV from the joint velocity feature extraction unit 1244.
Then, the joint image feature classification unit 1245 combines the left hand position feature vector fLP, the right hand position feature vector fRP, the left hand velocity feature vector fLV, and the right hand velocity feature vector fRV. Furthermore, the joint image feature classification unit 1245 inputs the combined feature vector to the element task estimation model M.
The joint image feature classification unit 1245 estimates, for each work performance time period, the component work estimated by the component work estimation model M, which has the highest probability, as the component work being performed by the worker during that work time period.
Then, the joint image feature classification unit 1245 outputs an estimation result set QS, which is a set of the estimation results qs for each work time period, to the estimation result processing unit 125.

***Description of Effects of the Embodiment***
In this embodiment, the estimation device 100 performs a two-stage process of detecting an operation execution time period and estimating an element operation being performed during the detected operation execution time period.
In detecting the task performance time period, the estimation device 100 detects the task performance time period by analyzing the appearance of the hand, the gripping state of the tool, and the amount of displacement of the hand. Therefore, the estimation device 100 can accurately detect the task performance time period even if the occurrence frequency and/or performance time of elemental tasks included in the task vary.
Because the work performance time period can be accurately detected in this manner, when estimating the component work, the estimation device 100 can accurately estimate the component work being performed during the work performance time period by using a partial image corresponding to the work performance time period.
Therefore, according to this embodiment, even if a task includes a plurality of elemental tasks, each of which has a different attribute, it is possible to accurately estimate the elemental task performed by a worker.

In addition, in this embodiment, the estimation device 100 estimates the element work using an image recognition model. Therefore, according to this embodiment, an existing trained model can be used to estimate the element work. In other words, since there is no need to train a model from scratch, the amount of data required for training can be reduced, and the burden of data collection can be reduced.

Embodiment 2.
In this embodiment, a learning device that generates the element work estimation model M described in the first embodiment will be described.
In this embodiment, differences from the first embodiment will be mainly described.
It should be noted that matters not explained below are the same as those in the first embodiment.

**Configuration Description**
12 illustrates an example of a functional configuration of the learning device 200 according to this embodiment. The learning device 200 operates in a learning phase. The learning phase is a phase preceding the estimation phase in which the estimation device 100 according to the first embodiment operates.
The learning device 200 is connected to an imaging device 210 .
The imaging device 210 is similar to the imaging device 110 described in the first embodiment.
That is, the imaging device 210 outputs an image of the worker's hands to the learning device 200.
The imaging device 210 is, for example, worn on the head of the worker and captures a first-person perspective image. In this embodiment, the imaging device 210 is assumed to be worn on the head of the worker. However, the imaging device 210 does not have to be worn on the head of the worker as long as it can capture an image of the worker's hands.

**Explanation of an Example of the Hardware Configuration of the Learning Device 200**
The learning device 200 according to this embodiment is a computer having a hardware configuration exemplified in FIG.
As shown in FIG. 23, the learning device 200 includes, as hardware, a processor 901, a main memory device 902, an auxiliary memory device 903, and a communication device 904.
The functions of the components such as the joint position time-series data acquisition unit 220 and the joint velocity calculation unit 221 shown in FIG. 12 are realized by, for example, a program.
The auxiliary storage device 903 stores programs that realize the functions of these components.
These programs are loaded from the auxiliary storage device 903 to the main storage device 902. Then, the processor 901 executes these programs to perform the operations of these components.

**Explanation of an Example of the Functional Configuration of the Learning Device 200**
FIG. 13 shows an example of the functional configuration of the learning device 200.
The learning device 200 includes a joint position time series data acquisition unit 220, a joint velocity calculation unit 221, a joint time series data imaging unit 222, a learning data generation unit 223, a preprocessing statistics calculation unit 224, an element work estimation model generation unit 225, a learning data storage unit 230, a preprocessing statistics storage unit 231 and an element work estimation model storage unit 232.

**Explanation of joint position time-series data acquisition unit 220**
The joint position time-series data acquiring section 220 performs the same operation as the joint position time-series data acquiring section 120 described in the first embodiment.
That is, the joint position time-series data acquisition unit 220 acquires an image V obtained by capturing an image of the worker's hand from the imaging device 210. The image V is an image captured during the work engagement time in the learning phase. The image V includes frames in which the worker's hand is captured and frames in which the worker's hand is not captured.
The joint position time series data acquisition unit 220 generates joint position time series data HPT from the video V.
Then, the joint position time series data acquisition unit 220 outputs the joint position time series data HPT to the joint velocity calculation unit 221 and the joint time series data imaging unit 222.
The joint position time series data HPT according to this embodiment is similar to the joint position time series data HPT described in the first embodiment.

**Explanation of the joint velocity calculation unit 221**
The joint velocity calculation unit 221 performs the same operation as the joint velocity calculation unit 121 described in the first embodiment.
The joint velocity calculation unit 221 acquires the joint position time series data HPT from the joint position time series data acquisition unit 220 .
The joint velocity calculation unit 221 then calculates the difference (velocity) in the time direction for each joint in the joint position time series data HPT, and outputs the joint velocity time series data HVT indicating the calculation result to the joint time series data visualization unit 222.
The joint velocity time series data HVT according to this embodiment is similar to the joint velocity time series data HVT described in the first embodiment.

**Explanation of the joint time-series data imaging unit 222**
The joint time-series data imaging unit 222 operates in the same manner as the joint time-series data imaging unit 122 described in the first embodiment.
That is, the joint time series data imaging unit 222 acquires the joint position time series data HPT from the joint position time series data acquisition unit 220. The joint time series data imaging unit 222 also acquires the joint velocity time series data HVT from the joint velocity calculation unit 221.
The joint time-series data imaging unit 222 images the joint position time-series data HPT to generate a left wrist joint position image LJPI and a right wrist joint position image RJPI. The joint time-series data imaging unit 122 images the joint velocity time-series data HVT to generate a left wrist joint velocity image LJVI and a right wrist joint velocity image RJVI.
Then, the joint time-series data visualization unit 222 outputs the left hand joint position image LJPI, the right hand joint position image RJPI, the left hand joint velocity image LJVI, and the right hand joint velocity image RJVI to the learning data generation unit 223.
The left hand joint position image LJPI and right hand joint position image RJPI and the left hand joint velocity image LJVI and right hand joint velocity image RJVI in this embodiment are similar to the left hand joint position image LJPI and right hand joint position image RJPI and the left hand joint velocity image LJVI and right hand joint velocity image RJVI described in embodiment 1.

**Explanation of the learning data generation unit 223**
The learning data generation unit 223 acquires a left wrist joint position image LJPI, a right wrist joint position image RJPI, a left wrist joint velocity image LJVI, and a right wrist joint velocity image RJVI from the joint time-series data imaging unit 222.
In addition, the learning data generation unit 223 acquires label information LBLinf from, for example, the user of the learning device 200.
The label information LBLinf includes a plurality of labels LBL. Each label LBL includes a start time ts, an end time te, and an element task type typ. Here, the user of the learning device 200 manually associates the start time ts and the end time te with the element task type typ. However, the start time ts and the end time te with the element task type typ may be associated with each other by other methods.
The start time ts indicates the start time of the work execution time period.
The end time te indicates the start time of the work execution time period.
The element work type typ indicates the type of element work being performed in the work performance time period.

The learning data generation unit 223 extracts, from the left hand joint position image LJPI, an image region from an image position corresponding to the start time ts to an image position corresponding to the end time te for each label LBL of the label information LBLinf, as a left hand joint position label image sLJPI. Then, the learning data generation unit 223 associates the extracted left hand joint position label image sLJPI with the label LBL.
In addition, the learning data generation unit 223 extracts, from the right hand joint position image RJPI, an image region from an image position corresponding to the start time ts to an image position corresponding to the end time te for each label LBL of the label information LBLinf, as a right hand joint position label image sRJPI. Then, the learning data generation unit 223 associates the extracted right hand joint position label image sRJPI with the label LBL.
Furthermore, for each label LBL in the label information LBLinf, an image region from an image position corresponding to the start time ts to an image position corresponding to the end time te is extracted as a left hand joint velocity label image sLJVI from the left hand joint velocity image LJVI. Then, the learning data generation unit 223 associates the extracted left hand joint velocity label image sLJVI with the label LBL.
Furthermore, the learning data generating unit 223 extracts, from the right hand joint velocity image RJVI, an image region from an image position corresponding to the start time ts to an image position corresponding to the end time te for each label LBL in the label information LBLinf, as a right hand joint velocity label image sRJVI. Then, the learning data generating unit 223 associates the extracted right hand joint velocity label image sRJVI with the label LBL.
Furthermore, the learning data generation unit 223 stores multiple pairs of left hand joint position label images sLJPI and labels LBL, multiple pairs of right hand joint position label images sRJPI and labels LBL, multiple pairs of left hand joint velocity label images sLJVI and labels LBL, and multiple pairs of right hand joint velocity label images sRJVI and labels LBL in the learning data storage unit 230 as learning data sIs.

**Explanation of Pre-Processing Statistics Calculation Unit 224**
The pre-processing statistics calculation unit 224 acquires the learning data sIs from the learning data storage unit 230 .
Specifically, the preprocessing statistics calculation unit 224 acquires, as the learning data sIs, multiple pairs of left hand joint position label images sLJPI and labels LBL, multiple pairs of right hand joint position label images sRJPI and labels LBL, multiple pairs of left hand joint velocity label images sLJVI and labels LBL, and multiple pairs of right hand joint velocity label images sRJVI and labels LBL.

Then, the pre-processing statistics calculation unit 224 calculates the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS as pre-processing statistics from the learning data sIs.
Specifically, the preprocessing statistics calculation unit 224 calculates the average luminance value in the multiple left hand joint position label images sLJPI included in the learning data sIs for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the left hand joint position label image sLJPI is the left hand joint position luminance average value LPM(C).
In addition, the preprocessing statistics calculation unit 224 calculates the average luminance value in the multiple right hand joint position label images sRJPI included in the learning data sIs for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the right hand joint position label image sRJPI is the right hand joint position luminance average value RPM(C).
Furthermore, the preprocessing statistics calculation unit 224 calculates the standard deviation of luminance in the multiple left hand joint position label images sLJPI included in the learning data sIs for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the left hand joint position label image sLJPI is the left hand joint position luminance standard deviation LPS(C).
In addition, the preprocessing statistics calculation unit 224 calculates the standard deviation of luminance in the multiple right hand joint position label images sRJPI included in the learning data sIs for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the right hand joint position label image sRJPI is the right hand joint position luminance standard deviation RPS(C).
Note that, instead of using the average luminance value and standard deviation of the left hand and the average luminance value and standard deviation of the right hand as described above, the average luminance value and standard deviation of the left hand and the average luminance value and standard deviation of the right hand may be used. In the following, an example in which the average luminance value and standard deviation of the left hand and the average luminance value and standard deviation of the right hand are used will be described, but the following description also applies to the case in which the average luminance value and standard deviation of the left hand and the average luminance value and standard deviation of the right hand are used.

In addition, the preprocessing statistics calculation unit 224 calculates the average luminance value in the multiple left hand joint velocity label images sLJVI included in the learning data sIs for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the left hand joint velocity label image sLJVI is the left hand joint velocity luminance average value LVM(C).
In addition, the preprocessing statistics calculation unit 224 calculates the average brightness value of the multiple right hand joint velocity label images sRJVI included in the learning data sIs for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the right hand joint velocity label image sRJVI is the right hand joint velocity brightness average value RVM(C).
Furthermore, the preprocessing statistics calculation unit 224 calculates the standard deviation of luminance in the multiple left hand joint velocity label images sLJVI included in the learning data sIs for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the left hand joint velocity label image sLJVI is the left hand joint velocity luminance standard deviation LVS(C).
In addition, the preprocessing statistics calculation unit 224 calculates the standard deviation of luminance in the multiple right hand joint velocity label images sRJVI included in the learning data sIs for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the right hand joint velocity label image sRJVI is the right hand joint velocity luminance standard deviation RVS(C).
Note that, instead of using the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand as described above, the average luminance value and standard luminance deviation for both the left and right hands may be used. In the following, an example will be described in which the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand are used, but the following description also applies to the case in which the average luminance value and standard luminance deviation for both the left and right hands are used.

The pre-processing statistics calculation unit 224 stores the joint position luminance mean value PM, the joint position luminance standard deviation PS, the joint velocity luminance mean value VM and the joint velocity luminance standard deviation VS in the pre-processing statistics storage unit 231.

**Explanation of the element work estimation model generation unit 225**
The element work estimation model generation unit 225 acquires the learning data sIs from the learning data storage unit 230. Specifically, the element work estimation model generation unit 225 acquires, as the learning data sIs, a plurality of pairs of left hand joint position label images sLJPI and labels LBL, a plurality of pairs of right hand joint position label images sRJPI and labels LBL, a plurality of pairs of left hand joint velocity label images sLJVI and labels LBL, and a plurality of pairs of right hand joint velocity label images sRJVI and labels LBL.
In addition, the element work estimation model generation unit 225 acquires the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS from the preprocessing statistics storage unit 231.

Next, the element work estimation model generation unit 225 resizes the left hand joint position label image sLJPI included in the learning data sIS into an image having a certain width and height.
Furthermore, the element work estimation model generation unit 225 standardizes the pixel values of the resized left hand joint position labeled image sLJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS.
Specifically, the element task estimation model generation unit 225 standardizes the pixel values of the resized left hand joint position labeled image sLJPI using the left hand joint position luminance average value LPM(C) and the left hand joint position luminance standard deviation LPS(C).

Furthermore, the element work estimation model generation unit 225 resizes the right hand joint position label image sRJPI included in the learning data sIS into an image having a certain width and height.
Furthermore, the element work estimation model generation unit 225 standardizes the pixel values of the resized right hand joint position labeled image sRJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS.
Specifically, the element task estimation model generation unit 225 standardizes the pixel values of the resized right hand joint position labeled image sRJPI using the right hand joint position luminance average value RPM(C) and the right hand joint position luminance standard deviation RPS(C).

Furthermore, the element work estimation model generation unit 225 resizes the left hand joint velocity label image sLJVI included in the learning data sIS into an image having a certain width and height.
Furthermore, the element work estimation model generation unit 225 standardizes the pixel values of the resized left hand joint velocity labeled image sLJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS.
Specifically, the element work estimation model generation unit 225 standardizes the pixel values of the resized left hand joint velocity labeled image sLJVI using the left hand joint velocity brightness average value LVM(C) and the left hand joint velocity brightness standard deviation LVS(C).

Furthermore, the element work estimation model generation unit 225 resizes the right hand joint velocity label image sRJVI included in the learning data sIS into an image having a certain width and height.
Furthermore, the element work estimation model generation unit 225 standardizes the pixel values of the resized right hand joint velocity labeled image sRJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS.
Specifically, the element work estimation model generation unit 225 standardizes the pixel values of the resized right hand joint velocity labeled image sRJVI using the right hand joint velocity brightness average value RVM(C) and the right hand joint velocity brightness standard deviation RVS(C).

Next, the element work estimation model generation unit 225 inputs each of the resized and standardized label images to a predefined neural network. The label images to be input to the neural network are the resized and standardized left hand joint position label image sLJPI, the resized and standardized right hand joint position label image sRJPI, the resized and standardized left hand joint velocity label image sLJVI, and the resized and standardized right hand joint velocity label image sRJVI.
The neural network is a combination of a pre-trained convolutional neural network and a fully connected layer. The number of layers and the type of layers to be used in the pre-trained convolutional neural network are arbitrarily specified by the user of the learning device 200.

The element task estimation model generation unit 225 acquires a feature map of each label image extracted by the convolutional neural network.
The feature map of the left hand joint position label image sLJPI after resizing and standardization is called the feature map fLP.
The feature map of the right hand joint position label image sRJPI after resizing and standardization is called the feature map fRP.
The feature map of the left wrist joint velocity label image sLJVI after resizing and standardization is called the feature map fLV.
The feature map of the right wrist joint velocity label image sRJVI after resizing and standardization is referred to as the feature map fRV.

The element work estimation model generation unit 225 vectorizes and combines these feature maps fLP, fRP, fLV, and fRV.
Furthermore, the element work estimation model generation unit 225 inputs the combined feature map to a neural network, learns the weights of the neural network, and generates an element work estimation model M.
Then, the element work estimation model generation unit 225 stores the element work estimation model M in the element work estimation model storage unit 232 .

**Explanation of the learning data storage unit 230**
The learning data storage unit 230 stores the learning data sIs.

**Explanation of Pre-Processing Statistics Storage Unit 231**
The pre-processing statistics storage unit 231 stores the pre-processing statistics (the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS).

**Explanation of the element work estimation model storage unit 232**
The element work estimation model storage unit 232 stores the element work estimation model M.

**Explanation of an example of the internal configuration of the element work estimation model generation unit 225**
FIG. 13 shows an example of the internal configuration of the element work estimation model generation unit 225.
An example of the internal configuration of the element work estimation model generation unit 225 will be described with reference to FIG.
Note that, in FIG. 13, of the functional components of the learning device 200, only the functional components necessary for explaining an example of the internal configuration of the element work estimation model generation unit 225 are shown.

The element work estimation model generation unit 225 has, as its internal components, a joint position image acquisition unit 2251, a joint velocity image acquisition unit 2252, a joint position feature extraction unit 2253, a joint velocity feature extraction unit 2254, and a joint feature classification learning unit 2255.
The joint position image acquisition unit 2251 corresponds to the joint position image acquisition unit 1241 shown in FIG.
The joint velocity image acquisition unit 2252 corresponds to the joint velocity image acquisition unit 1242 shown in FIG.
The joint position feature extraction unit 2253 corresponds to the joint position feature extraction unit 1243 shown in FIG.
The joint velocity feature extractor 2254 corresponds to the joint velocity feature extractor 1244 shown in FIG.
The joint image feature classification learning unit 2255 corresponds to the joint image feature classification unit 1245 shown in FIG.

**Explanation of the joint position image acquisition unit 2251**
The joint position image acquisition unit 2251 acquires the learning data sIs from the learning data storage unit 230.
Specifically, the joint position image acquisition unit 2251 acquires, as the learning data sIs, multiple pairs of left hand joint position label images sLJPI and labels LBL, multiple pairs of right hand joint position label images sRJPI and labels LBL, multiple pairs of left hand joint velocity label images sLJVI and labels LBL, and multiple pairs of right hand joint velocity label images sRJVI and labels LBL.
Furthermore, the joint position image acquisition section 2251 acquires preprocessing statistics from the preprocessing statistics storage section 231. Specifically, the joint position image acquisition section 2251 acquires a joint position luminance average value PM, a joint position luminance standard deviation PS, a joint velocity luminance average value VM, and a joint velocity luminance standard deviation VS as the preprocessing statistics.

Then, the joint position image acquisition unit 2251 resizes the left hand joint position label image sLJPI to an image having a certain width and height. Furthermore, the joint position image acquisition unit 2251 standardizes the pixel values of the resized left hand joint position label image sLJPI using the left hand joint position luminance average value LPM(C) and the left hand joint position luminance standard deviation LPS(C) for each coordinate axis C.
The joint position image acquisition unit 2251 also resizes the right hand joint position label image sRJPI to an image having a certain width and height. Furthermore, the joint position image acquisition unit 2251 standardizes the pixel values of the resized right hand joint position label image sRJPI using the right hand joint position luminance average value RPM(C) and the right hand joint position luminance standard deviation RPS(C) for each coordinate axis C.
The joint position image acquisition unit 2251 outputs the resized and standardized left hand joint position label image sLJPI and the resized and standardized right hand joint position label image sRJPI to the joint position feature extraction unit 2253.
Note that, instead of using the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand as described above, the average luminance value and standard luminance deviation for both the left hand and the right hand may be used. In the following, an example will be described in which the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand are used, but the following description also applies to the case in which the average luminance value and standard luminance deviation for both the left hand and the right hand are used.

In addition, the joint position image acquisition section 2251 outputs the left hand joint velocity label image sLJVI, the right hand joint velocity label image sRJVI, the joint velocity brightness average value VM, and the joint velocity brightness standard deviation VS to the joint velocity image acquisition section 2252.
Here, an example is described in which the joint position image acquisition unit 2251 acquires a left hand joint velocity label image sLJVI, a right hand joint velocity label image sRJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS, and outputs these to the joint velocity image acquisition unit 2252.
Alternatively, the joint velocity image acquisition unit 2252 may acquire the left hand joint velocity label image sLJVI and the right hand joint velocity label image sRJVI from the learning data storage unit 230, and acquire the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS from the preprocessing statistics storage unit 231. In this case, the joint position image acquisition unit 2251 acquires only the left hand joint position label image sLJPI and the right hand joint position label image sRJPI from the learning data storage unit 230, and acquires only the joint position luminance average value PM and the joint position luminance standard deviation PS from the preprocessing statistics storage unit 231.

**Explanation of the joint velocity image acquisition unit 2252**
The joint velocity image acquisition section 2252 acquires a left hand joint velocity label image sLJVI, a right hand joint velocity label image sRJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS from the joint position image acquisition section 2251.
Then, the joint velocity image acquisition unit 2252 resizes the left hand joint velocity labeled image sLJVI to an image having a certain width and height. Furthermore, the joint velocity image acquisition unit 2252 standardizes the pixel values of the resized left hand joint velocity labeled image sLJVI using the left hand joint velocity brightness average value LVM(C) and the left hand joint velocity brightness standard deviation LVS(C) for each coordinate axis C.
The joint velocity image acquisition unit 2252 also resizes the right hand joint velocity labeled image sRJPI to an image with a certain width and height. Furthermore, the joint velocity image acquisition unit 2252 standardizes the pixel values of the resized right hand joint velocity labeled image sRJVI using the right hand joint velocity brightness average value RVM(C) and the right hand joint velocity brightness standard deviation RVS(C) for each coordinate axis C.
The joint velocity image acquisition unit 2252 outputs the resized and standardized left hand joint velocity label image sLJVI and the resized and standardized right hand joint velocity label image sRJVI to the joint velocity feature extraction unit 2254.

**Explanation of the joint position feature extraction unit 2253**
The joint position feature extraction unit 2253 acquires a resized and standardized left hand joint position label image sLJPI and a resized and standardized right hand joint position label image sRJPI from the joint position image acquisition unit 2251.
Then, the joint position feature extraction unit 2253 inputs the resized and standardized left hand joint position label image sLJPI to a pre-trained convolutional neural network to obtain a feature vector fLP.

The joint position feature extraction unit 2253 may also perform network learning, i.e., calculation of network weights, based on the loss value propagated from the joint image feature classification unit 2255 described later. For example, the weights of all layers may be updated using the weights of a pre-trained network as initial values.
For calculating the weights, the joint position feature extraction unit 2253 uses, for example, the commonly used backpropagation algorithm.
Furthermore, the joint position feature extraction unit 2253 may use other known methods to calculate the weights.

As a learning process, the joint position feature extraction unit 2253 repeats the acquisition of the feature map fLP, the calculation of the loss value L, and the calculation of the weights a predetermined number of times.
Finally, the joint position feature extraction unit 2253 obtains a feature map fLP of the left hand joint position label image sLJPI after resizing and standardization.
In addition, the joint position feature extraction unit 2253 performs similar processing on the right hand joint position label image sRJPI after resizing and standardization, and obtains a feature map fRP of the right hand joint position label image sRJPI after resizing and standardization.
The joint position feature extraction unit 2253 outputs the feature map fLP and the feature map fRP to the joint image feature classification learning unit 2255.

**Description of the joint velocity feature extraction unit 2254**
The joint velocity feature extraction unit 2254 acquires a resized and standardized left hand joint velocity label image sLJVI and a resized and standardized right hand joint velocity label image sRJVI from the joint velocity image acquisition unit 2252.
Then, the joint velocity feature extraction unit 2254 performs the same process as the joint position feature extraction unit 2253 to obtain a feature map fLV of the left hand joint velocity label image sLJVI after resizing and standardization. Also, the joint velocity feature extraction unit 2254 obtains a feature map fRV of the right hand joint velocity label image sRJVI after resizing and standardization.
Then, the joint velocity feature extraction unit 2254 outputs the feature map fLV and the feature map fRV to the joint image feature classification learning unit 2255.

**Explanation of the joint image feature classification learning unit 2255**
The joint image feature classification learning unit 2255 acquires the feature maps fLP and fRP from the joint position feature extraction unit 2253.
In addition, the joint image feature classification learning unit 2255 acquires the feature maps fLV and fRV from the joint velocity feature extraction unit 2254.
Then, the joint image feature classification learning unit 2255 vectorizes and combines the feature maps fLP, fRP, fLV, and fRV.
Furthermore, the joint image feature classification learning unit 2255 inputs the combined feature map to a neural network, learns the weights of the neural network, and generates an element task estimation model M.
The joint image feature classification learning unit 2255 uses, for example, cross-entropy, which is generally used for classification learning, to calculate the loss value L.
The joint position feature extraction unit 2253 may use a loss value corresponding to imbalance data, such as focal loss, so as to accommodate variations in the amount of data depending on the type of label LBL.
In addition, the joint image feature classification learning unit 2255 may use a loss function that enables distance learning based on the relationship between the angle between feature vectors and the element task type typ of the label LBL, so as to deal with variability in the data.
Then, the joint image feature classification learning unit 2255 stores the element work estimation model M in the element work estimation model storage unit 232.

*** Operation Description ***
Next, an example of the operation of the learning device 200 according to the present embodiment will be described.
FIG. 14 is a flowchart showing an example of the operation of the learning device 200.

First, in step S21, the joint position time series data acquisition unit 220 generates joint position time series data HPT from the image V from the imaging device 210.

Next, in step S22, the joint velocity calculation unit 221 generates joint velocity time series data HVT from the joint position time series data HPT.

Next, in step S23, the joint time series data imaging unit 222 images the joint position time series data HPT to generate a left wrist joint position image LJPI and a right wrist joint position image RJPI. The joint time series data imaging unit 222 also images the joint velocity time series data HVT to generate a left wrist joint velocity image LJVI and a right wrist joint velocity image RJVI.

Next, in step S24, the learning data generation unit 223 generates learning data sIs.

Next, in step S25, the preprocessing statistics calculation unit 224 calculates the preprocessing statistics.

Finally, in step S26, the element work estimation model generation unit 225 generates an element work estimation model M using the learning data sIs and the pre-processing statistics.
Finally, the element work estimation model generation unit 225 stores the element work estimation model M in the element work estimation model storage unit 232 .

***Description of Effects of the Embodiment***
According to this embodiment, it is possible to generate an element work estimation model M for estimating the element work being performed in a work execution time period from a partial image corresponding to the work execution time period.

Embodiment 3.
In this embodiment, a learning device 300 will be described as a modified example of the learning device 200 described in embodiment 2. More specifically, in this embodiment, the learning device 300 generates a trained model that is used by the work performance time zone detection unit 123 described in embodiment 1 to detect a work performance time zone.
In this embodiment, differences from the second embodiment will be mainly described.
It should be noted that matters not explained below are the same as those in the second embodiment.

**Configuration Description**
FIG. 15 shows an example of the functional configuration of a learning device 300 according to this embodiment.
In FIG. 15, an image capturing device 310 is similar to the image capturing device 210 shown in FIG.
12. Moreover, the joint position time-series data acquisition unit 320 is similar to the joint position time-series data acquisition unit 220 shown in FIG.
12. Moreover, the joint velocity calculation unit 321 is similar to the joint velocity calculation unit 221 shown in FIG.
12. The joint time-series data imaging unit 322 is similar to the joint time-series data imaging unit 222 shown in FIG.
For this reason, detailed explanations of these will be omitted.
Similar to the learning device 200, the learning device 300 also has the hardware configuration illustrated in FIG.

**Description of the grip tool time series data generating unit 325 The grip tool time series data generating unit 325 performs the same processing as the grip tool time series data generating unit 126 described in the first embodiment.

That is, the gripping tool time-series data generator 325 acquires the video V or the sensor information RRS.
The gripped tool time-series data generator 325 uses the video V or the sensor information RRS to identify whether the worker's hand is gripping a tool. If the worker's hand is gripping a tool, the gripped tool time-series data generator 325 identifies the type of tool the worker's hand is gripping.
The gripping tool time series data generating unit 325 identifies whether or not the worker's hand is gripping a tool, using a method similar to that of the gripping tool time series data generating unit 126. The gripping tool time series data generating unit 325 also identifies the type of tool being held by the worker's hand, using a method similar to that of the gripping tool time series data generating unit 126.

The gripped tool time-series data generating unit 325 outputs the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 indicating the classification results to the learning data generating unit 323.
In addition, the gripping tool time-series data generating unit 325 stores the left hand gripping tool data LTO1 and the right hand gripping tool data RTO1 in the gripping tool information storage unit 333.
The left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 generated by the grip tool time series data generating unit 325 are similar to the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 generated by the grip tool time series data generating unit 126.

**Explanation of the learning data generation unit 323**
The learning data generating unit 323 performs the same processing as the work execution time zone detecting unit 123 described in the first embodiment.

That is, the learning data generation unit 323 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data visualization unit 322.
Furthermore, the learning data generating unit 323 acquires left hand gripped tool data LTO1 and right hand gripped tool data RTO1 from the gripped tool time-series data generating unit 325.

Then, like the task execution time zone detection unit 123, the learning data generation unit 323 performs appearance status determination, tool gripping status determination, and displacement amount determination.
Furthermore, similar to the work performance time zone detection unit 123, the learning data generation unit 323 divides the work engagement time into a work performance time zone and a non-work time zone based on at least one of the results of the appearance status determination, the results of the tool holding status determination, and the results of the displacement amount determination.
The work time in this embodiment is the work time in the learning phase, that is, the work time in this embodiment is the time during which the worker is engaged in work in the learning phase.
Moreover, the work execution time period in this embodiment is a work execution time period in the learning phase, that is, a work execution time period in this embodiment is a time period during which a worker performs any one of the elemental works in the learning phase.
The non-working time period in this embodiment is a non-working time period in the learning phase, i.e., a non-working time period in this embodiment is a time period in the learning phase when the worker is not performing any element work.
In this embodiment, the learning data generating unit 323 detects a plurality of task execution time periods.

Next, the learning data generation unit 323 extracts, for each task performance time period, from the left hand joint position image LJPI, partial images constituting partial videos captured in each task performance time period as extracted left hand joint position images extLJPI.
Furthermore, the learning data generation unit 323 extracts, for each work performance time period, from the left hand joint velocity image LJVI, partial images constituting partial videos captured in each work performance time period as extracted left hand joint velocity images extLJVI.
Furthermore, the learning data generation unit 323 extracts, for each work performance time period, from the right hand joint position image RJPI, a partial image constituting a partial video captured in each work performance time period as an extracted right hand joint position image extRJPI.
Furthermore, the learning data generation unit 323 extracts, for each work performance time period, from the right hand joint velocity image RJVI, partial images constituting partial videos captured in each work performance time period as extracted right hand joint velocity images extRJVI.
Then, the learning data generation unit 323 stores the multiple extracted left hand joint position images extLJPI, the multiple extracted left hand joint velocity images extLJVI, the multiple extracted right hand joint position images extRJPI, and the multiple extracted right hand joint velocity images extRJVI in the learning data storage unit 330 as learning data sIt.
Furthermore, the learning data generating unit 323 stores a plurality of work execution time periods in the learning data storage unit 330 as learning data sIu.

**Explanation of Pre-Processing Statistics Calculation Unit 324**
The pre-processing statistics calculation unit 324 acquires the learning data sIt from the learning data storage unit 330 .
Then, the pre-processing statistics calculation unit 324 calculates the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS as pre-processing statistics from the learning data sIt.
Specifically, the preprocessing statistics calculation unit 324 calculates the average luminance value in the multiple extracted left hand joint position images extLJPI included in the learning data sIt for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the multiple extracted left hand joint position images extLJPI is the left hand joint position luminance average value LPM(C).
In addition, the preprocessing statistics calculation unit 324 calculates the average luminance value in the multiple extracted right hand joint position images extRJPI included in the learning data sIt for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the multiple extracted right hand joint position images extRJPI is the right hand joint position luminance average value RPM(C).
Furthermore, the preprocessing statistics calculation unit 324 calculates the standard deviation of luminance in the multiple extracted left hand joint position images extLJPI included in the learning data sIt for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted left hand joint position images extLJPI is the left hand joint position luminance standard deviation LPS(C).
Furthermore, the preprocessing statistics calculation unit 324 calculates the standard deviation of luminance in the multiple extracted right hand joint position images extRJPI included in the learning data sIt for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted right hand joint position images extRJPI is the right hand joint position luminance standard deviation RPS(C).

Furthermore, the preprocessing statistics calculation unit 324 calculates the average luminance value in the multiple extracted left hand joint velocity images extLJVI included in the learning data sIt for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted left hand joint velocity images extLJVI is the left hand joint velocity luminance average value LVM(C).
Furthermore, the preprocessing statistics calculation unit 324 calculates the average brightness value in the multiple extracted right hand joint velocity images extRJVI included in the learning data sIt for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted right hand joint velocity images extRJVI is the right hand joint velocity brightness average value RVM(C).
Furthermore, the preprocessing statistics calculation unit 324 calculates the standard deviation of luminance in the multiple extracted left hand joint velocity images extLJVI included in the learning data sIt for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted left hand joint velocity images extLJVI is the left hand joint velocity luminance standard deviation LVS(C).
Furthermore, the preprocessing statistics calculation unit 324 calculates the standard deviation of luminance in the multiple extracted right hand joint velocity images extRJVI included in the learning data sIt for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted right hand joint velocity images extRJVI is the right hand joint velocity luminance standard deviation RVS(C).

Unless otherwise specified strictly, the left wrist joint position luminance average value LPM(C) and the right wrist joint position luminance average value RPM(C) will be collectively referred to as the joint position luminance average value PM.
Similarly, the left hand joint position luminance standard deviation LPS(C) and the right hand joint position luminance standard deviation RPS(C) are collectively referred to as the joint position luminance standard deviation PS.
Furthermore, the left wrist joint velocity luminance average value LVM(C) and the right wrist joint velocity luminance average value RVM(C) are collectively referred to as the joint velocity luminance average value VM.
Furthermore, the left wrist joint velocity luminance standard deviation LVS(C) and the right wrist joint velocity luminance standard deviation RVS(C) are collectively referred to as the joint velocity luminance standard deviation VS.

Note that, instead of using the average luminance value and standard deviation of the left hand and the average luminance value and standard deviation of the right hand as described above, the average luminance value and standard deviation of the left and right hands may be used. Below, an example is described in which the average luminance value and standard deviation of the left hand and the average luminance value and standard deviation of the right hand are used, but the following description also applies to the case in which the average luminance value and standard deviation of the left and right hands are used.

The pre-processing statistics calculation unit 324 stores the joint position luminance mean value PM, the joint position luminance standard deviation PS, the joint velocity luminance mean value VM and the joint velocity luminance standard deviation VS in the pre-processing statistics storage unit 331.

**Explanation of the work execution time period detection model generation unit 326**
The work execution time period detection model generation unit 326 acquires the learning data sIt and the learning data sIu from the learning data storage unit 330 .
Specifically, the work performance time zone detection model generation unit 326 acquires a plurality of extracted left hand joint position images extLJPI, a plurality of extracted left hand joint velocity images extLJVI, a plurality of extracted right hand joint position images extRJPI, and a plurality of extracted right hand joint velocity images extRJVI as the learning data sIt.
Furthermore, the work performance time period detection model generation unit 326 acquires true values of a plurality of work performance time periods as the learning data sIu.
In addition, the task execution time period detection model generation unit 326 acquires the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS from the preprocessing statistics storage unit 331.
In addition, the task execution time zone detection model generation unit 326 acquires the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 from the gripped tool information storage unit 333.

Then, the work performance time zone detection model generation unit 326 generates a learned model, called a work performance time zone detection model DM, which is used by the work performance time zone detection unit 123 to detect the work performance time zone.

Specifically, the activity performance time period detection model generation unit 326 resizes the extracted left hand joint position image extLJPI included in the learning data sIt into an image having a certain width and height.
Furthermore, the activity performance time period detection model generation unit 326 standardizes the pixel values of the resized extracted left hand joint position image extLJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS.
Specifically, the task performance time period detection model generation unit 326 standardizes the pixel values of the resized extracted left hand joint position image extLJPI using the left hand joint position luminance average value LPM(C) and the left hand joint position luminance standard deviation LPS(C).

In addition, the activity performance time period detection model generation unit 326 resizes the extracted right hand joint position image extRJPI included in the learning data sIt into an image with a certain width and height.
Furthermore, the task performance time period detection model generation unit 326 standardizes the pixel values of the resized extracted right hand joint position image extRJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS.
Specifically, the task performance time period detection model generation unit 326 standardizes the pixel values of the resized extracted right hand joint position image extRJPI using the right hand joint position luminance average value RPM(C) and the right hand joint position luminance standard deviation RPS(C).

Furthermore, the task execution time period detection model generation unit 326 resizes the extracted left hand joint velocity image extLJVI included in the learning data sIt into an image with a certain width and height.
Furthermore, the task performance time period detection model generation unit 326 standardizes the pixel values of the resized extracted left hand joint velocity image extLJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS.
Specifically, the task performance time zone detection model generation unit 326 standardizes the pixel values of the resized extracted left hand joint velocity image extLJVI using the left hand joint velocity brightness average value LVM(C) and the left hand joint velocity brightness standard deviation LVS(C).

Furthermore, the task execution time period detection model generation unit 326 resizes the extracted right hand joint velocity image extRJVI included in the learning data sIt into an image with a certain width and height.
Furthermore, the task performance time period detection model generation unit 326 standardizes the pixel values of the resized extracted right hand joint velocity image extRJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS.
Specifically, the task performance time zone detection model generation unit 326 standardizes the pixel values of the resized extracted right hand joint velocity image extRJVI using the right hand joint velocity brightness average value RVM(C) and the right hand joint velocity brightness standard deviation RVS(C). As described above, the order of resizing and standardization may be reversed.

Next, the task performance time zone detection model generation unit 326 inputs each of the resized and standardized images to a predefined neural network. The images input to the neural network are the resized and standardized extracted left hand joint position image extLJPI, the resized and standardized extracted right hand joint position image extRJPI, the resized and standardized extracted left hand joint velocity image extLJVI, and the resized and standardized extracted left hand joint velocity image extLJVI.
The neural network used here is assumed to have the functions of a convolutional neural network and a region proposal network used in Faster-RCNN, for example.

Furthermore, the work performance time zone detection model generation unit 326 calculates a loss value L. The work performance time zone detection model generation unit 326 uses a loss function to calculate the difference between the true values of multiple work performance time zones included in the learning data sIu and multiple candidate values (predicted values) of the work performance time zones described below, as the loss value L. As the simplest loss function, the work performance time zone detection model generation unit 326 can use a squared error function.
After calculating the loss value L, the work execution time period detection model generation unit 326 updates the weights of the neural network using the backpropagation method.
The work performance time zone detection model generation unit 326 repeats these processes a predetermined number of times to generate the work performance time zone detection model DM.
The work performance time zone detection model generation unit 326 stores the work performance time zone detection model DM in the work performance time zone detection model storage unit 332 .

**Explanation of learning data storage unit 330**
The learning data storage unit 330 stores the learning data sIt and the learning data sIu.

**Explanation of Pre-Processing Statistics Storage Unit 331**
The pre-processing statistics storage unit 331 stores the pre-processing statistics (the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS).

**Explanation of the work execution time period detection model storage unit 332**
The work performance time zone detection model storage unit 332 stores the work performance time zone detection model DM.

**Explanation of the gripping tool information storage unit 333**
The gripped tool information storage unit 333 stores left hand gripped tool data LTO1 and right hand gripped tool data RTO1.

**Explanation of an example of the internal configuration of the work execution time period detection model generation unit 326**
FIG. 16 shows an example of the internal configuration of the work execution time period detection model generation unit 326 .
An example of the internal configuration of the work execution time period detection model generation unit 326 will be described with reference to FIG.
Note that, in FIG. 16, of the functional components of the learning device 300, only the functional components necessary for explaining an example of the internal configuration of the work execution time period detection model generation unit 326 are shown.

The work performance time zone detection model generation unit 326 has, as its internal components, a joint position image acquisition unit 3261, a joint velocity image acquisition unit 3262, a joint position feature extraction unit 3263, a joint velocity feature extraction unit 3264, an appearance status determination unit 3265, a tool holding status determination unit 3266, a work performance time zone determination unit 3268, a proposal learning unit 3269, and a regression learning unit 3270.

**Explanation of the appearance status determination unit 3265**
The appearance status determination unit 3265 performs appearance status determination in the same manner as the appearance status determination unit 1231 shown in FIG.

The appearance status determination unit 3265 acquires a left wrist joint position image LJPI, a left wrist joint velocity image LJVI, a right wrist joint position image RJPI, and a right wrist joint velocity image RJVI from the joint time-series data imaging unit 322.
Then, the appearance status determination unit 3265 analyzes the left hand joint position image LJPI and/or the left hand joint velocity image LJVI to generate a left hand appearance time period set LS.
Similarly, the appearance status determination unit 3265 analyzes the right hand joint position image RJPI and/or the right hand joint velocity image RJVI to generate a right hand appearance time period set RS.
The left hand appearance time period set LS is a set of time periods when a left hand appears in the image, and the right hand appearance time period set RS is a set of time periods when a right hand appears in the image.
The appearance status determination unit 3265 outputs the left hand appearance time period set LS and the right hand appearance time period set RS to the work implementation time period determination unit 3268.
The appearance status determination unit 3265 may determine the appearance status using the joint position time series data HPT and/or the joint velocity time series data HVT.
16, input of the joint position time series data HPT and/or the joint velocity time series data HVT to the appearance status determination unit 3265 is omitted.

**Explanation of the tool gripping state determination unit 3266**
The tool holding status determination unit 3266 performs tool holding status determination in the same manner as the tool holding status determination unit 1232 shown in FIG.

The tool holding status determination unit 3266 acquires the left hand held tool data LTO1 and the right hand held tool data RTO1 from the held tool information storage unit 333.
Then, based on the left hand held tool data LTO1 and the right hand held tool data RTO1, the tool holding status determination unit 3266 extracts, in chronological order, the time periods when a tool is being held and the time periods when a tool is not being held for each of the left hand and right hand.
In addition, if the type of tool held by the worker is changing, the tool holding status determination unit 3266 treats the time period during which the tool is held as a different time period for each type of tool.
The tool holding status determination unit 3266 outputs a left hand tool holding status determination result LTS indicating the extraction result for the left hand and a right hand tool holding status determination result RTS indicating the extraction result for the right hand to the work execution time zone determination unit 3268.

**Explanation of the work execution time zone determination unit 3268**
The work implementation time zone determining unit 3268 performs the same processing as the work implementation time zone determining unit 1234 shown in FIG.

The operation implementation time zone determination unit 3268 acquires a left hand appearance time zone set LS and a right hand appearance time zone set RS from the appearance status determination unit 3265.
In addition, the work execution time zone determination unit 3268 acquires the left hand tool holding situation determination result LTS and the right hand tool holding situation determination result RTS from the tool holding situation determination unit 3266.
Then, the work execution time zone determining unit 3268 divides the work engagement time into a work execution time zone and a non-work time zone based on these.
The work implementation time zone determination unit 3268 outputs a work implementation time zone set FS, which is a set of work implementation time zones, to the joint position image acquisition unit 3261. The work implementation time zone set FS indicates the start time and end time of each work implementation time zone.

**Explanation of the joint position image acquisition unit 3261**
The joint position image acquisition unit 3261 performs the same processing as the joint position image acquisition unit 1241 shown in FIG.

That is, the joint position image acquisition unit 3261 acquires the work implementation time zone set FS from the work implementation time zone determination unit 3268.
In addition, the joint position image acquisition section 3261 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging section 322.
Furthermore, the joint position image acquisition unit 3261 acquires preprocessing statistics from the preprocessing statistics storage unit 331. Specifically, the joint position image acquisition unit 3261 acquires a joint position luminance average value PM, a joint position luminance standard deviation PS, a joint velocity luminance average value VM, and a joint velocity luminance standard deviation VS as the preprocessing statistics.

Then, the joint position image acquisition unit 3261 extracts partial images constituting a partial video captured during the work performance time period from each of the left hand joint position image LJPI and the right hand joint position image RJPI for each work performance time period included in the work performance time period set FS.
Then, the joint position image acquisition unit 3261 resizes each partial image to an image having a certain width and height for each working time period. Furthermore, the joint position image acquisition unit 3261 standardizes the pixel values of the resized partial image extracted from the left hand joint position image LJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. Furthermore, the joint position image acquisition unit 3261 standardizes the pixel values of the resized partial image extracted from the right hand joint position image RJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. As described above, the order of resizing and standardization may be reversed.
The joint position image acquisition section 3261 outputs the partial images after resizing and standardization for each of the left hand joint position image LJPI and the right hand joint position image RJPI to the joint position feature extraction section 3263.

In addition, the joint position image acquisition section 3261 outputs the task execution time period set FS, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the joint velocity brightness average value VM and the joint velocity brightness standard deviation VS to the joint velocity image acquisition section 3262.
Here, an example is described in which the joint position image acquisition unit 3261 acquires a work performance time period set FS, a left hand joint velocity image LJVI, a right hand joint velocity image RJVI, a joint velocity brightness average VM and a joint velocity brightness standard deviation VS, and outputs these to the joint velocity image acquisition unit 3262.
Alternatively, the joint velocity image acquisition unit 3262 may acquire a work performance time period set FS from the work performance time period determination unit 3268, acquire a left hand joint velocity image LJVI and a right hand joint velocity image RJVI from the joint time series data imaging unit 322, and acquire a joint velocity brightness average value VM and a joint velocity brightness standard deviation VS from the pre-processing statistics storage unit 331.
In this case, the joint position image acquisition unit 3261 acquires only the left hand joint position image LJPI and the right hand joint position image RJPI from the joint time series data imaging unit 322, and acquires only the joint position luminance average value PM and the joint position luminance standard deviation PS from the preprocessing statistics storage unit 331.

**Explanation of joint velocity image acquisition unit 3262**
The joint velocity image acquisition section 3262 performs the same processing as the joint velocity image acquisition section 1242 shown in FIG.

That is, the joint velocity image acquisition unit 3262 acquires the task execution time period set FS, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the joint velocity brightness average value VM and the joint velocity brightness standard deviation VS from the joint position image acquisition unit 3261.
Then, the joint velocity image acquisition unit 3262 extracts partial images that constitute partial videos captured during the work performance time period from each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI for each work performance time period included in the work performance time period set FS.
Then, the joint velocity image acquisition unit 3262 resizes each partial image to an image having a constant width and height for each work time period. Furthermore, the joint velocity image acquisition unit 3262 standardizes the pixel values of the resized partial image extracted from the left hand joint velocity image LJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. Furthermore, the joint velocity image acquisition unit 3262 standardizes the pixel values of the resized partial image extracted from the right hand joint velocity image RJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. As described above, the order of resizing and standardization may be reversed.
The joint velocity image acquisition unit 3262 outputs the resized and standardized partial images of each of the left wrist joint velocity image LJVI and the right wrist joint velocity image RJVI to the joint velocity feature extraction unit 3264.

**Explanation of the joint position feature extraction unit 3263**
The joint position feature extraction unit 32633 performs the same processing as the joint position feature extraction unit 1243 shown in FIG.

That is, the joint position feature extraction section 3263 acquires, from the joint position image acquisition section 3261, partial images after resizing and standardization for each of the left hand joint position image LJPI and the right hand joint position image RJPI.
Then, the joint position feature extraction unit 3263 inputs partial images of each of the left hand joint position image LJPI and the right hand joint position image RJPI to a pre-trained convolutional neural network.
The joint position feature extraction unit 3263 extracts a joint position feature vector, which is a feature vector, from each partial image.
Here, a feature vector obtained from a partial image of the left hand joint position image LJPI is referred to as a left hand position feature vector fLP, and a feature vector obtained from a partial image of the right hand joint position image RJPI is referred to as a right hand position feature vector fRP.
The joint position feature extraction unit 3263 outputs the left hand position feature vector fLP and the right hand position feature vector fRP to the proposal learning unit 3269.

**Explanation of the joint velocity feature extraction unit 3264**
The joint velocity feature extractor 3264 performs the same processing as the joint velocity feature extractor 1244 shown in FIG.

That is, the joint velocity feature extraction section 3264 acquires, from the joint velocity image acquisition section 3262, partial images after resizing and standardization for each of the left wrist joint velocity image LJVI and the right wrist joint velocity image RJVI.
Then, the joint velocity feature extraction unit 3264 inputs the partial images of each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI to a pre-trained convolutional neural network. Note that the convolutional neural network used here may be the same as or different from the convolutional neural network used by the joint position feature extraction unit 3263.
The joint velocity feature extraction unit 3264 extracts a joint velocity feature vector, which is a feature vector, from each partial image.
Here, the feature vector obtained from a partial image of the left hand joint velocity image LJVI is referred to as a left hand velocity feature vector fLV, and the feature vector obtained from a partial image of the right hand joint velocity image RJVI is referred to as a right hand velocity feature vector fRV.
The joint velocity feature extraction unit 3264 outputs the left hand velocity feature vector fLV and the right hand velocity feature vector fRV to the proposal learning unit 3269.

**Explanation of the proposed learning unit 3269**
The proposal learning unit 3269 acquires a left hand position feature vector fLP and a right hand position feature vector fRP from the joint position feature extraction unit 3263. In addition, the proposal learning unit 3269 acquires a left hand velocity feature vector fLV and a right hand velocity feature vector fRV from the joint velocity feature extraction unit 3264.
In addition, the proposal learning unit 3269 acquires the learning data sIt from the learning data storage unit 330.

Then, the proposal learning unit 3269 converts the left hand position feature vector fLP, the right hand position feature vector fRP, the left hand velocity feature vector fLV, and the right hand velocity feature vector fRV into a map (FW×FH×FC) format. The map obtained by the proposal learning unit 3269 converting these feature vectors into a map format is called a feature map FM.
Then, the proposal learning unit 3269 performs a process of proposing candidates for the work execution time period for the feature map FM.
The proposal learning unit 3269 can use a proposal region network used in Faster-RCNN, which proposes multiple candidates for object existence regions, for the process of proposing candidates for work time periods. The output of the proposal process is a candidate for work time periods. A candidate for work time periods is represented by a tuple having two elements, a start time and a duration. The start time is the start time of the candidate for work time periods. In addition, the duration is the duration of the candidate for work time periods.
The proposal learning unit 3269 outputs a plurality of candidates for work execution time periods (a set of tuples) to the regression learning unit 3270 as proposal PROPS.

**Description of Regression Learning Unit 3270**
The regression learning unit 3270 obtains the suggestions PROPS from the suggestion learning unit 3269 .
Furthermore, the regression learning unit 3270 acquires the learning data sIu from the learning data storage unit 330. The learning data sIu is the true value of each of the multiple work execution time periods obtained by the learning data generation unit 323. The learning data sIu is also a collection of tuples having two elements, a start time and a duration. The start time in the learning data sIu is the start time of the work execution time period (true value). Furthermore, the duration in the learning data sIu is the duration of the work execution time period (true value).

Next, the regression learning unit 3270 obtains a tuple of the start time and duration for each candidate work time slot from the proposed PROPS. Furthermore, the regression learning unit 3270 uses the information of each tuple to process the partial feature map sFM extracted from the feature map FM through a fully connected layer of a convolutional neural network, and outputs an estimation result of the start time and duration of the work time slot. Then, the regression learning unit 3270 calculates a probability indicating whether the output result of the regression learning unit 3270 is a work time slot or not and a degree of confidence that it is a work time slot.
The regression learning unit 3270 selects a candidate having a degree of certainty greater than a predetermined threshold value from among a plurality of candidates for the work execution time period. The regression learning unit 3270 may also select a candidate having an index (e.g., Intersection of Units) indicating the proportion of an overlapping area with a true work execution time period greater than a predetermined threshold value from among a plurality of candidates for the work execution time period. Furthermore, the regression learning unit 3270 may apply a non-maximum value suppression process based on a probability that indicates the degree of certainty.

Next, the regression learning unit 3270 calculates the difference between the work implementation time slot (predicted value) of the selected candidate and the true value of the work implementation time slot acquired as the learning data sIu.
The obtained difference is used as a loss value L for training the neural network in the proposal training unit 3269 and for training the neural network in the regression training unit 3270 .
The regression learning unit 3270 may use an L1 loss such as Faster-RCNN to calculate the loss value L. Any method for calculating the loss value may be used as long as it properly calculates the difference between the predicted value and the true value of the work execution time period.
After calculating the loss value L, the regression learning unit 3270 updates the weights of the neural network using the backpropagation method.
The regression learning unit 3270 repeats these processes a predetermined number of times to generate an operation execution time period detection model DM.
The regression learning unit 3270 stores the work performance time zone detection model DM in the work performance time zone detection model storage unit 332 .

*** Operation Description ***
Next, an example of the operation of the learning device 300 according to the present embodiment will be described.
FIG. 17 is a flowchart showing an example of the operation of the learning device 300.

First, in step S31, the joint position time series data acquisition unit 320 generates joint position time series data HPT from the image V from the imaging device 310.

Next, in step S32, the joint velocity calculation unit 321 generates joint velocity time series data HVT from the joint position time series data HPT.

Next, in step S33, the joint time series data imaging unit 322 images the joint position time series data HPT to generate a left wrist joint position image LJPI and a right wrist joint position image RJPI. The joint time series data imaging unit 322 also images the joint velocity time series data HVT to generate a left wrist joint velocity image LJVI and a right wrist joint velocity image RJVI.

Next, in step S34, the learning data generation unit 323 generates learning data sIt and learning data sIu.

Next, in step S35, the preprocessing statistics calculation unit 324 calculates the preprocessing statistics.

Finally, in step S36, the work performance time zone detection model generation unit 326 generates the work performance time zone detection model DM using the learning data sIt, the learning data sIu, and the pre-processing statistics.
Finally, the work performance time zone detection model generation unit 326 stores the work performance time zone detection model DM in the element work estimation model storage unit 232 .

***Description of Effects of the Embodiment***
According to this embodiment, an activity performance time zone detection model DM can be generated based on the learning data sIt and the learning data sIu. This eliminates the need to perform rule-based activity performance time zone detection. In other words, it eliminates the need to determine a threshold value for activity performance time zone detection. As a result, the burden on the user of the estimation device 100 can be reduced.

Embodiment 4.
FIG. 18 shows an example of a functional configuration of an estimation device 400 according to this embodiment.
In FIG. 18, the work performance time zone detection model storage unit 433 stores the work performance time zone detection model DM shown in the third embodiment.
Furthermore, the work performance time zone detection unit 423 acquires the work performance time zone detection model DM from the work performance time zone detection model storage unit 433, and detects the work performance time zone using the work performance time zone detection model DM. That is, the work performance time zone detection unit 423 inputs the left hand joint position image LJPI, left hand joint velocity image LJVI, right hand joint position image RJPI, right hand joint velocity image RJVI, left hand held tool data LTO1, and right hand held tool data RTO1 into the work performance time zone detection model DM to detect the work performance time zone.
The components other than the work implementation time zone detection unit 423 and the work implementation time zone detection model storage unit 433 are similar to the components of the same names shown in Fig. 7. Therefore, the description of these components will be omitted.
Similarly to the estimation device 100, the estimation device 400 also has the hardware configuration illustrated in FIG.

***Description of Effects of the Embodiment***
As in the third embodiment, there is no need to determine a threshold value for detecting an operation execution time period, so the burden on the user of the estimation device 400 can be reduced.
Furthermore, by using the task execution time zone detection model DM, the task execution time zone can be accurately detected even for tasks in which element tasks are combined in a complex manner.

Embodiment 5.
19 shows an example of a functional configuration of a learning device 500 according to this embodiment. The learning device 500 is a combination of the learning device 200 described in the second embodiment and the learning device 300 described in the third embodiment.
The learning data generating unit 523 generates the learning data sIs, learning data sIt, and learning data sIu. The methods of generating the learning data sIs, learning data sIt, and learning data sIu are the same as those described in the second and third embodiments.
The preprocessing statistics calculation unit 524 calculates preprocessing statistics for each of the learning data sIs and the learning data sIt. The method of calculating the preprocessing statistics is the same as that described in the second and third embodiments.
The components other than the learning data generating unit 523 and the preprocessing statistics calculating unit 524 are similar to the components of the same names shown in Fig. 12 and Fig. 15. Therefore, the description of these components will be omitted.
Similarly to the learning devices 200 and 300, the learning device 500 also has the hardware configuration illustrated in FIG.

The learning for generating the work execution time period detection model DM and the learning for generating the element work estimation model M may be performed independently of each other, or may be performed simultaneously as multitask learning.
When learning for generating the task performance time zone detection model DM and learning for generating the element task estimation model M are performed simultaneously, part of the network structure and/or weights may be shared between the two learnings.

Furthermore, for the order of a plurality of element tasks that is the detection and classification result as the estimation result, a loss value L for the order may be calculated by comparing the estimation result with a true value (the true order of the element tasks), and the loss value L may be used for training each model.
The loss value L regarding the order may be calculated using, for example, an edit distance or a difference between graphs.
It is also desirable to change the calculation method of the loss value L for the sequence depending on the type of work. For example, when calculating the loss value L for an assembly work, it is possible to impose a large loss value L on the occurrence of an impossible sequence of element works in carrying out the assembly work.

***Description of Effects of the Embodiment***
According to this embodiment, the work execution time period detection model DM and the element work estimation model M can be generated efficiently.

Although the first to fifth embodiments have been described above, two or more of these embodiments may be combined for implementation.
Alternatively, one of these embodiments may be partially implemented.
Alternatively, two or more of these embodiments may be partially combined and implemented.
Furthermore, the configurations and procedures described in these embodiments may be modified as necessary.

Additional hardware configuration information
Finally, a supplementary explanation of the hardware configurations of the estimation device 100, the learning device 200, the learning device 300, the estimation device 400, and the learning device 500 will be given.
The following provides a supplementary explanation of the hardware configurations of the estimation device 100 and the learning device 200. The following explanation of the estimation device 100 also applies to the estimation device 400. Similarly, the following explanation of the learning device 200 also applies to the learning device 300 and the learning device 500.

A processor 801 shown in FIG. 22 is an integrated circuit (IC) that performs processing.
The processor 801 is a central processing unit (CPU), a digital signal processor (DSP), or the like.
The main memory device 802 shown in FIG. 22 is a RAM (Random Access Memory).
The auxiliary storage device 803 shown in FIG. 22 is a ROM (Read Only Memory), a flash memory, a HDD (Hard Disk Drive), or the like.
The communication device 804 shown in FIG. 22 is an electronic circuit that executes data communication processing.
The communication device 804 is, for example, a communication chip or a NIC (Network Interface Card).

The auxiliary storage device 803 also stores an OS (Operating System).
At least a part of the OS is executed by the processor 801 .
The processor 801 executes at least a part of the OS and executes programs that realize the respective functional components of the estimation device 100 .
The processor 801 executes the OS, thereby performing task management, memory management, file management, communication control, and the like.
In addition, at least one of information, data, signal values, and variable values indicating the results of processing of each functional component of the estimation device 100 is stored in at least one of the main memory device 802, the auxiliary memory device 803, and a register and cache memory within the processor 801.
Furthermore, the programs for realizing the respective functional components of the estimation device 100 may be stored in portable recording media such as magnetic disks, flexible disks, optical disks, compact disks, Blu-ray (registered trademark) disks, DVDs, etc. Then, the portable recording media in which the programs for realizing the respective functional components of the estimation device 100 are stored may be distributed.

Furthermore, at least any "part" of the functional components of the estimation device 100 may be read as a "circuit" or a "step" or a "procedure" or a "process" or a "circuitry".
Furthermore, the estimation device 100 may be realized by a processing circuit. The processing circuit is, for example, a logic IC (Integrated Circuit), a GA (Gate Array), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).
In this case, the functional components of the estimation device 100 are each realized as part of a processing circuit.

The processor 901 shown in FIG. 23 is an IC that performs processing.
The processor 901 is a CPU, a DSP, or the like.
The main memory device 902 shown in FIG. 23 is a RAM.
The auxiliary storage device 903 shown in FIG. 23 is a ROM, a flash memory, a HDD, or the like.
The communication device 904 shown in FIG. 23 is an electronic circuit that executes data communication processing.
The communication device 904 is, for example, a communication chip or a NIC.

The auxiliary storage device 903 also stores the OS.
At least a part of the OS is executed by the processor 901 .
The processor 901 executes at least a part of the OS and executes programs that realize each functional component of the learning device 200.
The processor 901 executes the OS, thereby performing task management, memory management, file management, communication control, and the like.
In addition, at least one of information, data, signal values, and variable values indicating the results of processing of each functional component of the learning device 200 is stored in at least one of the main memory device 902, the auxiliary memory device 903, a register within the processor 901, and a cache memory.
Furthermore, the programs that realize the functional components of learning device 200 may be stored on portable recording media such as magnetic disks, flexible disks, optical disks, compact disks, Blu-ray (registered trademark) disks, DVDs, etc. Then, portable recording media on which the programs that realize the functional components of learning device 200 are stored may be distributed.

Furthermore, at least any "part" of the functional components of the learning device 200 may be read as a "circuit" or a "step" or a "procedure" or a "process" or a "circuitry".
Furthermore, the learning device 200 may be realized by a processing circuit. The processing circuit is, for example, a logic IC, a GA, an ASIC, or an FPGA.
In this case, the functional components of the learning device 200 are each realized as part of a processing circuit.

In this specification, the higher-level concept of a processor and a processing circuit is called "processing circuitry."
That is, a processor and a processing circuit are each specific examples of "processing circuitry."

11 Time division unit, 12 Estimation unit, 100 Estimation device, 110 Imaging device, 120 Joint position time series data acquisition unit, 121 Joint velocity calculation unit, 122 Joint time series data imaging unit, 123 Work execution time zone detection unit, 1231 Appearance status determination unit, 1232 Tool gripping status determination unit, 1233 Displacement amount determination unit, 1234 Work execution time zone determination unit, 124 Element work estimation unit, 1241 Joint position image acquisition unit, 1242 Joint velocity image acquisition unit, 1243 Joint position feature extraction unit, 1244 Joint velocity feature extraction unit, 1245 Joint image feature classification unit, 125 Estimation result processing unit, 126 Grip tool time series data generation unit, 130 Element work estimation model storage unit, 131 Pre-processing statistics storage unit, 132 Estimation result storage unit, 200 Learning device, 210 Imaging device, 220 Joint position time series data acquisition unit, 221 Joint velocity calculation unit, 222 Joint time series data imaging unit, 223 Learning data generation unit, 224 Pre-processing statistics calculation unit, 225 Element work estimation model generation unit, 2251 Joint position image acquisition unit, 2252 Joint velocity image acquisition unit, 2253 Joint position feature extraction unit, 2254 Joint velocity feature extraction unit, 2255 Joint image feature classification learning unit, 230 Learning data storage unit, 231 Pre-processing statistics storage unit, 232 Element work estimation model storage unit, 300 Learning device, 310 Imaging device, 320 Joint position time series data acquisition unit, 321 Joint velocity calculation unit, 322 Joint time series data imaging unit, 323 Learning data generation unit, 324 Pre-processing statistics calculation unit, 325 Grasping tool time series data generation unit, 326 Task execution time zone detection model generation unit, 3261 Joint position image acquisition unit, 3262 joint velocity image acquisition unit, 3263 joint position feature extraction unit, 3264 joint velocity feature extraction unit, 3265 appearance status determination unit, 3266 tool gripping status determination unit, 3268 task execution time zone determination unit, 3269 proposal learning unit, 3270 regression learning unit, 330 learning data storage unit, 331 pre-processing statistics storage unit, 332 task execution time zone detection model storage unit, 333 gripped tool information storage unit, 400 estimation device, 410 imaging device, 420 joint position time series data acquisition unit, 421 joint velocity calculation unit, 422 joint time series data imaging unit, 423 task execution time zone detection unit, 424 element work estimation unit, 425 estimation result processing unit, 426 gripped tool time series data generation unit, 430 element work estimation model storage unit, 431 estimation result storage unit, 432 Pre-processing statistics storage unit, 433 task performance time zone detection model storage unit, 500 learning device, 510 imaging device, 520 joint position time series data acquisition unit, 521 joint velocity calculation unit, 522 joint time series data imaging unit, 523 learning data generation unit, 524 pre-processing statistics calculation unit, 525 task performance time zone detection model generation unit, 526 gripping tool time series data generation unit, 527 element task estimation model generation unit, 530 learning data storage unit, 531 pre-processing statistics storage unit, 532 task performance time zone detection model storage unit, 533 gripping tool information storage unit, 534 element task estimation model storage unit, 801 processor, 802 main memory device, 803 auxiliary memory device, 804 communication device, 901 processor, 902 main memory device, 903 auxiliary memory device, 904 communication device.

Claims (11)

a time division unit that divides a work engagement time into a work performance time zone during which the worker is performing any one of the plurality of elemental tasks and a non-work time zone during which the worker is not performing any one of the plurality of elemental tasks, using at least one of a result of a displacement amount determination that determines an amount of displacement of the hand of a worker during a work engagement time that is a time during which the worker is engaged in a task including a plurality of elemental tasks, each of which has non-uniform attributes, while the worker is engaged in the task, a result of a tool gripping state determination that determines a gripping state of a tool in the hand of the worker during the work engagement time, and a result of an appearance state determination that determines an appearance state of the hand of the worker in an image captured during the work engagement time;
A plurality of partial images constituting a partial video, which is a portion of the video captured during the work execution time period, are acquired, and the plurality of partial images are standardized using an average value and a standard deviation of the luminance of the learning images of the hand used in learning when generating the trained model, and resized to a specified size;
Inputting a plurality of processed partial images, which are the plurality of partial images after standardization and resizing, into the trained model to extract feature vectors of the plurality of processed partial images;
and an estimation unit that inputs feature vectors of the plurality of processed partial images into the trained model to estimate the elemental work being performed by the worker during the work performance time period. The estimation unit is
As the plurality of partial images, a hand position partial image which is a partial image regarding a position of the hand of the worker, and a hand speed partial image which is a partial image regarding a speed of the hand of the worker are obtained,
The hand position partial image is standardized using the average value and standard deviation of the luminance of the learning image for the hand position used in learning when generating the trained model, and resized to the specified size;
The hand speed partial image is standardized using the average value and standard deviation of the luminance of the learning image for the hand speed used in learning when generating the trained model, and resized to the specified size;
A processed hand position partial image, which is the hand position partial image after standardization and resizing, is input to the trained model to extract a feature vector of the processed hand position partial image;
A processed hand velocity partial image, which is the hand velocity partial image after standardization and resizing, is input to the trained model to extract a feature vector of the processed hand velocity partial image;
2. The estimation device according to claim 1, further comprising: a feature vector of the post-processing hand position partial image and a feature vector of the post-processing hand speed partial image; and inputting the feature vector obtained by the combination into the trained model to estimate the component work being performed by the worker during the work performance time period. The time division unit
The estimation device according to claim 1 , wherein, when a result of the displacement amount determination is used, a time period during which the displacement amount of the worker's hand is less than a threshold value is designated as the work performance time period. The time division unit
The estimation device according to claim 1 , wherein, when a result of the tool gripping state determination is used, a time period during which the worker's hands are gripping the tool is designated as the work performance time period. The time division unit
The estimation device according to claim 4, wherein, when the type of tool held by the worker's hand changes over time, each time period during which the worker's hand holds a different type of tool is designated as a different work performance time period. The time division unit
The estimation device according to claim 1 , wherein, when a result of the appearance status determination is used, a time period during which the worker's hands are present in the image is designated as the work performance time period. The time division unit
The estimation device according to claim 1, wherein at least one of the results of the displacement amount determination, the results of the tool gripping status determination, and the results of the appearance status determination is applied to the trained model to divide the work engagement time into the work performance time period and the non-work time period. Generate candidates for the work execution time period;
A learning device that generates a trained model of claim 1 by learning the difference between the true value of the work performance time period obtained by performing at least one of a process corresponding to the displacement amount determination, a process corresponding to the tool holding status determination, and a process corresponding to the appearance status determination, and the candidate value of the work performance time period. Calculating the average and standard deviation of the luminance of the learning images ;
A learning device that generates the trained model of claim 1 by learning using the calculated average value and standard deviation . a computer uses at least one of a result of a displacement amount determination that determines an amount of displacement of the hand of a worker engaged in work including a plurality of elemental tasks, each of which has non-uniform attributes, during a work engagement time during which the worker is engaged in the work, a result of a tool gripping state determination that determines a state in which the tool is gripped by the hand of the worker during the work engagement time, and a result of an appearance state determination that determines an appearance state of the hand of the worker in an image captured during the work engagement time, to divide the work engagement time into a work performance time period that is a time period during which the worker is performing any one of the plurality of elemental tasks and a non-work time period that is a time period during which the worker is not performing any one of the plurality of elemental tasks;
The computer acquires a plurality of partial images constituting a partial video that is a portion of the video captured during the work execution time period, and standardizes the plurality of partial images using an average value and a standard deviation of the luminance of a learning image of a hand used in learning when generating a trained model, and resizes the plurality of partial images to a specified size ;
Inputting a plurality of processed partial images, which are the plurality of partial images after standardization and resizing, into the trained model to extract feature vectors of the plurality of processed partial images;
An estimation method for inputting feature vectors of the plurality of processed partial images into the trained model to estimate the elemental work being performed by the worker during the work performance time period. a time division process for dividing a work engagement time into a work performance time zone during which the work worker is performing any one of the plurality of elemental tasks and a non-work time zone during which the work worker is not performing any one of the plurality of elemental tasks, using at least one of the following: a displacement amount determination result for determining an amount of displacement of the hand of a work worker during a work engagement time zone during which the work worker is engaged in a task including a plurality of elemental tasks, each of which has non-uniform attributes; a tool holding state determination result for determining a tool holding state of the hand of the work worker during the work engagement time zone; and an appearance state determination result for determining an appearance state of the hand of the work worker in an image captured during the work engagement time zone;
A plurality of partial images constituting a partial video, which is a portion of the video captured during the work execution time period, are acquired, and the plurality of partial images are standardized using an average value and a standard deviation of the luminance of the learning images of the hand used in learning when generating the trained model, and resized to a specified size;
Inputting a plurality of processed partial images, which are the plurality of partial images after standardization and resizing, into the trained model to extract feature vectors of the plurality of processed partial images;
and an estimation program that causes a computer to execute an estimation process of inputting feature vectors of the plurality of processed partial images into the trained model and estimating the elemental work being performed by the worker during the work performance time period.

Images