JP7080285B2 - Operation identification device, operation identification method and operation identification program - Google Patents

Description

The present invention relates to a technique for specifying an operation content of a subject from image data taken by the subject.

In the industrial field, there is a need for processing such as measurement of cycle time, which is the time for workers to assemble products, and analysis of work contents for detecting work omissions or non-routine work that is not routine work. Currently, these processes are mainly performed manually. Therefore, a lot of human cost is required, and only a limited range can be processed.

Patent Document 1 describes that a camera and a three-dimensional sensor attached to a human head are used to extract features of human motion and automatically perform motion analysis.

Japanese Unexamined Patent Publication No. 2016-099982 Japanese Unexamined Patent Publication No. 2017-199303 Japanese Unexamined Patent Publication No. 2017-174093 Japanese Unexamined Patent Publication No. 2016-42332

In Patent Document 1, a camera is attached to a person's head. However, in the industrial field, it is avoided to attach unnecessary objects to a part of the worker's body during the work because it may hinder the work.
It is an object of the present invention to make it possible to identify when each work constituting a series of work starts and finishes without attaching unnecessary objects to the worker's body.

The operation specifying device according to the present invention is
Regarding the video data for learning in which a worker who has performed a series of operations composed of a plurality of operations is photographed, the image data at each time constituting the video data for learning and the operation of the worker at each time are performed. An image acquisition unit that acquires label information indicating the contents, and
Targeting the image data at each time constituting the video data for learning acquired by the image acquisition unit, operation information which is skeletal information representing the posture of the worker is obtained from the image data at the target time. The skeleton extraction part to be extracted and
For the image data at each time, the pair of the operation information extracted from the image data at the target time by the skeleton extraction unit and the label information indicating the operation of the worker at the target time is learned as learning data. When the skeletal information is input, the motion information similar to the input skeletal information is specified, and a learning model that outputs the label information corresponding to the specified motion information is generated. Equipped with a learning department
The image acquisition unit acquires image data at each time constituting the target video data for the target video data obtained by photographing the target worker who has performed a series of operations composed of a plurality of operations.
The skeleton extraction unit extracts target information, which is skeleton information representing the posture of the worker, from the image data at the target time, targeting the image data at each time constituting the video data of the target.
moreover,
For the image data at each time constituting the video data of the target, the target information extracted from the image data at the target time is input to the learning model generated by the learning unit, and the learning model is used. By acquiring the output label information and specifying the operation content indicated by the acquired label information as the operation content performed by the target worker, each operation constituting the series of operations can be performed. It is equipped with an operation specifying part that specifies when it started and when it ended.

In the present invention, the target information, which is the skeletal information representing the posture of the subject, is extracted from the image data, and the motion content indicated by the motion information, which is the skeleton information similar to the target information, is set as the motion content performed by the subject. Identify. Therefore, it is possible to specify when each work constituting the series of work started and ended without attaching unnecessary objects to the worker's body.

The block diagram of the operation specifying apparatus 10 which concerns on Embodiment 1. FIG. The flowchart of the registration process which concerns on Embodiment 1. Explanatory drawing of image data which concerns on Embodiment 1. FIG. The explanatory view of the skeleton information 43 which concerns on Embodiment 1. FIG. An explanatory diagram of a registration process according to the first embodiment. The explanatory view of the operation information table 31 which concerns on Embodiment 1. FIG. The flowchart of the specific process which concerns on Embodiment 1. An explanatory diagram of a specific process according to the first embodiment. The block diagram of the operation specifying apparatus 10 which concerns on modification 1. FIG. The block diagram of the operation specifying apparatus 10 which concerns on modification 3. The block diagram of the operation specifying apparatus 10 which concerns on Embodiment 2. FIG. The flowchart of the learning process which concerns on Embodiment 2. The flowchart of the specific process which concerns on Embodiment 2. The block diagram of the operation specifying apparatus 10 which concerns on modification 5.

Embodiment 1.
*** Explanation of configuration ***
The configuration of the operation specifying device 10 according to the first embodiment will be described with reference to FIG.
The operation specifying device 10 is a computer.
The operation specifying device 10 includes hardware of a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware via a signal line and controls these other hardware.

The processor 11 is an IC (Integrated Circuit) that performs processing. Specific examples of the processor 11 are a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit).

The memory 12 is a storage device that temporarily stores data. As a specific example, the memory 12 is a SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory).

The storage 13 is a storage device for storing data. As a specific example, the storage 13 is an HDD (Hard Disk Drive). The storage 13 includes SD (registered trademark, Secure Digital) memory card, CF (Compact Flash, registered trademark), NAND flash, flexible disk, optical disk, compact disc, Blu-ray (registered trademark) disk, DVD (Digital Versaille Disk), and the like. It may be a portable recording medium.

The communication interface 14 is an interface for communicating with an external device. As a specific example, the communication interface 14 is a port of Ethernet (registered trademark), USB (Universal Serial Bus), HDMI (registered trademark, High-Definition Multimedia Interface). The communication interface 14 may be provided separately for each data to be communicated. For example, HDMI (registered trademark) may be provided for communicating image data described later, and USB may be provided for communicating label information described later.

The operation specifying device 10 includes an image acquisition unit 21, a skeleton extraction unit 22, an operation information registration unit 23, an operation specifying unit 24, and an output unit 25 as functional components. The functions of each functional component of the operation specifying device 10 are realized by software.
The storage 13 stores a program that realizes the functions of each functional component of the operation specifying device 10. This program is read into the memory 12 by the processor 11 and executed by the processor 11. As a result, the functions of each functional component of the operation specifying device 10 are realized.

Further, the storage 13 stores the operation information table 31.

In FIG. 1, only one processor 11 is shown. However, the number of processors 11 may be plural, and the plurality of processors 11 may execute programs that realize each function in cooperation with each other.
As a specific example, the operation specifying device 10 may include a CPU and a GPU as the processor 11. In this case, as will be described later, the skeleton extraction unit 22 that performs image processing is realized by the GPU, and the remaining image acquisition unit 21, operation information registration unit 23, operation identification unit 24, and output unit 25 are used. May be realized by the CPU.

*** Explanation of operation ***
The operation of the operation specifying device 10 according to the first embodiment will be described with reference to FIGS. 2 to 8.
The operation of the operation specifying device 10 according to the first embodiment corresponds to the operation specifying method according to the first embodiment. Further, the operation of the operation specifying device 10 according to the first embodiment corresponds to the processing of the operation specifying program according to the first embodiment.
The operation of the operation specifying device 10 according to the first embodiment includes a registration process and a specifying process.

The registration process according to the first embodiment will be described with reference to FIG.
(Step S11: Image acquisition process)
The image acquisition unit 21 acquires one or more sets of image data captured by the person 42 performing the target motion by the photographing device 41 and label information indicating the target motion via the communication interface 14. As shown in FIG. 3, in the first embodiment, the image data is acquired by photographing the entire body of the person 42 who is performing the target motion by the photographing device 41 from the front of the subject.
The image acquisition unit 21 writes the set of the acquired image data and the label information in the memory 12.

(Step S12: Skeleton extraction process)
The skeleton extraction unit 22 reads the image data acquired in step S11 from the memory 12. The skeleton extraction unit 22 extracts skeleton information 43 representing the posture of the person 42 as motion information from the image data. As shown in FIG. 4, in the first embodiment, the skeletal information 43 indicates the coordinates of a plurality of joints such as the neck and shoulders of the person 42, or the relative positional relationship of the plurality of joints.
The skeleton extraction unit 22 writes the extracted operation information in the memory 12.

(Step S13: Operation information registration process)
The operation information registration unit 23 reads the operation information extracted in step S12 and the label information of the same set as the image data from which the operation information is extracted from the memory 12. The operation information registration unit 23 associates the read operation information with the label information and writes it in the operation information table 31.

(Step S14: End determination process)
The skeleton extraction unit 22 determines whether or not all the sets acquired in step S11 have been processed.
When the skeleton extraction unit 22 processes all the sets, the skeleton extraction unit 22 ends the registration process. On the other hand, if there is a set that has not been processed, the skeleton extraction unit 22 returns the process to step S12 and executes the process for the next set.

By executing the registration process, a set of a plurality of operation information and label information is accumulated in the operation information table 31.
For example, as shown in FIG. 5, in step S11, the image acquisition unit 21 describes the image data at each time constituting the video data obtained by shooting the person who performed the series of operations, and the image data at that time and the time. Acquires a set with label information indicating the movement of a person indicated by image data. Then, in step S12, the skeleton extraction unit 22 extracts operation information from the image data to be processed, and in step S13, the operation information registration unit 23 associates the same set of label information and operation information with the image data to be processed. And write to the operation information table 31. As a result, as shown in FIG. 6, the associated operation information and label information are accumulated in the operation information table 31 for the operation at each time in the series of operations.
In step S11, the image acquisition unit 21 also uses the image data at each time to form the image data obtained by photographing the person who has performed the unsteady work that is not normally performed in the series of work, as well as the image data at that time. You may acquire a set with the label information which shows the action of the person which the image data of the time shows. As a result, even for non-routine work, the associated operation information and label information are accumulated in the operation information table 31 for the operation at each time.

The specific process according to the first embodiment will be described with reference to FIG. 7.
(Step S21: Image acquisition process)
The image acquisition unit 21 acquires one or more image data captured by the subject via the communication interface 14. In the first embodiment, similarly to the image data acquired in step S11, the image data acquired in step S21 is acquired by photographing the entire body of the subject from the front of the subject by the photographing device 41.
The image acquisition unit 21 writes the acquired image data to the memory 12.

(Step S22: Skeleton extraction process)
The skeleton extraction unit 22 reads the image data acquired in step S21 from the memory 12. The skeleton extraction unit 22 extracts the skeleton information 43 representing the posture of the target person from the image data as the target information.
The skeleton extraction unit 22 writes the extracted target information in the memory 12.

(Step S23: Operation specifying process)
The motion specifying unit 24 identifies the motion content indicated by the motion information, which is skeletal information similar to the target information extracted in step S22, as the motion content performed by the target person.
Specifically, the operation specifying unit 24 searches the operation information table 31 for operation information similar to the target information. Similarity means that when the skeletal information 43 indicates the coordinates of a plurality of joints, the Euclidean distance between the coordinates of the same joint is short in the target information and the motion information. Further, when the skeletal information 43 indicates the relative positional relationship between the plurality of joints, it means that the Euclidean distance between each joint indicated by the target information and the Euclidean distance between each joint indicated by the motion information are close. .. Then, the motion specifying unit 24 identifies the motion content indicated by the label information associated with the motion information hit by the search as the motion content performed by the target person.

For example, the motion specifying unit 24 calculates the degree of similarity with the target information for all the motion information stored in the motion information table 31. Then, the motion specifying unit 24 treats the motion information having the highest degree of similarity as motion information that hits the search. If there is no motion information whose similarity is higher than the threshold value, the motion specifying unit 24 may not have the motion information that hits the search.
When the relative positional relationship between specific joints characterizes the movement, weighting may be performed so that the difference in Euclidean distance for the specific joint greatly affects the degree of similarity. That is, when the skeletal information 43 indicates the coordinates of a plurality of joints, weighting is performed so that the difference in the Euclidean distance between the coordinates in the target information and the coordinates in the motion information for a specific joint greatly affects the similarity. You may go. Further, when the skeletal information 43 indicates the relative positional relationship of a plurality of joints, weighting may be performed so that the difference in Euclidean distance between specific joints greatly affects the degree of similarity.

(Step S24: Output processing)
The output unit 25 outputs the operation content specified in step S23 to a display device or the like connected via the communication interface 14. The output unit 25 may output label information indicating the operation content.
If there is no operation information that hits the search, the output unit 25 outputs information indicating that the operation content cannot be specified.

(Step S25: End determination process)
The skeleton extraction unit 22 determines whether or not all the image data acquired in step S21 have been processed.
When the skeleton extraction unit 22 has processed all the image data, the skeleton extraction unit 22 ends the registration process. On the other hand, if there is unprocessed image data, the skeleton extraction unit 22 returns the process to step S22 and executes the process for the next image data.

For example, as shown in FIG. 8, in step S21, the image acquisition unit 21 acquires image data at each time that constitutes video data obtained by photographing a person who has performed a series of operations. Then, in step S22, the skeleton extraction unit 22 extracts the target information from the image data to be processed, and in step S23, the operation information registration unit 23 searches for the operation information similar to the target information and specifies the operation content. .. This makes it possible to specify the operation content at each time in a series of operations.
At this time, it is also possible to specify when the target work was started and when it was completed. In addition, when the subject performs non-routine work during a series of work, it is possible to identify that the non-routine work has been performed.

*** Effect of Embodiment 1 ***
As described above, the motion specifying device 10 according to the first embodiment extracts the target information, which is the skeleton information representing the posture of the target person, from the image data obtained by photographing the target person from the front, and has a skeleton similar to the target information. The operation content indicated by the operation information, which is information, is specified as the operation content performed by the target person. Therefore, the operation specifying device 10 according to the first embodiment can analyze a series of operations by inputting video data including a plurality of image data and specifying the operation contents for each image data. .. As a result, it is possible to perform processes such as cycle time measurement and work content analysis without attaching unnecessary objects to the worker's body.

*** Other configurations ***
<Modification 1>
In the first embodiment, as shown in FIG. 1, the operation specifying device 10 is one device. However, the operation specifying device 10 may be a system composed of a plurality of devices.
As a specific example, as shown in FIG. 9, the operation specifying device 10 may be a system composed of a registration device having a function related to registration processing and a specific device having a function related to specific processing. In this case, the operation information table 31 may be stored in a storage device provided outside the registration device and the specific device, or may be stored in the storage of either the registration device and the specific device.
In FIG. 9, the hardware in the registration device and the specific device is omitted. Similar to the operation specifying device 10, the registration device and the specifying device include a processor, a memory, a storage, and a communication interface as hardware.

<Modification 2>
In the first embodiment, the data photographed by the photographing apparatus 41 is used as the image data. However, as the image data, three-dimensional image data obtained by a sensor such as a depth sensor may be used.

<Modification 3>
In the first embodiment, each functional component is realized by software. However, as a modification 3, each functional component may be realized by hardware. The difference between the third modification and the first embodiment will be described.

With reference to FIG. 10, the configuration of the operation specifying device 10 according to the modification 3 will be described.
When each functional component is realized by hardware, the operation specifying device 10 includes an electronic circuit 15 in place of the processor 11, the memory 12, and the storage 13. The electronic circuit 15 is a dedicated circuit that realizes the functions of each functional component, the memory 12, and the storage 13.

Examples of the electronic circuit 15 include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA (Gate Array), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable Gate Array). is assumed.
Each functional component may be realized by one electronic circuit 15, or each functional component may be distributed and realized by a plurality of electronic circuits 15.

<Modification example 4>
As a modification 4, some functional components may be realized by hardware, and other functional components may be realized by software.

The processor 11, the memory 12, the storage 13, and the electronic circuit 15 are referred to as a processing circuit. That is, the function of each functional component is realized by the processing circuit.

Embodiment 2.
The second embodiment is different from the first embodiment in that the learning model 32 is generated based on the operation information and the label information, and the label information corresponding to the target information is specified by the learning model 32. In the second embodiment, these different points will be described, and the same points will be omitted.

*** Explanation of configuration ***
The configuration of the operation specifying device 10 according to the second embodiment will be described with reference to FIG. 11.
The operation specifying device 10 is different from the operation specifying device 10 shown in FIG. 1 in that the learning unit 26 is provided instead of the operation information registration unit 23. Further, the operation specifying device 10 is different from the operation specifying device 10 shown in FIG. 1 in that the storage 13 stores the learning model 32 instead of the operation information table 31.

*** Explanation of operation ***
The operation of the operation specifying device 10 according to the second embodiment will be described with reference to FIGS. 12 to 13.
The operation of the operation specifying device 10 according to the second embodiment corresponds to the operation specifying method according to the second embodiment. Further, the operation of the operation specifying device 10 according to the second embodiment corresponds to the processing of the operation specifying program according to the second embodiment.
The operation of the operation specifying device 10 according to the second embodiment includes a learning process and a specific process.

The learning process according to the second embodiment will be described with reference to FIG.
The process from step S31 to step S32 is the same as the process from step S11 to step S12 in FIG. Further, the process of step S34 is the same as the process of step S14 of FIG.

(Step S33: Learning model generation process)
The learning unit 26 learns a plurality of sets of the motion information extracted in step S32 and the label information of the same set as the image data from which the motion information is extracted as learning data. As a result, when the skeleton information 43 is input, the learning unit 26 identifies the motion information similar to the input skeleton information 43 and outputs the learning model 32 corresponding to the specified motion information. Generate. As the learning method based on the learning data, an existing machine learning model or the like may be used. The learning unit 26 writes the generated learning model 32 to the storage 13.
When the learning model 32 has already been generated, the learning unit 26 updates the learning model 32 by giving learning data to the generated learning model 32.

In step S31, not only the pair of the image data and the label information but also the image data may be input. In this case, the motion information is extracted from the image data in step S32, and only the motion information is given to the learning model 32 as learning data in step S33. In this way, it is possible to obtain a certain learning effect even when the label information does not exist.

The specific process according to the second embodiment will be described with reference to FIG.
The process from step S41 to step S42 is the same as the process from step S21 to step S22 in FIG. Further, the processing from step S44 to step S45 is the same as the processing from step S24 to step S25 in FIG.

(Step S43: Operation specifying process)
The operation specifying unit 24 inputs the target information extracted in step S42 into the learning model 32 stored in the storage 13, and acquires the label information output from the learning model 32. Then, the operation specifying unit 24 specifies the operation content indicated by the acquired label information as the operation content performed by the target person. That is, the motion specifying unit 24 specifies the motion content indicated by the label information inferred from the target information by the learning model 32 as the motion content performed by the target person.

*** Effect of Embodiment 2 ***
As described above, the motion specifying device 10 according to the second embodiment generates the learning model 32, and specifies the label information corresponding to the target information by the learning model 32. Therefore, it is possible to efficiently specify the label information corresponding to the target information.

*** Other configurations ***
<Modification 5>
In the second embodiment, as shown in FIG. 11, the operation specifying device 10 is one device. However, as in the first modification, the operation specifying device 10 may be a system composed of a plurality of devices.
As a specific example, as shown in FIG. 14, the operation specifying device 10 may be a system composed of a registration device having a function related to learning processing and a specific device having a function related to specific processing. In this case, the learning model 32 may be stored in a storage device provided outside the learning device and the specific device, or may be stored in the storage of either the learning device and the specific device.
In FIG. 14, the hardware in the registration device and the specific device is omitted. Similar to the operation specifying device 10, the learning device and the specifying device include a processor, a memory, a storage, and a communication interface as hardware.

<Modification 6>
In the second embodiment, as shown in FIG. 11, the operation specifying device 10 includes a processor 11, a memory 12, a storage 13, and a communication interface 14 as hardware. The operation specifying device 10 may include a CPU, a GPU, a processor for learning processing, and a processor for inference processing as the processor 11. In this case, the skeleton extraction unit 22 that performs image processing is realized by the GPU, and the learning unit 26 related to the learning of the learning model 32 is realized by the learning processing processor, and the operation specification that makes inferences by the learning model 32 is performed. The unit 24 may be realized by a processor for inference processing, and the remaining image acquisition unit 21 and the learning unit 26 may be realized by a CPU.

The above can be summarized as follows.
The operation specifying device according to the first and second embodiments is
An image acquisition unit that acquires image data about the target person,
A skeleton extraction unit that extracts target information, which is skeleton information representing the posture of the target person, from the image data acquired by the image acquisition unit.
It is provided with an operation specifying unit that specifies the operation content indicated by the operation information, which is the skeleton information similar to the target information extracted by the skeleton extraction unit, as the operation content performed by the target person.

The skeletal information indicates the coordinates of a plurality of joints of the subject.

The skeletal information indicates the relative positional relationship between the plurality of joints of the subject.

The operation specifying unit searches for the operation information similar to the target information from a storage device that stores the operation information in association with the label information indicating the work content, and associates the operation information with the operation information that hits the search. The operation content indicated by the label information is specified as the operation content performed by the target person.

The operation specifying device further
It is provided with an operation information registration unit that associates the operation information, which is the skeleton information extracted from the image data of a person performing the target operation, with the label information indicating the target operation, and registers the operation information in the storage device. ..

The operation specifying device further
By learning a plurality of sets of the motion information, which is the skeletal information extracted from the image data of the person performing the target motion, and the label information indicating the target motion as training data, the skeletal information can be obtained. When input, it is provided with a learning unit that identifies the operation information similar to the input skeleton information and generates a learning model that outputs the label information corresponding to the specified operation information.
The motion specifying unit inputs the target information into the learning model generated by the learning unit, acquires the label information output from the learning model, and obtains the operation content indicated by the acquired label information. , The operation content performed by the subject is specified.

The image acquisition unit acquires video data including a plurality of image data about the target person, and obtains video data.
The skeleton extraction unit extracts the target information from the target image data for each of the plurality of image data included in the video data.
The operation specifying unit specifies the operation content performed by the target person indicated by the target image data based on the target information extracted from the target image data for each of the plurality of image data.

The operation specifying method according to the first and second embodiments is
The image acquisition unit of the operation identification device acquires image data about the target person, and
The skeleton extraction unit of the motion specifying device extracts the target information, which is the skeleton information representing the posture of the target person, from the image data.
The operation specifying unit of the operation specifying device specifies the operation content indicated by the operation information, which is the skeleton information similar to the target information, as the operation content performed by the target person.

The operation specifying program according to the first and second embodiments is
The image acquisition process for acquiring image data about the target person and the image acquisition unit
A skeleton extraction process in which the skeleton extraction unit extracts target information, which is skeleton information representing the posture of the target person, from the image data acquired by the image acquisition process.
The motion specifying unit performs an motion specifying process for specifying the motion content indicated by the motion information, which is the skeleton information similar to the target information extracted by the skeleton extraction process, as the motion content performed by the target person. Make the computer function as an operation identification device.

10 operation identification device, 11 processor, 12 memory, 13 storage, 14 communication interface, 15 electronic circuit, 21 image acquisition unit, 22 skeleton extraction unit, 23 operation information registration unit, 24 operation identification unit, 25 output unit, 26 learning unit. , 31 motion information table, 32 learning model, 41 imaging device, 42 people, 43 skeleton information.

Claims (4)

Regarding the video data for learning in which a worker who has performed a series of operations composed of a plurality of operations is photographed, the image data at each time constituting the video data for learning and the operation of the worker at each time are performed. An image acquisition unit that acquires label information indicating the contents, and
The skeletal information representing the posture of the worker from the image data at the target time for the image data at each time constituting the video data for learning acquired by the image acquisition unit . A skeletal extraction unit that extracts motion information, which is skeletal information indicating the relative positional relationship of multiple joints of a worker ,
For the image data at each time, the pair of the operation information extracted from the image data at the target time by the skeleton extraction unit and the label information indicating the operation of the worker at the target time is learned as learning data. When the skeletal information is input, the motion information similar to the input skeletal information is specified, and a learning model that outputs the label information corresponding to the specified motion information is generated. Equipped with a learning department
The image acquisition unit acquires image data at each time constituting the target video data for the target video data obtained by photographing the target worker who has performed a series of operations composed of a plurality of operations.
The skeleton extraction unit is targeting the image data at each time constituting the video data of the target, and is skeleton information representing the posture of the worker from the image data at the target time, and is the skeleton information of the worker. Target information, which is skeletal information showing the relative positional relationship of multiple joints, is extracted.
moreover,
For the image data at each time constituting the video data of the target, the target information extracted from the image data at the target time is input to the learning model generated by the learning unit, and the learning model is used. By acquiring the output label information and specifying the operation content indicated by the acquired label information as the operation content performed by the target worker, each operation constituting the series of operations can be performed. An operation specifying device including an operation specifying unit that specifies when it started and when it ended. The operation according to claim 1, wherein the learning model outputs not only label information indicating operation contents constituting the series of operations but also label information indicating unsteady operations which are operation contents not constituting the series of operations. Specific device. Regarding the video data for learning in which the image acquisition unit of the operation specifying device captures a worker who has performed a series of operations composed of a plurality of operations, the image data at each time constituting the video data for learning and the image data at each time are used. The label information indicating the operation content of the worker at each time is acquired, and the label information is acquired.
The skeleton extraction unit of the motion specifying device targets the image data at each time constituting the video data for learning, and is skeleton information representing the posture of the worker from the image data at the target time. , Extracting motion information, which is skeletal information indicating the relative positional relationship of a plurality of joints of the worker ,
The learning unit of the operation specifying device targets the image data at each time, the operation information extracted from the image data at the target time by the skeleton extraction unit, and a label indicating the operation of the worker at the target time. When the skeletal information is input by learning a set with the information as training data, the operation information similar to the input skeletal information is specified, and the label corresponding to the specified operation information is specified. Generate a learning model that outputs information
The image acquisition unit of the operation specifying device is the image data of each time constituting the target video data with respect to the target video data obtained by photographing the target worker who has performed a series of operations composed of a plurality of operations. To get,
The skeleton extraction unit of the operation specifying device is targeting the image data at each time constituting the video data of the target, and is skeleton information representing the posture of the worker from the image data at the target time. , Extract target information which is skeletal information showing the relative positional relationship of a plurality of joints of the worker .
The operation specifying unit of the operation specifying device inputs the target information extracted from the image data of the target time into the learning model for the image data at each time constituting the video data of the target, and the above-mentioned. By acquiring the label information output from the learning model and specifying the operation content indicated by the acquired label information as the operation content performed by the target worker, the series of operations is configured. A behavioral identification method that identifies when each task starts and ends. Regarding the video data for learning in which the image acquisition unit has taken a picture of a worker who has performed a series of operations composed of a plurality of operations, the image data at each time constituting the video data for learning and the image data at each time An image acquisition process for acquiring label information indicating the operation content of the worker, and
The skeleton extraction unit targets the image data at each time constituting the video data for learning acquired by the image acquisition process, and from the image data at the target time, the skeleton information representing the posture of the worker. The skeleton extraction process for extracting motion information, which is skeletal information indicating the relative positional relationship between a plurality of joints of the worker ,
For the image data at each time, the pair of the operation information extracted from the image data at the target time by the skeleton extraction process and the label information indicating the operation of the worker at the target time is learned as training data. When the skeletal information is input, the motion information similar to the input skeletal information is specified, and a learning model that outputs the label information corresponding to the specified motion information is generated. Perform learning processing and
In the image acquisition process, with respect to the target video data obtained by shooting the target worker who has performed a series of operations composed of a plurality of operations, the image data at each time constituting the target video data is acquired.
In the skeleton extraction process, the image data at each time constituting the video data of the target is targeted, and the skeleton information representing the posture of the worker from the image data at the target time is the skeleton information of the worker. Target information, which is skeletal information showing the relative positional relationship of multiple joints, is extracted.
moreover,
The operation specifying unit inputs the target information extracted from the image data at the target time into the learning model generated by the learning process for the image data at each time constituting the video data of the target. By acquiring the label information output from the learning model and specifying the operation content indicated by the acquired label information as the operation content performed by the target worker, the series of operations can be performed. An operation specifying program that makes a computer function as an operation specifying device that performs an operation specifying process that specifies when each of the constituent tasks started and ended.

Images