WO2022153597A1 - Task level conversion device, task level conversion method, and task level conversion program - Google Patents

Abstract

Provided is a task level conversion device capable of converting a plurality of low-level tasks into high-level tasks. This task level conversion device comprises: a generation unit 13 that generates sequence data of target operation information 19b that associates a target operation detected from time-series information on tasks performed by an operator with accompanying information that accompanies the target operation; and a conversion/output unit 14 that converts the sequence data of the target operation information 19b into high-level task sequence data including a plurality of low-level tasks and outputs the high-level task sequence data on the basis of the sequence data of the target operation information 19b and a trained model 19d that converts low-level task sequence data specified by a combination of the target operation and the accompanying information into the high-level task sequence data.

Description

Work level conversion device, work level conversion method and work level conversion program

The present invention relates to a work level conversion device, a work level conversion method, and a work level conversion program.

At production sites such as factories, it is performed to analyze the movements of workers, recognize the work performed by the workers, and evaluate the contents of the work. For example, in Patent Document 1 below, an object is associated with a body part of a worker who has worked with the object, the work performed by the worker is recognized, and the work is performed based on the recognition result. A work recognition device for evaluating the content is disclosed.

Japanese Patent No. 6444573

By the way, the work performed at the production site can be divided into work levels with different particle sizes. For example, the boxing work can be divided into a work of fixing the box to a jig and a work of storing the product in the box. In this case, the boxing work is the upper level work, and the work of fixing the box to the jig and the work of storing the product in the box are the lower level work. The work of fixing the box to the jig, which is divided into the lower level work, can be further divided into the work of gripping the box, the work of transporting the box, and the work of adjusting the box. In this case, the work of fixing the box to the jig is the work of the upper level, and the work of gripping the box, the work of transporting the box, and the work of adjusting the box are the work of the lower level.

When evaluating the content of work, the content of evaluation may be set for each work level. Therefore, when recognizing work, if different levels of work can be recognized, more effective evaluation can be performed.

However, in Patent Document 1, the work of the same level is only recognized, and the work to be recognized cannot be converted into the work of a different level.

Here, in Japanese Patent Application No. 2019-200826 by the applicant of the present application, there is a method of converting upper-level work into a plurality of lower-level work by dividing the work based on the target operation into work for each part. Have been described. However, there is no description of a method for converting a plurality of lower-level tasks into higher-level tasks.

Therefore, the present invention provides a work level conversion device, a work level conversion method, and a work level conversion program capable of converting a plurality of lower level work into higher level work.

The work level conversion device according to one aspect of the present disclosure generates series data of target motion information in which the target motion detected from the time series information related to the work of the worker and the incidental information incidental to the target motion are associated with each other. The generator, the converter that converts the lower-level work series data specified by the combination of the target operation and incidental information into the upper-level work series data composed of multiple lower-level works, and the target operation information. Based on the series data, it includes a conversion output unit that converts the series data of the target operation information into higher-level work series data and outputs the data.

According to this aspect, the series data of the target motion information is generated based on the target motion and incidental information detected from the time series information related to the worker's motion, and the lower level specified by the combination of the target motion and the incidental information. Based on the converter that converts the work series data to the higher level work series data and the series data of the target operation information, the series data of the target operation information can be converted into the higher level work series data and output.

In the above embodiment, the converter uses the work information indicating the correspondence between the lower level work series data and the upper level work series data as the teacher data, and inputs the series data of the target operation information as the input to the upper level work series. It may be composed of a trained model trained to be converted into data and output.

This makes it possible to acquire higher-level work series data by inputting the series data of the target motion information into the trained model.

In the above aspect, the conversion output unit may further output start timing information indicating whether or not higher-level work has been started by the target operation for each target operation.

This makes it possible to identify when higher level work has started.

In the above aspect, the incidental information may include information that can be recognized from the time-series information and information that is annotated in the time-series information.

This makes it possible to improve the accuracy of identifying the target motion.

The work level conversion method according to another aspect of the present disclosure is a method executed by a processor, and corresponds to a target operation detected from time-series information related to a worker's work and incidental information incidental to the target operation. Generating the series data of the attached target operation information and converting the lower level work series data specified by the combination of the target operation and incidental information into the upper level work series data composed of multiple lower level work. This includes converting the sequence data of the target motion information into higher-level work sequence data and outputting the sequence data based on the converter to be converted and the sequence data of the target motion information.

According to this aspect, the series data of the target motion information is generated based on the target motion and incidental information detected from the time series information related to the worker's motion, and the lower level specified by the combination of the target motion and the incidental information. Based on the converter that converts the work series data to the higher level work series data and the series data of the target operation information, the series data of the target operation information can be converted into the higher level work series data and output.

The work level conversion program according to another aspect of the present disclosure is a sequence of target motion information in which a computer associates a target motion detected from time series information related to a worker's work with incidental information incidental to the target motion. A generator that generates data, a converter that converts lower-level work series data specified by a combination of target actions and incidental information into higher-level work series data composed of multiple lower-level works, and a target. Based on the series data of the operation information, it functions as a conversion output unit that converts the series data of the target operation information into higher-level work series data and outputs it.

According to this aspect, the series data of the target motion information is generated based on the target motion and incidental information detected from the time series information related to the worker's motion, and the lower level specified by the combination of the target motion and the incidental information. Based on the converter that converts the work series data to the higher level work series data and the series data of the target operation information, the series data of the target operation information can be converted into the higher level work series data and output.

According to the present invention, it is possible to provide a work level conversion device, a work level conversion method, and a work level conversion program capable of converting a plurality of lower level work into higher level work.

It is a figure which illustrates the outline of the system including the work level conversion apparatus which concerns on embodiment of this invention. It is a figure which illustrates the functional structure of the system including the work level conversion apparatus. It is a figure which illustrates the target operation information stored in the work level conversion apparatus. It is a figure which illustrates the work information stored in the work level conversion apparatus. It is a figure which illustrates the hardware composition of the work level conversion apparatus. It is a flowchart for demonstrating an example of the work level conversion process executed by the work level conversion apparatus.

Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, referred to as “the present embodiment”) will be described with reference to the drawings. In each figure, those having the same reference numerals have the same or similar configurations.

§1 Application example First, an example of a situation in which the present invention is applied will be described with reference to FIG. The system 100 including the work level conversion device 10 according to the present embodiment captures the operation of the worker A performed in a certain work area R with the image sensors 20a, 20b, and 20c, and acquires the captured moving image for the work level conversion. The device 10 outputs the upper level work based on the target operation corresponding to the lower level work detected from the moving image. The upper and lower work levels will be described below.

When work levels with different particle sizes are set, the work set to the coarser (higher) work level becomes the higher level work, and the work set to the finer (lower) work level becomes the lower level work. It will be work.

For example, if the "boxing work" is composed of "the work of fixing the box to the jig" and "the work of storing the product in the box", the "boxing work" becomes the higher level work and the "box". "The work of fixing the product to the jig" and "the work of storing the product in the box" are the lower level work.

In addition, when the lower level "work of fixing the box to the jig" is composed of "work of gripping the box", "work of transporting the box", and "work of adjusting the box", " "The work of fixing the box to the jig" is the work of the upper level, and "the work of gripping the box", "the work of transporting the box" and "the work of adjusting the box" are the lower level work.

Here, gripping is defined as an action of moving a hand to a work object and grasping it, and corresponds to, for example, an action of going to grasp a part or an instrument. Transport is defined as the action of moving a work object to a destination, and corresponds, for example, to the action of transporting parts or instruments toward an assembled product. Adjustment is defined as an action that shifts work to a target state, and corresponds, for example, an action of assembling parts.

In the present invention, a model is trained to learn the correspondence between lower-level work series data and higher-level work series data, a trained model for work level conversion is generated, and the trained model is detected from a moving image. The work series data is input, and the converted higher level work series data is output from the trained model.

This makes it possible to convert work series data consisting of a plurality of works detected from a moving image into higher level work series data.

That is, according to the work level conversion device 10 according to the present embodiment, it is possible to convert a plurality of lower level work into higher level work.

§2 Configuration example [Functional configuration]
Next, with reference to FIG. 2, an example of the functional configuration of the system 100 including the work level conversion device 10 according to the present embodiment will be described. The system 100 includes three image sensors 20a, 20b, and 20c, and a work level conversion device 10. In the following, when it is not necessary to separately describe the three image sensors 20a, 20b, and 20c, they are described as the image sensor 20. The work level conversion device 10 has, for example, an acquisition unit 11, a detection unit 12, a generation unit 13, a conversion output unit 14, and a storage unit 19 as functional configurations. The storage unit 19 stores, for example, the moving image 19a, the target motion information 19b, the work information 19c, and the trained model 19d.

Details of each functional configuration of the system 100 including the work level conversion device 10 will be sequentially described below.

<Image sensor>
The image sensor 20 is, for example, a general-purpose camera, and captures a moving image including a scene in which the worker A is operating in the work area R. The image sensor 20 has, for example, a detection unit as a functional configuration. The detection unit detects the movement of the worker A and outputs a moving image showing the movement as time-series information.

Each image sensor 20a, 20b, 20c is arranged so that the entire area of the work area R and the whole body of the worker A can be photographed. In this case, for example, each of the image sensors 20a, 20b, 20c may be arranged so that the entire area of the work area R and the whole body of the worker A can be photographed, or each of the image sensors 20a, 20b, 20c may be arranged. However, the work area R and a part of the worker A may be photographed, and the moving images thereof may be combined so as to cover the entire work area R and the whole body of the worker A. Further, the image sensors 20a, 20b, and 20c may photograph the work area R and the worker A at different magnifications. It is not necessary to provide three image sensors 20, but at least one or more image sensors 20 may be provided.

<Acquisition department>
The acquisition unit 11 acquires time-series information (moving image in the present embodiment) related to the operation performed by the worker A from the image sensor 20. The time-series information acquired by the acquisition unit 11 is transmitted to the storage unit 19 and stored as a moving image 19a. The acquisition unit 11 also acquires the moving image 19a stored in the storage unit 19.

<Detector>
The detection unit 12 recognizes the position on the image in which the hand of the worker A exists from the moving image 19a, and detects the target motion based on the movement at the recognized position. As the target operation, for example, gripping, transporting, and adjusting can be set. Further, as a movement corresponding to gripping, for example, a movement for grasping a box or a part is set, as a movement corresponding to transportation, for example, a movement for carrying a box or a part is set, and as a movement corresponding to adjustment, for example, a box. Alternatively, the movement of fixing or incorporating the component can be set. The target motion and the motion corresponding to the target motion can be appropriately set according to the work content, and it is preferable to store the correspondence between them in the storage unit 19 in advance.

Specifically, the detection unit 12 recognizes the position of the worker A's hand on the moving image 19a, and when it is determined that the worker A is moving to grab the part, the detection unit 12 detects the grip as the target motion. do.

<Generator>
The generation unit 13 generates the target motion information 19b in which the target motion detected by the detection unit 12 and the incidental information incidental to the target motion are associated with each other. The target operation information 19b will be described with reference to FIG.

The target operation information 19b is series data having, for example, a NO item, a target operation item, and ancillary information items as data items. The NO item stores the order in which the target operation occurs. It should be noted that the time in which the target operation occurs and the elapsed time from the start of the work may be stored instead of the order in which the target operation occurs. The target action item stores the content of the target action. The incidental information item stores incidental information incidental to the target operation. As the incidental information, for example, information that can be recognized from the moving image 19a and information that is annotated in the moving image 19a can be included. Specifically, it corresponds to an object such as a part or a box, a contact separation state between the part or the box and a hand, a worker, a work process, information known at the time of shooting a moving image, and the like.

For example, in the target motion information in which the NO in FIG. 3 is "1", "grasping" is stored as the target motion, and "box" is stored as incidental information. From this target operation information, it can be specified that the worker is performing the work of gripping the box at the time of NO "1". Further, in the target motion information in which NO is "4", "grasping" is stored as the target motion, and "part 1" is stored as incidental information. From this target operation information, it can be specified that the operator is performing the work of gripping the component 1 at the time of NO "4". Further, in the target operation information in which NO is "6", "transportation" is stored as the target operation, and "part 1" is stored as incidental information. From this target operation information, it can be identified that the worker is performing the work of gripping the box at the time of NO "6".

<Conversion output unit>
Returning to the description of FIG. The conversion output unit 14 inputs the series data of the target operation information 19b generated by the generation unit 13 into the trained model 19d, and outputs the converted higher-level work series data from the trained model 19d.

The trained model 19d is a trained model for work level conversion. For example, the model is trained to generate the correspondence between the lower level work series data and the upper level work series data. As the trained model, for example, a deep learning model used in the field of natural language processing (NLP) such as Transformer can be used.

In the present embodiment, as an example, learning is performed using work information 19c indicating the correspondence between lower-level work series data specified by a combination of target motion and incidental information and higher-level work series data as teacher data. Generate finished model 19d.

The work information 19c will be described with reference to FIG. In FIG. 4, "level 1", "level 2", and "level 3" are set as work levels as an example. "Level 1" is the lowest level, "Level 3" is the highest level, and "Level 2" is the middle level.

In "Level 1", lower-level work series data in which the combination of the content of the target operation, which is the minimum unit of the work, and the incidental information incidental to the target operation is arranged in chronological order is set.

In "level 2", the upper level work series data corresponding to each time series data constituting the lower level work series data is set. In the work series data of "level 2", start timing information indicating the timing at which each work of "level 2" is started is set. For example, "1" is set for the data for which the work has been started, and "0" is set for the other data.

For example, the three data in the upper part of level 2 are common in the work of "fixing the box to the jig", and "1" is set in the start timing information of the data in the uppermost part. Therefore, by specifying the work in which "1" is set in the start timing information, it is possible to specify the timing when the work of "fixing the box to the jig" is started.

In the five data at the bottom of level 2, the work of "putting part 1 in the box" and the work of "putting part 2 in the box" are arranged alternately. For example, "putting part 1 in the box" with the right hand. Such work series data is set when the work is performed and the work of "putting the part 2 in the box" is performed in parallel with the left hand. In such work, the start timing information corresponding to the timing when the work of "putting the part 1 in the box" is started with the right hand and the timing when the work of "putting the part 2 in the box" is started with the left hand is " 1 ”will be set.

If each work item includes "level 1" incidental information as in "level 2", the setting of the incidental information can be omitted.

In "level 3", the highest level work series data corresponding to each time series data constituting the work series data of "level 1" and "level 2" is set. When one work is set as the highest level as in "level 3", each setting of incidental information and start timing information can be omitted.

Here, when the conversion output unit 14 outputs the high-level work series data after conversion, the conversion output unit 14 may further output the start timing information for each work (target operation).

[Hardware configuration]
Next, an example of the hardware configuration of the work level conversion device 10 according to the present embodiment will be described with reference to FIG. The work level conversion device 10 inputs a CPU (Central Processing Unit) 10a corresponding to an arithmetic unit, a RAM (Random Access Memory) 10b and a ROM (Read only Memory) 10c corresponding to a storage unit 19, and a communication device 10d. It has a device 10e and a display device 10f. Each of these configurations is connected so that data can be transmitted and received to and from each other via a bus. Although the case where the work level conversion device 10 is composed of one computer will be described in the present embodiment, the work level conversion device 10 may be realized by using a plurality of computers.

The CPU 10a executes a program stored in the RAM 10b or the ROM 10c, and functions as a control unit that calculates and processes data. The CPU 10a receives various input data from the input device 10e and the communication device 10d, displays the result of calculating the input data on the display device 10f, and stores it in the RAM 10b or ROM 10c.

The RAM 10b is composed of, for example, a semiconductor storage element and stores rewritable data. The ROM 10c is composed of, for example, a semiconductor storage element, and stores readable and non-rewritable data.

The communication device 10d is an interface for connecting the work level conversion device 10 to an external device. The communication device 10d is connected to the image sensor 20 by, for example, a communication network such as a LAN (Local Area Network) or the Internet, and receives a moving image from the image sensor 20.

The input device 10e is an interface for receiving data input from the user, and may include, for example, a keyboard, a mouse, and a touch panel.

The display device 10f is an interface for visually displaying the calculation result or the like by the CPU 10a, and can be configured by, for example, an LCD (Liquid Crystal Display).

The program may be stored in a storage medium readable by a computer such as RAM 10b or ROM 10c and provided, or may be provided via a communication network connected by the communication device 10d. The work level conversion device 10 operates the acquisition unit 11, the detection unit 12, the generation unit 13, and the conversion output unit 14 shown in FIG. 2 by executing the program by the CPU 10a. It should be noted that these physical configurations are examples and do not necessarily have to be independent configurations. For example, the work level conversion device 10 may include an LSI (Large-Scale Integration) in which the CPU 10a and the RAM 10b or ROM 10c are integrated.

§3 Operation example FIG. 6 is a flowchart showing an example of a work level conversion process executed by the work level conversion device 10 according to the present embodiment.

First, the acquisition unit 11 of the work level conversion device 10 acquires a moving image, which is time-series information regarding the operation of the worker A, from the image sensor 20 (step S101). The acquired moving image is stored in the storage unit 19 as the moving image 19a.

Subsequently, the detection unit 12 of the work level conversion device 10 recognizes the position on the image in which the hand of the worker A exists based on the moving image 19a, and detects the target motion (step S102).

Subsequently, the generation unit 13 of the work level conversion device 10 generates the target motion information 19b based on the target motion detected in step S102 and the incidental information incidental to the target motion (step S103). The generated target operation information 19b is stored in the storage unit 19.

Subsequently, the conversion output unit 14 of the work level conversion device 10 inputs the series data of the target operation information 19b generated in step S103 into the trained model 19d, and inputs the converted higher level work series data. Output from the trained model 19d (step S104). Then, the work level conversion process is completed.

As described above, according to the work level conversion device 10 according to the present embodiment, the series data of the target motion information 19b is generated based on the target motion and incidental information detected from the moving image 19a relating to the motion of the worker A. The sequence of the target motion information 19b based on the trained model 19d that converts the lower level work sequence data specified by the combination of the target motion and the incidental information into the upper level work sequence data and the sequence data of the target motion information 19b. Data can be converted to higher-level work series data and output.

Therefore, according to the work level conversion device 10 according to the present embodiment, it is possible to convert a plurality of lower level work into higher level work.

§4 Modifications The present invention is not limited to the above-described embodiment, and can be implemented in various other forms without departing from the gist of the present invention. For this reason, the above embodiments are merely exemplary in all respects and are not to be construed in a limited manner.

For example, in the above-described embodiment, a trained model (transducer) is used when converting lower-level work series data to higher-level work series data, but the present invention is not limited to this. For example, the work information table (transducer) as shown in FIG. 4 may be used to convert the lower level work series data into the upper level work series data.

Further, in the above-described embodiment, the case where the time-series information is a moving image has been described, but the time-series information is not limited to the moving image. Specifically, the time-series information is information on coordinate values indicating the movement of the worker A measured by motion capture provided in place of the image sensor 20, or an acceleration sensor or gyro sensor is used instead of the image sensor 20. It may be information indicating the operation of the worker A measured by being attached to the worker A. Further, the time-series information is information indicating the change state of the pressure value measured by the pressure sensor arranged in the work area R instead of the image sensor 20, or the photoelectric arranged in the work area R instead of the image sensor 20. It may be information indicating a change in an event detected by a sensor. Further, the time series information is not limited to using each of the above information individually, and two or more of the above information may be combined.

Moreover, the embodiment of the present invention can also be described as the following appendix. However, the embodiments of the present invention are not limited to the embodiments described in the following appendices. Moreover, the embodiment of the present invention may be a form in which the description between the appendices is replaced or combined.

[Appendix 1]
A generation unit (13) that generates series data of the target motion information (19b) in which the target motion detected from the time series information related to the worker's work and the incidental information incidental to the target motion are associated with each other.
A converter that converts lower-level work series data specified by a combination of the target operation and the incidental information into higher-level work series data composed of a plurality of the lower-level works, and the target operation information ( A conversion output unit (14) that converts the series data of the target operation information (19b) into the higher-level work series data and outputs the data based on the series data of 19b).
(10).

[Appendix 2]
The converter uses work information (19c) indicating the correspondence between the lower level work series data and the upper level work series data as teacher data, and inputs the series data of the target operation information (19b) as input. , Consists of a trained model (19d) trained to convert and output to the higher level work sequence data.
The work level conversion device (10) according to Appendix 1.

[Appendix 3]
The conversion output unit (14) further outputs start timing information indicating whether or not a higher level work is started by the target operation for each target operation.
The work level conversion device (10) according to Appendix 2.

[Appendix 4]
The incidental information includes information that can be recognized from the time-series information and information that is annotated with the time-series information.
The work level conversion device (10) according to Appendix 2.

[Appendix 5]
The method executed by the processor
Generating the series data of the target motion information (19b) in which the target motion detected from the time series information related to the worker's work and the incidental information incidental to the target motion are associated with each other.
A converter that converts lower-level work series data specified by a combination of the target operation and the incidental information into higher-level work series data composed of a plurality of the lower-level works, and the target operation information ( Based on the series data of 19b), the series data of the target operation information (19b) is converted into the higher level work series data and output.
Work level conversion method including.

[Appendix 6]
Computer,
A generator (13) that generates series data of the target motion information (19b) in which the target motion detected from the time series information related to the worker's work and the incidental information incidental to the target motion are associated with each other.
A converter that converts lower-level work series data specified by a combination of the target operation and the incidental information into higher-level work series data composed of a plurality of the lower-level works, and the target operation information ( A conversion output unit (14), which converts the series data of the target operation information (19b) into the higher-level work series data and outputs it based on the series data of 19b).
A work level conversion program that functions as.

10 ... Work level conversion device, 10a ... CPU, 10b ... RAM, 10c ... ROM, 10d ... Communication device, 10e ... Input device, 10f ... Display device, 11 ... Acquisition unit, 12 ... Detection unit, 13 ... Generation unit, 14 ... conversion output unit, 19 ... storage unit, 19a ... moving image, 19b ... target operation information, 19c ... work information, 19d ... trained model, 20 ... image sensor, 100 ... system, A ... worker, R ... work area

Claims (6)

1. A generator that generates series data of target motion information that associates the target motion detected from the time series information related to the worker's work with the incidental information incidental to the target motion.
   A converter that converts lower-level work sequence data specified by a combination of the target motion and the incidental information into higher-level work sequence data composed of a plurality of the lower-level tasks, and the target motion information. Based on the series data, the conversion output unit that converts the series data of the target operation information into the higher level work series data and outputs it.
   A work level converter equipped with.
2. The converter uses work information indicating the correspondence between the lower level work sequence data and the upper level work sequence data as teacher data, and uses the sequence data of the target operation information as input to perform the upper level work. Consists of trained models trained to convert to series data and output
   The work level conversion device according to claim 1.
3. The conversion output unit further outputs start timing information indicating whether or not a higher level work is started by the target operation for each target operation.
   The work level conversion device according to claim 1 or 2.
4. The incidental information includes information that can be recognized from the time-series information and information that is annotated with the time-series information.
   The work level conversion device according to any one of claims 1 to 3.
5. The method executed by the processor
   Generating the series data of the target motion information that associates the target motion detected from the time series information related to the worker's work with the incidental information incidental to the target motion.
   A converter that converts lower-level work sequence data specified by a combination of the target motion and the incidental information into higher-level work sequence data composed of a plurality of the lower-level tasks, and the target motion information. Based on the series data, the series data of the target operation information is converted into the higher level work series data and output.
   Work level conversion method including.
6. Computer,
   A generator that generates series data of target motion information that associates the target motion detected from the time series information related to the worker's work with the incidental information incidental to the target motion.
   A converter that converts lower-level work sequence data specified by a combination of the target motion and the incidental information into higher-level work sequence data composed of a plurality of the lower-level tasks, and the target motion information. A conversion output unit that converts the series data of the target operation information into the higher-level work series data and outputs it based on the series data.
   A work level conversion program that functions as.

Images