WO2024029411A1

WO2024029411A1 - Work feature amount display device, work feature amount display method, and work feature amount display program

Info

Publication number: WO2024029411A1
Application number: PCT/JP2023/027284
Authority: WO
Inventors: 健太西行
Original assignee: オムロン株式会社
Priority date: 2022-08-03
Filing date: 2023-07-25
Publication date: 2024-02-08
Also published as: JP2024021467A

Abstract

This work feature amount display device comprises: a feature amount acquisition unit that acquires the feature amount for each body part of a worker when the worker carries out a series of predetermined work; and a control unit that controls a display unit to display the feature amount of a selected body part selected from among the body parts of the worker.

Description

Work feature display device, work feature display method, and work feature display program

The disclosed technology relates to a work feature display device, a work feature display method, and a work feature display program.

Japanese Patent Laid-Open No. 2022-077870 discloses that a first moving image in which a person performing a specific action is photographed is played back and displayed, and a frame corresponding to the specific action is displayed on the frame of the first moving image that is played back and displayed. an input receiving unit that accepts an instruction input indicating a related position during playback and display; a second feature point group including one or more feature points included in the second moving image; and a second feature point group including one or more feature points included in the first moving image. The degree of similarity between the first moving image and the second moving image is calculated based on the comparison result of the feature amount with a first feature point group including two or more feature points, and the calculated similarity between the first moving image and the second moving image is calculated. a determination unit that determines whether the specific motion is included in the second moving image based on a degree of similarity; Disclosed is a motion recognition system including the determination unit that assigns a larger weight to the comparison result of the feature amount as the feature point is closer to the indicated position.

The above-mentioned Japanese Patent Laid-Open No. 2022-077870 discloses a technology that performs motion recognition with high accuracy by increasing the weight of a part related to a specific motion specified by a user in the process of recognizing a person's motion. ing.

However, with the technology described in Japanese Patent Application Laid-open No. 2022-077870, it is difficult for a user who has no knowledge of motion recognition to specify an appropriate location that contributes to improving accuracy, and it is difficult to use it for motion analysis of work. There was a problem.

The disclosed technology has been made in view of the above points, and provides a work feature display device, a work feature display method, and a work feature display method that can select and display worker feature values suitable for work motion analysis. The purpose of the present invention is to provide a work feature value display program.

A first aspect of the disclosure is a work feature amount display device, which includes a feature amount acquisition unit that acquires a feature amount for each part of the worker when the worker performs a predetermined series of tasks; and a control unit that controls the display unit to display the feature amount of the selected part selected from among the parts of the person.

A second aspect of the disclosure is the first aspect, further comprising a reduction unit that reduces the feature amount to one-dimensional series data, and the control unit controls the control unit so that the series data is displayed on the display unit. Control.

A third aspect of the disclosure, in the first aspect or the second aspect, includes a worker reception unit that receives the worker.

A fourth aspect of the disclosure, in any of the first to third aspects described above, includes a selected site receiving section that receives the selected site.

A fifth aspect of the disclosure is, in any one of the first to third aspects, a standard working time acquisition unit that acquires a standard working time for the series of tasks, and a standard working time acquisition unit that acquires a standard working time for the series of tasks, and a The control unit includes a peak detection unit that detects a position, and a work time calculation unit that calculates a work time for the series of work based on the interval between each detected peak for each part, and the control unit Control is performed so that the feature amount of the selected region with which the difference between the working time and the working time is the smallest is displayed on the display unit.

In a sixth aspect of the disclosure, in the fifth aspect, the working time calculation unit sets the median value of the intervals between the detected peaks as the working time.

A seventh aspect of the disclosure is a work feature amount display method, in which a computer acquires feature amounts for each part of the worker when the worker performs a predetermined series of tasks, and A process including controlling the feature amount of the selected part selected from each part to be displayed on the display unit is executed.

An eighth aspect of the disclosure is a work feature amount display program, which causes a computer to acquire feature amounts for each part of the worker when the worker performs a predetermined series of tasks, and displays the feature amount of the worker. A process including controlling the feature amount of the selected part selected from among the parts to be displayed on the display unit is executed.

According to the disclosed technology, it is possible to select and display the characteristic amount of the worker suitable for the motion analysis of the work.

FIG. 1 is a configuration diagram of a work feature amount display system. FIG. 2 is a configuration diagram showing the hardware configuration of a work feature amount display device. FIG. 2 is a functional block diagram of a work feature amount display device. It is a figure showing an example of a display screen. It is a figure showing an example of a display screen. It is a figure showing an example of a display screen. FIG. 3 is a diagram for explaining estimation of a starting point. It is a flowchart of work feature amount display processing.

Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the drawings. In addition, the same reference numerals are given to the same or equivalent components and parts in each drawing. Further, the dimensional ratios in the drawings may be exaggerated for convenience of explanation and may differ from the actual ratios.

FIG. 1 shows the configuration of a work feature quantity display system 10. The work feature display system 10 includes a work feature display device 20 and a camera 30.

The work feature quantity display device 20 is a device that displays motion feature quantities calculated based on moving images photographed by the camera 30.

As an example, the worker W takes out the work object M placed on the workbench TB and performs a predetermined series of tasks in the work space S. The series of operations performed by the worker W include various operations in one work cycle, such as grasping, transporting, assembling, inspecting, tightening screws with a screwdriver, and attaching labels to components, for example.

The camera 30 is a photographing device capable of photographing, for example, RGB color moving images. The camera 30 is installed at a position where the movement of the worker W and the entire workbench TB can be easily recognized.

Furthermore, in this embodiment, a case will be described in which there is one camera 30, but a configuration in which a plurality of cameras 30 are provided may be used.

FIG. 2 is a block diagram showing the hardware configuration of the work feature amount display device 20 according to the present embodiment. As shown in FIG. 2, the work feature amount display device 20 includes a controller 21. As shown in FIG. The controller 21 is composed of a device including a general computer.

As shown in FIG. 2, the controller 21 includes a CPU (Central Processing Unit) 21A, a ROM (Read Only Memory) 21B, a RAM (Random Access Memory) 21C, and an input/output interface (I/O) 2. Equipped with 1D. The CPU 21A, ROM 21B, RAM 21C, and I/O 21D are connected to each other via a bus 21E. Bus 21E includes a control bus, an address bus, and a data bus.

Furthermore, an operation section 22, a display section 23, a communication section 24, and a storage section 25 are connected to the I/O 21D.

The operation unit 22 includes, for example, a mouse and a keyboard.

The display unit 23 is composed of, for example, a liquid crystal display.

The communication unit 24 is an interface for performing data communication with an external device such as the camera 30.

The storage unit 25 is composed of a nonvolatile external storage device such as a hard disk. As shown in FIG. 2, the storage unit 25 stores a work feature display program 25A, a motion feature database 25B, and the like.

The CPU 21A is an example of a computer. A computer here refers to a processor in a broad sense, and can be a general-purpose processor (e.g., CPU) or a dedicated processor (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: FI eld Programmable Gate array, programmable logic device, etc.).

Note that the work feature amount display program 25A is realized by being stored in a non-volatile non-transitory recording medium, or distributed via a network, and installed on the work feature amount display device 20 as appropriate. You may.

Examples of non-volatile non-transitional recording media include CD-ROM (Compact Disc Read Only Memory), magneto-optical disk, HDD (Hard Disk Drive), DVD-ROM (Digital Versatile Disc Read Only Memory), and DVD-ROM (Digital Versatile Disc Read Only Memory). Memory), flash memory, memory Cards, etc. are assumed.

FIG. 3 is a block diagram showing the functional configuration of the CPU 21A of the work feature quantity display device 20. As shown in FIG. 3, the CPU 21A functionally includes a motion feature acquisition section 40, a control section 41, a reduction section 42, a worker reception section 43, a selected part reception section 44, a standard work time acquisition section 45, It includes functional units such as a peak detection unit 46 and a work time calculation unit 47.

The CPU 21A functions as each functional unit shown in FIG. 3 by reading and executing the work feature amount display program 25A stored in the storage unit 25.

The motion feature acquisition section 40 reads motion features for each part of the worker W when the worker W performs a predetermined series of tasks from the motion feature database 25B stored in advance in the storage section 25. Obtained by The motion feature database 25B is a database that stores motion features for each part and for each worker.

The motion feature amount can be calculated as a motion vector series, for example, based on a moving image captured by the camera 30 of the worker W performing a predetermined series of tasks. Note that instead of the camera 30, the motion feature amount of the worker W may be acquired using a motion sensor or the like. Here, the parts of the worker W are parts including the joints that make up the skeleton of the worker W, such as the face, neck, shoulders, elbows, wrists, hips, knees, and ankles of the worker W. .

In calculating the motion vector series, in order to avoid being influenced by the background, the clothes of the worker W, etc., the posture of the worker W is estimated based on the moving image, and the estimated posture is converted into a skeletal sequence.

As a method for estimating the posture of the worker W and converting the estimated posture into a skeletal sequence, a known method called OpenPose described in Reference 1 below can be used. The skeletal series is time-series data that includes coordinates of feature points such as body parts and joints of the worker W, and labels representing the body parts of the feature points. For example, the feature points include the face of the worker W, such as eyes and nose, and joints such as the neck, shoulders, elbows, wrists, hips, knees, and ankles.

OpenPose uses a learned model that uses a moving image as input and a skeletal sequence as output, and is trained using a large number of moving images as training data. As a learning method for obtaining such a trained model, a known method such as CNN (Convolutional Neural Networks) is used, for example.

(Reference 1) "OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", Zhe Cao, Student Member, IEEE, Gines Hidalgo, Student Member, IEEE, Tomas Simon, Shih-En Wei, and Yaser Sheikh, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE.

Here, in factory work, various workers with different body types perform the work, so the difference in body shape has a large influence. In order to avoid being affected by differences in body shape, this embodiment uses a method called MotionRetargeting described in Reference 2 below to convert a skeletal sequence obtained from a video image into a motion representation representing a motion feature amount. Convert to vector series.

(Reference 2) K. Aberman, R. Wu, D. Lischinski, B. Chen, and D. Cohen-Or, “Learning character-agnostic motion for motion retargeting in 2d,” TOG, vol.38, no.4 , p.75, 2019.

In MotionRetargeting, a skeletal sequence is input and an encoder is used to output feature vectors of three components: motion, body shape, and camera viewpoint. In this embodiment, for example, in order to reduce the influence of body shape and camera viewpoint, Only the feature vector of the motion component is used.

In addition, in order to remove the influence of noise in pose estimation, the skeletal sequence is subjected to three preprocessing processes: time series interpolation processing, time series smoothing, and lower body interpolation processing before conversion to a motion vector series. You may do so.

In time-series interpolation processing, if there is a joint point for which pose estimation has failed, the joint point of the previous frame is copied. In time series smoothing, in order to remove noise in pose estimation, series data is smoothed using a Gaussian filter. OpenPose used in this embodiment estimates not only the posture of the upper body of a person but also the posture of the lower body. When working in a factory, workers often work on a desk-top workbench, so their lower body is often shielded by the desk, resulting in loss of joints in the lower body. Encoders that extract motion features, such as MotionRetargeting, take as input the skeletal sequence of a person's entire body, so if lower body joint points are missing, the motion component feature vector cannot be properly output. There are cases. Therefore, interpolation processing for the lower body may be performed. Specifically, as the interpolation process for the lower body, the joint points of at least one of both knees and both legs may be interpolated with a length proportional to the length of the torso of the person.

The motion features of each part calculated in this way are stored in advance in the storage unit 25 as a motion feature database 25B for each worker.

The control unit 41 controls the display unit 23 to display the motion feature amount of the selected part selected from among the parts of the worker W.

The reduction unit 42 reduces the motion features acquired by the motion feature acquisition unit 40 into one-dimensional series data. In this case, the control unit 41 controls the reduced one-dimensional series data to be displayed on the display unit 23. Note that various known methods can be used as the reduction method.

The worker reception unit 43 receives a worker whose motion feature amount is to be displayed from among a plurality of workers.

The selected part receiving unit 44 receives the selected part selected from among the parts of the worker.

FIG. 4 shows an example of a display screen of motion feature amounts displayed on the display unit 23. The display screen D shown in FIG. 4 includes a pull-down menu M1 for accepting a worker whose motion feature amount is to be displayed from among a plurality of workers, and a selection part for displaying a motion feature amount from among each part of the worker. , a display area R1 for checking the part numbers assigned to the parts of the worker, and a display area R2 for displaying the reduced one-dimensional series data as a graph.

By opening the pull-down menu M1, the user can select a desired worker from among the plurality of workers whose motion features are stored in the motion feature database 25B. The example in FIG. 4 shows a state in which a worker with worker ID "A_02" is selected.

Further, by opening the pull-down menu M2, the user can select a desired part from among the motion features of each part of the worker selected in the pull-down menu M1. In the display area R1, part numbers assigned in advance to parts of the worker are displayed. The user refers to display area R1 and selects a desired part number from pull-down menu M2. The example in FIG. 4 shows a state in which [5, 6, 7] represented by a set of part numbers is selected, that is, a state in which the left arm is selected. Note that the set of part numbers may be composed of a plurality of part numbers or a single part number. Furthermore, although a plurality of predetermined sets of part numbers are displayed in the pull-down menu M2, the user may be able to directly input any part number.

When a worker is selected from the pull-down menu M1 and a body part is selected from the pull-down menu M2, series data obtained by condensing the motion features of the body part of the selected worker into one dimension is displayed as a graph G in the display area R2. be done. The horizontal axis of the graph G is time, and the vertical axis is a one-dimensional motion feature.

This allows the user to confirm whether the periodicity of the motion feature of the selected region is high or low.

For example, in the example of FIG. 5, [2, 3, 4, 5, 6, 7] is selected as the part number. That is, the right arm and left arm are selected. Looking at the graph G in the display area R2, it can be confirmed that the variation in the intervals between the peaks of the motion feature amount is relatively small. Therefore, the user cannot understand that the motion features of the parts represented by [2, 3, 4, 5, 6, 7] are suitable for analyzing the work of worker "A_02". can.

On the other hand, in the example of FIG. 6, [0, 1, 2, 3, 4, 5, 6, 7, 8] is selected as the part number. Looking at the graph in the display area R2, it can be seen that the variation in the peak interval of the motion feature amount is relatively large. Therefore, the user considers that the motion features of the parts represented by [0, 1, 2, 3, 4, 5, 6, 7, 8] are not suitable for analyzing the work of worker "A_02". be able to understand that.

In addition, instead of the user selecting a part from the pull-down menu M2, if the automatic button B is pressed as shown in FIG. 7, the part suitable for the work analysis is automatically selected, and the selected part The motion feature amount may be displayed in the display area R2.

In this case, the standard working time acquisition unit 45 acquires the standard working time for a series of tasks by reading it from the storage unit 25, for example. The standard working time can be obtained, for example, by measuring the working time when a standard worker performs a series of tasks, and is stored in the storage unit 25 in advance. Note that the standard working time may be acquired by having the user input the standard working time on the display screen D.

Then, the peak detection unit 46 detects the position of the peak of the motion feature amount for each part. Note that various known methods can be used to detect the peak. FIG. 7 shows an example of peak position detection. In the example of FIG. 7, peaks P1 to P6 are detected.

The work time calculation unit 47 calculates the work time of a series of tasks for each region based on the interval between each detected peak. At this time, the work time calculation unit 47 may take the median value of the intervals between the detected peaks as the work time. For example, in the example of FIG. 7, if the appearance times of peaks P1 to P7 are [100,950,1300,1800,2650,3500], the intervals between adjacent peaks are [850,350,500,850]. , 850]. In this case, the median interval between adjacent peaks is "850", which is the working time for one cycle of the series of works. In this way, by setting the median interval between adjacent peaks as the working time, it is possible to eliminate the influence of noise peaks such as peak P3 shown in FIG. 7 and calculate the working time with high accuracy. .

Then, the control unit 41 displays the motion feature amount of the selected part in which the difference between the standard work time acquired by the standard work time acquisition unit 45 and the work time calculated by the work time calculation unit 47 is the smallest in the display area R2. Control what is displayed. At this time, the selected region can be highlighted by displaying the region number of the selected region in the pull-down menu M2 or by making the color of the region number of the selected region different from other region numbers in the display area R1. Make it easy to understand which part it is. This eliminates the need for the user to select various parts from the pull-down menu M2, display the graph G, and judge whether or not the motion features are suitable for analysis of the work. Therefore, the user can quickly grasp the part suitable for analysis of the work.

Next, the work feature display process executed by the CPU 21A of the work feature display device 20 will be described with reference to the flowchart shown in FIG. Note that the work feature value display process shown in FIG. 8 is repeatedly executed.

In step S100, the CPU 21A determines whether a worker has been selected from the pull-down menu M1. Then, if a worker is selected, the process moves to step S101, and if a worker is not selected, the process waits until a worker is selected.

In step S101, the CPU 21A determines whether a body part has been selected from the pull-down menu M2. Then, if a part is selected, the process moves to step S102, and if no part is selected, the process moves to step S105.

In step S102, the CPU 21A acquires the motion feature amount of the part selected in step S101 of the worker selected in step S100 by reading it from the motion feature database 25B.

In step S103, the CPU 21A reduces the motion feature amount acquired in step S102 to a one-dimensional motion feature amount.

In step S104, the CPU 21A displays the motion feature quantity reduced to one dimension in step S108 as a graph G in the display area R2 of the display screen D.

In this way, the user can display the motion feature amount in the display area R2 by selecting the worker and the part, and can analyze the work by checking the periodicity of the motion feature amount of the selected part. You can decide whether it is suitable or not.

In step S105, the CPU 21A determines whether automatic button B has been pressed. Then, if the automatic button B is pressed, the process moves to step S106, and if the automatic button B is not pressed, the process moves to step S101.

In step S106, the CPU 21A acquires the standard working time of the series of tasks by reading it from the storage unit 25.

In step S107, the CPU 21A detects the peak position of the motion feature amount for each part. Specifically, for each part of the worker selected in step S100, motion features are read from the motion feature database 25B, reduced to one-dimensional motion features, and based on the reduced motion features. Detect the position of the peak.

In step S108, the CPU 21A calculates the working time of the series of tasks for each part of the worker selected in step S100, based on the interval between each peak detected in step S107.

In step S109, the CPU 21A compares the standard working time obtained in step S104 and the working time for each part calculated in step S106. Then, the motion feature amount of the selected part with the smallest difference between the standard work time acquired in step S104 and the work time calculated in step S106 is displayed in the display area R2.

As a result, the user does not have to select various parts on his/her own and display the motion feature amounts to check the periodicity, and can quickly grasp the part suitable for analysis of the work.

In this way, in this embodiment, the movement of a selected part selected from among the parts of the worker among the movement features of each part of the worker when the worker performs a predetermined series of tasks is determined. The feature amount is controlled to be displayed on the display unit 23. Thereby, it is possible to display the worker's motion feature amount suitable for the motion analysis of the work.

Note that the above embodiments are merely illustrative examples of the configuration of the present disclosure. The present disclosure is not limited to the above-described specific form, and various modifications can be made within the scope of the technical idea.

For example, in the above embodiment, a case has been described in which a motion feature amount calculated based on a video image of a worker working is reduced to a one-dimensional motion feature amount and displayed. It is not limited to this. For example, the motion feature may be a sound recorded by a microphone installed in a place where the worker is working. This is because it is sometimes possible to understand the periodicity of work from the operation sounds of machines operated by workers. Further, as described above, acceleration data detected by an acceleration sensor such as a motion sensor may be used as the motion feature amount. Furthermore, a motion feature amount that is a combination of these motion feature amounts may be used.

Further, in the above embodiment, a case has been described in which control is performed so that the motion feature amount of the selected part selected from among the parts of the worker is displayed on the display unit 23. Control may be performed so that the feature amount is displayed on the display unit 23. For example, the display unit 23 may be controlled to display other feature amounts such as a skeletal coordinate series of the worker or a velocity vector series of the skeletal coordinates. In this case, the user may be able to select the feature amount to be displayed on the display unit 23 from among the motion feature amount, the skeletal coordinate series, and the velocity vector series of the skeletal coordinates.

Further, the work feature amount display processing that was executed by the CPU in the above embodiment by reading the software (program) may be executed by various processors other than the CPU. In this case, the processor is a PLD (Programmable Logic) whose circuit configuration can be changed after manufacturing, such as an FPGA (Field-Programmable Gate Array).
Examples include a dedicated electric circuit, which is a processor having a circuit configuration specifically designed to execute recognition processing, such as a device (device), and an application specific integrated circuit (ASIC). Further, the work feature amount display processing may be executed by one of these various processors, or by a combination of two or more processors of the same type or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). combinations of etc.). Further, the hardware structure of these various processors is, more specifically, an electric circuit that is a combination of circuit elements such as semiconductor elements.

Note that the disclosure of Japanese Patent Application No. 2022-124302 is incorporated herein by reference in its entirety. In addition, all documents, patent applications, and technical standards mentioned herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually indicated to be incorporated by reference. , herein incorporated by reference.

Claims

a feature amount acquisition unit that obtains feature amounts for each part of the worker when the worker performs a predetermined series of tasks;
a control unit that controls the feature amount of the selected part selected from the respective parts of the worker to be displayed on a display unit;
A work feature display device equipped with.
comprising a reduction unit that reduces the feature amount to one-dimensional series data,
The work feature display device according to claim 1, wherein the control unit controls the series data to be displayed on the display unit.
The work feature amount display device according to claim 1, further comprising a worker reception section that receives the worker.
The work feature quantity display device according to any one of claims 1 to 3, further comprising a selected part receiving section that receives the selected part.
a standard working time acquisition unit that acquires standard working time for the series of tasks;
a peak detection unit that detects the position of the peak of the feature amount for each part;
a work time calculation unit that calculates the work time of the series of work based on the interval of each detected peak for each part;
Equipped with
The control unit includes:
The work feature amount according to any one of claims 1 to 3, wherein the work feature amount is controlled so that the feature amount of the selected part having the smallest difference between the standard work time and the work time is displayed on the display unit. Display device.
The work feature value display device according to claim 5, wherein the work time calculation unit sets the work time to be a median value of intervals between detected peaks.
The computer is
Obtaining feature amounts for each part of the worker when the worker performs a predetermined series of tasks,
A method for displaying a work feature amount, which performs a process including: controlling a feature amount of a selected part selected from among the parts of the worker to be displayed on a display unit.
to the computer,
Obtaining feature amounts for each part of the worker when the worker performs a predetermined series of tasks,
A work feature quantity display program that executes a process including controlling so that a feature quantity of a selected part selected from among the parts of the worker is displayed on a display unit.