JP2024005533A - Work management device and work management method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a work management device which can accurately determine that a work object is arranged at a prescribed position.

SOLUTION: A work management device 2 comprises: an image acquisition unit 242 which acquires a photographed image generated by imaging the work in which a worker arranges a work object in a prescribed designated area T by an imaging device; an operation portion detection unit 243 which detects a position of an operation portion of the worker imaged in the photographed image; a determination unit 244 which specifies a first photographed image captured at first timing before the operation portion detected by the operation portion detection unit 243 enters the designated area T and a second photographed image captured at second timing after the operation portion enters the designated area T, and determines that the work object is arranged in the designated area T when a difference between the image of the designated area T in the first photographed image and the image of the designated area T in the second photographed image is equal to or greater than a prescribed threshold; and an output unit 245 which outputs a determination result of the determination unit 244.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to a work management device and a work management method.

Conventionally, images obtained by photographing the work performed by a worker are analyzed to detect when a worker's hand enters a predetermined monitoring area or when a part is assembled in a predetermined position by the worker. A work analysis system is known (for example, see Patent Document 1).

JP2021-163293

For example, when a worker moves an object from a certain position to a predetermined position and places the object, it is important to determine whether the object has been placed at the predetermined position. becomes. However, the conventional technology has a problem in that it is not possible to accurately determine whether the object is placed at a predetermined position.

Therefore, the present invention has been made in view of the above, and an object of the present invention is to provide a work management device and a work management method that can accurately determine whether a work object is placed at a predetermined position.

A work management device according to one embodiment of the present invention includes: an image acquisition unit that acquires a captured image generated by an imaging device capturing a task in which a worker places a work object in a predetermined designated area; a working part detection unit that detects the position of the working part of the worker in the photograph; and a first image taken at a first timing before the working part detected by the working part detection unit enters the designated area. image and a second captured image taken at a second timing after the operating part enters the specified area, and identify the image of the specified area in the first captured image and the second captured image in the second captured image. a determination unit that determines that the work object is placed in the designated area when a difference between the designated area and the image is greater than or equal to a predetermined threshold; and an output unit that outputs a determination result of the determination unit. .

The determining unit is configured to identify, as the first captured image, a captured image at the first timing in which the operating part does not enter the specified area, and to specify the captured image at the second timing in which the operating part does not enter the designated area as the first captured image. The captured image may be specified as the second captured image, and the determination may be made based on a difference between the specified first captured image and the second captured image.

The determination unit determines, as the first timing, a time that is a first set time before a predetermined reference time when the operating part enters the specified area, and determines a second set time from the predetermined reference time as the first timing. A later time may be determined as the second timing.

The imaging device may generate a captured image including a plurality of the specified areas, and the determination unit may determine the first timing and the second timing for each of the specified areas.

The motion part detection section creates skeletal information of the body including a plurality of motion points based on the image of the worker's body shown in the captured image, and the output section generates skeletal information of the body including a plurality of motion points. A predetermined motion point related to the content of the work performed by the worker may be displayed on the image display unit by being superimposed on a body part corresponding to the predetermined motion point in the body image of the worker. good.

The operating part detection unit creates skeletal information of the body based on the image of the worker's body shown in the captured image, and determines the content of the work performed by the worker, which is specified based on the skeletal information. The position of the motion part may be detected based on the position of one or more motion points associated with the motion part.

The motion part detection section refers to a storage section that stores a plurality of work contents in association with a plurality of predetermined motion points related to the work contents, and detects the work contents received from the user. In response to an input, the predetermined operation point related to the inputted content of the work may be determined.

The work management device further includes a work specifying unit that specifies the content of the work performed by the worker based on the movement indicated by the skeletal information created by the working part detecting unit, and the working part detecting unit includes: The position of the motion part may be detected based on the position of the motion point related to the content of the work specified by the work specifying unit.

The determination unit is configured to determine the timing at which the output from the detection means for detecting that the worker has picked up the work object from the object arrangement area where the work object is arranged, or from the reception timing. The first timing may be a timing before or after a predetermined time.

A work management method according to an embodiment of the present invention includes an image acquisition step of acquiring a captured image generated by an imaging device capturing a task of placing a work object in a predetermined designated area by a worker; a working part detecting step of detecting the position of the working part of the worker in the photograph; and a first image taken at a first timing before the working part detected in the working part detecting step enters the specified area. image and a second captured image taken at a second timing after the operating part enters the specified area, and identify the image of the specified area in the first captured image and the second captured image in the second captured image. a determination step of determining that the work object is placed in the designated area when a difference between the designated area and the image is greater than or equal to a predetermined threshold; and an output step of outputting a determination result in the determination step. and each step is executed by a computer.

According to the present invention, it is possible to provide a work management device and a work management method that can accurately determine whether a work object is placed at a predetermined position.

1 is a diagram showing the configuration of a work management system according to an embodiment of the present invention. FIG. 3 is a diagram showing a work performed by a worker viewed from above. FIG. 3 is a diagram showing a captured image taken by a camera, and an inventory area and a specified area displayed on the image. FIG. 3 is a diagram showing skeletal information created based on an image of a worker's hand. This is an example of a data table in which the contents of work and operation points are associated with each other, which is stored in the storage unit. FIG. 6 is a schematic diagram for explaining determination of product placement based on a difference between captured images by a determination unit. It is a flowchart of an example of operation of a work management system. FIG. 3 is a diagram showing a captured image taken by a camera, and a plurality of inventory areas and a plurality of designated areas displayed on the image. It is a block diagram showing the composition of the modification of the embodiment of the present invention.

<First embodiment>
Embodiments of the present invention will be described below with reference to the drawings.
In the following description, the same or corresponding elements will be denoted by the same or corresponding symbols, and overlapping description will be omitted. FIG. 1 is a diagram showing the configuration of a work management system S100 according to an embodiment of the present invention. FIG. 2 is a diagram showing the work performed by the worker viewed from above. FIG. 3 is a diagram showing a captured image taken by the camera 1, and the inventory area S and designated area T displayed on the image.

[Overview of work management system S100]
The work management system S100 includes a camera 1 and a work management device 2, as shown in FIG. In this embodiment, the work management system S100 takes pictures of product sorting work performed by a worker. Specifically, in this work, as shown in FIGS. 1 and 2, a worker selects work objects from a plurality of inventory boxes 11-1, 11-2, and 11-3 (also simply referred to as inventory boxes 11). This is a task of picking up a certain product 5, moving the product 5 to the packaging box 12, and arranging it.

The work management system S100 of this embodiment uses a captured image taken at a timing before the worker's hand enters a predetermined designated area T corresponding to the packaging box 12, and a captured image taken at a timing before the worker's hand enters the designated area T corresponding to the packaging box 12. One of the features is that it is determined that the product 5 has been placed in the packaging box 12 based on the difference with the captured image taken at a later timing.

In this type of work performed by a worker, it is assumed that although the worker's hand holding the product 5 passes over the packaging box 12, the product 5 is not placed in the packaging box 12. Therefore, in a method that determines that the product 5 is placed in the packaging box 12 simply on the condition that the worker's hand enters the designated area T, it cannot be correctly determined that the product 5 is placed in the packaging box 12.

On the other hand, according to the work management system S100 of the present embodiment, the details will be described later, but the captured image before the worker's hand entered the designated area T and the captured image before the worker's hand entered the designated area T. Since it is determined that the product 5 has been placed on the basis of the difference between the captured images and the captured images after the image capture, it is possible to accurately determine that the product 5 has been placed in the packaging box 12.

[Description of each part of the work management system S100]
(Camera 1)
The camera 1 is an imaging device having an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 1 is supported by a support (not shown) and photographs the inventory box 11, the packaging box 12, the product 5, and the hands of the worker from above at a predetermined frame rate. The camera 1 transmits the generated captured image to the work management device 2.

(Work management device 2)
The work management device 2 is, for example, a computer in which a computer program is installed. The work management device 2 includes, for example, an input section 21, a display section 22, a storage section 23, and a control section 24, as shown in FIG.

The input unit 21 is a device for receiving input from a worker. The input unit 21 is any device such as a keyboard, mouse, touch panel, etc., for example.

The display unit 22 is any image display unit that displays various types of information, and is, for example, one or more displays. The input section 21 and the display section 22 may be configured as a touch panel display. The display unit 22 displays, for example, a moving image that is a plurality of continuous captured images generated by the camera 1 in real time. In addition to being configured as a display, the display unit 22 may be configured to display various information by, for example, projecting an image onto a predetermined subject by a projection unit (not shown).

The display unit 22 displays an image as shown in FIG. 3, for example. The image in FIG. 3 includes an inventory area S and a designated area T in addition to the captured image generated by the camera 1. The inventory area S is an area with a size corresponding to the inventory box 11, and in this example, three inventory areas S are set corresponding to each inventory box 11. The designated area T is an area set to a size corresponding to the packaging box 12, and in this example, one designated area T is set corresponding to the packaging box 12. The settings of the inventory area S and designated area T will be described later.

The storage unit 23 is a storage medium that stores various data, and includes a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk, and the like. The storage unit 23 stores a computer program executed by the control unit 24. The storage unit 23 stores various data that the control unit 24 refers to when performing calculation processing, as will be described later.

(Each functional unit of the control unit 24)
The control unit 24 includes a CPU (Central Processing Unit). For example, by executing a computer program stored in the storage unit 23, the control unit 24 controls the area setting unit 241, image acquisition unit 242, motion part detection unit 243, determination unit 244, and output as shown in FIG. 245.

The area setting unit 241 is a functional unit for setting the inventory area S and designated area T. The area setting unit 241 receives, for example, the range specified by the user using the input unit 21 as the inventory area S and the designated area T, and stores the setting information in the storage unit 23. Specifically, the user sets the inventory area S and designated area T, for example, by inputting the coordinates of two diagonal points of a rectangle.

The image acquisition unit 242 acquires a captured image generated by the camera 1. The captured image acquired by the image acquisition unit 242 is stored in the storage unit 23, as an example.

The operating part detection unit 243 detects the position of the hand, which is an example of the operating part of the worker, shown in the captured image. FIG. 4 is a diagram showing skeletal information created based on an image of a worker's hand. The operating part detection unit 243 creates skeletal information based on the image of the worker's body shown in the captured image. For example, the operating part detection unit 243 creates three-dimensional skeletal information of the operating part in real time based on the body image captured in the captured image, without using markers attached to the worker's body. .

The "skeletal information" includes, for example, a plurality of motion points corresponding to the positions of joints, fingertips, and the like. In FIG. 4, among the plurality of motion points of the worker's one hand (right hand), the tip of the thumb and the joint of the thumb are shown as motion points P11 and P12, respectively, and the tip of the index finger and the joint of the index finger are motion points, respectively. They are indicated by points P21 and P22.

In this embodiment, the determination unit 244 detects whether the operator's moving part is within the designated area T, as will be described later. In order to detect whether or not the worker's moving part falls within the designated area T, it is necessary to specify which position of the working part falls within the designated area T. Therefore, in one example, the motion part detection unit 243 detects the position of the worker's motion part based on the positions of one or more motion points related to the content of the work, which are identified based on the skeletal information.

As a specific example, in the case of a work in which the product 5 is held and moved between the index finger and the thumb, the motion part detection unit 243 detects the position of the worker's hand based on the positions of the motion point P11 and the motion point P21. That is, the motion point P11 and the motion point P21 serve as reference positions for detecting the position of the worker's hand. Note that the motion part detection unit 243 may detect the position of the hand based on the position of either the motion point P11 or the motion point P21 instead of the plurality of motion points.

In this embodiment, the position of the hand is detected based on the motion point related to the work content, so in the above example of the work content, the hand position is detected based on the motion point of the wrist, for example. This makes it possible to perform more appropriate detection depending on the content of the work than in the case of conventional methods.

Which motion point among the plurality of motion points is important for detecting a motion part in a target task differs depending on the content of the task. For example, if the task is to move a large product, it is preferable to detect the motion points of each of the right and left hands, and if the task is to move a small product, it is assumed that only the motion of one hand needs to be detected.

Therefore, the storage unit 23 may store a plurality of work contents and a plurality of predetermined operation points related to the work contents in association with each other. In this case, the motion part detection section 243 may refer to the storage section 23 and determine motion points related to the work content. Specifically, the motion part detection unit 243 may determine a motion point related to the input work content in response to the work content input received from the user. The work details are input via a predetermined graphical user interface, for example. An example in which the content of the work is specified by the work management system S100 will be described later.

FIG. 5 is an example of a data table stored in the storage unit 23 in which work contents and operation points are associated with each other. In this table, work contents, target parts (motion parts), and motion points are associated with each other. "Work content" indicates the type of work content, "motion part" indicates the part of the worker's body that should be detected in the work, and "motion points" that should be used as a reference in the work are included in the skeletal information. Indicates one or more operating points. The motion part detection unit 243, for example, refers to the data table in FIG. 5 to determine motion points related to the work content.

The determining unit 244 is a functional unit that determines whether the product 5, which is the work target, has been placed in the designated area T. The determination unit 244 determines that the product 5 has been placed based on the difference between the two captured images in which the designated area T is captured.

FIG. 6 is a schematic diagram for explaining the determination of product placement based on the difference between captured images by the determination unit 244, and the horizontal axis is the time axis. The determination unit 244 determines whether the first captured image is captured at a first timing before the worker's hand enters the designated area T, and the first captured image captured at a second timing after the worker's hand enters the designated area T. The second captured image is specified. Specifically, the determination unit 244 identifies the captured image at the first timing in which the hand of the worker does not enter the designated area T as the first captured image. Further, the determination unit 244 identifies the captured image at the second timing in which the hand of the worker does not enter the designated area T as the second captured image. In this way, by using a captured image that does not include the hand, the presence or absence of the product 5 can be satisfactorily determined without the product 5 being hidden by the hand.

In order to identify a captured image that does not include a hand, the determination unit 244 specifies that the operator's hand has entered the specified area T by a first set time a1 before a predetermined reference time t. The time is determined as the first timing, and the time after the second set time a2 from the predetermined reference time t is determined as the second timing. In this way, by setting the first and second timings to be timings with a predetermined time margin with respect to the reference time t when the hand is in the designated area T, complicated image processing is not required. Therefore, the first timing and the second timing can be determined.

The "predetermined reference time t" is an arbitrary timing within the time period when the worker's hand is within the designated area T, and is determined by the determination unit 244 based on the captured image, for example. As an example, when the determination unit 244 detects that the worker's hand has entered the designated area T, and subsequently detects that the worker's hand has left the designated area T, the determination unit 244 determines that the worker's hand has entered the designated area T. An arbitrary timing between the time when the worker's hand enters the designated area T and the time when the worker's hand leaves the designated area T is determined as a reference time t.

By the way, at what timing the worker's hand enters the designated area T and at what timing the hand moves out of the designated area T depends on, for example, the content of the work and the size of the designated area T. Dependent. Therefore, the work management device 2 may be configured such that the user can input the first set time a1 and the second set time a2, thereby changing the first timing and the second timing. Specifically, the determination unit 244 receives the first set time a1 and second set time a2 input by the user, and uses the received first set time a1 and second set time a2 to determine the first timing and the second set time a2. Determine the second timing.

Further, the storage unit 23 stores a plurality of first set times a1 and second set times a2 in association with the content of a plurality of tasks, and the determination unit 244 refers to the storage unit 23 to The first set time a1 and second set time a2 may be determined accordingly. The work content may be input by the user, or may be estimated by the work management device 2 based on the captured image.

(difference calculation processing)
The determination unit 244 performs a difference calculation process on the first captured image at the first timing and the second captured image at the second timing identified by the above processing, and when the difference is greater than or equal to a predetermined threshold, It is determined that product 5 has been placed. Specifically, as an example, the determination unit 244 determines the difference between the image of the specified area T in the first captured image and the image of the specified area T in the second captured image. According to such processing, for example, when the product 5 is placed correctly in the packaging box 12, the difference becomes greater than or equal to a predetermined threshold, and the determination unit 244 can detect that the product 5 is placed in the designated area T. .

The difference is expressed, for example, by the number of pixels with different pixel values between the first captured image and the second captured image at the first timing, and the predetermined threshold value is determined based on the number of pixels corresponding to the size of the product 5. There is. The predetermined threshold is, for example, the minimum number of pixels that is assumed to correspond to the size of the product 5.

The output unit 245 is a functional unit that outputs various information processed by the control unit 24. The output unit 245 causes the display unit 22 to display various information, for example. The output unit 245 may also output various types of information to an external device (not shown) connected via a network.

For example, the output unit 245 outputs to the display unit 22 a captured image generated by capturing the worker's work with the camera 1. As described above, in this embodiment, the motion part detection unit 243 creates the skeleton information of the worker's hand. For example, the output unit 245 may display motion points included in the skeleton information and lines connecting the motion points in a superimposed manner on the image of the worker's hand.

The output unit 245 may also preferentially display, among the motion points included in the skeleton information, motion points that are highly relevant to the target work content, and may not display the others. In FIG. 3, among the motion points included in the worker's hand skeleton information, motion points P11 and P21 are displayed, and the other motion points are not displayed on the screen. In this way, the output unit 245 may display a predetermined motion point out of the plurality of motion points in a superimposed manner on the body part corresponding to the motion point in the image of the worker's body. Regarding which of the plurality of motion points to display, the output unit 245 may specify a motion point related to the content of the work performed by the worker, and display the specified motion point.

The output unit 245 also outputs the determination result of the determination unit 244. The output unit 245 outputs, for example, a notification indicating that the product 5 has been placed in the designated area T. Specifically, the output unit 245 has an image identification function for identifying each product 5, and may output the result of identifying the product 5 placed in the designated area T.

(Example of operation of work management system S100)
The operation of the work management system S100 configured as above will be explained. FIG. 7 is a flowchart of an example of the operation of the work management system S100. It is assumed that, at the start of the operation of the work management system S100, in the work management device 2, an inventory area S and a designated area T are set at positions corresponding to the inventory boxes 11 and packaging boxes 12. Further, it is assumed that the content of the work to be performed by the worker has already been input by the user, as an example.

First, in step S1, the camera 1 continuously photographs the work of the worker to generate captured images, and the image acquisition unit 242 acquires the captured images.

Next, in step S2, the operating part detection unit 243 creates three-dimensional skeletal information of the worker's hand based on the image of the worker's body captured in the captured image. Based on the content of the work input by the user, the motion part detection unit 243 also refers to the data table illustrated in FIG. 5 to identify motion points related to the target work content. Here, as an example, a motion point P11 and a motion point P12 of one hand of the worker are specified.

The determination unit 244 determines whether the worker's hand has entered the designated area T based on the positions of the motion point P11 and the motion point P12. If the determination in step S2 is No, the determination unit 244 repeats the processing in step S1 and step S2.

If the determination in step S2 is Yes, then in step S3, the determination unit 244 determines whether the worker's hand has come out of the designated area T based on the positions of the motion point P11 and the motion point P12. If the determination in step S3 is No, the determination unit 244 repeats the process in step S3.

If the determination in step S3 is Yes, then in step S4, the determining unit 244 determines, for example, based on the time when the worker's hand entered the designated area T and the time when the worker's hand left the designated area T. A reference time t (see FIG. 6) within the designated area T is specified. Then, the determination unit 244 determines a time a preset first time a1 before as the first timing, and a time after a second set time a2 from a predetermined reference time t as the second timing. decide. The determining unit 244 identifies the first captured image and the second captured image according to the determined first timing and second timing.

Next, in step S5, the determination unit 244 performs a difference calculation process on the first captured image and the second captured image identified in the above step. Then, in step S6, if the difference is greater than or equal to a predetermined threshold as a result of the difference calculation process, the determination unit 244 determines that the product 5 is placed in the designated area T, and the output unit 245 outputs the result. If the difference is less than a predetermined threshold, the determination unit 244 determines that the product 5 is not placed in the specified area T, and the output unit 245 outputs the result.

Through the series of operations described above, the work management system S100 can determine that the product 5 that the worker took out from the inventory box 11 has been placed in the packaging box 12.

Note that although a specific example of the operation of the work management system S100 has been described above according to the flowchart of FIG. 7, the present invention is not limited to the content or order of the specific processing described above; The processing contents and processing order may be changed as appropriate within the range.

(effect)
As explained above, in the configuration of the present embodiment, the work management device 2 uses the first captured image taken at the timing before the worker's hand enters the designated area T, and the first captured image taken before the worker's hand enters the designated area T. It is determined that the product 5 has been placed in the packaging box 12 based on the difference with the second captured image taken at the timing after entering T.

In a method that determines that the product 5 has been placed in the packaging box 12 based on the worker's hand entering the designated area T, for example, if the worker's hand simply passes through the designated area T, the product 5 If the product 5 is not placed in the packaging box 12, it cannot be correctly determined that the product 5 has been placed in the packaging box 12.

In contrast, the work management device 2 of the present embodiment uses two captured images before and after the worker's hand is in the designated area T, and determines the product 5 based on the difference between these captured images. Since it is determined that the product 5 has been placed in the packaging box 12, it can be determined with high accuracy that the product 5 has been placed in the packaging box 12.

In particular, since the determining unit 244 identifies captured images at a time when the worker's hand does not enter the specified area T as the first captured image and the second captured image, the product 5 is not hidden by the worker's fielder. It becomes possible to accurately determine whether the product 5 has been placed.

Further, the determination unit 244 uses the first set time a1 and the second set time a2 (FIG. 6) based on a predetermined reference time t when the worker's hand enters the designated area T to obtain a first captured image. Since the timing of the second captured image and the second captured image are determined, the first timing and the second timing can be determined without requiring complicated image processing.

<Modification 1>
In the above embodiment, an example in which there is only one designated area T has been described, but the work management system S100 can also good. Such a system can be suitably used, for example, when a plurality of packaging boxes 12 of different sizes are arranged, and an operator places a predetermined product 5 into each packaging box 12.

FIG. 8 is a diagram showing a captured image taken by a camera, and a plurality of inventory areas and a plurality of designated areas displayed on the image. In the example of FIG. 8, a plurality of designated areas T1 and T2 (also simply referred to as designated area T) are set according to a plurality of packaging boxes 12. As shown in FIG. 8, when there are a plurality of packaging boxes 12 of different sizes, for example, a large designated area T1 and a small designated area T2 will exist correspondingly. The time that the worker's hand is in the designated area T depends on the content of the work, but is also related to the size of the designated area T. Therefore, in one embodiment of the present invention, the determination unit 244 may determine the first timing and the second timing for each designated area.

The determination unit 244 may determine the first timing and second timing for each designated area T, for example, according to inputs received from the user (for example, the first set time a1 and the second set time a2). As another aspect, the determination unit 244 may determine the first timing and second timing for each designated area T based on at least one parameter of the content of the work and the size of the designated area T. .

Note that the size of the designated area T may be input by the user, or may be specified by the work management device 2, for example, by image analysis. Specifically, as an example, the determination unit 244 may identify each packaging box 12 shown in the captured image, and specify a range along the outline of the packaging box 12 as the designated area T. The determination unit 244 dynamically identifies the size of an object (packing box 12, for example) moving on a conveyance means such as a sorter or a belt conveyor by image analysis, and specifies the designated area T. It may be. Note that the area setting unit 241 may perform the above processing instead of the determination unit 244.

Regarding the specification of the designated area T as described above performed by the determination unit 244 or the area setting unit 241, the determination unit 244 or the area setting unit 241 once specifies the designated area T of the predetermined object, and then re-identifies the designated area T of the object. The designated area T may be respecified based on the image. For example, it is assumed that the worker's hand hits the packaging box 12 during work and the position of the packaging box 12 moves. In this case, the determination unit 244 or the area setting unit 241 specifies the designated area T again, so that the work management device 2 can set the designated area T at the correct position according to the position of the packaging box 12. .

Specifically, the determination unit 244 refers to a storage unit that stores a plurality of first set times a1 and second set times a2 in association with at least one of the content of the work and the size of the designated area T. Then, the first timing and second timing for each designated area T may be determined.

According to such a configuration, the first timing and second timing of each designated area T are determined according to the size of the designated area T, for example, so that the possibility that the designated area T has not been touched is eliminated. High-quality captured images can be suitably identified. As a result, it becomes possible to place products with high precision based on the difference between the captured images.

<Modification 2>
FIG. 9 is a block diagram showing the configuration of a modified example of the embodiment of the present invention. In the above embodiment, it has been explained that the work management device 2 performs various processes based on the work content input by the user. However, as shown in FIG. 9, the work specifying unit 246 provided in the work management device 2 may specify the content of the work. In FIG. 9, the configuration of the work management system S100 is the same as that of the first embodiment, except that the control unit 24 of the work management device 2 includes a work specifying unit 246, so a duplicate explanation will be omitted.

The work specifying unit 246 is a functional unit for specifying the content of the work that the worker is doing. The work identifying unit 246 identifies the content of the work performed by the worker, for example, based on the movement indicated by the skeletal information of the worker's body created by the motion part detecting unit 243. Based on the movement pattern of a predetermined movement point of the skeleton information created by the movement part detection unit 243, the work specifying unit 246 stores, for example, a plurality of movement patterns of the movement point and a plurality of corresponding contents of the work in association with each other. The content of the work may be specified by referring to the storage unit 23 for the work. The "movement pattern" may be related to, for example, the speed at which the operating point moves and/or the time that the operating point stays within the specified area T.

The operating part detection unit 243 detects the position of the operating part, as described in the first embodiment, based on the position of the operating point related to the work content specified by the work specifying unit 246 in this way.

In this way, the work management device 2 is configured to specify the work content based on the movement of the working part of the worker's hand, and detect the position of the working part based on the relevant action point of the specified work content. According to the invention, the content of the work can be specified without the user inputting the content of the work, and moreover, the movement part of the worker is detected based on the position of the movement point appropriate according to the content of the work. As a result, the difference calculation process is performed on the captured images at a suitable timing according to the content of the work, and it becomes possible to accurately determine whether the product 5 has been placed.

<Modification 3>
In the above embodiment, it has been explained that the first captured image and the second captured image are captured at timings before and after the predetermined reference time t (see FIG. 6) when the worker's hand enters the designated area T. . However, the determination unit 244 may specify the first captured image and the second captured image to be used in the difference calculation process, for example, using the timing when the worker takes the product 5 from the inventory box 11 as a trigger.

Specifically, the determination unit 244 sets the reception timing at which the output from the detection means for detecting that the worker has picked up the product 5 from the inventory box 11, which is the object placement area where the product 5 is placed, as the first timing. It may be one timing. The detection means is, for example, a device that measures the weight of the inventory box 11 or an infrared sensor that detects the movement of the worker's hand. The first timing is not strictly limited to the timing at which the output from the detection means is received; for example, the determination unit 244 may set the first timing to be a timing that is a predetermined time earlier or later than the reception timing.

In determining the first timing and/or the second timing, the distance from the inventory box 11 to the designated area T may be taken into consideration. Therefore, the determination unit 244 may determine the first timing and/or the second timing based on the distance between the inventory box 11 and the designated area T. Specifically, the determination unit 244 refers to the storage unit that stores the distances between the plurality of inventory boxes 11 and the specified area T and the plurality of first timings and/or second timings in association with each other, and determines the first timing. The timing and/or the second timing may be determined.

For example, in a configuration in which the determination unit 244 determines a timing that is a predetermined time later than the reception timing at which the output from the detection means is received as the first timing, the distance between the inventory box 11 and the designated area T is long. If the distance between the inventory box 11 and the specified area T is short, the first timing is, for example, "3 seconds after the reception timing", and if the distance between the inventory box 11 and the designated area T is short, the first timing is "0.5 seconds after the reception timing", for example. The determining unit 244 may determine the first timing to be the first timing, the longer the distance between the inventory box 11 and the specified area T is, the longer the time has elapsed from the reception timing. .

Although the specific configuration of the present invention has been described above as embodiments and modified examples, the present invention is limited to the above and can be modified in various ways. For example, in the work in this embodiment, the worker moves the product 5 from the inventory box 11 to the packaging box 12, but the location where the inventory product 5 is placed is It is not limited to a box, but may be a shelf, for example. Furthermore, instead of being placed in a box, the products 5 may be placed in a predetermined sorting tray, for example.

The work management system of the present invention can be used not only for sorting products but also for assembling parts, for example. Work management system S100 may detect a body part other than the worker's hand as a working part. Here, the "worker's operating part" may literally be a part of the worker's body, but depending on the content of the work, it may also include objects to be detected such as tools, implements, or jigs used by the worker. It is also assumed that it is better to detect it using the work management system S100. Therefore, the work management device 2 may detect a detection target such as a tool, tool, or jig used by a worker as an operating part, and may perform various processes as described above based on the detection result. .

When assembling parts, it is desirable to correctly determine that the parts have been attached to a specific position. Therefore, similarly to the first embodiment, the first captured image taken at the first timing before the worker's movement part detected by the movement part detection unit 243 enters a predetermined designated area, and If the difference between the part and the second captured image taken at the second timing after the component is inserted is equal to or greater than a predetermined threshold, the determination unit 244 may determine that the component is placed at a predetermined position. When attaching a component, the determination unit 244 may not only simply detect the presence or absence of the component, but may also specify the location of the component based on the difference between captured images, for example.

The determination unit 244 of the work management device 2 learns training data in which the movement pattern of the worker's movement part and the work content corresponding to the movement pattern of the work are associated with each other. It may include a learned model that outputs the work content, and the work content may be estimated using the learned model.

<Modification 4>
As illustrated in FIG. 8, when there are a plurality of packaging boxes 12 into which the product 5 can be placed, there is a risk that the operator may make a mistake in the packaging box 12 into which the product 5 should be placed. Therefore, for example, the determination unit 244 has a function of identifying the product 5 placed in the designated area T corresponding to each packaging box 12, and detects whether the desired product 5 is placed in the target designated area T. You can. For example, the determination unit 244 refers to product distribution information stored in the storage unit 23 that indicates which product 5 should be placed in which specified area T, and determines whether the desired product 5 is placed in the target specified area T. detect whether

As an example, when the determining unit 244 detects that the product 5 to be placed in the designated area T1 (see FIG. 8) is placed in the designated area T2, the determining unit 244 determines that the product 5 that should be placed in the designated area T1 (see FIG. 8) is placed in the designated area T2. For both the area T1 and the designated area T2 where the worker's hand has entered, the captured image at the first timing and the captured image at the second timing are identified as in the first embodiment, and the difference between these images is determined. Arithmetic processing may also be performed.

The determination unit 244 may specify in which designated area T the product 5 is placed (or not placed) based on the result of the difference calculation process. Further, the determination unit 244 may specify at which position within the specified area T2 the incorrectly placed product 5 is located.

The output unit 245 outputs the above identification result to the display unit 22 as an example. Although the manner in which the output unit 245 outputs the identification result is not limited to a specific manner, as an example, the output unit 245 outputs an image indicating the position of the incorrectly placed product 5 identified by the determination unit 244. Information or text information may be displayed on the display unit 22. Specifically, for example, the output unit 245 projects an image, text, etc. onto the packaging box 12 in which the incorrectly placed product 5 is located, thereby informing the worker that the product 5 is incorrectly placed. It may be possible to inform the user that the product 5 is placed and/or the location of the product 5.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes can be made within the scope of the gist. be. For example, all or part of the device can be functionally or physically distributed and integrated into arbitrary units. In addition, new embodiments created by arbitrary combinations of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiment resulting from the combination have the effects of the original embodiment.

Although the above description has focused on the invention of a work management device, this specification also discloses the invention of the above technical idea expressed as an invention of a method and an invention of a computer program. Specifically, this specification discloses a work management method that includes an image acquisition step, a motion part detection step, a determination step, and an output step.

1 Camera 2 Work management device 11 Inventory box 12 Packing box 21 Input section 22 Display section 23 Storage section 24 Control section 241 Area setting section 242 Image acquisition section 243 Operating part detection section 244 Judgment section 245 Output section 246 Work specification section a1 1st Set time a2 Second set time P11 Operating point P12 Operating point P21 Operating point P22 Operating point S Inventory area t Reference time T, T1, T2 Specified area

Claims (10)

an image acquisition unit that acquires a captured image generated by an imaging device capturing a work in which a worker places a work object in a predetermined designated area;
a working part detection unit that detects the position of the working part of the worker shown in the captured image;
a first captured image captured at a first timing before the operating region detected by the operating region detection unit enters the designated area; and a first captured image captured at a second timing after the operating region enters the designated area. If the difference between the image of the specified area in the first captured image and the image of the specified area in the second captured image is greater than or equal to a predetermined threshold, a determination unit that determines that the work object is placed at
an output unit that outputs the determination result of the determination unit;
A work management device comprising: The determination unit includes:
A captured image at the first timing in which the operating part is not within the designated area is specified as the first captured image, and an image captured at the second timing in which the operating part is not in the designated area is identified as the first captured image. Identification as the second captured image,
making the determination based on the identified difference between the first captured image and the second captured image;
The work management device according to claim 1. The determination unit determines, as the first timing, a time that is a first set time before a predetermined reference time when the operating part enters the specified area, and determines a second set time from the predetermined reference time as the first timing. determining a later time as the second timing;
The work management device according to claim 1 or 2. The imaging device generates a captured image including the plurality of designated areas,
The determination unit determines the first timing and second timing for each designated area,
The work management device according to claim 3. The motion part detection unit creates skeletal information of the body including a plurality of motion points based on the image of the worker's body shown in the captured image,
The output unit outputs a predetermined motion point related to the content of the work performed by the worker among the plurality of motion points to a body part corresponding to the predetermined motion point in the body image of the worker. superimpose them and display them on the image display section,
The work management device according to claim 1 or 2. The operating part detection unit creates skeletal information of the body based on the image of the worker's body shown in the captured image, and determines the content of the work performed by the worker, which is specified based on the skeletal information. detecting the position of the operating part based on the position of one or more operating points associated with the
The work management device according to claim 1 or 2. The motion part detection section refers to a storage section that stores a plurality of work contents in association with a plurality of predetermined motion points related to the work contents, and detects the work contents received from the user. in response to an input, determining the predetermined operation point related to the inputted content of the work;
The work management device according to claim 6. The operation part detection unit further includes a work specifying unit that specifies the content of the work performed by the worker based on the movement indicated by the skeletal information created by the movement part detection unit. detecting the position of the operating part based on the position of the operating point related to the content of the work;
The work management device according to claim 6. The determination unit includes:
A reception timing at which an output from a detection means for detecting that the worker has picked up the work object from an object arrangement area where the work object is arranged, or a predetermined time earlier than the reception timing. or a later timing is set as the first timing;
The work management device according to claim 1 or 2. an image acquisition step of acquiring a captured image generated by an imaging device capturing a work in which a worker places a work object in a predetermined designated area;
a working part detection step of detecting the position of the working part of the worker shown in the captured image;
a first captured image captured at a first timing before the operating region detected in the operating region detection step enters the designated area; and a first captured image captured at a second timing after the operating region enters the designated area. If the difference between the image of the specified area in the first captured image and the image of the specified area in the second captured image is greater than or equal to a predetermined threshold, a determination step of determining that the work object is placed at
an output step for outputting the determination result in the determination step;
A work management method in which each step is performed by a computer.