JP7119687B2 - Work process control system, work process control method, and work process control program - Google Patents

Description

The present invention relates to a work process management system, a work process management method, and a work process management program.

Techniques for managing the progress of work have been proposed in order to improve work efficiency at work sites in the manufacturing industry.

For example, in Japanese Unexamined Patent Application Publication No. 2002-100000, a server stores a work content and a progress plan table in which the scheduled execution time is assigned to each of a plurality of work processes, and receives the progress status of the work content from each process. Then, the server judges whether or not the received progress status is behind the plan in the progress schedule table, and displays the work contents whose progress status is behind schedule and the work contents which are not behind schedule. are displayed as a progress table. As a result, the progress of work at the work site can be managed in near real time. Furthermore, in Patent Document 1, a worker whose work is progressing ahead of schedule is directed to a process in which a worker who is behind schedule is present to assist the work, thereby preventing delays in the work time of each of a plurality of processes. is doing.

JP 2013-117881 A

In the prior art, the progress of work is managed in near-real time by determining whether the progress is ahead of or behind the plan in the progress schedule.

However, with such a conventional technique, the degree of progress of work in each process cannot be known. For this reason, a worker who comes to support a certain process wastes time until the start of work because he or she does not know when to start the setup process. In spite of this, the progress of the previous process is bad and the work is wasted because it cannot be done. Furthermore, when a worker goes to another process to assist, the process that the worker was originally working on may be delayed.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a work process management system that can grasp the progress of work in a certain process in real time and notify other processes, It is to provide a work process control method and a work process control program.

The above objects of the present invention are achieved by the following means.

(1) a photographing unit for photographing an area where a worker works;
a work recognition unit that acquires moving image data from the imaging unit and recognizes the work performed by the worker in the first step from the movement of the worker in the moving image data;
a progress determination unit that determines the progress of the work in the first step from the recognized work of the worker;
When the degree of progress determined by the degree-of-progress determination unit reaches a predetermined degree of progress, the degree of progress is transferred to an information terminal that transmits information to a worker performing work in a second step different from the first step. a notification processing unit that notifies that has reached the predetermined degree of progress;
has
The work recognition unit acquires the coordinate values of the joints of both wrists of the worker from the moving image data, and the difference between the acquired coordinate values of the joints of the right wrist and the acquired coordinate values of the joints of the left wrist. A work process control system that recognizes specific work content from

(2) The work recognizing unit, within a frame of moving image data of the worker photographed from above, is arranged along a first axis corresponding to the horizontal direction of the worker and a second axis intersecting the first axis. In the configured two-dimensional coordinate system, when the coordinate values of one wrist joint on the second axis do not change and the coordinate values of the other wrist joint on the second axis change , The work process management system according to the above (1), which recognizes the work of removing the tape from a predetermined place or the work of returning the tape to a predetermined place when packing .

(3) The work recognizing unit, within a frame of moving image data of the worker photographed from above, is arranged along a first axis corresponding to the left-right direction of the worker and a second axis intersecting the first axis. In the constructed two-dimensional coordinate system, if the coordinate values in the second axis of both wrist joints do not change for a certain period of time, or if one wrist joint moves linearly with respect to the other wrist joint The work process management system according to the above (1 ), which recognizes the task of attaching a tape when packing if the work is done.

(4) The first step consists of a plurality of tasks, and a standard work time required for each task out of the plurality of tasks or a task combining some of the tasks. Having a storage unit that stores standard work time,
The progress determination unit determines the time from the start time of the work in the first step to the time when the work recognition unit recognizes the one work or the grouped work among the plurality of works, and the standard comparing the work time with the work time to determine the progress of the first step indicating whether the work recognized by the work recognition unit is ahead of or behind the standard work time;
The work process management system according to any one of (1) to (3) above, wherein the notification processing unit notifies the determined progress of the first process to the information terminal.

(5) The first step consists of a plurality of tasks, and a standard work time required for each task out of the plurality of tasks or a task combining some of the tasks. Having a storage unit that stores the standard work time and stores the proficiency level of each worker for a plurality of workers,
The work recognition unit identifies a worker who performs the work of the first step from the moving image data,
The progress determination unit determines the one work or the combined work among the plurality of works by the work recognition unit based on the skill level of the identified worker and the start time of the work in the first step. Predicting the work end time of the first step from the time until the recognized time and the standard work time,
The work process management system according to any one of (1) to (4) above, wherein the notification processing unit notifies the information terminal of the predicted work finish time of the first process.

(6) The information terminal is a signal lamp or display installed in the workplace of the second step, or a mobile information terminal carried by the worker of the second step, (1) to The work process management system according to any one of (5).

(7) step (a) of acquiring video data of an area where the worker is working;
a step (b) of recognizing the work performed by the worker in the first step from the motion of the worker in the moving image data;
Step (c) of determining the progress of the work in the first step from the recognized work of the worker;
When the determined degree of progress reaches a predetermined degree of progress, the degree of progress is transferred to an information terminal that transmits information to a worker performing work in a second step different from the first step. step (d) of notifying that the
has
The step (b) acquires the coordinate values of the joints of both wrists of the worker from the moving image data, and compares the acquired coordinate values of the right wrist joint and the acquired coordinate values of the left wrist joint. A work process management method that recognizes specific work content from differences .

(8) In step (b) , a first axis corresponding to the left-right direction of the worker and a second axis intersecting the first axis in a frame of moving image data photographing the worker from above. In a two-dimensional coordinate system composed of In this case, the work process management method according to (7) above, wherein the work of removing the tape from a predetermined place or the work of returning the tape to a predetermined place is recognized when packing .

(9) In step (b) , a first axis corresponding to the left-right direction of the worker and a second axis intersecting the first axis in a frame of video data of the worker photographed from above. , if the coordinate values of both wrist joints on the second axis do not change for a certain period of time, or if one wrist joint is linear with respect to the other wrist joint The work process management method according to (7 ) above , wherein when the product is moving, it is recognized as the task of applying tape during packing .

(10)
The first step consists of a plurality of tasks, and the standard work time required for each task out of the plurality of tasks or the standard work time required for each task in which some of the plurality of tasks are combined is stored in advance,
comparing the standard work time with the time from the start time of the work in the first step to the time of recognition of the one work out of the plurality of works or the grouped work in the step (b); a step (e) of determining the progress of the first step indicating whether the work recognized in the step (b) is ahead of or behind the standard work time;
The work process management method according to any one of the above (7) to (9), wherein in the step (d), the information terminal is notified of the determined progress of the work in the first process.

(11) The first step consists of a plurality of tasks, and a standard work time required for each task out of the plurality of tasks or a task combining some of the tasks Pre-store the standard work time and the proficiency level of each worker for multiple workers,
a step (f) of identifying a worker who is working in the first step from the moving image data;
the proficiency level of the identified worker, the time from the start time of the work in the first step to the time when the one work out of the plurality of work or the grouped work is recognized in the step (b) , and a step (g) of predicting the work finish time of the first process from the standard work time,
The work process management method according to any one of the above (7) to (10), wherein in the step (d), the predicted work finish time of the first process is notified to the information terminal.

(12) A program for causing a computer to execute the work process control method according to any one of ( 7 ) to (11) above.

According to the present invention, the work performed by the worker in the first process is recognized from the moving image data of the worker in the first process, the progress of the first process is determined from the recognized work, and the predetermined progress is determined. When the time comes, the operator in the second process different from the first process is notified to the information terminal that can be seen. As a result, the worker of the second process can know the progress of the first process in real time. Therefore, it is possible to reduce or eliminate wasteful preparation and delay in starting work.

BRIEF DESCRIPTION OF THE DRAWINGS It is the schematic which shows the whole structure of the work process management system of embodiment. It is a functional block diagram for explaining the function of the work process management system of the embodiment. It is a figure which shows an example of the image which image|photographed the packing process. It is an explanatory view for explaining the work contents of a packing process. It is an explanatory view for explaining the work contents of a packing process. It is an explanatory view for explaining the work contents of a packing process. 4 is a graph showing Y-axis coordinate values of left and right wrist joints R1 and L1 in chronological order. It is a proficiency level table which shows an example of the proficiency level of individual workers. FIG. 10 is a diagram showing an example of a screen display of a portable information terminal that has received a progress notification; It is a flow chart which shows the processing procedure of the process control method. FIG. 11 is a flowchart showing the processing procedure of the process control method following FIG. 10; FIG. FIG. 5 is a schematic diagram showing the overall configuration of Modification 1; FIG. 11 is a schematic diagram showing the overall configuration of Modification 2;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and overlapping descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may differ from the actual ratios.

FIG. 1 is a schematic diagram showing the overall configuration of the work process management system of the embodiment. FIG. 2 is a functional block diagram for explaining functions of the work process management system of the embodiment.

This work process management system 1 includes camera units 11 to 13 respectively installed in the work areas of a plurality of processes 51 to 53, a server 20 to which the camera units 11 to 13 are connected, and work performed in the processes 51 to 53. It has a portable information terminal 41 carried by a person or a laminated signal lamp 42 provided in the work place of processes 51 to 53.

A plurality of processes 51 to 53 are processes after manufacturing a product (article), which are an inspection process 51, a packing process 52, and a palletizing process 53. All of these steps 51-53 require work by workers 31-33. In the inspection step 51, the worker 31 inspects the produced article. In the packing process 52, after the inspection, the worker 32 packs the articles that have passed the inspection into cardboard boxes. In the palletizing process 53, the worker 33 places the packed cardboard boxes on the pallet. These steps 51 to 53 are merely examples, and the present invention can be applied to other steps.

The camera units 11 to 13 are all the same, and consist of a camera 15 and a computer (COMP: Computer) 16 integrated with the camera 15 .

The camera 15 becomes the imaging unit 101 (see FIG. 2). The camera 15 photographs the areas where the workers 31 to 33 work in the steps 51 to 53, respectively. Therefore, the camera units 11 to 13 are installed at positions where the movements of the workers performing the work in each of the steps 51 to 53 can be photographed. For example, when the workers 31 to 33 mainly move their arms up and down to perform the work, it is preferable to install the camera at a position where the vertical movements of the arms of the workers 31 to 33 can be photographed. Moreover, when the workers 31 to 33 mainly move their arms back and forth to perform the work, it is preferable to install the camera at a position where the back and forth movements of the arms of the workers 31 to 33 can be photographed. The same applies to the actions of the other workers 31 to 33, and is installed at a position where the contents of the work performed by the workers 31 to 33 can be photographed. A general color movie camera can be used as the camera 15 itself.

The computer 16 functions as a work recognition unit 102 and a progress determination unit 103 (see FIG. 2) (details will be described later). The computer 16 includes a processor (CPU: Central Processing Unit or GPU: Graphics Processing Unit) that executes various arithmetic processing, ROM (Read Only Memory), RAM (Random Access Memory) and HDD (Hard Disk Drive) or SSD ( A storage device (storage unit 111) configured by appropriately combining a Solid State Drive, etc. is provided. In addition, in order to receive video data from the camera 15, for example, PCI Express, HDMI (registered trademark) (High-Definition Multimedia Interface), and other video signal interfaces (or a plurality of these) are provided.

The computer 16 may be installed in the same housing as the camera 15, or may be integrated by connecting the housing of the camera 15 and the housing of the computer 16, for example. For this reason, it is preferable that this computer 16 be a small-sized computer such as a one-board computer or a one-chip computer. However, the computer 16 does not necessarily have to be integrated with the camera, and may be a personal computer connected to the camera 15 in a separate housing via a video signal cable. However, in this embodiment, the camera 15 and the computer 16 are in one-to-one correspondence, and the moving image data from one camera 15 is used to function as the work recognition unit 102 and the progress determination unit 103 .

This computer 16 also has an interface for connecting with the server 20 . Examples include Ethernet (registered trademark), Bluetooth (registered trademark) based on standards such as IEEE 1394, and wireless communication interfaces such as IEEE 802.11.

The server 20 functions as a notification processing unit 104 (see FIG. 2) (details will be described later). The server 20 is also a computer, and includes a processor, a storage device (storage unit 112), and various interfaces (the hardware configuration is the same as the computer 16 integrated in the camera 15 described above, so the description is omitted).

The server 20 has an interface for connecting with the computer 16 of the camera units 11-13. It is similar to the computer 16 of the camera units 11-13.

The server 20 is also connected to the mobile information terminals 41 possessed by the workers 31 to 33 via the wireless communication interface. In addition, the server 20 is connected to stacked signal lamps (hereinafter referred to as signal lamps 42) in the work areas of the respective processes 51-53.

The mobile information terminal 41 and the signal lamp 42 are the information terminal 105 that receives and displays the notification from the server 20 when the progress of the preceding process reaches a predetermined progress (see FIG. 2).

The mobile information terminal 41 is, for example, a smartphone, a tablet, or a dedicated terminal having a display. The portable information terminal 41 receives notification from the server 20 that the degree of progress of the preceding process has reached a predetermined degree of progress, and displays it on the display.

A signal lamp 42 is provided in each process. The signal lamp 42 changes the lighting color of the signal lamp 42 upon receiving a notification from the server 20 indicating that the degree of progress of the preceding process has reached a predetermined degree of progress. For example, a notification that the progress of the previous process has been completed normally is colored green, a notification that the progress of the previous process is behind schedule is colored yellow, and a notification that the work of the previous process has stopped is colored red. Of course, the color for lighting the signal lamp 42 may be changed according to the content of the notification.

Next, with reference to FIG. 2, functions of the work process management system of the embodiment will be described.

The imaging unit 101 is the camera 15 as already described.

The work recognition unit 102 acquires video data from the camera 15 and recognizes the work of the worker.

Recognition of the worker's work is performed from the moving image data from the camera 15 . For example, using skeleton recognition technology, the positions (coordinate values) of specific joints of the worker are obtained, and the work content is recognized from the joint positions (coordinates).

Skeleton recognition technology includes, for example, Open Pose. Open Pose is a technology for detecting keypoints (feature points) and estimating relationships between keypoints, announced by CMU (Carnegie Mellon University) at CVPR2017, an international conference on computer vision. By using open poses, it is possible to determine the coordinates of feature points on the human body, such as the positions of human joints (for example, http://leapmind.io/blog/2018/01/22/%E4%BD%93 %E3%81%AE%E4%BD%8D%E7%BD%AE%E3%82%84%E5%90%91%E3%81%8D%E3%81%8C%E6%8E%A8%E5 %AE%9A%E3%81%A7%E3%81%8D%E3%82%8Bopenpose%E3%82%92%E7%B4%B9%E4%BB%8B/). A similar skeleton recognition technology is Deep Pose (see https://www.slideshare.net/mitmul/deeppose-human-pose-estimation-via-deep-neural-networks).

Recognition of a worker's work using the skeleton recognition technology will be described with a specific example. FIG. 3 is a diagram showing an example of an image of the packing process.

The image example of FIG. 3 shows one frame of video (moving image data) captured by the camera 15 in the packing process 52 . The photographing area 200 includes the area where the worker 32 works. Here, the camera 15 is installed so as to photograph the worker 32 from almost directly above. In moving image data, an XY two-dimensional coordinate system is set within the frame screen.

A work area 201 for detecting that the worker 32 has come to the packing process 52 is set within the imaging area 200 . A work area 201 is an area (including movement for work) that exists when the worker 32 works in this process, and is set by coordinate values of a two-dimensional coordinate system. Similarly, the position where the cardboard box 80 to be worked on is placed is also set as a work area 202 by coordinate values. Furthermore, a tool area 203 in which tools required for work are placed is set by coordinate values. The tool here is a packing adhesive tape 90 (OPP (Oriented Polypropylene) tape, so-called gummed tape, etc.) that is attached when the cardboard box 80 is closed.

A worker 32 is present in this photographing area 200 . In this image, the skeleton of the worker 32 is displayed by skeleton recognition technology. In this image, a right shoulder 301R, a right upper arm 302R, a right forearm 303R, a left shoulder 301L, an upper left arm 302L, and a left forearm 303L are shown as the skeleton of the worker 32. FIG.

The skeleton recognition technology can obtain the current coordinate values of these skeletons and joints (coordinate values within each frame in moving image data). Here, the coordinate values of the wrist joints R1 and L1 and the coordinate values of the shoulder joints R2 and L2 are obtained. The coordinate values of wrist joints R1 and L1 are used to recognize hand motion. Also, the coordinate values of the shoulder joints R2 and L2 are used to recognize the position of the worker 32. FIG. Of course, since the coordinate values of other skeletons and joints are also obtained, they are used as necessary.

The work recognition unit 102 recognizes the positions (coordinate values) of the wrists and shoulders of the worker 32 using the skeleton recognition technology.

Then, the work recognition unit 102 recognizes what work is being performed from the obtained positions (coordinate values) of the wrists and shoulders of the worker 32 . There are two ways to do this. The first is a method of recognizing from time-series changes in coordinate values of specific joints of the worker 32 . The second method is to recognize whether or not the coordinate values of a specific joint of the worker 32 have entered a predetermined area corresponding to the work content.

First, a method of recognition based on time-series changes in coordinate values of specific joints of the worker will be described.

4 to 6 are explanatory diagrams for explaining the work contents of the packing process. 4 to 6 are examples of images in the packing process 52 as in FIG. 3 described above, and show the contents of the work following FIG.

Here, FIG. 3 described above is used as the start of the work. FIG. 4 shows the operation of removing adhesive tape 90 from tool area 203 . FIG. 5 shows the operation of closing the flap 81 of the corrugated cardboard box 80 and sticking the adhesive tape 90 thereon. FIG. 6 shows the operation of returning the adhesive tape 90 to the tool area 203 after the taping is completed.

FIG. 7 is a graph showing the Y-axis coordinate values of the left and right wrist joints R1 and L1 in chronological order. The vertical axis is the Y-axis coordinate value, and the horizontal axis is time. From FIG. 7, it can be seen that the left and right wrist joints R1 and L1 move in the Y-axis direction. The Y-axis direction here is the Y-axis direction on the screen of the moving image data, as shown in FIGS.

First, the start of work is the state shown in FIG. 3, which is the start of packing work for one piece. Looking at FIG. 7, both the Y-axis coordinate values of the wrist joints R1 and L1 are high at the beginning of the time series. This is because he held the empty cardboard box 80 with both hands and placed it on the work area 202 of the workbench.

A pattern in which both the Y-axis coordinate values of the wrist joints R1 and L1 are high at the beginning of such a time series is stored in advance in the storage unit 111 as a task of "start packing one piece". Then, if the work recognition unit 102 detects Y-axis coordinate values of the wrist joints R1 and L1 similar to this pattern from the moving image data, it recognizes that the work of "start packing one piece" has been performed. .

Next, looking at FIG. 7, only the wrist joint R1 has a Y-axis coordinate value that suddenly increases and then decreases. This is because he picked up the adhesive tape 90 with his right hand as shown in FIG.

Therefore, a pattern in which the Y-axis coordinate value of only this wrist joint R1 suddenly increases and immediately decreases is stored in the storage unit 111 as "tape picking work", and the work recognition unit 102 recognizes the work from the moving image data. used for

Subsequently, looking at FIG. 7, the Y-axis coordinate values of the wrist joints R1 and L1 both remain at approximately the same position, and time has passed. Moreover, although the wrist joints R1 and L1 move differently during the exercise, the wrist joints R1 and L1 remain in the same position for a certain amount of time over a total of three times (P1 to P3). As shown in FIG. 5, this is the work of closing the flap 81 of the corrugated cardboard box 80 and pasting the adhesive tape 90 (pasted tape 91). Here, P1 to P3 are performed three times, P1 for closing the flaps 81 together and applying tape, and P2 and P3 for applying tapes to the corners of the side portions of the flaps 81 . That is, here, the so-called "H sticking" is used for the cardboard box 80 (see the stuck tape 91 in FIG. 6). Therefore, the pattern in which the Y-axis coordinate values of the wrist joints R1 and L1 remain at approximately the same position and a certain amount of time elapses is stored in the storage unit 111 as "tape work", and the work recognition unit 102 is used to recognize work from video data. The number of taping operations depends on how the tape is applied. For example, in the case of pasting in a cross shape, the Y-axis coordinate values of the wrist joints R1 and L1 are at almost the same position twice. Also, when the tape is applied to the portion where the flaps 81 are closed together, the Y-axis coordinate values of the wrist joints R1 and L1 are at approximately the same position only once.

Here, the expression that the Y-axis coordinate values of the wrist joints R1 and L1 are at "substantially the same position" means that the Y-axis coordinate values of the wrist joints R1 and L1 are within the allowable error range.

In many cases, the tape application work is performed by holding the tape with one hand and applying the tape while rotating and pulling the tape roll with the other hand. Therefore, the Y-axis coordinate values of the wrist joints R1 and L1 are often at the same position. However, depending on the operator 32, the position of the hand holding the tape and the hand pulling the tape roll may be misaligned (the Y-axis coordinate values of the wrist joints R1 and L1 may be misaligned), or the cardboard box 80 may be placed obliquely. do. Therefore, the Y-axis coordinate values of the wrist joints R1 and L1 may differ slightly. Therefore, an allowable error range is provided for the difference between the Y-axis coordinate value of the wrist joint R1 and the Y-axis coordinate value of the wrist joint L1.

This can be represented by the following formula (1), where "R1y" is the Y-axis coordinate value of the right wrist joint R1 and "L1y" is the Y-axis coordinate value of the left wrist joint L1.

|R1y−L1y|≦error range (1)
Such an error range is preferably determined by, for example, experimentation or actual results. A specific value of the error range varies depending on the size of the cardboard box 80 and the operator, but is, for example, about 1 to 10 cm.

Although only the Y-axis coordinate values of the wrist joints R1 and L1 are focused here, it is determined whether or not the positions of the wrist joints R1 and L1 move substantially linearly from both the X-axis and the Y-axis. good too. As noted above, taping operations often involve one hand moving away from the other in a straight line. Therefore, if the wrist joint L1 moves almost linearly with respect to the wrist joint R1 based on the movement of the wrist joints R1 and L1, that is, the changes in the X-axis and Y-axis values of the wrist joints R1 and L1, it is recognized as a taping operation. can. In this case as well, it is preferable to set an allowable error range similar to the above-described error range for substantially linear.

Further, when the rectangular parallelepiped cardboard box 80 is placed obliquely, the coordinate axes are rotated (from the Y axis to the Y′ axis) according to the extending direction of the side of the placed cardboard box 80, and the coordinates after rotation are These determinations may be made at the Y'-axis coordinate values in the system. Also, if a system recognizes a three-dimensional coordinate system from video data, the vertical direction is taken as the Z-axis, and the Z-axis coordinate values of the wrist joints R1 and L1 are at approximately the same position. You may do so. In the three-dimensional coordinate system, when the Z-axis coordinate values of the wrist joints R1 and L1 are at substantially the same position, it means that both hands are moving substantially horizontally. This corresponds to the operation of holding the tape with one hand and applying the tape while rotating and pulling the tape roll with the other hand.

Next, looking at FIG. 7, the Y-axis coordinate value of only the wrist joint R1 suddenly increases and then decreases again. This is because the used adhesive tape 90 was returned to the tool area 203 with the right hand.

Therefore, a pattern in which the Y-axis coordinate value of only the wrist joint R1 for the second time suddenly increases and immediately decreases is stored in the storage unit 111 as a "tape return operation", and the operation recognition unit 102 recognizes the operation from the moving image data. used to

Furthermore, a series of operations of removing the tape, applying the tape, and returning the tape are performed immediately after the start of the operation (the operation of packing the articles has not yet been performed). Therefore, this series of operations is the work of attaching the adhesive tape 90 to the bottom surface of the corrugated cardboard box 80 to form the bottom cover. By storing such a series of movement patterns in the storage unit 111, the work recognition unit 102 detects movement similar to this series of patterns from the moving image data, and sticks tape on the bottom surface of the cardboard box 80. Recognize it as work.

Next, looking at FIG. 7, the Y-axis coordinate values of the wrist joints R1 and L1 change irregularly. Such movement occurs because the left and right hands move toward and away from the cardboard box 80 in order to pack articles into the cardboard box 80 .

Therefore, such a pattern in which the Y-axis coordinate values of the wrist joints R1 and L1 change irregularly along the time series is stored in the storage unit 111 as "item packing work", and the work recognition unit 102 recognizes the moving image data. Used to recognize work from.

Subsequently, referring to FIG. 7, again, there are tape removal work, tape application work (P4 to P6), and tape return work. The final tape return operation is shown in FIG. In FIG. 6, the tape 91 attached to the corrugated cardboard box 80 that has been packed has an H shape when viewed from above. This is the second time for a series of operations consisting of tape removing operations, tape applying operations (P4 to P6), and tape returning operations. From this, the work recognition unit 102 detects a movement similar to this series of patterns from the moving image data, and if it is the second time, it recognizes that it is the work of applying tape to close the upper surface of the cardboard box 80. do.

Thereafter, the Y-axis coordinate values of wrist joints R1 and L1 are both low. This is the action of removing the hand from the corrugated cardboard box 80 because the packing work for one item has been completed.

When the movements of the hands in the packing work are followed in chronological order in this way, there are characteristics for each work content, and these characteristics can be grasped as changes in the Y-axis coordinate values of the wrist joints R1 and L1. Therefore, as already described, the Y-axis coordinate values of the wrist joints R1 and L1 for each task are stored as patterns in the storage unit 111, and when the actual task is performed, the movements of the wrist joints R1 and L1 are determined from the moving image data. Each task can be recognized by detecting movements similar to

In particular, the taping work is a characteristic motion in which the Y-axis coordinate values of the wrist joints R1 and L1 are at approximately the same position and remain constant for a certain period of time. Therefore, the taping operation can be easily distinguished from other operations. In this example, since the taping operations are performed six times (P1 to P6), only these six taping operations may be recognized.

Next, a method of recognition based on whether or not the coordinate values of a specific joint of the worker have reached a predetermined position will be described.

Work is started, for example, when it is detected that the worker 32 has entered the work area 201 . For this, for example, when the shoulder joint R2 or L2 of the worker 32 enters the work area 201, it is determined that the work has started. Further, in the packing process 52, the worker 32 may not leave the work area 201 in order to continuously perform a plurality of packing operations. In such a case, work is started, for example, when it is detected that the cardboard box 80 is placed on the work area 202 (the work in this case is the cardboard box 80). Alternatively, in the packing process 52, the work may be started at the time when the first adhesive tape 90 is removed (described later).

Subsequently, the operator 32 goes to pick up the adhesive tape 90, sticks the tape, and then goes to return the tape. Therefore, when the coordinate values of the wrist joint R1 enter and exit the tool area 203 twice, it can be understood that the operation of removing the tape and returning the tape has been performed. From this, it can be estimated that the work of attaching the adhesive tape 90 to the corrugated cardboard box 80 was performed between tape removal and tape return. As previously described, the packaging step 52 first forms the bottom of the cardboard, then stuffs it with items and then tapes the top lid. Therefore, the coordinate values of the wrist joint R1 enter the tool area four times in total.

In this way, the work content can be recognized from the positions (coordinate values) of the shoulder joint and the wrist joint as specific joints and the coordinate values of the region corresponding to the work content.

Each of the above work content recognition methods may be performed independently, or a time-series determination and a determination in a predetermined area may be combined. For example, the start of packing work for one item is determined by the worker 32 entering the work area 201 or placing the cardboard box 80 in the work area 202 . Also, when the wrist joint R1 enters the tool area 203, it is used as a trigger for taping. When the Y-axis coordinate values are almost the same at the time, it is recognized as a taping operation.

Such recognition of work is also performed by the work recognition unit 102 in other processes. For example, if the inspection process 51 is a surface inspection of an article, recognition is made by carrying the article into the work area 201, entering the inspection area by the operator 31, or taking out a necessary tool. Further, in the palletizing process 53, for example, when stacking four cardboard boxes 80 on a pallet, the same operation is repeated four times. Therefore, if the same operation is repeated four times in chronological order, it is recognized that the loading operation for one pallet is completed.

The work recognition unit 102 transmits the recognized work content to the progress determination unit 103 each time the work content (including the start of work) is recognized (each time the work content is determined).

The progress determination unit 103 determines the current progress of the work from the received work content. In order to determine the degree of progress, the progress determination unit 103 pre-stores the work content and the degree of progress corresponding thereto in the storage unit 111 . Then, the progress determination unit 103 obtains the progress by comparing the work content from the work recognition unit 102 with the stored work content.

For example, when the degree of progress is to be obtained from the taping work, as shown in FIG. 7, there are six taping tasks P1 to P6. Therefore, the degree of progress is defined in association with the tape pasting operations P1 to P6. For example, the tape application work P1 has a progress of 1%, the tape application work P2 has a progress of 5%, the tape application work P3 has a progress of 10%, and then there is the stuffing operation, and the tape application work P4 has a progress of 90%. For example, the degree of progress of the taping work P5 is 95%, and the degree of progress is 100% when the last tapeing work P6 is completed.

Such a definition of the degree of progress is, for example, by having a model worker actually perform the work, obtaining each work content and the degree of progress at that time, and storing the degree of progress corresponding to the work content in the storage unit 111. Remember. In addition, for example, each work content may be associated with the degree of progress based on the results of past work, or the degree of progress assumed in design may be associated with the content of work.

The progress determination unit 103 can also determine the progress of each work along with the progress. Here, the progress indicates whether the actual work progresses faster or lags behind the standard progress time of the work.

For this purpose, the storage unit 111 stores the standard work time. The standard work time is, for example, the time from start to finish of all work when one process consists of a plurality of works. Also, even for one process consisting of a plurality of tasks, the standard work time may be determined for each task or for each task combining some of the plurality of tasks.

The progress determination unit 103 records the time when the work recognition unit 102 recognizes the work content, and calculates the time from the work start time (described later) to the time when the work content is recognized, and the standard work read from the storage unit 111 . Contrast time. Based on this, it is determined whether the actual work is advanced or delayed with respect to the standard work time.

The work start time is basically the start time of the first work in the process, in order to obtain the progress from start to finish in one process. However, when the progress status is to be obtained for each work (or a plurality of works grouped together), the work start time of each work (or a plurality of work grouped together) may be used.

An example will be explained. For example, the packaging process shown in FIG. 7 consists of multiple operations, as already explained. And there are 6 times of taping work from P1 to P6. Therefore, for these six taping operations P1 to P6, the standard work time for bottom taping work that summarizes the first three times, the standard work time for top taping work that summarizes the latter three times, and the bottom taping work The standard work time for the packing work between the work and the top surface taping work is memorized.

Then, the progress determination unit 103 can determine whether each task out of a plurality of tasks or each group of tasks is early or late by comparing the actual task recognition time with the standard task time.

One example is shown in Table 1. Here, the first tape removing work is the start of one packing work, the first tape returning work to the second tape removing work is the packing work, and the end of the tape applying work P6 is the end of one packing work.

As shown in Table 1, the recognized work is 5 seconds earlier until the end of the taping work P3, that is, until the end of the bottom taping. However, there is a one minute delay in packing work. Finally, the work was completed with a delay of 50 seconds. From this progress, it can be inferred that the taping work alone will progress faster than the standard, but the stuffing work will be delayed. Also, when the progress status is known, it is notified to the information terminal 105 of the next process (second process), so that it becomes possible to know whether the preceding process (first process) is progressing quickly or late.

Furthermore, the progress determination unit 103 predicts the work end time based on the worker's individual proficiency level and the standard work time. For this reason, the storage unit 111 stores the standard work time and the proficiency level of each worker in advance. The progress determination unit 103 reads the standard work time and the proficiency level of the individual worker from the storage unit 111 and predicts the work end time.

FIG. 8 is a proficiency level table showing an example of individual proficiency levels of workers. Here, the proficiency level of an individual worker is shown as a multiplication rate for the standard work time. In addition, the proficiency level is divided into the tape application work P3 from the start of the work and the tape application work P6, which is the actual end of the work after the packing work is finished. Such a proficiency level table is stored in the storage unit 110 in advance.

To predict the work finish time, it is necessary to identify the individual worker. For this purpose, for example, the worker may input personal identification information (ID (identification)) each time he/she enters the process. It may be automatically read by a reader or the like.

In addition, since the camera 15 photographs the worker in the embodiment, individual identification may be performed using the moving image data. For this purpose, for example, face recognition technology using moving image data can be used. In this case, the camera 15 is installed at a position where not only the movement of the operator's hands and arms described above but also the face can be photographed (a separate camera may be provided for face authentication). Existing technology can be used for face recognition technology. Further, since it is possible to recognize the skeleton of the face even in the above-described open pose, an individual may be identified from the facial skeleton recognized in the open pose. By using such face recognition technology and skeleton recognition technology, an individual worker can be identified and specified only by photographing with the camera 15 without ID input or a wireless ID tag.

The progress determination unit 103 acquires the proficiency level of the worker currently in the process by identifying the individual worker and referring to the proficiency level table, and multiplies it by the standard work time to determine the completion of the work. Predict time.

The work end time is predicted, for example, from the time when each work content is recognized (the predicted end time is called the work end prediction time). In addition, as shown in FIG. 8, since the proficiency level is divided into two according to the work content, the actual time required to tape the bottom surface of the cardboard box 80 from the start of the work to the tape application P3 can be understood. . Also, since the stuffing of the articles is practically completed by tape application P6, the time until the end of the packing work is predicted.

The progress determination unit 103 transmits the determined progress, the progress, and the predicted work end time to the notification processing unit 104 (server 20) immediately after determination. However, the progress status and estimated work completion time will be sent only when requested.

The notification processing unit 104 notifies the information terminal 105 of the progress received from the progress determination unit 103 . The notification destination is either the portable information terminal 41 held by the worker of the next process or the signal lamp 42 provided at the work place of the next process. If both are provided, both may be notified. The notification timing is stored in the storage unit 112 in advance. For example, if the degree of progress is 0%, the worker of the next process who received the notification knows that the work has started in the previous process. Also, if the degree of progress is 80%, it can be seen that the worker of the next process who received the notification will finish the work in the previous process soon. Furthermore, if the degree of progress is 100%, it can be seen that the worker of the next process who received the notification has finished the work in the previous process and the goods will be coming soon. Of course, the notification timing may be determined arbitrarily.

The portable information terminal 41 or the signal lamp 42, which is the information terminal 105 that has received the notification of the degree of progress, displays a predetermined display. For example, when a notification is given only once at a progress rate of 80%, portable information terminal 41 displays a message to that effect and emits a predetermined sound. In the signal light 42, the green or yellow display may be changed to red until then. The display colors may be set accordingly. Also, the degree of progress may be notified multiple times in one step. In that case, the predetermined degree of progress can be, for example, 0% (start of work), 10%, 50%, 80%, 90%, or the like. Also, for example, notification may be made at a certain degree of progress such as every 5% or 10%. Of course, the predetermined degree of progress that triggers such notification may be arbitrarily determined.

FIG. 9 is a diagram illustrating an example of a screen display of a mobile information terminal that has received a progress notification.

Since the portable information terminal 41 has a display 411, the degree of progress is displayed there, for example, as a bar graph as shown in FIG. Also, at this time, the color of the graph may be changed according to the degree of progress.

In the case of the signal lamp 42, for example, the lighting color is changed according to the degree of progress. 0% progress (start of work) is green, 50% is yellow, 100% is red, and so on. Further, each color may be blinked.

Further, if the progress determination unit 103 obtains the progress status and the estimated work end time, the notification processing unit 104 receives them and notifies the information terminal 105 of them. The timing of the notification may be the same timing as the progress level, or may be notified at the timing of receiving the progress status and the predicted work end time from the progress determination unit 103 . In particular, by notifying the progress status and the predicted work end time from the progress determination unit 103 at the timing, the contents of the progress status and the predicted work end time can be known in real time. As for the progress status and the predicted work completion time, the portable information terminal 41 may display the received progress status and the predicted work completion time as they are in characters, graphs, or the like. On the other hand, for the signal lamp 42, for example, it is preferable to determine in advance the color to be lit and the blinking state (multiple colors may be used at the same time) in correspondence with the progress and the predicted work completion time. For example, in progress indicators, red is later than normal, yellow is faster than normal, green is normal, and so on. Similarly, it is preferable to specify that the predicted work completion time will be later than the standard, earlier than the standard, or that the lighting color is different for each of the standards.

Next, the processing procedure of the process control method using the system of this embodiment will be described. 10 and 11 are flow charts showing the processing procedure of the process control method. Here, the packing process 52 (first process) will be described as an example. Also, in this flowchart, processing procedures of the work recognition unit 102, the progress determination unit 103, and the notification processing unit 104 described above will be described. This flowchart also includes a process of predicting the work end time.

First, the work recognition unit 102 starts acquiring moving image data from the camera 15 that captures the work site of the packing process 52 (S11). From then on, moving image data from the camera 15 is constantly acquired.

Subsequently, the work recognition unit 102 determines whether the worker 32 has entered the work area 201 or the cardboard box 80 has been placed in the work area 202 (S12). Here, when the worker 32 enters the work area 201 or the cardboard box 80 is placed in the work area 202 (S12: YES), the start of packing work for one piece is recognized, and at the same time the individual worker is identified ( S13).

Subsequently, the work recognition unit 102 determines whether or not the wrist joint R1 has entered the tool area 203 from the X and Y axis coordinate values (represented as R1xy in the figure) of the wrist joint R1 from the moving image data (S14). If it is included (S14: YES), the tape picking operation is recognized (S15). Then, it is transmitted to the progress determination unit 103 . If the wrist joint R1 is not within the tool area 203 in S14 (S14: NO), the process of S14 is continued. In this process, it is determined whether or not the wrist joint R1 has entered the tool area 203. This is because, as a work standard, the tool area 203 is arranged on the right side according to the operator's dominant hand (right-handedness). This is because the user puts the adhesive tape 90 down and picks it up with the right hand. Of course, other work standards and the joints determined according to the operator's dominant hand are changed.

The progress determination unit 103 reads the standard work time and the proficiency level of the worker 32 from the storage unit 111, predicts the work end time using the time when the tape picking is recognized as the work start time, and notifies the notification processing unit 104 of the predicted end time. Send to The notification processing unit 104 notifies the information terminal 105 of the predicted work end time (S16).

Subsequently, the work recognition unit 102 determines whether the Y-axis coordinate values (denoted as R1y and L1y in the figure) of both wrist joints R1 and L1 are at approximately the same position within an allowable error range from the moving image data ( S17). If the positions are not substantially the same (S17: NO), the process of S17 is continued.

If the Y-axis coordinate values of both wrist joints R1 and L1 are at approximately the same position within the allowable error range in S17 (S17: YES), the work recognition unit 102 determines whether or not a predetermined time has elapsed ( S18). If the predetermined time has not elapsed (S18: NO), the process returns to S17 and continues.

If the predetermined time has passed (S18: YES), the work recognition unit 102 recognizes the tape sticking work and transmits it to the progress determination unit 103 (S19). The progress determination unit 103 counts the number of recognized taping operations (S20). The progress determination unit 103 determines the progress based on the counted number of tape application operations. Here, the progress is 1% at count number 1, the progress rate is 5% at count number 2, and the progress rate is 10% at count number 3 (S21). The progress determining unit 103 transmits the determined progress to the notification processing unit 104, and if the notification processing unit 104 reaches a predetermined progress, notifies the information terminal 105 of the progress (S22). Assuming that the predetermined degree of progress here is, for example, every 10% of progress, the notification processing unit 104 will notify the user when the degree of progress of 10% is received.

Subsequently, the work recognition unit 102 determines whether or not the wrist joint R1 has entered the tool area 203 from the X- and Y-axis coordinate values of the wrist joint R1 from the moving image data (S23). YES), the tape return operation is recognized (S24). Recognition of the tape return operation at this stage completes the application of the tape to the bottom surface of the cardboard box 80 . If the wrist joint R1 is not in the tool area 203 in S23 (S23: NO), the process returns to S17 and continues.

Subsequently, the work recognition unit 102 determines whether or not the wrist joint R1 has entered the tool area 203 from the X- and Y-axis coordinate values of the wrist joint R1 from the moving image data (S25). YES), the tape picking operation is recognized (S26). After the goods are packed into the corrugated cardboard box 80 by recognizing the taping work, it is recognized that the taping work on the upper surface has started. If the wrist joint R1 is not within the tool area 203 in S25 (S25: NO), the process of S25 is continued.

Subsequently, the work recognition unit 102 determines from the moving image data whether the Y-axis coordinate values of both wrist joints R1 and L1 are at approximately the same position within an allowable error range (S27). If the positions are not substantially the same (S27: NO), the process of S27 is continued.

If the Y-axis coordinate values of both wrist joints R1 and L1 are at approximately the same position within the allowable error range in S27 (S27: YES), the work recognition unit 102 determines whether or not a predetermined time has elapsed ( S28). If the predetermined time has not elapsed (S28: NO), the process returns to S27 and continues.

If the predetermined time has passed (S28: YES), the work recognition unit 102 recognizes the tape sticking work and transmits it to the progress determination unit 103 (S29). The progress determination unit 103 counts the number of recognized taping operations (S30). The progress determination unit 103 determines the progress based on the counted number of tape application operations. Here, the previous count is continued, and the progress is 90% at count number 4, the progress rate is 95% at count number 5, and the progress rate is 100% at count number 6 (S31). The progress determining unit 103 transmits the determined progress to the notification processing unit 104, and the notification processing unit 104 notifies the information terminal 105 of the progress (S32). Since the predetermined degree of progress is set to every 10% of degree of progress as described above, the notification processing unit 104 notifies when the degree of progress of 90% and 100% is received.

Thereafter, the progress determining unit 103 determines whether or not the progress is 100% (S33), and if the progress is not 100% (S33: NO), the process returns to S27 and continues. On the other hand, if the degree of progress is 100% (S33: YES), the process is terminated (all counts are reset upon termination of the process), and the process is started from the start for the next packing task.

The embodiment described above has the following effects.

This embodiment recognizes the work performed by the worker in one process (first process), determines the degree of progress, and if the determined degree of progress reaches a predetermined degree of progress, the other process ( The notification is made to the information terminal 105 that can be viewed by the worker in the second step).

As a result, the worker of the second process can know the progress of the first process in real time. Therefore, for example, the worker of the next process can make arrangements for his/her own process according to the degree of progress of the previous process. Also, for example, if the progress of the previous process is not so advanced and the setup of the own process is completed early, it is possible to go to another process to support. Also, a worker who is assisting in another process can return to his own process in a timely manner so as not to cause a delay in the next process if the progress of the previous process approaches 100%. In particular, by providing each worker with a portable information terminal 41 as the information terminal 105, even a worker who has gone to a place where the signal lamp 42 cannot be seen directly (for example, to support another process) can be in charge of the work. You can see the progress of the previous process of the current process.

In addition, since this embodiment recognizes the work performed by the worker from the moving image data of the camera 15, it is not necessary to attach special devices or pointers to the worker's work clothes, safety helmet, or the like. Therefore, there is no (or less) burden on workers. In addition, since the worker does not have to input the start and end of each task, the burden on the worker just to confirm the contents of the company's business is eliminated (or reduced).

In addition, since the progress of the work is obtained and notified to the worker in the next process, the worker in the next process can know whether the work is progressing quickly or behind in the previous process.

In addition, since the work end time (time) is predicted from the proficiency level of the individual worker, the work end time that varies depending on the worker in the previous process can be known.

Further, in the present embodiment, the computer 16 integrated with the camera 15 executes the work recognition unit 102 for recognizing the work from the moving image data and the progress determination unit 103 for determining the degree of progress from the work content. This eliminates the need to send data with a large amount of data, such as video data, to the server 20, so that the communication load with the server 20 can be reduced. Especially when there are a plurality of processes and the camera units 11 to 13 are arranged for each process, the amount of communication can be reduced.

(Modification 1)
In the above-described embodiment, the functions of the work recognition unit 102, the progress determination unit 103, and the notification processing unit 104 are divided between the computer 16 of the camera units 11 to 13 and the server 20. However, the present invention is not limited to such division of functions. Each function and hardware configuration will be described below as a modified example.

In Modified Example 1, the computer 16 of the camera units 11 to 13 has the function of the notification processing section 104 in addition to the work recognition section 102 and the progress determination section 103 . FIG. 12 is a schematic diagram showing the overall configuration of Modification 1. As shown in FIG. It should be noted that in Modification 1, the functions of the work recognition unit 102, the progress determination unit 103, and the notification processing unit 104 are the same as those of the above-described embodiment, and thus description thereof will be omitted. A functional block diagram is also omitted.

In the work process management system 2 of Modification 1, camera units 11 to 13 in which a computer 16 is integrated with a camera 15 are installed at processes 51 to 53, respectively. Computers 16 of camera units 11 to 13 are connected to personal digital assistants 41, which are information terminals 105, held by workers in steps 51 to 53, and signal lamps 42 in steps 51 to 53, by wire or wirelessly. For this purpose, the computer 16 is equipped with a communication interface.

The computer 16 has all the functions of the work recognition unit 102 , the progress determination unit 103 and the notification processing unit 104 .

This eliminates the need for the server 20 and simplifies the hardware configuration of the entire system in addition to the same effects as in the embodiment.

As the camera-integrated computer 16 having a communication function, for example, a smart phone, a tablet, or the like can be used. Then, the present invention can be achieved by causing a smartphone or tablet to execute a program that implements the functions of each part described above. In addition, it is not necessarily limited to the computer 16 integrated with a camera, but a form in which a camera and a computer (so-called personal computer) are provided in each process, and the computer is connected to a portable information terminal and a signal lamp in other processes by wire or wirelessly. may be

(Modification 2)
In modification 2, the server 20 has all the functions of the work recognition unit 102, the progress determination unit 103, and the notification processing unit 104. FIG. FIG. 13 is a schematic diagram showing the overall configuration of Modification 2. As shown in FIG. Also in the second modification, the functions of the work recognition unit 102, the progress determination unit 103, and the notification processing unit 104 are the same as those of the above-described embodiment, and therefore descriptions thereof will be omitted. A functional block diagram is also omitted.

In the work process management system 3 of Modification 2, a camera 15 is installed at each process 51 to 53 and connected to the server 20 by wire or wirelessly. In addition, the server 20 is connected by wire or wirelessly to the personal digital assistants 41, which are the information terminals 105, possessed by the workers of the processes 51 to 53, and to the signal lamps 42 of the processes 51 to 53. FIG. Moving image data from the camera 15 is directly transmitted to the server 20. - 特許庁

The server 20 has all the functions of the work recognition unit 102 , the progress determination unit 103 and the notification processing unit 104 .

Thereby, in addition to the same effect as the embodiment, for example, it becomes possible to centrally manage a plurality of processes in a plurality of factories by a single server. In particular, since moving image data is also transmitted to the server 20, it is possible to transmit (transfer) the moving image data to, for example, another computer via the server 20 and view the status of each process there.

In addition, as in Modification 2, a cloud server on the Internet can be used as the server 20 for the configuration in which the server 20 is responsible for all functions. When using a cloud server, the cloud server may be composed of, for example, a plurality of servers, and the plurality of servers exchange data with each other, and the work recognition unit 102, the progress determination unit 103, and It is also possible to distribute the functions of the notification processing unit 104 . Distributed processing enables real-time processing with no (or little) delay in processing, even if the number of steps and the amount of data are large. Moreover, since there is no need to attach a computer to the camera 15, a general movie camera, surveillance camera, or simple so-called web camera can be used as they are.

Furthermore, as a modification, the computer 16 integrated with a camera may be assigned the function of the work recognition unit 102 , and the server 20 may be assigned the functions of the progress determination unit 103 and the notification processing unit 104 . In this case, the hardware configuration is the same as the system configuration shown in FIG.

Although the embodiments of the present invention and their modifications have been described above, the present invention can be further modified in various ways.

For example, instead of using one camera 15 for one process, one camera 15 for photographing a plurality of workers in a plurality of processes may be used. Skeleton recognition techniques such as the open pose and deep pose described above can simultaneously recognize the skeletons of a plurality of people appearing in one moving image data. Therefore, even when a plurality of workers are photographed by one camera 15, it is possible to recognize the work being performed individually for each worker and obtain the degree of progress. is not limited to the next process in the flow of goods. For example, the previous process may be notified of the progress of the next process. In other words, the second process is the previous process and the first process is the next process in terms of the product flow. Then, the preceding process (second process) is notified of the progress of the next process (first process). By doing so, when a delay occurs in the next process, such a situation can be immediately known in the previous process, and it is possible to go to the next process for support. For this reason, for example, it is possible to prevent stagnation in the previous process and excessive holding of inventory.

Also, the second process of the notification destination may be not only a process immediately after or immediately before the first process, but may be a process further after or before the first process.

In addition, recognition of the work performed by the worker may be performed using not only the moving image data of the camera 15 but also other sensors or the like. For example, a technology for estimating a person's posture (especially the direction of the arm) can be used by obtaining a distance value to each part of a person using a lidar (Light Detection and Ranging) that obtains a distance value to an object. . In this case, the rider becomes the imaging unit 101 .

Further, when there is a signal light 42, the signal light 42 is not limited to three colors, and the signal light 42 of two colors, the signal light 42 of one color, etc. can be applied. Signal lamps 42 of four colors or five colors may also be used.

Also, instead of or in addition to the signal light 42, a display or another computer having a display may be installed to receive and display progress notifications from the server 20. FIG. In this case, the display or other computer is connected to server 20 via a wired or wireless interface.

Either one of the portable information terminal 41 and the signal lamp 42 provided in each of the steps 51 to 53 may be provided as the information terminal 105, or different information terminals 105 may be provided according to the steps and workers. good.

Moreover, although the embodiment has been described by taking the packing process as an example, the present invention is of course not limited to the packing process. For example, it is suitable for each process of a production line in which deterioration occurs when left in the middle of an assembly line. For example, some chemicals used to manufacture or process the article must be used immediately after being mixed or prepared. Therefore, by applying the present invention, the degree of progress of the steps before and after can be known, so that the chemicals can be mixed and prepared according to the degree of progress.

Furthermore, the conditions and numerical values used in the description of the embodiments are for illustrative purposes only, and the present invention is not limited to these conditions and numerical values.

Various modifications are possible for the present invention based on the configuration described in the claims, and these are also within the scope of the present invention.

11-13 camera unit,
15 camera,
20 servers,
31-33 workers,
41 personal digital assistants,
42 stacked signal lights,
51 inspection process,
52 packing process,
53 palletizing process,
80 cardboard boxes,
90 adhesive tape,
101 imaging unit,
102 work recognition unit,
103 progress determination unit,
104 notification processing unit,
105 information terminal,
200 imaging area,
201 work area;
202 work area,
203 Tool area.

Claims (12)

a photographing unit for photographing an area in which a worker works;
a work recognition unit that acquires moving image data from the imaging unit and recognizes the work performed by the worker in the first step from the movement of the worker in the moving image data;
a progress determination unit that determines the progress of the work in the first step from the recognized work of the worker;
When the degree of progress determined by the degree-of-progress determination unit reaches a predetermined degree of progress, the degree of progress is transferred to an information terminal that transmits information to a worker performing work in a second step different from the first step. a notification processing unit that notifies that has reached the predetermined degree of progress;
has
The work recognition unit acquires the coordinate values of the joints of both wrists of the worker from the moving image data, and the difference between the acquired coordinate values of the joints of the right wrist and the acquired coordinate values of the joints of the left wrist. A work process control system that recognizes specific work content from The work recognition unit includes a first axis corresponding to the left-right direction of the worker and a second axis intersecting the first axis in a frame of moving image data of the worker photographed from above. In the two-dimensional coordinate system, when the coordinate value of one wrist joint on the second axis does not change and the coordinate value of the other wrist joint on the second axis changes, pack 2. The work process management system according to claim 1, which recognizes the work of removing the tape from a predetermined place or the work of returning the tape to a predetermined place . The work recognition unit includes a first axis corresponding to the left-right direction of the worker and a second axis intersecting the first axis in a frame of moving image data of the worker photographed from above. In a two-dimensional coordinate system, when the coordinate values of both wrist joints on the second axis do not change for a certain period of time, or when one wrist joint moves linearly with respect to the other wrist joint 2. The work process management system according to claim 1 , wherein if there is, it is recognized as a task of applying tape when packing . The first step consists of a plurality of tasks, and the standard work time required for each task out of the plurality of tasks or the standard work time required for each task in which some of the plurality of tasks are combined has a storage unit that stores
The progress determination unit determines the time from the start time of the work in the first step to the time when the work recognition unit recognizes the one work or the grouped work among the plurality of works, and the standard comparing the work time with the work time to determine the progress of the first step indicating whether the work recognized by the work recognition unit is ahead of or behind the standard work time;
4. The work process management system according to any one of claims 1 to 3, wherein said notification processing unit notifies said information terminal of the determined progress of said first process. The first step consists of a plurality of tasks, and the standard work time required for each task out of the plurality of tasks or the standard work time required for each task in which some of the plurality of tasks are combined and a storage unit that stores the proficiency level of each worker for a plurality of workers,
The work recognition unit identifies a worker who performs the work of the first step from the moving image data,
The progress determination unit determines the one work or the combined work among the plurality of works by the work recognition unit based on the skill level of the identified worker and the start time of the work in the first step. Predicting the work end time of the first step from the time until the recognized time and the standard work time,
5. The work process management system according to claim 1, wherein said notification processing unit notifies said information terminal of the predicted work finish time of said first process. Any one of claims 1 to 5, wherein the information terminal is a signal lamp or display installed in the workplace of the second process, or a portable information terminal carried by the worker of the second process. 1. The work process control system according to 1. Step (a) of acquiring video data of an area in which the worker is working;
a step (b) of recognizing the work performed by the worker in the first step from the motion of the worker in the moving image data;
Step (c) of determining the progress of the work in the first step from the recognized work of the worker;
When the determined degree of progress reaches a predetermined degree of progress, the degree of progress is transferred to an information terminal that transmits information to a worker performing work in a second step different from the first step. step (d) of notifying that the
has
The step (b) acquires the coordinate values of the joints of both wrists of the worker from the moving image data, and compares the acquired coordinate values of the right wrist joint and the acquired coordinate values of the left wrist joint. A work process management method that recognizes specific work content from differences . The step (b) consists of a first axis corresponding to the horizontal direction of the worker and a second axis intersecting the first axis in a frame of moving image data photographing the worker from above. In the two-dimensional coordinate system, when the coordinate value of one wrist joint on the second axis does not change and the coordinate value of the other wrist joint on the second axis changes, 8. The work process management method according to claim 7, wherein the work of removing the tape from a predetermined place or the work of returning the tape to a predetermined place is recognized . The step (b) consists of a first axis corresponding to the horizontal direction of the worker and a second axis intersecting the first axis in a frame of moving image data photographing the worker from above. In the two-dimensional coordinate system, if the coordinate values of both wrist joints on the second axis do not change for a certain period of time, or if one wrist joint moves linearly with respect to the other wrist joint 8. The work process management method according to claim 7, wherein, if there is, it is recognized as a task of attaching a tape when packing . The first step consists of a plurality of tasks, and the standard work time required for each task out of the plurality of tasks or the standard work time required for each task in which some of the plurality of tasks are combined is stored in advance,
comparing the standard work time with the time from the start time of the work in the first step to the time of recognition of the one work out of the plurality of works or the grouped work in the step (b); a step (e) of determining the progress of the first step indicating whether the work recognized in the step (b) is ahead of or behind the standard work time;
10. The work process management method according to any one of claims 7 to 9, wherein in said step (d), the information terminal is notified of the determined progress of the work in said first process. The first step consists of a plurality of tasks, and the standard work time required for each task out of the plurality of tasks or the standard work time required for each task in which some of the plurality of tasks are combined , and the proficiency level of each worker is stored in advance for a plurality of workers,
a step (f) of identifying a worker who is working in the first step from the moving image data;
the proficiency level of the identified worker, the time from the start time of the work in the first step to the time when the one work out of the plurality of work or the grouped work is recognized in the step (b) , and a step (g) of predicting the work finish time of the first process from the standard work time,
11. The work process management method according to any one of claims 7 to 10, wherein in said step (d), said information terminal is notified of the predicted work finish time of said first process. A program for causing a computer to execute the work process control method according to any one of claims 7 to 11.

Images