WO2019087638A1

WO2019087638A1 - Information processing device and information processing method

Info

Publication number: WO2019087638A1
Application number: PCT/JP2018/036361
Authority: WO
Inventors: 悠美子内田
Original assignee: キヤノン株式会社
Priority date: 2017-11-01
Filing date: 2018-09-28
Publication date: 2019-05-09
Also published as: JP2019086827A

Abstract

In the present invention, in the case where a detection process is performed for detecting a defective operation from a photographed image of an operator sequentially executing a series of operations included in an operation sequence, when the defective operation is detected from the photographed image in the detection process, the detection process is suspended until an instruction from the operator is detected. When the instruction is detected after the suspension, whether an operation to be detected after resumption of the detection process is set to an operation after the series of operations which have been executed, or to an operation related to the operation which has been detected as defective is specified on the basis of the operation detected as defective, and the detection process is resumed for said operation to be detected.

Description

Information processing apparatus, information processing method

The present invention relates to a technique for performing execution determination of work in a process using a captured image.

Manual assembly of products in a factory consists of a large number of processes and operations, which may result in defects such as omissions and errors in operations. Since a defect in work leads to a defect in the product, if there is a defect in work, it is necessary to re-execute the defective work. However, it is difficult to determine whether or not there is a defect in work just by checking the product after assembly. For this reason, there is a need for a system for determining work deficiencies while performing work and prompting workers to re-execute.

Patent Document 1 discloses a technique of photographing the site of an assembly operation, analyzing the movement of a worker from a photographed image, and giving a warning when a defect in operation is detected. As a method for detecting a defect in work, the movement of the worker in normal time is defined in advance, the defined movement is compared with the movement acquired from the photographed video, and the defect in work is detected when the difference is large. The method is used. As described above, in a system in which a worker is notified of a warning during work execution, it is assumed that a worker receives a warning and re-executes a work that has been defective. However, when re-executing the work, the worker moves differently from the normal time, so if the same video analysis as in the normal work is continued, there is a problem that the work is erroneously detected as a work defect. . In the point which concerns, patent document 1 is not disclosed about the control method of the operation | movement detection process after warning generation | occurrence | production, and is not solved about this subject.

Patent Document 2 discloses a system that continuously monitors outdoor and indoor activities of a monitoring target person such as an elderly person by cooperation of a robot and a wristwatch. When the person to be monitored is indoors, behavior monitoring is performed using the video taken by the robot, and when the person to be monitored goes out, the monitoring by the robot is interrupted and the behavior is measured by the wristwatch. When the person to be monitored returns home, the robot analyzes the data measured by the wristwatch, and if there is something wrong with the behavior when going out, the monitoring by the robot is resumed after reporting to the security company.

Patent Document 3 discloses a technique of photographing an area for managing important assets and confidential information, detecting a difference from a normal operation state by analyzing a monitoring video, and performing an alarm. Furthermore, at the time of work by the administrator, in order to prevent generation of an alarm, there is provided a function of canceling the monitoring state based on the contents of video analysis. Specifically, the proper work image set in advance and the photographed visitor work image are compared, and if they match, the monitoring state is canceled and control is performed so as not to generate an alarm until the work is completed. It is a thing.

Japanese Patent Application Publication No. 2003-167613 JP 2012-38339 A JP, 2016-149108, A

The techniques disclosed in Patent Document 2 and Patent Document 3 relate to techniques for temporarily interrupting video surveillance and resuming it later, but depending on the status at the time of surveillance suspension and the captured video during suspension, surveillance resumes It does not control later monitoring processing.

The present invention has been made in view of such a problem, and provides a technique for correctly performing the work determination even in the case where the worker re-executes the defective work and then resumes the normal work. Do.

One aspect of the present invention includes processing means for executing detection processing for detecting a defect in work from a photographed image of a worker sequentially performing a series of work included in a work sequence, the processing means including the detection When the processing detects a defect in work from the photographed video, the detection processing is interrupted until an instruction from the operator is detected, and when the instruction is detected after the interruption, the target of the detection processing is resumed after the detection processing is resumed. Target work from the continuation of already performed work to the target work or work from the work related to the work for which the deficiency is detected, or specified based on the work for which the deficiency is detected, The detection process for the target work is resumed.

According to the configuration of the present invention, even in the case where the worker re-executes the defective work and then resumes the normal work, it is possible to correctly determine the execution of the work.

Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings. In the attached drawings, the same or similar configurations are denoted by the same reference numerals.

The accompanying drawings are included in the specification, constitute a part thereof, show embodiments of the present invention, and are used together with the description to explain the principle of the present invention.
FIG. 2 is a block diagram showing a configuration example of a computer device. The block diagram which shows the structural example of a work monitoring system. The figure explaining the operation example of the work monitoring system. The figure explaining the operation example of the work monitoring system. The figure explaining the operation example of the work monitoring system. The figure explaining the operation example of the work monitoring system. FIG. The figure which shows the structural example of a table. The figure which shows the structural example of a table. The figure which shows the structural example of a table. The figure which shows the structural example of a table. The figure which shows the structural example of a table. 6 is a flowchart of work execution determination processing performed by the information processing apparatus 200. 5 is a flowchart showing details of processing in step S402. 5 is a flowchart showing details of processing in step S402. The figure explaining step S510. The flowchart which shows the details of the processing in step S403. The flowchart which shows the details of the processing in step S404. The figure explaining the operation example of the work monitoring system. The flowchart which shows the details of the processing in step S403. The flowchart which shows the details of the processing in step S404. The figure explaining 3rd Embodiment. The figure explaining 3rd Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configuration described in the claims.

First Embodiment
First, a configuration example of a task monitoring system according to the present embodiment will be described using the block diagram of FIG. 1B. As shown in FIG. 1B, the task monitoring system according to the present embodiment includes an imaging device 100 for imaging a worker who is sequentially performing a series of tasks, and a photographed image (moving image) of the worker by the imaging device 100. And the information processing apparatus 200 that performs the analysis of The information processing apparatus 200 executes a detection process of detecting a defect in work from a photographed image of a worker who is sequentially performing a series of work included in the work sequence. At this time, when the information processing apparatus 200 detects a defect in work from the captured image by the detection process, the information processing apparatus 200 interrupts the detection process until an instruction from the operator is detected. Then, when the information processing apparatus 200 detects the instruction after the interruption, the information processing apparatus 200 specifies a target work to be a target of the detection process after resuming the detection process based on the work for which the deficiency is detected, and performs the detection process for the target work. Resume The following description of the operation of the information processing apparatus 200 is merely an example of such an operation.

First, the imaging device 100 will be described. The image pickup apparatus 100 picks up an operator who is sequentially executing a series of tasks, and the images (photographed images) of each frame of the photographed image by the image pickup apparatus 100 are sequentially sent out to the information processing apparatus 200. Be done. The imaging apparatus 100 may perform imaging so as to include the whole of the worker, or imaging so as to include the work site (for example, the entire arm) of the worker (not necessarily including other than the work site) You may do it.

Next, the information processing apparatus 200 will be described. The acquisition unit 108 acquires a captured image (captured image 109) of each frame sent from the imaging device 100 and stores (buffers) the image in the memory 198.

The work determination unit 110 sequentially refers to the captured image of each frame stored in the memory 198 and determines whether the worker has performed each work constituting the work process assigned to the worker in a prescribed order. The determination is made based on the standard work sequence 111 or the temporary work sequence 112. The standard work sequence 111 and the temporary work sequence 112 will be described later.

The detection unit 113 detects a gesture performed by the operator to switch the operation mode of the information processing apparatus 200 by sequentially referring to the captured image of each frame stored in the memory 198. There are three operation modes of the information processing apparatus 200: “normal mode”, “warning mode”, and “standby mode”. The mode control unit 114 controls switching of the operation mode of the information processing device 200 according to the detection result of the gesture by the detection unit 113.

The information processing apparatus 200 in the normal mode performs a detection process of detecting a defect in work by the worker from a video captured by the imaging device 100 by the worker. Here, “detection processing for detecting a defect in work” refers to work execution determination processing including detection of a missing work and detection of a work that does not satisfy “conditions that a correct work should meet”. is there. When the information processing apparatus 200 detects a defect in work, the information processing apparatus 200 shifts the operation mode of the information processing apparatus 200 from the normal mode to the warning mode.

The information processing apparatus 200 in the warning mode interrupts the process of detecting a defect in work, notifies the worker of information related to the work in which the defect is detected, and urges re-execution of the work in which the defect is detected. When an operator receives notification of information related to a task for which a defect has been detected from the information processing apparatus 200, the operator can shift the operation mode of the information processing apparatus 200 from the warning mode to the standby mode to re-execute the task Make a gesture of 1. When the information processing apparatus 200 in the warning mode detects a first gesture by the operator from a video shot by the imaging apparatus 100, the information processing apparatus 200 interrupts notification of information related to the work for which a defect is detected, and the operation mode of the information processing apparatus 200 Move from warning mode to standby mode.

The information processing apparatus 200 in the standby mode performs a detection process of detecting a second gesture by the worker from a video captured by the imaging apparatus 100. When the operation mode of the information processing apparatus 200 shifts to the standby mode, the worker re-executes the work for which the deficiency is detected. In the standby mode, since the information processing apparatus 200 is in the standby state without performing the process of detecting the incomplete work, even if the worker re-executes the work, the work is not detected as the incomplete work. . Then, when the re-execution of the work is completed, the worker performs a second gesture to shift the operation mode of the information processing device 200 from the standby mode to the normal mode. When the information processing apparatus 200 in the standby mode detects the second gesture from the video shot by the imaging apparatus 100, the operator notifies the worker of the work to be performed next, and the operation mode of the information processing apparatus 200 is set. Shift from standby mode to normal mode.

The notification control unit 115 controls the various types of notification described above. The generation unit 116 generates the temporary work sequence 112 described above. The control unit 199 controls the operation of the entire information processing apparatus 200. The memory 198 stores various data for realizing various processes by the information processing apparatus 200.

In FIG. 1B, the numbers of the imaging devices 100 and the information processing devices 200 are “1”, but the numbers of the imaging devices 100 and the information processing devices 200 are not limited to “1”. Although the imaging device 100 and the information processing device 200 are separate devices in FIG. 1B, the present invention is not limited to this, and the imaging device 100 and the information processing device 200 may be integrated into one device.

Next, an operation example of the work monitoring system according to the present embodiment will be described using FIGS. 2A to 2D. FIG. 2A is a bird's-eye view showing how a worker 201 is performing assembly work (work steps assigned to the worker 201) on the work object 204 on the work bench 203. FIG. Further, an imaging device 100 is provided above the work table 203 in order to capture an image including the worker 201 (a part or all) and the work object 204.

In the normal mode, the work determination unit 110 performs a detection process of detecting a defect in the work performed by the worker 201 from the image captured by the worker 201 by the imaging device 100. Then, when the work determination unit 110 detects an incomplete work, the mode control unit 114 shifts the operation mode of the information processing device 200 from the normal mode to the warning mode. In the warning mode, the operation determination unit 110 interrupts the process of detecting the incompleteness of the operation, and the notification control unit 115 notifies the worker of information related to the operation for which the incompleteness is detected, and urges reimplementation of the incomplete operation. . In FIG. 2A, since the defect of the operation of work ID = 3 was detected among the operations constituting the work of assembling the work object 204, "There was a defect in the work ID3 screw tightening work. Please redo the work." Is output from the speaker to prompt re-execution of the work. In this way, by notifying the worker 201 of the incomplete work, the worker 201 can grasp the incomplete work, that is, the work to be performed again.

Here, a large number of operation steps exist in the assembly operation such as assembling a product in a factory, and each operation step is composed of a plurality of operations (operation groups). Furthermore, for each work that constitutes one work process, the order of execution is defined. Each worker is in charge of one work process, and it is assumed that the work of the same work process is repeatedly performed. The work monitoring system according to the present embodiment is for performing, for a specific work process determined in advance, execution determination of each work that constitutes the work process.

Since it is necessary to consider the order of work in order to judge the quality of the assembly work performed by the worker, it is necessary to analyze the order of those occurrences. In order to analyze the order of occurrence, it is generally used to collate a sequence of data related to the movement of the worker extracted from the video. In order to collate a sequence, a length of a fixed length is required, and collation is performed on a group of a plurality of tasks.

In this embodiment, this "group of a plurality of tasks" is defined as a "phase". FIG. 2B shows a configuration example of one work process assigned to the worker 201. In FIG. 2B, this one work process can be performed in this order of nine work of work 1, work 2, work 3, work 4, work 5, work 6, work 7, work 8, work 9, and so on. It is a prescribed work process. The tasks 1 to 4 constitute a phase 1, the tasks 5 to 6 constitute a phase 2, and the tasks 7 to 9 constitute a phase 3.

The information processing device 200 detects the occurrence of a prescribed event in a captured image by the imaging device 100 as a phase break (end). The occurrence of a prescribed event is, for example, the occurrence of a prescribed change in the position and / or posture of the work object 204. For example, it is assumed that it is known that the work object 204 is reversed between the work 4 and the work 5 in the work process of FIG. 2B and the work object 204 is rotated between the work 6 and the work 7 . At this time, when the information processing apparatus 200 detects that the work object 204 is turned over from the captured video, it determines that there is a break in phase 1 and that the work object 204 is rotated from the captured video. If it is detected, it is determined that there is a phase 2 break. Then, each time the information processing apparatus 200 detects a division of a phase, the information processing apparatus 200 performs a process of detecting a defect in work in the phase (determination of execution of the work in the phase). In the example of FIG. 2B, when the information processing apparatus 200 detects that the work object 204 is turned over from the captured video, it determines that there is a break in phase 1, and works 1 to 4 constituting phase 1 It is judged whether or not it was carried out normally in this order.

Here, an example of switching of the operation mode in the information processing apparatus 200 will be described with reference to FIG. 2C. When the information processing apparatus 200 in the normal mode detects a phase 1 break from the captured image, it is determined whether the operations 1 to 4 constituting the phase 1 are normally performed in this order. Here, it is assumed that the information processing apparatus 200 determines that there is a defect in the operation 3 among the operations 1 to 4. At this time, the information processing apparatus 200 switches the operation mode of the information processing apparatus 200 from the normal mode to the warning mode, interrupts execution determination processing of the work, and notifies that there is a defect in the work 3. Encourage re-implementation. When receiving the notification, the worker performs a first gesture to shift the operation mode of the information processing apparatus 200 from the warning mode to the standby mode in order to re-perform the work 3. When the information processing apparatus 200 recognizes the first gesture from the captured video, the notification that there is a defect in the work 3 is interrupted, and the operation mode of the information processing apparatus 200 is shifted from the warning mode to the standby mode. Then, the worker re-performs work 3, and when re-execution of work 3 is completed, the operator performs a second gesture for shifting the operation mode of the information processing device 200 from the standby mode to the normal mode. When the information processing apparatus 200 recognizes the second gesture from the captured image, the operator is notified of the first operation “operation 5” of phase 2 which is the phase following phase 1 as the operation to be performed next by the operator. . Then, the information processing device 200 shifts the operation mode of the information processing device 200 from the standby mode to the normal mode. Then, the worker sequentially performs the work after the work 5.

Here, the example of FIG. 2C is based on the premise that the related work associated with the work for which the defect is detected is not present in the same work process. An example of switching the operation mode of the information processing apparatus 200 in the case where another work (related work) related to the work having a defect exists in the same work process will be described using FIG. 2D. Here, the related work related to the work A is a work that must be performed after the work A is performed, and is determined in advance by characteristics of the contents of the work. For example, in the case where there is a rule that the cleaning operation must be re-executed after the cutting operation is re-performed, the “cleaning operation” is a related operation of the “cutting operation”. In FIG. 2D, the related work of the work 2 is the work 4.

When the information processing apparatus 200 in the normal mode detects a phase 1 break from the captured image, it is determined whether the operations 1 to 4 constituting the phase 1 are normally performed in this order. Here, it is assumed that the information processing apparatus 200 determines that there is a defect in the operation 2 among the operations 1 to 4. At this time, the information processing apparatus 200 switches the operation mode of the information processing apparatus 200 from the normal mode to the warning mode, interrupts execution determination processing of the work, and notifies that there is a defect in the work 2. Encourage re-implementation. When receiving the notification, the worker performs a first gesture to shift the operation mode of the information processing apparatus 200 from the warning mode to the standby mode in order to re-perform the work 2. When the information processing apparatus 200 recognizes the first gesture from the captured image, the notification that there is a defect in the work 2 is interrupted, and the operation mode of the information processing apparatus 200 is shifted from the warning mode to the standby mode. Then, the worker re-performs task 2, and when re-execution of task 2 is completed, the operator performs a second gesture to shift the operation mode of the information processing apparatus 200 from the standby mode to the normal mode.

Here, the work 4 is finished in the previous normal mode, but is a work that must be performed again after the work 2 is performed again. Therefore, when the information processing apparatus 200 recognizes the second gesture from the photographed image, the information processing apparatus 200 notifies the worker of the task 4 which is a related task of the task 2 as a task to be performed next. Then, the information processing device 200 shifts the operation mode of the information processing device 200 from the standby mode to the normal mode. Then, the worker sequentially performs the work after the work 4.

Next, the process of determining whether an operation by a worker is performed by the information processing apparatus 200 will be described with reference to FIG. 4 showing a flowchart of the process. It is assumed that the operation mode of the information processing apparatus 200 is set to the normal mode at the start of the process according to the flowchart of FIG. 4.

In step S401, the control unit 199 sets information specifying the work process to be performed by the operator. For example, the control unit 199 sets the work process input by the operator or the manager of the factory operating the operation unit (not shown). The subsequent processing is performed for this set processing step.

In step S402, the information processing device 200 performs the normal mode operation. Details of the process in step S402 will be described using the flowcharts of FIGS. 5A and 5B. In step S501, the work determination unit 110 reads captured images in the order of storage from the memory 198. Further, the control unit 199 initializes a variable F representing an ID of a phase to be currently processed to one.

In step S502, the work determination unit 110 determines whether or not the phase of phase ID = F has been started (switched to the phase of phase ID = F) in the captured image read from the memory 198 in step S501. to decide.

Various processes can be applied to the process for determining whether or not a phase has been started in the captured image (a phase has been switched). For example, the presence or absence of a change in the position or posture of the work object in the captured image is determined by template matching of the captured image and the template image for phase switching determination defined in advance, and the result is determined It can be determined whether the phase has started. In template matching, the template image of the work object is moved on the captured image, and the similarity between the template image and a partial image overlapping the template image in the captured image is obtained. The similarity can be obtained, for example, as a sum of absolute values of differences between pixel values of corresponding pixel positions in the template image and the partial image, but the method of obtaining the similarity is not limited to a specific method. Then, the position of the template image for which the maximum similarity degree is obtained on the captured image can be obtained as the position of the work target. It should be noted that images of work objects of various postures can be used as template images, and the posture corresponding to the template image having the highest degree of similarity with the captured image can be obtained as the posture of the work target in the captured image. . In addition, it is possible to determine that there is no work target in the captured image when a threshold is provided for the similarity and a similarity equal to or higher than the threshold can not be obtained.

In this way, since the position and orientation of the work object can be estimated from the captured image, the work object can be determined by calculating the difference between the position and orientation of the work object determined for the captured image of the previous frame. It is possible to obtain changes in the position and posture of For example, if the distance between the position of the work target determined for the captured image of the previous frame and the position of the work target determined for the captured image of the current frame is equal to or greater than the threshold, the work target is between these two frames. It can be determined that it is moving. On the other hand, if the distance is less than the threshold, it can be determined that the work object is stationary between the two frames. The phase is started when the change in the position or posture of the work object obtained in this way satisfies the start condition (the condition for switching from the current phase to the next phase) of the phase (here, phase 1). It can be determined that (the current phase has been switched to the next phase). The start conditions include, for example, moving the work object in a prescribed direction, turning over the work object, causing the work object to appear, and causing it to disappear.

Note that the method for determining whether or not a prescribed phase has started in the captured image (whether or not the current phase has switched to the next phase) is not limited to a specific method, for example, a method such as machine learning It is available.

If it is determined that the phase of phase ID = F is started in the captured image read from the memory 198 in step S501, the process proceeds to step S503. On the other hand, when it is determined that the phase of the phase ID = F is not started in the captured image read from the memory 198 in step S501, the process returns to step S501.

In step S503, the work determination unit 110 acquires the standard work sequence 111 as the comparison sequence. However, the work determination unit 110 creates the temporary work sequence 112 in the standby mode, and thereafter acquires the temporary work sequence 112 as the collation sequence when the process proceeds from step S806 to step S503.

In step S504, the work determination unit 110 reads captured images in the order of storage from the memory 198. In step S505, the work determination unit 110 extracts the area (hand area) of the worker's hand from the captured image read from the memory 198 in step S504. For example, the work determination unit 110 extracts the hand area from the captured image by the background difference method. In the background subtraction method, a moving body (human body) is extracted from a difference image between a target captured image and a background image, and a hand region is acquired from the extracted human body excluding a head, a moving body, an arm and the like. Note that the method of extracting (acquiring) the hand region is not limited to a specific method, and for example, a method of using machine learning, a method of extracting a color region of the skin and detecting the hand region, skin using infrared rays There is also a method of detecting a hand area by extracting a temperature area. A technique for extracting a hand region from a captured image is a known technique, and any technique may be used in this step to extract the hand region from the captured image.

In step S506, the work determination unit 110 determines in which of the work areas on the captured image the hand area extracted in step S505 is included. As shown in FIG. 3A, real areas 301 to 304 are set on the work bench 203, and in the information processing apparatus 200, the hands of the operator 201 are real areas X (any one of the real areas 301 to 304). If it is located at, it is determined that work X has been performed. The numbers in the real areas 301 to 304 are the IDs (area IDs) of the real areas, respectively. An area (working area) on an image captured by the imaging apparatus 100 corresponding to each of the real areas 301 to 304 is uniquely defined based on parameters such as the position and orientation of the imaging apparatus 100 and the angle of view. The information defining the work area is, as shown in FIG. 3B, a memory 198 as a table in which an area ID 305 of the real area corresponding to the work area is associated with image coordinates (coordinate values 306) of four corners of the work area. Are registered in advance. However, the work determination unit 110 determines which of the respective work areas defined by the coordinate values 306 on the captured image is included in the hand area on the captured image.

Then, the work determination unit 110 registers the area ID corresponding to the work area including the hand area in the detection area sequence 315 illustrated on the left side of FIG. 3F. As shown on the left side of FIG. 3F, in the detection area sequence 315, area IDs are arranged in the order of registration. In FIG. 3F, in the detection area sequence 315, the area IDs are arranged in the order of 1, 1, 2, 2,. This means that the hand area passes in the order of work area with area ID = 1 → work area with area ID = 1 → work area with area ID = 2 → work area with area ID = 2, ... There is.

Further, as shown in FIG. 3C, the memory 198 is required to pass the operator's hand region because it is determined that the work corresponding to the work ID 307 is performed for each work ID 307 which is an ID unique to the work. The area ID 305 of the work area, the work ID (related work ID) 308 of the related work of the work corresponding to the work ID 307, and the name of the work of the work ID 307 are registered in association with each other. In the table of FIG. 3C, “1” is registered as the area ID of the work area to which the hand area of the worker should pass in order to determine that the work of the work ID = 1 is performed. That is, when the hand area passes through the work area corresponding to the real area 301, it is determined that the work corresponding to the work ID = 1 is performed. Further, since there is no related work of the work of the work ID = 1, information “-” indicating that is registered in this table. In addition, “cleaning” is registered in the table as the name of the work of work ID = 1. Further, in the table of FIG. 3C, “3” is registered as the area ID of the work area to which the worker's hand area should pass in order to determine that the work of work ID = 3 is performed. That is, when the hand area passes through the work area corresponding to the real area 303, it is determined that the work corresponding to the work ID = 3 has been performed. Further, in this table, since the related work of the work with work ID = 3 is the work of work ID = 4, “4” is registered as the work related ID. Also, in this table, “screw tightening” is registered as the name of the work of work ID = 3.

In step S507, the work determination unit 110 performs the same process as step S502 described above on the captured image read from the memory 198 in step S504 to determine whether or not the phase has been switched in the captured image. . If the phase is switched in the captured image, the process proceeds to step S509. If the phase is not switched in the captured image, the process returns to step S504.

In step S509, the work determination unit 110 collates the comparison sequence with the detection area sequence 315, and determines whether there is any defect in the “work area through which the hand area has passed” represented by the detection area sequence 315.

An exemplary configuration of the standard work sequence 111 is shown in FIG. 3D. The standard work sequence 111 is a sequence in which the works constituting the phase corresponding to the phase ID 310 are arranged in the order of execution for each phase ID 310, and the area of the work area corresponding to each work in the work sequence 311. It is a table which matches and manages the area sequence 312 showing the sequence of ID305. For example, as the work sequence 311 of the phase of phase ID 310 = 1 (phase 1), a sequence “1 → 2 → 3 → 4” in which the work IDs of the tasks 1 to 4 constituting phase 1 are arranged in order of execution is registered. It is done. Also, as the area sequence 312 of the phase of phase ID 310 = 1 (phase 1), it is the area ID of the work area to which the hand area of the worker should pass in order to be judged that the work of work ID = 1 is performed. It is determined that the work of the work ID = 3, which is the area ID of the work area to which the worker's hand area should pass in order to be judged that the work of “1” and the work ID = 2 is performed The work area where the worker's hand area should pass in order to be judged that the work with the work ID of "3" and the work ID = 4 is performed. A sequence in which the area ID "4" is arranged in this order is registered. The same applies to phases other than 1 in the phase ID. Such standard work sequence 111 is prepared in advance and registered in the memory 198.

An exemplary configuration of the temporary work sequence 112 is shown in FIG. 3E. The temporary work sequence 112 corresponds to a work sequence 313 in which the work determined by the information processing apparatus 200 as the work to be performed in the phase corresponding to the phase ID 310 is arranged in order of execution and each work in the work sequence 313. It is a table which associates and manages an area sequence 314 representing a sequence of the area ID 305 of the work area. The temporary work sequence 112 is created in the standby mode and deleted after use. The tables shown in FIGS. 3B to 3D are prepared in advance for each work process and registered in the memory 198.

In step S 509, therefore, the area sequence 312 (314) corresponding to the value of phase ID = variable F in the comparison sequence is compared with the detection area sequence 315. This matching can use sequence matching called DP matching. The DP matching is a method of measuring the similarity of two data strings (a phase ID = area sequence 312 (314) corresponding to the value of variable F in the matching sequence, detection area sequence 315). Such DP matching makes it possible to calculate the degree of similarity while taking account of partial expansion and contraction and noise in the sequence. The matching method is not limited to DP matching, and another matching method may be used. Also, the detection area sequence 315 may be deleted after completion of step S509.

In step S510, the work determination unit 110 determines, based on the collation result in step S509, whether or not there is a defect in performing the work that constitutes the phase corresponding to the value of phase ID = variable F. For example, as shown in FIG. 6, it is assumed that a DP matching result between the area sequence “1 2 3 4” and the detection area sequence “1 1 2 2 4 1 4” is obtained. In this case, it can be seen that there is nothing in the detection area sequence that corresponds to "3" in the area sequence, and as a result, it is clear that there is a defect in the work corresponding to area ID = 3. In addition, various processes can be considered as a process for determining whether there is a defect in the implementation of the work.

As a result of the determination in step S510, if there is a defect in the implementation of the work constituting the phase corresponding to the value of phase ID = variable F, the process proceeds to step S511. On the other hand, if there is no defect in the implementation of the work constituting the phase corresponding to the value of phase ID = variable F, the process proceeds to step S513.

In step S513, the work determination unit 110 determines whether the value of the variable F = (total number of phases in the work process), that is, whether or not work execution determination has been performed for all phases in the work process. . In the case of FIG. 2B, it is determined whether or not the value of variable F = 3. If the result of this determination is that the value of variable F = (total number of phases in the work process), the process returns to step S501. On the other hand, if the value of variable F <(the total number of phases in the work process), the process proceeds to step S514. If the value of the variable F = (the total number of phases in the work process) in step S513, the value of the variable F may be initialized to an appropriate value. In step S514, the control unit 199 increments the value of the variable F by one. Then, the process proceeds to step S503.

In step S511, the work determination unit 110 specifies a work sequence 311 (313) corresponding to the value of phase ID = variable F from the table of FIG. 3D (the table of FIG. 3E). Then, the work determination unit 110 indicates, for each work ID included in the specified work sequence 311 (313), information indicating whether or not a defect in the work corresponding to the work ID is detected (execution determination result) Create a table where is registered. The area ID corresponding to the work ID can be uniquely identified because each work ID in the work sequence 311 (313) corresponds to each area ID in the area sequence 312 (314) on a one-to-one basis. However, if the area ID corresponding to the work ID is an area ID determined not to be in the collation sequence in the collation processing in step S510, information indicating that a defect in the work corresponding to the work ID is detected (implementation Register the judgment result). On the other hand, if the area ID corresponding to the work ID is an area ID determined to be in the collation sequence in the collation processing in step S510, information indicating that the defect of the work corresponding to the work ID is not detected ( Register the implementation judgment result).

For example, it is assumed that the standard work sequence 111 is used as the matching sequence, and the value of the variable F = 1. At this time, as shown on the right side of FIG. 3F, the work determination unit 110 determines the execution determination result of the work ID 307 = 1, 2, 3, 4 included in the standard work sequence 111 corresponding to phase ID = 1 in the table of FIG. Create a table in which 316 is registered. If the execution determination result 316 is "OK", it indicates that no defect was detected in the corresponding operation. On the other hand, when the implementation determination result 316 is "NG", it indicates that a defect is detected in the corresponding operation. The configuration for acquiring the corresponding implementation determination result for each work ID is not limited to a specific configuration.

In step S512, the mode control unit 114 shifts the operation mode of the information processing device 200 from the normal mode to the warning mode. Then, the process proceeds to step S403. That is, the transition from step S402 to step S403 is performed when there is a defect in the implementation of the work that constitutes the phase corresponding to the value of phase ID = variable F.

In the present embodiment, one work area is associated with one work, but a plurality of work areas may be associated with one work. In this case, it is possible to determine whether the work has been correctly performed using the ratio of the work area determined to include the hand area among the work areas corresponding to the respective works. Further, the execution determination of the work is not limited to the one using the work area, and for example, the lack of work may be detected by comparing the feature quantity of the hand movement with the feature quantity of the correct hand movement. That is, as long as it can be determined whether or not each work is defective, the configuration therefor is not limited to a specific configuration.

Returning to FIG. 4, in step S403, the information processing device 200 operates in the warning mode. Details of the process in step S403 will be described using the flowchart of FIG.

In step S701, the notification control unit 115 specifies the work ID whose execution determination result 316 corresponds to "NG" in the table created in step S511 above, and detects a defect in the work corresponding to the specified work ID. Report that it was done. In the case where notification of a defect in work is performed by voice output, it is assumed that voice data has been recorded in advance for all work, and the notification can be performed by reproducing voice data corresponding to the work ID. This notification is continued while the information processing apparatus 200 operates in the warning mode. In addition, the notification method of the defect of the work to the worker in the present embodiment is not limited to the voice notification, and may be a lighting pattern of a lamp connected to the information processing apparatus 200 or a display notification on a display device such as a liquid crystal monitor. Good.

In step S702, when the detection unit 113 reads the captured images in the order of storage from the memory 198 and recognizes the above-described first gesture by the worker from the read captured image, the process proceeds to step S704 via step S703. Go to On the other hand, when the above-described first gesture by the operator is not recognized from the captured image read out in step S702, the process proceeds to step S702 via step S703.

Known techniques can be used to recognize gestures. For example, it is possible to detect a hand area or an arm area from a captured image, and to determine whether or not a predetermined gesture has been performed based on a time-series change in movement. By calculating feature quantities for each image and accumulating information of a plurality of images in time series, it is possible to obtain time-sequential motion features. Also, the gesture is not limited to recognition from the image, and for example, the motion of a part such as a hand is measured using a sensor, and if the measured motion matches the motion of a predetermined gesture, the predetermined gesture is You may judge that it was done.

In step S704, the work determination unit 110 suspends (stops) the processing for detecting a defect in work from the captured image (execution determination process of the work), and in step S705, the mode control unit 114 operates the information processing apparatus 200. Move the mode from warning mode to standby mode. Then, the process proceeds to step S404. That is, the transition from step S403 to step S404 is performed when the first gesture is recognized from the captured image in the warning mode.

Returning to FIG. 4, in step S404, the information processing apparatus 200 operates in the standby mode. Details of the process in step S404 will be described using the flowchart of FIG.

In step S801, when the detection unit 113 reads the captured images in the order of storage from the memory 198 and recognizes the second gesture by the worker from the read captured image, the process proceeds to step S803 via step S802. Go to On the other hand, if the above-described second gesture by the operator is not recognized from the captured image read out in step S801, the process proceeds to step S801 via step S802. The recognition of the gesture is performed using a known technique as described above. The first gesture and the second gesture may be the same gesture or may be different gestures. The first gesture and the second gesture may be performed by the same person or different persons.

In step S 803, the work determination unit 110 specifies the work ID whose execution determination result 316 corresponds to “NG” in the table created in step S 511 above, and the related work ID corresponding to the work ID is FIG. 3C. It is determined whether it is registered in the table. In the case of FIG. 3C, if the operation determination result 316 is the operation ID = 1 corresponding to “NG”, the related operation ID is not registered, and the operation determination result 316 is the operation ID = 3 corresponding to “NG”. If there is, related work ID = 4 is registered.

As a result of this determination, if the related work ID corresponding to the work ID corresponding to “NG” is registered in the table of FIG. 3C as the execution determination result 316, the process proceeds to step S804, and if not registered, the process The process proceeds to step S805. The table of FIG. 3F may be deleted after the process of step S803 is completed.

In step S804, as shown in FIG. 3E, the generation unit 116 puts the related work ID at the top, and following it, the standard work sequence 111 corresponding to phase ID = (value of variable F + 1) in the table of FIG. Create the added work sequence 313. Further, the generation unit 116 takes, as an area sequence 314, a sequence in which the area IDs (obtained from the table in FIG. 3C and the table in FIG. 3D) corresponding to the work ID are arranged in the arrangement order of each work ID included in the work sequence 313. create. Then, the generation unit 116 generates, as a temporary work sequence 112, a table in which the phase ID 310 (= value of variable F + 1), the work sequence 313, and the area sequence 314 are associated.

In step S805, the notification control unit 115 notifies the worker of the work to be performed next. When the process proceeds from step S803 to step S805, the notification control unit 115 specifies a work sequence 311 corresponding to phase ID = (value of variable F + 1) from the table of FIG. 3D. Then, the notification control unit 115 notifies of the work ID of the first work order in the specified work sequence 311 and the work name corresponding to the work ID in the table of FIG. 3C. On the other hand, when the process proceeds from step S804 to step S805, the notification control unit 115 specifies a work sequence 313 corresponding to phase ID = (value of variable F + 1) from the table of FIG. 3E. Then, the notification control unit 115 notifies of the work ID whose execution order is first in the specified work sequence 313 and the work name corresponding to the work ID in the table of FIG. 3C.

As described above, when the temporary work sequence 112 is not created, the standard work sequence 111 is used as a collating sequence after returning to the normal mode. Therefore, the information related to the first work of the next phase in the standard work sequence 111 To inform On the other hand, when the temporary work sequence 112 is created, since the temporary work sequence 112 is used as a comparison sequence after returning to the normal mode, information on the leading work is notified in the temporary work sequence 112. The notification method is not limited to a specific method, and as described above, may be notification by voice or notification by display. Also, the content to be notified is not limited to the specific content.

In step S806, the mode control unit 114 shifts the operation mode of the information processing device 200 from the standby mode to the normal mode. Furthermore, the control unit 199 increments the value of the variable F by one.

Then, the process proceeds to step S402 (more specifically, step S503). When the process proceeds from step S806 to step S503, the temporary work sequence 112 generated in step S804 described above is acquired (used instead of the standard work sequence 111) as the collation sequence from step S503. become.

As described above, according to the present embodiment, when notification of an operation implementation defect is notified, the operation mode shifts to a warning mode and a standby mode in which processing different from the normal mode is performed, and execution of the operation is performed while the operator re-executes the operation. Suspend the judgment process. Then, after the re-execution of the work, it is determined which work should be resumed and the worker is notified, and a defect in work after the re-execution is also detected. Further, the operator can easily notify the information processing apparatus 200 of a warning comprehension and completion of the reworking by a gesture or the like by a mode shift instruction using a video.

<Modification>
The notification control unit 115 notifies the operator of a message prompting the user to return to the normal mode (for example, a message prompting to perform the second gesture) when the duration time of the standby mode exceeds the prescribed time. good. With such a configuration, when the operator forgets a gesture for instructing transition to the normal mode, the gesture can be prompted.

In the first embodiment, the information processing apparatus 200 may notify the worker when the duration of the standby mode exceeds the prescribed time in the processing of the standby mode. When the information processing apparatus 200 is configured as such, when the operator forgets the instruction to shift to the normal mode, the mode shift instruction can be prompted. In the above case, since the worker has usually performed the work with forgetting the mode transition instruction, the state where the standard sequence and the work performed by the worker do not match at the time of return to the normal mode continues, and an erroneous warning is given. It may continue to be notified. In order to solve such an exceptional case, the information processing apparatus 200 resets the phase position of the work execution determination process to the initial state “first phase” or “worker designated phase”. May be provided. Similar to the mode shift instruction, the reset instruction and the ID of the phase to be reset can be received by a specific gesture or the like of the worker.

In the first embodiment, the mode transition instruction is performed by a gesture. However, the mode transition instruction may be performed by another method. For example, by causing the imaging device 100 to capture an object such as a marker and the information processing device 200 recognizes an object in a captured image, the mode may be shifted to a mode according to the type of the recognized object. Alternatively, a plurality of objects may be set in advance in the imaging range of the imaging device 100, and the operator may instruct mode transition by pointing to one of the plurality of objects. In this case, the information processing apparatus 200 recognizes an object pointed to by the worker from the captured image, and shifts the mode to a mode according to the type of the recognized object.

Further, in the present embodiment, an instruction from the operator can be received in addition to the mode transition instruction. For example, when the worker can not hear the notification about the work resumption, the instruction to re-notify may be performed. In this case, when the information processing apparatus 200 detects a re-notification instruction from the operator, the information content of the immediately preceding notification is notified again.

Furthermore, the operator may be notified of the operation mode of the information processing apparatus 200. For example, the operator may be notified of the operation mode of the information processing apparatus 200 by the lighting pattern of the lamp connected to the information processing apparatus 200 or the display content of the display screen of the information processing apparatus 200. According to such a configuration, the operator can grasp whether the mode shift instruction issued by the operator has been correctly detected by the information processing apparatus 200.

Second Embodiment
Hereinafter, differences from the first embodiment will be described, and it is assumed that the second embodiment is the same as the first embodiment unless otherwise specified. In the first embodiment, the execution determination of the work is interrupted and not performed during the warning mode and the standby mode. However, in the case where the assembly work is performed at a very high speed, it is conceivable that the worker has already started the work of the next phase when the information processing apparatus 200 notifies of a work defect in a certain phase. . In this case, it is assumed that the worker finishes the work that is currently underway, and then re-does the work for which the defect has been notified.

The present embodiment takes into consideration the time lag between the work completion timing of the worker and the notification timing of the execution determination result of the work, and the work performed by the worker until the instruction to shift from the warning mode to the standby mode is input. A temporary work sequence 112 is generated based on.

Here, an operation example of the work monitoring system according to the present embodiment will be described with reference to FIG. When the information processing apparatus 200 in the normal mode detects a break of phase 1 (between work 4 and work 5) from the captured image, whether or not the works 1 to 4 constituting the phase 1 are normally performed in this order Make an implementation decision. It is assumed that there is a defect in work 3 by this implementation determination. In this case, the operation mode of the information processing apparatus 200 shifts from the normal mode to the warning mode, the operator is notified of the deficiency of the operation 3 and the re-execution of the operation 3 is prompted. However, the worker carries out each work in a relatively short time, and it takes time to acquire a photographed image and perform processing such as execution determination of the work of phase 1 and the like. Therefore, as shown in FIG. 9, when the information processing apparatus 200 detects a break in phase 1 and determines the execution of the work, and notifies the defect in the work 3 detected by the execution determination, the worker has already made the phase 2 Work 5 of is carried out. In this embodiment, in such a case, the operator completes the operation 5 and then instructs the shift to the standby mode to perform the operation 3 again. Then, when the information processing apparatus 200 shifts to the normal mode, since the work 5 has already been performed, the information processing apparatus 200 instructs the work restart from the work 6.

In the present embodiment, the information processing apparatus 200 performs processing according to the flowchart of FIG. 10 in the warning mode (that is, in the present embodiment, processing according to the flowchart of FIG. 10 is performed in step S403).

In step S1001, the notification control unit 115 gives notification of an operation defect as in step S701 described above. In step S1002, the work determination unit 110 reads captured images in the order of storage from the memory 198. The captured image read out in step S1002 is a captured image after the phase has been switched, and in the case of FIG. 9 is a captured image corresponding to the start of task 5.

In step S1003, the work determination unit 110 extracts the hand area from the captured image read in step S1002 by performing the same process as step S505 described above. In step S1004, the work determination unit 110 performs the same process as step S506 described above to determine which of the work areas on the captured image the hand area extracted in step S1003 is included. That is, in steps S1003 and S1004, it is possible to detect the work performed between the transition to the warning mode and the transition to the standby mode, and to register the corresponding area ID in the detection area sequence 315. Steps S1005 to S1007 are the same as steps S703 to S705, respectively, and thus the description of these processes is omitted.

Further, in the present embodiment, the information processing apparatus 200 performs processing according to the flowchart of FIG. 11 in the standby mode (that is, in the present embodiment, processing according to the flowchart of FIG. 11 is performed in step S404).

Step S1101 is the same as step S801 described above, and thus the description of this step is omitted. When the above-described second gesture by the worker is recognized from the captured image read out in step S1101, the process proceeds to step S1103 via step S1102. On the other hand, if the above-described second gesture by the worker is not recognized from the captured image read out in step S1101, the process proceeds to step S1101 via step S1102.

In step S1103, the work determination unit 110 acquires the area sequence 312 corresponding to phase ID = (value of variable F + 1) from the table in FIG. 3D, and the area sequence 312 and the detection area sequence 315 generated in step S1004. ,. For example, the above DP matching is used for the matching. By such collation, it is determined whether the detection area sequence 315 generated in step S1004 is included in the area sequence 312 corresponding to phase ID = (value of variable F + 1). That is, whether or not the work performed by the worker between the phase switching detection (step S507) and the notification (step S1001) is included in the work sequence 311 corresponding to phase ID = (value of variable F + 1). To judge. This is to determine whether or not the operator has implemented a part of the next phase (the phase corresponding to phase ID = (value of variable F + 1)) after phase switching detection (step S507). It is equivalent to. For example, it is assumed that there is a partial sequence corresponding to the detection area sequence 315 generated in step S1004 in the area sequence 312 corresponding to phase ID = (value of variable F + 1) as a result of DP matching. At this time, it can be determined that the work corresponding to the partial sequence has already been performed.

As a result of the collation in step S 1103, if there is work already performed in the area sequence 312 corresponding to phase ID = (value of variable F + 1), the process proceeds to step S 1105 via step S 1104. On the other hand, if there is no work performed in the area sequence 312 corresponding to phase ID = (value of variable F + 1) as a result of the collation in step S1103, the process proceeds to step S1106 via step S1104. .

In step S1005, the work determination unit 110 selects a work ID corresponding to the work that has already been identified in the check in step S1003 from the standard work sequence 111 corresponding to phase ID = (value of variable F + 1) in the table of FIG. 3D. The deleted work sequence is created as a work sequence 313.

In step S1106, the work determination unit 110 identifies the work ID whose execution determination result 316 corresponds to "NG" in the table created in step S511, and the related work ID corresponding to the work ID is the table in FIG. 3C. It is judged whether it is registered in. As a result of this judgment, if the related work ID corresponding to the work ID corresponding to “NG” in the execution judgment result 316 is registered in the table of FIG. 3C, the process proceeds to step S1107, and if not registered, the process The process proceeds to step S1108.

In step S1107, when the work sequence 313 is created in step S1105, the generation unit 116 adds a related work ID at the top of the work sequence 313 as the work sequence 313 again. In addition, when the work sequence 313 is not generated in step S1105, the generation unit 116 performs the same operation as that in step S804. That is, the generation unit 116 generates the work sequence 313 by adding the standard work sequence 111 corresponding to the phase ID = (value of variable F + 1) in the table of FIG. .

Then, the generation unit 116 generates the area sequence 314 in the same manner as step S 804 above, and sets a table in which the phase ID 310 (= value of variable F + 1), the work sequence 313 and the area sequence 314 are associated as a temporary work sequence 112. Generate

Steps S1108 and S1109 are the same as steps S805 and S806, respectively. In step S1108, when the temporary work sequence 112 is not created, the notification control unit 115 specifies the work sequence 311 corresponding to phase ID = (value of variable F + 1) from the table of FIG. 3D. Then, the notification control unit 115 notifies of the work ID of the first work order in the specified work sequence 311 and the work name corresponding to the work ID in the table of FIG. 3C. On the other hand, when the temporary work sequence 112 is created, the notification control unit 115 specifies the work sequence 313 corresponding to the phase ID = (value of variable F + 1) from the table of FIG. 3E. Then, the notification control unit 115 notifies of the work ID whose execution order is first in the specified work sequence 313 and the work name corresponding to the work ID in the table of FIG. 3C.

As described above, according to the present embodiment, there is a time lag between the work completion timing of the worker and the notification timing of the execution determination result of the work, and the worker has performed the next work before the notification of the work deficiency. Even in the case, it is possible to correctly give instructions for resuming work after reworking. And according to such a configuration, it is possible to eliminate unnecessary re-work regarding the work that has already been performed.

Third Embodiment
In the first and second embodiments, the execution determination of the work is performed using the switching of the phase as a trigger, but the trigger for determining the execution of the work is not limited to the switching of the phase. For example, when one process is short, or when one process is handled as one phase, etc., the execution determination of the work for the process may be performed when one process is completed. In this case, since one standard work sequence is defined for one process, and the standard work sequence is repeatedly performed, a temporary work sequence is always generated for this standard work sequence.

When one process is configured with operations 1 to 4, an operation of the information processing apparatus 200 that performs the execution determination of the operation on the one process when the one process is completed will be described using FIG. 12A. Do.

As shown in FIG. 12A, it is assumed that a defect is detected in operation 2 because the execution determination of operations 1 to 4 is performed when operations 1 to 4 are completed. Further, it is assumed that an operation 4 is defined as a related operation of the operation 2. When a defect in work 2 is detected in the normal mode, the operation mode of the information processing apparatus 200 shifts to a warning mode, and since there is a defect in work 2, notification is given to prompt the operator to re-perform work 2. When the worker instructs to shift to the standby mode, the operation mode of the information processing device 200 shifts from the warning mode to the standby mode. In the standby mode, after the operator re-performs work 2, the instruction to shift to the normal mode is issued. At this time, the information processing apparatus 200 sets “4, 1, 2, 3, 4” as a temporary work sequence in which “work 4,” which is a related work, is added to the head of the work sequence “1, 2, 3, 4”. After generating and returning to the normal mode, this temporary work sequence is used. In addition, as notification to the worker, after the execution instruction of the operation 4 is given, the operation start from the operation 1 can be instructed to the next target object.

In addition, the execution determination of the work may be performed for each work constituting one process. For example, for each work from the top of one standard work sequence, the execution determination result of the work is sequentially collated, and it is possible to determine that the work is not performed when a work not corresponding to the standard work sequence occurs. In this case, one standard work sequence is defined for one process, and the temporary work sequence can be generated on the basis of a sequence obtained by cutting the work determined to be performed from the standard work sequence.

Further, in the first and second embodiments, in the warning mode, notification is provided to prompt re-execution of only the work for which a defect has been detected. However, if there is a related work associated with the work for which the defect is detected, in addition to a notification prompting re-execution of the work for which the defect is detected, a notification for promoting re-execution of the related work related to the work is performed. It is good. The operation of the information processing apparatus 200 in the case of performing such notification will be described using FIG. 12B.

Similar to FIG. 9, it is assumed that there is a division of phase 1 between work 4 and work 5 and work 4 is defined as a related work of work 2. When the information processing apparatus 200 in the normal mode detects a break of phase 1 (between work 4 and work 5) from the captured image, whether or not the works 1 to 4 constituting the phase 1 are normally performed in this order Make an implementation decision. It is assumed that there is a defect in work 2 due to this implementation determination. When a defect is detected in operation 2, the operation mode of the information processing apparatus 200 shifts to the warning mode, and since the operation 2 is defective, the operation 4 is a related operation not only to the operation 2 but also to the operation 2 Also perform notification to encourage reimplementation. When the worker instructs to shift to the standby mode, the operation mode of the information processing device 200 shifts from the warning mode to the standby mode. In the standby mode, after the operator re-performs the work 2 and the work 4, an instruction to shift to the normal mode is issued. After returning to the normal mode, since the operation is performed from the operation 5 which is the leading operation of the next phase, the notification to prompt the operator to perform the operation 5 is performed. The information processing apparatus 200 may not generate the temporary work sequence to which the related work is added. In this embodiment, although it can not be determined whether the related work has been correctly re-executed, the system configuration can be simplified.

Fourth Embodiment
In each of the embodiments and the modifications described above, the data generated by the information processing apparatus 200, such as the temporary work sequence 112, may be appropriately deleted if it becomes unnecessary. In the above description, for example, when using the standard work sequence 111 instead of the temporary work sequence 112, the temporary work sequence 112 may be deleted. Conversely, you may save without deleting.

Further, in each of the embodiments and the modifications described above, the photographed video of the worker is acquired from the imaging device 100, but the present invention is not limited to this. For example, when the captured video of the recorded operator is stored in the external device, the acquisition unit 108 may obtain the captured video from the external device.

Further, the configurations of various tables shown in FIGS. 3B to 3F are merely examples, and the numerical values in the tables and the configurations of the tables are not limited to the examples of FIGS. 3B to 3F. Also, several tables may be combined into one table, or one table may be divided into several tables.

Fifth Embodiment
Each functional unit in the information processing apparatus 200 illustrated in FIG. 1B may be implemented by hardware. However, the acquisition unit 108, the detection unit 113, the work determination unit 110, the mode control unit 114, the notification control unit 115, the generation unit 116, and the control unit 199 may be implemented by software (computer program). In this case, a computer device having a memory 198 and having a processor capable of executing such a computer program is applicable to the information processing device 200 described above. An exemplary hardware configuration of a computer apparatus applicable to the information processing apparatus 200 will be described using the block diagram of FIG. 1A.

The CPU 101 executes processing using computer programs and data stored in the ROM 102 and the RAM 103. Thus, the CPU 101 controls the entire operation of the computer apparatus, and executes or controls each process described above as being performed by the information processing apparatus 200.

The ROM 102 stores computer programs and data that do not require rewriting (such as computer programs and data of the BIOS). The RAM 103 has an area for storing a computer program or data loaded from the storage unit 104 and data received from the outside via the communication unit 107. Further, the RAM 103 has a work area used when the CPU 101 executes various processes. Thus, the RAM 103 can appropriately provide various areas.

The storage unit 104 is a large-capacity information storage device represented by a hard disk drive. The storage unit 104 stores an operating system (OS) and computer programs and data for causing the CPU 101 to execute or control each process described above as being performed by the information processing apparatus 200.

The computer program stored in the storage unit 104 includes the functions of the acquisition unit 108, the detection unit 113, the operation determination unit 110, the mode control unit 114, the notification control unit 115, the generation unit 116, and the control unit 199 described above. Contains a computer program to make it happen.

The data stored in the storage unit 104 includes one handled as known information in the above description and one described as information stored in the memory 198. That is, the data stored in the storage unit 104 includes various tables of FIGS. 3B to 3F including the standard work sequence 111 and the temporary work sequence 112.

The computer program and data described as being stored in the storage unit 104 are appropriately loaded to the RAM 103 according to control of the CPU 101, and are to be processed by the CPU 101. The above memory 198 can be implemented by the ROM 102, the RAM 103, and the storage unit 104.

The input unit 105 is configured of a user interface such as a keyboard, a mouse, and a touch panel screen, and can input various instructions to the CPU 101 by the user's operation.

The output unit 106 is a device for outputting various types of information, such as a display device that displays the processing result of the information processing device 200 as an image or characters, a device that performs various notifications as described above, and the like.

The communication unit 107 is for performing data communication with an external device via a network such as a LAN or the Internet. For example, the above-described imaging device 100 can be connected to the communication unit 107, and an image captured by the imaging device 100 is stored in the RAM 103 or the storage unit 104 via the communication unit 107.

The CPU 101, the ROM 102, the RAM 103, the storage unit 104, the input unit 105, the output unit 106, and the communication unit 107 are all connected to the bus 180. The configuration shown in FIG. 1A is only an example of the hardware configuration of the computer apparatus applicable to the information processing apparatus 200. For example, a DVD-ROM / RAM or a USB memory may be used as the information storage device, or the input unit 105 may be omitted, and instead, an operation input may be received via a touch panel screen. In this case, the touch panel screen realizes both functions of information display and operation acceptance.

In addition, you may use combining suitably one part or all of each embodiment and each modification which were demonstrated above. In addition, a part or all of each embodiment and each modification described above may be selectively used.

[Modification]
The work determination method in the above-mentioned embodiment is not limited to the method using which work area the hand area is in. For example, the work content may be specified by machine learning using a learned learning model based on an image of a frame that constitutes a moving image input from the imaging device 100. Specifically, a learning model in which motion features of a plurality of work contents set in advance are learned is created, and a score for each work content is obtained for an image of one frame. The task determination unit 110 specifies the task content with the largest score as the task content corresponding to the frame. In addition, in preparation of the above-mentioned learning model, various methods exist and the method is not specifically limited. For example, in the case of using a method called deep learning, a large amount of data obtained by photographing the correct work content is prepared first, and a model is created by inputting these data into a learning network.

The work determination unit 110 may appropriately correct the work content determination result obtained for each frame. The correction process is a process of correcting a false recognition and the like, and can be corrected based on the continuity before and after the work determination result.

The task determination unit 110 can determine the task deficiency by collating the sequence of tasks. Under the present circumstances, you may comprise so that an operation implementation defect may be determined for every operation | work, without dividing a process into a phase. For example, one standard work sequence and the detection result are sequentially collated from the front, and it is possible to determine that the work is incomplete when work not corresponding to the standard work sequence occurs.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

The present invention is not limited to the above embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Accordingly, the following claims are attached to disclose the scope of the present invention.

The present application claims priority based on Japanese Patent Application No. 2017-211934 filed on Nov. 1, 2017, the entire contents of which are incorporated herein by reference.

108: acquisition unit 110: operation determination unit 113: detection unit 114: mode control unit 115: notification control unit 116: generation unit 198: memory 199: control unit

Claims

The processing means is provided with a processing means for executing detection processing for detecting a defect in work from a photographed image of a worker sequentially performing a series of work included in the work sequence,
The processing means
The detection processing is interrupted until detection of an instruction from the worker is detected when a defect in work is detected from the photographed video by the detection processing, and when the instruction is detected after the interruption, the detection processing is resumed after the detection processing is resumed. Target work to be the target of the work from the continuation of the performed work to the target work or from the work related to the work for which the deficiency is detected, or specified based on the work for which the deficiency is detected And resuming the detection process for the target work.
2. The information processing apparatus according to claim 1, wherein the processing means, upon detecting a defect in work from the photographed video, notifies information on the work until detecting a first instruction.
The processing means is provided with a processing means for executing detection processing for detecting a defect in work from a photographed image of a worker sequentially performing a series of work included in the work sequence,
The processing means
When the defect in work is detected from the photographed video by the detection process, notification of information related to the work is performed, and when the first instruction is detected, the notification is interrupted, and when the second instruction is detected after the interruption, An information processing apparatus characterized by specifying a target work to be a target of the detection process after resuming the detection process based on the work in which the deficiency is detected, and resuming the detection process for the target work.
4. The processing method according to any one of claims 1 to 3, wherein among the plurality of tasks included in the task sequence, the task included in the order of execution after the task for which the insufficiency has been determined is specified as the target task. An information processing apparatus according to any one of the preceding items.
The processing means targets the related work associated with the work for which the deficiency is detected, and the work included in the order of execution after the work for which the deficiency has been determined among the plurality of works included in the work sequence. The information processing apparatus according to any one of claims 1 to 3, wherein the information processing apparatus is specified as an operation.
The processing means notifies after the first instruction that the duration until the input of the second instruction is equal to or longer than a prescribed time, and is included in the work sequence according to the input of the third instruction. 4. The information processing apparatus according to claim 3, wherein among the plurality of tasks to be executed, the task in the order after the task specified in the first task or the third instruction in the execution order is specified as the target task.
The processing unit is configured to start the detection processing for the work included in the work sequence and to perform notification of a defect in the work or until the first instruction is input. The information processing apparatus according to claim 3, wherein the remaining work obtained by deleting the work performed by the work group from the work group in the order of execution subsequent to the work for which the insufficiency has been determined is specified as the target work. .
4. The processing method according to any one of claims 1 to 3, wherein the processing means specifies, as the target work, the related work associated with the work for which the deficiency is detected and the work included in the work sequence. The information processing apparatus according to claim 1.
4. The information processing apparatus according to claim 3, wherein the processing means, when detecting the second instruction, notifies of information related to the first work in the execution order among the target works.
The information processing apparatus according to any one of claims 4 to 7, wherein the processing means detects the occurrence of a prescribed event from the photographed video as a work break.
11. The information processing apparatus according to claim 10, wherein the event is a change in a specified position and / or posture of an object to be worked by the worker.
The information processing apparatus according to claim 3, wherein the first instruction and the second instruction are any one or more of a gesture and a marker.
The information processing apparatus according to claim 3, wherein the first instruction and the second instruction are instructions according to a result of recognition of an object included in the captured video.
The information according to claim 3, wherein the first instruction and the second instruction are instructions according to a result of recognition of a designated object among a plurality of objects included in the captured video. Processing unit.
The processing means is characterized in that the instruction of the re-notification of the information related to the work in which the defect is detected from the photographed image is determined from the photographed video, and the notification content immediately before is notified again according to the instruction of the re-notification. The information processing apparatus according to any one of claims 1 to 14, wherein:
The information processing apparatus according to any one of claims 1 to 15, wherein the processing means uses one or more of a voice, a lighting pattern of a lamp, and a display to notify information related to an operation for which a defect is detected. The information processing apparatus according to claim 1.
The information processing apparatus according to any one of claims 1 to 16, wherein the processing means notifies an operator of an operation mode of the information processing apparatus.
An information processing method performed by the information processing apparatus;
The processing means of the information processing apparatus comprises a processing step of executing a detection processing of detecting a defect in work from a photographed image of a worker who is sequentially performing a series of work included in a work sequence;
In the processing step,
The detection processing is interrupted until detection of an instruction from the worker is detected when a defect in work is detected from the photographed video by the detection processing, and when the instruction is detected after the interruption, the detection processing is resumed after the detection processing is resumed. Target work to be the target of the work from the continuation of the performed work to the target work or from the work related to the work for which the deficiency is detected, or specified based on the work for which the deficiency is detected And resuming the detection process for the target work.
A computer program for causing a computer to function as the processing means according to any one of claims 1 to 17.