JP2020068009A - Operation detection device, operation detection method, and operation detection system - Google Patents

Abstract

To provide an operation detection device, an operation detection method, and an operation detection system capable of detecting accurately in real time an operation of each operator on a production line without relying on manual work.SOLUTION: An operation detection device according to an embodiment includes: an object detection model acquisition part; an object information acquisition part; a detection image input part; and an operation detection part. The object detection model acquisition part acquires an object detection model based on a plurality of training images. The object information acquisition part acquires object information on the basis of the plurality of training images and the object detection model acquired by the object detection model acquisition part. The detection image input part inputs a detection image for detecting an operation of an operator. The operation detection part detects the operation on the basis of the object information acquired by the object information acquisition part and the detection image input by the detection image input part.SELECTED DRAWING: Figure 3

Description

  Embodiments of the present invention relate to an operation detection device, an operation detection method, and an operation detection system.

  At present, production lines for manufacturing or assembling various products are widely used in various industries. In each production line, each production unit intensively processes the work of either segment so as to improve the operation efficiency and the production amount. Generally, a plurality of stations can be arranged in a production line depending on factory demand, product complexity, manufacturing or assembly complexity, and the like. The operator at each station manually repeats operations different from those of other operators. After completing the operation at one station, the operator flows to the next station to perform the operation and finally completes the manufacturing / assembly.

  Therefore, the operator's operation at each station of the whole production line is directly related to the quality of the product and the efficiency of the whole production line. In order to guarantee the quality of products and the efficiency of the whole production line, it is usual to set various working processes at each station of the whole production line, and to prepare a corresponding work manual, by comprehensively considering various factors. Prepare The work manual usually includes a plurality of actions and reference times included in the work process. However, in reality, the operator's operation is a manual operation, and there are instability factors in the operating environment. Therefore, it is possible to guarantee for each operator that all repeated operations are the same. Have difficulty. Further, it is difficult to ensure that each operation of the repeated operations is strictly conforming to the work manual. And, there is a difference in skill among the operators. Therefore, such reproducibility cannot be guaranteed. Variations in work manuals and differences in skills may not only cause quality problems such as product defects, but may also affect the production line balance of the entire production line. The production line balance is a technical means and method that averages the operation of each station and adjusts the work load so that each operation time becomes as close as possible, and its quality depends on the production efficiency and product quality of the production line. Directly related.

  Therefore, it is necessary to acquire the production line balance information by grasping the working state of the operator of each station on the production line. At present, grasping the working state of each station on the production line is mainly performed by an IE engineer (also called an industrial engineer) by manually measuring the man-hours for each station by a stopwatch. With such a method, it is possible to intuitively observe the presence or absence of the operation by the operator on the station, and the operator on the station records the information such as the start time and the end time when the operation including a plurality of operations is performed. can do.

  However, the method in which the IE engineer manually records and grasps the working state in the production line has the following problems. The reliance on human eye observation and manual operation reduces the accuracy of the recorded data and limits the amount of recorded data; IE engineers affect the psychology of the operator on the station when measuring And the operations performed at such times cannot reflect normal operations; the balance of the production line can only be obtained after the IE engineer has completed the measurement and performed the analytical process. , It is not possible to monitor the production line in real time; since the time recorded by the IE engineer is finite, not every abnormal operation or misoperation on the production line is necessarily recorded, which results in product defects. When it occurs, it is difficult to determine the cause of the defect, and the cause of the product defect is timely and efficiently. It is difficult to fix to.

  Therefore, an operation detection device, an operation detection method, and an operation detection system that detect and grasp an operation of an operator on a production line without relying on the acquisition have been earnestly desired.

Chinese Patent No. 1037542528 Chinese Patent No. 103793054

  The present invention has been made to solve the problems existing in the above-mentioned conventional techniques, and an object thereof is an operation capable of accurately and in real time detecting the operation of each operator on a production line without relying on human labor. An object is to provide a detection device, an operation detection method, and an operation detection system.

  The operation detection device according to the embodiment includes a target detection model acquisition unit, a target information acquisition unit, a detected image input unit, and an operation detection unit. The target detection model acquisition unit acquires a target detection model based on a plurality of training images. The target information acquisition unit acquires target information based on the plurality of training images and the target detection model acquired by the target detection model acquisition unit. The detected image input unit inputs a detected image for detecting an operator's operation. The operation detection unit detects the operation based on the target information acquired by the target information acquisition unit and the detected image input by the detected image input unit.

It is a figure which shows the structure of the operation detection apparatus of the 1st Embodiment of this invention. It is a figure which shows one training image with which the target of this invention was marked. It is an operation | movement flowchart of the operation detection apparatus of 1st Embodiment. It is a figure which shows the structure of the operation detection part of Example 1 of the 1st Embodiment of this invention. It is a figure which shows an example of the event generation method of this invention. It is a figure which shows an example of the detail of the object type of this invention. It is a figure which shows an example of the detail of the object type of this invention. It is a figure which shows an example of the detail of the object type of this invention. It is a figure which shows an example of the detail of the object type of this invention. It is a figure which shows an example of the event of this invention. It is a figure which shows an example of the operation mode produced | generated based on the event of this invention. It is an operation | movement flowchart of the operation detection apparatus of Example 1 of the 1st Embodiment of this invention. It is a figure which shows an example of the event which concerns on the operation detection apparatus of Example 1 of the 1st Embodiment of this invention. It is a figure which shows the structure of the operation detection part of Example 2 of the 1st Embodiment of this invention. It is a figure which shows an example of the operation mode produced | generated based on the event of this invention. It is a figure which shows an example of the operation mode produced | generated based on the event of this invention. It is a figure which shows an example of the operation mode sequence produced | generated based on the event of this invention. 5 is an operation flow chart of the operation detecting device of the second example of the first exemplary embodiment of the present invention. FIG. 8 is a diagram showing an example of an event related to the operation detection device of Example 2 of the first exemplary embodiment of the present invention. FIG. 8 is a diagram showing an example of an operation mode generated by the operation detection device of Example 2 of the first exemplary embodiment of the present invention based on the operation mode. FIG. 8 is a diagram showing an example of an operation mode generated by the operation detection device of Example 3 of the first exemplary embodiment of the present invention based on an event and an operation mode. 7 is an operation flowchart of the operation detection device of Example 3 of the first exemplary embodiment of the present invention. FIG. 6 is a diagram showing a configuration of an operation detection device according to a second exemplary embodiment of the present invention. 7 is an operation flowchart of the operation detecting device according to the second embodiment of the present invention. It is a figure which shows the structure of the operation detection system of this invention.

  Hereinafter, embodiments will be described with reference to the drawings. The following description exemplifies a device and a method for embodying the technical idea of the embodiment, and the technical idea of the embodiment is the structure, shape, arrangement, and material of the components described below. It is not limited to the above. Modifications that can be easily conceived by those skilled in the art are naturally included in the scope of the disclosure. In order to make the description clearer, in the drawings, the size, thickness, plane size, shape, or the like of each element may be changed and schematically shown in the actual embodiment. In some drawings, elements having different dimensional relationships and ratios from each other may be included. In a plurality of drawings, corresponding elements may be given the same reference numerals, and redundant description may be omitted. Although some elements may be given a plurality of names, examples of these names are merely examples, and giving other names to these elements is not denied. Further, it is not denied that another name is given to an element to which a plurality of names are not given. In the following description, “connection” means not only direct connection but also indirect connection through other elements.

  Hereinafter, an operation detection device, an operation detection method, and an operation detection system of the present invention will be described with reference to the drawings.

(First embodiment)
(Configuration of operation detection device 100)
First, the configuration of the operation detection device 100 according to the first embodiment will be described with reference to FIG. Further, the operation detection device includes various parts, and in FIG. 1, only parts related to the technical idea of the first embodiment are shown, and other parts are omitted.

  Further, although the operation detecting device can be configured to detect various operations, in the present specification, the operation detecting device is used in an assembly production line of electronic products to show the operation state of the operator of each station in the production line. The detection will be described as an example. The operation detection device of the first embodiment detects the operation based on a moving image obtained by imaging the operation of the operator of each station in the production line.

  As shown in FIG. 1, the operation detection device 100 includes a target detection model acquisition unit 101, a target information acquisition unit 102, a detected image input unit 103, and an operation detection unit 104.

  The target detection model acquisition unit 101 is for acquiring a target detection model obtained based on a plurality of training images.

  Here, the target detection model acquisition unit 101 acquires the target detection model from an external device (not shown) of the operation detection device 100 via wired or wireless communication.

  The external device is a target detection model generation member that can generate a target detection model using, for example, the following method. The external device processes a moving image, which is a training image group obtained by capturing the operation of the operator of each station on the production line, extracts a plurality of training images, and outputs the training images to the training images of the plurality of training images. Mark one or more objects. An example of marking is shown, for example, in FIG. Objects such as "machine", "hand", and "hand" are marked on one training image. The external device performs deep learning based on a plurality of training images including time-series information while marking one or more targets, and generates (trains) a target detection model. Here, the method of generating the target detection model has been described as deep learning, but this is an example, and the present invention is not limited to this. If an accurate object detection model can be obtained, various methods that are known or may appear in the future can be used.

  The target information acquisition unit 102 acquires target information from a plurality of training images based on the target detection model acquired by the target detection model acquisition unit 101. For example, by using a target detection model trained by deep learning for a plurality of training images, target information included in a plurality of training images can be acquired. The target information that can be acquired is related to the target detection model, and the target detection model can be intentionally obtained based on the necessary target information. Here, the target information includes at least a target name, a target type, a target position, and a change in the target position (for example, a moving direction or a posture).

  An example of the detection image input unit 103 may be a camera that can input, as a detection image, a moving image obtained by capturing the operation of the operator of each station on the production line. Further, the example of the detection image input unit 103 is a data receiving device that can receive a moving image obtained by capturing an operation of an operator of each station in a production line as a detection image from outside via wired or wireless communication. It may be a functional wired or wireless communication module / component.

  The operation detection unit 104 detects an operation based on the target information acquired by the target information acquisition unit 102 and the detected image input by the detected image input unit 103. Here, the operation detection unit 104 can detect an operation, for example, by performing processing such as image processing or mode identification on the detected image based on the target information acquired by the target information acquisition unit 102. . Here, the operation means that the operator of each station in the production line uses a hand, a tool, or the like to put something in or out of a work piece, a component, a container, or the like in the electronic product assembly production line, and attach, position, fix, or otherwise perform various operations. Is to perform the operation. That is, the operation can appear due to changes in the object and its position.

  Further, although the target detection model acquisition unit 101, the target information acquisition unit 102, and the operation detection unit 104 are different components or modules, some or all of them can be combined into one component or module.

(Operation of the operation detection device 100)
Hereinafter, the operation detection apparatus 100 according to the first embodiment of the present invention and the operation detection method used by the operation detection apparatus 100 will be described based on an operation flowchart.

  FIG. 3 is an operation flowchart of the operation detection device 100 according to the first embodiment of this invention.

  As shown in FIG. 3, in step S300, the operation detection apparatus 100 causes the object detection model acquisition unit 101 to acquire an object detection model based on a plurality of training images. After that, the process of the operation detection device 100 proceeds to step S302.

  In step S302, the operation detection apparatus 100 causes the target information acquisition unit 102 to acquire target information based on the plurality of training images and the target detection model acquired by the target detection model acquisition unit 101. After that, the process of the operation detection device 100 proceeds to step S304. Here, it is assumed that the target information including the target name, the target type, the target position, the change in the target position, and the change in the relative position between the targets is acquired.

  In step S304, the operation detection apparatus 100 inputs the moving image obtained by capturing the operation of the operator of each station in the production line as the detected image by the detected image input unit 103. After that, the process of the operation detection device 100 proceeds to step S306.

  In step S306, the operation detection device 100 causes the operation detection unit 104 to detect an operation based on the target information acquired by the target information acquisition unit 102 and the detected image input by the detected image input unit 103. Then, the process of the operation detection device 100 ends.

  According to the operation detection device 100 according to the present embodiment, the operation detection is performed without relying on human hands by using the moving image obtained by capturing the operation of the operator of each station in the production line. In addition, it is possible to avoid the problems caused by the IE engineer performing manual operation observation. Further, by using an object detection model based on a method such as deep learning, the object involved in the operation of each operator on the production line reflected in the detected image is accurately detected, positioned, tracked, and Target information can be acquired. Thereby, the operation can be detected accurately, timely and efficiently, the operation abnormality can be detected without exception so as to predict or detect the product defect, and the operator on the production line to evaluate the operator. The operation of can be tracked. Further, by detecting the operation accurately and in a timely manner, the specific operation in the moving image can be searched for and positioned using the operation. This can contribute to improving the production line by analyzing and evaluating the operation of the operator.

  Hereinafter, each example of the operation detection device 100 of the first exemplary embodiment will be described. Hereinafter, corresponding elements will be denoted by the same reference numerals and redundant description will be omitted.

(Example 1)
Hereinafter, a first example of the operation detection device 100 according to the first embodiment of the present invention will be described.

  In the first embodiment, the structure of the operation detection device 100 includes the target detection model acquisition unit 101, the target information acquisition unit 102, the detected image input unit 103, and the operation detection unit 104, in the same manner as the structure shown in FIG. Those details will not be repeated.

  Further, the structure of the operation detection unit 104 in the first embodiment includes an event generation unit 1041, an operation mode generation unit 1043, and an operation detection unit 1045, as shown in FIG.

  The event generation unit 1041 generates an event based on the target information acquired by the target information acquisition unit 102. Here, an event is referred to as an event that occurs when a target interaction, that is, a relationship between targets occurs due to a change in a target position, or an event that occurs when a relationship changes between targets, and that a target appears or disappears in an image (imaging field of view). Including events.

  The event generation will be described below by taking an electronic product assembly production line as an example. FIG. 5 shows an example of a method of generating an event in an electronic product assembly and production line. That is, generally speaking, an event is generated based on a target, a target position and a target relationship.

  As shown in FIG. 5, the target type may be “hand”, “container”, “workpiece”, “part”, or “tool” in the electronic product assembly and production line. Here, it is classified as "hand", "container", "workpiece", "part", and "tool". The object type classification method is an example and is not limited to this, and other classification can be performed according to the object related to the production line and / or the actual need. FIG. 6 shows an example of details of the target type “container”. FIG. 7 shows an example of details of the target type “workpiece”. FIG. 8 shows an example of details of the target type “part”. FIG. 9 shows an example of details of the target type “tool”.

  Further, as shown in FIG. 5, the target relationship may be, for example, “top”, “bottom”, “left”, “right”, “center”.

  In addition, an event is generated based on the target position and the target relationship, and, for example, based on the target, the target position and the target relationship as described above, “take an item from the front”, “operate on the machine”, “screw” It is possible to recognize a target interaction such as “use a turn”, “use a scanning gun”, “operate on a main plate”, etc., and generate an event generated from the recognized target instruction. Further, it is possible to generate an event that an object such as "UsbPCB appears", "Main board appears", "Main board disappears", "Airframe appears" or the like appears or disappears in the image (field of view). FIG. 10 shows an event "operation is mounted on the aircraft-upper left". As shown in FIG. 10, the event type of the event is “operate on the work piece”, the target for generating the event is the work piece “machine”, and “the operation on the machine is The position of the operation related to the event “attached-upper left” corresponds to the target position, “horizontal left part vertical upper part”, and the SOP time of this event is “indeterminate”. Here, the SOP time is a reference time required for the occurrence of the event "operation is mounted on the aircraft-upper left", and the SOP time may be set before shipment of the device or set by the user after shipment. Alternatively, adjustment may be performed. Further, in FIG. 10, the SOP time is “indeterminate”, but the SOP time may be a specific time length, for example, 1.0 second, 2.0 seconds or the like.

  The operation mode generation means 1043 also generates an operation mode based on the event generated by the event generation means 1041. Here, the operation mode is how to configure an operation by an event. Further, the number of operation modes is one or more. The operation mode may be generated based on one event, or the operation mode may be generated based on one or more events. FIG. 11 shows an example of the operation mode generated based on the event. As shown in FIG. 11, the operation with the operation name “attach SYS PCB to COVER” is generated based on two events, and the event with the number “1” is performed after the event with the number “0”. FIG. 11 is an example, and the present invention is not limited to this.

  Further, the operation detection unit 1045 detects an operation based on the detection image input by the detection image input unit 103 and the operation mode generated by the operation mode generation unit 1043. For example, when an operation matching the operation mode generated by the operation mode generation unit 1043 as shown in FIG. 11 is identified from the detection image input by the detection image input unit 103, the operation is detected. As a method of detecting an operation suitable for the operation mode from the detected image based on the operation mode, for example, a technique such as a conventional mode identification can be applied, and details thereof will not be described here.

(Operation of Example 1)
Hereinafter, an operation detection method, which is an operation performed by the operation detection apparatus 100 according to the first embodiment, will be described with reference to FIG. Since steps S3060, S3062, and S3064 shown in FIG. 12 are operations of the operation detection unit 104, the operation flow shown in FIG. 12 can be obtained by using steps S3060, S3062, and S3064 instead of step S306 in FIG. It is what is done.

  As shown in FIG. 12, after the operation is started, the operation detection device 100 executes steps S300, S302, and S304. The operations of steps S300, S302, and S304 executed by the operation detection device 100 are the same as the operations of steps S300, S302, and S304 of FIG. 3 according to the first embodiment described above, and the description thereof will not be repeated here. Absent. After the detection image is input in step S304, the process of the operation detection device 100 proceeds to step S3060.

  In step S3060, the operation detection apparatus 100 causes the event generation unit 1041 to generate an event based on the target information acquired by the target information acquisition unit 102 in step S302. After that, the process of the operation detection device 100 proceeds to step S3062. Here, it is assumed that the event generation means 1041 generates two events shown in FIG. 13, for example.

  In step S3062, the operation detection device 100 causes the operation mode generation means 1043 to generate an operation mode based on the event (for example, as shown in FIG. 13) generated by the event generation means 1041 in step S3060. After that, the process of the operation detection apparatus 100 proceeds to step S3064. Here, the operation mode generation means 1043 is provided so as to generate the operation mode shown in FIG.

  In step S3064, the operation detection apparatus 100 causes the operation detection unit 1045 to detect the detection image input by the detection image input unit in step S304, and the operation mode generated in step S3062 by the operation mode generation unit 1043 (for example, shown in FIG. 11). So as to detect the operation based on and.

  The operation detection device 100 according to the present example has the technical effects of the first embodiment as described above. The operation detection device 100 uses an object detection model based on a method such as deep learning to accurately detect, position, and track the object related to the operation of each operator on the production line that is reflected in the detected image. An event can be generated by acquiring the target information of. As a result, the event can be detected, and the operation can be detected accurately, in a timely and efficient manner. It is possible to improve the detection efficiency against the detection of repeated operations by the operator on the production line and to contribute to the real-time detection of the operations. It is possible to detect the abnormal operation on the production line more quickly, by contributing to the adjustment of the production line and the production rate. This allows defective products to be found in a timely and efficient manner and adjusted as soon as possible to avoid producing more defective products.

(Example 2)
Example 2 of the operation detection device 100 according to the first exemplary embodiment will be described below.

  The structure of the operation detection device 100 according to the second embodiment includes a target detection model acquisition unit 101, a target information acquisition unit 102, a detected image input unit 103, and an operation detection unit 104, as in the structure shown in FIG. Those details will not be repeated.

  Further, the configuration of the operation detection unit 104 in the second embodiment is different from that of the first embodiment shown in FIG. 4 as shown in FIG. 14 in that an operation mode generation means 1042 is further provided.

  The operation mode generation means 1042 generates an operation mode based on the event generated by the event generation means 1041. Here, the operation mode is how to configure an operation by an event. Further, the number of operation modes may be one or more, and the plurality of operation modes form an operation mode sequence. The operation mode may be generated based on one event, or the operation mode may be generated based on an event sequence that is one or more events. 15 and 16 each show an example of an operation mode generated based on an event. As shown in FIG. 15, the action with the action name “scan” is generated based on two events, and the event with the number “1” is performed after the event with the number “0”. As shown in FIG. 16, the action with the action name “screw and fix” is also generated based on two events, and the event with the number “0” is followed by the event with the number “1”. Done.

  Further, the operation mode generation means 1042 may generate an operation mode sequence as shown in FIG.

  Further, in the second embodiment, the operation mode generation unit 1043 generates the operation mode based on the operation mode generated by the operation mode generation unit 1042. The operation detection unit 1045 detects an operation based on the detection image input by the detection image input unit 103 and the operation mode generated by the operation mode generation unit 1043.

(Operation of Example 2)
Hereinafter, an operation detection method, which is an operation performed by the operation detection device 100 according to the second embodiment, will be described with reference to FIG. Since steps S3060, S3061, S3063, and S3065 shown in FIG. 18 are operations of the operation detection unit 104, the operation flow shown in FIG. 18 is different from step S306 in FIG. Is obtained by using.

  As shown in FIG. 18, after the operation starts, the operation detection device 100 executes steps S300, S302, and S304. The operations of steps S300, S302, and S304 executed by the operation detection device 100 are the same as the operations of steps S300, S302, and S304 of FIG. 3 according to the first embodiment described above, and the description thereof will not be repeated here. Absent. After the detection image is input in step S304, the process of the operation detection device 100 proceeds to step S3060.

  In step S3060, the operation detection apparatus 100 causes the event generation unit 1041 to generate an event based on the target information acquired by the target information acquisition unit 102 in step S302. After that, the process of the operation detection device 100 proceeds to step S3061. Here, it is assumed that the event generation unit 1041 generates four events shown in FIG. 19, for example.

  In step S3061, the operation detection device 100 causes the operation mode generation unit 1042 to generate an operation mode based on the event (for example, shown in FIG. 19) generated by the event generation unit 1041 in step S3060. After that, the process of the operation detection device 100 proceeds to step S3063. Here, the operation mode generation means 1042 is assumed to generate the two operation modes shown in FIGS. 15 and 16.

  In step S3063, the operation detection device 100 causes the operation mode generation unit 1043 to generate an operation mode based on the operation mode (for example, shown in FIGS. 15 and 16) generated by the operation mode generation unit 1042 in step S3061. After that, the process of the operation detection device 100 proceeds to step S3065. Here, the operation mode generation means 1043 is assumed to generate the operation mode shown in FIG.

  In step S3065, the operation detection apparatus 100 causes the operation detection unit 1045 to detect the detection image input by the detection image input unit in step S304, and the operation mode generated in step S3063 by the operation mode generation unit 1043 (for example, shown in FIG. 20). ) And the operation is detected.

  The operation detection device 100 according to the present embodiment has the technical effects of the first embodiment and the first embodiment as described above. It should be noted that an object detection model based on a method such as deep learning is used to accurately detect, position, and track the object related to the operation of each operator on the production line reflected in the detected image. Furthermore, by generating the event by acquiring the target information of the target, and thereby generating the operation mode based on the event, it is possible to detect the operation mode and detect the operation in a timely and efficient manner. it can. Modes of operation are more complete acts than events. As described above, with respect to the detection in which the operation by the operator on the production line is repeated a plurality of times, it is not necessary to repeatedly detect the target and generate the event, as compared with the first embodiment, and the operation is performed based on only the generated operation mode. Can be quickly detected. Furthermore, it is possible to further improve the detection efficiency, further contribute to the real time property of detecting the operation, further contribute to the real time grasp and adjustment of the production line balance, and further contribute to the improvement of the productivity. It is also possible to detect abnormal operation on the production line faster. This allows defective products to be found in a more timely and effective manner and adjusted as soon as possible to avoid producing more defective products.

(Example 3)
Example 3 of the operation detection device 100 according to the first exemplary embodiment will be described below.

  The structure of the operation detection device 100 according to the third embodiment is the same as that of the second embodiment, and the details thereof will not be repeated here and only the differences will be described.

  The difference between the third embodiment and the second embodiment is that the operation mode generation means 1043 shown in FIG. 14 is based on the event generated by the event generation means 1041 and the operation mode generated by the operation mode generation means 1042. , Is to generate the operation mode.

(Operation of Example 3)
FIG. 22 is a flow of the operation detecting method which is the operation of the third embodiment executed by the operation detecting device 100. In FIG. 22, the same operations as the operations performed by the operation detecting device 100 according to the second embodiment are designated by the same reference numerals, and the duplicated description will be omitted.

  As shown in FIG. 22, the difference from the operation flowchart of FIG. 18 of the operation detection device 100 of the second embodiment is that step S3066 is used instead of step S3063.

  In step S3066, the operation detection apparatus 100 is operated by the operation mode generation means 1043 based on the event generated by the event generation means 1041 in step S3060 and the operation mode generated by the operation mode generation means 1042 in step S3061. After the mode is generated, the process proceeds to step S3065. Here, it is assumed that the operation mode shown in FIG. 21 is generated.

  Then, in step S3065, the operation detection device 100 causes the operation detection unit 1045 to detect the detection image input by the detection image input unit 103 in step S304, and the operation mode generated in step S3063 by the operation mode generation unit 1043 (for example, in FIG. (Shown in FIG. 21) and the operation is detected.

  The operation detecting device 100 according to the present embodiment has the technical effects of the first embodiment, the first embodiment, and the second embodiment as described above, and a duplicate description will be omitted here.

(Second embodiment)
Hereinafter, the operation detection device 100A according to the second embodiment will be described.

(Structure of operation detection device 100A)
FIG. 23 is a diagram showing the configuration of the operation detection device 100A of the second embodiment. In FIG. 23, the same elements as those in FIG. 1, which is a diagram showing the structure of the operation detection device 100 according to the first embodiment, are designated by the same reference numerals, and a duplicate description will be omitted.

  The difference between the operation detection device 100A of the second embodiment and the operation detection device 100 of the first embodiment is that the operation detection device 100A according to the second embodiment includes an operation time detection unit 105. .

  The operation time detection unit 105 detects a start time and an end time of the operation based on the operation detected by the operation detection unit 104.

  For example, the operation time detection unit 105 specifies the start time and the end time of the operation based on the operation detected by the operation detection unit 104 based on the time information included in the detected image.

  In addition, the operation time detection unit 105, based on the operation detected by the operation detection unit 104, the operation start time and the operation start time according to the time information included in the event and / or the operation mode of the operation mode that generates the operation. Detect the end time.

(Operation of operation detection device 100A)
FIG. 24 is an operation flowchart of the operation detection device 100A of the second embodiment. In FIG. 24, the same operation as the operation shown in FIG. 3, which is the operation performed by the operation detection device 100 according to the first embodiment, is denoted by the same reference numeral, and duplicated description will be omitted.

  As shown in FIG. 24, the difference from the operation shown in FIG. 3, which is the operation performed by the operation detection device 100 of the first embodiment, is that after executing steps S300 to S306, the process should be completed. Instead, the process proceeds to step S240.

  In step S240, the operation time detection unit 105 detects the start time and end time of the operation based on the operation detected by the operation detection unit 104, and then ends the process.

  According to the operation detection device 100A of the second embodiment, the technical effects of the first embodiment and each example can be provided, and in addition to that, after detecting the operation, the operation start time and The end time can be detected. Therefore, for example, without relying on an IE engineer, information on the operation time of the operator of each station on the production line can be obtained in real time, accurately and efficiently, and the balance of the production line can be obtained in real time and accurately. It can be analyzed so that it can be adjusted in a timely manner to guarantee the production rate and the quality of the products manufactured on the production line.

  The specific expression form of the operation detection device and the operation detection method has been described above in detail, but the specific expression form is not limited to this, and the operation detection system, the integrated circuit, the program, and the program are recorded. It may be expressed in various forms such as a medium.

  FIG. 25 is a diagram showing the configuration of the operation detection system.

  As shown in FIG. 25, the operation detection system 1000 includes a training image acquisition device 200, a target detection model generation device 300, and an operation detection device 100.

  Here, since the operation detection device 100 is the above-described operation detection device 100, detailed description thereof will be omitted here. In addition, the operation detection system may of course include the operation detection device 100A.

  The training image acquisition device 200 is for acquiring a plurality of training images in which an object is imaged. The training image acquisition device 200 may be a camera that captures a moving image as a training image by capturing the operation of the operator of each station in the production line. Further, the training image acquisition device 200 has a data receiving function, which is a wired or wireless communication module having a data receiving function, which obtains a moving image as a training image by externally capturing an operation of an operator of each station in a production line via wired or wireless communication. It may be a component.

  The target detection model generation device 300 generates a target detection model based on the plurality of training images acquired by the training image acquisition device 200. The target detection model generation device 300 can generate a target detection model using the following method, for example. That is, a plurality of training images are extracted by processing a moving image that is a training image group obtained by capturing the operation of the operator of each station in the production line, and the extracted training images are marked. After marking one or more targets on each training image of the multiple training images, deep learning is performed based on the training images containing the time-series information, which are marked on the one or more targets. , Generate an object detection model (train). Here, the method of generating an object detection model by deep learning is an example, and the method is not limited to this. If an accurate object detection model can be obtained, various known methods or those that may appear in the future may appear. Any method can be used.

  The operation detection system has the technical effect of a corresponding operation detection device, depending on the operation detection device included, and will not be described repeatedly here.

  According to the operation detection device, the operation detection method, and the operation detection system provided by the present invention, the operation of each operator on the production line can be accurately detected in real time without depending on the acquisition, and the balance of the production line can be improved. It is possible to accurately detect the working time information of each operator on the production line so as to make adjustments, to detect operation abnormalities without fail to predict or detect product defects, and to operate the operator on the production line. Can contribute to improving the production line by analyzing and evaluating the operation of the operator.

(Modification)
Although the respective embodiments have been described above, the present invention is not limited to the embodiments, examples, and modified examples. For example, a person skilled in the art appropriately adds or deletes components or changes the design of each of the embodiments, examples, and modified examples described above, and appropriately combines the features of each embodiment, example, or modified example. Also, as long as it matches the technical idea of the present invention, it is included in the scope of the present invention.

  For example, in the second embodiment, the operations may be detected by applying Examples 1 to 3 of the first embodiment.

  It should be noted that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements within a range not departing from the gist of the invention in an implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, the constituent elements of different embodiments may be combined appropriately.

100 ... Operation detection device, 101 ... Target detection model acquisition unit, 102 ... Target information acquisition unit, 103 ... Detected image input unit.

Claims (9)

A target detection model acquisition unit that acquires a target detection model based on a plurality of training images,
A target information acquisition unit that acquires target information based on the plurality of training images and the target detection model acquired by the target detection model acquisition unit,
A detection image input unit for inputting a detection image for detecting the operation of the operator,
An operation detection unit that detects the operation based on the target information acquired by the target information acquisition unit and the detected image input by the detected image input unit,
An operation detection device including. The operation detection unit,
Event generating means for generating an event based on the target information acquired by the target information acquiring section;
Operation mode generation means for generating an operation mode based on the event generated by the event generation means,
An operation detection unit that detects the operation based on the detection image input by the detection image input unit and the operation mode generated by the operation mode generation unit,
The operation detection device according to claim 1, further comprising: The operation detection unit,
Event generating means for generating an event based on the target information acquired by the target information acquiring section;
Operation mode generation means for generating an operation mode based on the event generated by the event generation means,
Operation mode generation means for generating an operation mode based on the operation mode generated by the operation mode generation means,
An operation detection unit that detects the operation based on the detection image input by the detection image input unit and the operation mode generated by the operation mode generation unit,
The operation detection device according to claim 1, further comprising: The operation mode generation means,
The operation detection device according to claim 3, wherein an operation mode is generated based on the event generated by the event generation unit and the operation mode generated by the operation mode generation unit.   The operation detection device according to claim 1, further comprising an operation time detection unit that detects an operation start time and an operation end time based on the operation detected by the operation detection unit. .   The operation detection device according to claim 1, wherein the target information includes at least a target name, a target type, a target position, and a change in the target position.   The operation detection device according to claim 1, wherein the detected image is a moving image. A training image acquisition device for acquiring a plurality of training images in which an object is captured;
Based on a plurality of training images acquired by the training image acquisition device, a target detection model generation device for generating a target detection model,
An operation detection device according to any one of claims 1 to 7,
An operation detection system including. A target detection model acquisition step of acquiring a target detection model based on a plurality of training images,
A target information acquisition step of acquiring target information based on the plurality of training images and the target detection model acquired by the target detection model acquisition step;
A detection image input step of inputting a detection image for detecting the operation of the operator,
An operation detection step of detecting the operation based on the target information acquired by the target information acquisition step and the detected image input by the detected image input step;
An operation detection method comprising:

Images