JP2020022101A - Monitoring device, production line, and control method of monitoring device - Google Patents

Abstract

To simplify a work for confirming a recorded moving image, without modifying a system on the side where a production line including an automatic machine is monitored.SOLUTION: Without changing a system to be monitored, and a system on the monitored side, a moving image M shot by means of a camera 21 is recorded by means of a recorder 22. For example, a moving image processing section placed in the recorder 22 detects a repetition part of the recorded original moving image (M), and divides the original moving image (M) into partial moving images in units of the detected repetition part. Detection of the repetition part is carrier out by applying a circulation detection method of Floyd to change with time of representative values from multiple still-images clipped from the original moving image, for example. The partial moving images are formed as multiple partial moving image files, or by imparting chapter data in units of the repetition part. Specific partial moving images specified by user operation are reproduced by means of a moving image reproducer.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a monitoring device, a production line, and a control method of a monitoring device that use a camera and a recording device that records a moving image captured by the camera.

  2. Description of the Related Art A monitoring device using a camera and a recording device that records a moving image captured by the camera has been known. This type of monitoring device is used for various purposes such as monitoring of an automatic machine as a production device arranged in a production line, in addition to being used in stores, residential buildings, roads, intersections, level crossings, and the like.

  Particularly, in a production line, an automatic machine as a production device, such as a robot arm, a monitoring device that performs moving image capturing / recording is used for monitoring a supply / discharge stocker and various transport devices. For example, an automatic machine such as a robot arm, a supply / discharge stocker, or various transport devices performs a process of processing a material or a process of assembling various articles and parts thereof without manual intervention. The monitoring devices of the automatic machines continue to photograph the operation of these automatic machines in the same manner as the monitoring devices used in other applications. The monitoring device of the automatic machine is used particularly for the purpose of visually confirming the normal operation of the automatic machine or for analyzing the cause of the abnormality when an abnormality occurs in the automatic machine.

  In some applications, the camera and the recording device of the monitoring device continue to operate for a long period of time and record a long moving image. The length of the video can be several days, sometimes months, or years. When analyzing a captured moving image, an automatic analysis procedure using image processing may be applicable to such a long-time moving image, but in many cases, the user plays back the moving image and the image is captured. It is often performed by confirming (viewing) the state of the subject. In that case, there is a problem that it is difficult for the user to find a moving image that the user wants to check as the recorded moving image is enormous.

  In particular, in monitoring an automatic machine, in consideration of the above problem, a monitoring device is connected to a detection device that detects an abnormal state of a production line, and when an abnormal state is detected, control is performed so that a corresponding moving image cannot be overwritten. A configuration has been proposed (for example, Patent Document 1 below). Further, a configuration having an image capturing device that captures an image according to a signal generated by a facility device similar to an automatic machine referred to in the present specification is also known (for example, Patent Document 2 below). In this configuration, when it is determined that the facility device is necessary, moving image shooting is performed.

JP-A-2006-122319 JP-A-6-86288

  In the monitoring device of Patent Literature 1, it is necessary to connect a detection device that detects an abnormal state to the automatic machine. In this case, the automatic machine needs to be configured so that the detection device can be installed. Alternatively, if the detection device is installed in an automatic machine that is already operating, mechanical or electrical remodeling may be required in some cases. Such a change may not be allowed depending on the specifications and operating conditions of the automatic machine. Further, in the monitoring device of Patent Document 2, the automatic device and the monitoring device of the automatic device must be electrically connected so that a control signal can be transmitted and received. In addition, it is necessary to install control software for a monitoring device in an automatic machine. Similar to the above, such a change may not be allowed depending on the specifications and operating state of the automatic machine.

  An object of the present invention is to make it possible to easily check a recorded moving image without modifying a monitored system, for example, a production line in which an automatic machine is arranged, such as a production line. It is in.

  In order to solve the above problem, in the present invention, a camera, a recording device that records a moving image captured by the camera, and detects a repeated portion from the original video captured by the camera, the detected repeated portion as a unit And a moving image processing unit that divides the original moving image into partial moving images.

  According to the above configuration, a recorded moving image can be easily divided into partial moving images for a long time, and a monitored system, for example, a monitored system such as a production line in which an automatic machine is arranged is modified. The user can easily check the recorded moving image without any need.

It is explanatory drawing which showed the automatic machine and the monitoring apparatus of the automatic machine. It is an explanatory view showing a control system of an automatic machine and a monitoring device of the automatic machine. It is explanatory drawing which showed the mode that an automatic machine operates. It is explanatory drawing which showed the mode that an automatic machine operates. It is explanatory drawing which showed the mode that an automatic machine operates. It is explanatory drawing which showed the mode that an automatic machine operates. It is explanatory drawing which showed the mode that an automatic machine operates. FIG. 4 is an explanatory diagram showing a frame in which the state of FIG. 3 is photographed by a camera. FIG. 5 is an explanatory diagram showing a frame obtained by photographing the state of FIG. 4 with a camera. It is explanatory drawing which showed the frame which image | photographed the state of FIG. 5 with the camera. FIG. 7 is an explanatory diagram showing a frame in which the state of FIG. 6 is photographed by a camera. It is explanatory drawing which showed the frame which image | photographed the state of FIG. 7 with the camera. FIG. 4 is an explanatory diagram showing a technique for dividing a moving image using a deductive algorithm. FIG. 9 is an explanatory diagram showing a divided partial moving image group and a state of reproduction thereof. (A)-(c) is explanatory drawing which showed the technique of dividing a moving image using a learning system and a discriminator. FIG. 3 is a block diagram illustrating an example of a specific configuration of a control device that can be used for moving image division.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings. The configuration described below is merely an example, and for example, a detailed configuration can be appropriately changed by those skilled in the art without departing from the spirit of the present invention. Also, the numerical values taken up in the present embodiment are reference numerical values and do not limit the present invention.

  The monitoring device for an automatic machine according to the present embodiment captures and records the state of a production operation performed by the automatic machine arranged as a production device on a production line. The recorded moving image can be suitably used for visual recognition of the normal operation of the automatic machine and for analysis of the cause of the abnormality when an abnormality occurs in the automatic machine. However, the technology of the present invention is not necessarily limitedly applied to a monitoring device that targets an automatic machine, and can be implemented even if another monitoring target is other than an automatic machine or a production line. Things.

  FIG. 1 shows an automatic machine 1 arranged together with a monitoring device 2 in an article production line. The monitoring device 2 includes a camera 21 and a recorder 22 as a recording device for a moving image captured by the camera 21, and monitors the surroundings of the automatic machine 1 with the camera 21 by capturing a moving image. The shooting frame rate of a moving image captured by the camera 21 is arbitrary, and the technology of the present embodiment can be implemented even in a case where the frame rate is appropriate for intermittent shooting of a still image in some cases.

  In FIG. 1, the automatic machine 1 includes a PLC (programmable logic controller) 11, a robot 12, a supply stocker 13, a discharge stocker 14, and a work table 15 arranged on a base plate 16. A part A and a part B for manufacturing an article are placed on the supply stocker 13. The PLC 11 controls the operations of the robot 12, the supply stocker 13, and the discharge stocker 14 to perform an operation of manufacturing an article from the component A and the component B.

  The robot 12 has a configuration having a plurality of movable parts and a drive system, and includes, for example, a vertical articulated robot arm as shown and a hand 121 as an end effector mounted on the tip of the robot arm. The robot 12 can operate each movable part by each drive system in response to a command from the PLC 11, and can take a position and a posture according to the purpose.

  The hand 121 attached to the tip of the robot 12 constitutes a gripper capable of gripping the components A and B by a finger method, an air suction method, or the like.

  The supply stocker 13 and the discharge stocker 14 are also a part of the automatic machine 1 and have a plurality of movable parts and a drive system. The supply stocker 13 is driven based on a command from the PLC 11 so as to make it easier for the robot 12 to take the placed component group, or to be in a position and orientation for receiving the supply of the component group from outside the automatic machine. The discharge stocker 14 is driven based on a command from the PLC 11 so that the robot 12 can easily place a component group that has been operated in some way, or a position and orientation for discharging the component group to the outside of the automatic machine.

  The work table 15 is used when the robot 12 temporarily places the acquired component group and performs operations such as processing and assembly. The robot 12, the supply stocker 13, the discharge stocker 14, and the worktable 15 are fixed at predetermined positions on a surface plate 16, and a desirable positional relationship is maintained. Specifications such as rigidity and levelness of the surface plate 16 are appropriately determined according to the performance and specifications required for the automatic machine 1.

  The monitoring device 2 in FIG. 1 includes a camera 21 and a recorder 22. The camera 21 is electrically connected to the recorder 22 by a control signal or the like. From the recorder 22 to the camera 21, a signal or the like for instructing shooting is transmitted. The captured moving image M (FIG. 2) is transmitted from the camera 21 to the recorder 22.

  The user arranges the camera 21 so as to include the robot 12, the supply stocker 13, the discharge stocker 14, and the worktable 15 in the field of view. The arrangement of the camera 21 may be farther than the illustrated position, and the camera 21 may be fixed to the surface plate 16 or the robot 12. Further, in FIG. 1, the camera 21 is arranged using a camera mount like a tripod. However, the camera 21 may be arranged in such a way as to be mounted on a ceiling, and the automatic machine 1 may be monitored from above.

  The camera 21 photographs the operation of the robot 12, the supply stocker 13, the discharge stocker 14, the worktable 15, and the components A and B, and the captured moving image M (FIG. 2) is recorded on the recorder 22.

  FIG. 2 shows a connection relationship between the automatic machine and a monitoring device of the automatic machine. It should be noted that, in the present embodiment, for example, the PLC 11 and the recorder 22 are not connected, and the monitoring device 2 and the automatic machine 1 to be monitored are not connected.

  The PLC 11 and the robot 12 in the automatic machine 1 are connected via a robot driver 122. Similarly, the supply stocker 13 is provided with a supply stocker driver 131 and the discharge stocker 14 is provided with a discharge stocker driver 141 for connection with the PLC 11. In general, the output signal of the PLC 11 is not high voltage and large current enough to directly drive a robot or a stocker, and each of these drivers has a function of driving signal timing and power amplification. Although each of these drivers is simply shown by a block diagram, it may have a configuration of a control device disposed between the PLC 11 and each of the driven parts. For example, in the case of the robot 12, a robot controller may be arranged between the PLC 11 and the robot 12, but the function of the robot driver 122 of the present embodiment can be realized by the robot controller. This is the same for the driver of the supply / discharge stocker.

  As described above, the camera 21 and the recorder 22 of the monitoring device 2 are connected to each other. However, in the present embodiment, the camera 21 and the recorder 22 do not need to be connected to the system of the automatic machine 1 to be monitored. It is not necessary to mutually transmit and receive an abnormality detection signal and a kind of synchronization signal.

  3 to 7 show how the automatic machine 1 operates, and FIGS. 8 to 12 show moving image frames shot by the camera 21 in the states of FIGS. 3 to 7, respectively. Hereinafter, first, an example of a manufacturing process performed by the automatic machine 1 will be described with reference to FIGS. The operation of the automatic machine 1 exemplified here is to assemble the parts A and B on the worktable 15 and discharge the parts A and B to the discharge stocker 14.

  FIG. 3 shows a first step performed by the automatic machine 1. The first step is an operation of gripping one of the components A on the supply stocker 13. Here, the robot 12 and the supply stocker 13 are cooperatively driven by a command from the PLC 11, and the component A on the supply stocker 13 is gripped by the hand 121.

  FIG. 4 shows a second step performed by the automatic machine 1. In the second step, the component A is transferred to the work table 15 and placed. In response to a command from the PLC 11, the robot 12 transfers the hand 121 and the component A held by the hand 121 to the worktable 15. When the hand 121 releases the gripping of the component A, the component A is placed on the worktable 15.

  FIG. 5 shows a third step performed by the automatic machine 1. In the third step, an operation of gripping one of the components B on the supply stocker 13 is performed. Here, the robot 12 and the supply stocker 13 are cooperatively driven by a command from the PLC 11, and the hand 121 grips the component B.

  FIG. 6 shows a fourth step performed by the automatic machine 1. This fourth step corresponds to assembling of the component B to the component A. Here, the robot 12 places the hand 121 and the component B gripped by the hand 121 on the component A on the work table 15 and assembles them according to a command from the PLC 11. The mutual positioning of the components A and B is performed, for example, via a fitting portion (not shown in detail) such as a claw provided on these components. When the positioning of the components A and B is completed, the hand 121 releases the grip of the component B.

  After that, the robot 12 fixes and assembles the part B to the part A by mechanical joining means such as screw fastening and bonding. At this time, if necessary, the robot 12 may perform an operation of automatically exchanging the hand 121 with another hand having a screw suction or fastening function. An article (part A + B) manufactured by assembling the part B to the part A is gripped by the hand 121.

  FIG. 7 shows a fifth step performed by the automatic machine 1. In the fifth step, an operation of placing the manufactured article, that is, the part A + B on the discharge stocker 14 is performed. Here, in response to a command from the PLC 11, the robot 12 performs an operation of placing the hand 121 and the components A + B held by the hand 121 on the discharge stocker 14. The hand 121 releases the gripping of the components A + B on the discharge stocker 14, whereby the manufactured articles (components A + B) are transferred to the discharge stocker 14.

  The automatic machine 1 repeats the operations of the first to fifth steps shown in FIGS. 4 to 7 to produce articles (parts A + B) one after another. In this case, the automatic machine 1 continuously receives the supply of the components A and B to the supply stocker 13 from the outside, and discharges the components A + B to the outside via the discharge stocker 14.

  Next, referring to FIGS. 8 to 12, the moving image frames captured by the camera 21 in the states of FIGS. 3 to 7 will be described. When the automatic machine 1 captures images of executing the first to fifth steps as shown in FIGS. 3 to 7 from the camera 21, a moving image M (FIG. 2) including frames as shown in FIGS. Will be recorded.

  8 to 12, only one frame image is shown for convenience of illustration. However, in the actual moving image M, the operations between FIGS. 8 to 9, 9 to 10, 10 to 11, and 11 to 12 are of course recorded as many moving image frames. is there. Also, if the automatic machine 1 repeatedly executes one set of the first to fifth steps (FIGS. 3 to 7), the moving image M continuously recording the same will be changed from the state of FIG. Images during the transition to the state are also included.

  The moving image M also includes a state in which the automatic machine 1 is not operating, depending on the shooting timing of the monitoring device 2 of the automatic machine. Alternatively, when an abnormality occurs in the automatic machine 1, the moving image M may include an image of a state of the automatic machine 1 different from any of the above.

  However, the monitoring device 2 of the present embodiment records moving images for a sufficiently long time than the first to fifth steps (FIGS. 3 to 7) repeated many times. In this case, the moving image M includes a repetition portion in which the automatic machine 1 of the subject repeats the operation of FIGS. 8 to 9 to 10 to 11 to 12.

  In the present embodiment, a repeated portion is detected from the recorded original moving image (M) recorded in the recorder 22 as a recording device without interruption for a long time, and the original moving image (M) is set in units of the detected repeated portion. Is divided into partial videos. The repetition portion of the original moving image (M) is a portion of the original moving image (M) in which a subject that repeats a specific operation is photographed.

  The detection of the repetition part of the original moving image (M) and the division processing into the partial moving image are performed by the moving image processing unit. The moving image processing unit may be a unit that processes the moving image M recorded in the recorder 22 by providing an independent stand-alone image processing device. A configuration having a function may be used.

  An image processing device different from the recorder 22 that realizes the moving image processing unit or the recorder 22 having the function of the moving image processing unit can be configured as shown in FIG. 16, for example. Hereinafter, a case where the control system of FIG. 16 is configured as the recorder 22 will be described as an example.

  FIG. 16 illustrates an example of a configuration of an image processing apparatus as a moving image processing unit that detects a repeated portion of the original moving image (M) and performs a division process into partial moving images. The control system in FIG. 16 can be constituted by PC hardware including a CPU 601 as main control means, a ROM 602 as a storage device, and a RAM 603. The ROM 602 can store a control program of the CPU 601 and constant information for realizing a manufacturing procedure described later. The RAM 603 is used as a work area of the CPU 601 when executing the control procedure.

  An external storage device 606 is connected to the control system of FIG. The external storage device 606 includes an HDD, an SSD, and an external storage device of another system mounted on a network. For example, the moving image M (FIG. 2) can be recorded (recorded) in the external storage device 606.

  A control program of the CPU 601 for realizing the control procedure (for example, FIG. 8) of the present embodiment can be stored in the above-described external storage device 606 or a storage unit such as the ROM 602 (for example, an EEPROM area). . In this case, a control program of the CPU 601 for realizing a control procedure described later can be supplied to each of the above-described storage units via the network interface 607 and updated to a new (another) program. Alternatively, a control program of the CPU 601 for realizing a control procedure to be described later is supplied to each of the above storage units via various storage means such as a magnetic disk, an optical disk, and a flash memory, and a drive device therefor. Also, the contents can be updated. Various storage units, storage units, or storage devices in a state in which the control program of the CPU 601 for realizing the above control procedure is stored constitute a computer-readable recording medium storing the control procedure of the present invention. Become.

  When the control system in FIG. 16 is configured as the recorder 22, the camera 21 is connected to the CPU 601 via, for example, a network interface 607 and a network 608. The network interface 607 can be configured using a communication standard based on wired communication such as IEEE 802.3 and wireless communication such as IEEE 802.11 and 802.15. The network interface 607 can also be used for communication with a supervisory control device such as a PLC 11 arranged for production control and management on a production line, a management server, and the like.

  In the monitoring device of FIG. 16, an operation unit 604 and a display device 605 are connected as a UI device (user interface device). The operation unit 604 can be configured by a terminal such as a handy terminal, a device such as a keyboard, a jog dial, and a pointing device (or a control terminal including the same). As the display device 605, a display device of any type can be used as long as it can display and output a data display described later or a display screen similar thereto, in addition to a liquid crystal type, for example.

  Several methods are conceivable for the detection of the repeated portion of the original moving image (M) and the division process into the partial moving image performed by the moving image processing unit configured as described above.

  FIG. 13 shows a technique for dividing a moving image using a deductive algorithm. The left side of FIG. 13 is a flowchart showing the flow of the processing procedure, and the right side shows the data being handled. In the present embodiment, a technique for dividing a moving image using a deductive algorithm as shown in FIG. 13 is referred to as a deductive technique. The illustrated control procedure can be stored as a control program of the CPU 601 in, for example, the ROM 602 or the external storage device 606 described above.

  In the deduction method of FIG. 13, step S51 indicates a process in which the camera 21 and the recorder 22 record the moving image M. It is assumed that the moving image M includes at least two or more times that the automatic machine 1 repeatedly executes the process.

  Next, in step S52, a luminance change 521 is calculated as a representative value from the moving image M acquired in step S51. In this step S52, for example, the moving image M is handled as a still image group. For all the still images of the moving image M, the average value of the luminance values of all the pixels of each still image, that is, the average luminance is calculated. When the average luminance as the representative value is listed in the time direction, a temporal change 521 in the representative value (average luminance) of each still image is obtained.

  Next, in step S53, a noise filter is applied to the luminance change 521 obtained in step S52, and a filtered luminance change 531 is obtained. In the noise filter processing in step S53, a change that is too steep or a change that is too slow is removed from the luminance change 521. Factors that cause the change to be too steep are assumed to be noise mixed into the camera 21, a case where the user is reflected in the shooting range of the camera 21, and the like. In addition, as a factor that causes a change that is too slow, it is assumed that the automatic machine 1 is exposed to outdoor environmental light. The type and order of the filter applied in the noise filter processing in step S53 may be determined by trial and error, for example, by performing an experiment in advance so that remarkable repetition can be detected in the luminance change 531 after the filter.

  In step S54, the luminance and time resolutions of the luminance change 531 after the filtering are reduced, and the luminance change 541 after the resolution reduction is obtained. By reducing the resolution in this way, the accuracy of the Floyd circulation detection method used for the circulation detection (Step S55 described below) in the present embodiment is improved.

  Next, in step S55, the Floyd circulation detection method is applied to the luminance change 541 after the resolution reduction acquired in step S54. As is well known, the Floyd circulation detection method is an algorithm for detecting a repeated sequence from an arbitrary sequence. According to the Floyd circulation detection method, the repeated portion R can be detected from the luminance change 541 after the resolution is reduced. This repetition portion R corresponds to the portion of FIGS. 8 to 9 to 10 to 11 (FIG. 8) in the previous moving image frame.

  However, if the Floyd circulation detection method is applied at random, the initial value of the circulation may not be uniquely determined. For example, there is a possibility that the portions shown in FIGS. 9 to 10 to 11 to 12 to 8 (to 9) are detected as the repeated portion R. As a countermeasure, it is conceivable to adjust the timing of the start of operation of the automatic machine 1 and the start of monitoring of the monitoring device 2. It is not always necessary to use a synchronization signal for this timing adjustment. For example, it is possible to use a means such that the operation of the automatic machine 1 and the monitoring of the monitoring device 2 are started at the same time. However, it goes without saying that a synchronization signal may be used. In this case, for example, the automatic machine 1 and the monitoring device 2 may transmit and receive the synchronization signal via the network 608 in FIG. By such timing adjustment, the initial elements of repetition included in the moving image M can be specified as shown in FIG. 8, and the repetition portion R is determined as the repetition part R by the Floyd circulation detection method in FIGS. The portion shown in FIG. 12 (to FIG. 8) can be detected.

  Further, the automatic machine 1 may not be able to carry out the intended process for some reason. This is the case where, for example, after executing one cycle corresponding to FIGS. 8 to 9 to 10 to 11 (to FIG. 8), an emergency stop is performed due to some abnormal operation. In such a case, the repetition portion R cannot be detected in the process of step S55. When the repeated portion R cannot be detected as described above, the process is terminated without executing the moving image division (S56). Alternatively, the recorder 22 may perform control to set the moving image M to be undeletable. As described above, in the case where the repeated portion R is not detected, the recorder 22 controls the moving image M so that the moving image M cannot be deleted, so that the user may be able to easily analyze the inconvenience of the automatic machine 1 after the fact.

  The success or failure of the detection of the repetition part R is determined by, for example, allowing the user to input a rough required time of one cycle of the process, and determining whether the required time of the repetition part R is substantially the same as that input by the user. It may be performed by determining whether

  Next, when the detection of the repeated portion R is successful, step S56 is executed. This step S56 is a process of dividing the moving image M at the boundary of the repeated portion R acquired in step S55, and obtaining partial moving images corresponding to the repeated portions R respectively. Each moving image in the moving image group is a partial moving image for one cycle performed by the automatic machine 1, corresponding to FIGS. 8 to 9 to 10 to 11 (to FIG. 8).

  FIG. 14 illustrates an example of a user interface suitable for the user to confirm (view) the divided partial moving image. When the recorder 22 has a configuration as shown in FIG. 16, for example, a GUI (Graphical User Interface) as shown in FIG. 14 can be configured by the display device 605 (and the operation unit 604). However, the GUI as shown in FIG. 14 may be implemented by a device that is connectable to the recorder 22 and is not the recorder 22.

  In the GUI shown in FIG. 14, the file manager 61 is activated and the recording folder 62 is opened. The file manager 61 may use a file manager used in an OS having various GUIs. In the recording folder 62, a partial moving image group 63 divided by the above-described deduction method is stored. If the file name of the partial moving image group 63 includes information on the shooting time of the head of the moving image, for example, as shown in FIG. In FIG. 14, the metaphor (GUI display) of the partial moving image group 63 is a display of, for example, a file name by a character string. However, the partial moving image group 63 may be displayed using an icon display or a combination of an icon and a file name.

  When one of the moving images in the partial moving image group 63 is clicked, the moving image reproducing unit 64 is activated. For the moving image reproducing unit 64, a moving image reproducing application used in an OS having various GUIs may be used. A procedure in which an application capable of handling the file is started when the metaphor of the file is clicked on the file manager 61 in this manner is a method generally used in OSs having various GUIs.

  Since each partial moving image of the partial moving image group 63 is a partial moving image corresponding to the repeated portion R as described above, the reproduction of the moving image 65 starts from the image of FIG. become. The reproduction of the moving image 65 continues to each frame of FIGS. 9 to 10 to 11 to 12 and ends again at or immediately before FIG.

  At this time, it is assumed that the partial moving image group 63 includes a moving image of a process of manufacturing a component A + B determined to be defective in a subsequent process of the automatic machine 1. Further, it is assumed that the execution time of the process of the automatic machine 1 that manufactured the part A + B can be acquired. If division into partial moving images has already been completed in units of repetitive operations, it is extremely easy to select and reproduce a moving image having time information close to the execution time via a GUI as shown in FIG. . Note that it is not impossible to specify the time and reproduce the original moving image (M) recorded for a long time from that position. However, according to the GUI as shown in FIG. 14, in addition to the one-cycle partial moving image in question, one or several other one-cycle partial moving images before and after it can be designated and immediately reproduced. In addition, it is extremely easy to simultaneously arrange and reproduce the partial moving image in another cycle such as the one before and after the one in addition to the partial moving image in question.

  The serial number is described on the manufactured part A + B, and the execution time of the process of the automatic machine 1 that manufactured the part can be obtained from the serial number, or the manufacturing time can be directly stamped. Production lines are not uncommon. If there is such an agreement, it is easy to obtain the execution time of the process of the automatic machine 1 that manufactured the part.

  According to the GUI as shown in FIG. 14, the user can reproduce a partial moving image having time information close to the execution time in the partial moving image group 63. Therefore, for example, if it is desired to confirm the process of a specific one cycle in which the component A + B is manufactured, the corresponding partial moving image can be easily reproduced from the start of the process. By checking the reproduced partial moving image, the user can easily confirm the cause of the defect such as the gripping of the component by the robot 12 being displaced or the screw fastening of the assembly being insufficient.

  As described above, according to the monitoring apparatus for an automatic machine of the present embodiment, a repetitive portion is detected by automatic processing from a long-time original moving image captured without taking a picture (recording), and the detected repetitive portion is defined as a unit. The original moving image can be divided into partial moving images. For this reason, the user can easily find a moving image that he / she wants to watch without performing a dedicated design or remodeling of the automatic machine to be monitored, and can appropriately support the user's work of analyzing the monitored object.

  In the above description, the process of dividing the original moving image M into partial moving images in units of the detected repetition portion is generated as partial moving image files of different files as shown in the example of the file manager in FIG. I thought that. However, instead of generating a partial moving image file of another file, a method of generating a partial moving image by generating chapter data in units of a repetition part detected from the original moving image M may be used. According to this configuration, there is a possibility that the storage capacity of the external storage device 606 can be saved as compared with a method of storing a partial moving image file of another file in addition to the original moving image M.

  FIGS. 15A to 15C show a method of detecting a repeated portion of an original moving image and dividing the moving image using a learning system and a discriminator. A method of dividing a moving image using such learning control is hereinafter referred to as a learning method. In the learning method of the present embodiment, for example, a method of reinforcement learning control such as supervised learning is used. As a specific software architecture, a configuration similar to an artificial neural network can be used. In this type of reinforcement learning, a Q learning table or the like for calculating a validity score of discrimination may be used.

  In the learning method of the present embodiment, for example, at the same time as the photographing by the camera 21, for example, the leading part of the process is specified by using a still image photographing a state corresponding to the leading state of one cycle of a repeated part. To detect a repeated portion.

  That is, in the learning method of the present embodiment, the specific state of the subject photographed in the original moving image is changed from the still image cut out from the original moving image to the specific state through learning control using a group of still images photographed. Identify the corresponding specific still image. Then, based on the position of the identified specific still image in the original moving image, a repeated portion of the original moving image is detected, and the original moving image is divided into the partial moving images in units of the detected repeated portion.

  FIG. 15A corresponds to the first process of the learning method. Step S711 in FIG. 15A is a process of configuring a classifier capable of determining a start image of a series of steps of the automatic machine 1 as an artificial neural network.

  FIG. 15B shows the configuration of the discriminator 721 created in step S711 of FIG. 15A and the input / output relationship between the teacher still image group 722 and the teacher output group 723 used in step S711.

  The discriminator 721 can be configured as an artificial neural network. The discriminator 721 includes, for example, an input layer 721i, an intermediate layer 721m, and an output layer 721o. For example, when a still image is input to the input layer 721i, the output layer 721o outputs yes or no information to the teacher output group 723 through abstract information processing by the middle layer 721m. The network shape of the discriminator 721 is designed artificially according to the required performance. Note that the network configuration of the discriminator 721 in FIG. 15B is an example, and another network configuration may be used.

  The teacher still image group 722 is a still image group including at least the images of FIGS. 8 to 12 cut out from the moving image M captured by the camera 21 of the automatic machine 1 in advance. When acquiring the teacher still image group 722, there is a possibility that a highly robust discriminator 721 can be acquired by photographing while changing the conditions of light and components in various ways. In addition, it is necessary to prepare a sufficient number of teacher still image groups 722 to acquire the desired classifier 721.

  The teacher output group 723 allows the user to look at the teacher still image group 722 in advance and, for example, answer yes for an image similar to the image in FIG. 8 at the beginning of the process and answer no for other images. This is data that accumulates results. Of course, the beginning of the process does not necessarily need to be the image of FIG. 8, and the teacher output group 723 may be created using the images of FIGS.

  In the learning method of FIG. 15, the discriminator 721 is trained using the teacher still image group 722 and the teacher output group 723 prior to the actual monitoring process. Specifically, regardless of which still image of the teacher still image group 722 is input, the weight of the discriminator 721 related to the network connection is operated until a value equal to the teacher output group 723 is output. In this learning step, a technique such as back propagation or an auto encoder can be used as appropriate. Through such a learning process, the discriminator 721 can identify whether a new image that is not included in the teacher still image group 722 is an image similar to, for example, the target image in FIG.

  FIG. 15C shows a second process of the learning method, which includes a procedure for sequentially detecting the head of a repetitive portion being executed by a subject being photographed in synchronization with the photographing of a moving image. In the process of FIG. 15C, the partial moving image is divided, or at least the division position is specified, by sequentially detecting the beginning of the repetitive portion using the discriminator 721 created in the first process (learning step). I do. Note that the process of FIG. 15C is described as a loop process of photographing and recording one partial moving image corresponding to one repeated portion.

  First, in step S731, the camera 21 and the recorder 22 start capturing a partial moving image. In this learning method, it is possible to execute the processing required for dividing the moving image without waiting for a long time until the end of capturing the entire moving image.

  Next, in step S732, one almost latest still image is extracted from the moving image being captured. This still image is hereinafter referred to as a still image I.

  Subsequently, in step S733, the discriminator 721 is applied to the still image I to determine whether or not the still image I is an image similar to the head of a repeated part of one cycle. If it is not determined in step S733 that the still image I is similar to FIG. 8, the process returns to step S732, in which case, for example, a captured image is not recorded.

  If it is determined in step S733 that the still image I is similar to the image in FIG. 8, the shooting of the partial moving image ends in step S734. In step S734, the shooting of the moving image M ends, but the process immediately returns to step S731 to start shooting of the subsequent partial moving image. In the process of FIG. 15, the partial moving image may be recorded as a partial moving image file, or the partial moving image may be defined by generating chapter data.

  In the learning method described above, unlike the above-described deduction method, it is not necessary for the moving image M to include the state in which the automatic machine 1 performs all the processes twice or more. Here, it is assumed that a retry step intended to return from a failure in the execution is incorporated in any of the steps performed by the automatic machine 1. The retry step is an operation in which, for some reason, when the robot 12 fails to grip the component A, the robot 12 again attempts to grip the component A. The deduction method corresponds to a retry process in which the repetition of the luminance change is largely broken, and it is not easy to realize the desired moving image division. On the other hand, in the learning method, even when the automatic machine 1 performs the retry process, the head of the repeated part can be specified by analogy with the state of the head of the process included in the prepared teacher image. Is likely to be generated,

  The user can also select and reproduce the partial moving image group obtained by the learning method in FIG. 15 using the same user interface as in FIG. 14 described above. Also in the present embodiment, the user can easily find a desired moving image and can assist the user who analyzes the system to be monitored satisfactorily without making a dedicated design or modification for the automatic machine to be monitored.

  The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. Processing can also be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

  In the above description, a configuration for monitoring a production line for producing articles, particularly an automatic machine arranged on the production line has been described. However, the above-described technology of detecting a repeated portion of the original moving image and dividing the original moving image into partial moving images in units of the detected repeated portion is applied to moving images obtained by shooting various monitoring targets regardless of the monitoring target. can do.

  DESCRIPTION OF SYMBOLS 1 ... Automatic machine, 2 ... Monitoring device, 11 ... PLC, 12 ... Robot, 13 ... Supply stocker, 14 ... Discharge stocker, 15 ... Work table, 21 ... Camera, 22 ... Recorder, 601 ... CPU, 602 ... ROM, 603 ＲＡＭ RAM, 604 操作 operation unit, 605 表示 display device, 606 外部 external storage device, A, B 部品 parts.

Claims (15)

Camera and
A recording device for recording a moving image taken by the camera,
A monitoring apparatus comprising: a moving image processing unit that detects a repeated portion from an original moving image captured by the camera and divides the original moving image into partial moving images in units of the detected repeated portion.   2. The monitoring apparatus according to claim 1, wherein the repeated portion of the original moving image is a portion of the original moving image in which a subject that repeats a specific operation is captured. 3.   3. The monitoring device according to claim 1, wherein the partial moving image is generated as a plurality of partial moving image files each recording the repeated portion.   The monitoring device according to claim 1, wherein the partial moving image is generated by generating chapter data corresponding to the repeated portion of the original moving image.   The monitoring device according to any one of claims 1 to 4, further comprising: an operation unit for allowing a user to specify the specific partial moving image; and a moving image reproducing device, wherein the specific unit specified by the operation unit is provided. A monitoring device for reproducing the partial moving image by the moving image reproducing device.   6. The monitoring device according to claim 1, wherein the moving image processing unit calculates a representative value from each of a plurality of still images cut out from the original moving image, and calculates a change in the representative value over time. 7. A monitoring device that detects repeated parts of the original moving image by applying the Floyd circulation detection method.   7. The monitoring device according to claim 6, wherein the representative value is an average luminance of all pixels of the still image.   The monitoring device according to claim 1, wherein the moving image processing unit performs learning control using a group of still images obtained by capturing a specific state of a subject captured in the original moving image. Identifying a specific still image corresponding to the specific state from a still image cut out from the original moving image, and detecting a repeated portion of the original moving image based on a position of the identified specific still image in the original moving image. A monitor that divides the original moving image into the partial moving images in units of the repeated portion.   9. The monitoring device according to claim 8, wherein the learning control is performed by artificial neural network means.   9. The monitoring device according to claim 8, wherein the learning control is performed by reinforcement learning control.   11. A production line comprising the monitoring device according to any one of claims 1 to 10, wherein the production line is arranged so that the camera photographs the production unit, and the production unit repeatedly executes the production unit by the moving image processing unit. A production line on which the partial moving image on which is recorded is generated from an original moving image.   An article manufacturing method for manufacturing an article using the production equipment arranged on the production line according to claim 11. In a control method of a monitoring device including a camera and a recording device that records a moving image captured by the camera,
A control device, a detection step of detecting a repeated portion of the original moving image captured by the camera,
A moving image processing step in which the control device divides the original moving image into partial moving images in units of a repetition part detected in the detection step.   A control program for causing the control device to execute each step of the monitoring device control method according to claim 13.   A computer-readable recording medium storing the control program according to claim 14.

Images