WO2020250501A1

WO2020250501A1 - Data extraction device, control method for data extraction device, information processing program, and recording medium

Info

Publication number: WO2020250501A1
Application number: PCT/JP2020/008289
Authority: WO
Inventors: 火炎木焦; 慎也藤本
Original assignee: オムロン株式会社
Priority date: 2019-06-14
Filing date: 2020-02-28
Publication date: 2020-12-17

Abstract

The present invention efficiently uses image pickup data acquired by capturing images of the implementation states of a plurality of work steps while suppressing a communication load on a higher-level system located outside of a master-slave control system with an apparatus to be used in the work steps as a slave. An information processing device (10) uses step information acquired from a PLC (20) via a control network (50) to extract partial image pickup data (OD) from basic image pickup data (BI).

Description

Data extraction device, control method of data extraction device, information processing program, and recording medium

The present invention relates to a data extraction device or the like that extracts data that can be used for improving work processes from imaged data obtained by photographing the implementation status of a plurality of work processes.

Conventionally, there has been known an attempt to use imaging data obtained by imaging the implementation status of a work process at a production site such as a factory for improvement of the work process. In particular, there is known an attempt to extract and use the imaging data obtained by imaging the implementation status of a desired work process from the imaging data obtained by imaging the implementation status of a plurality of work processes. For example, in Patent Document 1 below, from work video data obtained by photographing the situation of a work site where a plurality of work processes are carried out by an operator, planning data related to each work process is used to perform a predetermined work process. A technique for extracting predetermined work video data related to a work process determined to be applicable is disclosed.

Japanese Patent Publication "Japanese Patent Laid-Open No. 2019-23803"

However, in the conventional technology as described above, since the image data obtained by capturing the situation of the entire work site is transmitted to the upper system outside the master-slave control system in which each device at the production site is a slave, communication to the upper system is performed. There is a problem that the load becomes very large.

That is, the prior art uses the planning data for each work process to extract the predetermined work video data related to the work process determined to correspond to the predetermined work process. Since such planning data is arranged in a higher-level system located outside the master-slave control system in which each device at the production site is a slave, the conventional technique is imaging data in which the situation of the entire work site is captured. Needs to be uploaded to the higher system. Since the amount of image data obtained by photographing the situation of the entire work site is extremely large, and in particular, the amount of data of the imaged data for detailed analysis is enormous, the image data obtained by photographing the situation of the entire work site is used as the higher-level system. The communication load when uploading to is very large.

One aspect of the present invention has been made in view of the above-mentioned problems, and while suppressing the communication load on the host system located outside the master-slave control system in which the equipment used in the work process is a slave. The purpose is to efficiently use the imaging data obtained by photographing the implementation status of a plurality of work processes.

In order to solve the above-mentioned problems, the data extraction device according to one aspect of the present invention includes a first acquisition unit that acquires basic data which is imaging data obtained by photographing the implementation status of a plurality of work processes, and the plurality of operations. The second acquisition unit that acquires the process information generated from the operation result indicating the content and result of the operation executed by the device used for each execution of the process, and the operation performed by the device using the process information. An operation determination unit that determines whether or not meets a predetermined operation standard, and imaging data indicating the implementation status of the work process corresponding to the operation determined by the operation determination unit to satisfy the predetermined operation standard. A device including an extraction unit that extracts data from the basic data, and a device connected to a master in a master-slave control system in which the device is a slave via an internal bus, or a slave and the master. It is a device different from the device connected to the master via a control network connected communicably, and the second acquisition unit acquires the information generated or collected by the master as the process information.

In order to solve the above problems, the control method of the data extraction device according to one aspect of the present invention includes a first acquisition step of acquiring basic data which is imaging data obtained by photographing the implementation status of a plurality of work processes, and the above-mentioned. The second acquisition step of acquiring process information generated from the operation result indicating the content and result of the operation executed by the device used for each execution of the plurality of work processes, and the device performing the process using the process information. An imaging showing an operation determination step for determining whether the said operation satisfies a predetermined operation standard and an execution status of the work process corresponding to the operation determined to satisfy the predetermined operation standard in the operation determination step. The data extraction device includes an extraction step of extracting data as partial data from the basic data, and the data extraction device is a device connected to a master in a master-slave control system having the device as a slave via an internal bus, or the device. A device different from the device, which is connected to the master via a control network that communicably connects the slave and the master, and the second acquisition step obtains the information generated or collected by the master. Obtained as process information.

According to one aspect of the present invention, while suppressing the communication load on the host system located outside the master-slave control system in which the device used in the work process is a slave, the imaging of the implementation status of a plurality of work processes is captured. It has the effect of being able to use the data efficiently.

It is a block diagram which shows the main part structure of the information processing apparatus and the like which concerns on Embodiment 1 of this invention. It is a figure which shows the whole outline of the control system including the information processing apparatus of FIG. It is a figure which shows the image of the basic imaging data acquired by the information processing apparatus of FIG. It is a flow diagram explaining an example of the process executed by the information processing apparatus of FIG. It is a figure which shows the method which the information processing apparatus of FIG. 1 identifies a worker and a monitoring area using the basic imaging data (image). It is a figure which shows an example of the information which shows the correspondence relation between the operation period of an apparatus and the stay period of a worker for each work process. This is a screen example showing the correspondence between the operating period of the device and the staying period of the worker for a plurality of work processes in a list format. It is a figure which shows an example of the work site where a plurality of monitoring areas are arranged corresponding to the execution order of a plurality of work processes. It is a figure which shows an example of the information which shows the movement progress of a worker in a work site. It is a figure which shows an example of the box-and-whisker plot which shows the distribution in a certain day of the stay period for each work process required to produce one certain product. It is a figure which shows an example of the box-and-whisker plot which shows the distribution of one stay period in a certain day for each of a plurality of workers belonging to a certain worker group. It is a figure which shows an example of the bar graph which shows the ratio of the total stay period corresponding to a work process to the total time of a certain day for each work process. It is a block diagram which shows the main part structure of the information processing apparatus and the like which concerns on Embodiment 2 of this invention. It is a figure which shows an example of the information which the information processing apparatus of FIG. 13 uses when performing correction with respect to the basic image pickup data (captured image). FIG. 5 is a flow chart illustrating an example of correction processing executed by the information processing apparatus of FIG. 13. It is a figure which shows an example of the flow line analysis result before execution of the correction process illustrated in FIG. It is a figure which shows an example of the flow line analysis result after execution of the correction process illustrated in FIG. It is a figure which shows an example of the analysis using the combination of the stay period and the operation time.

[Embodiment 1]
Hereinafter, embodiments according to one aspect of the present invention (hereinafter, also referred to as “the present embodiment”) will be described with reference to FIGS. 1 to 12. The same or corresponding parts in the drawings are designated by the same reference numerals, and the description thereof will not be repeated. In the present embodiment, for example, the information processing device 10 will be described as a typical example of the data extraction device. In order to facilitate understanding of the information processing device 10 according to one aspect of the present invention, first, an outline of the control system 1 and the like including the information processing device 10 will be described with reference to FIG. In the following description, "n" indicates "an integer of 1 or more", and "m" indicates "an integer of 1 or more and n or less".

§1. Application example (about work site and work process)
FIG. 2 is a diagram showing an overall outline of the control system 1 and the like including the information processing device 10. The ceiling camera 30 shown in FIG. 2 is installed on the ceiling of the work site WS and generates basic imaging data BI which is imaging data of the entire work site WS.

The work site WS is a production site such as a factory, and in the work site WS, for example, various products have a plurality of work processes Pr (1), Pr (2), Pr (3), ... Pr (n). ) To be produced. Each of the plurality of work processes Pr (1), Pr (2), Pr (3), ... Pr (n) carried out at the work site WS is, for example, a "painting" process and a "main work assembly". The process, the process of "incorporating the main work into the main body", and the process of "inspection".

When it is necessary to distinguish each of the plurality of work process Prs for the work process Pr, the symbols are "(1)", "(2)", "(3)", ..., "(N)". To distinguish by adding a subscript such as ". For example, "work process Pr (1)", "work process Pr (2)", "work process Pr (3)", ..., "Work process Pr (n)" are described to distinguish them. When it is not necessary to distinguish each of the plurality of work process Prs, it is simply referred to as "work process Pr".

(About the monitoring area)
The work site WS includes a plurality of monitoring areas Ar (1), Ar (2), Ar (3), ... Ar (n). Each of the plurality of monitoring areas Ar (1), Ar (2), Ar (3), ... Ar (n) has a plurality of work steps Pr (1), Pr (2), Pr (3), ... .. It is associated with each of Pr (n). That is, the monitoring area Ar (m) is an area in which the worker Pe performs the work Op (m) related to the execution of the work process Pr (m), for example, in the area where the device 40 (m) is arranged. is there. For example, the worker Pe performs the work Op (m) related to the work process Pr (m) in the monitoring area Ar (m) using the device 40 (m).

Similar to the work process Pr, when it is necessary to distinguish each of the plurality of monitoring areas Ar, the symbols are "(1)", "(2)", "(3)", ..., "( It is distinguished by adding a subscript such as "n)", and when it is not necessary to distinguish it, it is simply referred to as "monitoring area Ar".

(About basic imaging data)
A ceiling camera 30, which is a wide area imaging camera, is installed on the ceiling of the WS at the work site. However, it is not essential that the ceiling camera 30 is installed on the ceiling of the work site WS, and the ceiling camera 30 may be installed at a position where the entire work site WS can be overlooked. The ceiling camera 30 gives a bird's-eye view of the entire work site WS and generates basic imaging data BI which is imaging data of the entire work site WS. The basic imaging data BI includes a plurality of analysis target regions Aa (1), Aa (1), each of which corresponds to a plurality of monitoring regions Ar (1), Ar (2), Ar (3), ... Ar (n). 2), Aa (3), ... Aa (n) are preset.

Similar to the monitoring area Ar, when it is necessary to distinguish each of the plurality of analysis target areas Aa, the codes are "(1)", "(2)", "(3)", ..., " (N) ”and other subscripts are added to distinguish them, and when it is not necessary to distinguish them, they are simply referred to as“ analysis target area Aa ”.

The analysis target area Aa is the target area for image analysis of the basic imaging data BI by the information processing device 10. By setting the analysis target area Aa in the basic imaging data BI, it is possible to efficiently execute the image analysis for the basic imaging data BI for recognizing the status of the monitoring area Ar. However, it is not essential to set the analysis target area Aa in the basic imaging data BI. The analysis target area Aa may be set for the basic imaging data BI by the information processing apparatus 10 according to the user operation.

(About work and workers)
When each of the plurality of work steps Pr (1), Pr (2), Pr (3), ... Pr (n) is carried out, a plurality of work Ops (1) and Op (2) are performed by the worker Pe. , Op (3), ... Op (n) are performed respectively. Each of the plurality of work steps Pr (1), Pr (2), Pr (3), ... Pr (n) includes a plurality of standard work Ops (1), Op (2), Op (3). ), ... Each of Op (n) is associated in advance. However, among the work Ops performed by the worker Pe, there may be work Ops other than the "standard work Op previously associated with the work process Pr", that is, the work op cannot be associated with the work process Pr in advance. , Non-standard work Op is also possible. When it is necessary to distinguish each of multiple work Ops, add subscripts such as "(1)", "(2)", "(3)", ..., "(n)" to the code. When it is not necessary to distinguish them, they are simply referred to as "work Op".

Further, at the work site WS, for example, there is one or more worker Pe who executes the work Op (m) related to the execution of the work process Pr (m), and the worker Pe is covered by, for example, the worker Pe. It is identified by the worker ID attached to the crown of the hat. Specifically, the worker Pe (1) and the worker Pe (2) existing in the work site WS are the worker ID: Pe (1) attached to the hat worn by the worker Pe (1). Each is identified by the worker ID: Pe (2) attached to the hat worn by the worker Pe (2). When it is necessary to distinguish each of a plurality of worker Pes, subscripts such as "(1)", "(2)", "(3)", ..., "(n)" are added to the code. When it is not necessary to distinguish them, they are simply referred to as "worker Pe".

FIG. 3 is a diagram showing an image of basic imaging data BI acquired by the information processing device 10 from the ceiling camera 30. As shown in FIG. 3, the information processing apparatus 10 executes image analysis on the basic imaging data BI, determines whether or not the worker Pe exists in the work site WS, and the worker Pe is in the work site WS. If it is determined that it exists, the worker ID of the worker Pe existing in the work site WS is specified.

(About operation and equipment)
In carrying out each of the plurality of work steps Pr (1), Pr (2), Pr (3), ... Pr (n), a plurality of operations Ac (1), Ac (2), depending on the device 40. Each of Ac (3), ... Ac (n) is executed. That is, each of the plurality of work processes Pr (1), Pr (2), Pr (3), ... Pr (n) has a plurality of operations Ac (1), Ac (2), Ac (3). , ... Each of Ac (n) is associated with each other. When it is necessary to distinguish each of the plurality of operation Acs, add subscripts such as "(1)", "(2)", "(3)", ..., "(n)" to the code. When it is not necessary to distinguish them, they are simply referred to as "operation Ac".

Further, each of the plurality of devices 40 is used for each execution of the plurality of work process Pr, that is, the work process Pr and the device 40 are associated in advance. For example, one or more devices 40 (m) are used to carry out the work process Pr (m). That is, one or more devices 40 (1) are used to carry out the work process Pr (1), and specifically, the devices 40 (1-1), 40 (1-2), and 40 (1). The three devices 40 (1) of -3) may be used. Similarly, one or more devices 40 (2) are used to carry out the work process Pr (2), and one or more devices 40 (3) are used to carry out the work process Pr (3). When it is necessary to distinguish between the plurality of devices 40, each of which is associated with each of the plurality of work process Prs, the symbols "(1)", "(2)", "(3)", ..., A subscript such as "(n)" is added to distinguish them, and when it is not necessary to distinguish them, they are simply referred to as "device 40".

Here, one device 40 may be used to carry out a plurality of work steps Pr. This means that the variables "p", "q", "x", and "y" each indicate an "integer of 1 or more", "q" is different from "p", and "y" is "x". It can be paraphrased as follows. That is, even if the device 40 (px) used to carry out the work process Pr (p) and the device 40 (qy) used to carry out the work process Pr (q) are the same device 40. Good.

(About master-slave control system)
The plurality of devices 40 used to carry out the plurality of work process Prs at the work site WS are controlled by the PLC (Programmable Logic Controller) 20 as a line controller. That is, a control system 1 is constructed as a master-slave control system in which the PLC 20 is used as a master and each of the plurality of devices 40 is a slave, and each of the plurality of devices 40 can communicate with the PLC 20 via a network (control network 50). It is connected to the. The PLC 20 is called a "master" in the sense that it manages data transmission via the control network 50. The “master” and “slave” are defined by paying attention to the control function of data transmission on the control network 50, and what kind of information is transmitted and received between the devices is not particularly limited.

The PLC 20 is a control device (controller) that controls the entire control system 1, and is communicably connected to each of a plurality of devices 40. The PLC 20 acquires information from each of the plurality of devices 40 as input devices (measurement devices) as input data. The PLC 20 executes arithmetic processing using the acquired input data according to a user program incorporated in advance. The PLC 20 executes the arithmetic processing to determine the control content for the control system 1, for example, determines the control content for each of the plurality of devices 40 as output devices such as actuators, and corresponds to the control content. The control data is output to each of the plurality of devices 40. The PLC 20 repeatedly executes the acquisition of input data from each of the plurality of devices 40 and the acquisition of control data to each of the plurality of devices 40 in a predetermined cycle (control cycle). For example, a display unit and an operation unit (not shown) may be connected to the PLC 20. The display unit is composed of a liquid crystal panel or the like capable of displaying an image, and the operation unit is typically composed of a touch panel, a keyboard, a mouse or the like.

The device 40 is a slave in the control system 1 as a master-slave control system with the PLC 20 as the master. The device 40 is an input device that repeatedly transmits input data to the PLC 20 at predetermined control cycles, or receives repeated control data from the PLC 20 at predetermined control cycles and operates according to the received control data. It is a device. The device 40 may be, for example, a sensor (for example, a photoelectric sensor) as an input device for transmitting a detection result or the like to the PLC 20 as input data, or a bar code reader for transmitting a reading result, and is inspected. It may be an inspection machine (tester) that transmits the results. Further, the device 40 may be a PT (Programmable Terminal) to which a plurality of input devices are connected. Further, the device 40 may be a robot or the like as an output device that executes screw tightening, picking, and the like.

The control network 50 transmits various data received by or transmitted by the PLC 20 and can typically use various Industrial Ethernet®, sometimes referred to as a field network. .. As industrial Ethernet (registered trademark), for example, EtherCAT (registered trademark), Profile IRT, MECHATROLINK (registered trademark) -III, Powerlink, SERCOS (registered trademark) -III, CIP Motion and the like are known. Any of these may be adopted. Further, a field network other than Industrial Ethernet (registered trademark) may be used. For example, if motion control is not performed, DeviceNet, CompoNet / IP (registered trademark), or the like may be used.

In the following, the control system 1 in which data is transmitted and received between the PLC 20 (master) and the device 40 (slave) by sequentially transferring data frames on the control network 50 will be described. That is, by sequentially transferring data frames on the control network 50 in a predetermined control cycle, data is repeatedly transmitted and received between the PLC 20 and the device 40 in each control cycle. By sequentially transferring data frames on the control network 50, data may be transmitted and received between a plurality of devices 40, that is, between a plurality of slaves.

(About process information)
In the control system 1 which is a master-slave control system in which the device 40 is a slave, the master PLC 20 is repeated, for example, at predetermined control cycles, and the content and result of the operation Ac executed by the device 40 from the slave device 40. The operation result La indicating the above is received. That is, the device 40 repeats the operation result La indicating the content and result of the operation Ac actually executed when the work process Pr is executed at a predetermined cycle, and transmits the operation result La to the PLC 20. For example, the device 40 (m) repeats the operation result La (m) indicating the content and result of the operation Ac (m) executed when the work process Pr (m) is executed in the control cycle, and transmits the operation result La (m) to the PLC 20. To do.

The PLC 20 acquires, for example, a measurement result that is the result of a measurement operation executed by the device 40 as an input device (measurement device) as an operation result La of the device 40. Further, when the device 40 is an inspection machine, the PLC 20 sets the result of the inspection operation executed by the device 40 as, for example, the inspection result such as "satisfied or did not meet the inspection standard" as the operation result of the device 40. Obtained as La. Further, the PLC 20 acquires, for example, the result of the output operation executed by the device 40 as the output device as the operation result La of the device 40. When the device 40 is a robot that executes screw tightening, picking, etc., the PLC 20 acquires an operation result La such as a screw tightening number and a picking result (picking success or picking error) as an operation result La of the device 40.

The PLC 20 repeatedly receives an operation result La indicating the content and result of the operation Ac actually executed by the device 40 from the device 40 at a predetermined cycle, and uses the received operation result La as process information. It transmits (that is, transfers) to the information processing device 10. Further, the PLC 20 transmits information generated by using the operation result La repeatedly received from the device 40 at a predetermined cycle to the information processing apparatus 10 as process information.

Further, the PLC 20 may transmit the operation result La repeatedly received from the device 40 at a predetermined cycle to the outside of the control system 1 as process information. For example, the PLC 20 connects the operation result La repeatedly received from the device 40 in a predetermined cycle to a MES (Manufacturin g Execution System) or the like as process information, and is an in-house LAN (Local Area Network) shown in FIG. ) May be sent.

In the following description, the determination for the operation result La (particularly the operation Ac) included in the process information (for example, the presence / absence of an error and the determination of the presence / absence of execution completion within the range of the standard operation time Tta) is determined from the PLC 20. An example of the information processing apparatus 10 that has received the process information will be described. Here, "determination of whether or not execution is completed within the range of standard operating time Tta" means whether or not the execution of operation Ac is completed between the standard lower limit operating time lower limit Ttal and the standard upper limit operating time upper limit Ttah. Is to be determined.

Details will be described later, but in the example described below, the information processing apparatus 10 operates from the process information, in particular, from the operation result La, the operation start time Tms of the operation Ac executed by the device 40 when the work process Pr is executed, and the operation. Specify the completion time Tme and the operation period Da. Then, the information processing apparatus 10 executes the determination for the operation Ac by using the specified operation start time Tms, the operation completion time Tme, the operation period Da, and the predetermined operation reference Sa corresponding to the operation Ac.

However, the above-mentioned determination for the operation result La (particularly, the operation Ac) included in the process information may be executed by the PLC 20, and the PLC 20 includes the result of the above-mentioned determination in the process information or the process information. Instead of, the information may be transmitted to the information processing apparatus 10.

In the above description, the operation start time Tms is the time when the device 40 used for the work process Pr starts the execution of the operation Ac when the work process Pr is executed, and the operation completion time Tme is the operation completion time Tmes of the operation Ac. It is the time when the execution is completed. The operation period Da is a period from the operation start time Tms to the operation completion time Tme.

(About systems and devices other than master-slave control systems)
FIG. 2 shows an in-house LAN system, another network system, and the like, in addition to the control system 1 as a master-slave control system. The in-house LAN is connected to a process information DB (Database), which is also called MES. In the process information DB, information indicating "standard operation to be executed by the device 40 used to carry out the work process Pr" is stored as the operation reference Sa. Although the details will be described later, the operation reference Sa includes the execution order information Si, the error reference Te, and the standard operating time Tta (specifically, the standard lower limit operating time lower limit Ttal and the standard upper limit operating time upper limit Ttah). .. In the following description, among "actual operation Ac performed by the device 40 used to carry out the work process Pr", operations other than "standard operation to be performed by the device 40 used to carry out the work process Pr". Is referred to as an operation Ac that "satisfies the operation standard Sa". In other words, the operation Ac that "satisfies the operation standard Sa" is "an operation that the device 40 used to carry out the work process Pr does not normally perform".

Further, in the example shown in FIG. 2, an event management device 60 that monitors and manages various events that occur at the work site WS is connected to the process information DB as the MES via the in-house LAN. However, it is not essential that the event management device 60 is connected to the process information DB via the in-house LAN, and the event management device 60 may not be provided.

Further, the PLC 20 is connected to the process information DB via the in-house LAN. Although not shown, the process information DB and the information processing device 10 may be connected to each other. Further, in addition to the MES, an ERP (Enterprise Resources Planning), a WMS (Warehouse Management System), or the like (not shown) may be connected to the in-house LAN.

In FIG. 2, a video storage server or the like is connected to the process information DB via an "other network" that is different from the control network 50 and the in-house LAN. The information processing device 10 is connected to the moving image storage server or the like via another network, and the partial imaging data OD transmitted from the information processing device 10 is stored in the moving image storage server or the like. Further, an external device 70 realized by a PC (Personal Computer) or the like is connected to the moving image storage server or the like, and the external device 70 displays, for example, partial imaging data OD and visualizes process information or the like. That is, the external device 70 displays a list of information necessary for improving the work process Pr, and performs partial imaging of information indicating the bottleneck work process Pr, the date and time of the error that occurred in the work process Pr, and the like. Displayed in association with data OD.

As described above, the ceiling camera 30 captures the entire work site WS to generate basic imaging data BI, and the generated basic imaging data BI is, for example, via a communication cable such as a USB (Universal Serial Bus) cable. , Is transmitted to the information processing device 10.

The information processing device 10 is a data extraction device realized by, for example, a PC or the like, which combines the process information acquired from the PLC 20 and the basic imaging data BI acquired from the ceiling camera 30 to enable efficient use of both. Is. The information processing device 10 can be said to be a device that visualizes process information including "operation result La indicating the contents and results of actual operation Acs of a plurality of devices 40 at the work site WS" by combining with the basic imaging data BI.

In the conventional technique, although image analysis may be performed on the image data obtained by capturing the "work Op performed by the worker Pe at the work site WS", it is associated with the process information of a plurality of devices 40 at the work site WS. There is no such thing, and the accuracy of estimation regarding work Op by image analysis is low. Further, in the conventional technique of arranging sensors in each of a plurality of work process Prs to visualize the process information of the device 40, the bottleneck of the work process Pr can be visualized, but the bottleneck is associated with the work Op performed by the worker Pe. It is difficult to understand the cause of the neck. Further, when the image data obtained by imaging the "work Op performed by the worker Pe at the work site WS" and the process information of the device 40 are associated with each other by a system higher than the master-slave control system, such a higher system is used. It is very large and not easy to realize.

On the other hand, the information processing apparatus 10 acquires process information including the operation result La indicating the content and result of the operation Ac of the device 40 used for carrying out the work process Pr from the PLC 20, and obtains the acquired process information as 1. It is combined with the basic imaging data BI acquired from the ceiling camera 30 of the table. The information processing device 10 combines the process information including the operation result La showing the contents and results of each operation Ac of the plurality of devices 40 with the basic imaging data BI that images the entire work site WS to perform a plurality of operations. The analysis related to each of the steps Pr can be performed efficiently and precisely.

That is, in addition to visualizing the process information, the information processing apparatus 10 extracts, for example, a bottleneck work process Pr, and captures an image of the implementation status of the extracted work process Pr (partial imaging data OD). And the process information of the extracted work process Pr are combined. Therefore, for example, the user can easily identify the cause of the bottleneck and the cause of the defect in the work process Pr that is the bottleneck, and can efficiently perform the improvement work of the work process Pr. become able to. In addition, the information processing device 10 can also be used for traceability when a defect occurs.

Further, the information processing apparatus 10 improves the analysis system for both the analysis for the basic imaging data BI and the analysis for the process information by combining the basic imaging data BI and the process information of each of the plurality of devices 40. Can be done.

Here, in order to realize the conventional technique as described in Patent Document 1 mentioned as the background technique for the information processing apparatus 10, it is necessary to introduce a very expensive device into the entire work site WS. It was. On the other hand, the information processing apparatus 10 uses the PLC 20 that has been used in the work site WS for a long time and the ceiling camera 30 that is installed on the ceiling of the work site WS, so that the realization cost is low. Further, the information processing device 10 may use the ceiling camera 30 that captures only the desired work process Pr (that is, the desired monitoring area Ar), or the ceiling camera 30 used by the information processing device 10 may be relocated. Is also easy.

Further, the technique of Patent Document 1 calculates the incapacity time, which is the time when the worker could not perform the work, by analyzing the work video data taken by a plurality of cameras. The technique of Patent Document 1 is to subtract the incapacity time calculated by analyzing the work video data from the apparent work time measured based on the operator's operation on the work instruction terminal that indicates the work procedure to the worker. Then, the actual work time that was actually possible to work is calculated. That is, the technique of Patent Document 1 relates to the device 40 used to carry out the work process Pr in order to grasp the correspondence between the actual operation Ac of the device 40 used to carry out the work process Pr and the worker Pe. Specifically, an operation instruction of the worker Pe is required for the work instruction terminal.

On the other hand, the information processing apparatus 10 uses the process information, specifically, the operation result La of the device 40 to set the operation start time Tms at which the device 40 starts the operation Ac and the operation Ac. Specify the completed operation completion time Tme. That is, the information processing device 10 specifies the operation start time Tms and the operation completion time Tme of the device 40 without requiring the operation work of the worker Pe to the work instruction terminal that indicates the work procedure to the worker Pe.

Specifically, the information processing device 10 is communicably connected to the PLC 20 via a control network 50 that connects the master PLC 20 and the slave device 40. The information processing apparatus 10 receives process information from the PLC 20 including the content and the operation result La indicating the content and result of the “actual operation Ac executed by the device 40 used to carry out the work process Pr”. Here, the information processing device 10 needs to be communicably connected to the PLC 20 via the control network 50, but the information processing device 10 is "a master-slave whose master is the PLC 20 and whose slave is the device 40." It does not have to be a slave in the "control system".

Further, the information processing device 10 is communicably connected to the ceiling camera 30 via, for example, a USB (Universal Serial Bus) cable. The information processing device 10 acquires the basic imaging data BI in which the entire work site WS is imaged from the ceiling camera 30.

Further, the information processing apparatus 10 acquires an operation standard Sa indicating "a standard operation to be executed by the device 40 used for executing the work process Pr" from the process information DB. Then, the information processing apparatus 10 uses the acquired operation reference Sa to perform a determination on "the actual operation Ac executed by the device 40 used to carry out the work process Pr". The information processing apparatus 10 corresponds to the work process Pr corresponding to the operation Ac (m) determined to "satisfy the operation standard Sa (m)", that is, "usually not executed when the work process Pr (m) is executed". The imaging data Id (m) that images the implementation status of (m) is extracted from the basic imaging data BI as partial imaging data OD.

Therefore, since the information processing device 10 does not transmit the basic imaging data BI to the in-house LAN connected to the MES or the like, the communication in the in-house LAN is not overwhelmed by the transmission of the basic imaging data BI having a huge amount of data. ..

§2. Configuration Example So far, the outline of the control system 1 and the like has been described with reference to FIGS. 2 and 3. Before explaining the details of the information processing apparatus 10 with reference to FIG. 1 and the like, the outline of the information processing apparatus 10 is organized as follows in order to facilitate the understanding of the information processing apparatus 10.

That is, the information processing device 10 (data extraction device) includes a first acquisition unit 110 that acquires basic imaging data BI (basic data), which is imaging data obtained by photographing the implementation status of a plurality of working steps Pr, and a plurality of working steps. The second acquisition unit 120 that acquires the process information generated from the operation result La indicating the content and result of the operation Ac executed by the device 40 used for each execution of Pr, and the device 40 using the process information. The execution status of the operation determination unit 130 that determines whether the executed operation Ac satisfies the predetermined operation standard Sa, and the work process Pr corresponding to the operation Ac that is determined by the operation determination unit 130 to satisfy the predetermined operation standard Sa. An extraction unit 150 that extracts the indicated imaging data Id from the basic imaging data BI as partial imaging data OD (partial data) is provided. The information processing device 10 is connected to a PLC 20 which is a master in a master-slave control system in which the device 40 is a slave via an internal bus, or a control network 50 which connects the device 40 and the PLC 20 in a communicable manner. It is a device different from the device 40 connected to the PLC 20. In the information processing apparatus 10, the second acquisition unit 120 acquires the information generated or collected by the PLC 20 as the process information.

According to the above configuration, the information processing device 10 is a device connected to the PLC 20 via an internal bus or a device connected to the PLC 20 via the control network 50, which is different from the device 40, and has the following processing. To execute. That is, the information processing apparatus 10 uses the process information generated or collected by the PLC 20 to perform the work process Pr corresponding to the operation Ac determined to satisfy the predetermined operation reference Sa from the basic imaging data BI. The imaging data Id indicating the above is extracted.

That is, the information processing device 10 acquires the process information from the PLC 20 via the internal bus or the control network 50. Then, the information processing apparatus 10 uses the acquired process information to obtain the desired imaging data Id from the basic imaging data BI without transmitting the basic imaging data BI to the outside of the control system 1 which is the master-slave control system. Is extracted as partial imaging data OD.

Therefore, the information processing device 10 extracts the partial imaging data OD from the basic imaging data BI without imposing a communication load on the host system (for example, the in-house LAN in FIG. 2) located outside the master-slave control system. , The effect is that the basic imaging data BI can be used efficiently.

The information processing device 10 further includes a transmission unit 160 that transmits the imaging data Id extracted as the partial imaging data OD by the extraction unit 150 to the outside of the master-slave control system.

According to the above configuration, the information processing apparatus 10 transmits the imaging data Id extracted as the partial imaging data OD from the basic imaging data BI to the outside of the master-slave control system. Here, it is clear that the data amount of the imaging data Id extracted as the partial imaging data OD is smaller than the data amount of the basic imaging data BI.

Therefore, the information processing device 10 has the effect of enabling efficient use of the basic imaging data BI in the higher-level system while suppressing the communication load on the higher-level system located outside the master-slave control system. ..

The information processing apparatus 10 whose outline has been described so far will be described next with reference to FIG. 1 for details of its configuration, and then the processing executed by the information processing apparatus 10 will be described with reference to FIG. To go.

(Details of information processing device)
FIG. 1 is a block diagram showing a main configuration of an information processing device 10 and the like included in the control system 1. As shown in FIG. 1, the information processing apparatus 10 has a first acquisition unit 110, a second acquisition unit 120, an operation determination unit 130, a storage unit 140, an extraction unit 150, and a transmission unit 160 as functional blocks. And an image analysis unit 170. The information processing device 10 may include an operation reception unit or the like that accepts user operations in addition to the above-mentioned functional blocks, but in order to ensure the simplicity of the description, the information processing device 10 is not directly related to the present embodiment. Is omitted from the description and block diagram. However, the information processing apparatus 10 may have the omitted configuration in accordance with the actual situation of implementation.

In the first acquisition unit 110, the second acquisition unit 120, the operation determination unit 130, the extraction unit 150, the transmission unit 160, and the image analysis unit 170, for example, the CPU (central processing unit) or the like has a ROM (read only memory). , NVRAM (non-Volatile random access memory) or the like can be realized by reading a program stored in a storage device (storage unit 140) into a RAM (random access memory) or the like (not shown) and executing the program. First, the first acquisition unit 110, the second acquisition unit 120, the operation determination unit 130, the extraction unit 150, the transmission unit 160, and the image analysis unit 170 in the information processing apparatus 10 will be described.

The first acquisition unit 110 acquires the basic imaging data BI, which is the imaging data obtained by the ceiling camera 30 capturing the entire work site WS, from the ceiling camera 30, and outputs the acquired basic imaging data BI to the extraction unit 150. To do.

The second acquisition unit 120 acquires the process information from the PLC 20 and outputs the acquired process information to the operation determination unit 130. The process information is "the" operation result La of the device 40 "acquired by the PLC 20 from the device 40" and "the information generated by the PLC 20 using the acquired" operation result La of the device 40 "". That is, the process information includes the operation result La, and the operation result La is information indicating the content and result of the operation Ac actually executed by the device 40 when the work process Pr is executed. When the PLC 20 uses the "operation result La (m) of the device 40 (m)" to execute the operation determination for the operation Ac (m) executed by the device 40 (m), the process information is based on the PLC 20. The determination result of the operation determination for the operation Ac (m) may be included.

Using the process information acquired from the second acquisition unit 120, the operation determination unit 130 asks "whether the predetermined operation standard Sa is satisfied" for the actual operation Ac executed by the device 40 used for executing the work process Pr. Judgment (operation judgment). That is, when the operation determination unit 130 acquires the process information (particularly, the operation result La) from the second acquisition unit 120, the operation reference unit 130 sets the predetermined operation reference Sa corresponding to the operation Ac included in the process information as the operation reference of the storage unit 140. Obtained by referring to Table 141. Then, the operation determination unit 130 determines whether or not the operation Ac included in the process information satisfies the predetermined operation reference Sa stored in the operation reference table 141.

For example, the operation determination unit 130 indicates that "an error (in-process error or inter-process error) has occurred" or "the operation period Da required for execution is not within the standard operation time Tta" for a certain operation Ac. ", It is determined that the certain operation Ac satisfies the operation reference Sa. Specifically, when the operation determination unit 130 confirms that "the operation period Da is less than the standard lower limit operation time lower limit Ttal" or "the operation period Da exceeds the standard upper limit operation time upper limit Ttah", there is a certain amount. It is determined that the operation Ac satisfies the operation reference Sa.

The operation determination unit 130 specifies the work process Pr (m) corresponding to the operation Ac (m) for the operation Ac (m) determined to satisfy the operation reference Sa (m), and the specified work process Pr (m). ) Is notified to the extraction unit 150 (extraction target information).

The operation determination unit 130 sets, for example, an operation start time Tms (m) which is "a time when the device 40 (m) starts" an operation Ac (m) determined to satisfy the operation reference Sa (m) ". , Specifically, from the process information, from the operation result La (m) of the device 40 (m). Further, the operation determination unit 130 is, for example, an operation completion time Tme (m) which is "a time when the device 40 (m) completes the" operation Ac (m) determined to satisfy the operation reference Sa (m) "". ) Is specified from the process information, specifically, from the operation result La (m) of the device 40 (m). The operation determination unit 130 sets the operation start time Tms (m) and the operation completion time Tme (m) specified for the “operation Ac (m) determined to satisfy the operation reference Sa (m)” to the “operation reference Sa (m)”. ) Is notified to the extraction unit 150 as extraction target information indicating the work process Pr (m) corresponding to the operation Ac (m) determined to satisfy. The time (period) from the operation start time Tms (m) to the operation completion time Tme (m) of the "operation Ac (m) determined to satisfy the operation standard Sa (m)" is also referred to as "cutting time". That is, the information indicating the "cutting time" is an example of the extraction target information. The time (period) from the operation start time Tms (m) of the operation Ac (m) to the operation completion time Tme (m) means "the period during which the operation Ac (m) is executed", and the operation period Da ( Also called m).

The operation determination unit 130 specifies the work process Pr (m) corresponding to the operation Ac (m) for the operation Ac (m) determined not to satisfy the operation reference Sa (m), and the specified work process Pr ( The image analysis unit 170 is notified of the information (non-extraction target information) indicating m). That is, the operation determination unit 130 indicates the work process Pr (m) corresponding to the operation Ac (m) that "no error occurred" and "execution was completed within the range of the standard operation time Tta (m)". The information (non-extracted target information) is notified to the image analysis unit 170. For example, the operation determination unit 130 confirms that "the operation period Da is within the range from the standard lower limit operation time lower limit Ttal to the standard upper limit operation time upper limit Ttah", and the work process Pr corresponding to the operation Ac (m) ( The image analysis unit 170 is notified of the information indicating m).

Here, the operation determination unit 130 notifies the image analysis unit 170 of the extraction target information, so that the image analysis unit 170 is subjected to operations corresponding to the operation Ac other than the “operation Ac determined to satisfy the operation reference Sa”. The process Pr may be specified. For example, the motion determination unit 130 may cause the image analysis unit 170 to specify a time other than the “cutting time” by notifying the image analysis unit 170 of information indicating the “cutting time”.

The operation determination unit 130 determines the operation start time Tms (m), the operation completion time Tme (m), and the operation period Da (m) of the operation Ac (m) specified by using the operation result La (m). You may communicate with the image analysis unit 170. Further, the motion determination unit 130 determines whether or not the motion Ac (m) is an "automated motion" by using the motion result La (m), and communicates the determination result to the image analysis unit 170. You may.

The image analysis unit 170 performs work performed by the worker Pe in the work process Pr for the work process Pr corresponding to the operation Ac other than "the operation Ac determined by the operation determination unit 130 to" satisfy the operation reference Sa "". It is determined whether the Op is a standard work (standard work). Specifically, the image analysis unit 170 first captures an image of the implementation status of the work process Pr corresponding to the operation Ac other than "the operation Ac determined by the operation determination unit 130 to" satisfy the operation reference Sa "". Image analysis is executed for the data ID. Then, the image analysis unit 170 uses the result of the image analysis and the process information to determine whether the work Op performed by the worker Pe is a standard work.

The image analysis unit 170 executes a flow line analysis capable of precisely grasping the existence and movement of the worker Pe as an image analysis for the image capture data ID (basic image data BI), and is the center of the worker Pe. Coordinates, worker ID, etc. may be detected.

The image analysis unit 170 identifies the work process Pr corresponding to the work Op corresponding to the work Op determined to be non-standard work (non-standard work), and information indicating the specified work process Pr (extraction target information). Is notified to the extraction unit 150.

The extraction unit 150 acquires the extraction target information indicating the work process Pr corresponding to the “operation Ac determined to satisfy the operation standard Sa” from the operation determination unit 130, and uses the acquired extraction target information to perform basic imaging. Partial imaging data OD is extracted from the data BI. That is, the extraction unit 150 uses the extraction target information acquired from the operation determination unit 130 to perform the work process Pr (m) corresponding to the “operation Ac (m) determined to satisfy the operation reference Sa (m)”. The imaging data Id (m) that images the implementation status is extracted from the basic imaging data BI as partial imaging data OD. For example, the extraction unit 150 is from the operation start time Tms (m) to the operation completion time Tme (m) of the “operation Ac (m) determined to satisfy the operation reference Sa (m)” from the operation determination unit 130. Get the time (that is, the cut time). Then, the extraction unit 150 extracts the imaging data Id corresponding to the acquired cutting time from the basic imaging data BI as the partial imaging data OD.

Further, the extraction unit 150 acquires the extraction target information indicating the work process Pr corresponding to the “work Op determined to be non-standard work (non-standard work)” from the image analysis unit 170, and extracts the acquired information. Partial imaging data OD is extracted from the basic imaging data BI using the target information.

The extraction unit 150 outputs the partial imaging data OD extracted from the basic imaging data BI, that is, the imaging data Id extracted as the partial imaging data OD from the basic imaging data BI to the transmission unit 160.

The transmission unit 160 transmits the imaging data Id extracted as the partial imaging data OD from the basic imaging data BI by the extraction unit 150 to the outside of the control system 1, and transmits it to, for example, the moving image storage server shown in FIG.

The transmission unit 160 may also transmit data indicating the distribution of at least one of the stay period Do and the operation period Da in a predetermined period to, for example, an external display device (for example, the external device 70 in FIG. 2). Good. Then, the transmission unit 160 may display the distribution of at least one of the stay period Do and the operation period Da in a predetermined period on an external display device. Details will be described later with reference to FIGS. 10 to 12.

The storage unit 140 is a storage device that stores various data used by the information processing device 10. The storage unit 140 includes (1) a control program, (2) an OS program, (3) an application program for executing various functions of the information processing device 10, and (4). Various data to be read when the application program is executed may be stored non-temporarily. The data (1) to (4) above may be, for example, ROM (read only memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (registered trademark) (Electrically EPROM), HDD (Hard Disc Drive), or the like. It is stored in a non-volatile storage device. The information processing device 10 may include a temporary storage unit (not shown). The temporary storage unit is a so-called working memory that temporarily stores data used for calculation and calculation results in the process of various processes executed by the information processing device 10, and is volatile storage such as RAM (Random Access Memory). It consists of devices. Which data is stored in which storage device is appropriately determined from the purpose of use, convenience, cost, physical restrictions, and the like of the information processing device 10. The storage unit 140 further stores the operation reference table 141 and the analysis target area table 142.

The operation standard table 141 stores the operation standard Sa corresponding to the "operation Ac executed by the device 40 used to carry out the work process Pr". For example, the operation reference Sa (m) is associated with the operation Ac (m) executed by the device 40 (m) used to carry out the work process Pr (m) in advance, and the operation reference table 141 shows. The operation reference Sa (m) associated with the operation Ac (m) is stored.

The operation standard Sa stored in the operation standard table 141 may be acquired from the process information DB or the like shown in FIG. 2, or may be set and stored by the user. For example, the information processing apparatus 10 acquires the operation reference Sa directly from the process information DB shown in FIG. 2 or from the process information DB via the PLC 20, and stores the acquired operation reference Sa in the operation reference table 141. May be good. The operation reference Sa stored in the operation reference table 141 includes the execution order information Si, the error reference Te, and the standard operation time Tta (specifically, the standard lower limit operation time lower limit Ttal and the standard upper limit operation time upper limit Ttah). ,including.

The execution order information Si is information indicating in what order the plurality of work process Prs should be executed for the plurality of work process Prs, that is, a plurality of operations corresponding to the plurality of work process Prs. Information indicating the order in which Ac should be executed.

The execution order information Si (m) is, for example, "The work process Pr (m) should be executed after the execution of the work process Pr (m-1) is completed and before the start of the implementation of the work process Pr (m + 1). ". Here, since the work process Pr and the operation Ac are associated with each other, the execution order information Si (m) states that "the operation Ac (m) is the operation Ac (m) after the execution of the operation Ac (m-1) is completed. It can be said that it should be executed before the start of execution of m + 1). "

The error standard Te is a standard set in advance for each "operation Ac executed by the device 40 used to carry out the work process Pr", and specifically, the "device 40 used to carry out the work process Pr" is used. Information indicating "contents and results of standard operations to be performed". For example, "work process Pr (m): equipment 40 (m) used to carry out the inspection process: operation Ac (m) performed by the inspection machine: inspection operation" includes "error criterion Te: inspection result-good". Is preset. Further, in "work process Pr (m): equipment 40 (m) used for performing the picking process: operation Ac (m) executed by the picking machine: picking operation", "error criterion Te: picking success" is set in advance. It is set. Further, "Work process Pr (m): Equipment used for carrying out the screw tightening process 40 (m): Operation executed by the screw tightening machine Ac (m): Screw tightening operation" includes "Error reference Te: Rotation speed". Is from p to q (> p) times ”is preset.

That is, in the error reference Te, there is an abnormality in the operation result La that the device 40 used for carrying out a certain work process Pr detects an abnormality and the operation Ac of the device 40 used for carrying out a certain work process Pr. This is a standard for detecting at least one of the operation result La, which is the result of the operation.

The standard operation time Tta is information indicating "a period normally required from the start to the completion of the operation Ac to be executed by the device 40 when used to carry out the work process Pr". More specifically, the standard operating time Tta is the period from the standard lower limit operating time lower limit Ttal to the standard upper limit operating time upper limit Ttah. The standard lower limit operating time lower limit Ttal is the lower limit of "the period normally required from the start to the completion of the operation Ac to be executed by the device 40 when used to carry out the work process Pr". Further, the standard upper limit operation time upper limit Ttah is an upper limit of "a period normally required from the start to the completion of the operation Ac to be executed by the device 40 when used to carry out the work process Pr".

For example, the standard operation time Tta (m) is associated with the work process Pr (m), that is, the operation Ac (m) corresponding to the work process Pr (m). Specifically, the standard lower limit operating time lower limit Ttal (m) and the standard upper limit operating time upper limit Ttah (m) are associated with the operation Ac (m).

This is because "the device 40 (m) used to carry out the work process Pr (m) executes the operation Ac (m) within the standard operation time Tta (m) when the work process Pr (m) is carried out. It should be completed. " That is, it is shown that the device 40 (m) should complete the execution of the operation Ac (m) within the range from the standard lower limit operating time lower limit Ttal (m) to the standard upper limit operating time upper limit Ttah (m). There is.

The above-mentioned execution order information Si, error reference Te, and standard operating time Tta (specifically, standard lower limit operating time lower limit Ttal and standard upper limit operating time upper limit Ttah), which are operation reference Sa, are steps that are MES. It may be stored in the information DB. In that case, the information processing apparatus 10 may store the operation reference Sa acquired from the process information DB in the operation reference table 141.

The analysis target area table 142 stores a plurality of analysis target areas Aa, each of which corresponds to each of the plurality of monitoring areas Ar. For example, an analysis target area Aa for confirming the status of the monitoring area Ar (m) corresponding to the work process Pr (m) is set for the basic imaging data BI, and the analysis target area table 142 shows the work process Pr. The monitoring area Ar (m) associated with (m) is stored. Each of the plurality of analysis target areas Aa stored in the analysis target area table 142 may be updated according to the user operation.

§3. Operation Example FIG. 4 is a flow diagram showing an example of a process executed by the information processing device 10 (in other words, a control method executed by the information processing device 10). In order to facilitate the understanding of the processing shown in FIG. 4, the outline is as follows. That is, the processing (control method) executed by the information processing apparatus 10 includes the first acquisition step (S120) of acquiring the basic imaging data BI (basic data) which is the imaging data obtained by photographing the execution status of the plurality of work steps Pr. The second acquisition step (S110) for acquiring the process information generated from the operation result La indicating the content and result of the operation Ac executed by the device 40 used for each execution of the plurality of work process Pr, and the process information. When the operation determination step (S130, S140, and S190) for determining whether the operation Ac executed by the device 40 satisfies the predetermined operation reference Sa and the operation determination step Satisfy the predetermined operation reference Sa are satisfied. The extraction step (S170) of extracting the imaging data Id indicating the implementation status of the work process Pr corresponding to the determined operation Ac from the basic imaging data BI as the partial imaging data OD is included. Here, the information processing device 10 is a device connected to the PLC 20 which is a master in a master-slave control system in which the device 40 is a slave via an internal bus, or a control network 50 which connects the device 40 and the PLC 20 so as to be communicable. It is a device different from the device 40, which is connected to the PLC 20 via the device 40. Then, in the second acquisition step, the information generated or collected by the PLC 20 is acquired as the process information.

According to the above configuration, the control method controls the information processing device 10 connected to the PLC 20 via the internal bus, or the information processing device 10 connected to the PLC 20 via the control network 50, which is different from the device 40. It is a method and executes the following processing. That is, the control method uses the process information generated or collected by the PLC 20 to determine the implementation status of the work process Pr corresponding to the operation Ac determined to satisfy the predetermined operation reference Sa from the basic imaging data BI. The imaged data Id shown is extracted.

That is, the control method is executed by the information processing device 10 that acquires the process information from the PLC 20 via the internal bus or the control network 50. Then, the control method uses the acquired process information to obtain the desired imaging data Id from the basic imaging data BI without transmitting the basic imaging data BI to the outside of the control system 1 which is a master-slave control system. It is extracted as partial imaging data OD.

Therefore, the control method extracts the partial imaging data OD from the basic imaging data BI without imposing a communication load on the host system (for example, the in-house LAN in FIG. 2) located outside the master-slave control system. It has the effect that the basic imaging data BI can be used efficiently.

Next, the details of the processing executed by the information processing apparatus 10 whose outline has been described above will be described with reference to FIG. As shown in FIG. 4, first, the second acquisition unit 120 acquires the process information from the PLC 20 (S110). Specifically, the second acquisition unit 120 is the actual operation Ac (m) executed by the device 40 (m) used for carrying out the work process Pr (m) during the execution of the work process Pr (m). The operation result La (m) indicating the content and the result of the above is acquired from the PLC 20 as process information. Further, the first acquisition unit 110 acquires basic imaging data from the ceiling camera 30 (S120).

(Judgment for process information)
The operation determination unit 130 uses the process information acquired from the PLC 20 by the second acquisition unit 120 to ask "whether the predetermined operation standard Sa is satisfied" for the operation Ac executed by the device 40 when the work process Pr is executed. judge. Specifically, the operation determination unit 130 uses the operation result La (m) included in the process information to perform the operation Ac (m) actually executed by the device 40 (m) when the work process Pr (m) is executed. ) Satisfies the operation standard Sa (m). "

(S130: Presence or absence of process error)
As a determination of "whether the operation Ac (m) satisfies the operation reference Sa (m)", the operation determination unit 130 uses the operation result La (m) to first perform the "work process Pr (m)". Is there a process error in the operation Ac (m) executed by the device 40 (m)? ”(S130). That is, the operation determination unit 130 determines "whether there is an in-process error or an inter-process error in the operation Ac (m) actually executed by the device 40 (m) when the work process Pr (m) is executed".

(S130-1: Judgment of in-process error)
The operation determination unit 130 uses the operation result La (m) indicating the content and result of the actual operation Ac (m) executed by the device 40 (m) when the work process Pr (m) is executed, and uses the operation result La (m) to perform "operation Ac (m). Whether an in-process error occurred in m) ”is determined.

As described above, the error reference Te (m) is set in advance for the operation Ac (m). The operation determination unit 130 acquires the error reference Te (m) by referring to the operation reference table 141 stored in the storage unit 140. Then, the operation determination unit 130 compares the acquired error reference Te (m) with the actual operation Ac (m) executed by the device 40 (m) when the work process Pr (m) is executed, and "operates". Whether an in-process error has occurred in Ac (m) ”is determined. When the operation determination unit 130 determines that "an in-process error has occurred" for the operation Ac (m), it determines that "the operation Ac (m) satisfies the operation reference Sa (m)".

In carrying out the work process Pr (m), a plurality of operations Ac (m) (1), Ac (m) (2), Ac (m) (3), ..., Ac (m) (x) are executed. The same applies when it is done. For example, the actual operations Ac (m) performed by each of the devices 40 (m) (1), 40 (m) (2), and 40 (m) (3) used to carry out the work process Pr (m). ) (1), Ac (m) (2), and Ac (m) (3), the operation determination unit 130 executes the following determination. That is, the operation determination unit 130 uses the operation Ac (m) (1), Ac (m) (2), and Ac (m) (3), and the error criteria Te (m) (1), Te ( Compare each of m) (2) and Te (m) (3). The error reference Te (m) (1) is an error reference Te preset in the operation Ac (m) (1), and the error reference Te (m) (2) is set in the operation Ac (m) (2). , The error reference Te (m) (3) is an error reference Te preset in the operation Ac (m) (3).

When it is determined that "the operation Ac (m) (1) satisfies the error reference Te (m) (1)", the operation determination unit 130 determines that "the operation Ac (m) (1) is the operation reference Sa (m) (1). 1) is satisfied. " Further, when it is determined that "the operation Ac (m) (2) satisfies the error reference Te (m) (2)", the operation determination unit 130 determines that "the operation Ac (m) (2) is the operation reference Sa (m)". ) (2) is satisfied. " Further, when it is determined that "the operation Ac (m) (3) satisfies the error reference Te (m) (3)", the operation determination unit 130 determines that "the operation Ac (m) (3) is the operation reference Sa (m)". ) (3) is satisfied. "

For example, "work process Pr (m): equipment 40 (m) used to carry out the inspection process: operation Ac (m) executed by the inspection machine: inspection operation" includes "error standard Te (m): inspection result". -Good" is preset. Therefore, in response to the inspection operation (inspection operation result) of the inspection machine as "inspection result-defective" in the inspection process, the operation determination unit 130 determines that "an in-process error has occurred in the inspection operation". It is determined that "the operation standard Sa is satisfied".

Further, for example, "work process Pr (m): device 40 (m) used for carrying out the picking process: operation Ac (m) executed by the picking machine: picking operation" includes "error reference Te (m): "Successful picking" is preset. Therefore, in response to the picking operation (picking operation result) with the "picking failure" of the picking machine in the picking process, the operation determination unit 130 determines that "an in-process error has occurred in the picking operation" and "operations". It is determined that "the standard Sa is satisfied".

Further, for example, "work process Pr (m): equipment 40 (m) used for carrying out the screw tightening process: operation executed by the screw tightening machine Ac (m): screw tightening operation" includes "error reference Te ( m): The number of rotations is from p to q (> p) times ”is preset. Therefore, in contrast to the screw tightening operation (screw tightening operation result) when the screw tightening machine "rotates r (> q) times" in the screw tightening process, the operation determination unit 130 "sets the screw tightening operation. It is determined that "an internal error has occurred" and "the operation standard Sa is satisfied".

That is, at least that the operation determination unit 130 detects an abnormality by executing the operation Ac (m) by the device 40 (m) and that the operation Ac (m) of the device 40 (m) has an abnormality. When one of them is detected, it is determined that an in-process error has occurred in the operation Ac (m). Then, the operation determination unit 130 determines that the operation Ac (m) for which the in-process error has occurred is "satisfied with the operation reference Sa (m)".

As described above, the operation determination unit 130 includes "the actual operation Ac (m) executed by the device 40 (m) when the work process Pr (m) is performed" and the error associated with this operation Ac (m). It is compared with the reference Te (m) to determine whether an in-process error has occurred in the operation Ac (m). When the operation determination unit 130 determines that "an in-process error has occurred in the actual operation Ac (m) executed by the device 40 (m) when the work process Pr (m) is executed", the operation Ac (m) is determined. It is determined that the operation standard Sa (m) is satisfied.

(S130-2: Judgment of inter-process error)
The operation determination unit 130 uses the operation result La (m) indicating the content and result of the actual operation Ac (m) executed by the device 40 (m) when the work process Pr (m) is executed, and uses the operation result La (m) to perform "operation Ac (m). For m), it is determined whether a process skip (inter-process error) has occurred.

For example, among the plurality of work process Prs, there is a work process Pr that should be carried out in a predetermined order. Specifically, a situation such as "the work process Pr (m) should be performed after the work process Pr (m-1) is performed and before the work process Pr (m + 1) is performed" is assumed. In this case, the operation Ac (m) corresponding to the work process Pr (m) corresponds to the work process Pr (m + 1) after the execution of the operation Ac (m-1) corresponding to the work process Pr (m-1) is completed. Action to be performed Before the start of execution of Ac (m + 1), it must be executed.

When it is determined that such "operations Ac (m) to be executed in a predetermined order are not executed in a predetermined order", the operation determination unit 130 determines that the operation Ac (m) is not executed in a predetermined order. Regarding (m), it is determined that an inter-process error (process skipping) has occurred.

One typical example of an inter-process error is when "the operation Ac (m) that should be executed before the execution of the subsequent operation Ac (m + 1) is not executed before the execution of the operation Ac (m + 1) is started". Is. In addition, when "the operation Ac (m) that should be executed after the execution of the preceding operation Ac (m-1) is completed is not executed after the execution of the operation Ac (m-1) is completed", an inter-process error occurs. It can be mentioned as a typical example. In these inter-process errors, "the operation Ac (m) to be executed is not executed", and in particular, "the operation Ac (m) to be executed in a predetermined order is predetermined". It is also called "process skipping" because it is not executed in order.

Here, as described above, in the operation reference table 141, as the operation reference Sa, "execution order information indicating in what order the plurality of work process Prs should be executed for a plurality of work process Prs". "Si" is stored. That is, the operation reference table 141 stores the execution order information Si indicating in what order each of the plurality of operation Acs corresponding to each of the plurality of work process Prs should be executed.

The operation determination unit 130 refers to the operation reference table 141 and acquires the execution order information Si (m). For example, "the operation Ac (m) is the operation Ac after the execution of the operation Ac (m-1) is completed. It should be executed before the start of execution of (m + 1). ”The execution order information Si (m) is acquired. Then, the operation determination unit 130 uses the acquired execution order information Si (m) and the process information to perform the operation Ac (m) to be executed by the device 40 (m) used to execute the work process Pr (m). ), It is determined whether or not an inter-process error has occurred.

Specifically, in the operation determination unit 130, the device 40 (m-1) is based on the operation result La (m-1) of the device 40 (m-1) used for carrying out the work process Pr (m-1). The operation completion time Tme (m-1) of the executed operation Ac (m-1) is specified. Further, the operation determination unit 130 starts the operation start time of the operation Ac (m + 1) executed by the device 40 (m + 1) from the operation result La (m + 1) of the device 40 (m + 1) used for executing the work process Pr (m + 1). Identify Tms (m + 1). Further, the operation determination unit 130 is based on the operation result La (m) of the device 40 (m) used for carrying out the work process Pr (m), and the operation period Da of the operation Ac (m) executed by the device 40 (m). (M) is specified, in other words, the period during which the operation Ac (m) is not executed is specified.

When the operation determination unit 130 confirms that the operation period Da (m) of the operation Ac (m) does not exist between the operation completion time Tme (m-1) and the operation start time Tms (m + 1), the operation determination unit 130 “operates”. For Ac (m), an inter-process error (process skipping) has occurred. " Further, when the operation determination unit 130 confirms that the operation period Da (m) of the operation Ac (m) does not exist within a predetermined time from the operation completion time Tme (m-1), the operation determination unit 130 confirms that “the operation Ac (m)”. , An inter-process error (process skipping) has occurred. " Further, when the operation determination unit 130 confirms that the operation period Da (m) of the operation Ac (m) does not exist before the operation start time Tms (m + 1), “an inter-process error (for the operation Ac (m)) Process skipping) has occurred. " Then, the operation determination unit 130 determines that "the operation Ac (m) satisfies a predetermined operation reference Sa (m)" with respect to the operation Ac (m) that has been determined that "an error between processes has occurred".

As described above, the operation determination unit 130 uses the operation result La (m) to perform a process error (process) for "the operation Ac (m) executed by the device 40 (m) when the work process Pr (m) is performed". It is determined whether or not there is an internal error and an inter-process error) (S130).

(Yes in S140: When an error between processes occurs)
When the operation determination unit 130 determines that the operation Ac (m) has a process error (Yes in S130), the operation determination unit 130 determines whether the process error generated in the operation Ac (m) is an "inter-process error" (S140). ). When the operation determination unit 130 determines that the process error generated in the operation Ac (m) is an "inter-process error" (Yes in S140), an "inter-process error" occurs, that is, "process skipping" occurs. The operation Ac (m) determined to have occurred is specified (S150).

The operation determination unit 130 determines that the “process skipping” has occurred, that is, with respect to the operation Ac (m) determined to satisfy the operation reference Sa (m), the work process Pr (corresponding to the operation Ac (m)). For example, the following information is generated as the extraction target information indicating m). That is, the operation determination unit 130 specifies the time point at which the action Ac (m) that was not actually executed should have been originally executed as the "flag time point". For example, the operation determination unit 130 has elapsed a predetermined grace time from the operation completion time Tme (m-1) of the “operation Ac (m-1) to be executed in the order immediately before the operation Ac (m)”. The time point is specified as the "flag time point". The operation determination unit 130 sets the “cutting time” from the time when a predetermined period (for example, 30 seconds) is traced back from the specified flag time to the flag time, that is, the operation Ac (m) which satisfies the operation reference Sa (m). It is determined as the extraction target information indicating the corresponding work process Pr (m) (S160). The operation determination unit 130 notifies the extraction unit 150 of the determined “cutting time”.

The extraction unit 150 extracts an imaging moving image (imaging data Id) corresponding to the "cutting time" determined in S160 from the basic imaging data BI as partial imaging data OD (S170). The transmission unit 160 may transmit the image pickup data Id extracted as the partial image pickup data OD from the basic image pickup data BI by the extraction unit 150 to the outside of the control system 1.

That is, the operation determination unit 130 states that "the operation Ac (m) corresponding to the operation result La (m) that the process was not executed in the order in which it should be executed" is, that is, "the operation Ac (m) in which the process skip occurs". "Satisfies the predetermined operation standard Sa (m)".

According to the above configuration, the information processing apparatus 10 determines that the operation Ac corresponding to the operation result La that is not executed in the order in which it should be executed satisfies the predetermined operation reference Sa.

Therefore, the information processing apparatus 10 extracts from the basic imaging data BI the imaging data Id indicating the execution status of the work process Pr corresponding to the operation Ac that was not executed in the order to be executed as the partial imaging data OD. It has the effect of being able to. For example, in the information processing apparatus 10, the implementation status of the work process Pr (m-1) executed immediately before the work process Pr (m) corresponding to the operation Ac (m) that was not executed in the order to be executed. The imaging data Id (m-1) indicating the above is extracted as the partial imaging data OD. Further, for example, in the information processing apparatus 10, the implementation status of the work process Pr (m + 1) executed immediately after the work process Pr (m) corresponding to the operation Ac (m) that was not executed in the order to be executed. The imaging data Id (m + 1) indicating the above is extracted as the partial imaging data OD.

(No in S140: When an in-process error occurs)
When the operation determination unit 130 determines that the process error generated in the operation Ac (m) is not an "inter-process error" (No in S140), it is assumed that an "in-process error" has occurred in the operation Ac (m). The operation Ac (m) in which the “in-process error” has occurred is specified (S220). The operation determination unit 130 starts (that is, operation start time Tms (m)) and completes (that is, operation completion time Tme (m)) of the work process Pr (m) corresponding to the operation Ac (m) specified in S220. Therefore, the "cutting time" is determined (S200). The operation determination unit 130 sets the "cutting time", that is, the operation start time Tms (m) and the operation completion time Tme (m) of the operation Ac (m) in which the "in-process error" occurs, to the operation Ac (m). The extraction unit 150 is notified as the extraction target information indicating the work process Pr (m) corresponding to the above. The extraction unit 150 extracts the imaging moving image (imaging data Id) corresponding to the “cutting time” determined in S220 from the basic imaging data BI as the partial imaging data OD (S170).

That is, the operation determination unit 130 uses the operation result La (m) to detect an abnormality in the operation Ac (m) of the device 40, and indicates that the operation Ac (m) of the device 40 has an abnormality. When at least one is detected, it is determined that an in-process error has occurred in the operation Ac (m). Then, the operation determination unit 130 determines that the operation Ac (m) for which the in-process error has occurred is "satisfied with the operation reference Sa (m)", and notifies the extraction unit 150 of the determination result. In other words, the operation determination unit 130 has at least one of (A) the operation result La that the device 40 has detected an abnormality and (B) the operation result La that the operation Ac executed by the device 40 has an abnormality. It is determined that the operation Ac corresponding to the above satisfies the predetermined operation reference Sa.

According to the above configuration, the information processing apparatus 10 corresponds to at least one of (A) the operation result La when an abnormality is detected and (B) the operation result La when the executed operation Ac has an abnormality. It is determined that the operation Ac to be performed satisfies the predetermined operation reference Sa.

Therefore, the information processing apparatus 10 corresponds to at least one of (A) the operation result La that the abnormality is detected and (B) the operation result La that the executed operation Ac has an abnormality from the basic imaging data BI. It is effective that the imaging data Id indicating the implementation status of the work step Pr can be extracted as the partial imaging data OD.

(No in S130: When no process error has occurred-Judgment of standard operating time)
When the operation determination unit 130 determines that no process error has occurred in the operation Ac (m) (No in S130), the operation determination unit 130 identifies and identifies the operation Ac (m) determined that no process error has occurred. The operation period Da (m) of the operation Ac (m) is calculated (S180). Specifically, for the operation Ac (m) determined that no process error has occurred, the operation determination unit 130 starts the operation start time Tms (m) and the operation start time Tms (m) from the operation result La (m) of the operation Ac (m). The operation completion time Tme (m) is specified. From the specified operation start time Tms (m) and operation completion time Tme (m), the operation determination unit 130 sets the operation period Da for the operation Ac (m) executed by the device 40 when the work process Pr (m) is executed. (M) is calculated.

Further, the operation determination unit 130 refers to the operation reference table 141 and refers to the standard operation time Tta (m) corresponding to the operation Ac (m) (specifically, the standard lower limit operation time lower limit Ttal (m) and the standard upper limit. The operating time upper limit Ttah (m)) is acquired. The operation determination unit 130 compares the acquired standard operation time Tta (m) with the operation period Da (m) calculated in S180, and the operation period Da (m) is outside the range of the standard operation time Tta (m). (That is, whether it is within the range) is determined (S190). That is, in the operation determination unit 130, the "operation period Da (m) required for the device 40 (m) to actually execute the operation Ac (m)" is set to "the operation Ac (m) of the device 40 (m)". It is determined whether or not it is out of the range of "standard operating time Tta (m)" which is normally required to execute. Specifically, the operation determination unit 130 determines "whether the operation period Da (m) is less than the standard lower limit operation time lower limit Ttal (m)", and "the operation period Da (m) is the standard upper limit operation". Whether the time upper limit Ttah (m) is exceeded "is determined.

In the operation determination unit 130, the operation period Da (m) of the operation Ac (m) is not within (that is, outside the range) of the standard operation time Tta (m) preset for the operation Ac (m). ) (Yes in S190), it is determined that the operation Ac (m) satisfies the predetermined operation reference Sa (m). Specifically, the operation determination unit 130 states that "the operation period Da (m) is less than the standard lower limit operation time lower limit Ttal (m)" or "the operation period Da (m) is the standard upper limit operation time upper limit Ttah (m)". When it is confirmed that "exceeds", it is determined that the operation Ac (m) satisfies the operation reference Sa (m).

The operation determination unit 130 starts (that is, operation start time Tms (m)) and completes (that is, operation start time Tms (m)) of the work process Pr (m) corresponding to the operation Ac (m) determined to satisfy the predetermined operation reference Sa (m). , The "cutting time" is determined from the operation completion time Tme (m) (S200). The operation determination unit 130 sets the "cutting time", that is, the operation start time Tms (m) and the operation completion time Tme (m), and the extraction target information indicating the work process Pr (m) corresponding to the operation Ac (m). To notify the extraction unit 150. The extraction unit 150 extracts the imaging moving image (imaging data Id) corresponding to the “cutting time” determined in S220 from the basic imaging data BI as the partial imaging data OD (S170).

That is, the operation determination unit 130 has a range of standard operation time Tta in which the operation period Da required to complete the execution of the operation Ac executed by the device 40 is preset as the time required to complete the execution. If it is not inside, the operation Ac determines that the predetermined operation reference Sa is satisfied. In other words, if the operation period Da (m) required for the operation determination unit 130 to actually complete the execution is not within the range of the standard operation time Tta (m), the operation Ac (m) is the operation reference Sa. It is determined that (m) is satisfied.

According to the above configuration, the information processing apparatus 10 determines that the operation Ac in which the operation period Da actually required to complete the execution is not within the range of the standard operation time Tta satisfies the predetermined operation reference Sa. To do. Specifically, the information processing apparatus 10 determines that the operation Ac (m) for which "the operation period Da (m) is less than the standard lower limit operation time lower limit Ttal (m)" satisfies the operation reference Sa (m). To do. Further, the information processing apparatus 10 determines that the operation Ac (m) for which "the operation period Da (m) exceeds the standard upper limit operation time upper limit Ttah (m)" satisfies the operation reference Sa (m).

Therefore, the information processing apparatus 10 indicates from the basic imaging data BI the implementation status of the work process Pr corresponding to the operation Ac in which the operation period Da required to actually complete the execution is not within the range of the standard operation time Tta. It has the effect that the imaging data Id can be extracted as the partial imaging data OD. Specifically, the information processing apparatus 10 has a work process Pr corresponding to the operation Ac (m) in which "the operation period Da (m) is less than the standard lower limit operation time lower limit Ttal (m)" from the basic imaging data BI. The imaging data Id (m) indicating the implementation status of (m) can be extracted as the partial imaging data OD. Further, the information processing apparatus 10 has a work process Pr (m) corresponding to the operation Ac (m) in which "the operation period Da (m) exceeds the standard upper limit operation time upper limit Ttah (m)" from the basic imaging data BI. The imaging data Id (m) indicating the implementation status of the above can be extracted as the partial imaging data OD.

(No in S190: When the standard operating time is not exceeded-Image analysis)
When the operation determination unit 130 determines that the operation period Da of the operation Ac (m) that has determined that no process error has occurred is within the range of the standard operation time Tta (m) (No in S190), "Operation Ac" is determined. (M) does not satisfy the operation standard Sa (m). " As described above, when the operation period Da of the operation Ac (m) is within the range from the standard lower limit operation time lower limit Ttal (m) to the standard upper limit operation time upper limit Ttah (m), the operation determination unit 130 has an operation period Da. Is determined to be within the standard operating time Tta (m).

When the operation determination unit 130 confirms that no process error has occurred in the operation Ac (m) and the operation period Da (m) is within the range of the standard operation time Tta (m), the operation determination unit 130 "operates". Ac (m) does not satisfy the operation standard Sa (m). " The operation determination unit 130 specifies the work process Pr (m) corresponding to the operation Ac (m) for the operation Ac (m) determined not to satisfy the operation reference Sa (m), and the specified work process Pr (m). ) Is notified to the image analysis unit 170 of the information (non-extraction target information).

The image analysis unit 170 describes the work process Pr (m) corresponding to the operation Ac (m) in which "the process error has not occurred and the operation period Da (m) is within the range of the standard operation time Tta (m)". The following determination is made by image analysis with respect to the image pickup data ID (m) obtained by imaging the implementation status of m). That is, the image analysis unit 170 identifies the work Op (m) performed by the worker Pe (x) at the time of executing the work step Pr (m) by image analysis on the image pickup data ID (m), and the work Op (m). ) Is a non-standard operation (S210).

When it is determined that the work Op (m) specified by the image analysis on the imaged data Id (m) is not a non-standard work (that is, a standard work) (No in S210), the information processing apparatus 10 ends the process. .. When it is determined that the work Op (m) is a non-standard work (Yes in S210), the image analysis unit 170 determines the "cutting time" for the work Op (m) (S200). That is, the image analysis unit 170 "cuts time" for the work Op (m) from the start (that is, the stay start time Tos (m)) and the completion (that is, the stay end time Toe (m)) of the work Op (m). To decide. The details will be described later, but at the start of the stay, Tos (m) states that "the worker Pe who performed the work Op (m) has come to exist in the monitoring area Ar (m). It was confirmed by image analysis. " Further, at the end of the stay, Toe (m) stated, "The image analysis unit 170 confirmed by image analysis that the worker Pe who performed the work Op (m) no longer exists in the monitoring area Ar (m)." At the time.

The image analysis unit 170 notifies the extraction unit 150 of the determined "cutting time", that is, the stay start time Tos (m) and the stay end time Toe (m) as extraction target information indicating the work Op (m). To do. The extraction unit 150 extracts the imaging moving image (imaging data Id) corresponding to the “cutting time” determined in S220 from the basic imaging data BI as the partial imaging data OD (S170).

That is, the image analysis unit 170 is subjected to image analysis on the basic imaging data BI, particularly the imaging data Id (m) in the basic imaging data BI, so that the worker Pe (x) can perform the work step Pr (m). It is determined whether the work Op (m) performed is the standard work performed when the work step Pr (m) is carried out. The extraction unit 150 uses the imaging data Id (m) for the situation in which the work Op (m) determined by the image analysis unit 170 to be non-standard work (standard work) is performed as the partial imaging data OD. Extracted from the basic imaging data BI. An example of the image analysis executed by the image analysis unit 170 and the determination process of whether or not it is a standard operation will be described below.

(Image analysis processing (1): determination of the presence of workers, etc.)
The image analysis unit 170 analyzes the basic imaging data BI, which is the imaging data of the entire work site WS by the ceiling camera 30, and determines whether or not the worker Pe exists in the monitoring area Ar. The image analysis unit 170 has an image analysis of the monitoring area Ar (m) of the work site WS with respect to the analysis target area Aa (m) of the basic imaging data BI, and the worker Pe exists in the monitoring area Ar (m). Judge whether or not.

That is, the image analysis unit 170 determines whether or not the worker Pe exists in the work site WS by image analysis on the basic imaging data BI, and if it determines that the worker Pe exists in the work site WS, further work. The monitoring area Ar in which the person Pe exists is specified.

(Identification of worker identification information)
When the image analysis unit 170 determines that "the worker Pe exists in the monitoring area Ar (m)", the worker Pe existing in the monitoring area Ar (m) is further analyzed by image analysis on the analysis target area Aa (m). Worker ID may be specified. That is, the image analysis unit 170 performs image analysis on the basic imaging data BI in which the worker ID attached to the crown of the hat or the like is imaged, so that the worker Pe of the worker Pe existing at the work site WS Identify the ID. The image analysis unit 170 performs image analysis on the basic imaging data BI to "whether the worker Pe exists in the work site WS", "where is the monitoring area Ar where the worker Pe exists", and "the monitoring area Ar". The worker ID of the worker Pe existing in the above can be specified.

(Specification of stay period)
When the image analysis unit 170 determines that the worker Pe exists in the monitoring area Ar that was previously determined by the image analysis for the analysis target area Aa that the worker Pe does not exist. Is specified as Tos at the start of stay. Further, the image analysis unit 170 determines that the worker Pe does not exist in the monitoring area Ar which was previously determined to have the worker Pe by the image analysis for the analysis target area Aa. The time when this happens is specified as the Toe at the end of the stay.

In the image analysis unit 170, the worker Pe determines the time from the stay start time Tos (m) to the stay end time Toe (m) for the monitoring area Ar (m) associated with the work process Pr (m). Identify as the time that existed. In the following description, the time from the start time of stay Tos (m) (x) to the end time of stay Toe (m) (x), which is the time when the worker Pe (x) was present in the monitoring area Ar (m). "Time" is also referred to as stay period Do (m) (x).

(Calculation of frequency of stay)
In addition to specifying (measuring) the stay period Do for each work process Pr (that is, for each monitoring area Ar), the image analysis unit 170 further determines the stay period Do for each work process Pr (that is, for each monitoring area Ar). The frequency may be calculated.

The image analysis unit 170 grasps the time from the stay start time Tos (m) to the stay end time Toe (m) for the monitoring area Ar (m) as "one stay period Do (m)". "When it is confirmed that the worker Pe (1), which has not existed in the monitoring area Ar (m), exists in the monitoring area Ar (m) (= Tos (m) (1) at the start of stay) From ", when it is confirmed that the worker Pe (1) that had existed in the monitoring area Ar (m) no longer exists in the monitoring area Ar (m) (= Toe (m) at the end of the stay). The time until "(1))" is "one stay period Do (m) (1)".

The image analysis unit 170 sets “x + y + ... + z = n” and sets the stay period Do (m) (1) of the worker Pe (1) “n” times for a certain time at predetermined time intervals. You may calculate how many times the interval stay period Do (m) (1) was out of "n" times. For example, the image analysis unit 170 confirms that the stay period Do (m) (1) of 0 to 4 minutes was "x" times out of the stay period Do (m) (1) of "n" times. .. Further, the image analysis unit 170 confirms that the stay period Do (m) (1) of 5 to 9 minutes has been "y" times. Similarly, with "p" as an integer of "1" or more, the image analysis unit 170 has a stay period Do (m) (1) of "5 (p-1)" to "5p-1" of "z". Confirm that it was a time.

Then, the image analysis unit 170 describes the frequency of the stay period Do (m) (1) of 0 to 4 minutes for the "n" stay period Do (m) (1) of the worker Pe (1) from 5 minutes. Frequency of stay period Do (m) (1) for 9 minutes, ..., Calculate the frequency of stay period Do (m) (1) for "5p-1" from "5 (p-1)" May be good.

The image analysis unit 170 extracts the stay period Do (m) having a frequency lower than the predetermined standard as the "non-standard" stay period Do (m) for the "n" stay period Do (m). May be good. The image analysis unit 170 may extract the imaging data Id (m) corresponding to the extracted “non-standard” stay period Do (m) from the basic imaging data BI as partial imaging data OD.

(Specification of work process)
As described above, the monitoring area Ar and the work process Pr are associated in advance, that is, each of the monitoring areas Ar (1), Ar (2), Ar (3), ... Ar (n) is It is associated with each of the work processes Pr (1), Pr (2), Pr (3), ... Pr (n).

Therefore, when the image analysis unit 170 analyzes the basic imaging data BI and confirms that the worker Pe (1) exists in the monitoring area Ar (m), for example, the monitoring in which the worker Pe (1) exists. The work process Pr (m) corresponding to the region Ar (m) can be specified.

(Image analysis processing (2): Correspondence between stay period and operation period)
The image analysis unit 170 uses the process information to perform "a stay period Do in which the worker Pe exists in the monitoring area Ar associated with the work process Pr" and "the device 40 executes the work process Pr." Information associated with the operation period Da of the operation Ac is generated.

Specifically, when the image analysis unit 170 confirms that the worker Pe (1) exists in the monitoring area Ar (m) by image analysis, for example, the work process Pr corresponding to the monitoring area Ar (m). (M) is specified, and further, the operation Ac (m) corresponding to the work process Pr (m) is specified. Then, in the image analysis unit 170, the worker Pe is placed in the monitoring area Ar (m) by using the operation period Da (m) of the operation Ac (m) specified by the operation determination unit 130 from the operation result La (m). Information may be generated in which the existing stay period Do (m) and the operation period Da (m) are associated with each other.

(Image analysis process (3): work estimation and judgment)
Using the process information, the image analysis unit 170 estimates the work Op performed by the worker Pe in the “stay period Do in which the worker Pe exists in the monitoring area Ar associated with the work process Pr”, and concretely. First, it is estimated what kind of work Op the worker Pe has executed. In particular, the image analysis unit 170 uses the operation result La of the “equipment 40 used to carry out the work process Pr” and uses the “stay period in which the worker Pe exists in the monitoring area Ar associated with the work process Pr”. The work Op performed by the worker Pe in "Do" is estimated. Then, the image analysis unit 170 determines whether or not the estimated work Op is a standard work (standard work) to be performed when the work process Pr is performed.

(3-1. Work performed at the start of operation)
Before the "equipment 40 (m) used for carrying out the work process Pr (m)" starts the execution of the operation Ac (m), the device 40 (m) is to start the operation Ac (m). In general, it is necessary for the worker Pe to perform a start instruction work to the device 40 (m). That is, at the operation start time Tms (m) at which the device 40 (m) starts executing the operation Ac (m), the monitoring area Ar (m) generally associated with the work process Pr (m). Worker Pe needs to be present in.

Therefore, when the image analysis unit 170 confirms that the worker Pe exists in the monitoring area Ar (m) at the operation start time Tms (m), the image analysis unit 170 confirms that the worker Pe exists in the monitoring area Ar (m). It is estimated that the start instruction work of the operation Ac (m) was executed on the device 40 (m). " The work Op (m) executed by the worker Pe existing in the monitoring area Ar (m) at the operation start time Tms (m) includes, for example, preparatory work for carrying out the work process Pr (m), pressing the start button, and the like. The work of instructing the start of the operation, the work of confirming whether the device 40 (m) has started the operation Ac (m) smoothly, and the like. The work of putting the work into the device 40 (m), such as setting the work, is an example of the preparatory work for carrying out the work process Pr (m). These operations are "normal operations performed by the worker Pe before and after the time when the device 40 (m) used for carrying out the work process Pr (m) starts executing the operation Ac (m)". , Also called "standard work".

That is, when the image analysis unit 170 confirms that the stay period Do (m) existing in the monitoring area Ar (m) overlaps with the operation start time Tms (m), the worker Pe confirms that the stay period Do The work Op (m) performed by the worker Pe during (m) is a start instruction work or the like. " Then, the image analysis unit 170 determines that "the start instruction work or the like performed by the worker Pe during the stay period Do (m) overlapping the operation start time Tms (m) is a standard work".

(3-2. About the work to be done when the operation is completed)
After the "equipment 40 (m) used for carrying out the work process Pr (m)" completes the execution of the operation Ac (m), the work for causing the device 40 (m) to complete the operation Ac (m). In general, it is necessary for the person Pe to perform the completion instruction work to the device 40 (m). That is, at the operation completion time Tme (m) in which the device 40 (m) completes the execution of the operation Ac (m), the monitoring area Ar (m) generally associated with the work process Pr (m). Worker Pe needs to be present in.

Therefore, when the image analysis unit 170 confirms that the worker Pe exists in the monitoring area Ar (m) at the operation completion time Tme (m), the image analysis unit 170 confirms that the worker Pe exists at the operation completion time Tme (m). It is estimated that the completion instruction work of the operation Ac (m) was executed to the device 40 (m). " The work Op (m) executed by the worker Pe existing in the monitoring area Ar (m) at the operation completion time Tme (m) is, for example, after the completion instruction work such as pressing the end button and the completion of the work process Pr (m). Post-work, confirmation work of whether the device 40 (m) has completed the smooth operation Ac (m), and the like. The work for which the work process Pr (m) has been completed, that is, the work for collecting the work for which the operation Ac (m) by the device 40 (m) used for carrying out the work process Pr (m) has been completed is the work process Pr (m). This is an example of post-work after the completion of. These operations are "normal operations performed by the worker Pe before and after the time when the device 40 (m) used for carrying out the work process Pr (m) completes the execution of the operation Ac (m)". , Also called "standard work".

That is, when the image analysis unit 170 confirms that the stay period Do (m) existing in the monitoring area Ar (m) overlaps with the operation completion time Tme (m), the worker Pe confirms that the stay period Do The work Op (m) performed by the worker Pe during (m) is a completion instruction work or the like. " Then, the image analysis unit 170 determines that "the completion instruction work or the like performed by the worker Pe during the stay period Do (m) overlapping the operation completion time Tme (m) is a standard work".

(3-3. Work performed when the operation is not executed)
After the "equipment 40 (m) used to carry out the work process Pr (m)" completes the "p" th execution of the operation Ac (m) and before the "p + 1" th execution is started. , It is conceivable that the worker Pe behaves as follows. That is, the worker Pe stops at the monitoring area Ar (m), performs the setup work for starting the "p + 1" th execution, and after that work, once leaves the monitoring area Ar (m), and then for a while. It is conceivable to return to the monitoring area Ar (m) and start the "p + 1" th execution.

Therefore, the image analysis unit 170 starts the operation start time Tms (m) (m) of the "p + 1" th operation Ac (m) from the operation completion time Tme (m) (p) of the "p" th operation Ac (m). After confirming that the period of stay Do (m) exists before p + 1), the following estimation is performed. That is, the image analysis unit 170 states that the worker Pe performs setup work and the like during the stay period Do (m) between the operation completion time Tme (m) (p) and the operation start time Tms (m) (p + 1). It was running. " Since the setup work and the like are "normal work performed by the worker Pe between the operation completion time Tme (m) (p) and the operation start time Tms (m) (p + 1)", the image analysis unit. 170 determines that "setup work and the like are standard work".

In other words, the image analysis unit 170 has a stay period Do (m) that does not overlap with the operation period Da (m) of the device 40 (m), and is within a predetermined time after the end of the stay period Do (m). The following estimation is performed for the stay period Do (m) in which the operation period Da (m) appears. That is, the image analysis unit 170 presumes that "the worker Pe was performing the setup work or the like during such a stay period Do (m)", and "work during such a stay period Do (m)". The work Op (m) performed by the person Pe is a standard work. "

(3-4. Work performed while the operation is being executed)
The image analysis unit 170 was performed by the worker Pe who was present in the monitoring area Ar (m) while the "equipment 40 (m) used for carrying out the work process Pr (m)" was executing the operation Ac (m). It is determined whether or not the work Op (m) is a standard work. In particular, the image analysis unit 170 uses the content of the operation Ac (m) executed by the “equipment 40 (m) used to carry out the work process Pr (m)” to execute the “operation Ac (m)”. While there, the work Op (m) of the worker Pe who "exists in the monitoring area Ar (m)" is determined.

(3-4-1. Judgment of whether or not the operation is automated)
First, the image analysis unit 170 uses the “operation result La (m) indicating the content and result of the operation Ac (m) executed by the device 40 (m) in carrying out the work process Pr (m)” to perform the operation Ac. It is determined whether or not (m) is an "automated operation".

Here, the "automated operation" is an operation Ac (m) executed by the device 40 (m) when the work process Pr (m) is performed, and is an operation Ac (m) having the following properties. ). That is, "automated operation" means "additional work Op (m) by the worker Pe during the execution of the operation (m) by the device 40 (m), that is, during the operation period Da (m). Is not normally required. ”Operation Ac (m).

For example, the "automated inspection operation" is a work from the start of the "automated inspection operation" by the device 40 (m), which is an automatic inspection machine, to the completion of the "automated inspection operation". No additional work Op (m) by the person Pe is usually required. That is, the "automated inspection operation" usually does not require an additional work Op (m) by the worker Pe from the start of the execution to the completion of the execution, that is, the operation of the "automated inspection operation". No additional work Op (m) by the worker Pe during the period Da (m) is usually required. Therefore, the "automated inspection operation" is an "automated operation".

On the other hand, the operation Ac (m) that is the "operation Ac (m) executed by the device 40 (m) when the work process Pr (m) is performed" and is not the "automated operation" is ". Also referred to as "non-automated operation" or "manual operation", that is, "non-automated operation". The "non-automated operation" is an operation Ac (m) executed by the device 40 (m) when the work process Pr (m) is performed, and is an operation Ac (m) having the following properties. is there. That is, "non-automated operation" means "additional work Op (m) by the worker Pe during the execution of the operation (m) by the device 40 (m), that is, during the operation period Da (m). ) Is usually required. ”Operation Ac (m).

For example, the "assembly assisting operation" is an operation Ac (m) that assists the assembly work Op (m) by the worker Pe, and after the device 40 (m) starts the "assembly assisting operation", the "assembly assisting operation" is performed. The assembly work Op (m) by the worker Pe is required until the "operation" is completed. That is, the "assembly auxiliary operation" usually requires an additional work Op (m) by the worker Pe from the start of execution to the completion of execution, that is, the operation period Da (m) of the "assembly auxiliary operation". Additional work Op (m) by the worker Pe in the middle is usually required. Therefore, the "assembly auxiliary operation" is a "non-automated operation".

Here, the image analysis unit 170 only needs to be able to grasp whether or not the operation Ac (m) is an "automated operation", and determines whether or not the operation Ac (m) is an "automated operation". It is not essential for the image analysis unit 170 to carry out the above. For example, the operation determination unit 130 uses the operation result La (m) to determine whether or not the operation Ac (m) is an "automated operation", and notifies the image analysis unit 170 of the determination result. You may. Further, the PLC 20 may use the operation result La (m) to determine whether or not the operation Ac (m) is an "automated operation", and notify the image analysis unit 170 of the determination result. ..

(3-4-2. Work performed during the execution of automated operations)
When the image analysis unit 170 determines that the operation Ac (m) executed by the device 40 (m) during the execution of the work process Pr (m) is an "automated operation", the image analysis unit 170 stays with the operation period Da (m). It is confirmed whether the period Do (m) overlaps with the predetermined time or more.

As described above, the operation Ac (m), which is an "automated operation", "usually does not require an additional work Op (m) by the worker Pe during the operation period Da (m)". The operating period Da (m) and the stay period Do (m) usually do not overlap for a predetermined time or longer.

Therefore, when the image analysis unit 170 confirms that the operation period Da (m) and the stay period Do (m) overlap for a predetermined time or more, it is not normal when the work process Pr (m) is performed. A situation has occurred. " That is, if the stay period Do (m) overlaps with the operation period Da (m) of the operation Ac (m), which is an "automated operation", the image analysis unit 170 will "stay period". The work Op (m) performed by the worker Pe during Do (m) is a non-standard work. "

For example, if the operation period Da (m) of the operation Ac (m), which is an "automated operation", overlaps with the operation period Da (m) of a predetermined time or longer and a short stay period Do (m), the stay period The image analysis unit 170 makes the following determinations regarding the work Op (m) performed by the worker Pe on the Do (m). That is, the image analysis unit 170 estimates that "the worker Pe is performing maintenance work or repair work on the device 40 (m)", and "the worker during the stay period Do (m)". The work Op (m) performed by Pe is a non-standard work. "

For example, if the operation period Da (m) of the operation Ac (m), which is an "automated operation", overlaps with the operation period Da (m) of a predetermined time or longer and a long stay period Do (m), the stay period The image analysis unit 170 makes the following determinations regarding the work Op (m) performed by the worker Pe on the Do (m). That is, the image analysis unit 170 presumes that "the worker Pe may be wasting time until the operation Ac (m), which is the" automated operation ", is completed." The work Op (m) performed by the worker Pe during the period Do (m) is a non-standard work. "

(3-4-3. Work performed during execution of non-automated operations)
When the image analysis unit 170 determines that the operation Ac (m) executed by the device 40 (m) during the execution of the work process Pr (m) is an "non-automated operation", the operation period Da (m) is set. Always check if it overlaps with the length of stay Do (m).

As described above, the operation Ac (m), which is an "unautomated operation", is "usually requires an additional work Op (m) by the worker Pe during the operation period Da (m)". , The operation period Da (m) always overlaps with the stay period Do (m).

Therefore, when the image analysis unit 170 confirms that the non-overlapping operating period Da (m) exists in the staying period Do (m), the image analysis unit 170 confirms that the non-overlapping operating period Da (m) exists. In (m), an unusual situation has occurred. " That is, when the image analysis unit 170 confirms that the operation period Da (m) that does not overlap with the stay period Do (m) exists, the image analysis unit 170 "requires the stay period Do before and after the operation period Da (m). The work Op (m) performed by the worker Pe during m) is a non-standard work. "

For example, in the operation period Da (m) of the operation Ac (m), which is an “unautomated operation”, which does not overlap with the stay period Do (m), the operation Ac (m) that should have been executed originally. However, it may be interrupted for some reason.

(3-5. Arrangement of work estimation and judgment)
As described above, the image analysis unit 170 uses the process information, and in particular, using the operation result La of the device 40 of the “device 40 used to carry out the work process Pr”, “stay period Do”. What kind of work Op was executed by the worker Pe in the above? ”Is estimated. Then, the image analysis unit 170 determines whether or not the estimated work Op is a standard work (standard work) to be performed when the work process Pr is performed.

That is, the image analysis unit 170 uses the process information, in particular, using the operation result La of the device 40 of the “device 40 used to carry out the work process Pr”, the operation start time Tms and the operation completion time of the device 40. The Tme and the operation period Da are specified. Then, the image analysis unit 170 confirms whether each of the specified operation start time Tms, operation completion time Tme, and operation period Da and the stay period Do overlap, and based on the confirmation result, the stay period Do In, the estimation and determination of the work Op performed by the worker Pe is executed.

Further, the image analysis unit 170 determines the content of the operation Ac of the device 40 during the operation period Da, specifically, whether or not the operation Ac executed by the device 40 during the operation period Da is an “automated operation”. , The operation result La of the device 40 is used for determination. As described above, the image analysis unit 170 may acquire the determination result of whether or not the operation Ac executed by the device 40 during the operation period Da is an “automated operation” from the operation determination unit 130.

When the image analysis unit 170 confirms that the operation period Da has a stay period Do that overlaps for a predetermined time or more, the image analysis unit 170 uses the contents of the operation Ac of the device 40 in the operation period Da to make the stay period Do. The estimation and determination of the work Op performed by the worker Pe is executed. That is, the image analysis unit 170 uses the content of the operation Ac (the result of determining whether or not the operation Ac is an "automated operation") to "residerate the operation period Da for a predetermined time or more." The work content of the work Op in "Do" is estimated, and it is determined whether the work Op is a standard work.

Here, regarding the "work Op performed by the worker Pe during the stay period Do", the content of the work Op is estimated depending on the estimation method and the determination method described above, and whether it is "standard work" or "non-standard work". It is also conceivable that the work Op cannot determine whether or not. The image analysis unit 170 may determine the work Op, for which the determination method described above could not determine whether the work is "standard work" or "non-standard work", as "non-standard work". ..

As described above, in the image analysis unit 170, the work Op performed by the worker Pe during the execution of the work process Pr is performed by the image analysis of the basic imaging data BI when the work process Pr is performed. Determine if it is a standard task to be performed. The extraction unit 150 extracts the imaging data Id indicating the situation in which the work Op determined to be non-standard work by the image analysis unit 170 is performed from the basic imaging data BI as the partial imaging data OD.

According to the above configuration, in the information processing apparatus 10, the work Op performed by the worker Pe at the time of executing the work process Pr is performed at the time of executing the work process Pr by image analysis on the basic imaging data BI. Determine if it is a standard task. Then, when the information processing apparatus 10 determines that the work Op is not a standard work, the information processing apparatus 10 extracts the imaging data Id indicating the situation in which the work Op determined to be non-standard work is performed from the basic imaging data BI. To do. That is, the information processing apparatus 10 obtains the desired imaging data Id from the basic imaging data BI by performing image analysis on the basic imaging data BI without transmitting the basic imaging data BI to the outside of the PLC20 slave control system. Extract as.

(Specific example of image analysis processing)
FIG. 5 is a diagram showing a method in which the image analysis unit 170 identifies the worker Pe and the monitoring area Ar by using the basic imaging data BI. As shown in FIG. 5, the image analysis unit 170 performs image analysis on the basic imaging data BI to determine whether a worker Pe exists for each of the plurality of monitoring areas Ar corresponding to each of the plurality of work process Prs. To judge. Then, with respect to the monitoring area Ar determined that the worker Pe exists, the worker ID of the worker Pe existing in the monitoring area Ar is specified.

In the example shown in FIG. 5, the image analysis unit 170 executes the following determination by image analysis on the basic imaging data BI, particularly by image analysis on a plurality of analysis target regions Aa. That is, the image analysis unit 170 determines that the worker Pe exists in the monitoring area Ar (2) and the monitoring area Ar (4) from the monitoring area Ar (1) to the monitoring area Ar (5). In the image analysis unit 170, the worker ID of the worker Pe existing in the monitoring area Ar (2) is "worker Pe (2)", and the work of the worker Pe existing in the monitoring area Ar (4) It is specified that the person ID is "worker Pe (1)".

FIG. 6 is a diagram showing an example of "information in which the operation period Da and the stay period Do are associated with each work process Pr" generated by the image analysis unit 170. As described above, the image analysis unit 170 uses the process information to perform "a stay period Do in which the worker Pe exists in the monitoring area Ar associated with the work process Pr" and "a device for carrying out the work process Pr". The information associated with the operation period Da of the operation Ac executed by 40 is generated. For example, the image analysis unit 170 identifies the “stay period Do (m) in which the worker Pe exists in the monitoring area Ar (m)” and the operation determination unit 130 using the operation result La (m). Information associated with "the operation period Da (m) of Ac (m)" may be generated.

The image analysis unit 170 distinguishes whether or not the operation Ac (m) executed during the operation period Da (m) is an "automated operation", and distinguishes between the stay period Do (m) and the operation period Da (m). You may generate the information associated with. As described above, by distinguishing whether or not the operation Ac (m) is an "automated operation", the image analysis unit 170 performs the work Op (m) performed by the worker Pe during the stay period Do (m). ) Is a "standard work" or not can be determined more precisely.

FIG. 6A shows an image analysis unit for “work Op (m) performed by worker Pe during the stay period Do (m)” when the movement Ac (m) is “automated movement”. An example of the result of the estimation performed by 170 is shown.

Regarding (A) of FIG. 6, the image analysis unit 170 indicates that the work Op (m) performed by the worker Pe during the stay period Do (m) overlapping the operation start time Tms (m) is the start instruction work or the like. Judged as "standard work". The image analysis unit 170 determines that the work Op (m) performed by the worker Pe during the stay period Do (m) overlapping the operation completion time Tme (m) is a "standard work" such as a completion instruction work. .. The image analysis unit 170 has a stay period Do (m) that does not overlap with the operation period Da (m), and the stay period Da (m) appears within a predetermined time after the end of the stay period Do (m). It is determined that the work Op performed by the worker Pe on Do (m) is a "standard work" such as a setup work.

If the operation period Da (m) of the operation Ac (m), which is an "automated operation", overlaps with the operation period Da (m) of a predetermined time or longer and a short stay period Do (m), the image analysis unit 170 Determines that "work Op (m) is non-standard work such as repair work".

If the operation period Da (m) of the operation Ac (m), which is an "automated operation", overlaps with the operation period Da (m) of a predetermined time or longer and a long stay period Do (m), the image analysis unit 170 Determines that "work Op (m) is a non-standard work such as waiting for inspection".

FIG. 6B shows an operation Op (m) performed by the worker Pe during the stay period Do (m) when the operation Ac (m) is an “unautomated operation (that is, a manual operation)”. ) ”, An example of the result of estimation executed by the image analysis unit 170 is shown.

In the image analysis unit 170, the work Op (m) performed by the worker Pe during the stay period Do (m) overlapping at least one of the operation start time Tms (m) and the operation completion time Tme (m) is the start instruction work. Etc. or it is judged to be "standard work" such as completion instruction work. The image analysis unit 170 has a stay period Do (m) that does not overlap with the operation period Da (m), and the stay period Da (m) appears within a predetermined time after the end of the stay period Do (m). It is determined that the work Op performed by the worker Pe on Do (m) is a "standard work" such as a setup work.

When it is confirmed that the operation period Da (m) of the operation Ac (m), which is an "unautomated operation", is interrupted during the stay period Do (m), the image analysis unit 170 indicates the "stay period". The work Op performed by the worker Pe on Do (m) is a non-standard work. "

FIG. 7 shows a screen example showing the correspondence between the operation period Da and the stay period Do in a list format for a plurality of work process Prs. As illustrated in FIG. 7, the information processing device 10 displays the correspondence between the operation period Da and the stay period Do, for example, on an external display device (external device 70, etc. in FIG. 2) for a plurality of work process Prs. It may be displayed in a list format. By displaying the correspondence between the operation period Da and the stay period Do in a list format for a plurality of work process Prs, the information processing apparatus 10 allows the user to discover the work process Pr that is a bottleneck and improve it. Can be facilitated to consider.

FIG. 8 is a diagram showing an example of a work site WS in which a plurality of monitoring areas Ar are arranged in correspondence with the execution order of a plurality of work process Prs. The image analysis unit 170 changes the movement process of the worker Pe at the work site WS by performing image analysis on the basic imaging data BI, and particularly by performing image analysis on each of the plurality of analysis target areas Aa corresponding to the plurality of work steps Pr. The information shown may be generated. Such information indicating the movement progress of the worker Pe is a bottleneck especially for the work site WS in which a plurality of monitoring areas Ar are arranged in correspondence with the execution order of the plurality of work processes Pr. It is useful for discovering work process Pr and considering improvement.

FIG. 9 is a diagram showing an example of information indicating the movement progress of the worker Pe at the work site WS, which is generated by the image analysis unit 170 by image analysis on the basic imaging data BI. With the "information indicating the movement progress of the worker Pe at the work site WS" as illustrated in FIG. 9, the user can perform the following examination, for example.

That is, the user can use the worker Pe from the monitoring area Ar (6) to the monitoring area Ar (7), from the monitoring area Ar (7) to the monitoring area Ar (8), and from the monitoring area Ar (8) to the monitoring area Ar (7). ), Confirm that the movement from the monitoring area Ar (7) to the monitoring area Ar (6) is repeated. When the monitoring area Ar (6) corresponds to the work process Pr (6): inspection process, the user can use the following, for example, from the “information indicating the movement progress of the worker Pe at the work site WS” illustrated in FIG. The situation can be estimated. That is, in the waiting time generated in the inspection process or the like, the worker Pe performs the work Op (6) in the inspection process: the work Op in the work process Pr (7) and the work process Pr (8) in parallel with the inspection work. It can be estimated that (7) and work Op (8) are being performed.

Further, as illustrated in FIG. 9, the user configures "1 cycle" from "information indicating the movement progress of the worker Pe at the work site WS" when, for example, the same movement progress occurs repeatedly. It is possible to specify a plurality of work process Prs. Similarly, the user can identify the "(cycle) switching step" from the movement progress of the worker Pe between cycles.

(Display of analysis results related to stay period and operation period)
As illustrated in FIGS. 10, 11, and 12, the information processing device 10 displays information indicating the distribution or the like of at least one of the stay period Do and the operation period Da in a predetermined period on an external display device or the like ( For example, it may be displayed on the external device 70) of FIG. For example, the transmission unit 160 transmits data indicating the distribution of at least one of the stay period Do and the operation period Da in a predetermined period to an external display device or the like, and transmits the data in the form of a box plot, a histogram, or the like. Then, it may be displayed on an external display device or the like.

(1-1. Distribution and total value of "stay period for each work process" in a predetermined period)
FIG. 10 is a box-and-whisker plot showing the distribution of the period of stay Do for each work process Pr required to produce one product P (x) selected by the user on a certain day. For example, when the user selects an arbitrary product P (x), the information processing device 10 requires a period of stay for each work process Pr required to produce one selected product P (x) on a display device or the like. For Do, a boxplot showing the distribution in the period selected by the user is displayed.

In the example shown in FIG. 10, a boxplot is used to show "the distribution of the stay period Do for each work process Pr required to produce one product P (x) in the period selected by the user". However, the information processing apparatus 10 may show the above-mentioned distribution using a histogram.

As described above, in the information processing apparatus 10, the worker Pe exists in the monitoring area Ar associated with the work process Pr, that is, the stay period Do for each work process Pr is determined by the image analysis for the analysis target area Aa. The period of stay Do, which indicates the period of stay, is calculated.

For example, a certain product P (x) goes through five work processes Pr (1) to Pr (5) of "screw tightening", "soldering", "assembly 1", "assembly 2", and "packing". Suppose it is produced. The information processing apparatus 10 calculates the stay period Do required to produce one product P (x) each time one product P (x) is produced for each of the above-mentioned five work process Prs. That is, each time the information processing apparatus 10 produces one product P (x), it corresponds to each of the work processes Pr (1) to Pr (5) required to produce one product P (x). Each of the length of stay Do (1) to Do (5) is calculated.

Then, the information processing device 10 provides information indicating how the stay periods Do (1) to Do (5) required to produce one product P (x) are distributed on a certain day. As illustrated in FIG. 10, the boxplot format is used to display the information on an external display device or the like.

In the box-and-whisker plot shown in FIG. 10, the period of stay Do (1) for the "screw tightening" process (= work process Pr (1)) required to produce one product P (x) on a certain day. It is shown that the maximum of is "66 seconds" and the minimum of the period of stay Do (1) is "49 seconds". Similarly, for the "soldering" process (= work process Pr (2)) required to produce one product P (x) on that one day, the maximum stay period Do (2) is "74". It is shown that the minimum of "seconds" and the length of stay Do (2) is "50 seconds".

Further, among the stay period Do (1) to Do (5) corresponding to each of the above-mentioned five work steps Pr, the maximum stay period Do is the stay period Do (4), that is, "assembly". It is "90 seconds" which is the maximum of the stay period Do (4) required for "2". Therefore, "90 seconds" is the so-called "tact time for producing one product P (x)".

Further, in FIG. 10, the information processing apparatus 10 displays the total value of each of the stay periods Do (1) to Do (5) required to produce one product P (x) on a certain day. I'm letting you. For example, for the "screw tightening" process (= work process Pr (1)) required to produce one product P (x), the total value of the stay period Do (1) on a certain day is "12000 seconds". Is shown to be. Similarly, for the "soldering" process (= work process Pr (2)) required to produce one product P (x), the total value of the stay period Do (2) on that one day is "13000". It is shown to be "seconds".

In the example shown in FIG. 10, the distribution of the period of stay Do for each work process Pr required to produce one product P (x) is (1) “on a certain day”, and (2). The total value "on a certain day" is shown.

However, the information processing apparatus 10 has (1) the distribution of the period of stay Do for each work process Pr required to produce one product P (x), and (1) the distribution "in the period selected by the user", and (2) The total value "in the period selected by the user" may be displayed. For example, the information processing apparatus 10 distributes (1) "in a certain week" of the period of stay Do for each work process Pr required to produce one product P (x) according to the user's choice. , And (2) the total value "in a certain week" may be displayed. Further, the information processing apparatus 10 distributes (1) "in a certain month" of the period of stay Do for each work process Pr required to produce one product P (x) according to the user's selection. , And (2) the total value "in a certain month" may be displayed.

(1-2. Distribution and total value of "operation period for each work process" in a predetermined period)
The information processing apparatus 10 is a boxplot showing the distribution of the "operating period Da" for each work process Pr required for producing one product P (x) selected by the user in the period selected by the user. May be displayed on a display device or the like.

As described above, the information processing apparatus 10 determines the operation period Da for each work process Pr from the process information, particularly from the operation result La, that is, the operation period Da of the operation Ac executed by the device 40 when the work process Pr is executed. Is calculated.

For example, assuming that a certain product P (x) is produced through the above-mentioned five work steps Pr (1) to Pr (5), the information processing apparatus 10 has the information processing apparatus 10 for each of the above-mentioned five work steps Pr. The operating period Da required to produce one product P (x) is calculated. That is, each time the information processing apparatus 10 produces one product P (x), it corresponds to each of the work processes Pr (1) to Pr (5) required to produce one product P (x). Each of the operation periods Da (1) to Da (5) is calculated.

Then, the information processing apparatus 10 has (1) "on a certain day" distribution of the operating period Da for each work process Pr, which is required to produce one product P (x), and (2). ) The total value "on a certain day" may be displayed.

That is, the information processing apparatus 10 has (1) the distribution of the operation period Da for each work process Pr required to produce one product P (x), and (1) the distribution “in the period selected by the user”, and (2) The total value "in the period selected by the user" may be displayed. For example, the information processing apparatus 10 distributes (1) "in a certain week" of the operation period Da for each work process Pr required to produce one product P (x) according to the user's choice. , And (2) the total value "in a certain week" may be displayed. Further, the information processing apparatus 10 distributes (1) "in a certain month" of the operation period Da for each work process Pr required to produce one product P (x) according to the user's selection. , And (2) the total value "in a certain month" may be displayed.

The information processing apparatus 10 uses a box plot to express "the distribution of the operating period Da for each work process Pr required to produce one product P (x) in the period selected by the user". It may be expressed using a histogram.

Further, the information processing apparatus 10 displays "distribution of the stay period Do and the operation period Da for each work process Pr required for producing one product P (x) in the period selected by the user" on one screen. It may be displayed at the same time inside.

As described above with reference to FIG. 10, in the information processing apparatus 10, the worker Pe is placed in the area (monitoring area Ar) for performing the work Op related to the execution of (1) the work process Pr for each work process Pr. A predetermined period for at least one of the period of stay (that is, the period of stay Do) and (2) the period during which the operation Ac related to the execution of the work process Pr is executed by the device 40 (that is, the operation period Da). The external device is displayed with information indicating at least one of the distribution and the total value in.

According to the above configuration, the information processing apparatus 10 displays on the external device information indicating at least one of the distribution and the total value in a predetermined period for at least one of the stay period Do and the operation period Da for each work process Pr. Let me.

Therefore, the information processing apparatus 10 can provide the user with information indicating at least one of the distribution and the total value in a predetermined period for at least one of the stay period Do and the operation period Da for each work process Pr. Play the effect of.

Here, the information processing device 10 may display information indicating the distribution in the predetermined period on an external device in the form of a box plot or a histogram. Further, the information processing apparatus 10 has at least one of the distribution and the total value in the predetermined period for at least one of the stay period Do and the operation period Da for each work process PrPr required to produce one predetermined product P. The information indicating the above may be displayed on the external device. At least one of the above-mentioned "predetermined period" and "predetermined product P" may be selected by the user.

(2-1. Distribution and total value of "stay period for each worker" in a predetermined period)
FIG. 11 shows one stay period for each of the plurality of worker Pes belonging to the worker group GP (x) selected by the user, regardless of which work process Pr the stay period Do is. It is a box plot showing the distribution of Do in a certain day. In the example shown in FIG. 11, for each of the workers 1 to 5 (that is, the workers Pe (1) to Pe (5)) belonging to the worker group GP (x), one stay period in a certain day. How the Do is distributed is shown using a box plot.

For example, when the user selects an arbitrary worker group GP (x), the information processing device 10 causes the display device or the like to display the following information. That is, the information processing device 10 displays a box-and-whisker plot showing the distribution of one stay period Do of each of the plurality of worker Pes belonging to the selected worker group GP (x) in the period selected by the user. Let me. That is, the information processing device 10 causes an external display device to display information indicating the distribution of one stay period in the period selected by the user for each worker Pe.

In the example shown in FIG. 11, a boxplot is used to show "a distribution of one stay period Do for each worker Pe in a period selected by the user", but the information processing apparatus 10 has a histogram. May be used to indicate the above distribution.

In the box-and-whisker plot shown in FIG. 11, the maximum period of one stay of the worker 1 (that is, the worker Pe (1)) Do (worker Pe (1)) on a certain day is "66 seconds". It is shown that the minimum stay period Do (worker Pe (1)) is "49 seconds". Similarly, the maximum of one stay period Do (worker Pe (2)) of the worker 2 (that is, the worker Pe (2)) on that one day is "74 seconds", and the stay period Do (work). The minimum of person Pe (2)) is shown to be "50 seconds".

Further, in FIG. 11, the information processing device 10 displays the total value of one stay period Do for each worker Pe on a certain day. For example, it is shown that the total value of one stay period Do (worker Pe (1)) of the worker 1 on a certain day is "12000 seconds". Similarly, it is shown that the total value of one stay period Do (worker Pe (2)) of the worker 2 on that one day is "13000 seconds".

In the example shown in FIG. 11, the distribution of (1) "on a certain day" and (2) the total value of (2) "on a certain day" of one stay period Do for each worker Pe is shown. However, the information processing device 10 calculates the total value of (1) the distribution of the one-time stay period Do for each worker Pe, "in the period selected by the user", and (2) "in the period selected by the user". , May be displayed. For example, the information processing apparatus 10 has a distribution of (1) "in a certain week" and (2) a total value of "in a certain week" of one stay period Do for each worker Pe according to the user's choice. May be displayed. Further, the information processing apparatus 10 has (1) a distribution of "in a certain month" and (2) a total value of "in a certain month" of one stay period Do for each worker Pe according to the user's selection. May be displayed.

(2-2. Distribution and total value of "operating period for each device" in a predetermined period)
The information processing device 10 displays a box-and-whisker plot showing the distribution of one operation period Da in the period selected by the user for each of the plurality of devices 40 belonging to the device group GM (x) selected by the user. It may be displayed on a device or the like. That is, the information processing device 10 may display information indicating the distribution of one operation period Da for each device 40 in the period selected by the user on a display device or the like.

For example, when the five devices 40 of the devices 40 (1) to 40 (5) belong to the device group GM (x), when the device group GM (x) is selected by the user, the information processing device 10 causes the information processing device 10. , Display the following information on a display device or the like. That is, the information processing device 10 causes the display device or the like to display a box-and-whisker plot showing the distribution of one operation period Da in the period selected by the user for each of the devices 40 (1) to 40 (5). ..

Specifically, in the information processing device 10, how the operation period Da (1) required for the device 40 (1) to execute the operation Ac (1) once is distributed in the period selected by the user. Is displayed on a display device or the like using a box plot. Similarly, the information processing apparatus 10 determines how the operation period Da (2) required for the device 40 (2) to execute the operation Ac (2) once is distributed in the period selected by the user. , Display on a display device or the like using a box plot. The information processing device 10 shows how the operation period Da (m) required for the device 40 (m) to execute the operation Ac (m) once is distributed in the period selected by the user. Instead of the figure, a histogram may be used for display on a display device or the like.

Further, the information processing apparatus 10 sets the total of the operation period Da (1) required for the device 40 (1) to execute the operation Ac (1) once in the period selected by the user, for example, on a certain day. The total of a certain week, the total of a certain month, or the total of a certain month may be displayed on a display device or the like.

Further, the information processing apparatus 10 has (1) distribution of "one stay period Do for each worker Pe" and "one operation period Da for each device 40" and "in a period selected by the user". , (2) The total value "in the period selected by the user" may be displayed simultaneously on one screen.

As described with reference to FIG. 11 so far, the information processing apparatus 10 stays in (1) an area (monitoring area Ar) for each worker Pe to perform work Op related to the execution of the work process Pr. For at least one of the period (that is, the period of stay Do) and (2) the period during which the operation Ac related to the execution of the work process Pr is being executed (that is, the operation period Da) for each device 40, in a predetermined period. Display information indicating at least one of the distribution and the total value on the external device.

According to the above configuration, the information processing apparatus 10 has information indicating at least one of the distribution and the total value in a predetermined period for at least one of the stay period Do for each worker Pe and the operation period Da for each device 40. Is displayed on the external device.

Therefore, the information processing device 10 provides the user with information indicating at least one of the distribution and the total value in a predetermined period for at least one of the stay period Do for each worker Pe and the operation period Da for each device 40. It has the effect of being able to.

Here, the information processing device 10 may display information indicating the distribution in the predetermined period on the external device in the form of a box plot or a histogram. Further, the information processing device 10 is a worker group GP to which a plurality of worker Pes belong, and for each of the plurality of worker groups Pe belonging to the worker group GP selected by the user, a predetermined period of stay Do is set. Information indicating at least one of the distribution and the total value in the period may be displayed on the external device. Further, the information processing device 10 is a device group GM to which the plurality of devices 40 belong, and for each of the plurality of devices 40 belonging to the device group GM selected by the user, the distribution of the operation period Da in a predetermined period and Information indicating at least one of the total values may be displayed on the external device.

(3. Comparison between stay period / operation period and free time in the specified period)
FIG. 12 is a diagram showing an example of a bar graph showing the ratio of the total stay period Do corresponding to the work process Pr to the total time of a certain day for each work process Pr. In FIG. 12, "free time" refers to a time other than "stay period Do".

The information processing device 10 calculates the total stay period Do corresponding to the work process Pr for each work process Pr for the "period selected by the user". Then, the information processing apparatus 10 displays information indicating the ratio of the calculated "total stay period Do in the period selected by the user" to the "total time of the period selected by the user" for each work process Pr. Display on the device etc.

In the example shown in FIG. 12, "a certain day" is selected as the "period selected by the user". The information processing device 10 displays the ratio of the total stay period Do to the total time (eg, 8 hours) of a certain day in the form of a bar graph for each work process Pr. Etc. are displayed. However, it is not essential for the information processing device 10 to display the above-mentioned ratio for each work process Pr in the form of a bar graph, and the information processing device 10 displays the above-mentioned ratio for each work process Pr in an arbitrary display format. It should be displayed.

In FIG. 12, it is shown that the total stay period Do (1) of the “screw tightening” process (= work process Pr (1)) occupies about 60% of the total time of “a certain day”. ing. Similarly, it is shown that the total stay period Do (4) of the "electrical inspection" process (= work process Pr (4)) occupies about 90% of the total time of "a certain day". There is.

The information processing device 10 displays information indicating the ratio of the "total operating period Da in the period selected by the user" to the "total time of the period selected by the user" on the display device or the like for each work process Pr. You may let me. For example, in the information processing apparatus 10, for each work process Pr, the ratio of "the total of the operation period Da corresponding to the work process Pr in the period selected by the user" to the "total time of the period selected by the user" A bar graph or the like may be displayed.

Further, in the information processing apparatus 10, the ratio of "the total of each of the stay period Do and the operation period Da in the period selected by the user" to the "total time of the period selected by the user" for each work process Pr. It may be displayed simultaneously on one screen.

As has been described with reference to FIG. 12, the information processing apparatus 10 has an area (monitoring area Ar) for performing (1) work Op related to the execution of the work process Pr for each work process Pr in a predetermined period. The total period during which the worker Pe stayed (that is, the period of stay Do), and (2) the period during which the operation Ac related to the execution of the work process Pr was executed by the device 40 (that is, the operation period Da). Information indicating the ratio of at least one of the totals to the total time (total, that is, "total time") during which the work can be performed or the operation Ac can be performed in the predetermined period is provided to the external device. Display it.

According to the above configuration, in the information processing apparatus 10, at least one of the total stay period Do and the total operation period Da for each work process Pr in the predetermined period occupies the total time in the predetermined period. The information indicating the above is displayed on the external device.

Therefore, the information processing apparatus 10 provides the user with information indicating that at least one of the total of the stay period Do and the total of the operation period Da for each work process Pr in the predetermined period occupies the total time. It has the effect of being able to. The information processing device 10 may display information indicating a ratio to the total time on the external device in the form of a bar graph.

[Embodiment 2]
Other embodiments of the present invention will be described below with reference to FIGS. 13-18. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the above embodiment, and the description thereof will be omitted.

FIG. 13 is a block diagram showing a configuration of a main part of the information processing apparatus 1010 according to the second embodiment of the present invention. The information processing device 1010 in the present embodiment includes an image analysis unit 171 in place of the image analysis unit 170 included in the information processing device 10 in the first embodiment. Since the information processing device 1010 has the same configuration as the above-mentioned information processing device 10 except that the image analysis unit 171 is provided instead of the image analysis unit 170, the image analysis unit 171 will be focused on below. The explanation will be given.

The basic imaging data BI acquired by the first acquisition unit 110 of the information processing apparatus 1010 is moving image data obtained by photographing the implementation status of a plurality of work processes Pr.

The image analysis unit 171 includes a correction filter 210 that corrects the result of the flow line analysis for each of the plurality of frames (that is, the plurality of captured images) constituting the captured moving image generated from the basic imaging data BI. The image analysis unit 171 is the same as the image analysis unit 170 included in the information processing apparatus 10 except that the correction filter 210 is provided.

The correction filter 210 corrects the result of the flow line analysis (Objective Detection) using an object detection algorithm such as YOLO (YouLookOnlyOnse) for the basic imaging data BI. Specifically, the correction filter 210 removes noise (an imaged object other than the worker Pe) from the result of the flow line analysis, and obtains the result of the flow line analysis for a certain frame (a certain captured image). Complement using the detection results of a predetermined number of frames immediately before a certain frame. The details of the correction by the correction filter 210 will be described below with reference to FIGS. 14 to 17.

(Setting of analysis target area)
FIG. 14 shows an example of information used when the correction filter 210 corrects each of a plurality of frames (a plurality of captured images) constituting the captured moving image generated from the basic imaging data BI. It is a figure. As described above, a plurality of analysis target regions Aa, each of which corresponds to each of the plurality of work process Prs, are set in the captured image (frame). In the example shown in FIG. 14, the analysis target areas Aa (1) to Aa (5), each of which corresponds to each of the work steps Pr (1) to Pr (5), are set in the frame (captured image). .. Specifically, the analysis target area Aa (m) corresponds to the monitoring area Ar (m) in which the work Op (m) related to the execution of the work step Pr (m) is performed, that is, the analysis target area Aa (m). ) Is associated with the work process Pr (m). For example, each of the plurality of cells in the cell line manufacturing process corresponds to each of the plurality of monitoring areas Ar.

Further, in the example shown in FIG. 14, the analysis target area Aa (0) corresponding to the “passage in the work site WS” is set in the frame (captured image). The analysis target areas Aa (0) to Aa (5) are also referred to as detection areas (detection ranges). Then, in the frame (captured image), the regions other than the analysis target regions Aa (0) to Aa (5) are designated as the analysis target regions Aa (-1). Since the analysis target area Aa (-1) does not correspond to the monitoring area Ar or the passage in the work site WS, the worker Pe should not be detected normally. Also referred to as "outside the detection range (non-detection area)".

The image analysis unit 171 (particularly, the correction filter 210) analyzes each of the plurality of frames (plurality of captured images) constituting the captured moving image generated from the basic imaging data BI in the analysis target regions Aa (-1) to Aa (1) to Aa ( Divide into 5). Then, the image analysis unit 171 (particularly, the correction filter 210) executes the correction process illustrated in FIG. 15 on the results of the flow line analysis of each of the plurality of frames (plurality of captured images) described above, and each of them The result of the flow line analysis is corrected to generate a highly accurate detection result.

(Example of correction processing)
FIG. 15 is a flow chart illustrating an example of correction processing executed by the image analysis unit 171 (particularly, the correction filter 210). The correction process is executed as a pre-process of a process of specifying the coordinates of the worker Pe for each of a plurality of frames (a plurality of captured images) constituting the captured moving image generated from the basic imaging data BI.

The correction filter 210 applies the flow line analysis result for "a certain frame" among the plurality of frames constituting the above-mentioned captured moving image to "outside the detection range (that is, the analysis target area Aa (-1))". Whether or not the worker Pe) is detected ”(S310). Then, when the correction filter 210 determines that "there is a human detection outside the detection range" for the flow line analysis result for a certain frame (Yes in S310), the correction filter 210 removes the detection of the human outside the detection range (S320). .. These processes correspond to "false detection of humans outside the detection range", that is, the correction filter 210 detects humans outside the detection range (that is, the analysis target area Aa (-1)). Remove from the flow line analysis result.

When it is determined that "there is no human detection outside the detection range" (No in S310) or the processing of S320 is completed, the correction filter 210 "immediately" -1 for the "certain frame". It is determined whether "not detected this time even though there was a detection other than" (S330).

Here, as described above, the information processing apparatus 1010 acquires the basic imaging data BI, which is the moving image data obtained by the ceiling camera 30 capturing the entire work site WS, from the ceiling camera 30. The captured moving image generated from the basic imaging data BI is composed of a plurality of frames (a plurality of captured images).

Therefore, the "most recent" of "there was a detection in" most recent "other than" -1 "" means "the currently detected frame (that is, the currently corrected frame) in the flow line analysis for the basic imaging data BI. It means a frame detected immediately before "a certain frame") "that is the target of processing. In other words, the "most recent" means the immediately preceding frame of the "certain frame" that constitutes the captured moving image generated from the basic imaging data BI.

Therefore, "there was a detection other than" -1 "in the latest" means that a human (worker Pe) was detected in a frame other than "outside the detection range" of the "most recent frame". Means. That is, "there was a detection other than" -1 "in the latest" means that "at least one of the analysis target areas Aa (0) to Aa (5)" of the "most recent frame of a certain frame" is a human. It means that (worker Pe) has been detected.

In S330, the correction filter 210 first determines whether a human (worker Pe) is detected in "at least one of the analysis target areas Aa (0) to Aa (5)" of the "most recent frame" of a certain frame. Check. When it is confirmed that a human (worker Pe) has been detected in "at least one of the analysis target areas Aa (0) to Aa (5)" of the "most recent frame" of a certain frame, the correction filter 210 In addition, check the following contents. That is, the correction filter 210 does not detect a human (worker Pe) in any of the analysis target areas Aa (0) to Aa (5) in the "flow line analysis result for the" this time "frame" ( = Check whether or not "Undetected has occurred") ". In other words, the correction filter 210 determines whether or not the operator Pe is detected in any of the analysis target areas Aa (0) to Aa (5) in the flow line analysis result for the current frame (= a certain frame). Check.

If it is determined that no detection has occurred this time (Yes in S330) even though there was a detection other than "-1" in the latest, the correction filter 210 complements with the latest value (S340). That is, although the worker Pe was detected in at least one of the analysis target areas Aa (0) to Aa (5) of the latest frame, in the flow line analysis for this frame, the analysis target areas Aa (0) to Aa (5) If the worker Pe is not detected in any of Aa (5), the process of S340 is executed.

In S340, the correction filter 210 first identifies the analysis target area Aa (x) (where “x” is an integer of any of “0” to “5”) in which the worker Pe is detected for the latest frame. To do. Then, the correction filter 210 complements the result of the flow line analysis for the current frame with the detection result for the latest frame. That is, the correction filter 210 also obtains data that the worker Pe is detected in the analysis target area Aa (x) in which the worker Pe is detected for the latest frame, as a result of detecting the current frame. Generate as.

That is, in the flow line analysis for the frame this time, if the worker Pe is not detected in any of the analysis target areas Aa (0) to Aa (5), the correction filter 210 executes the following processing. That is, the correction filter 210 confirms whether or not the worker Pe has been detected in any of the analysis target areas Aa (0) to Aa (5) for the latest frame. When the correction filter 210 confirms, for example, that "worker Pe was detected in the analysis target area Aa (0) of the latest frame", "worker Pe is detected in the analysis target area Aa (0) of the current frame". Generate a detection result of "detected". Similarly, when the correction filter 210 confirms that "worker Pe has been detected in the analysis target area Aa (1) of the latest frame", for example, "work in the analysis target area Aa (1) of the current frame". The detection result that "person Pe has been detected" is generated.

In S330 and S340, an example is shown in which the detection result in the "most recent" frame of a certain frame is used to complement the result of the flow line analysis of the certain frame, but the correction filter 210 is immediately before the certain frame. The detection result in "plurality" of frames may be used. That is, the correction filter 210 may supplement the result of the flow line analysis for a certain frame by using the detection result of the worker Pe in the "predetermined number" of frames immediately before the certain frame, and the "predetermined number". May be either "1" or "an integer greater than or equal to 2".

For example, even though the worker Pe was detected in any of the analysis target areas Aa (0) to Aa (5) in succession in "plurality" of frames immediately before a certain frame, the analysis of a certain frame was performed. When the worker Pe is not detected in any of the target areas Aa (0) to Aa (5), the correction filter 210 may execute the above-mentioned complement.

When the determination process of S330 is "No", or when the process of S340 is completed, the correction filter 210 executes the determination process of S350, that is, determines whether "inclusion" has occurred ( S350).

Here, "inclusion" means that a plurality of bounding boxes BB (Bounding Boxes) are installed in the flow line analysis for detecting the worker Pe from the frame, and the centers of the plurality of bounding boxes BB are installed close to each other. Is to be done. That is, "inclusion" is a state in which a plurality of bounding boxes BB having a distance of each center of a predetermined value or less are installed.

When the correction filter 210 determines that inclusion has occurred (Yes in S350), the plurality of bounding boxes BB in which inclusion has occurred are unified into the largest bounding box BB (S360).

When the determination process of S350 is "No", or when the process of S360 is completed, the correction filter 210 ends the correction process, and when the next frame (that is, the next captured image) is acquired, the S310 is displayed. Go back and repeat the process.

After the correction processing for the flow line analysis result by the correction filter 210 described above, the image analysis unit 171 executes the detection of the worker Pe in the same manner as the image analysis unit 170. That is, the image analysis unit 171 determines whether or not the worker Pe exists in the monitoring area Ar, and if it determines that the worker Pe exists in the monitoring area Ar, the worker Pe of the worker Pe existing in the monitoring area Ar. Specify the ID, position (coordinates), and length of stay Do. For example, the image analysis unit 171 specifies the center coordinates of the bounding box BB (when S360 is executed, the bounding box BB after the execution of S360) installed in the flow line analysis as the coordinates of the worker Pe.

The information processing apparatus 1010 described with reference to FIGS. 14 and 15 can be organized as follows. That is, the information processing device 1010 further includes a correction filter 210 (correction unit) in addition to the configuration of the information processing device 10. The correction filter 210 uses the captured image generated from the basic imaging data BI as (1) an area where the worker Pe performs a work op (for example, monitoring areas Ar (1) to Ar (5)) or the worker Pe. A detection region (for example, analysis target regions Aa (0) to Aa (5)) in which the passage through which the passage passes is imaged, and (2) a non-detection region (that is, an analysis target) that is a region other than the detection region. It is divided into areas Aa (-1)). Then, the correction filter 210 deletes the detection in the analysis target region Aa (-1) from the result of the flow line analysis for detecting the worker Pe from the captured image.

According to the above configuration, the information processing apparatus 1010 detects the captured image in a detection region (for example, analysis target regions Aa (0) to Aa (5)) and a non-detection region (that is, analysis target region Aa (-1)). ) And. Then, the information processing apparatus 1010 deletes the detection in the non-detection region from the result of the flow line analysis on the captured image. That is, the information processing apparatus 1010 deletes the noise "detection in the non-detection region (that is, the analysis target region Aa (-1))" from the result of the flow line analysis for the captured image.

Therefore, the information processing apparatus 1010 realizes highly accurate detection of the worker Pe by deleting noise from the result of the flow line analysis for the captured image, that is, realizes highly accurate analysis of the work process Pr. It has the effect of being able to.

In the information processing apparatus 1010, the first acquisition unit 110 acquires the basic imaging data BI which is the moving image data obtained by photographing the implementation status of the plurality of work processes Pr. Then, when the worker Pe is not detected in any of the analysis target areas Aa (0) to Aa (5) of a certain frame in the flow line analysis for the basic imaging data BI, the correction filter 210 complements the following. Execute the process. That is, the correction filter 210 complements the result of the flow line analysis for a certain frame by using the detection result of the worker Pe in a predetermined number of frames immediately before the certain frame.

According to the above configuration, the information processing apparatus 1010 executes the flow line analysis on the basic imaging data BI which is the moving image data obtained by photographing the execution status of the plurality of work processes Pr. Then, when the worker Pe is not detected in any of the analysis target areas Aa (0) to Aa (5) of a certain frame, the information processing apparatus 1010 executes the following complementary processing. That is, the information processing apparatus 1010 complements the result of the flow line analysis for a certain frame by using the detection result of the worker Pe in a predetermined number of frames immediately before the certain frame.

For example, the worker Pe was not detected in any of the analysis target areas Aa (0) to Aa (5) of a certain frame, but the worker Pe was detected in the "most recent" frame of the certain frame. If so, the information processing apparatus 1010 executes the following complementary processing. That is, the information processing apparatus 1010 complements the result of the flow line analysis for a certain frame with the detection result of the "most recent" frame, and the analysis target regions Aa (0) to Aa (5) of the certain frame. ), The detection result that the worker Pe is detected is generated. The information processing apparatus 1010 continuously detects the worker Pe in any of the analysis target areas Aa (0) to Aa (5) in the "plurality" of frames immediately before the certain frame, but the certain frame. When the worker Pe does not detect any of the analysis target areas Aa (0) to Aa (5) in the above, the above-mentioned complement may be executed.

Here, the basic imaging data BI is moving image data obtained by photographing the implementation status of a plurality of work processes Pr. Further, the analysis target areas Aa (0) to Aa (5) are areas in which the worker Pe performs work Op (for example, monitoring areas Ar (1) to Ar (5)) or passages through which the worker Pe passes. Is the imaged area. Therefore, even if the worker Pe is detected in any of the analysis target areas Aa (0) to Aa (5) in a predetermined number of frames immediately before a certain frame by the flow line analysis for the basic imaging data BI. For example, even in a certain frame, the worker Pe is usually detected in any of the analysis target areas Aa (0) to Aa (5). This is because it is usually difficult to imagine a situation in which the worker Pe suddenly disappears from the monitoring areas Ar (1) to Ar (5) and the passage where the worker Pe performs the work Op.

Therefore, even though the worker Pe is detected in any of the analysis target areas Aa (0) to Aa (5) of a predetermined number of frames immediately before a certain frame, the analysis target area Aa (0) of a certain frame is detected. If the worker Pe is not detected in any of Aa (5), it is highly possible that the result of the flow line analysis of a certain frame contains an error. Therefore, the information processing apparatus 1010 complements the result of the flow line analysis of the certain frame with the detection result of a predetermined number of frames immediately before the certain frame.

That is, when there is a high possibility that the result of the flow line analysis of a certain frame is incorrect, the information processing apparatus 1010 uses the result of the flow line analysis of the certain frame as the detection result of a predetermined number of frames immediately before. It can be complemented and the certain frame can be analyzed with high accuracy. Therefore, the information processing apparatus 1010 realizes highly accurate detection of the worker Pe by correcting the result of the flow line analysis of a certain frame, that is, realizes highly accurate analysis of the work process Pr. It has the effect of being able to.

When a plurality of bounding boxes BB having a distance of each center of a predetermined value or less are installed in the flow line analysis of the captured image, the correction filter 210 sets the plurality of bounding boxes BB and the bounding box BB having the largest area. Unify to.

According to the above configuration, when a plurality of bounding boxes BB having a distance of each center of a predetermined value or less are installed in the flow line analysis of the captured image, the information processing apparatus 1010 uses the plurality of bounding boxes BB. , Unify to the bounding box BB with the largest area.

Here, in the flow line analysis for the captured image, when a plurality of bounding boxes BB having a distance of each center of a predetermined value or less are installed, one worker Pe is erroneously detected as a plurality of worker Pes. It may have been. Therefore, the information processing apparatus 1010 unifies the plurality of bounding boxes BB into the bounding box BB having the largest area when a plurality of bounding boxes BB having a distance of each center of a predetermined value or less are installed.

That is, when there is a high possibility that one worker Pe is erroneously detected as a plurality of worker Pes in the flow line analysis of the captured image, the information processing apparatus 1010 has only the bounding box BB having the largest area. The number of detected worker Pe is set to one. Therefore, the information processing apparatus 1010 realizes highly accurate detection of the worker Pe by modifying the number of bounding boxes BB installed in the flow line analysis for the captured image, that is, highly accurate for the work process Pr. It has the effect of being able to realize analysis.

(Effect of correction processing)
FIG. 16 is a diagram showing an example of the flow line analysis result before correction by the correction filter 210. As shown in FIG. 16, in the flow line analysis result before correction, the presence other than the worker Pe (human) is detected as noise Z1 at the lower right of the screen, that is, noise Z1 is generated. Such noise Z1 may be erroneously detected as an operator (human).

Further, in the flow line analysis result before correction shown in FIG. 16, inclusion Z2 is generated on the right side of the center of the screen. Specifically, in FIG. 16, the bounding box BB1 and the bounding box BB2 are installed on the right side of the center of the screen with their centers substantially overlapping. When such inclusion Z2 occurs, one worker Pe may be mistakenly detected as a plurality of workers, for example, two workers.

FIG. 17 is a diagram showing an example of the analysis result after correction by the correction filter 210. As shown in FIG. 17, the noise Z1 generated at the lower right of the screen of FIG. 16 is removed by the correction filter 210 as “detection outside the detection range (that is, detection in the analysis target area Aa (-1))”. ing. Further, the bounding box BB1 and the bounding box BB2 in FIG. 16 are unified into the larger bounding box BB1 by the correction filter 210, and the inclusion Z2 generated on the right side of the center of the screen of FIG. 16 is eliminated in FIG. Has been done.

It is known that flow line analysis using an object detection algorithm such as YOLO tends to generate noise or the like, but the correction filter 210 removes noise or the like and corrects the flow line analysis. The position of the worker Pe is specified with high accuracy. That is, the correction filter 210 visualizes the process information with high accuracy, and by improving the visualization accuracy, it is possible to acquire more process information.

Then, the correction process by the correction filter 210 makes it possible to specify the position of the worker Pe with high accuracy, and therefore, the period during which the worker Pe stays in the monitoring area Ar corresponding to the work process Pr. The length of stay Do, which is, will also be specified with high accuracy. Hereinafter, using FIG. 18, the analysis performed by the information processing apparatus 1010 (particularly, the image analysis unit 171) by combining the stay period Do and the operation period Da specified with high accuracy by the correction process by the correction filter 210. The processing will be described.

(Arrangement of analysis processing examples for each work process)
FIG. 18 is a diagram showing an example of analysis performed by the information processing apparatus 1010 (particularly, the image analysis unit 171) using the combination of the stay period Do and the operation period Da. In FIG. 18, the state “1” of the device 40 indicates that “the device 40 is executing the operation Ac corresponding to the work process Pr”, that is, “the device 40 is in operation”. There is. The state "0" of the device 40 indicates that "the device 40 has not executed the operation Ac corresponding to the work process Pr", that is, "the device 40 is stopped".

Further, in FIG. 18, the state “1” of the person (worker Pe) means that “worker Pe is staying in the monitoring area Ar corresponding to the work process Pr”, that is, “worker Pe is staying in the monitoring area Ar”. I am staying. " The state "0" of the person (worker Pe) means that "the worker Pe is not staying in the monitoring area Ar corresponding to the work process Pr", that is, "the worker Pe is absent". Is shown.

(1. Analysis for manual process)
When the operation Ac executed by the device 40 when the work process Pr is executed is an "non-automated operation", that is, when the work process Pr is a manual process, information is shown in FIG. 18 (A). The processing apparatus 1010 performs the following analysis.

That is, when "device 40 is operating (device 40 state: 1)" and "worker Pe is staying (human state: 1)", the information processing device 1010 (particularly, the image analysis unit 171) It is determined that the worker Pe is "working" (performing) the work Op corresponding to the work process Pr. That is, in the work process Pr which is a manual process, when the stay period Do and the operation period Da overlap, the image analysis unit 171 performs the work Op corresponding to the work process Pr in the stay period Do. Judge that

When "device 40 is operating (device 40 state: 1)" and "worker Pe is absent (person state: 0)", the information processing device 1010 is "interrupted" in the work process Pr. Is determined. That is, in the work process Pr which is a manual process, when the operation period Da does not overlap with the stay period Do, the image analysis unit 171 determines that the work process Pr is interrupted in the operation period Da.

When "device 40 is stopped (device 40 state: 0)" and "worker Pe is staying (person state: 1)", the information processing device 1010 means that the worker Pe corresponds to the work process Pr. It is determined that the "work" or "setup" of the work to be performed is performed. That is, in the work process Pr which is a manual process, when the stay period Do does not overlap with the operation period Da, the image analysis unit 171 indicates that the worker Pe in the stay period Do is the work Op itself corresponding to the work process Pr, or , It is determined that the setup for the work Op is being performed.

When "device 40 is stopped (device 40 state: 0)" and "worker Pe is absent (person state: 0)", the information processing device 1010 has a work process Pr of "Null". That is, it is determined that the work process Pr has not been carried out. That is, the image analysis unit 171 determines that the work process Pr, which is a manual process, is not executed during the period other than the stay period Do and the operation period Da.

(2. Analysis for automation process)
When the operation Ac executed by the device 40 when the work process Pr is executed is an "automated operation", that is, when the work process Pr is an automated process, information processing is performed as shown in FIG. 18B. Device 1010 performs the following analysis.

That is, when "device 40 is operating (device 40 state: 1)" and "worker Pe is staying (human state: 1)", the information processing device 1010 (particularly, the image analysis unit 171) The worker Pe determines that it is "wasteful". That is, in the work process Pr which is an automation process, when the stay period Do and the operation period Da overlap, the image analysis unit 171 determines that the worker Pe is wasted in the stay period Do.

When "device 40 is operating (device 40 state: 1)" and "worker Pe is absent (person state: 0)", the information processing device 1010 is set to the work process Pr such as "feed operation". It is determined that the corresponding "normal operation Ac" is being executed. That is, in the work process Pr which is an automation process, when the operation period Da does not overlap with the stay period Do, the image analysis unit 171 corresponds to the work process Pr such as "feed operation" in the device 40 in the operation period Da. It is determined that the operation Ac is being executed.

When "device 40 is stopped (device 40 state: 0)" and "worker Pe is staying (person state: 1)", the information processing device 1010 means that the worker Pe corresponds to the work process Pr. It is determined that the work to be performed is "set up". That is, in the work process Pr which is an automation process, when the stay period Do does not overlap with the operation period Da, the image analysis unit 171 indicates that the worker Pe in the stay period Do is for the work Op corresponding to the work process Pr. It is determined that the setup is being performed.

When "device 40 is stopped (device 40 state: 0)" and "worker Pe is absent (person state: 0)", the information processing device 1010 has a work process Pr of "Null". That is, it is determined that no work has been performed on the work process Pr. That is, the image analysis unit 171 determines that the work process Pr, which is an automation process, is not executed during the period other than the stay period Do and the operation period Da.

As described with reference to FIG. 18, the information processing apparatus 1010 (particularly, the image analysis unit 171) visualizes the waste process, that is, the wasteful stay period Do of the worker Pe. In particular, the information processing apparatus 1010 can perform highly accurate and detailed analysis of the work process Pr by removing noise from the flow line analysis result by the correction filter 210 and complementing the flow line analysis result. it can.

§4. Modification example (Integrating the information processing device and PLC)
So far, as shown in FIG. 2, an example in which the information processing device 10 (or the information processing device 1010) and the PLC 20 are connected to each other so as to be communicable with each other via the control network 50 has been described. However, it is not essential to connect the information processing device 10 (or the information processing device 1010) and the PLC 20 via the control network 50.

For example, the information processing device 10 (or information processing device 1010) and the PLC 20 may be connected to each other so as to be able to communicate with each other via an internal bus, and the information processing device 10 (or the information processing device 1010) and the PLC 20 may be connected to each other. May be integrated. That is, the information processing device 10 (or information processing device 1010) may be configured as an IPC (Industrial PC) in which the information processing device 10 (or information processing device 1010) and the PLC 20 are integrated.

(Analysis of process information by PLC)
In the description so far, the determination for the operation result La (particularly the operation Ac) included in the process information (for example, the presence / absence of an error and the determination of the presence / absence of execution completion within the range of the standard operation time Tta) is determined by PLC20. The information processing device 10 (or the information processing device 1010) that received the process information from For example, in the description so far, it is determined whether the operation period Da of the operation Ac (m) is within the range from the standard lower limit operation time lower limit Ttal (m) to the standard upper limit operation time upper limit Ttah (m). , Information processing device 10 (or information processing device 1010).

Specifically, in the examples described so far, when the operation determination unit 130 included in the information processing device 10 (or the information processing device 1010) executes the work process Pr from the process information, particularly from the operation result La. The operation start time Tms, the operation completion time Tme, and the operation period Da of the operation Ac executed by the device 40 were specified. Then, the operation determination unit 130 included in the information processing device 10 (or the information processing device 1010) uses the specified operation start time Tms, operation completion time Tme, operation period Da, and operation reference Sa corresponding to the operation Ac. , The judgment for the actual operation Ac was executed.

However, the above-mentioned determination for the operation result La (particularly, the operation Ac) included in the process information may be executed by the PLC 20, and the PLC 20 includes the result of the above determination in the process information or the process information. Alternatively, the information may be transmitted to the information processing device 10 (or the information processing device 1010). For example, the operation determination unit 130 is provided in the PLC 20, and the information processing device 10 (or the information processing device 1010) acquires the result of the above-mentioned determination for the operation result La (particularly, the operation Ac) included in the process information from the PLC 20. You may. The information processing apparatus 10 (or the information processing apparatus 1010) may be able to extract the partial imaging data OD from the basic imaging data BI by using the result of the above-mentioned determination for the operation result La (particularly, the operation Ac) included in the process information. It is not essential that the information processing device 10 (or the information processing device 1010) executes the determination for the operation result La.

[Example of realization by software]
Functional blocks of the information processing device 10 and the information processing device 1010 (specifically, first acquisition unit 110, second acquisition unit 120, operation determination unit 130, extraction unit 150, transmission unit 160, image analysis unit 170, image analysis. The unit 171 and the correction filter 210) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit). May be good.

In the latter case, the information processing device 10 and the information processing device 1010 each have a CPU that executes instructions of a program that is software that realizes each function, and the above program and various data are readablely recorded by a computer (or CPU). It is equipped with a ROM (Read Only Memory) or a storage device (these are referred to as "recording media"), a RAM (Random Access Memory) for developing the above program, and the like. Then, the object of the present invention is achieved by the computer (or CPU) reading the program from the recording medium and executing the program. As the recording medium, a "non-temporary tangible medium", for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, the program may be supplied to the computer via an arbitrary transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

(Additional notes)
The data extraction device according to one aspect of the present invention includes a first acquisition unit that acquires basic data that is imaging data obtained by photographing the implementation status of a plurality of work processes, and an apparatus used for each execution of the plurality of work processes. The second acquisition unit that acquires the process information generated from the operation result indicating the content and result of the operation executed by the device, and whether or not the operation executed by the device satisfies a predetermined operation standard by using the process information. Extraction of the motion determination unit to be determined and the imaging data indicating the implementation status of the work process corresponding to the motion determined to satisfy the predetermined motion criteria by the motion determination unit as partial data from the basic data. The own device includes a unit and a device, and the own device is connected to a master in a master-slave control system in which the device is a slave via an internal bus, or via a control network that communicably connects the slave and the master. It is a device different from the device connected to the master, and the second acquisition unit acquires the information generated or collected by the master as the process information.

According to the above configuration, the data extraction device is a device connected to the master via an internal bus, or a device different from the device connected to the master via the control network, and is described below. Executes the processing of. That is, the data extraction device uses the process information generated or collected by the master to perform the work process corresponding to the operation determined to satisfy the predetermined operation standard from the basic data. The imaging data indicating the above is extracted.

That is, the data extraction device acquires the process information from the master via the internal bus or the control network. Then, by using the acquired process information, the data extraction device extracts desired imaging data as the partial data from the basic data without transmitting the basic data to the outside of the master-slave control system. ..

Therefore, the data extraction device efficiently uses the basic data by extracting the partial data from the basic data without imposing a communication load on the host system located outside the master-slave control system. It has the effect of being able to.

In the data extraction device according to one aspect of the present invention, the work performed by the operator during the execution of the work process is a standard work performed during the execution of the work process by image analysis of the basic data. The work process further includes an image analysis unit that determines whether or not the data is, and the extraction unit further corresponds to the work that is determined by the image analysis unit to be non-standard work from the basic data. The imaging data indicating the implementation status of the above may be extracted as the partial data.

According to the above configuration, in the data extraction device, the work performed by the worker at the time of carrying out the work process is performed at the time of carrying out the work process by image analysis of the basic data. Determine if it is a standard task. Then, when the data extraction device determines that the work is not the standard work, the data extraction device obtains imaging data indicating the implementation status of the work process corresponding to the work determined to be not the standard work. Extract from the data. That is, the data extraction device extracts desired imaging data from the basic data as the partial data by image analysis of the basic data without transmitting the basic data to the outside of the master-slave control system.

The data extraction device according to one aspect of the present invention may further include a transmission unit that transmits the imaging data extracted as the partial data by the extraction unit to the outside of the master / slave control system.

According to the above configuration, the data extraction device transmits the imaging data extracted as the partial data from the basic data to the outside of the master / slave control system. Here, it is clear that the amount of imaging data extracted as the partial data is smaller than the amount of basic data.

Therefore, the data extraction device has the effect of enabling efficient use of the basic data in the higher-level system while suppressing the communication load on the higher-level system located outside the master-slave control system. ..

In the data extraction device according to one aspect of the present invention, the operation determination unit said that (A) the operation result that the device detected an abnormality, and (B) the operation executed by the device had an abnormality. It may be determined that the operation corresponding to at least one of the operation results satisfies the predetermined operation standard.

According to the above configuration, the data extraction device corresponds to at least one of (A) an operation result of detecting an abnormality and (B) an operation result of having an abnormality in the executed operation. Determines that the predetermined operating standard is satisfied.

Therefore, the data extraction device is a work process corresponding to at least one of (A) an operation result of detecting an abnormality and (B) an operation result of having an abnormality in the executed operation from the basic data. It has the effect that the imaging data indicating the implementation status can be extracted as the partial data.

In the data extraction device according to one aspect of the present invention, even if the operation determination unit determines that the operation corresponding to the operation result that the operation is not executed in the order to be executed satisfies the predetermined operation standard. Good.

According to the above configuration, the data extraction device determines that the operation corresponding to the operation result that the data extraction device was not executed in the order to be executed satisfies the predetermined operation standard.

Therefore, the data extraction device can extract imaging data indicating the execution status of the work process corresponding to the operation that was not executed in the order to be executed from the basic data as the partial data. It works. For example, the data extraction device extracts imaging data indicating the execution status of the work process executed immediately before the work process corresponding to the operation not executed in the order to be executed as the partial data. Further, for example, the data extraction device extracts imaging data indicating the execution status of the work process executed immediately after the work process corresponding to the operation not executed in the order to be executed as the partial data. ..

In the data extraction device according to one aspect of the present invention, the operation determination unit determines in advance that the time actually required to complete the execution of the operation executed by the device is the time required to complete the execution. If it is not within the set standard operating time range, it may be determined that the operation satisfies the predetermined operating standard.

According to the above configuration, the data extraction device determines that the operation in which the time actually required to complete the execution is not within the range of the standard operation time satisfies the predetermined operation standard. Specifically, the data extraction device "the time actually required to complete the execution is less than the standard lower limit operating time preset as the lower limit of the time required to complete the execution." It is determined that the operation satisfies the predetermined operation standard. In addition, the data extraction device "exceeds the standard upper limit operating time preset as the upper limit of the time required to complete the execution of the time actually required to complete the execution." It is determined that the predetermined operation standard is satisfied.

Therefore, from the basic data, the data extraction device obtains imaging data indicating the execution status of the work process corresponding to the operation in which the time required to actually complete the execution is not within the range of the standard operation time. It has the effect that it can be extracted as the partial data. Specifically, the data extraction device uses the basic data to determine the implementation status of the work process corresponding to the operation, "the time actually required to complete the execution is less than the standard lower limit operation time". The shown imaging data can be extracted as the partial data. In addition, the data extraction device uses the basic data to show imaging data indicating the implementation status of the work process corresponding to the operation in which "the time actually required to complete the execution exceeds the standard upper limit operation time". Can be extracted as the partial data.

In the data extraction device according to one aspect of the present invention, for each work process, (1) the period during which the worker stayed in the area where the work related to the execution of the work process was performed, and (2) the work process. The external device may display information indicating at least one of the distribution and the total value in a predetermined period for at least one of the periods during which the operation according to the execution of the above is performed by the device.

According to the above configuration, in the data extraction device, for each work process, (1) the period during which the worker stayed in the area where the work related to the execution of the work process was performed (“stay period”), and. , (2) Display on the external device information indicating at least one of the distribution and the total value in a predetermined period for at least one of the periods (“operation period”) in which the operation related to the execution of the work process is executed by the device. Let me.

Therefore, the data extraction device can provide the user with information indicating at least one of the distribution and the total value in a predetermined period for at least one of the stay period and the operation period for each of the work processes. Play the effect of.

Here, the data extraction device may display information indicating the distribution in the predetermined period on the external device in the form of a box plot or a histogram. In addition, the data extraction device has at least one of the distribution and the total value in the predetermined period for at least one of the stay period and the operation period for each work process Pr required to produce one predetermined product. The information indicating the above may be displayed on the external device. At least one of the above-mentioned "predetermined period" and "predetermined product" may be selected by the user.

The data extraction device according to one aspect of the present invention includes (1) the period of stay in the area where the work related to the execution of the work process is performed for each worker, and (2) the work for each device. For at least one of the periods during which the operation related to the execution of the process was executed, information indicating at least one of the distribution and the total value in the predetermined period may be displayed on the external device.

According to the above configuration, the data extraction device (1) stays in the area where the work related to the execution of the work process is performed for each worker (“stay period”), and (2). For at least one of the periods during which the operation related to the execution of the work process (“operation period”) is executed for each device, information indicating at least one of the distribution and the total value in the predetermined period is displayed on the external device. ..

Therefore, the data extraction device provides the user with information indicating at least one of the distribution and the total value in a predetermined period for at least one of the stay period for each worker and the operation period for each device. It has the effect of being able to.

Here, the data extraction device may display information indicating the distribution in the predetermined period on the external device in the form of a box plot or a histogram. Further, the data extraction device is a worker group to which the plurality of the workers belong, and for each of the plurality of the workers belonging to the worker group selected by the user, during the predetermined period of the stay period. Information indicating at least one of the distribution and the total value may be displayed on the external device. Further, the data extraction device is a device group to which the plurality of devices belong, and for each of the plurality of devices belonging to the device group selected by the user, the distribution and total value of the operation period in a predetermined period. Information indicating at least one of the above may be displayed on the external device.

The data extraction device according to one aspect of the present invention includes, for each work process, the total period during which the worker stayed in the area where the work related to the execution of the work process was performed in a predetermined period. , (2) At least one of the total periods during which the operation related to the execution of the work process is performed by the device is the total time during which the work can be performed or the operation can be performed in the predetermined period. Information indicating the ratio to (total, that is, "total time") may be displayed on the external device.

According to the above configuration, in the data extraction device, the period during which the worker stayed in the area where (1) the work related to the execution of the work process was performed for each work process in a predetermined period (“stay”). The work is performed in the predetermined period for at least one of the total of (2) the total of (2) the period in which the operation related to the execution of the work process is executed by the device (“operation period”). The external device is made to display information indicating the ratio of the time obtained or the operation can be executed to the total time (“total time”).

Therefore, the data extraction device provides the user with information indicating the ratio of at least one of the total stay period and the total operation period for each work process in the predetermined period to the total time. It has the effect of being able to. The data extraction device may display information indicating a ratio to the total time on the external device in the form of a bar graph.

The data extraction device according to one aspect of the present invention captures an image generated from the basic data in (1) an area where the worker performs the work or an area where a passage through which the worker passes is imaged. The detection region is divided into a detection region (2) and a non-detection region which is a region other than the detection region, and the detection in the non-detection region is deleted from the result of the flow line analysis for detecting the worker from the captured image. A correction unit may be further provided.

According to the above configuration, the data extraction device divides the captured image into the detection region and the non-detection region, and deletes the detection in the non-detection region from the result of the flow line analysis on the captured image. It is further provided with a correction unit for processing. That is, the data extraction device deletes the noise "detection in the non-detection region" from the result of the flow line analysis for the captured image.

Therefore, the data extraction device realizes highly accurate detection of the worker by removing noise from the result of the flow line analysis of the captured image, that is, realizes highly accurate analysis of the work process. It has the effect of being able to.

In the data extraction device according to one aspect of the present invention, the first acquisition unit acquires the basic data which is moving image data obtained by photographing the implementation status of the plurality of work processes, and in the flow line analysis for the basic data. When the worker is not detected in the detection area of a certain frame, the correction unit uses the detection result of the worker in a predetermined number of frames immediately before the certain frame. The result of the flow line analysis for the frame may be complemented.

According to the above configuration, the data extraction device executes a flow line analysis on the basic data which is moving image data obtained by photographing the execution status of the plurality of work processes. Then, when the worker is not detected in the detection area of a certain frame, the data extraction device uses the detection result of the worker in a predetermined number of frames immediately before the certain frame to obtain the above. Complements the result of flow line analysis for a certain frame.

For example, if the worker is not detected in the detection area of the certain frame, but the worker is detected in the frame closest to the certain frame, the data extraction device is said to be. The following complementation is performed on the result of the flow line analysis for the frame. That is, the data extraction device complements the result of the flow line analysis for the certain frame with the detection result of the latest frame, and detects the worker in the detection area of the certain frame. Generate detection results. Further, the data extraction device continuously detects the worker in the detection area in a plurality of frames immediately before the certain frame, but the worker detects the worker in the detection area of the certain frame. If not, the above complement may be performed.

Here, the basic data is moving image data obtained by photographing the implementation status of the plurality of work processes, and the detection area is an area where the worker performs the work or the worker passes through the detection area. The area where the passage is imaged. Therefore, if the worker is detected in the detection area in a predetermined number of frames immediately before the certain frame by the flow line analysis for the basic data, the detection area is also in the certain frame. The worker is usually detected. This is because it is usually difficult to imagine a situation in which the worker suddenly disappears from the area where the worker performs the work and from the passage.

Therefore, when the worker is detected in the detection area of a predetermined number of frames immediately before the certain frame, but the worker is not detected in the detection area of the certain frame, the worker is not detected. It is highly possible that the result of the flow line analysis of the frame contains an error. Therefore, the data extraction device complements the result of the flow line analysis of the certain frame with the detection result of a predetermined number of frames immediately before the certain frame.

That is, when there is a high possibility that the result of the flow line analysis of the certain frame is incorrect, the data extraction device obtains the result of the flow line analysis of the certain frame in a predetermined number of frames immediately before. The certain frame can be analyzed with high accuracy by complementing with the detection result. Therefore, the data extraction device realizes highly accurate detection of the worker by correcting the result of the flow line analysis of the certain frame, that is, realizes highly accurate analysis of the work process. It has the effect of being able to.

In the data extraction device according to one aspect of the present invention, when a plurality of bounding boxes (Bounding Boxes) in which the distance between the centers is equal to or less than a predetermined value are installed in the flow line analysis of the captured image, the correction unit May unify the plurality of bounding boxes into the bounding box having the largest area.

According to the above configuration, when a plurality of bounding boxes in which the distance between the centers is equal to or less than a predetermined value are installed in the flow line analysis of the captured image, the data extraction device uses the plurality of bounding boxes. , Unify to the bounding box with the largest area.

Here, in the flow line analysis for the captured image, when a plurality of bounding boxes in which the distance between the centers is equal to or less than a predetermined value are installed, one said worker erroneously detects as the plurality of said workers. It may have been. Therefore, when a plurality of bounding boxes having a distance of each center of a predetermined value or less are installed, the data extraction device unifies the plurality of bounding boxes into the bounding box having the largest area.

That is, when there is a high possibility that one worker is erroneously detected as a plurality of workers in the flow line analysis of the captured image, the data extraction device has only the bounding box having the largest area. The number of the detected workers is set to one, leaving the above. Therefore, the data extraction device realizes highly accurate detection of the worker by modifying the number of bounding boxes installed in the flow line analysis for the captured image, that is, highly accurate analysis of the work process. It has the effect of being able to realize.

The control method of the data extraction device according to one aspect of the present invention includes a first acquisition step of acquiring basic data which is imaging data obtained by photographing the implementation status of a plurality of work processes, and each of the plurality of work processes. The second acquisition step of acquiring the process information generated from the operation result indicating the content and result of the operation performed by the device used, and the operation performed by the device using the process information set a predetermined operation standard. The basic data includes an operation determination step for determining whether or not the operation is satisfied, and imaging data indicating the implementation status of the work process corresponding to the operation determined to satisfy the predetermined operation standard in the operation determination step as partial data. The data extraction device includes an extraction step of extracting from, and the data extraction device enables communication between a device connected to a master in a master-slave control system having the device as a slave via an internal bus, or the slave and the master. It is a device different from the device connected to the master via the connected control network, and the second acquisition step acquires the information generated or collected by the master as the process information.

According to the above configuration, the control method is the data extraction device connected to the master via an internal bus, or the data extraction device connected to the master via the control network and different from the device. It is a control method of, and executes the following processing. That is, the control method uses the process information generated or collected by the master to determine the implementation status of the work process corresponding to the operation determined to satisfy the predetermined operation standard from the basic data. The imaged data shown is extracted.

That is, the control method is executed by the data extraction device that acquires the process information from the master via the internal bus or the control network. Then, in the control method, by using the acquired process information, desired imaging data is extracted as the partial data from the basic data without transmitting the basic data to the outside of the master / slave control system.

Therefore, in the control method, the basic data is efficiently used by extracting the partial data from the basic data without imposing a communication load on the host system located outside the master-slave control system. It has the effect of being able to.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

10 Information processing device (data extraction device)
20 PLC (master)
40 equipment (equipment, slave)
50 Control network 110 1st acquisition unit 120 2nd acquisition unit 130 Operation judgment unit 150 Extraction unit 160 Transmission unit 170 Image analysis unit 171 Image analysis unit 210 Correction filter (correction unit)
1010 Information processing device (data extraction device)
Ac operation Aa (-1) Analysis target area (non-detection area)
Aa (0) Analysis target area (detection area)
Aa (1) Analysis target area (detection area)
Aa (2) Analysis target area (detection area)
Aa (3) Analysis target area (detection area)
Aa (4) Analysis target area (detection area)
Aa (5) Analysis target area (detection area)
BB Bounding Box BI Basic Imaging Data (Basic Data)
Da operating period (time actually required to complete execution)
La operation result OD partial imaging data (partial data)
Pr work process Sa operation standard Tta standard operation time S110 2nd acquisition step S120 1st acquisition step S130 operation judgment step S140 operation judgment step S190 operation judgment step S170 extraction step

Claims

The first acquisition unit that acquires basic data, which is imaging data that captures the implementation status of multiple work processes,
A second acquisition unit that acquires process information generated from an operation result indicating the content and result of the operation executed by the equipment used for each execution of the plurality of work processes.
Using the process information, an operation determination unit that determines whether the operation performed by the device satisfies a predetermined operation standard, and an operation determination unit.
An extraction unit that extracts image data indicating the implementation status of the work process corresponding to the operation determined by the operation determination unit to satisfy the predetermined operation standard from the basic data as partial data.
With
The own device is connected to the master via an internal bus or a control network that communicably connects the slave and the master in a master-slave control system in which the device is a slave. , It is a device different from the above device,
The second acquisition unit is a data extraction device that acquires information generated or collected by the master as the process information.
An image analysis unit that determines whether or not the work performed by the worker during the execution of the work process is a standard work performed during the execution of the work process by image analysis of the basic data. Further prepare
The extraction unit further extracts image data indicating the implementation status of the work process corresponding to the work determined by the image analysis unit to be non-standard work from the basic data as the partial data. The data extraction device according to claim 1.
The data extraction device according to claim 1 or 2, further comprising a transmission unit that transmits the imaging data extracted as the partial data by the extraction unit to the outside of the master-slave control system.
The operation determination unit has the operation corresponding to at least one of (A) an operation result that the device has detected an abnormality and (B) an operation result that the operation executed by the device has an abnormality. The data extraction device according to any one of claims 1 to 3, which is determined to satisfy the predetermined operation standard.
The data according to any one of claims 1 to 4, wherein the operation determination unit determines that the operation corresponding to the operation result that the operation is not executed in the order to be executed satisfies the predetermined operation standard. Extractor.
The operation determination unit does not have the time actually required to complete the execution of the operation executed by the device within the standard operating time preset as the time required to complete the execution. The data extraction device according to any one of claims 1 to 5, wherein it is determined that the operation satisfies the predetermined operation standard.
For each work process
(1) The period during which the worker stayed in the area where the work related to the implementation of the work process was performed.
and,
(2) Claims 1 to 6 for causing an external device to display information indicating at least one of the distribution and the total value in a predetermined period for at least one of the periods during which the operation related to the execution of the work process is executed by the device. The data extraction device according to any one item.
(1) For each worker, the period of stay in the area where the work related to the implementation of the work process is performed, and
(2) Claim 1 for causing an external device to display information indicating at least one of the distribution and the total value in a predetermined period for at least one of the periods during which the operation related to the execution of the work process is executed for each of the devices. 7. The data extraction device according to any one of 7.
For each work process in a predetermined period,
(1) The total period during which the worker stayed in the area where the work related to the implementation of the work process is performed,
and,
(2) At least one of the total periods during which the operation related to the execution of the work process is performed by the device is the entire time during which the work can be performed or the operation can be performed in the predetermined period. The data extraction device according to any one of claims 1 to 8, wherein information indicating the proportion of the data is displayed on an external device.
The captured image generated from the basic data
(1) An area in which the worker performs the work, or a detection area in which the passage through which the worker passes is imaged.
(2) It is divided into a non-detection area, which is an area other than the detection area.
The data extraction device according to claim 2, further comprising a correction unit for deleting the detection in the non-detection region of the result of the flow line analysis for detecting the worker from the captured image.
The first acquisition unit acquires the basic data, which is moving image data obtained by photographing the implementation status of the plurality of work processes.
When the worker is not detected in the detection area of a certain frame in the flow line analysis for the basic data, the correction unit detects the worker in a predetermined number of frames immediately before the certain frame. The data extraction device according to claim 10, wherein the result is used to supplement the result of the flow line analysis for a certain frame.
In the flow line analysis for the captured image, when a plurality of bounding boxes in which the distance between the centers is equal to or less than a predetermined value are installed,
The data extraction device according to claim 10 or 11, wherein the correction unit unifies the plurality of bounding boxes into a bounding box having the largest area.
It is a control method of the data extraction device.
The first acquisition step to acquire the basic data, which is the imaging data obtained by photographing the implementation status of multiple work processes, and
A second acquisition step of acquiring process information generated from an operation result indicating the content and result of the operation executed by the equipment used for each execution of the plurality of work processes, and
Using the process information, an operation determination step for determining whether the operation performed by the device satisfies a predetermined operation standard, and an operation determination step.
An extraction step of extracting from the basic data as partial data the imaging data indicating the implementation status of the work process corresponding to the operation determined to satisfy the predetermined operation standard in the operation determination step.
Including
The data extraction device is connected to the master in a master-slave control system having the device as a slave via an internal bus, or to the master via a control network that communicably connects the slave and the master. It is a connected device different from the above device,
The second acquisition step is a control method for acquiring information generated or collected by the master as the process information.
An information processing program for operating a computer as the data extraction device according to any one of claims 1 to 12, and for operating the computer as each part.
A computer-readable recording medium on which the information processing program according to claim 14 is recorded.